Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

  • SuspiciousCarrot78@aussie.zoneOP
    link
    fedilink
    English
    arrow-up
    1
    ·
    21 hours ago

    Yeah. Though I think theres a new strix out soon (Medusa? Gorgon? Something like that).

    Its a bit like my P40. On paper, it has 24GB. But that 24gb is capped at 400GB/s and the ai compute is what…Pascal era?

    AI = Good, fast, cheap - pick 2

    • robber@lemmy.ml
      link
      fedilink
      English
      arrow-up
      1
      ·
      3 hours ago

      Well compared to the strix, 400GB/s is not that bad, I think with fast system RAM and expert offloading you could squeeze quite something out of it when running stuff in the 100b-a10b regions.

      Your bigger problem is going to be future software support.