I just found https://www.arliai.com/ who offer LLM inference for quite cheap. Without rate-limits and unlimited token generation. No-logging policy and they have an OpenAI compatible API.

I’ve been using runpod.io previously but that’s a whole different service as they sell compute and the customers have to build their own Docker images and run them in their cloud, by the hour/second.

Should I switch to ArliAI? Does anyone have some experience with them? Or can recommend another nice inference service? I still refuse to pay $1.000 for a GPU and then also pay for electricity when I can use some $5/month cloud service and it’d last me 16 years before I reach the price of buying a decent GPU…

Edit: Saw their $5 tier only includes models up to 12B parameters, so I’m not sure anymore. For larger models I’d need to pay close to what other inference services cost.

  • tpWinthropeIII@lemmy.worldEnglish
    2·
    23 hours ago

    I know that people are using P40 and P100 GPUs. These are outdated but still work with some software stacks / applications. The P40 GPU, once very cheap for the amount of VRAM, is no longer as cheap as it was probably because folks have been picking them up for inference.

    I’m getting a lot done with an NVidia GTX 1080 which only has 8GB VRAM. I can run a quant of dolphin Mixtral 7x8B and it works well enough. It takes minutes to load, almost too long for me, but after that I get 3-5 TPS with some acceptable delay between questions.

    I can even run Miqu quants at 2 or 3 bits. It’s super smart even at these low quant levels.

    llama 3.1 8B runs great with this 1080 8BG GPU at 4_K_M and also 5 or 6_K_M. I believe I can run gemma 9B f16 at 8 bpw.

    • hendrik@palaver.p3x.deOPEnglish
      1·
      9 hours ago

      Hmmh, I lately use mistral-nemo which is 12B parameters. Since I’m more a programmer than a gamer, I didn’t put a graphics card into my PC, and I believe it’s too old to accomodate any recent one. (older PCIe generation, only x8 ports) I’d have to replace everything. And then I might as well go for a Radeon RX 7900 XTS or something. That’s $1.000(?) but has 24GB of VRAM. I don’t think buying an entire PC and then going for an old GPU will make me happy. And thanks to llama.cpp I get about 2 tokens per second just on the CPU. It’d have to be a considerable step up to be worth it. And last time I checked even a P40 was like $300+ and it’s super old and unclear if it’ll continue to be supported in the major frameworks. I’m not sure. I still lean towards paying for cloud GPU compute.

      Thanks for the numbers on your setup. That certainly helps weighing my options. Maybe some of my friends have some upgrades planned and want to give me their older 8GB NVidia cards…