Hosted API vs renting a GPU for a 35B model
Updated · prices checked on that date
A 4-bit 35B-A3B model fits on one 48 GB GPU, so self-hosting is realistic. Whether it pays off depends on how many tokens you use every month.
Key points
- The 4-bit weights of a 35B-A3B model take about 23 GiB, so one 48 GB card is enough for a 64,000-token context.
- A rented L40S costs $0.79–$1.09 an hour on RunPod: about $577–$796 a month around the clock.
- At Dolphin LLM prices, $577 buys roughly 2.6 billion tokens a month in a typical 80/20 input/output mix.
- Self-hosting makes sense at steady, very high volume; below that the hosted API costs less and needs no upkeep.
What hardware the model needs
The official 4-bit (GPTQ-Int4) weights of Qwen3.5-35B-A3B total about 22.7 GiB. On a 48 GB card that leaves roughly 20 GB for the KV cache, which is enough for a 64,000-token context with a few parallel requests. Longer contexts or more parallel users need more memory or more cards.
Rental prices
| GPU | Where | Per hour | Per month, 24/7 |
|---|---|---|---|
| L40S 48 GB | RunPod Community / Secure | $0.79 / $1.09 | $577 / $796 |
| L40S 48 GB | Vast.ai (median of 4 offers) | $0.80 | $584 |
| RTX A6000 48 GB | RunPod Community / Secure | $0.33 / $0.59 | $241 / $431 |
| A100 80 GB | RunPod Secure | $1.79 | $1,307 |
| H100 80 GB | RunPod Secure (PCIe) | $2.89 | $2,110 |
Monthly figures are hourly price × 730 hours, without storage, traffic or the time the card sits idle.
Break-even
A typical mix of 80% input and 20% output tokens costs $0.22 per million tokens on Dolphin LLM. So the monthly price of a rented card buys:
| Monthly budget | Equals on Dolphin LLM | Per day |
|---|---|---|
| $241 (RTX A6000, Community) | about 1.1 billion tokens | about 36 million |
| $577 (L40S, Community) | about 2.6 billion tokens | about 87 million |
| $796 (L40S, Secure) | about 3.6 billion tokens | about 120 million |
If you use fewer tokens than this, the hosted API is cheaper. Above it, a rented card can win, provided it is busy most of the day.
Costs that are not on the price list
- Setup and updates: inference server, drivers, model downloads, monitoring.
- Idle time: a rented card costs the same at night when nobody sends requests.
- Peaks: one card serves a limited number of parallel requests; more users mean more cards.
Which one to choose
- Self-host if you have steady, very high volume, need full control of the stack or a context longer than 64,000 tokens.
- Use Dolphin LLM if your volume is below the break-even point, varies from day to day, or you do not want to run servers.
Sources
Prices change. Check each provider's page before you decide.
Questions and answers
Can I run the same model on my own computer?
Yes, with a GPU that has enough memory: the 4-bit weights take about 23 GiB, so a 48 GB card is comfortable and a 24 GB card is too small for useful context.
Is the hosted API slower than a local GPU?
It adds network latency on top of generation time. Generation speed depends on the load on the shared servers; for chat and most batch work the difference is small.