The only way to be safe from these issues is if you run the models locally, yourself.
I don’t think it’s really necessary to run the models locally unless you’re worried about privacy. A rented GPU from somewhere like RunPod is probably fine. They have no incentive to quantize your models (why would they want you to use cheaper hardware?), and it would be hard for them to do even if they wanted to since you’re running arbitrary code.
I’m actually in the middle of running an experiment using GPT-OSS-20B on rented A40′s from RunPod (not using API access for exactly the reasons in your post), and it’s a little bit more annoying but not that expensive[1], and I would be pretty surprised if they were messing with the code I’m running. You can even parallelize via SkyPilot, although RunPod’s quality control is pretty terrible so you need to make sure the machine actually booted and has a GPU (but it’s obvious when RunPod machines break—you’re not going to get seemingly good results, the code just won’t run at all).
The conversation can’t be shared because it used Deep Research mode, but:
For batch workloads (thousands of prompts, spin up → run → tear down, no interactive debugging), renting GPUs costs roughly the same per token as the API, sometimes less. GPT-OSS-120B on a single H100 runs ~$0.14–0.46/M output vs OpenRouter’s ~$0.10–0.19/M. DeepSeek-R1 in reference FP8 needs 8×H200 (685GB of weights won’t fit in 8×H100′s 640GB) and lands near $3/M output vs OpenRouter’s $2.50/M — parity. On B200 both models are plausibly several times cheaper than the API.
Three conditions on that:
You have to saturate the GPU. Use an offline batch API (LLM.generate() over the whole prompt list), not an HTTP server with a rate-limited client. Batch-of-1 throughput is 10–100× worse and blows up the per-token cost entirely.
Startup overhead scales with the model. For a 63GB model it’s a few dollars; for DeepSeek’s 685GB it’s more like $10–25 of billed time pulling weights, which is a real fraction of a short run. Use hf_transfer, or a network volume if you’re iterating.
Claude is a little bit optimistic here because RunPod’s quality control is so bad, but I think it’s within the right order of magnitude.
When I wrote “locally, yourself” I was thinking of either having or renting GPUs to run your models. I should have written “on hardware you control.”
For my use case, I sadly care a lot about “interactive debugging”. I’m iterating on prompts, control protocols, and models, so if I was running on rented GPUs they would be idling for a long time between experiments.
I don’t know too much about the details of running models on rented GPUs, but after looking into OpenRouter for the post, I am inclined to learn. Thanks for sharing!
I don’t think it’s really necessary to run the models locally unless you’re worried about privacy. A rented GPU from somewhere like RunPod is probably fine. They have no incentive to quantize your models (why would they want you to use cheaper hardware?), and it would be hard for them to do even if they wanted to since you’re running arbitrary code.
I’m actually in the middle of running an experiment using GPT-OSS-20B on rented A40′s from RunPod (not using API access for exactly the reasons in your post), and it’s a little bit more annoying but not that expensive[1], and I would be pretty surprised if they were messing with the code I’m running. You can even parallelize via SkyPilot, although RunPod’s quality control is pretty terrible so you need to make sure the machine actually booted and has a GPU (but it’s obvious when RunPod machines break—you’re not going to get seemingly good results, the code just won’t run at all).
The conversation can’t be shared because it used Deep Research mode, but:
For batch workloads (thousands of prompts, spin up → run → tear down, no interactive debugging), renting GPUs costs roughly the same per token as the API, sometimes less. GPT-OSS-120B on a single H100 runs ~$0.14–0.46/M output vs OpenRouter’s ~$0.10–0.19/M. DeepSeek-R1 in reference FP8 needs 8×H200 (685GB of weights won’t fit in 8×H100′s 640GB) and lands near $3/M output vs OpenRouter’s $2.50/M — parity. On B200 both models are plausibly several times cheaper than the API.
Three conditions on that:
You have to saturate the GPU. Use an offline batch API (
LLM.generate()over the whole prompt list), not an HTTP server with a rate-limited client. Batch-of-1 throughput is 10–100× worse and blows up the per-token cost entirely.Startup overhead scales with the model. For a 63GB model it’s a few dollars; for DeepSeek’s 685GB it’s more like $10–25 of billed time pulling weights, which is a real fraction of a short run. Use
hf_transfer, or a network volume if you’re iterating.Claude is a little bit optimistic here because RunPod’s quality control is so bad, but I think it’s within the right order of magnitude.
You’re right, sorry.
When I wrote “locally, yourself” I was thinking of either having or renting GPUs to run your models. I should have written “on hardware you control.”
For my use case, I sadly care a lot about “interactive debugging”. I’m iterating on prompts, control protocols, and models, so if I was running on rented GPUs they would be idling for a long time between experiments.
I don’t know too much about the details of running models on rented GPUs, but after looking into OpenRouter for the post, I am inclined to learn. Thanks for sharing!