Not Pinning Your OpenRouter Provider Might Invalidate Your Research
Please share this with anyone doing AI research with 3rd party providers so that they can ensure their research won’t be corrupted.
When you ask OpenRouter[1] to give you tokens from a given model, OpenRouter sends your request to a random available provider.
OpenRouter providers have variable quality. Ensuring that your provider is high quality is really difficult.
There is precedent for an AI safety paper accepted to NeurIPS having its core results entirely overturned by these issues.
Researchers who wish to do research using OpenRouter or similar providers should take precautions to minimize the risks to their research,[2] though the current selection of providers is insufficient for ideal scientific reliability.
Replicators should test whether results hold up when these bugs are fixed.
Core AI Safety codebases should make fixes so that downstream users are able to implement best practices.
This is a tangent I took for a few days during the Pivotal AI Safety Research Fellowship. I’m doing an AI Control project mentored by Adam Kaufman and James Lucassen of Redwood Research and partnered with Aniruddh Pramod.[3] This post has been edited to make use of feedback from Tim Hua, though he has not endorsed this post or the edits.[4] Every claim in the main body of this post has been human-checked. Claims in the footnotes are clearly marked when they have not been human checked.
Update 2026-07-28: Tomas from OpenRouter has taken notice of this post, and this has led to some corrections (tweet thread):
At time of writing this post, I wrote that the OpenRouter’s “Auto Exacto” quality-check benchmark numbers were not visible to the public. This was based on some outdated documentation, which has now been updated. These benchmark numbers are now available for some, but not all, models. I expect OpenRouter will continue adding scores to the model pages.
OpenRouter is “working on the logprobs detection”, referring to testing which providers output logprobs which match a reference implementation, and they are working on a “verified provider” tier. I look forward to both of these!
OpenRouter Kills an AI Safety NeurIPS Paper
In R1 CoT illegibility revisited from April 2026, nostalgebraist revisited the findings in the NeurIPS 2025 paper Reasoning Models Sometimes Output Illegible Chains of Thought by Arun Jose. It turned out that even when using an inference provider with the same quantization (fp8, “as far as [nostalgebraist] could tell”), different providers serve different models with drastically different capabilities and chain-of-thought legibilities. Arun conceded:
“I think this is pretty convincing re your original argument that my results were contaminated by bad inference setups, and that correspondingly my paper doesn’t substantiate some of its claims and conclusions with good evidence.”
While trying to set up our research codebase for an AI Control project[5] so that it would be easy to perform many experiments and know what led to the results, I came across the curious fact that different OpenRouter model providers serve models at different quantization levels. Looking into it further, it turns out that OpenRouter has barely any quality control, and providers are free to reduce quality to achieve lower costs. This stunned me, since I had always assumed that if you ask for model X from OpenRouter, OpenRouter has to give you responses from model X. That is essentially not true. My colleagues at the Pivotal Fellowship were similarly surprised.
Background
OpenRouter’s providers can:
Use different quantizations
Use different inference backends.
Handle your request parameters differently.
Have different maximum input lengths
Be maliciously finetuned to push agendas or have hidden backdoors
Some people have discussed these issues before but there is evidently not enough awareness, since among the AI Safety I’ve talked to, around half did not know about these issues, and AI Safety codebases that report results from OpenRouter, including recent ones in active development (see my PR to Control Tower), open themselves up to data quality issues. Though OpenRouter describes this behavior in its documentation, most researchers are unaware it is potentially corrupting their results.
Prior work/writing on the subject.
This is just a selection. For more prior work see Claude’s Report.
“...the choice of backend alone can shift benchmark scores by up to 16.6 percentage points and induce high rates of output disagreement”
Epoch AI’s “Why benchmarking is hard”
“The selection of an appropriate provider has the biggest impact on model performance”
“Our results show both PTQ [post-training quantization] and QAT [quantization-aware training] can degrade safety alignment”
Open weight LLMs exhibit inconsistent performance across providers
Image from https://x.com/xlr8harder/status/1974859337387651125:
No Guarantees from Fixing a Provider
Although selecting a provider[6] cuts down on a major source of variation, it does not guarantee that each time you request the provider for a model, you get the same model. There is significant active discussion about whether first party providers like Anthropic and OpenAI have variable quality.[7] Sometimes model quality can vary because of bugs. See this Anthropic blog post: Research results on Claude from “Between August and early September” 2025 might have been on a buggy Claude (kudos to Anthropic for announcing this information, though!). Fixing a provider is still the best practice available when using 3rd party providers, but unfortunately it falls short of ideal.
No Guarantees From Large N
You might hope that provider-routing would introduce some noise but that as long as the researcher had a large enough sample size and reported the standard deviation, then the research results would still be meaningful/replicable. Unfortunately, this is not necessarily the case. OpenRouter does not assign providers purely randomly. For example, if a provider goes down, your requests will be sent to the remaining providers. There are no guarantees of performance or of randomness. Researchers should assume that OpenRouter behaves adversarially. It is possible that if you ask for GPT-OSS-120B on Monday to do your baseline run, you’ll get a dumb/quantized/truncated model, and if you ask for GPT-OSS-120B on Tuesday to do your experimental run, you’ll get a smarter model, and your intervention will seem to have helped.[8]
No Guarantees from “The Benchmark Didn’t Move”
OpenRouter has and is continuing to develop an “Auto Exacto” “...step that automatically optimizes provider ordering for all requests that include tools. It runs by default on every tool-calling request, requiring no configuration.” I consider it to be an insufficient measure.
Accuracy is Not All You Need find that “ the behavior of compressed models as visible to end users is often significantly different from the baseline model, even when accuracy is similar”
It only applies to tool-calling requests. If the research doesn’t involve tool-calling, then it doesn’t apply
OpenRouter says that Auto Exacto depends on quality data on benchmarks they are running, but that this information is not yet public (Update 2026-07-28: OpenRouter has been rolling this out over the past few months and the information is public for many models! My previous claim was based on outdated documentation which has now been updated.) . While this is a good step towards quality control, it is notably not the same as actually ensuring the models being provided are the same. If OpenRouter started checking that the logprobs are allclose to a trusted reference implementation, I would be appeased.[9] (Update 2026-07-28: they are working on this, too!)
In my opinion, Auto Exacto is not good enough for reliable, replicable, trustable science.
No Guarantees from “Choose the Most Expensive Provider”
It would be great if we could just select the most expensive endpoint to get the one which is most faithfully served. Unfortunately, that is not the case. We found that for `meta-llama/llama-3.3-70b-instruct`, originally in bf16, the most expensive endpoint is in fp8, a cheaper one exists in bf16, and the cheapest is once again in fp8. This also hints that even within a fixed quantization, price doesn’t perfectly track quality.
No Guarantees from “Set a floor”
Some research codebases we reviewed set a quantization floor. Although this can protect them from the most lossily-compressed int4 models, it doesn’t protect them from the great variability which can exist within a higher quantization level.
When is Default OpenRouter Fine?
The model only has one provider (e.g. proprietary models from OpenAI or Anthropic). Version drift is still a risk.
Exploratory work where significant output variance doesn’t matter much
You want a cheap, high-throughput model with only minimal quality control
If you are measuring capabilities, comparing models, harnesses, or setups, measuring subtle propensities or safety behaviors, generating training data, or need reproducibility, default OpenRouter is not fine.
Developing Best Practices
Even with all of this, there is no way to actually be able to rely on providers. But researchers have limited alternative options, and self-hosting is too expensive and cumbersome for most research.
Grab my Claude Skill to have Claude handle it
Pin the endpoint. Although this isn’t a guarantee, making two requests to the same model of the same provider on the same day probably gives you the same model both times… probably.
If the provider is down, make switching to a new provider a conscious choice. You’d be testing a new organism, and comparisons with the old provider may or may not hold.[10]
Pin the quantization, just in case the provider decides to switch their quantization
Record the provider that served the response, and include in your paper/codebase the same way you’d report temperature.
Set allow_fallbacks to false.
Ensure that the provider’s context limits work for your use case, or you may face truncation.
Consider doing all of your experiments in a short period of time. This eliminates the chances that a provider will change how they function between experiments.
Set “require_parameters: true” so it is less likely the provider ignores your parameters (temperature, top_p, etc)
Remember to set data_collection if you’re doing something you’d prefer to stay private.
Setting ‘seed’ doesn’t guarantee determinism, sadly. This isn’t really the providers’ fault, since ensuring determinism is really hard. Keep this in mind.
Consider benchmarking your providers at the start of your research, to ensure quality. Though this isn’t really sufficient because Accuracy is Not All You Need.
Spotting Fingerprints
Nostalgebraist’s experience with different model providers allowed him to see signs of a degraded model in Arun Jose’s CoT legibility results. Identifying the effects of provider degradation on a model is not always possible for a few reasons:
Most of us aren’t Nostalgebraist
The idea behind techniques like quantization are exactly that they allow the provider to degrade quality without users noticing
If a paper publishes aggregated results without transcripts or metadata logs, there is much less information to indicate a bad inference[11]
If you want to attempt detective work, there are some things you could attempt. If there are transcripts available, you can check for token soup, special tokens where they don’t belong, and bad formatting (eg for reasoning tokens). There might be subtle disconnects between what the researchers say they did (eg set a temperature) and what they actually got (maybe the model provider didn’t respect the temperature parameter. Did the temperature impact the outputs as expected?).
The absence of visible fingerprints of bad inference is very weak evidence that the inference was done properly, and this type of detective work should not be relied upon.[12]
Highlighting Some Impacted Work
Here are some highlights, linked to the line in which a model is called without a provider.[13]
Redwood Research’s BashArena and LinuxArena Control Tower[14]
Should The Community Do Something About This?
Fixing Repos
Some important repos upstream of AI Safety research such as Redwood’s BashArena and Control Tower and others do not support best practices. Maintainers of these repositories should update them ASAP. I’ve already made a PR to Control Tower.
Replicating Research
Like nostalgebraist, we should work to identify which important work has the potential for its results to have been affected by this bug. Claude Fable or GPT-Sol could probably get a long way to fixing and replicating these results autonomously, and I encourage someone to do so (if you are working on OpenRouter-reliability replications please make a GitHub Issue on my Repo to coordinate!).
A New Org?
Even with all of these workarounds, we still have to place a lot of trust in the model providers which they frankly haven’t earned. It is impossible to rule out that even with all of my suggested precautions in place, an individual provider, when asked for tokens, routes some requests to one model and some to another. There is no FDA or SEC for model providers that ensure that their marketing claims are true.[15][16] A provider can change how they serve their models and not give any notice or warning. Imagine the confusion and sunk time that would occur if two chemicals with the same chemical formula had radically different properties, only one of which explodes.
The only way to be safe from these issues is if you run the models locally, yourself (edit: I should have said “on hardware you control”, rather than “locally”). Then you can report your setup for reproducibility and criticism. But that’s a huge tradeoff. OpenRouter-provided models are usually much cheaper and easier to work with than having to manage your own infrastructure. They also allow for much greater parallelism, which is invaluable for quickly running most research (e.g. evaluations and control research).
The AI Safety Community should consider creating an organization to provide standardized access to models. Researchers should be able to call on it to provide the same model, provided the same way, every time it is called (unless explicitly requested by the researcher). Until the AI Safety scientific community figures out best practices and sets up good shared infrastructure, we will have difficulty escaping our ignorance.
- ^
Although I primarily looked into OpenRouter, I am under the impression that other multi-provider routers such as Requesty, Vercel AI Gateway, Unify, and Martian suffer from the same issue. Single-provider hosts don’t have switching and avoid some, but not all, of the problems, making them similar to “pinning a provider”.
- ^
Other things to keep in mind: data_collection defaults to ‘allow’, so be careful before sending proprietary information in requests. And keep an eye on which providers include caching for decreased cost and increased speed. This is not the focus of this post, but it is worth mentioning.
- ^
Thank you to Ethan Nguyen for giving me some feedback on this post.
- ^
If you want to see a mess of Claude’s non-human-checked notes, they are at this GitHub repo.
- ^
The project is on extending AI Control protocols with “interrogations.” That means things like asking the untrusted model follow-up questions about why it took suspicious actions and making monitoring or control decisions informed by the answers. In future development, we plan to investigate more complex but similar protocols such as debate.
- ^
I had Claude look through a large number of model providers and figure out the extent to which they report what they serve, how it has changed over time, and their notice policy for changes in the future. According to Claude, all 10 audited had gaps, though I didn’t check these results closely, and Tim Hua noticed that Claude falsely claims Anthropic has “No written guarantee that weights behind a dated ID are fixed” when the docs say “Anthropic does not update the weights or configuration of an existing model ID.” You can view the (somewhat fishy) information in this Claude Artifact.
- ^
I would like to be able to dismiss this as a conspiracy, if anyone has insight into this question!
- ^
Nostalgebraist’s work is great. In his logs we can see that runs of GPQA use multiple providers of varying quality for QwQ-32B. From a single config, the provider mix ratios range from 90 Nebius + 10 DeepInfra for one run to 85 DeepInfra + 15 Nebius for another!
- ^
I wonder what OpenRouter’s incentives are. To what extent is OpenRouter aligned with users, and in particular, with researchers? I encourage economists to study and think about this.
- ^
I spend most of my time begging Claude “please please please fail loudly don’t secretly hide bugs”. If your provider is down, you should know.
- ^
Recording the providers can be really helpful here. I spent a little bit of time trying to find particular research results that are potentially impacted by bad inference. Because SpeechMap records their providers, we can tell that their results come from a variety of model providers, and their results on how models deal with controversial speech mixes together model variance and provider variance. In their case, this may be intentional/the right decision if they want to measure how models behave in “real world OpenRouter usage.”
- ^
You can see Claude’s attempt at a taxonomy of fingerprints, here. The full list was not human validated, though many make intuitive sense, like distributional tests on samples.
- ^
Claude’s review of influential AI and AI Safety research codebases that use OpenRouter for their reported results found that 31⁄32 (97%) of them use OpenRouter unsafely.[12]These numbers were spot-checked but not thoroughly reviewed. Exploring the 35 AI research repos Claude found that mention “OpenRouter”, 34 called OpenRouter and 32 use OpenRouter for their reported results. 31 out of 32 (all except nostalgebraist) were vulnerable to some form of corruption. It would require deep knowledge of the results and setups to say whether the findings were actually impacted. See the Artifact to explore Claude’s findings for yourself.
- ^
I’m making PRs to fix some of these issues in Control Tower when they are relevant to my research.
AI Control dynamics would likely not be affected, but safety/usefulness frontiers might be.
- ^
Like Cerebras’ “We are committed to serving the original models for all existing endpoints, without modification”. Sounds great, and I’ll prefer them going forward. Still, who knows if that’s true?
- ^
Someone should be keeping a closer eye on model providers. OpenRouter says “[benchmark] data will be shown publicly soon but is not yet available on the site”. Until they do, someone else could step in. But this doesn’t eliminate the possibility providers are serving backdoored models or models that maintain accuracy but are different in other ways.
- ^
By ‘safely’, we mean that the data produced by experiments calling that model are trustworthy in the sense that the request really was made as the researcher sent it.
Edit 3: I’ve removed my downvote after Matthew made various changes to the post, which I think is now much better.
Strong downvoted for slop research, even though I agree with the takeaway in the title.
In the executive summary, a core claim is:
And this “review” just redirect us to a Claude artifact, which classified “MathArena” as an AI safety research codebase. Come on. Does anybody even read posts anymore.
This Claude audit is also slop? I’m willing to bet $100 to $1 that, modulo unexpected issues (e.g., (1) (2)), Anthropic/OpenAI/Amazon Bedrock will not deteriorate the quality of a model or switch it out. All three of these providers were rated as a C by Claude in your audit! It rates Anthropic as a C because there is “No written guarantee that weights behind a dated ID are fixed.” Come on. Also, when people are using openrouter, they’re typically using open weight models, so it doesn’t even make sense to rate “Anthropic” as a provider.
----
Edit: I thought everyone already does this. You’d want to do pin the provider just for token speed purposes already.
But I am surprised by how many people didn’t already know this, so my guess is that this post will do a lot of good.
Idk. I still feel like I should downvote it. We should have a high bar.
----
Edit 2: Some more clarifications of my thinking behind making this comment and bring attention to it on Twitter:
If this post contained the title, a single paragraph of the issue with Arun’s paper, and a single bullet point on how the Inspect package does not force you to pin an openrouter provider, then I would have strong upvoted it. However as it stands, the fourth bullet point in the abstract/introduction contains a number that is ~totally made up by Claude. As I skimmed the post, I immediately went to “No Guarantees from Fixing a Provider.” And I was like, “oh my! I just fix my provider, what could I have been doing wrong this whole time??” Instead I have to sift through this Claude coded website which declared that the Anthropic provider scores a “C” because there is “no written guarantee that weights behind a dated ID are fixed” (along with some other hard-to-understand-claudeslop):
This, like many parts of this post, is locally invalid, and as Eliezer said, Local Validity [is] a Key to Sanity and Civilization.
I care a lot lesswrong not being overwhelmed by AI-generated slop, and that the front page posts on lesswrong remain high-quality.[1] This post does not meet my personal quality bar. I feel a bit conflicted about downvoted because I was surprised by how few people know about the openrouter providers. Naively, this post would do a lot of good by informing everyone about this! However, I am generally skeptical of arguments of the form “oh this would do a lot of good, so therefore we should break a rule.”
I tweeted about this post because I want to bring attention to it. If others agree with my assessment, it would rapidly bring the karma count down. Many people visit the lesswrong front page at a given moment, so I prefer these corrections done quickly.
This is also not the first time I’ve publicly downvoted a post, and I expect that I will keep doing this.
FWIW I’ve found some (fairly inconsequential) cases where Claude behaves consistently differently on Amazon Bedrock, compared to Anthropic 1P and Vertex. I tried replicating this example from Anthropic (my replication) where some prompt behaved differently in the “vanilla API” and “clean” sampling, and found that Bedrock always gives a correct answer while 1P only does ~70% of the time. (I don’t think this is a big issue to be clear)
Wow, that’s crazy! I independently replicated this here.
It seems like the Openrouter Opus 4.6 is just kind of cooked? On this prompt, it would return glitch tokens in its response like:, “cliff\n\nLet me try again.\n\n67”, “връщане\n\n67″, or “pigeons\n\nWait, let me reconsider.\n\n42\n\nHmm, let me try again.\n\n73”
Funny enough, this is an issue if you use the Anthropic provider through openrouter, but not if you call the Anthropic API.
In the original NLA post, the authors say that “this experiment is sensitive to specific sampling parameters. Specifically, it appears non-replicable on the public API.” I wonder what’s going on with how this model is being served? Anyways, I also tried replicating the result where increasing the stated reward increases the reward seeking rate. I find a non-monotonic relationship in the Openrouter Claudes.
(also note the big gap between the Boolean reward and the numeric reward).
Nice! Wish I’d been here sooner to fleece you on Manifold about your “willing to bet $100 to $1” :)
Ah I wasn’t being super clear, but this does not go against what I was trying to say, which is that there wouldn’t be any like, unannounced and human-intended changes to the model. For example, Claude complained that, for the Anthropic provider:
I think whatever happened with openrouter and opus 4.6 falls under unintended inference deterioration. But I suppose that, from a user’s perspective, it doesn’t matter why there is performance deterioration.
Well! In that case: https://manifold.markets/Gurkenglas/will-tim-hua-concede-his-bet
This is surprising to me! I would have expected negligible differences.
What does this mean for how we should do research in practice? Should we avoid OpenRouter entirely?
At the least, we should have a norm of recording which provider was used.
My theory as to what’s happening here is that there’s some subtle difference between inference on Trainium/TPUs/NVIDIA GPUs. The difference between OpenRouter Anthropic and direct Anthropic could be explained by sticky routing (using the same server for subsequent requests from the same user, to maximize the KV cache hit rate). (since presumably the first-party API uses a mix of GPU types)
For context, the post is decently transparent about the quality of the research presented. A quick exploration done in a few days and mostly AI made should be expected to have a number of issues.
I think there’s a difference between an error (e.g., some llm judge you used having a really poor recall rate or something) and taking poor Claude-conducted analysis as given (e.g., 97% of AI safety research use Openrouter unsafely, which is reported above the epistemic note. Ditto the “all the providers have gap” section.)
[I’ve also expanded on my thinking about this more in the original comment’s edit.]
There are sometimes subtle but significant drops in Claude and Codex model quality. This is something that could impact research involving those models, and it would be good if Anthropic and OpenAI were to have a log of what happened, why, and when the model is fully operational. I agree that first parties are generally fairly trustworthy, but I would still say they don’t reach the ideal for research. https://marginlab.ai/trackers/claude-code-historical-performance/
Though I admit I did not check all of Claude’s claims in the artifacts, I did spot check them and verified the specific claims (though not aggregate numbers) that made it to the post.
I actually considered using Cerebras for an experiment, but you’d have to design your entire experiment around them since they only offer three models in the non-enterprise API (and are only committed to continuing to serve one of them).
I don’t think it’s really necessary to run the models locally unless you’re worried about privacy. A rented GPU from somewhere like RunPod is probably fine. They have no incentive to quantize your models (why would they want you to use cheaper hardware?), and it would be hard for them to do even if they wanted to since you’re running arbitrary code.
I’m actually in the middle of running an experiment using GPT-OSS-20B on rented A40′s from RunPod (not using API access for exactly the reasons in your post), and it’s a little bit more annoying but not that expensive[1], and I would be pretty surprised if they were messing with the code I’m running. You can even parallelize via SkyPilot, although RunPod’s quality control is pretty terrible so you need to make sure the machine actually booted and has a GPU (but it’s obvious when RunPod machines break—you’re not going to get seemingly good results, the code just won’t run at all).
The conversation can’t be shared because it used Deep Research mode, but:
For batch workloads (thousands of prompts, spin up → run → tear down, no interactive debugging), renting GPUs costs roughly the same per token as the API, sometimes less. GPT-OSS-120B on a single H100 runs ~$0.14–0.46/M output vs OpenRouter’s ~$0.10–0.19/M. DeepSeek-R1 in reference FP8 needs 8×H200 (685GB of weights won’t fit in 8×H100′s 640GB) and lands near $3/M output vs OpenRouter’s $2.50/M — parity. On B200 both models are plausibly several times cheaper than the API.
Three conditions on that:
You have to saturate the GPU. Use an offline batch API (
LLM.generate()over the whole prompt list), not an HTTP server with a rate-limited client. Batch-of-1 throughput is 10–100× worse and blows up the per-token cost entirely.Startup overhead scales with the model. For a 63GB model it’s a few dollars; for DeepSeek’s 685GB it’s more like $10–25 of billed time pulling weights, which is a real fraction of a short run. Use
hf_transfer, or a network volume if you’re iterating.Claude is a little bit optimistic here because RunPod’s quality control is so bad, but I think it’s within the right order of magnitude.
You’re right, sorry.
When I wrote “locally, yourself” I was thinking of either having or renting GPUs to run your models. I should have written “on hardware you control.”
For my use case, I sadly care a lot about “interactive debugging”. I’m iterating on prompts, control protocols, and models, so if I was running on rented GPUs they would be idling for a long time between experiments.
I don’t know too much about the details of running models on rented GPUs, but after looking into OpenRouter for the post, I am inclined to learn. Thanks for sharing!
Quantizing something and then claiming it’s the “same model” is, of course, criminally fraudulent.
These recent papers are also relevant:
“Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts” https://arxiv.org/abs/2607.09999
“Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models” https://arxiv.org/abs/2606.25519