Adam Karvonen

• Adam Karvonen 23 Jul 2026 14:54 UTC

  3 points

  1

  in reply to: faul_sname’s comment on: faul_sname’s Shortform

  My understanding is that “GPT-5.6-Sol autonomously post-trained GPT-5.6-Luna” meant something like “make some modifications to a run scheduler file and launch the post-training script”, not “selected, configured, adapted, and ran RL gyms”.

  There’s more info /​ discussion here: https://​​x.com/​​nikolaj2030/​​status/​​2075297831376793764

• Adam Karvonen 22 Jul 2026 16:36 UTC

  5 points

  1

  in reply to: Petropolitan’s comment on: Fron­tier AI Models Still Fail at Ba­sic Phys­i­cal Tasks: A Man­u­fac­tur­ing Case Study

  I had some updates on Twitter here: https://​​x.com/​​a_karvonen/​​status/​​2049891045736055260

  Basically, there has been no progress since this post. Even Claude Fable 5 makes no improvement. Deepseek V4-Preview actually did a bit better than Claude Opus 4.8.

  Over the last two years there was some progress on visual ability to correctly identify more features of the part (although no model can consistently identify all features correctly) and little to no apparent progress on spatial or mechanical reasoning.

  Copy pasting the most relevant part:

  “Wow, Deepseek’s vision is pretty good and better than Claude’s.

  One of the trickiest vision parts is identifying there are two opposing flats, where you have to look very closely at the left part. Deepseek also identifies the cross hole on the upper end of the left part (which GPT-5.5-Pro misidentified as a slot instead of a hole).

  But then Deepseek totally botches it on the radial cross hole by saying “Stand the part perfectly upright, resting the bottom flat face on the parallels.”

  Where it wants to drill the cross-hole in the top of the left part axially, even though the hole is radial.

  This is the type of mistake that all LLMs make that makes me believe they have very poor spatial reasoning ability.”

• Adam Karvonen 22 Jun 2026 16:10 UTC

  5 points

  0

  in reply to: nostalgebraist’s comment on: NLA ex­pla­na­tions can be short­ened with­out harm­ing reconstruction

  Yeah, I think the Matryoshka loss is more “mostly applicable” rather than “directly applicable”.

  With Matryoshka SAEs there are two types of prefixes. The first is the dictionary subsets, which in our implementation were fixed and chosen in advance, which I agree doesn’t make sense for an NLA. The second is that the SAE could have K active features, and it could allocate the K active features across subsets as desired. This seems close to your “graded NLA prefixes” proposal.

  There are also alternative forms of hierarchy that could be explored. For example, in Appendix C.1 of the Matryoshka paper there was an alternative implementation where the dictionary subset sizes where randomly sampled from a distribution when performing inference, which encouraged a more continuous hierarchy.

  Another random idea (which may not be good) is to just pass the first K tokens of the AV verbalization to the AR (where K is chosen randomly), which may encourage the AV to put the more important stuff earlier in the generation, with the fine-grained details towards the end. But it could also encourage weird compression.

• Adam Karvonen 22 Jun 2026 1:49 UTC

  11 points

  1

  in reply to: nostalgebraist’s comment on: NLA ex­pla­na­tions can be short­ened with­out harm­ing reconstruction

  Which would be pretty cool: a “multiscale” view of the activation laid out sequentially from left to right, with the coarsest-grained (highest MSE impact) information at the start and the finest-grained information at the end.

  This would be a sort of “Matryoshka NLA”, which could be pretty cool. Matryoshka SAEs improved on some SAE pathologies (primarily feature absorption). The Matryoshka loss function could probably be used directly in the NLA case.

• Adam Karvonen 15 Jun 2026 2:36 UTC

  3 points

  0

  in reply to: J Bostock’s comment on: SFT Drives Gem­ini’s Safety Properties

  Gwern has speculated that Bing Sydney was only post-trained with SFT and no RL /​ RLHF, which drove the crazy behavior.

Build­ing Bet­ter Ac­ti­va­tion Oracles

• Adam Karvonen 2 Jun 2026 16:07 UTC

  7 points

  1

  in reply to: 1a3orn’s comment on: 1a3orn’s Shortform

  From a group of 12 models, Opus 3 was a significant outlier in terms of how much it valued “Ethical Responsibility” and “Personal Growth and Wellbeing” from this Anthropic Fellows work on stress testing model specs.

  https://​​alignment.anthropic.com/​​2025/​​stress-testing-model-specs/​​

  It’s Figure 2, I was unable to add the image to this comment.

• Adam Karvonen 16 May 2026 17:41 UTC

  4 points

  0

  in reply to: Jiro’s comment on: Vot­ers are sur­pris­ingly open to talk­ing about AI risk

  Anecdotally when I talk to normal people about AI the main questions are “Will AI take my job?” and “Will AI take over the world /​ become Skynet /​ etc?”.

  Although job concerns are taken more seriously than loss of control.

• Adam Karvonen 27 Apr 2026 2:38 UTC

  2 points

  0

  in reply to: Adam Karvonen’s comment on: Can gov­ern­ments quickly and cheaply slow AI train­ing?

  One other thing I’ll mention: to fully exploit the valid non-determinism as a covert communication channel, the attacker needs two very strong affordances which may not be realistic:

  • The attacker would need an exact copy of the model weights outside the data center, which may fail in the closed weight setting

  • The ability to replay the inference process more deterministically than the verifier can replay inference during verification. However, it’s unclear why an attacker outside the datacenter would have a cleaner view of the inference process than a verifier with access to all internal activations, logits, statistics, etc from inside the datacenter. To fully exploit the channel, the attacker needs deviations to be deterministic and predictable from their perspective while still appearing nondeterministic to the verifier.

  Given all of this, I believe this goal (“Output re-computation that reduces covert channel capacity below 0.01%.”) from your intro may be achievable with simple re-computation approaches. Simple DiFR (Divergence From Reference) techniques may work with non-deterministic inference, as the verifier likely has more affordances than the attacker.

  If that doesn’t work, Deepseek demonstrated deterministic inference with “minimal performance overhead” in their Deepseek-V4 technical report.

• Adam Karvonen 24 Apr 2026 16:36 UTC

  24 points

  7

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  Most of those signs are Anthropic specific, and OpenAI has several signs pointing in the opposite direction. They have really been leaning into a message of “efficient frontier AI for everyone”, they reset rate limits multiple times per month as a meme, and Codex rate limits are already very generous.

  Anecdotally I’ve never heard of anyone hitting a Codex rate limit on a Max plan, and from what I hear most people struggle to hit 20% of their weekly rate limit with heavy usage.

• Adam Karvonen 21 Apr 2026 0:31 UTC

  3 points

  0

  on: Can gov­ern­ments quickly and cheaply slow AI train­ing?

  In current prototypes, the “unsuspicious error” (aka level of non-determinism) of output re-computation is 1%; so 1% of server outputs could be generated by our undeclared RL model without raising suspicion.

  I think this is incorrect. The normal non-determinism applies per token, so 1% of output tokens in a given sequence should be incorrect. If inference on a given sequence was ran with an incorrect model then a far higher percentage of tokens in that sequence would be incorrect. So just checking per sequence rather than per token should prevent this attack.

  You could plausibly do attacks where you try to find the 1% most useful tokens in a given sequence. However, we can also measure how unlikely each incorrect token is, which would only allow small deviations from the original model’s behavior, which would further limit utility.

  In practice, I’m guessing you can’t do any useful RL rollouts on undeclared models using the 1% legitimate non-determinism.

  Also, note that the overhead from deterministic inference isn’t that bad, as Horace He’s first pass was 60% overhead, and he thought it could be further reduced. So that’s another way to eliminate this threat.

• Adam Karvonen 15 Apr 2026 16:29 UTC

  50 points

  49

  on: Cur­rent AIs seem pretty mis­al­igned to me

  A related observation: many people still don’t trust AI to provide honest critical feedback if asked directly, and instead resort to anti-sycophancy tricks like “some moron wrote this thing, can you provide a brutally honest critique?”.

• Adam Karvonen 31 Mar 2026 16:40 UTC

  21 points

  17

  in reply to: No77e’s comment on: No77e’s Shortform

  There is also a significant category of “AI video passing itself off as a real video”, and many videos have people debating in the comments if it’s real or AI. This seems like it can erode trust and is generally negative.

• Adam Karvonen 31 Mar 2026 4:37 UTC

  7 points

  1

  on: Pan­gram (AI de­tec­tion soft­ware) can be evaded

  Claude Code can evade Pangram if you give it a prompt and a Pangram labs API key, so it can iterate against Pangram until it passes.

  But in general I think false positives are worse than false negatives here, as it’s always possible to iterate against Pangram to create a negative.

• Adam Karvonen 27 Mar 2026 14:33 UTC

  11 points

  5

  in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform

  One thing that stood out to me from the post was this line: “With less verifiable tasks, the gamemaster decides how successful the AI is.” This seems like a major assumption which could change the takeaways.

  I’ve been experimenting with using Opus to implement experiments with only high-level feedback. I often find once I dig into the low-level details that Opus has made some very poor decisions which would not be observable from only looking at the final outputs, and that my uplift has been significantly reduced by the time I am satisfied with the quality of the experiment.

  I’m not sure if these problems will be solved by just instructing the AI to provide a proof of correctness or a summary report, although this probably depends on the specific domain. For less verifiable tasks, verification could become a huge bottleneck that scales with task complexity and significantly reduces uplift.

• Adam Karvonen 10 Mar 2026 20:39 UTC

  9 points

  0

  on: Let­ting Claude do Au­tonomous Re­search to Im­prove SAEs

  Nice post! This seems like a good application for autonomous research—a good metric, tight feedback loops, and there hasn’t been a ton of research effort directed at it yet.

  Karpathy had an auto-research scaffold improve validation loss on LLMs and from what I can see it mostly just tweaks hyperparameters, but LLM pretraining is a much better studied area.

  The Top-K annealing was used in the Llama Scope paper as well: https://​​arxiv.org/​​pdf/​​2410.20526

• Adam Karvonen 5 Mar 2026 17:53 UTC

  5 points

  0

  on: Text Com­pres­sion Can Help Se­cure Model Weights

  My estimates on text traffic and model weight sizes:

  Text traffic

  OpenAI was doing ~8.64 trillion tokens per day in October 2025 on the API (not including ChatGPT)

  Google went from processing ~500 trillion tokens per month in May to ~1 quadrillion tokens per month in July.

  It’s uncertain how many of these are output tokens. Based on OpenRouter’s numbers, it seems like a plausible estimate is 10%, or 100 trillion output tokens per month (back in July). This is 200 TB per month, or 7 TB per day. I’m not sure how many data centers they have—if there’s 10 data centers, then it would be 700GB per day.

  It has been 8 months since then, so a naive extrapolation suggests this is now ~16x larger.

  For model weight sizes, Grok-3 and Grok-4 were 3 trillion parameters per Elon. This would be ~1.5-3TB, depending on if they use 4 or 8 bit datatypes. Grok-5 will be 6 trillion parameters, also per Elon.

  Anthropic’s Feb 2026 risk report discussed securing “multi-terabyte model weights”.

  So a reasonable estimate is 1-10TB of output tokens per datacenter per day, and 1-10TB model weight size.

• Adam Karvonen 27 Feb 2026 5:04 UTC

  2 points

  0

  in reply to: micahcarroll’s comment on: Real­is­tic Eval­u­a­tions Will Not Prevent Eval­u­a­tion Awareness

  I agree that “eval awareness” can be a unclear concept and that reasoning about reward process and if an interaction is an evaluation is a very useful and natural strategy for a model to learn during RL. I generally think of it as meaning “the model thinks it may be in an evaluation”, even if it does not verbalize this information. It also doesn’t even have to affect the model’s behavior. This belief that “this may be an evaluation” is enough to muddy the results of safety /​ alignment related evaluations.

  One potential distinction between eval awareness and reward hacking could be that the most concerning version of eval awareness could involve exploiting the gap between training /​ evaluation and deployment, rather than exploiting the reward signal. For example, a competent schemer could always behave correctly in training and only act misaligned in deployment. However, this regime is speculative and we don’t have evidence of this sort of deliberate scheming today.

• Adam Karvonen 24 Feb 2026 19:29 UTC

  4 points

  0

  in reply to: benjamin ar’s comment on: Real­is­tic Eval­u­a­tions Will Not Prevent Eval­u­a­tion Awareness

  I think your summary is basically correct.

  After writing this, it’s hard for me to imagine a world in which it’s possible to build a complete safety case for a model before deployment.

  I think that building a safety case with a very high level of assurance is extremely difficult, as there is a (probably small) chance that the model is a competent schemer that is patiently waiting for better conditions.

  I think Anthropic’s approach in their recent model system cards is a reasonable approach with multiple stacked layers of evidence: behavioral evals look good, interpretability techniques didn’t find anything major, model organisms research that purposely trains models with bad goals didn’t produce competent schemers, production traffic monitoring didn’t find anything, current models still aren’t very intelligent or coherent on agentic tasks, etc. These can stack to provide reasonable confidence, although they may struggle to provide very high assurance with future more capable models.

  That said, there are also many confounders that Anthropic includes in their system cards. For example, Claude Opus 4.6 is quite good at subtly sabotaging experiments without getting caught, and Claude Opus 4.6 is used extensively for technical safety work due to time pressure.

Real­is­tic Eval­u­a­tions Will Not Prevent Eval­u­a­tion Awareness