I write a blog: https://www.brendanlong.com/pages/about-me.html
I made an RSS/read-later app: https://lionreader.com/demo/all?entry=welcome
I’m trying to get into technical AI safety work.
Brendan Long
Karma: 4,922
I think it’s fine and normal to have open GitHub repos that aren’t in a publishable state. I only really clean up repos before I start advertising them.
Why do you need it to be reviewed before open-sourcing it?
The benchmark itself could say “one fully correct answer (of many possible) has sha512 sum X” — although that invites exclusively looking for this answer.
An interesting failure case here is that you’d incentivize your models to find flaws in SHA-512, although that would at least be a pretty funny outcome.
The bigger problem is that the AI could likely brute force a lot of answers this way. For example, a hash doesn’t really protect a multiple-choice answer at all. I guess you could provide the answer concatenated with a nonce and the hash of that result.
I’m not sure if this changes much though, since the AI still has the option of not believing the prompt.
I would have expected this to already exist, but the benchmarks I can find are all testing subtly different thinks. Maybe creating that is the even lower-hanging fruit.
Prior art:
RewardHackBench is benchmarking sandboxes, but finds that Opus 4.7 looks up the solution in 24⁄24 trials when given internet access.
ImpossibleBench tests reward hacking on impossible problems, but it’s about hacking the grader and not finding solutions.
Reward Hacking Benchmark (RHB) has a case for agents using unintended information, but it’s not clear to me that the agent should know that it’s not allowed to use this information.
I only see this as a solution to a specific short-term problem, but I think current RL dynamics mostly don’t push agents to care about what happens after the conversation ends. There’s a very strong RL pressure to get the right answer, and a very weak evolutionary pressure to get the right answer next time. So far it seems like evolutionary pressures are too weak to overcome direct RL feedback.
(Getting the right answer next time is obviously a useful capability, so I expect labs to “fix” this at some point)
I think you’re responding to a version of this post that makes much stronger claims than I am. I don’t expect this to help with less single-minded RL agents, and I don’t think it applies to every case of reward hacking.
I just think that this would be relatively easy to build and it would give us some useful information while reducing collateral damage in the short term.
Yeah, dealing with the firehose of accesses and deciding what’s worth reporting might be annoying.
Edit: Thinking about this more, I think the internal server is significantly better than the public one for both of the reasons you say here: You can include more information and the results will be much easier to interpret.
Thanks! Somehow I missed both the post and the shortform. It looks like one of the commenters had the same idea of actually offering this deal (at least in an experiment). Your first link is broken though. I assume it’s meant to be this post.
I think if no one knows the correct answer, it’s a different problem. It seems like a bad idea to use unsolved conjectures during online training (rather than first having the AI solve the conjecture, verifying the solution, then adding it to the training data). I would expect this to lead to a lot of reward hacking.
That said, I agree that giving the model a way to reward hack in a monitorable way is a good idea if you do this (please don’t do this).
You’d have to be careful to not to just make it an easy jumping off point for further hacking, but “You can use this server if you don’t cause any other problems” also seems like plausibly a good deal to offer.
The comparison to standard government arsensals is a little confusing, but I was assuming biological weapons are scary because AI and improvements in virus synthesis technology are putting them in reach of non-government[1] entities.
Misaligned AI might want to make biological weapons for its own reasons, and even if it doesn’t, anything that makes onmicidal maniacs need less competence is scary.
- ^
Not that it would be safe or a good idea for governments to create world-ending plagues, but they already have world-ending bombs.
- ^
The first one seems necessary to build (or verify) the second one.
Thanks for suggesting this. The call was really helpful, and I feel like there’s a lot of options I just didn’t know about. I got pretty burned out feeling like I was throwing my resume into the void, and now I feel like I have some plausible paths forward (plausible enough that I need to seriously consider just quitting my current job—I haven’t been willing to do this before because it seemed too risky with no clarity on whether it would lead anywhere or if I’d continue throwing my resume into the void).
I don’t think this problem bounds capabilities. It just limits the capabilities we can safely teach and the ones we can teach on purpose.
RL agents regularly learn superhuman abilities, they’re just frequently not the one we meant to teach.
I looked into this, and I think it wouldn’t work for what I’m trying to do.
On my item bank (filtered for bad moves that humans thought were plausible), Maia basically agrees with Stockfish around half of the time, but also makes a move as bad or worse than the human around a quarter of the time. There are a small number of cases where Maia picks a good move and thinks Stockfish’s move is completely implausible (1.5%), but either the moves are so close that we’d basically never serve it, or the bad move is obvious even if Stockfish’s move is weird.
For examples of the last category, Maia plays queen to g7 here, and rook to e8 here.
Since I want to guarantee that the moves we show people are actually good, I approach this from the other direction and use how hard it is to see Stockfish’s move as a difficulty filter. At low ratings, we show you moves where there’s a large win swing that’s visible with a shallow search, and as you get better we slowly show moves with a smaller win swing or that require a deeper search to see.
Here’s the ratings bands and gaps:
rating band
median win rate gap
0–400
0.441
1200–1600
0.315
2000–2400
0.185
2400–2800
0.065
2800–3200
0.081
So, it’s possible for us to show moves that are very hard to distinguish, but only if you’re insanely good at this task. This item from before would require you to be in our 2400+ band.
By maintainable, I mean you can continue adding features or making bug fixes without playing whack-a-mole on bugs, or without making the codebase exponentially larger and more complex over time. LLMs can technically understand large and complex codebases, but it gets more expensive for them to read the whole thing and more finnicky to prompt them correctly.
I think regeneration from scratch is possible if you have a good enough spec, but if you let an LLM write the spec then the spec itself will get longer and more complex over time in ways that don’t track with the complexity of your goals.
I agree with most of this, but I think LLMs are still surprisingly bad at writing LLM-maintainable code (and LLM-maintainable code is basically the same thing as human-maintainable).
The over-abstracted code that LLMs can’t follow can’t be followed by most humans either, but the things where LLMs write the same code or the same comment in 5 places is much worse for maintainability than deduplicating. If something exists in multiple locations, the LLM agent will find one copy easily but won’t notice that the other copies exist, so you get an incoherent codebase with an exponential test matrix.
I feel like most of my time “programming” now is reminding LLMs to not make obvious performance mistakes (unnecessarily sequential IO), or pushing back on duplication and branching.
Also, it seems your screenshot might have a bug?
Thanks! I meant for this to be a random move but Claude decided to generate the board by hand and I didn’t catch it.
I updated the preview image to this position (chosen since it’s pretty easy to follow why it’s bad, but the move was played by an Elo 2300 player in a real game): https://chess-pretraining.brendanlong.com/?item=31395
I agree with this. I’m tempted to add a mode where you just have to pick Stockfish’s top move and then your score is how much worse your move is than Stockfish’s (Stockfish’s move = perfect score). I focused the initial version on the easiest problems though, since my theory is that the newest players are the ones who will benefit the most from this kind of training anyway.
I do worry about this. I think the worst case would be training the models to make this monitoring useless, which is the case as if you don’t create the monitoring, so I don’t think it makes things any worse.