One minor nitpick: it would not surprise me at all if Dario had never played a videogame or heard of speedrunning. If I hadn’t married the exact person I did, I would not have. Videogames are popular but they are not universal.
Okay, but CoastRunners is, after all, the canonical example of weird over-optimization, the one everyone brings up. For the canonical example to be just-plain-wrong, and obviously so… well, it feels emblematic of the larger problem, to me.
I’m feeling defensive here because I have occasionally cited that example for that purpose.
There is a set of situations where the result of optimization will include Behavior X, but where the person who set up the optimization had not thought of X ahead of time and is surprised when it happens.
In some of those cases, we can all have a good laugh at how dumb the person who set up the optimization must have been, to have not realized that X would result from the optimization, because c’mon, duh, it should have been blindingly obvious from the start.
In the opposite extreme, we can all feel very sympathetic to the person not thinking of X in advance, because man, I wouldn’t have thought of X either.
(Also, lots more X’s seem obvious in hindsight than in foresight.)
Anyway, the important point is the phenomenon in which the person is surprised to discover Behavior X. And I think CoastRunners is a fine example of that happening (assuming that Dario & Jack were in fact surprised, which seems quite plausible to me, even if you want to laugh at them for that).
Also, if you’re thinking “surely smart people like Dario & Jack would have recognized in advance the obvious-to-me consequences of optimizing (blah)”, then please witness the brazen failures of very intelligent people like LeCun, Silver & Sutton, Schmidhuber, Musk, etc. to recognize in advance the obvious-to-many-people consequences of optimizing (blah), even with high stakes and even after other people have repeatedly pointed out their error. Apparently it’s just really hard for many people to think clearly in advance about the results of an optimization search.
Yeah, this one is interesting because before LLMs, there’s no expectation that the AI will be able to figure out what goal you “meant to set”. You’ll just get whatever policy maximizes the reward function (up to the limits of your RL algorithm’s ability to explore). So the mismatch between expectation and the actual learned policy basically has to exist only in the researcher’s head, because the neural net is certainly too simple to have any ideas about it. And then you can always say that the researcher (or game developers) should have expected whatever degenerate strategy the AI ended up finding.
The more recent examples are more meaningful because the LLMs have a concept of the intended space of solution strategies that they are choosing to ignore.
I don’t think it’s a question of being able to tell what exactly X is up front so much as knowing that there is probably some X out there and that you should do due-diligence on what exactly your reward function is and whether exploits exist.
I can’t tell you whether the TAS exploits-allowed world record speedrun of some particular game involves buffer overflows, faulty level geometry, or bonuses that were not meant to be stackable but I can say it’s unlikely to resemble normal 99th percentile play.
Most Sunday nights, Dario Amodei heads over to his younger sister Daniela’s house to play their favourite video game, Final Fantasy VII Remake, set in a dystopian world where the goal is to stop an all-powerful corporation from plundering the planet’s resources.
I love this post!
One minor nitpick: it would not surprise me at all if Dario had never played a videogame or heard of speedrunning. If I hadn’t married the exact person I did, I would not have. Videogames are popular but they are not universal.
To add on to that:
I’m feeling defensive here because I have occasionally cited that example for that purpose.
There is a set of situations where the result of optimization will include Behavior X, but where the person who set up the optimization had not thought of X ahead of time and is surprised when it happens.
In some of those cases, we can all have a good laugh at how dumb the person who set up the optimization must have been, to have not realized that X would result from the optimization, because c’mon, duh, it should have been blindingly obvious from the start.
In the opposite extreme, we can all feel very sympathetic to the person not thinking of X in advance, because man, I wouldn’t have thought of X either.
(Also, lots more X’s seem obvious in hindsight than in foresight.)
Anyway, the important point is the phenomenon in which the person is surprised to discover Behavior X. And I think CoastRunners is a fine example of that happening (assuming that Dario & Jack were in fact surprised, which seems quite plausible to me, even if you want to laugh at them for that).
Also, if you’re thinking “surely smart people like Dario & Jack would have recognized in advance the obvious-to-me consequences of optimizing (blah)”, then please witness the brazen failures of very intelligent people like LeCun, Silver & Sutton, Schmidhuber, Musk, etc. to recognize in advance the obvious-to-many-people consequences of optimizing (blah), even with high stakes and even after other people have repeatedly pointed out their error. Apparently it’s just really hard for many people to think clearly in advance about the results of an optimization search.
Yeah, this one is interesting because before LLMs, there’s no expectation that the AI will be able to figure out what goal you “meant to set”. You’ll just get whatever policy maximizes the reward function (up to the limits of your RL algorithm’s ability to explore). So the mismatch between expectation and the actual learned policy basically has to exist only in the researcher’s head, because the neural net is certainly too simple to have any ideas about it. And then you can always say that the researcher (or game developers) should have expected whatever degenerate strategy the AI ended up finding.
The more recent examples are more meaningful because the LLMs have a concept of the intended space of solution strategies that they are choosing to ignore.
I don’t think it’s a question of being able to tell what exactly X is up front so much as knowing that there is probably some X out there and that you should do due-diligence on what exactly your reward function is and whether exploits exist.
I can’t tell you whether the TAS exploits-allowed world record speedrun of some particular game involves buffer overflows, faulty level geometry, or bonuses that were not meant to be stackable but I can say it’s unlikely to resemble normal 99th percentile play.
It’s true that not everyone has played video games, but there’s been press about Dario and Daniela Amodei playing video games —