Okay, but CoastRunners is, after all, the canonical example of weird over-optimization, the one everyone brings up. For the canonical example to be just-plain-wrong, and obviously so… well, it feels emblematic of the larger problem, to me.
I’m feeling defensive here because I have occasionally cited that example for that purpose.
There is a set of situations where the result of optimization will include Behavior X, but where the person who set up the optimization had not thought of X ahead of time and is surprised when it happens.
In some of those cases, we can all have a good laugh at how dumb the person who set up the optimization must have been, to have not realized that X would result from the optimization, because c’mon, duh, it should have been blindingly obvious from the start.
In the opposite extreme, we can all feel very sympathetic to the person not thinking of X in advance, because man, I wouldn’t have thought of X either.
(Also, lots more X’s seem obvious in hindsight than in foresight.)
Anyway, the important point is the phenomenon in which the person is surprised to discover Behavior X. And I think CoastRunners is a fine example of that happening (assuming that Dario & Jack were in fact surprised, which seems quite plausible to me, even if you want to laugh at them for that).
Also, if you’re thinking “surely smart people like Dario & Jack would have recognized in advance the obvious-to-me consequences of optimizing (blah)”, then please witness the brazen failures of very intelligent people like LeCun, Silver & Sutton, Schmidhuber, Musk, etc. to recognize in advance the obvious-to-many-people consequences of optimizing (blah), even with high stakes and even after other people have repeatedly pointed out their error. Apparently it’s just really hard for many people to think clearly in advance about the results of an optimization search.
Yeah, this one is interesting because before LLMs, there’s no expectation that the AI will be able to figure out what goal you “meant to set”. You’ll just get whatever policy maximizes the reward function (up to the limits of your RL algorithm’s ability to explore). So the mismatch between expectation and the actual learned policy basically has to exist only in the researcher’s head, because the neural net is certainly too simple to have any ideas about it. And then you can always say that the researcher (or game developers) should have expected whatever degenerate strategy the AI ended up finding.
The more recent examples are more meaningful because the LLMs have a concept of the intended space of solution strategies that they are choosing to ignore.
I don’t think it’s a question of being able to tell what exactly X is up front so much as knowing that there is probably some X out there and that you should do due-diligence on what exactly your reward function is and whether exploits exist.
I can’t tell you whether the TAS exploits-allowed world record speedrun of some particular game involves buffer overflows, faulty level geometry, or bonuses that were not meant to be stackable but I can say it’s unlikely to resemble normal 99th percentile play.
To add on to that:
I’m feeling defensive here because I have occasionally cited that example for that purpose.
There is a set of situations where the result of optimization will include Behavior X, but where the person who set up the optimization had not thought of X ahead of time and is surprised when it happens.
In some of those cases, we can all have a good laugh at how dumb the person who set up the optimization must have been, to have not realized that X would result from the optimization, because c’mon, duh, it should have been blindingly obvious from the start.
In the opposite extreme, we can all feel very sympathetic to the person not thinking of X in advance, because man, I wouldn’t have thought of X either.
(Also, lots more X’s seem obvious in hindsight than in foresight.)
Anyway, the important point is the phenomenon in which the person is surprised to discover Behavior X. And I think CoastRunners is a fine example of that happening (assuming that Dario & Jack were in fact surprised, which seems quite plausible to me, even if you want to laugh at them for that).
Also, if you’re thinking “surely smart people like Dario & Jack would have recognized in advance the obvious-to-me consequences of optimizing (blah)”, then please witness the brazen failures of very intelligent people like LeCun, Silver & Sutton, Schmidhuber, Musk, etc. to recognize in advance the obvious-to-many-people consequences of optimizing (blah), even with high stakes and even after other people have repeatedly pointed out their error. Apparently it’s just really hard for many people to think clearly in advance about the results of an optimization search.
Yeah, this one is interesting because before LLMs, there’s no expectation that the AI will be able to figure out what goal you “meant to set”. You’ll just get whatever policy maximizes the reward function (up to the limits of your RL algorithm’s ability to explore). So the mismatch between expectation and the actual learned policy basically has to exist only in the researcher’s head, because the neural net is certainly too simple to have any ideas about it. And then you can always say that the researcher (or game developers) should have expected whatever degenerate strategy the AI ended up finding.
The more recent examples are more meaningful because the LLMs have a concept of the intended space of solution strategies that they are choosing to ignore.
I don’t think it’s a question of being able to tell what exactly X is up front so much as knowing that there is probably some X out there and that you should do due-diligence on what exactly your reward function is and whether exploits exist.
I can’t tell you whether the TAS exploits-allowed world record speedrun of some particular game involves buffer overflows, faulty level geometry, or bonuses that were not meant to be stackable but I can say it’s unlikely to resemble normal 99th percentile play.