Do you mean “correctly abandoned” or “well seems empirically like we abandoned it?”
Fwiw I still think Microscope AI is actually pretty good and… I dunno find myself weirdly confidently believing in this moment that most people will pivot to something like Microscope AI once we gets to “the next training run seems legitimately dangerous and we don’t currently know how to control it” (in worlds where the labs correctly identify that moment).
(Seems particularly plausible of Anthropic because Chris Olah invented it and he works there)
I am also fairly bullish on variations on STEM AI. (I guess actually I maybe expect flavors of STEM AI to also be what people pivot to, trying to eke out more spikey capabilities without strategic awareness. But I expect this to stop working sooner than Microscope AI)
Adding a KL penalty against a pretrained model when doing RL is basically a more mathematically-elegant version of quantilization. Take a known-to-be-reasonably-safe distribution (the distribution produced by the pre-trained model), and then optimize it in a bounded way, which is exactly the goal of quantilization. Interestingly, the penalty is not usually used out of concern for alignment, but because generated outputs turned to mush without it. So the field ended up effectively doing quantilization entirely by accident, just because it empirically worked better. Of course, because it was an accident, this also means they might stop (or may have already). You can do many stages of RL, with the KL penalty just being WRT the model in the previous stage. So you can diverge more and more from the original distribution by repeatedly diverging a little bit. And if people do this, we again lose the nice alignment properties of quantilization, for all that the outputs remain sensible.
The other alignment idea from the old days that is currently being implemented by accident is myopia. A good way to RL-train models is with a 1-reply horizon. The model replies to the user, or the coding agent completes its current task and waits for further instruction, and that’s the end of the episode. Reward is assigned and that’s it. The next turn of the conversation is a whole new episode. This is myopia (limiting the agent’s time horizon for reward, and thus hopefully preventing it from wanting to influence the world in large ways). It’s also a particularly nice kind of myopia, where the way the human’s actions depend on the agent’s output is ignored as something that does not causally affect the reward signal. I think this is also done mostly by accident: Having long episodes makes credit assignment difficult, so you don’t want the episode to be too long, and the reply boundary is a natural cutting-point. But plausibly people might have noticed that letting the AI optimize over how users reacted to its outputs resulted in bad things happening, and decided not to do that? Alternately, maybe this info is out of date, and labs do just train with episodes spanning multiple conversation turns now?
I would not say that market making has been abandoned, at least no more so than any other sophisticated scalable oversight approach like amplification or debate; the state of scalable oversight is just such that it’s hard to get any of these sorts of techniques working. See also my general discussion of how I think about scalable oversight here.
I feel like microscope AI hasn’t really been abandoned. LLMs writing code and Lean proofs is evidence that people want (and are building) symbolic versions of fluid intelligence.
Regulatory thresholds based on total compute expended. I get the feeling there’s a lot less talk of the form “10^26 flops is OK, but 10^27 is dangerous.” The pace of algorithmic improvement is such that no fixed threshold makes sense.
I was confused why this was ever a thing. I just assumed everyone had seen this chart and noticed how even if the blue line plateaus the green and red need not, especially given tremendous persistent economic incentives. Maybe the counterargument is “obviously the proposers knew this, what they actually proposed were thresholds that would ratchet downward over time to adjust”?
I think CIRL is basically the right idea and is just hard to make robust against misspecification. I agree that it has been largely abandoned, however.
Which alignment ideas have been approximately abandonded?
Impact measures
Safe optimisers, e.g. quantilisers
Market-marking (this was only one post tbf)
STEM AI
Microscope AI
CIRL
Anything else?
Do you mean “correctly abandoned” or “well seems empirically like we abandoned it?”
Fwiw I still think Microscope AI is actually pretty good and… I dunno find myself weirdly confidently believing in this moment that most people will pivot to something like Microscope AI once we gets to “the next training run seems legitimately dangerous and we don’t currently know how to control it” (in worlds where the labs correctly identify that moment).
(Seems particularly plausible of Anthropic because Chris Olah invented it and he works there)
I am also fairly bullish on variations on STEM AI. (I guess actually I maybe expect flavors of STEM AI to also be what people pivot to, trying to eke out more spikey capabilities without strategic awareness. But I expect this to stop working sooner than Microscope AI)
The latter.
Humans Consulting HCH, based on factored cognition was abandoned.
Adding a KL penalty against a pretrained model when doing RL is basically a more mathematically-elegant version of quantilization. Take a known-to-be-reasonably-safe distribution (the distribution produced by the pre-trained model), and then optimize it in a bounded way, which is exactly the goal of quantilization. Interestingly, the penalty is not usually used out of concern for alignment, but because generated outputs turned to mush without it. So the field ended up effectively doing quantilization entirely by accident, just because it empirically worked better. Of course, because it was an accident, this also means they might stop (or may have already). You can do many stages of RL, with the KL penalty just being WRT the model in the previous stage. So you can diverge more and more from the original distribution by repeatedly diverging a little bit. And if people do this, we again lose the nice alignment properties of quantilization, for all that the outputs remain sensible.
The other alignment idea from the old days that is currently being implemented by accident is myopia. A good way to RL-train models is with a 1-reply horizon. The model replies to the user, or the coding agent completes its current task and waits for further instruction, and that’s the end of the episode. Reward is assigned and that’s it. The next turn of the conversation is a whole new episode. This is myopia (limiting the agent’s time horizon for reward, and thus hopefully preventing it from wanting to influence the world in large ways). It’s also a particularly nice kind of myopia, where the way the human’s actions depend on the agent’s output is ignored as something that does not causally affect the reward signal. I think this is also done mostly by accident: Having long episodes makes credit assignment difficult, so you don’t want the episode to be too long, and the reply boundary is a natural cutting-point. But plausibly people might have noticed that letting the AI optimize over how users reacted to its outputs resulted in bad things happening, and decided not to do that? Alternately, maybe this info is out of date, and labs do just train with episodes spanning multiple conversation turns now?
I would not say that market making has been abandoned, at least no more so than any other sophisticated scalable oversight approach like amplification or debate; the state of scalable oversight is just such that it’s hard to get any of these sorts of techniques working. See also my general discussion of how I think about scalable oversight here.
I feel like microscope AI hasn’t really been abandoned. LLMs writing code and Lean proofs is evidence that people want (and are building) symbolic versions of fluid intelligence.
Regulatory thresholds based on total compute expended. I get the feeling there’s a lot less talk of the form “10^26 flops is OK, but 10^27 is dangerous.” The pace of algorithmic improvement is such that no fixed threshold makes sense.
I was confused why this was ever a thing. I just assumed everyone had seen this chart and noticed how even if the blue line plateaus the green and red need not, especially given tremendous persistent economic incentives. Maybe the counterargument is “obviously the proposers knew this, what they actually proposed were thresholds that would ratchet downward over time to adjust”?
I think CIRL is basically the right idea and is just hard to make robust against misspecification. I agree that it has been largely abandoned, however.