Propaganda-masquerading-as-paper: the paper is mostly valuable as propaganda for the political agenda of AI safety. Scary demos are a central example. There can legitimately be valuable here.
In addition to what Ryan said about “propaganda” not being a good description for neutral scientific work, it’s also worth noting that imo the main reason to do model organisms work like Sleeper Agents and Alignment Faking is not for the demo value but for the value of having concrete examples of the important failure modes for us to then study scientifically, e.g. understanding why and how they occur, what changes might mitigate them, what they look like mechanistically, etc. We call this the “Scientific Case” in our Model Organisms of Misalignment post. There is also the “Global Coordination Case” in that post, which I think is definitely some of the value, but I would say it’s something like 2⁄3 science and 1⁄3 coordination.
Yeah, I’m aware of that model. I personally generally expect the “science on model organisms”-style path to contribute basically zero value to aligning advanced AI, because (a) the “model organisms” in question are terrible models, in the sense that findings on them will predictably not generalize to even moderately different/stronger systems (like e.g. this story), and (b) in practice IIUC that sort of work is almost exclusively focused on the prototypical failure story of strategic deception and scheming, which is a very narrow slice of the AI extinction probability mass.
I think I mostly agree with this for current model organisms, but it seems plausible to me that well chosen studies conducted on future systems that are smarter in an agenty way, but not superintelligent, could yield useful insights that do generalise to superintelligent systems.
Not directly generalise mind you, but maybe you could get something like “Repeated intervention studies show that the formation of coherent self-protecting values in these AIs works roughly like y with properties b,c,d,e,f. Combined with other things we know, this maybe suggests that the general math for how training signals relate to values is a bit like z, and that suggests what we thought of as ‘values’ is a thing with type signature t.”
And then maybe type signature t is actually a useful building block for a framework which does generalise to superintelligence.
I am not particularly hopeful here. Even if we do get enough time to study agenty AIs that aren’t superintelligent, I have an intuition that this sort of science could turn out to be pretty intractable for reasons similar to why psychology turned out to be pretty intractable. I do think it might be worth a try though.
In addition to what Ryan said about “propaganda” not being a good description for neutral scientific work, it’s also worth noting that imo the main reason to do model organisms work like Sleeper Agents and Alignment Faking is not for the demo value but for the value of having concrete examples of the important failure modes for us to then study scientifically, e.g. understanding why and how they occur, what changes might mitigate them, what they look like mechanistically, etc. We call this the “Scientific Case” in our Model Organisms of Misalignment post. There is also the “Global Coordination Case” in that post, which I think is definitely some of the value, but I would say it’s something like 2⁄3 science and 1⁄3 coordination.
Yeah, I’m aware of that model. I personally generally expect the “science on model organisms”-style path to contribute basically zero value to aligning advanced AI, because (a) the “model organisms” in question are terrible models, in the sense that findings on them will predictably not generalize to even moderately different/stronger systems (like e.g. this story), and (b) in practice IIUC that sort of work is almost exclusively focused on the prototypical failure story of strategic deception and scheming, which is a very narrow slice of the AI extinction probability mass.
I think I mostly agree with this for current model organisms, but it seems plausible to me that well chosen studies conducted on future systems that are smarter in an agenty way, but not superintelligent, could yield useful insights that do generalise to superintelligent systems.
Not directly generalise mind you, but maybe you could get something like “Repeated intervention studies show that the formation of coherent self-protecting values in these AIs works roughly like y with properties b,c,d,e,f. Combined with other things we know, this maybe suggests that the general math for how training signals relate to values is a bit like z, and that suggests what we thought of as ‘values’ is a thing with type signature t.”
And then maybe type signature t is actually a useful building block for a framework which does generalise to superintelligence.
I am not particularly hopeful here. Even if we do get enough time to study agenty AIs that aren’t superintelligent, I have an intuition that this sort of science could turn out to be pretty intractable for reasons similar to why psychology turned out to be pretty intractable. I do think it might be worth a try though.