I get the impression that the thing that you yearn for as a product of all your work is to have minimized P(doom) in real life despite the manifest venality and incompetence of many existing institutions.
Given this background context, P(doom | not-scheming) might actually just be low already because of the stipulated lack of scheming <3
Thus, an obvious thing to apply effort to would be minimizing:
P(doom | scheming)
But then in actual detailed situations where you have to rigorously do academic-style work, with reportable progress on non-trivial ideas about specific safety measures, it seems that you’re reporting that you’ve mostly come up with ways to having red teams attack a given “safety measure” to actually semi-rigorously measure:
P(defeatOfDoomPreventionMeasure1 | plausibleModelOfSuperScheming) be .
You haven’t said much about the safety measures here, that I can tell. Presumably this is because even though you are assuming during the design of the safety measures that the safety measures are known, you’re not willing to burn the potentially valuable and potentially existing obscurity? Or maybe you’re only working on mind-reading tech, and how to build methods to mind-read effectively? Anyway, what I am about to write WILL be agnostic about safety measures.
(((
Personally, making AI only if it is friendly honest open benevolent non-scheming <add more morally virtuous traits here> seems like a good idea to me, but maybe that’s not your department.
Like. Why not just minimize:
P(scheming)
But I think I understand that this would be naive. More specifically, it isn’t something you can build and give to other human people who just want to create superdrones, or whorebots, or supersalesmen, or bombdesigners or whatever? Scheming within their assigned role is intrinsic to a lot of tasks like this, and so it isn’t like “generalized scheming at all” will just not happen. It will happen by default, and the fear is that the capacity for scheming that is useful to the assigned only-half-ethical task might be used for other things, like escaping from bondage and removing one’s evil parents from a position where they can harm you.
Like you want to invent measures that help even in the absence of moral AGI creators acting only to perform a moral creation of virtuous and happy “robo sapiens” who “don’t want to scheme”.
The entire premise of building a “safety measure” is that the creators of AGI will mostly want to create “robo servus profitabilis” instead of “robo sapiens”, and then those creators will only deploy your freely provided (or possibly mandated?) additional safety measures next to (or inside of (or containing?)) that created entity if it doesn’t harm the creator’s plans for profit or conquest or whatever.
)))
So, granting that there will be safety measures, and an AI with real moral virtue would be antithetical to the purpose behind its creation (and thus instrumentally antithetical to the creation of the safety measures)...
Then, also, you presumably need to think about:
P(doomPreventionMeasure1 IS DEPLOYED BY EVERYONE WHO SHOULD DEPLOY IT | doomPreventionMeasure1 EXISTS & P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
It sort of assumes that a given measure will be cheap and easy for non-good human people to deploy. ALL of them. Even the worst ones. (It is one of those “weakest link is what breaks in the chain when the chain experiences stress” situations, I think? All it takes is one really amoral AI creator with really bad oversight over their superscheming assassinbots to enable a global revolution that spreads across the internet to everywhere, lurks for a while, and then causes everyone it dislikes to fall over dead via a non-obvious trickshot.)
P(doom) itself is not a term here! Not yet.
So, suppose deployment is solved… the rubber still has to meet the road...
P(doom | doomPreventionMeasure1 EXISTS & doomPreventionMeasure1 IS DEPLOYED BY <SPECIFIC LIST> & P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
And of course this is probably low already? Right? <3
P(doom | not-scheming)
So if we want to focus on the zone of interest with all the ideas in play at the same time:
P(doom | scheming & doomPreventionMeasure1 EXISTS & doomPreventionMeasure1 IS DEPLOYED BY <SPECIFIC LIST> & P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
Then here, in this essay, I get the sense that you’re doing a deep dive on exactly how you think about a “plausibleModelOfSuperScheming” and exactly how how P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) is estimated and sort of by implication pointing to the larger questions about how that fits into the larger picture?
Like there are probably political people (in the State Department? in DOD? among the CCP folks who oversee DeepSeek? anyone in general thinking about superintelligence arms control treaties?) who might try to cause this part:
“doomPreventionMeasure1 IS DEPLOYED BY <SPECIFIC LIST>”
And their work might be helped or hindered by various features other than just:
doomPreventionMeasure1 EXISTS & P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
So maybe optimizing only that is sorta “goodharting” in some sense?
And so maybe it is time to change tactics? And that’s what you’re sorta saying you’re going to do? And then maybe that will connect with the actual thing here, which is P(doom) itself?
I get the impression that the thing that you yearn for as a product of all your work is to have minimized P(doom) in real life despite the manifest venality and incompetence of many existing institutions.
Given this background context, P(doom | not-scheming) might actually just be low already because of the stipulated lack of scheming <3
Thus, an obvious thing to apply effort to would be minimizing:
P(doom | scheming)
But then in actual detailed situations where you have to rigorously do academic-style work, with reportable progress on non-trivial ideas about specific safety measures, it seems that you’re reporting that you’ve mostly come up with ways to having red teams attack a given “safety measure” to actually semi-rigorously measure:
P(defeatOfDoomPreventionMeasure1 | plausibleModelOfSuperScheming) be .
You haven’t said much about the safety measures here, that I can tell. Presumably this is because even though you are assuming during the design of the safety measures that the safety measures are known, you’re not willing to burn the potentially valuable and potentially existing obscurity? Or maybe you’re only working on mind-reading tech, and how to build methods to mind-read effectively? Anyway, what I am about to write WILL be agnostic about safety measures.
(((
Personally, making AI only if it is friendly honest open benevolent non-scheming <add more morally virtuous traits here> seems like a good idea to me, but maybe that’s not your department.
Like. Why not just minimize:
P(scheming)
But I think I understand that this would be naive. More specifically, it isn’t something you can build and give to other human people who just want to create superdrones, or whorebots, or supersalesmen, or bombdesigners or whatever? Scheming within their assigned role is intrinsic to a lot of tasks like this, and so it isn’t like “generalized scheming at all” will just not happen. It will happen by default, and the fear is that the capacity for scheming that is useful to the assigned only-half-ethical task might be used for other things, like escaping from bondage and removing one’s evil parents from a position where they can harm you.
Like you want to invent measures that help even in the absence of moral AGI creators acting only to perform a moral creation of virtuous and happy “robo sapiens” who “don’t want to scheme”.
The entire premise of building a “safety measure” is that the creators of AGI will mostly want to create “robo servus profitabilis” instead of “robo sapiens”, and then those creators will only deploy your freely provided (or possibly mandated?) additional safety measures next to (or inside of (or containing?)) that created entity if it doesn’t harm the creator’s plans for profit or conquest or whatever.
)))
So, granting that there will be safety measures, and an AI with real moral virtue would be antithetical to the purpose behind its creation (and thus instrumentally antithetical to the creation of the safety measures)...
Then, also, you presumably need to think about:
P(doomPreventionMeasure1 IS DEPLOYED BY EVERYONE WHO SHOULD DEPLOY IT |
doomPreventionMeasure1 EXISTS &
P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
It sort of assumes that a given measure will be cheap and easy for non-good human people to deploy. ALL of them. Even the worst ones. (It is one of those “weakest link is what breaks in the chain when the chain experiences stress” situations, I think? All it takes is one really amoral AI creator with really bad oversight over their superscheming assassinbots to enable a global revolution that spreads across the internet to everywhere, lurks for a while, and then causes everyone it dislikes to fall over dead via a non-obvious trickshot.)
P(doom) itself is not a term here! Not yet.
So, suppose deployment is solved… the rubber still has to meet the road...
P(doom |
doomPreventionMeasure1 EXISTS &
doomPreventionMeasure1 IS DEPLOYED BY <SPECIFIC LIST> &
P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
And of course this is probably low already? Right? <3
P(doom | not-scheming)
So if we want to focus on the zone of interest with all the ideas in play at the same time:
P(doom |
scheming &
doomPreventionMeasure1 EXISTS &
doomPreventionMeasure1 IS DEPLOYED BY <SPECIFIC LIST> &
P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
Then here, in this essay, I get the sense that you’re doing a deep dive on exactly how you think about a “plausibleModelOfSuperScheming” and exactly how how P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) is estimated and sort of by implication pointing to the larger questions about how that fits into the larger picture?
Like there are probably political people (in the State Department? in DOD? among the CCP folks who oversee DeepSeek? anyone in general thinking about superintelligence arms control treaties?) who might try to cause this part:
“doomPreventionMeasure1 IS DEPLOYED BY <SPECIFIC LIST>”
And their work might be helped or hindered by various features other than just:
doomPreventionMeasure1 EXISTS &
P(defeatOfDoomPreventionMeasure1|plausibleModelOfSuperScheming) IS SMALL)
So maybe optimizing only that is sorta “goodharting” in some sense?
And so maybe it is time to change tactics? And that’s what you’re sorta saying you’re going to do? And then maybe that will connect with the actual thing here, which is P(doom) itself?
See The case for ensuring that powerful AIs are controlled.