Hi! I took some notes as I read. I’m sure some of them are addressed in what I didn’t read (I only read Plan A, no supplements), or I didn’t pay close enough attention to what I did read; if so, I will try to come back here and update. Here they are, cleaned up and expanded:
This scenario relies heavily on the control agenda working out. Are we sure of this even today? Will this still hold for AIs at the time of the pause?
Could the weights be stolen by distillation, etc., by parties other than the MSS? Would any such parties be liable to purposefully or accidentally leak it to other parties?
Could the US government just decide not to give a dividend? Is there some other use it would rather put the money to? Could it, if it wanted to? Would the public trust blind-verified sousveillance?
If the dangers of AI persuasion are countered by having your own AI, this seems to only move the battlefield up one layer—now AIs are trying to persuade other AIs. Do we know if the asymmetries favor defense here?
Is a basin of sanity a coherent or evidenced concept? I know that in the rationalist sphere we like to think we’ve gotten the path to sanity down, but, is it guaranteed that we’re right about that, or that it is memetically dominant, or that the AIs people use will guide them to it? To put it differently—would, for instance, a far leftist or rightist agree with you on what “sanity” means?
Does humanity actually know how to keep a promise and uphold a contract over the lightcone, or at least long enough for aligned ASI to enshrine it?
Incentivizing safety research: outer loop alignment failure? (Sorry, I can’t remember what this one means. Outer loop refers to the human decision making process, I’m pretty sure.)
Handoff assumes that we really truly have solved alignment once and for all. But, are we sure there are no Old Ones lurking in the unknown unknowns, no black swans to ambush us, etc., with the full might of a decade of humanity and vaguely trustworthy AI bent to the task? (I imagine the answer is simply “that’s the best we can do.”)
Insider: Wasn’t the kludge result from last December? I notice more broadly that a few concepts that are established in current safety discourse get brought up several years later as seemingly-new discoveries. I get that you can’t model discoveries we haven’t made, but swapping in ones we know of already feels a tiny bit off.
Insider: It’s depicted that the AI can be taught defensive-only cybersecurity with enough data filtering. But is it actually theoretically possible to have something that can do cyberdefense without cyberoffense? Or at least so differentially so, that defense is dominant in all attack theories?
How do you stop an AI from reasoning further than a few weeks?
Insider: is “one deduction from the frontier” actually safe?
“Adversarially robust neuralese”—does this make sense? Can it be assumed to be real?
The handoff is quite fast and centralized. If everything is so nice and tidy in 2038, are we still in such a precarious state that the deal could still break at any time, and a fast handoff is warranted?
Some further notes from after reading:
Is a “Long Reflection” best described (and dismissed) as a “Long Memetic War”?
What’s going on with mass surveillance in this scenario?
The scenario seems to assume verifiability is the only way forward for capabilities.
What’s been happening with cybersecurity by the time of the pause?
Generally, can this scenario be described as aligned to humanity? It seems like a lot gets gradually handed off to the AIs that I’m not sure I’d be fine with handing off on reflection. Did disempowerment already happen in 2031?
The scenario sort of feels like a bargain with the accelerationists.
What is your way of addressing algorithmic improvement? Is it robust against all scenarios?
Feels funky to suggest this, but have you considered using frontier AI to analyze and red-team your model?
Could a finer grained model of robot development help here?
The scenario seems to assume that no single critical mistake on our part, that the AI can sneak past us, is enough to end the game in the AI’s favor.
I chatted with Fable after I read the piece and I remember adding expansions to a few notes, but I tried to remove ideas it came up with when writing this up.
Hi! I took some notes as I read. I’m sure some of them are addressed in what I didn’t read (I only read Plan A, no supplements), or I didn’t pay close enough attention to what I did read; if so, I will try to come back here and update. Here they are, cleaned up and expanded:
This scenario relies heavily on the control agenda working out. Are we sure of this even today? Will this still hold for AIs at the time of the pause?
Could the weights be stolen by distillation, etc., by parties other than the MSS? Would any such parties be liable to purposefully or accidentally leak it to other parties?
Could the US government just decide not to give a dividend? Is there some other use it would rather put the money to? Could it, if it wanted to? Would the public trust blind-verified sousveillance?
If the dangers of AI persuasion are countered by having your own AI, this seems to only move the battlefield up one layer—now AIs are trying to persuade other AIs. Do we know if the asymmetries favor defense here?
Is a basin of sanity a coherent or evidenced concept? I know that in the rationalist sphere we like to think we’ve gotten the path to sanity down, but, is it guaranteed that we’re right about that, or that it is memetically dominant, or that the AIs people use will guide them to it? To put it differently—would, for instance, a far leftist or rightist agree with you on what “sanity” means?
Does humanity actually know how to keep a promise and uphold a contract over the lightcone, or at least long enough for aligned ASI to enshrine it?
Incentivizing safety research: outer loop alignment failure? (Sorry, I can’t remember what this one means. Outer loop refers to the human decision making process, I’m pretty sure.)
Handoff assumes that we really truly have solved alignment once and for all. But, are we sure there are no Old Ones lurking in the unknown unknowns, no black swans to ambush us, etc., with the full might of a decade of humanity and vaguely trustworthy AI bent to the task? (I imagine the answer is simply “that’s the best we can do.”)
Insider: Wasn’t the kludge result from last December? I notice more broadly that a few concepts that are established in current safety discourse get brought up several years later as seemingly-new discoveries. I get that you can’t model discoveries we haven’t made, but swapping in ones we know of already feels a tiny bit off.
Insider: It’s depicted that the AI can be taught defensive-only cybersecurity with enough data filtering. But is it actually theoretically possible to have something that can do cyberdefense without cyberoffense? Or at least so differentially so, that defense is dominant in all attack theories?
How do you stop an AI from reasoning further than a few weeks?
Insider: is “one deduction from the frontier” actually safe?
“Adversarially robust neuralese”—does this make sense? Can it be assumed to be real?
The handoff is quite fast and centralized. If everything is so nice and tidy in 2038, are we still in such a precarious state that the deal could still break at any time, and a fast handoff is warranted?
Some further notes from after reading:
Is a “Long Reflection” best described (and dismissed) as a “Long Memetic War”?
What’s going on with mass surveillance in this scenario?
The scenario seems to assume verifiability is the only way forward for capabilities.
What’s been happening with cybersecurity by the time of the pause?
Generally, can this scenario be described as aligned to humanity? It seems like a lot gets gradually handed off to the AIs that I’m not sure I’d be fine with handing off on reflection. Did disempowerment already happen in 2031?
The scenario sort of feels like a bargain with the accelerationists.
What is your way of addressing algorithmic improvement? Is it robust against all scenarios?
Feels funky to suggest this, but have you considered using frontier AI to analyze and red-team your model?
Could a finer grained model of robot development help here?
The scenario seems to assume that no single critical mistake on our part, that the AI can sneak past us, is enough to end the game in the AI’s favor.
I chatted with Fable after I read the piece and I remember adding expansions to a few notes, but I tried to remove ideas it came up with when writing this up.