Thanks for writing this, I agree that people have underinvested in writing documents like this. I agree with many of your points, and disagree with others. For the purposes of this comment, I’ll focus on a few key disagreeements.
My model of this variety of reader has an inside view, which they will label an outside view, that assigns great relevance to some other data points that are not observed cases of an outer optimization loop producing an inner general intelligence, and assigns little importance to our one data point actually featuring the phenomenon in question. Consider skepticism, if someone is ignoring this one warning, especially if they are not presenting equally lethal and dangerous things that they say will go wrong instead.
There are some ways in which AGI will be analogous to human evolution. There are some ways in which it will be disanalogous. Any solution to alignment will exploit at least one of the ways in which it’s disanalogous. Pointing to the example of humans without analysing the analogies and disanalogies more deeply doesn’t help distinguish between alignment proposals which usefully exploit disanalogies, and proposals which don’t.
Alpha Zero blew past all accumulated human knowledge about Go after a day or so of self-play, with no reliance on human playbooks or sample games.
It seems useful to distinguish between how fast any given model advances during training, and how fast the frontier of our best models advances. AlphaZero seems like a good example of why we should expect the former to be fast; but for automated oversight techniques, the latter is more relevant.
if a textbook from one hundred years in the future fell into our hands, containing all of the simple ideas that actually work robustly in practice, we could probably build an aligned superintelligence in six months.
Maybe one way to pin down a disagreement here: imagine the minimum-intelligence AGI that could write this textbook (including describing the experiments required to verify all the claims it made) in a year if it tried. How many Yudkowsky-years does it take to safely evaluate whether following a textbook which that AGI spent a year writing will kill you?
This situation you see when you look around you is not what a surviving world looks like. The worlds of humanity that survive have plans.
It would be great to have a well-justified plan at this point. But I think you’re also overestimating the value of planning, in a way that’s related to you using the phrase “miracle” to mean “positive model violation”. Nobody throughout human history has ever had a model of the future accurate enough to justify equivocating those two terms. Every big scientific breakthrough is a model violation to a bunch of geniuses who have been looking at the problem really hard, but not quite at the right angle. This is why I pushed you, during our debates, to produce predictions rather than postdictions, so that I could distinguish you from all the other geniuses who ran into big model violations.
Maybe one way to pin down a disagreement here: imagine the minimum-intelligence AGI that could write this textbook (including describing the experiments required to verify all the claims it made) in a year if it tried. How many Yudkowsky-years does it take to safely evaluate whether following a textbook which that AGI spent a year writing will kill you?
Hmm, okay, here’s a variant. Assume it would take N Yudkowsky-years to write the textbook from the future described above. How many Yudkowsky-years does it take to evaluate a textbook that took N Yudkowsky-years to write, to a reasonable level of confidence (say, 90%)?
If I know that it was written by aligned people? I wouldn’t just be trying to evaluate it myself; I’d try to get a team together to implement it, and understanding it well enough to implement it would be the same process as verifying whatever remaining verifiable uncertainty was left about the origins, where most of that uncertainty is unverifiable because the putative hostile origin is plausibly also smart enough to sneak things past you.
Sorry, I should have been clearer. Let’s suppose that a copy of you spent however long it takes to write an honest textbook with the solution to alignment (let’s call it N Yudkowsky-years), and an evil copy of you spent N Yudkowsky-years writing a deceptive textbook trying to make you believe in a false solution to alignment, and you’re given one but not told which. How long would it take you to reach 90% confidence about which you’d been given? (You’re free to get a team together to run a bunch of experiments and implementations, I’m just asking that you measure the total work in units of years-of-work-done-by-people-as-competent-as-Yudkowsky. And I should specify some safety threshold too—like, in the process of reaching 90% confidence, incurring less than 10% chance of running an experiment which kills you.)
Get me to adopt a solution wrong in a particular direction, like a design that hands the universe over to them? I can maybe figure out the first time through who’s out to get me, if it’s 200 Yudkowsky-years. If it’s 200,000 Yudkowsky-years I think I’m just screwed.
Get me to make any lethal mistake at all? I don’t think I can get to 90% confidence period, or at least, not without spending an amount of Yudkowsky-time equivalent to the untrustworthy source.
Thanks for writing this, I agree that people have underinvested in writing documents like this. I agree with many of your points, and disagree with others. For the purposes of this comment, I’ll focus on a few key disagreeements.
There are some ways in which AGI will be analogous to human evolution. There are some ways in which it will be disanalogous. Any solution to alignment will exploit at least one of the ways in which it’s disanalogous. Pointing to the example of humans without analysing the analogies and disanalogies more deeply doesn’t help distinguish between alignment proposals which usefully exploit disanalogies, and proposals which don’t.
It seems useful to distinguish between how fast any given model advances during training, and how fast the frontier of our best models advances. AlphaZero seems like a good example of why we should expect the former to be fast; but for automated oversight techniques, the latter is more relevant.
Maybe one way to pin down a disagreement here: imagine the minimum-intelligence AGI that could write this textbook (including describing the experiments required to verify all the claims it made) in a year if it tried. How many Yudkowsky-years does it take to safely evaluate whether following a textbook which that AGI spent a year writing will kill you?
It would be great to have a well-justified plan at this point. But I think you’re also overestimating the value of planning, in a way that’s related to you using the phrase “miracle” to mean “positive model violation”. Nobody throughout human history has ever had a model of the future accurate enough to justify equivocating those two terms. Every big scientific breakthrough is a model violation to a bunch of geniuses who have been looking at the problem really hard, but not quite at the right angle. This is why I pushed you, during our debates, to produce predictions rather than postdictions, so that I could distinguish you from all the other geniuses who ran into big model violations.
Infinite? That can’t be done?
Hmm, okay, here’s a variant. Assume it would take N Yudkowsky-years to write the textbook from the future described above. How many Yudkowsky-years does it take to evaluate a textbook that took N Yudkowsky-years to write, to a reasonable level of confidence (say, 90%)?
If I know that it was written by aligned people? I wouldn’t just be trying to evaluate it myself; I’d try to get a team together to implement it, and understanding it well enough to implement it would be the same process as verifying whatever remaining verifiable uncertainty was left about the origins, where most of that uncertainty is unverifiable because the putative hostile origin is plausibly also smart enough to sneak things past you.
Sorry, I should have been clearer. Let’s suppose that a copy of you spent however long it takes to write an honest textbook with the solution to alignment (let’s call it N Yudkowsky-years), and an evil copy of you spent N Yudkowsky-years writing a deceptive textbook trying to make you believe in a false solution to alignment, and you’re given one but not told which. How long would it take you to reach 90% confidence about which you’d been given? (You’re free to get a team together to run a bunch of experiments and implementations, I’m just asking that you measure the total work in units of years-of-work-done-by-people-as-competent-as-Yudkowsky. And I should specify some safety threshold too—like, in the process of reaching 90% confidence, incurring less than 10% chance of running an experiment which kills you.)
Depends what the evil clones are trying to do.
Get me to adopt a solution wrong in a particular direction, like a design that hands the universe over to them? I can maybe figure out the first time through who’s out to get me, if it’s 200 Yudkowsky-years. If it’s 200,000 Yudkowsky-years I think I’m just screwed.
Get me to make any lethal mistake at all? I don’t think I can get to 90% confidence period, or at least, not without spending an amount of Yudkowsky-time equivalent to the untrustworthy source.