I agree with Ryan Greenblatt that precise timelines for AGI don’t matter that much in terms of actionable information, but big jumps in the chance of things going crazy within a few years can matter a lot more.
I guess this would have to quantified, but probably I want to simply respond: “Wrong.”
I’ve been told several times by funders something like: “I don’t want to fund reprogenetics; AGI is coming too soon for that to matter.” Probably they are just incoherent, but if they were coherent, maybe they’d mean something like “90% chance of AGI in the next 20 years”. So then it would totally matter for decision-making whether that number is 90% or 40%! I would think.
In fact,
One of the few robust ways out is human intelligence amplification;
How do we know that human intelligence amplification is a robust way out?
Maybe more intelligent humans just race to AGI faster because of coordination problems. Maybe more intelligent terrorists release better bio-weapons. Maybe after every reasonable boost for biological human brains we still aren’t smart enough to solve alignment, perhaps because it’s totally intractable or perhaps because we can’t really get that much smarter, and we muddle along for awhile trying to stop everyone from building AGI until something out of left field wipes us out.
(I do favor human intelligence amplification/augmentation, robust imitation learning, and uploading as particularly promising paths)
I’m unsure whether or not we disagree, because I think our disagreement would be quantitative (probabilities, perhaps relative priorities) rather than qualitative (which considerations are coherent, plausible, relevant, etc.). My basic direct answer would be: Because effectful wise action is more difficult than effectful unwise action, and requires more ideas / thought / reflection, relatively speaking; and because generally humans want to do good things.
Maybe more intelligent humans just race to AGI faster because of coordination problems.
True, maybe.
I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI. So more people around / above that threshold is differentially good in terms of technical progress. (It could still be bad overall, e.g. if you think there’s a fixed-rate process of reflection that gets outrun faster in those worlds.)
I think having a lot more super smart people makes other aspects of the situation better. For example, the leaders of AGI capabilities research would be smarter—which is bad in that they make progress faster, but good in that they can consider arguments about X-risk better. Another example: it’s harder to give even a plausible justification for plunging into AGI if you already have a new wave of super smart people making much faster scientific progress in general, e.g. solving diseases.
I think / hope that current memecrafting means that young smart people growing up will have the background sense that AGI is special and is one of the ~three things you really do not mess around with ever. BUT, if in fact it becomes clear that young super bright kids are actually, currently, growing up much more enthusiastic about AGI, that cuts against this a lot, and would significantly cut against my motivation towards reprogenetics (well, it would make me prioritize much higher stuff that affects the memesphere of newborn geniuses).
Maybe more intelligent terrorists release better bio-weapons.
AFAIK this is largely sci-fi? I mean you can find a few examples, but very very few super smart people are trying to do huge amounts of damage like that; and it’s even way harder to get a large and/or very expert group together to try to do that; and without a large/expert group it’s really hard to get super-high-effect things working, especially in biology, like that.
Maybe after every reasonable boost for biological human brains we still aren’t smart enough to solve alignment, perhaps because it’s totally intractable or perhaps because we can’t really get that much smarter, and we muddle along for awhile trying to stop everyone from building AGI until something out of left field wipes us out.
Yeah, true, this is plausible; I still think our chances are significantly in those worlds, but your point stands.
For example, the leaders of AGI capabilities research would be smarter—which is bad in that they make progress faster, but good in that they can consider arguments about X-risk better.
This mechanism seems weak to me. For example, I think the leaders of all AI companies are considerably smarter than me, but I am still doing a better job than they are of reasoning about x-risk. It seems unlikely that making them even smarter would help.
(All else equal, you’re more likely to arrive at correct positions if you’re smarter, but I think the effect is weak.)
Another example: it’s harder to give even a plausible justification for plunging into AGI if you already have a new wave of super smart people making much faster scientific progress in general, e.g. solving diseases.
If enhanced humans could make scientific progress at the same rate as ASI, then ASI would also pose much less of an x-risk because it can’t reliably outsmart humans. (Although it still has the advantage that it can replicate and self-modify.) Realistically I do not think there is any level of genetic modification at which humans can match the pace of ASI.
That all isn’t necessarily to say that human intelligence enhancement is a bad idea; I just didn’t find the given reasons convincing.
We probably disagree a little. I’d bring up a few points. E.g. I’d point out that you and I reason about X-risk better than them in large part due to the fact that we pay attention to people who are smarter than us and who are good at / ahead of the curve on reasoning about X-risk. E.g. I’d point out that more intelligence leads to more intellectual slack (meaning, if you can think faster, a given argument becomes less costly in time (though maybe not in opportunity) to come to understand). E.g. I’d point out that wisdom (in this sense: https://www.lesswrong.com/posts/fzKfzXWEBaENJXDGP/what-is-wisdom-1) is bottlenecked on considering and integrating many possibilities, which is a difficult cognitive task.
But, I agree it’s not that strong a reason.
Realistically I do not think there is any level of genetic modification at which humans can match the pace of ASI.
Yeah I agree, but that’s not the relevant threshold here. More like, can humanity feasibly get smart enough soon enough to be making a bunch of faster progress in material conditions, such that justifications of the form “AGI would give us much faster progress in material conditions” lose most of their (perhaps only apparent) force? I think probably we can.
There were just four reasons right? Your three numbered items, plus “effectful wise action is more difficult than effectful unwise action, and requires more ideas / thought / reflection, relatively speaking; and because generally humans want to do good things”. I think that quotation was the strongest argument. As for numbered item #1, I don’t know why you believe it, but it doesn’t seem clearly false to me, either.
I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI.
Simply put, it’s a harder problem. More specifically, it’s got significantly worse feedback signals: it’s easier to tell when / on what tasks your performance is and is not going up, compared to telling when you’ve made a thing that will continue pursuing XYZ as it gets much smarter. You can also tell because progress in capabilities seems to accelerate given more resources, but that is (according to me) barely true or not true in alignment, so far.
My own experience (which I don’t expect you to update much on, but this is part of why I believe these things) is that I’m really smart and as far as I can tell, I’m too dumb to even really get started (cf. https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html). I’ve worked with people who are smarter than I am, and they also are AFAICT totally failing to address the problem. (To be clear, I definitely don’t think it’s “just about being smart”; but I do think there’s some threshold effect.) It’s hard to even stay focused on the problem for the years that it apparently takes to work through wrong preconceptions, bad ideas, etc., and you (or rather, I, and ~everyone I’ve directly worked with) apparently have to do that in order to understand the problem.
I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI.
This is also my guess, but I think required intelligence thresholds (for the individual scientists/inventors involved) are only weak evidence about relative problem difficulty (for society, which seems to me the relevant sort of “difficulty” here).
I’d guess the work of Newton, Maxwell, and Shannon required a higher intelligence threshold-for-making-progress than was required to help invent decent steam engines or rockets, for example, but it nonetheless seems to me that the latter were meaningfully “harder” for society to invent. (Most obviously in the sense that their invention took more person-hours, but I suspect they similarly required more experience of frustration, taking on of personal risk, and other such things which tend to make given populations less likely to solve problems in given calendar-years).
required intelligence thresholds (for the individual scientists/inventors involved) are only weak evidence about relative problem difficulty (for society, which seems to me the relevant sort of “difficulty” here).
This sounds right, yeah. If I had to guess, I would guess AGI alignment is both kinds of problem (Maxwell/Faraday equations, and rockets).
Oh, IDK. Thermonuclear weapons and viral human infectiousness gain of function research? I just mean, there’s probably a few things that we want everyone to know not to do; the Noahide Laws, if you will.
There is some definition of intelligence, where by that definition sufficiently intelligent humans wouldn’t build AGI that kills them and all their children, simply because getting killed by robots would suck. Of course, it’s not at all obvious that genetic engineering would get pointed at that definition of intelligence.
I guess this would have to quantified, but probably I want to simply respond: “Wrong.”
I’ve been told several times by funders something like: “I don’t want to fund reprogenetics; AGI is coming too soon for that to matter.” Probably they are just incoherent, but if they were coherent, maybe they’d mean something like “90% chance of AGI in the next 20 years”. So then it would totally matter for decision-making whether that number is 90% or 40%! I would think.
In fact,
One of the few robust ways out is human intelligence amplification;
the only method that we can have reasonable confidence can be made to work any time soon is reprogenetics (https://www.lesswrong.com/posts/jTiSWHKAtnyA723LE/overview-of-strong-human-intelligence-amplification-methods);
you don’t have to be super confident that AGI will take 50 years to think reprogenetics is worthwhile (https://tsvibt.blogspot.com/2022/08/the-benefit-of-intervening-sooner.html); and
reprogenetics is funding constrained.
How do we know that human intelligence amplification is a robust way out?
Maybe more intelligent humans just race to AGI faster because of coordination problems. Maybe more intelligent terrorists release better bio-weapons. Maybe after every reasonable boost for biological human brains we still aren’t smart enough to solve alignment, perhaps because it’s totally intractable or perhaps because we can’t really get that much smarter, and we muddle along for awhile trying to stop everyone from building AGI until something out of left field wipes us out.
(I do favor human intelligence amplification/augmentation, robust imitation learning, and uploading as particularly promising paths)
I’m unsure whether or not we disagree, because I think our disagreement would be quantitative (probabilities, perhaps relative priorities) rather than qualitative (which considerations are coherent, plausible, relevant, etc.). My basic direct answer would be: Because effectful wise action is more difficult than effectful unwise action, and requires more ideas / thought / reflection, relatively speaking; and because generally humans want to do good things.
True, maybe.
I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI. So more people around / above that threshold is differentially good in terms of technical progress. (It could still be bad overall, e.g. if you think there’s a fixed-rate process of reflection that gets outrun faster in those worlds.)
I think having a lot more super smart people makes other aspects of the situation better. For example, the leaders of AGI capabilities research would be smarter—which is bad in that they make progress faster, but good in that they can consider arguments about X-risk better. Another example: it’s harder to give even a plausible justification for plunging into AGI if you already have a new wave of super smart people making much faster scientific progress in general, e.g. solving diseases.
I think / hope that current memecrafting means that young smart people growing up will have the background sense that AGI is special and is one of the ~three things you really do not mess around with ever. BUT, if in fact it becomes clear that young super bright kids are actually, currently, growing up much more enthusiastic about AGI, that cuts against this a lot, and would significantly cut against my motivation towards reprogenetics (well, it would make me prioritize much higher stuff that affects the memesphere of newborn geniuses).
AFAIK this is largely sci-fi? I mean you can find a few examples, but very very few super smart people are trying to do huge amounts of damage like that; and it’s even way harder to get a large and/or very expert group together to try to do that; and without a large/expert group it’s really hard to get super-high-effect things working, especially in biology, like that.
Yeah, true, this is plausible; I still think our chances are significantly in those worlds, but your point stands.
This mechanism seems weak to me. For example, I think the leaders of all AI companies are considerably smarter than me, but I am still doing a better job than they are of reasoning about x-risk. It seems unlikely that making them even smarter would help.
(All else equal, you’re more likely to arrive at correct positions if you’re smarter, but I think the effect is weak.)
If enhanced humans could make scientific progress at the same rate as ASI, then ASI would also pose much less of an x-risk because it can’t reliably outsmart humans. (Although it still has the advantage that it can replicate and self-modify.) Realistically I do not think there is any level of genetic modification at which humans can match the pace of ASI.
That all isn’t necessarily to say that human intelligence enhancement is a bad idea; I just didn’t find the given reasons convincing.
We probably disagree a little. I’d bring up a few points. E.g. I’d point out that you and I reason about X-risk better than them in large part due to the fact that we pay attention to people who are smarter than us and who are good at / ahead of the curve on reasoning about X-risk. E.g. I’d point out that more intelligence leads to more intellectual slack (meaning, if you can think faster, a given argument becomes less costly in time (though maybe not in opportunity) to come to understand). E.g. I’d point out that wisdom (in this sense: https://www.lesswrong.com/posts/fzKfzXWEBaENJXDGP/what-is-wisdom-1) is bottlenecked on considering and integrating many possibilities, which is a difficult cognitive task.
But, I agree it’s not that strong a reason.
Yeah I agree, but that’s not the relevant threshold here. More like, can humanity feasibly get smart enough soon enough to be making a bunch of faster progress in material conditions, such that justifications of the form “AGI would give us much faster progress in material conditions” lose most of their (perhaps only apparent) force? I think probably we can.
All of them, or just these two?
There were just four reasons right? Your three numbered items, plus “effectful wise action is more difficult than effectful unwise action, and requires more ideas / thought / reflection, relatively speaking; and because generally humans want to do good things”. I think that quotation was the strongest argument. As for numbered item #1, I don’t know why you believe it, but it doesn’t seem clearly false to me, either.
So I wrote:
Simply put, it’s a harder problem. More specifically, it’s got significantly worse feedback signals: it’s easier to tell when / on what tasks your performance is and is not going up, compared to telling when you’ve made a thing that will continue pursuing XYZ as it gets much smarter. You can also tell because progress in capabilities seems to accelerate given more resources, but that is (according to me) barely true or not true in alignment, so far.
My own experience (which I don’t expect you to update much on, but this is part of why I believe these things) is that I’m really smart and as far as I can tell, I’m too dumb to even really get started (cf. https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html). I’ve worked with people who are smarter than I am, and they also are AFAICT totally failing to address the problem. (To be clear, I definitely don’t think it’s “just about being smart”; but I do think there’s some threshold effect.) It’s hard to even stay focused on the problem for the years that it apparently takes to work through wrong preconceptions, bad ideas, etc., and you (or rather, I, and ~everyone I’ve directly worked with) apparently have to do that in order to understand the problem.
This is also my guess, but I think required intelligence thresholds (for the individual scientists/inventors involved) are only weak evidence about relative problem difficulty (for society, which seems to me the relevant sort of “difficulty” here).
I’d guess the work of Newton, Maxwell, and Shannon required a higher intelligence threshold-for-making-progress than was required to help invent decent steam engines or rockets, for example, but it nonetheless seems to me that the latter were meaningfully “harder” for society to invent. (Most obviously in the sense that their invention took more person-hours, but I suspect they similarly required more experience of frustration, taking on of personal risk, and other such things which tend to make given populations less likely to solve problems in given calendar-years).
This sounds right, yeah. If I had to guess, I would guess AGI alignment is both kinds of problem (Maxwell/Faraday equations, and rockets).
what are the other ~two?
Oh, IDK. Thermonuclear weapons and viral human infectiousness gain of function research? I just mean, there’s probably a few things that we want everyone to know not to do; the Noahide Laws, if you will.
There is some definition of intelligence, where by that definition sufficiently intelligent humans wouldn’t build AGI that kills them and all their children, simply because getting killed by robots would suck. Of course, it’s not at all obvious that genetic engineering would get pointed at that definition of intelligence.