Why do so many futuristic forecasts fail to account for likely progress in other important fields? For instance, what happens to companies like Merge Labs and Neuralink in your scenario? If we have BCIs capable of delivering a +3–4 SD boost, wouldn’t that turn the world upside down even without any further progress in AI?
It seems quite likely to me that, in soft takeoff scenarios, we will experience a major cultural or technological shift driven by advances in other technological domains before the emergence of ASI.
I don’t currently think that BCIs will deliver a +3-4 SD boost. If I did, I probably would have written a different scenario. Can you say more about why you think they’ll have that big of an effect?
It’s not just about BCIs. There are a number of technologies capable of flipping the board on the path to singularity, beyond ASI itself. However, this forecast ignores them all and doesn’t explicitly explain what the overall research landscape in these fields will look like given the slow takeoff and the presence of powerful AI within about 10 years.
What are Eon.Systems and E11bio doing? (https://brainemulation.mxschons.com/) Does the presence of powerful AI capable of automating scientific progress significantly reduce the timeframe for WBE?
The same is true for BCI and what Merge Labs and Neuralink are doing. For example, it would be pretty obvious to try something like this (and again, we’ll have a powerful AI capable of generating better ideas): “If we train an AI to send similar patterns as lots of extra (developmentally integrated) neurons would, we’ve effectively boosted someone’s brain mass. I strongly suspect that the relevant algorithms for training such an AI are compactly specifiable, since it seems like neurons learn from simple local firing statistics.” (https://www.lesswrong.com/s/Wy8smXGZ7PsS9CpgP/p/ewZXQgzaCvzdSvtWE)
What is Michael Levin doing with his bioelectricity? Yudkowsky considers work on enhancing adult intelligence the most important priority after an AI pause — so what are he and MIRI actually doing on that front? What about the field of NeuroAI (Astera Institute, Allen Institute, Amaranth.foundation)?
My point is that fundamental breakthroughs in all these fields are turning the tide. And when we have 10+ years and a powerful AI, I think there’s a pretty good chance that some of this will work BEFORE AI takes off.
kman here. I should mention that I now think that post was very overly optimistic, and I don’t think it could be made to work without big breakthroughs in editing and delivery tech. I do still think there should be a huge project trying to make those breakthroughs, but it doesn’t seem like something a small project can make progress on. Adult enhancement seems in general a lot harder than germline engineering, which I think is quite likely to work within the next couple decades, and should be mentioned in a scenario with as long timelines as “scenario S”.
(I still think adult enhancement is worth pursuing as well. I’m currently starting a new org and planning to focus on roadmapping for adult cognitive enhancement for the next year; I hope it’ll produce a good overview of possible methods, constraints, problems that can be factored out, etc.)
(See also kman’s comment.) Besides the technical difficulties with trying the experiments, it’s quite unclear how much effect you can have on an adult brain via editing. There are important ~irreversible changes from childhood to adult brains, such as lock-in of many synapses through PNNs, pruning of long-range connections, and commitment to which axons to myelinate. (Adult brains are more capable than child brains in many respects, but this tends to be based on “crystallized intelligence”; to go bigger, I’m guessing you want more child-like learning capabilities; though that’s just speculation.) Even for IQ genes that are expressed in adult brains, it’s plausible their effects on IQ are mainly or linearly coming from childhood effects.
My uninformed impression of E11 is that it’s very cool, but also is not the bottleneck to WBE or similar. The connectome isn’t the issue, the issue is actually getting the neural behavior that does the important work. (Cf. maybe this: https://www.youtube.com/watch?v=FHQfmJEpRmU ) IDK about Eon, but my impression from listening to others is that their announcement was super misleading / nothingburger. Separately, I think WBE in particular is much more risky than most other HIA approaches.
What is Michael Levin doing with his bioelectricity?
I highly doubt this is relevant.
In general there is the issue that it’s hard to test what you’re interested in; and you’re not constrained to the natural manifold (as with reprogenetics), so there’s little guarantee that e.g. increasing reaction time would generalize to increased philosophical insight or whatnot.
All that said, of course we should be investing much more into this.
some kind of BCI reservoir compute with computations that the brain can learn to use well but that wetware is ill-suited to (someone gave the example of matrix inversions);
networking with some sort of neural organoids to get “more cortex”.
All of these are quite speculative in terms of whether they would help much and how feasible they are, so I’m not so optimistic.
BUT, reprogenetics is very likely to be able to give +2 SD boost, and pretty likely to be able to give +3-6 SD or more boost. (I think the main uncertainty is around something like “do genetic influences on intelligence saturate (i.e. hit diminishing returns) strongly with each other without saturating with non-genetic causes”. It’s hard to tell and I think it’s unlikely, but not totally implausible.) Further, I think this can work in one shot, i.e. on the first generation.
Germline engineering will likely be able to give us a boost like that at some point. This is probably the best current resource if you want to learn more.
Also wondering about this question. I think in a slow enough AI takeoff scenario these technologies start to matter and should be factored into scenario planning. But maybe they deliberately left it out as it would be too weird to regular readers.
(Imagine in the “you’re a everyday citizen” scenario the main character decides to get cognitive enhancement in 2035, it would make for a much less approachable read afterwards.)
Why do so many futuristic forecasts fail to account for likely progress in other important fields? For instance, what happens to companies like Merge Labs and Neuralink in your scenario? If we have BCIs capable of delivering a +3–4 SD boost, wouldn’t that turn the world upside down even without any further progress in AI?
It seems quite likely to me that, in soft takeoff scenarios, we will experience a major cultural or technological shift driven by advances in other technological domains before the emergence of ASI.
I don’t currently think that BCIs will deliver a +3-4 SD boost. If I did, I probably would have written a different scenario. Can you say more about why you think they’ll have that big of an effect?
It’s not just about BCIs. There are a number of technologies capable of flipping the board on the path to singularity, beyond ASI itself. However, this forecast ignores them all and doesn’t explicitly explain what the overall research landscape in these fields will look like given the slow takeoff and the presence of powerful AI within about 10 years.
For example:
What are gene_smith and kman doing in the Plan A world? (https://www.lesswrong.com/posts/JEhW3HDMKzekDShva/significantly-enhancing-adult-intelligence-with-gene-editing) We live in a world where AI significantly accelerates scientific discovery, but the problem of delivering edited genes to the brain remains unresolved until the singularity? Isn’t Significantly Enhancing Adult Intelligence With Gene Editing Possible?
What are Eon.Systems and E11bio doing? (https://brainemulation.mxschons.com/) Does the presence of powerful AI capable of automating scientific progress significantly reduce the timeframe for WBE?
The same is true for BCI and what Merge Labs and Neuralink are doing. For example, it would be pretty obvious to try something like this (and again, we’ll have a powerful AI capable of generating better ideas): “If we train an AI to send similar patterns as lots of extra (developmentally integrated) neurons would, we’ve effectively boosted someone’s brain mass. I strongly suspect that the relevant algorithms for training such an AI are compactly specifiable, since it seems like neurons learn from simple local firing statistics.” (https://www.lesswrong.com/s/Wy8smXGZ7PsS9CpgP/p/ewZXQgzaCvzdSvtWE)
What is Michael Levin doing with his bioelectricity? Yudkowsky considers work on enhancing adult intelligence the most important priority after an AI pause — so what are he and MIRI actually doing on that front? What about the field of NeuroAI (Astera Institute, Allen Institute, Amaranth.foundation)?
My point is that fundamental breakthroughs in all these fields are turning the tide. And when we have 10+ years and a powerful AI, I think there’s a pretty good chance that some of this will work BEFORE AI takes off.
kman here. I should mention that I now think that post was very overly optimistic, and I don’t think it could be made to work without big breakthroughs in editing and delivery tech. I do still think there should be a huge project trying to make those breakthroughs, but it doesn’t seem like something a small project can make progress on. Adult enhancement seems in general a lot harder than germline engineering, which I think is quite likely to work within the next couple decades, and should be mentioned in a scenario with as long timelines as “scenario S”.
(I still think adult enhancement is worth pursuing as well. I’m currently starting a new org and planning to focus on roadmapping for adult cognitive enhancement for the next year; I hope it’ll produce a good overview of possible methods, constraints, problems that can be factored out, etc.)
(See also kman’s comment.) Besides the technical difficulties with trying the experiments, it’s quite unclear how much effect you can have on an adult brain via editing. There are important ~irreversible changes from childhood to adult brains, such as lock-in of many synapses through PNNs, pruning of long-range connections, and commitment to which axons to myelinate. (Adult brains are more capable than child brains in many respects, but this tends to be based on “crystallized intelligence”; to go bigger, I’m guessing you want more child-like learning capabilities; though that’s just speculation.) Even for IQ genes that are expressed in adult brains, it’s plausible their effects on IQ are mainly or linearly coming from childhood effects.
My uninformed impression of E11 is that it’s very cool, but also is not the bottleneck to WBE or similar. The connectome isn’t the issue, the issue is actually getting the neural behavior that does the important work. (Cf. maybe this: https://www.youtube.com/watch?v=FHQfmJEpRmU ) IDK about Eon, but my impression from listening to others is that their announcement was super misleading / nothingburger. Separately, I think WBE in particular is much more risky than most other HIA approaches.
I don’t believe the claim about simulating something close enough to neurons; see the previous point. Re/ BCIs, see also: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=ff9c8PHtuFKfwE6Yj
I highly doubt this is relevant.
In general there is the issue that it’s hard to test what you’re interested in; and you’re not constrained to the natural manifold (as with reprogenetics), so there’s little guarantee that e.g. increasing reaction time would generalize to increased philosophical insight or whatnot.
All that said, of course we should be investing much more into this.
I agree that BCIs probably wouldn’t do that much. (Though:) There are a few threads that one can imagine, such as:
networking people together and having them spend a bunch of time learning how to use that;
prosthetically increased long-range connectivity;
some kind of BCI reservoir compute with computations that the brain can learn to use well but that wetware is ill-suited to (someone gave the example of matrix inversions);
networking with some sort of neural organoids to get “more cortex”.
All of these are quite speculative in terms of whether they would help much and how feasible they are, so I’m not so optimistic.
BUT, reprogenetics is very likely to be able to give +2 SD boost, and pretty likely to be able to give +3-6 SD or more boost. (I think the main uncertainty is around something like “do genetic influences on intelligence saturate (i.e. hit diminishing returns) strongly with each other without saturating with non-genetic causes”. It’s hard to tell and I think it’s unlikely, but not totally implausible.) Further, I think this can work in one shot, i.e. on the first generation.
If that’s a crux, happy to give more info.
Germline engineering will likely be able to give us a boost like that at some point. This is probably the best current resource if you want to learn more.
I agree with the overall point! We tried to think through a bunch of technological progress but no doubt were missing some important stuff.
See, for example:
- Military power
- Robots (mentioned throughout the secnario)
- Lie detectors
I haven’t looked into BCIs much at all it’s plausible they should play a bigger role earlier.
Also wondering about this question. I think in a slow enough AI takeoff scenario these technologies start to matter and should be factored into scenario planning. But maybe they deliberately left it out as it would be too weird to regular readers.
(Imagine in the “you’re a everyday citizen” scenario the main character decides to get cognitive enhancement in 2035, it would make for a much less approachable read afterwards.)