Why couldn’t the goal of prosaic alignment actually be to bring us to the point where alignment research itself is automated? After all, prosaic alignment research ought to take into account worlds where superalignment works.
Forza
Karma: 63
could be that some kinds of “WBE research”, especially ones such as connectomics that don’t try to reverse engineer the brain’s learning algorithm, are net-helpful. See Byrnes 2023: “Connectomics seems great from an AI x-risk perspective”. I a little bit believe this regarding connectomics. I don’t strongly believe this about connectomics or about any research, a priori, because either it doesn’t lead to true, extremely helpful, human WBEs, or else it does and it’s powerful and therefore it throws off powerful PBEs on the way there.
How do you feel about PBE type projects like this specifically for the purposes of solving alignment?
It’s not just about BCIs. There are a number of technologies capable of flipping the board on the path to singularity, beyond ASI itself. However, this forecast ignores them all and doesn’t explicitly explain what the overall research landscape in these fields will look like given the slow takeoff and the presence of powerful AI within about 10 years.
For example:
What are gene_smith and kman doing in the Plan A world? (https://www.lesswrong.com/posts/JEhW3HDMKzekDShva/significantly-enhancing-adult-intelligence-with-gene-editing) We live in a world where AI significantly accelerates scientific discovery, but the problem of delivering edited genes to the brain remains unresolved until the singularity? Isn’t Significantly Enhancing Adult Intelligence With Gene Editing Possible?
What are Eon.Systems and E11bio doing? (https://brainemulation.mxschons.com/) Does the presence of powerful AI capable of automating scientific progress significantly reduce the timeframe for WBE?
The same is true for BCI and what Merge Labs and Neuralink are doing. For example, it would be pretty obvious to try something like this (and again, we’ll have a powerful AI capable of generating better ideas): “If we train an AI to send similar patterns as lots of extra (developmentally integrated) neurons would, we’ve effectively boosted someone’s brain mass. I strongly suspect that the relevant algorithms for training such an AI are compactly specifiable, since it seems like neurons learn from simple local firing statistics.” (https://www.lesswrong.com/s/Wy8smXGZ7PsS9CpgP/p/ewZXQgzaCvzdSvtWE)
What is Michael Levin doing with his bioelectricity? Yudkowsky considers work on enhancing adult intelligence the most important priority after an AI pause — so what are he and MIRI actually doing on that front? What about the field of NeuroAI (Astera Institute, Allen Institute, Amaranth.foundation)?
My point is that fundamental breakthroughs in all these fields are turning the tide. And when we have 10+ years and a powerful AI, I think there’s a pretty good chance that some of this will work BEFORE AI takes off.
Why do so many futuristic forecasts fail to account for likely progress in other important fields? For instance, what happens to companies like Merge Labs and Neuralink in your scenario? If we have BCIs capable of delivering a +3–4 SD boost, wouldn’t that turn the world upside down even without any further progress in AI?
It seems quite likely to me that, in soft takeoff scenarios, we will experience a major cultural or technological shift driven by advances in other technological domains before the emergence of ASI.
That’s not true if you and your employees are on earth. It’s like saying, “I’m building data centers in neutral waters, and no one can manage me.” Nope. It doesn’t work that way. You’re subject to the jurisdiction of the country whose flag your ship flies
if you’re worried that a nuclear weapon with yield Y might ignite the atmosphere, it doesn’t help to first test a nuclear weapon with yield 0.1×Y, and then if the atmosphere hasn’t been ignited yet, next try testing one with yield 0.2×Y, etc.” I.e., I sometimes (not always!) see people talking about gradual AI progress without having a clear and plausible (to me) mechanism by which the earlier steps actually de-risk the later steps.
In this example, people believe that each subsequent nuclear test could solve the problem of atmospheric ignite in the future.
Why do you call this an “atrocity” rather than a “mistake”? It seems obvious to me that the people creating ASI do not agree with your assessment of the risks and believe they are doing good. They are not seeking to deliberately destroy humanity.
Do you believe that people who think a pause in the development of modern AI paradigm it’s -EV should consider you “the greatest evil”?
because you might be wrong, or people might genuinely not understand that you’re right? There are many people who believe that their lives are in danger of a lot of things: vaccines against covid, GMOs, climate change, etc. They also sincerely believe that they are right and can consider scientists evil because they endanger their lives and the lives of their family. But the problem is that if they start equating any people who disagree with them with the evil that deliberately wants to destroy them, it radicalizes them and promoting enmity. Is this a rational approach when every person, confident that technology N can kill him, starts openly hating the scientists who develop it?
I’ve always disliked the idea of people who assess risk differently being called “evil.” There’s a big difference between “I want to kill everyone, so I’m creating technology X” and “I disagree with the risk assessment of this technology and want to make the world a better place, so I’m creating it.”
I think the lesswrong community has become too radicalized in considering itself so epistemologically right that it calls people evil who simply disagree with them about the probability of p(doom).
UPD: The problem with this radicalization of the community is also that it repels many AI researchers who do not consider themselves evil and are less inclined to change their point of view on the risks posed by people who consider them evil and are probably not averse to killing them for disagreeing with them.
I don’t understand the argument about the trillion expected lives. Why is it assumed that after the singularity we will have trillions more human children? Why is this not a mind upload and the end of reproduction in its current form? And the people alive at the time of the singularity will simply exist in virtual worlds with AI agents?
I find it strange to ask an ASI to emulate a human so that you can raise in your virtual world. And conditionally 8 billion improved uploaded human minds may not strive to emulate as many variations of human brains as possible in order to fill the universe with them.
cryonics is expensive, unpopular and unavailable in most countries of the world. This is also a situation where young and rich people in first world countries buy themselves a reduction probability of their own death, at the expense of a guaranteed deprivation of the chances of life of the poor and old people.
I don’t understand what state do you think other technologies are in from the moment Agent 3 appeared until everything got out of control? What about Agent-3 help with WBE? Or increasing the intelligence of adults to solve the alignment?
Why in the worlds where we already have powerful enough agents that haven’t yet gotten out of control, we see them working on self-improvement, but don’t see game-changing progress in other areas?
That seems like a false dichotomy. People working on AI safety can also engage in governance and communicate their positions to the media and policymakers, likely garnering even more attention thanks to their technical expertise. In fact, the individuals who have attracted the most media attention recently are precisely those with technical expertise in AI safety. So, the choice isn’t necessarily between working full-time on technical alignment research and working full-time on governance.