As far as I understand, superbabies would be important if, as Yudkowsky believes, SOTA mankind is unlikely to solve alignment because “humans are not at the level of intelligence where thinking they have a solution strongly correlates with them actually having a solution.”
Yudkowsky-Soares’ longer quote
Humanity often gains its knowledge by struggling, and trying, and failing, and slowly accumulating knowledge. But it doesn’t have to be that way.
Einstein was not only able to figure out general relativity; he was able to figure it out by thinking hard about the problem, even before humanity put satellites in orbit and started seeing discrepancies in their clocks with their own two eyes (as discussed in Chapter 6). He had empirical evidence, but he was able to efficiently pinpoint the right answer in response to the first quiet whispers from the empirical record, rather than needing the truth to come banging at his door.
That pathway is rarer and harder to walk, but that kind of scientific genius does exist — albeit rarely, even among the world’s best and brightest.
Humans augmented one or two steps beyond the level of researchers like Einstein or John von Neumann might begin to accurately figure out their own flaws, and correct for them, in dozens of different ways.
They might notice when they were rationalizing or falling victim to confirmation bias. They might go past the point of ever expecting a clever-sounding idea to work when actually it does not work — to the point where whenever they expect to succeed, they do succeed. They might achieve a level of competence where they still make plenty of mistakes, but they aren’t systematically overconfident (or underconfident) in tricky new domains.
Is human intelligence enhancement really a possibility? It seems so to us, having spoken with a number of biotech researchers who think that there are promising near-term angles of attack. Carefully targeted biotech-focused AI might also help accelerate the work. But from our perspective, it remains very uncertain whether a plan like this would realistically pan out. What we feel more confident in saying is that it’s a highly leveraged option that deserves a lot more investment and exploration than it’s currently getting.
We are not recommending enhancing human intelligence as the only post-AI-shutdown strategy we think humanity should heavily invest in. Rather, this is just one of many examples, and the one we currently think holds the most promise. We strongly recommend that humanity look into multiple possible non-AI paths forward, rather than putting all its eggs in one basket.
The main problems which I see with similar arguments are the following:
Mankind saw GPT-5.4 Pro and an internal OpenAI model solve two Erdos problems by applying an unnatural combination of pre-existing discoveries. How likely is it that Einstein stood on the shoulders of giants like Riemann (who studied mathematical notions like the geometry of curved spaces) or Minkowsky (who also studied an abstract Minkowski space)?
I have a conjecture that a big chunk of Yudkowsky’s reasoning requires rewriting, yielding something like “Human values are contingent… on the very features that allowed human brains to become transformative” or “Squirrelly algorithms and superstimuli regularly appear in neural nets, but aren’t THAT immune to moral reflection”.
As far as I understand, superbabies would be important if, as Yudkowsky believes, SOTA mankind is unlikely to solve alignment because “humans are not at the level of intelligence where thinking they have a solution strongly correlates with them actually having a solution.”
Yudkowsky-Soares’ longer quote
Humanity often gains its knowledge by struggling, and trying, and failing, and slowly accumulating knowledge. But it doesn’t have to be that way.
Einstein was not only able to figure out general relativity; he was able to figure it out by thinking hard about the problem, even before humanity put satellites in orbit and started seeing discrepancies in their clocks with their own two eyes (as discussed in Chapter 6). He had empirical evidence, but he was able to efficiently pinpoint the right answer in response to the first quiet whispers from the empirical record, rather than needing the truth to come banging at his door.
That pathway is rarer and harder to walk, but that kind of scientific genius does exist — albeit rarely, even among the world’s best and brightest.
Humans augmented one or two steps beyond the level of researchers like Einstein or John von Neumann might begin to accurately figure out their own flaws, and correct for them, in dozens of different ways.
They might notice when they were rationalizing or falling victim to confirmation bias. They might go past the point of ever expecting a clever-sounding idea to work when actually it does not work — to the point where whenever they expect to succeed, they do succeed. They might achieve a level of competence where they still make plenty of mistakes, but they aren’t systematically overconfident (or underconfident) in tricky new domains.
Is human intelligence enhancement really a possibility? It seems so to us, having spoken with a number of biotech researchers who think that there are promising near-term angles of attack. Carefully targeted biotech-focused AI might also help accelerate the work. But from our perspective, it remains very uncertain whether a plan like this would realistically pan out. What we feel more confident in saying is that it’s a highly leveraged option that deserves a lot more investment and exploration than it’s currently getting.
We are not recommending enhancing human intelligence as the only post-AI-shutdown strategy we think humanity should heavily invest in. Rather, this is just one of many examples, and the one we currently think holds the most promise. We strongly recommend that humanity look into multiple possible non-AI paths forward, rather than putting all its eggs in one basket.
The main problems which I see with similar arguments are the following:
Mankind saw GPT-5.4 Pro and an internal OpenAI model solve two Erdos problems by applying an unnatural combination of pre-existing discoveries. How likely is it that Einstein stood on the shoulders of giants like Riemann (who studied mathematical notions like the geometry of curved spaces) or Minkowsky (who also studied an abstract Minkowski space)?
Humans becoming superintelligent could face a potential severe tension with scaling laws of neural net efficiency;
I have a conjecture that a big chunk of Yudkowsky’s reasoning requires rewriting, yielding something like “Human values are contingent… on the very features that allowed human brains to become transformative” or “Squirrelly algorithms and superstimuli regularly appear in neural nets, but aren’t THAT immune to moral reflection”.