FAQ: Isn’t AGI coming too soon for reprogenetics to help?
Introduction
I think reprogenetics (human germline genomic engineering) can be done in a widely acceptable and beneficial way, and should be pursued aggressively. In particular, as a strong background motivation of mine, I think accelerating strong reprogenetics is probably the best way to enable strong human intelligence amplification; and I think strong HIA is among the best ways to decrease existential risk from AGI.
A very common objection to caring much about reprogenetics is that AGI seems very likely to come soon—say, within a decade or two. (Here I mean “actual” AGI—the kind that probably doesn’t already exist—the kind that has fluid intelligence and AI advantages for recursive self-improvement, which together make it likely to take over the world shortly after being created.) The objection is fairly straightforward:
AGI will probably come within a decade or two. If that’s going to happen, then even if a new cohort of brilliant humans were born today, they would still be children, or would at best have barely begun contributing ideas for how to avoid extinction. Any supposed benefit, denominated in percentage points of AGI existential risk averted, is small. Therefore, reprogenetics is too slow; and if you’re going to work on human intelligence amplification, you should work on adult HIA.
Now, of course this is true to an extent. If AGI comes within 15 years, reprogenetics is almost totally useless. To the extent you believe that will happen, you believe reprogenetics is quantitatively less useful in expectation. Also, having a faster HIA method would of course be great.
However, I believe that this line of reasoning has led to a very mistaken underallocation of funding, talent, and other resources towards human intelligence amplification in general, and reprogenetics in particular. So, I would like to push back in a few ways:
On general strategic grounds, there’s altruistic alpha in working on HIA.
(See the section “HIA, part of your nutritionally complete portfolio”.)
You shouldn’t be that confident that AGI is coming within a decade or two.
(See the section “Against confident short timelines”.)
Putting HIA on a legibly fast and good trajectory might indirectly decrease the motivation to create AGI.
(See the section “HIA may indirectly slow down AGI capabilities”.)
Even given a high probability of AGI coming soon, HIA still has a large positive impact.
(See the section “HIA has substantial impact even with short timelines”.)
Other methods for strong HIA are not at all obviously faster than reprogenetics.
(See the section “Adult HIA methods aren’t fast either, absent big investment”.)
(Note that this isn’t a comprehensive fair-and-balanced report, but rather a collection of arguments in one direction. For example, I won’t here discuss reasons that HIA could increase existential risk from AGI. Also, some but not all of these arguments rely on the assumption that alignment is very difficult.)
HIA, part of your nutritionally complete portfolio
-
HIA is super neglected. Would you rather be the 3000th person working on AI safety, the 300th person working on AI regulation, or the 3rd person working on accelerating HIA?
-
There’s broad philanthropic alpha in working on HIA. It’s tractable (to accelerate), neglected, and important. In particular, because it’s somewhat taboo (though probably less than you think), there’s relatively low-hanging fieldbuilding fruit to pick.
-
We must attend to all plausible points of intervention.
The world cannot be allowed to be destroyed by AGI. We must consider intervening on this course of events at every possible point.
HIA is widely agreed (sometimes publicly, more often in private) to be one of the few real hopes for humanity’s survival. This is a place where we must not drop the ball. Currently the ball is being dropped.
-
Uncoordinated resource allocation leads to unbalanced portfolios.
A correct belief that Sector A is more important than Sector B would lead someone to focus on Sector A exclusively when deciding where to allocate resources. This kind of focus is reasonable, given that one can’t spend forever evaluating and comparing all the different possible sectors; you have to focus on one or a few.
People allocate talent, funding, and other resources without checking the preexisting portfolio of allocations. (This is understandable, since that portfolio isn’t written down in any 1 place or in any 10 places.) Thus, they don’t automatically track that a sector is underresourced.
It’s widely accepted that AGI is coming soon, and therefore HIA is not such a useful investment. Even if this belief is correct, this belief leads to HIA being very underresourced within the global joint philanthropic portfolio.
This is a collective mistake. This is as true now as it was when I first wrote it 3 years ago. The candle is still burning down.
Against confident short timelines
I don’t in fact think that confident short timelines (say, “>80% within 15 years” or similar) make much sense. As yet, I have not heard a clear and convincing case that AI research has already uncovered the engines that would produce a smarter-than-human general intelligence. The main arguments for AGI coming soon boil down, I think, to the rapid rise in capabilities and to the use of AI in AI research.
Regarding the rapid rise:
We have large gaps in performance between AIs and humans (sample complexity for learning, ability to generate novel good concepts).
Also, we have an apparent explanation for the rapid rise in capability: hoovering up big piles of internet data for quasi-imitation. Current AIs are skillful / knowledgeable approximately when there’s a bunch of human training data for that thing. Of course, you need more elements if you want to explain all of the rapid rise, such as unhobbling via harnesses and RL, as well as computer advantages (scaled RLVR, speed, parallelism, copying, inexpensiveness).
But the core explanation of “the bulk of these capabilities come from human demonstration” holds water, as far as I know. In particular, this explanation says that current AIs got their capabilities via some route other than accumulation of crystallized intelligence that was produced by their own fluid intelligence. Namely, they got their capabilities via “copying” (broadly construed) crystallized human intelligence originally generated by human fluid intelligence.
This seems to largely explain away the rapid rise in capabilities, without invoking current AIs having much fluid intelligence. Combined with the apparent gap, with humans still well ahead on general fluid intelligence, I don’t see how anyone gets to being very confident that we already have most or all of the ideas that would be needed for AGI.
Regarding the use of AI in AI research:
I expect the most important kinds of research to not be accelerated much, because they are not bottlenecked by coding in the first place. Unless you already think we’re close to making AGI for other reasons, so that what’s left is mainly the kind of research that is greatly accelerated by current or near-future AI, this partial acceleration shouldn’t change your timelines much.
For previous discussion, see “Do confident short timelines make sense?” and “Views on when AGI comes and on strategy to reduce existential risk”. I’m open to debating this with anyone who’d like to make a serious, public case for confident short timelines.
HIA may indirectly slow down AGI capabilities
The main stated justification for pursuing AGI capabilities is that AI / AGI would bring abundance for humanity. It stands to reason that if there were a credible, workable plan to get the (supposed) benefits of AGI without the huge existential risk (and other disempowerment of humans), then there would be less motive to develop AGI and it would be harder to justify pursuing AI capabilities. (See “5.1. Abundance makes less motive to make AGI”.)
I don’t know how large this effect would be. Presumably not very large. Presumably many people trying to increase AI capabilities just want money, power, status, or other selfish things, and would find some other justification. I’m not sure though; it’s hard to put upper or lower bounds on the importance of underlying spiritual currents in society of hope, motivation, the longer-term future, and so on.
HIA has substantial impact even with short timelines
Approximate summary of this section:
We should play to our outs; outs are scarce; HIA is an out.
-
In dire situations, if you’re trying to be strategic about winning, it’s advisable to play to your outs. That is, avoid the temptation to focus on incremental tractable gains that don’t actually increase the chances of overall success. Instead, aim at paths to overall success, including by setting yourself up to take advantage of such opportunities, even if any specific path is unlikely.
-
Working towards a pause / stop / ban on AGI progress is a top priority. But what is a pause for? We pause, and then what? How do we more robustly prevent existential risk? Hopefully, we can ban Red AGI research, but probably not Blue AGI research progress. (Though social / political pressure might possibly be able to significantly slow down even Blue research.) In the longer run, how do we avoid making unaligned AGI? HIA is a way to improve our long-run chances by giving humanity more brainpower to find good answers to that question.
-
Suppose you do get a pause on AGI progress, but you don’t have a plan for how to more robustly stop existential risk. More specifically, suppose that progress is not being made towards the outs. Then the pause is being wasted: for the moment you’re bailing water out of the boat as it leaks in, but at some point more and more leaks will be sprung. You have to also be patching the hull. Ten years from now, when you have hopefully updated toward slightly longer timelines, you’ll regret not getting started on the longer-term solutions back in 2026. I know I regret not working on this a decade ago, rather than bashing my head against the AGI alignment problem. HIA will make some progress on its own by default, but this argument goes through quantitatively—you’ll regret not having quantitatively accelerated HIA if you could have.
Another lens on this: If you have a single long-term out, then the value of pulling the long-term out forward in time by one year is equal to the value of pushing the pause to last for one year longer.
-
Very short timelines are pretty intractable. If AGI is actually coming in 5 years by default, unless you think alignment is easy, there just aren’t many outs. Of course we still want to try, but intractability does weigh in the prioritization calculation.
-
Pulling world-saving conditions forward in time is very valuable unless you’re very confident extinction will have already happened.
If you make an intervention such that the world would get saved in 40 years rather than 50 years, you’ve decreased existential risk by an amount equal to the probability of AGI coming between 40 and 50 years from now. Unless you have very confident short timelines to AGI, this still decreases total existential risk by a few percentage points.
Decreasing total existential risk by a percentage point is very valuable!
As an illustration, you could have a hazard rate of 5%. In other words, at the beginning of each year (thus, conditioning on the fact that you’ve survived so far), you think there’s a 5% chance that AGI comes within the next year. This distribution has mean <20 years and median <14 years. In this case, the world being saved at year 40 vs. at year 50 is worth a 5% decrease in existential risk.
As another illustration, you could have a hazard rate of 10%. This is an extremely aggressive (confident) timeline. This distribution has median <7 years. In this case, the world being saved at year 40 vs. at year 50 is still worth a ~1% decrease in existential risk! Plug into your calculator:
For a more detailed model with numbers, see “The benefit of intervening sooner”.
-
AGI alignment is extremely hard. Therefore solving AGI alignment is not that helpful of an out.
This shouldn’t be surprising. Minds, especially superhumanly intelligent minds, are the most complex entities that exist. It’s not surprising if getting them to do some really specific thing (make the universe end up great for humans) is extremely difficult, unlike any challenge humans have faced before. As an analogy, how hard would it be for the Ancient Greeks to get to the moon? What if they have only 15 years? Do you think that task is easier or harder than solving the mysteries of values, intelligence, creativity, self-modification, and so on, well enough to put a mind in an unnatural but stable state, within 15 or even 50 years?
The problem of AGI alignment has major structural and technical obstacles, crucial and insoluble deep mysteries, and appears to be cognitively unnatural to work on.
Therefore, “just solve alignment now, fast” is not a feasible out.
-
HIA is an out we can play to.
Alignment is very difficult because it’s a very difficult intellectual (conceptual, philosophical, technical) problem; it’s bottlenecked on really insightful creative careful thinking, not on money or other resources. So, strong HIA might lead to humanity being able to solve alignment.
If alignment is still too hard, HIA might lead to humanity being able to handle AGI existential risk in some other way. The first-rate intellects that benefit from strong HIA would probably have a better shot than us at thinking of ways to avoid existential risk.
For example, they might:
develop next-generation HIA
figure out how to keep in force a stronger ban on AGI
make progress on material problems, more immanently removing motivation to take risks with AGI
improve society’s decision-making in general, leading to a strong social consensus against risking AGI
Indeed, the whole point of making an aligned AGI in the first place is that you need intelligence, but you can’t amplify your own intelligence enough. Therefore, an alternative to making aligned AGI is to figure out how to amplify human intelligence enough.
Adult HIA methods aren’t fast either, absent big investment
A variant of the argument against reprogenetics goes like this:
AGI is coming very soon. Reprogenetics has a built-in delay of at least 15 or 20 years while the kids grow up, before it actually helps. Instead, we should work on adult HIA, which will be able to help much sooner.
Of course, overall, this logic is valid and compelling. If I believed we could do (strong) HIA sooner than a couple decades, I would work on that. If there were actually a Manhattan-Project-scale project for adult human intelligence amplification, I would probably drop what I was working on in reprogenetics and join that effort. I would have some hope of success—assuming that we would have available the scientists, equipment, experimental volunteers, and money that would be needed to run several experimental investigations in parallel.
However, short of a Manhattan Project, I’m somewhat skeptical of adult HIA being a faster bet than reprogenetics. There are several reasons, which I’ll list here. But I want to summarize the main reasons more briefly:
It’s pretty expensive to intervene on an adult brain. Further, it’s not actually clear how to intervene on adult brains to make them smarter, so you’d be doing exploratory research. It’s probably hard to make adult brains much smarter, so you’d probably have to do a lot of exploratory research. Each experiment probably takes at least months (the intervention wouldn’t work instantly), and more likely years (to develop new tech). This means that developing the method involves lots of costly, lengthy, serial experiments. So actually adult HIA takes a long time to develop—like, think decades.
The reasons in more detail:
-
Reprogenetics has good-enough data on intelligence; adult HIA does not.
To know what genes increase intelligence, we can look at the billions of humans already living. This doesn’t come remotely close to telling us everything there is to know about genes and intelligence, but it most likely has already told us enough to make a future child be likely to possess a world-class intellect. (What’s left to do is stem cell biotech.)
There’s no comparable large dataset for adult HIA interventions. If you tried an intervention, you would wait a year, collect some data—and then you would have one (1) datapoint. With a single experimental volunteer, you’d even run into challenges trying to collect longitudinal data. You’d need some way to measure cognitive capability that is at least robust to practice, let alone to adversarial hacking (e.g. if you want to do distributed science / self-experimentation).
It may be that any large increase in intelligence must take a long time (years, say), even if it can be done in adults. It might, for example, take a person many months or years to develop the new ways of thinking that are unlocked by an adult intervention. Without that time to develop new ways of thinking, there may not be many clear demonstrations that you’ve really been put on that trajectory. This further increases the time needed for experiments (increasing the serial time needed, and the fungible cost per datapoint).
Each new protocol may take months or years to develop. Thus, even though there’s a faster feedback loop of end-to-end testing for adult interventions compared to reprogenetics, the loop itself is slow.
Reprogenetics, in contrast, probably doesn’t need end-to-end testing, because of the abundant natural experiment data. Reprogenetics does require innovation in things like stem cell biology. But that innovation does have fast feedback loops—significantly faster than end-to-end adult HIA, in most cases. Further, these stem cell biotech problems can mostly be tested in animals, whereas end-to-end adult intelligence amplification interventions can’t really be tested in animals (except for safety), because there’s no great (human-generalizable) measurement of animal intelligence.
-
Reprogenetics would probably work for strong HIA; adult interventions may or may not work well for strong HIA.
There is a clear path for strong reprogenetics to work: in vitro gametogenesis (or another epigenomic correctness method), plus strong genomic vectoring (iterated meiotic selection, chromosome selection, and/or iterated CRISPR editing).
There is not a clear path for adult HIA. I do not believe anyone has a near-term-feasible way that we have strong reason to think would work. (I’m somewhat excluding whole brain emulation. It would work, though I think it is actually extremely difficult and would take longer than reprogenetics. But more importantly I think it’s very dangerous.) See “Overview of strong human intelligence amplification methods”.
Without a clear path to success, adult HIA is a less compelling bet. Making big bets like this is hard enough when there’s a clear path. Without a clear path, the probability of overall success is multiplicatively diminished (compared to just having the uncertainty around execution). And, it would be hard to coordinate talent and funding.
-
There are general technical obstacles to adult HIA.
Adult brains have lost various kinds of flexibility that child brains possess. Many neurites (long-range axons, dendrites, synapses) have been pruned; genesis of neural structures has greatly decreased; structures have solidified (e.g. via perineuronal nets).
The childhood developmental program is evolved for canalization—making important structures in the body form properly. This development program is no longer operating in adulthood. Therefore, tweaks to brain function in adulthood won’t be met by compensatory mechanisms that keep everything functioning well, compared to how much a genetic tweak will be met with compensatory mechanisms during development.
The genome of a single cell isn’t easy to access and manipulate, but it’s fairly easy compared to accessing an adult brain at scale. The blood-brain barrier, the skull, the meninges, and immune reactions make it very inconvenient to deliver drugs, transplant cells, or implant electrodes into the brain at scale.
-
Reprogenetics has strong momentum, in terms of scientific foundations, user motivation, and user on-ramp.
Reprogenetics is constituted by statistical genetics, reproductive science, and stem cell biology. Each of these fields makes progress on its own. E.g. there’s lots of work on building biobanks, constructing polygenic scores, sussing out genetic causality, characterizing germline cells, characterizing and supporting healthy gametogenesis and embryogenesis, reprogramming cells, producing gametes in vitro, measuring genes, editing genes, and so on. For this reason, reprogenetics is already fairly close to being technically feasible.
The technology stack of reprogenetics for decreasing disease risks is almost identical to the stack for increasing intelligence (except for the specific genes). While many people would be interested in giving their children genomic foundations for higher expected intelligence, even more people would like to give them foundations for long, healthy, mentally healthy lives. That means there’s a large personal, financial, and social incentive for everyone involved (scientists, funders, parents, regulators) to make this technology work well and at scale.
Most of the different hypothetical future reprogenetics technologies share most of their biotech stack and social stack (motivations, politics, regulation). On the other hand, adult HIA is somewhat more diverse (brain drugs vs. brain editing vs. stem cell transplants vs. BCIs). So investments in science and fieldbuilding projects for reprogenetics may be somewhat more synergistic with each other.
I don’t see such a large-scale motivator for adult HIA technologies; BCIs, brain drugs, brain editing, and stem cell transplants all seem fairly niche, maybe possibly useful at some point for treating some specific illnesses.
For these reasons, my expectation is that legal obstacles will be tractable, as the component technologies of reprogenetics are proven safe and effective.
Millions of babies are already born through IVF. Reprogenetics would be a small change to the procedure from the parents’s perspective (perhaps even less burdensome, via the advanced reprotech needed for reprogenetics). Thus there’s already a large user base who could easily adopt reprogenetics. Most adult HIA methods (anything involving brain surgery or gene editing) would probably be somewhat hard to scale up a lot; some may be quite easy to scale up (e.g. non-editing brain drugs). A comparison point might be other non-deep brain surgeries, or organ transplantation.
Scale is important for HIA because, however hard it is to amplify intelligence, it’s probably even harder to amplify other cognitive capacities such as wisdom, philosophical insight, sanity, creativity, persistence, curiosity, etc. If some such qualities are needed to achieve the progress that humanity most needs, and they can’t easily be intervened on, then they would have to arise by chance. The plan here is to make strong HIA feasible at scale; hopefully enough people will want to avail themselves of that opportunity; and hopefully some of the resulting people will be both first-rate intellects and also in possession of those other needed qualities.
-
Therefore, adult HIA is fairly likely to take a rather long time.
Due to the technical difficulties, iterations may be months or years. E.g. each scaling up of BCI electrode count by an order of magnitude may be a multi-year research project; several such scale-ups may be needed for true strong HIA via BCIs.
Due to the lack of a clear method, good abundant data, and fast training-robust intelligence tests, many iterations may be required to work out a method.
Due to the lack of motivation and a market, there may not be so many resources available. Therefore, exploratory experiments probably can’t be massively parallelized.
Therefore, adult HIA would likely require many serial experiments, or in other words, a long campaign of research.
Now, all of this having been said, I could easily be wrong. I’ve focused largely on reprogenetics, because that’s what I think will work. I’m enthusiastic about roadmapping and research on adult HIA in general, and on specific methods. Hopefully, those interested in adult HIA would take these as non-dealbreaker constraints, to chew through or work around or creatively break.
Takeaways
HIA should be among the top priorities in a promising portfolio for avoiding existential risk from AGI.
You probably shouldn’t have very confident short AGI timelines.
Even if you do have confident short timelines, you should still consider HIA a top priority.
HIA is extremely neglected—it has much less talent, funding, and other resources compared to what it ought to have.
There’s lots to work on, to accelerate HIA via reprogenetics. See e.g. “Berkeley Genomics Project seeking hires (and collabs)” and “Projects that might help accelerate strong reprogenetics”.
Essentially this isn’t happening right now. I think it very likely starts happening within a few years, but will be slow, maybe slower than humanity, and as a result won’t be able to quickly fix the problem of being slow.
Humanity should count as actual AGI, but it remains slow. Similarly, I think LLMs that do prosaic RSI of automatically building the next model (crucially including formulation of new RL tasks/environments/graders) will be actual AGI in the sense that they can (on their own, without humanity’s input) eventually generate and accumulate as much crystallized intelligence as humanity would.
Yet there is a bottleneck of the speed at which they learn novel deep skills (generate and crystallize new pieces of intelligence), going from new RL tasks to new models to new teaching moments that inspire new tasks for the new models that are ready to use the opportunity. This keeps the process at the slow pace of building models rather than at the fast pace of generating tokens, and so AI advantages don’t quickly snowball into fast RSI that changes the fundamental nature of the AI.
Eventually fast RSI happens, but it plausibly takes many years of slow-learning prosaic RSI (for it) to figure out how to make that happen, and humans could end up being faster at figuring it out. As with any basic research, there is no trend that meaningfully predicts how many years it takes. All this time, there’s actual AGI of slow-learning model building that can figure out anything eventually, but only slowly. This AGI might be the lesser factor of danger in triggering fast RSI, compared to the vast amount of compute its usefulness finances, which enables ambitious experiments and rapid scaling of prototypes.
Pretraining very sample inefficiently reconstructs cognitive skills that left evidence about their nature in the text. RL training can use relatively short formulations of tasks/environments/graders to generate even novel pieces of cognitive skills needed to solve the tasks (that the authors of the tasks didn’t necessarily have, as they didn’t necessarily know how to solve the tasks, or how to do so efficiently). In this way, RL training is both sample efficient (with respect to the short formulations of tasks, and the tiny insights each task promotes), and a way of producing novel pieces of intelligence that can be crystallized in the new models.
What’s currently missing is automated formulation of new RL tasks/environments/graders in response to gaps in existing crystallized intelligence (in the current model) with respect to the situations/problems it comes in contact with. I estimate that LLMs of 2031 will be more capable than Mythos 5 by about 3x as much as Mythos 5 is more capable than Opus 4.5+ (strictly before Opus 5). This is likely a gap notably wider than the Sonnet-Mythos gap, above the current frontier models that are already in a weight class capable of producing strong technical results and spontaneous competent cyberoffense. Taking that further step to the models of 2031 is probably insufficient for leaps of insight that quickly show how to make RSI go fast. But this is very likely enough for a base model of 2031 to be sufficiently teachable (using RL tasks) to end up learning how to formulate new RL tasks/environments/graders on its own, starting the slow-learning prosaic RSI process of building the next model automatically.
Thanks. (I have read this with interest, but don’t have much of an overall reply.) A couple comments:
It’s a weird middle ground, where it is fooming but pretty slowly. (I mean, as a human, I would think that, i.e. I would experience time on a scale that’s somewhat faster than humanity’s foom.) And it kinda only half has the A in AGI. If it fully had the A (or I guess, ISGI, in silico general intelligence) then it would foom much faster.
Maybe kinda. But my guess is that this would be kind of like calling [hominid evolution by natural selection on genetic variation] an AGI, or calling [the ecosystem of the Earth throughout all of time] an AGI. I mean it would probably be extremely faster & scarier than evolution, but still.
It reconstructs a lot of them to a significant extent, but very much misses a lot of them to a significant extent, I think.
This seems plausible-ish. I would weakly expect a lot of plateaus, e.g. due to highly correlated taste, but not strongly and I haven’t thought about it much. (Maybe you’re pricing that in to “slow”.)
Another potential route for this is that reprogenetics can help solve the fertility crisis, which can reduce the incentives to build AGI. My thinking here is that currently it’s pretty risky to have children (e.g. if even one of them ends up unhealthy or “unsuccessful” / low status, it’s a big drain on one’s own finances/status/happiness) which I think contributes to low desire to have more kids in a lot of societies, and low average fertility is a motivation for building AGI in order to stop/reverse the resulting economic or civilizational decline. Reprogenetics could help fix this root cause.
Although I suppose this depends on reprogenetics being cheap enough. If it’s too expensive (and can’t be or isn’t subsidized), it might have the opposite effect, where people feel like it’s a de facto requirement of parenthood (e.g. they feel really bad about their unenhanced kids having to compete with enhanced kids) and opt out of having children because they can’t afford it.
Interesting. This seems plausible as a (smallish?) contributing factor. (Though I don’t personally feel very grounded in what people actually say to themselves or each other about why they’re building AGI.)
This probably isn’t the main cause, since I think it’s probably a lot less risky today than at previous times in history (better nutrition, medical care, social safety nets, etc.). But it stands to reason that it’s a contributing factor.
Incidentally, reprogenetics and more broadly the advanced reprotech that’s involved, could help support fertility in ways beyond affecting the life outcomes of the child. For example:
In vitro oogenesis (making eggs from stem cells)
Enables women to get eggs with their DNA while avoiding egg retrieval. Egg retrieval is uncomfortable (ovarian swelling, hormones), painful (injections, surgery), expensive ($10k++), unreliable, and inconvenient (confusing system, blood draws, ultrasound monitoring, financing, etc.). IVO would be a much simpler process (just take one biopsy; the rest is in the lab) and in the long run should way less expensive.
Egg retrieval doesn’t work well for women >35yo (say), and doesn’t work at all for older women. Some women don’t have eggs for other reasons (e.g. cancer treatment). There are also couples who would have same-sex gametes (gay or trans). So IVO would make it go from impossible to possible for such people to have genetic kids.
In vitro spermatogenesis (making sperm from stem cells)
The same considerations apply as for IVO.
Further, older men have worse sperm (genetic damage and probably epigenetic aberration). This leads to higher rates of disorders in children. High-quality IVS, maybe with genomic vectoring to correct damage, could alleviate that.
Egg and zygote support.
Various labs and startups are working on tech such as in vitro maturation of eggs, in vitro ovarian follicle culture, and repairing damaged cell-division machinery needed for oogenesis and early embryogenesis.
These technologies decrease the attrition in the egg retrieval (or IVO) --> implantable embryo pipeline, quantitatively alleviating the costs of that pipeline.
One reprogenetic target would be “genes that support a healthy successful pregnancy”, helping to alleviate the burden of failed implantation or miscarriage.
This is a bit more sci-fi and less related to reprogenetics, but artificial wombs would relieve the harms / burdens of pregnancy (though it may take a while to end up less expensive than surrogacy).
For several reasons this seems important, and I think it’s achievable. The research is expensive, but once the methods are worked out, they probably can be innovated to be pretty inexpensive.
That would be a sad outcome. In the longer run, there’s probably some restructuring of society that would be good, that takes advantage of material abundance to make it so that parents are reasonably assured that their children’s lives will be well worth living, whether or not they use reprogenetics.
Yeah it’s a lot less risky today, but we’re also a lot more risk averse when it comes to children (e.g. not letting them travel/play unsupervised like we used to). Also, this reason seems to have played a large role in my own family’s fertility decisions.
Yeah sure does seem like it. I wonder why. It does seem bad for parents to feel like they maybe shouldn’t have children because they might be held responsible for not doing everything right (though of course they should be held responsible for doing some basics right). I want reprogenetics to generally be taken as supererogatory, partly for this reason.
I think another way in which more children being born is civilizationally good and postpones AGI is that people with children are somewhat busy with children, and have their priorities shifted. A tail risk for you seems fine; a tail risk for your kids, less so. People think with clearer heads about risks to their kids, and so will be less likely to push for acceleration. Also, people having kids lowers the costs of other people having kids. We’re in a kind of a crappy valley now with few people having kids, but if we bounce back to many people having kids, then the cost of childcare is lower due to huge economies of scale in having kids, and that helps a lot to motivate others.
I did update a little bit against this when I heard (vaguely) that a lot of senior researchers at frontier AI labs do have kids. On the other hand, now that I think of it, I wonder if there’s kinda morally speaking two categories of AI capabilities researchers:
Older ones; tend to have kids, doesn’t worry about downside risks, doesn’t think AGI is a thing
Younger ones; tend to not have kids, kinda worries about downside risks, does think AGI is a thing, but has some rationalization because of Sinclair’s Razor or similar
I can imagine the first to be easier to switch to blocking AI progress once they feel the AGI than the latter as they don’t have rationalizations in place.
There’s also a third category, that of successionists which would ~likely change their minds if they had kids or if a lot of people dear to them had kids etc. as that (in my experience) moves your aliefs to “there’s something worth fighting for in humanity even if all current people are on average awful because the original state is so pure and full of potential for wonder and good things that a universe without this is one less valuable”
That’s a good reason for hope! One would have thought they could be moved by the pedigree of those who have already flipped, such as Hinton etc., but here’s to hoping.
Plausible. Though to be honest, I have trouble getting into that mindset, and it’s plausible to me that it’s sufficiently skew to what seems natural to me, that it would not respond in this way. In other words, if someone is a staunch successionist, that suggests there’s already some wonky stuff generating that stance, which may not go away.
Are there many successionists at all? How common are they in actual AI capabilities research? Do they in fact tend to have kids? How old do they tend to be?
I don’t think that reprogenetics would solve the crisis. IIRC (upd: link) Asian countries have the same problem of an overcompetition for prestigious jobs which are scarce almost by definition. Even if enhancing the kids was cheap and efficient (which I doubt, as detailed in my comment), competition would shift to a competition among the enhanced kids.
The fertility crisis in countries like the USA was described in five roundups by Zvi and has many other causes like housing (which is also a problem in S. Korea!), lack of couples, etc.
When they enter the workforce, they would be mostly competing with older, unenhanced or less enhanced adults. Although I just realized this brings up another potential issue of parents waiting longer to have kids in order to get access to better enhancement tech, or being afraid of subsequent generations outcompeting their own kids.
This could be a somewhat sad dynamic. Though not that sad, and maybe even good, if the technology is actually following a very rapid increase.
For safety reasons, the strength of reprogenetics will most likely plateau after we get to “strong reprogenetics” (being able to nudge several polygenic traits by several SDs). Pushing too far on any polygenic score or on any trait carries big risks (too tall, too big of a brain, too high a metabolism, etc., would each come with health problems at some point). This Osborne effect should only persist while the derivative is substantial, and the higher the derivative is, the sooner the approximate plateau is reached.
More realistically, I think that in fact it’s usually not a good idea to wait to have kids for that reason (and I don’t especially plan to wait for that reason). Some reasons for that:
For many / most people, if they want to have kids, much or most of why they want that isn’t that sensitive to the changes that future reprogenetics can offer. For example, many people want their kids to be happy and healthy; sure intelligence or other capabilities sound nice, but a lot of people shrug their shoulders about it. Weaker reprogenetics can get much of the health benefits (though not much of the intelligence benefits). (Maybe they would have a different revealed preference when faced with the practical choice.)
There’s various reasons that sooner is better.
Personally, I want to get lots of practice parenting well, whether or not I’m going to have kids who benefit from strong HIA, but especially so if I am.
Personally, I want to do some parenting while I’m not very old.
Personally, I just want to have kids.
Having an older sibling is probably a solid way to give an HIA kid a good environment for intellectual growth by having an intellectual quasi-peer.
More generally, if the world trends upward in various ways, then there are probably advantages to getting a head start in time, like making an investment earlier.
I want to take an attitude that’s more wholesomely future-oriented. A bit more concretely, I don’t mind having my life be to a substantial degree in service of future children, and I would hope that many kids would take on an attitude like that. So it’s a good outcome to have a weak-HIA kid who will be the 15-year-old camp counselor for the 6-year-old strong-HIA kids, for example.
The development of the technology will probably be at least somewhat unpredictable, so you could easily be waiting 15 years, which many people would find regretful.
Society has a terrible track record with this kind of technology (see what liberal eugenics ended up looking like in practise—it wasn’t nazism, but it wasn’t anything positive, either).
Human intelligence is a very complicated system, playing out over the course of an individual’s life. All we have for the moment is evidence of small, local genetic effects. It would be a grave error to assume that these effects remain additive, without side-effects, when you combine them all. And measuring these side effects takes decades. Genetics isn’t like physics: you can’t go off-distribution (like sending a rocket to the moon) and assume the results hold up.
I think it wasn’t a coincidence that liberal eugenics went so wrong. The theory overpromised and was unsound, especially as you got into more details. It also played to people’s prejudices. This seems to be repeating the exact pattern—overpromising, unsound on the details, playing into people’s prejudices. Of course you don’t intend for things to go sour, and are advocating this for the best of motives—but the liberal eugenicists didn’t intend that, either, and good motives don’t guarantee good results.
At the very least, I’d want to see a detailed argument for why this won’t go wrong the way it’s gone wrong in the past. “This technology will only be used sensibly by sensible people under sensible governments” is not an assumption you get to make for free.
I don’t think it makes sense for society to write off a possible future technology because it shares some general ideas with a past evil. In particular, reprogenetics applied to diseases can alleviate enormous amounts of suffering and death—think something like preventing many millions of cases of disease, if the technology is adopted at scale. If you don’t believe this technical statement, we could try to clarify that (though there are existing papers that would do a fine job, and a more comprehensive FAQ hasn’t been written yet). If we admit the large upside, that’s already plenty of reason to consider the technology more in depth.
To elaborate, first I want to distinguish three things:
Reprogenetics is a class of assisted reproductive technologies, some current (polygenic embryo screening, mitochondrial replacement therapy (kinda)) and some future (e.g. iterated meiotic selection, chromosome selection, iterated gene editing). (See “Methods for strong human germline engineering” for more detail.)
Eugenics is an ideology, most active in the first half of the 20th century, that holds that there are Good and Bad traits—implicitly universally, and as judged by someone in power (the state, the doctors, the scientists, the elites, etc.).
Genomic emancipation is an ideology that says it’s good to make biotechnology to empower parents on behalf of their future children. More philosophically, it says: “The birthright of human spirits includes fuller self-unfolding via self-sovereignty over their own genomes.”. (See “Genomic emancipation”.)
Society does not have a terrible record with reprogenetic technology, because such technology has not existed until the past decade.
Society does have a terrible track record with the ideology of eugenics. One could generalize that, and say that society has a terrible track record with the general idea of somehow influencing heredity of future children.
However, this has a strange consequence of bundling genomic emancipation together with eugenics. Genomic emancipation negates the core idea of eugenics, and is therefore opposed to eugenics. Please see: “Genomic emancipation contra eugenics”.
This bundling is, in this sense, incoherent. One reason you may nevertheless want to bundle these opposed ideologies together is something like this:
Or in your own words:
I agree that this would be a bad outcome. I agree that it’s a very legitimate concern. However, I want to largely refute this frame.
Of course, it’s true that it doesn’t matter too much what I personally do, at all. Insofar as it does matter what I personally do, though, my motives definitely matter, because they’ll guide my actions and tell other people what to expect from my actions. But more to the point, I’m not saying “well, as long as I have pure motives, it’ll all work out fine”.
There is a fallacy here, but I’m not sure what the name for it is. It’s something like the “action-counterfactual fallacy”, or the “non-planning counterfactual fallacy”, or the “non-agentic counterfactual fallacy”, or “off-policy counterfactual” or “orphaned action” or something. In this fallacy, you argue against action X by saying “imagine action X is taken[, and everything else is held constant]; then the results would be poor”. The problem is that this is the wrong counterfactual. The actual counterfactual should be more like “imagine action X is taken[, and also the rest of the plan that generated X is followed]; then such and such would happen”. In this counterfactual, the results may be quite a lot better than in the fallacious counterfactual.
So, first of all, I’m not saying “I’ll do reprogenetics, yes… but with good intentions!”. I’m saying I’ll do reprogenetics, and here’s my philosophy, and also, here’s a bunch of actions which I think will make this go well, which flow from that philosophy. Just a few examples of concrete actions:
Lobby for “The principle of genomic liberty”.
Accelerate stronger forms of reprogenetics, which has various benefits including making it less expensive and therefore more widely accessible.
Fund research to improve cross-ancestry generalization of PGSes, to make reprogenetics more equally accessible.
Create innovation-positive ethical norms for reprogenetics, such as not pressuring parents into specific genomic choices; and enforce these norms (e.g. via a 3rd party auditor).
You wrote:
Allow me to make another distinction:
What is likely to happen, in a future with reprogenetics?
What are the different possibilities, for futures with reprogenetics?
What can be done to change between such possibilities?
What should be done to change between such possibilities?
If we were to follow the best plan we could muster, what is likely to happen in a future with reprogenetics?
If we were to follow the best plan we could muster, would the result be good in expectation? (What about in distribution, which is relevant to “Is this likely to be good for almost everyone, and awful for approximately no one?”, which is relevant to constructing acceptable civil compromises.)
You’re asking for an argument for why this won’t go wrong. I do think it is likely to go well.
However, I’m rather less interested in making a detailed argument for that, compared to the activity of constructing better ways to make it go well. That’s because:
It seems like a more interesting and useful activity; it leads to ideas for how to make the world better.
It doesn’t make much sense to argue that it “would be good”, due to the orphaned-action fallacy. What’s the counterfactual? We can’t even ask the right question without thinking through the positive vision in a bunch of detail.
I expect constructing a good vision to be more persuasive than just arguing that it “would be good”, and I do want to persuade more people to work on this. (Partly so that there are more people constructing the positive vision, working towards it if it’s feasible, steering away from bad versions, and hopefully noticing if the bad versions really are too likely. Please see “Appendix: Why envision genomic emancipation?”.)
Predicting the future is hard. A lot of the reason I think reprogenetics is ethically acceptable, morally good, and altruistically a good bet, is not some detailed confident prediction about the future, but rather a mesh of partial predictions, ethics, and other considerations.
More simply, the world is very complicated, so it’s basically impossible to give a detailed confident argument for what will happen in the future. I’ve listed all the perils I could think of in “Potential perils of germline genomic engineering”. Endlessly debating attempted predictions seems unproductive.
If there’s a plausible case that developing reprogenetics would be net bad for humanity, then of course that would argue strongly against proceeding without a clearer picture. I don’t see such a case.
Here are some of the downside risks I see that seem plausible, AND related to eugenics, AND somewhat difficult to prevent through organized action (by the sector, by the public, and by liberal governments):
There’s a “soft eugenics” of social pressure, where people make genomics choices partly because of social approval or disapproval for specific genomic choices. I think this is kinda bad, but I don’t think it comes close to flipping the sign of reprogenetics, unless it’s very strong. I think this should be combatted, but doesn’t come close to making a plausible case that reprogenetics would be net bad.
There’s imposition by authoritarian governments of policies about reproduction.
China, for reference, had mandatory contraception/sterilization as part of its one-child policy in the late 20th century and early 21st century, and employs forced sterilization as part of its persecution of the Uyghurs. Some countries, such as Saudi Arabia, have mandatory premarital screening for certain genetic conditions, with subsequent social pressure against marriage for double-carrier couples.
One could imagine a country that forces its citizens to use reprogenetics, and/or forces specific genomic choices for those who do use reprogenetics. This seems fairly plausible to happen in some forms. However, it seems to me that the plausible versions aren’t that bad, and the worst versions aren’t that plausible.
One possibility on the pareto frontier of plausible & bad according to me would be authoritarian regimes forcing their subjects to make their kids be more obedient, more conformist, less brave.
(Note, though, that they could already be doing that, as polygenic embryo screening has been available for about 5 years; time will tell.)
Increased apathy or cruelty towards the frail, because “you should have been genetically modified”. (I don’t know how plausible this is to actually affect too much material policy as a consequence of reprogenetics, but that possibility doesn’t seem wild, and would certainly be bad. As a touchpoint, prenatal screening for Down’s syndrome seems to have become widespread, and it has decreased the number of Down’s births; and also conditions for people with Down’s seem to have gotten better over the same period. So in that one case, reprogenetics doesn’t seem to cause apathy or cruelty towards the frail.)
Overall these seem bad if they happen, but not so bad and probable that they are at all in the ballpark of making reprogenetics net bad in expectation. Some of them might be alleviable. (E.g. we could push for international treaties against coercive reprogenetics. I wouldn’t especially hold out too much hope on that, given that China’s abuse of the Uyghurs hasn’t provoked enough response to stop it; but on the other hand China did crack down on embryo gene editing in the He Jiankui affair, seemingly in response to international pressure.)
Some other possibilities seem also not very likely:
For example: There’s a fairly deep association between eugenics and racism. Does that also suggest reprogenetics is associated with racism?
It seems like people who want to be racist are just going to be racist.
Reprogenetics is about effects of genes, measured by trait differences between individual people; “race differences” for traits don’t enter it at all.
You could draw some sort of subtle line between things, like: if we’re doing reprogenetics, then we’re saying genes have some sort of biological effect on some of a person’s traits; and so you think genes are real; and therefore you think race is real (??).
But IDK, I just don’t buy this causal arrow much. Would we also argue against clinical genetic screening (e.g. to direct testing and treatment) because thinking that genes are real is making people more racist? I mean, I don’t want to be too dismissive, I have not looked into this specific worry. Maybe that causal arrow does exist? But I’m skeptical that it’s a big arrow.
I think eugenical ideology stems from high modernism and picks up genes as a tool, not the other way around. Reprogenetics in the framework of genomic emancipation is a tool for just that—emancipation—which is anti-authoritarian, anti-high-modernist, pro-pluralism.
It would help me if you could name specific bad scenarios, or general principles, that you believe fairly confidently pose a serious risk here.
I think the sector of reprogenetics does have various responsibilities. It has responsibilities to:
notice, think about, and work to alleviate and avert potential harms of reprogenetics;
self-regulate to uphold the norms that would make reprogenetics beneficial for people and for humanity as a whole;
stop if there’s a major (fairly likely and quite bad) unanswered downside risk.
However, I don’t view it as a responsibility of the sector to get everyone on board with reprogenetics before proceeding, or to police the behavior of the entire world with regards to novel reprotech.
For my ideas about how the sector of reprogenetics, and society at large, should comport themselves with regards to reprogenetics, please see “Some practical norms for good development of reprogenetics”.
One might say:
To which I say, yes of course. I am not saying “everything’s automatically perfect by default”. I am also not saying “if only society did what I want, things would be great; therefore there’s no problem with reprogenetics, just put me in charge”. That’s not the procedure I’m proposing. The procedure I’m proposing has these elements, inter alia:
We (you, me, society, the public, scientists, policymakers, workers in the sector of reprogenetics) should envision what good futures with reprogenetics could look like.
The sector of reprogenetics should constitute itself, and thereby take on its appropriate responsibilities as part of society and humanity. For example, it should:
Make and enforce professional norms of behavior (genetic counseling, marketing, publishing, scientific validation, follow-up studies, data privacy, etc.).
Put resources into research that will widen access (e.g. biotech innovation to make the tech less expensive).
Listen and respond (with explanation and action) to concerns that people have.
These two things take a bunch of work and should get resources (funding and talent).
I think that with a high-effort, intentional version of this, the sector of reprogenetics actually can decide for itself to implement better versions of my and others’s plans that will fulfill its responsibilities to humans and humanity, and do so in collaboration with governments and societies to implement broader structures (laws, social attitudes) that further build out a good future with reprogenetics. There are some forces beyond the control even of all of those constituencies put together, but I don’t think that’s a strong argument against doing this.
(I take it you mean, early 20th century eugenics in liberal countries, rather than “liberal eugenics”?)
(True.)
Now, regarding this:
First, an overall comment: As I mentioned above, I think that this comparison is inapt. Comparing them because they both involve genes and reproduction is a poor comparison because that’s not what makes them really distinctive. It’s not the underlying engine of those behaviors. It’s not that there’s something about genes and reproduction that just makes people go crazy and want to forcibly sterilize people, deport all the brown people, etc. It’s that there’s a somewhat coherent ideology behind historical eugenics, and from that ideology flows much of that other behavior, in conjunction with associated attitudes such as high modernism / anti-pluralism. On that more salient dimension of comparison, genomic emancipation is anti-eugenics. Please see “Genomic emancipation contra eugenics”.
Now on some particulars:
Could you be more specific about the overpromising? Who is doing the overpromising (me in this post, other people somewhere, people in the future)? What is being promised, and what do you think is incorrect about that? To be clear, I think that:
The field is quite vulnerable to overpromising, because statistical genetics is new, the results will take years to show up and will be noisy, and there’s a lot of ways to misleadingly present claims.
There probably has already been some degree of overpromising from polygenic embryo screening companies. At least one company has been accused of presenting its polygenic predictors as being considerably more accurate than they really are; and it seems common for gains to be expressed as “the difference between the highest and lowest risk embryo” for some disease—which is kinda misleading, compared to presenting the expected gain (i.e. lowest vs. average).
Overpromising is quite bad for various reasons.
Therefore it’s important to only make accurate promises with appropriate uncertainties.
As with many other worries, this is also something to improve. I’m trying a bit to help the situation, mainly by trying to set up a 3rd party academic validator of PGS claims (though it sure would be nice if good scientists who could work on this didn’t have to worry so much about being vaguely associated with eugenics). Eventually the sector should have self-regulatory orgs, professional standards, legal regulation, public commitments and norms, etc.
But I don’t think this field has been especially bad in this dimension so far, compared to lots of other sectors. Lots of commercial sectors hype stuff up to investors, to customers, to potential hires, and to the public at large. I think that’s generally bad (because generally deceptive) and I don’t excuse it in reprogenetics, but I don’t see how it’s part of some pattern specifically in common with historical eugenics.
I think it is also harmful to underpromise about technologies. Don’t forget the basic facts here—if strong reprogenetics were widely adopted, it would save at least hundreds of thousands of lives, probably more. (This is a somewhat far away hypothetical, but it’s still important!) If you think this is overpromising, we may have a technical disagreement, which could be productive to clarify?
Same questions as above—who’s being unsound and how? Be specific please.
What do you mean by this? Who / what statements are playing into people’s prejudices, and how?
Hm. Are you sure they didn’t get what they intended? Some of them jumped ship, though my impression is that a lot of that was due to the science turning out to be super wrong. A lot of them were excited with how things were going, and wanted it to go more! But I’m not sure what you’re referring to, to be honest. Maybe you could point me to a source that gives some detail on this?
It’s my (weakly held) belief that some of the human intelligence genes act through lowering the happiness set point. Unhappy more often —> more problems to solve —> more practice solving problems —> better at solving problems. If true, even for a smallish subset of the human intelligence genes, then stacking that would be both horrifying and seems intuitively to me like it would fail past a certain point.
Evidence: ~all the cognitively-handicapped-from-birth people I’ve known are very happy on average, and the opposite holds on average for the unusually-smart folks I’ve known.
Curated. The argument here is straightforward and seems basically correct to me. I think I have shorter timelines than Tsvi does, at least if not conditioning on a successful pause/stop effort, but as Tsvi says:
Really, most of the HIA has substantial impact even with short timelines section is important to understand, and should carry the argument even for people with pretty confidently short timelines. (Unless they disagree with “Very short timelines are pretty intractable.” by way of things that can meaningfully be influenced by humans today.) The points at which people think marginal effort allocation to HIA stops making sense might differ, but right now, there need to be more people working on this.
A “Human intelligence amplification” search in google doesn’t turn up much, basically the wikipedia article and a less wrong post. The field probably needs a place to gather?
As an HIA noob, I think an HIA sub reddit, or another public forum (visible from search engines and LLMs) dedicated to the topic would be pretty useful.
At the moment it’s hard to get started as the knowledge is spread across many scientific fields and they tend to not ask the questions we’d care about for intelligence amplification.
So a forum with an intro “Human Intelligence Amplification 101” post containing a map of the fields of intelligence amplification, some resource recommendations, providing opportunities for noobs to ask dumb questions, but also for more advanced researchers to brainstorm, would be useful imo.
I think the forum being public and indexed is extremely important, so that all the useful knowledge doesn’t get lost in private discords or irc chats. And the barrier for entry should be very low (or there should be a clear delimitation between the newcomer and the advanced section), so that people from other fields don’t feel unwelcome and can get started on the topic.
An AI search gives these options:
Discourse forum (self-hosted or hosted), e.g. this type of thing: https://forum.image.sc/
ForumMagnum (LW OSS codebase); LW, EA forum, or https://progressforum.org/ (seems to have died); better for long-form
subreddit (less ownership, maybe worse for long-form, but way easier to set up)
Thoughts? Note that there is https://www.reddit.com/r/reprogenetics/, though it seems to be Gwern-only (empirically I mean).
I think the subreddit has a huge advantage in terms of visibility so I’d go with that at first, that way it acts as a gateway to HIA. Long-form/important content can still be cross-posted to LessWrong.
Then at some point in the future, the community can move to a dedicated ForumMagnum or equivalent, while the subreddit still exists as an entrypoint/question hub for newcomers and as a link to the main forum?
That sounds reasonable. Here: https://www.reddit.com/r/AmplifyHumanCognition/
Yeah I agree. This is one of many things I would like to do at some point.
I mean, another important point: even though adult HIA is presumably harder, adults can consent to HIA. Unborn humans / babies cannot. Even if you want to ignore morals, Adult HIA is just politically and legally far more feasible.
Adult HIA, while perhaps more difficult, has much faster test-->signal loops, so there’s also a strong benefit there—particularly if you’re concerned about AGI coming soon.
Future children can’t directly consent or otherwise give opinions on anything, up to and including their own existence. So, morally speaking, you have to do the best you can until they are able to understand, evaluate, and consent to things. There are many potential influences on a future child’s genome, and more generally, many potential influences on a future child in all zer aspects, from the environment and self-determination and randomness and so on. Parents make lots of choices for their future and present children; the fact that the children can’t directly consent doesn’t make all those choices immoral.
Parents do have a moral responsibility to listen to their future child’s consent as much as feasible, and society should support that. For example, the sector of reprogenetics should be set up to collect multigenerational feedback and make it available to parents and other decision-makers (lawmakers, professional standards setters, individual clinics). If parents make some genomic choices and then those children end up believing they were thereby wronged, they should say so, and this would inform what is reasonably presumed to be a general future child’s envelope of reasonable consent.
It’s far less technically feasible. I’m also somewhat skeptical of taking the current state of political and legal stances as some sort of strong / highly-stable indicator of the state of things in 10 years etc. Lots of things were illegal and/or stigmatized and then un-illegaled and unstigmatized; and reprogenetics has very strong and clear cases in favor of allowing and developing it (e.g. decreasing disease risks).
https://www.lesswrong.com/posts/iQzxxgJXXaAQjq7Jz/faq-isn-t-agi-coming-too-soon-for-reprogenetics-to-help#Adult_HIA_methods_aren_t_fast_either__absent_big_investment
I like it!
Curious if you have any thoughts on combining this with ideas of collective intelligence?
I think we had a discussion like this on one of your shortforms but I’ve read that intelligence genetically seems quite well selected for and that it might be hard (at least with random variation) to hit the right type of genes due to the optimisation pressure already applied there.
So it becomes more of a remove the negative genes rather than improve the positive genes type of thing.
If we then define collective inteliigence as the ability of a group of people or information processing systems to make wise and intelligent decisions there might be other things than just the individual intelligence that would be good to improve?
E.g listening ability and the ability to understand other people is one of the things that generally increase the C-factor (collective intelligence, see this paper) among people.
I also think things like trait openness or conscientiousness seems like quite good targets in terms of general throughput. There’s also arguments to be said for a correlation between intelligence and depression, so if your intelligence increasing correlates with trait neuroticism that might be bad?
You might then even at this point bring in ideas from cyborgism and similar into this entire strand…
I don’t think that the world will stop being susceptible to all x-risks within the next 20 years or so and so it clearly seems useful to have a bet in this direction. I do wonder whether individual intelligence is the thing to aim for but it likely won’t hurt.
Yeah, I don’t have much of a thought beyond “that sounds great”. Hopefully these sorts of things would stack, i.e. add up or even synergize.
Where did you read this? Curious to take a look. We almost know for a fact that one could nudge IQ in the region around the mean (say, within 2 SDs of the mean) for fairly small nudges (say, up to +.8 SD on IQ). There are causally (sibling) validated PGSes for IQ with hundreds of variants, and there’s no visible non-linearity up to about 2 SDs on the trait (see fig 2 in https://www.nature.com/articles/s41586-026-10516-5).
I’ve probably floated this as what might be going on in the past, but it doesn’t especially seem to actually be the case. IQ-positive variants have a broad spread of allele frequencies.
Yeah these seem great. In general though, there’s two problems with reprogenetics for personality:
The PGSes (and underlying trait measurements) are bad for now.
Personality seems like a somewhat more abusable trait to affect.
I think that these aren’t, in the end, dealbreaker obstacles, but they are problems. It would also be great if we had a good understanding of non-abusable personality traits that would help out the child and humanity.
So I had read Kevin Mitchell’s Innate and then followed up by looking at some of the papers back in 2023 or so.
It seems that some of these things are turning out to be false/not replicating? E.g the mutational load hypothesis so I think it’s more about going to the extremes, it might be hard to get 200 IQ geniuses all over the place. I think increasing the general populations IQ by 30-50 points remains quite feasible and that it would be great boon.
My certainty here is not that high fwiw.
I did a chat with fable about this and here’s the sources it looked at:
Sources on the “mutational load / asymmetry” view of intelligence genetics (the claim in Kevin Mitchell’s *Innate*), plus the newer work that bounds it:
**Theory**
- [Keller & Miller 2006, *Behavioral and Brain Sciences*](https://pubmed.ncbi.nlm.nih.gov/17094843/) — Why harmful, heritable mental disorders persist: new mutations arrive faster than selection removes them (polygenic mutation-selection balance). The theoretical backbone of the load argument.
**Evidence that intelligence was selected upward and that random variation skews negative**
- [Joshi et al. 2015, *Nature*](https://pubmed.ncbi.nlm.nih.gov/26131930/) — 354k people: more homozygous DNA (children of related parents) → lower cognition. Directional dominance is the signature of a trait under past directional selection.
- [Ganna et al. 2016, *Nature Neuroscience*](https://www.nature.com/articles/nn.4404) — In the general population, each ultra-rare disruptive mutation ≈ 3 months less education; ≈ 6.5 months if it hits a constrained, brain-expressed gene.
- [Kingdom et al. 2024, *Nature Genetics*](https://www.nature.com/articles/s41588-024-01710-0) — UK Biobank: rare damaging variants in developmental-disorder genes additively lower cognitive and socioeconomic outcomes; partially offset by a higher EA polygenic score.
**Evidence that bounds the strong version (“most heritable variance is load”)**
- [Common and rare variant associations with cognitive performance across development, *Nature Human Behaviour*](https://www.nature.com/articles/s41562-026-02491-8) — Common variants explain ~20% of variance; all rare coding variants exome-wide only ~1%.
- [Distinct genetic architecture in the tails of complex traits, *Nature* 2026](https://www.nature.com/articles/s41586-026-10516-5) — Middle of the distribution (~±2 SD) is common-variant, linear-PGS territory; the tails are dominated by rare large-effect variants. Reconciles the two views.
**Background / provenance of the idea**
- [West Hunter, “The genetics of stupidity” (2012)](https://westhunt.wordpress.com/2012/11/08/the-genetics-of-stupidity/) — Cochran’s version of the same load argument.
- [Yeo et al. 2011](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3027642/) — Early small study (n≈200) linking rare CNV deletions to IQ; the effect size didn’t hold up in larger samples.
I think of my target as being something like: make it be the case that, for parents who want to, they can safely give their kid an average expected IQ of 175 or so. My guess is that this is both feasible and safe (given a bunch of work), but I’m currently investigating how likely we should expect this to be feasible for different numbers (e.g. what’s the genetic architecture of intelligence at the high tail).
(The following is garbled / maybe wrong because it’s unfinished thinking.) It’s also plausible that IQ is just not a good description of things at the high tail. We just mean something like “can do thinky stuff at the level of 1-in-a-million geniuses” (which corresponds to a bit less than 5 SDs, i.e. a bit less than 175 IQ). Such people definitionally exist, so we don’t need to worry about “IQ”. One question is then something like “For 1-in-a-million geniuses, is their genius explained by non-direct-genetic effects to an unusually large extent compared to average?”. If not, then we’re implicitly saying that direct genetic effects aren’t hitting diminishing returns faster than non-direct-genetic effects. That in turn suggests we can push the direct genetic effects a bit further than that. If yes on the other hand, then we’re saying that genetic effects strongly co-saturate with each other right around the 2 or 3 SD mark, but not at all before (because we can observe them not doing that in the normal range), and somehow don’t cosaturate with environment / other non-genetic things. Which seems weird.
How is increasing intelligence not going to lead to changes in personality? E.g, I think there are good reasons to think that higher intelligence leads to higher degrees of rumination due to predictive processing reasons. If you’re better able to create an accurate model then it can be harder to break out of it as you’re already quite certain that the world works as you think it does.
There seems to be correlation between IQ and depression, this might be caused by the neuroticism connection but then the question of why the IQ and neuroticism connection exists is still there.
I probably woul be more fine with increasing traits that increases the possiblity of an individual being able to change their mind on things and come to their own conclusions about the world, some sort of empowerment thing. I think this is likely to involve an increasing in IQ but also conscientiousness, openness and a decrease in neuroticism at least some sort of clamping on neuroticism.
By “personality trait” I mean things like the Big Five, Dark Tetrad, and intuitive things like obedience, creativity, wisdom, etc.
I say they are “somewhat more abusable” because for some of these personality traits:
Plausibly some bad actors would want to push hard on that trait.
It would be pretty bad for the child if the child had an extreme version of that trait, even if it’s within the bounds of biological safety.
For intelligence, within bounds of safety, it seems fine / good for the child to be really smart; and it seems unlikely that many actors would want to make a child have very low intelligence.
But for obedience, for example, you could imagine an authoritarian regime that wants its subjects to be obedient / submissive; and this could get locked in. I don’t think that’s super likely to happen, but it would be a bad outcome.
Maybe? IDK what claims in this genre I’d want to put stock in, unless there’s scientific data.
Yeah totally, this sounds great. (Cf. https://www.lesswrong.com/posts/fzKfzXWEBaENJXDGP/what-is-wisdom-1 )
Maybe? At a wild guess, I’d expect that we’d have to make whole new psychometric constructs, but IDK.
Unclear to me why you put creativity and wisdom in the personality traits bucket. To me they seem to be as much “cognitive capacities” (or generators thereof), as g/intelligence.
On the g->personality thing, IIUC there’s strong support for high IQ correlating significantly with some facets of Opennesss to Experience, especially stuff like willingness to entertain new ideas (very unsurprising), although one can ask here whether it’s about high g influencing personality (or the other way around or a causal fork from some latent...) VS about the cognitive capacity allowing a latent trait of interest in ideas to manifest, where on lower levels of g there would be too much friction.
Agreed on both points.
Yeah, I don’t personally have a principled distinction. What we have, by my lay understanding, is that
Intuitively, there’s things that increase one’s capability in general, and then there’s things that steer / shape what one is interested in / tends to do (extraversion, interests, adventurousness, sadisticness, …).
Intuitively, there’s lots of dimensions of capabilities and somewhat-general factors of capabilities (such as wisdom, empathy, creativity, determination, sanity, cleverness...), which may kinda overlap / shade into being more like steering, especially e.g. curiosity.
But in fact when psychometricians try, mainly what they find is one big blob, g; the subfactorizations they’ve tried don’t explain much more than just g.
g is pretty robust (measurable, transferrable between measurements, test-retest, correlates with life outcomes); other psychometrics are not very robust and not very explanatory of other things.
But I’d like to improve psychometrics for those other things.
We should expect a pause or slowdown because the current trajectory is not sustainable.
I’m not sure, but I doubt this. It feels intuitively plausible to me that there will be some substantial market correction for various reasons (e.g. open source --> cheaper providers for a lot of what people would actually pay for; SWEs keep paying for the good stuff but a lot of enterprise and whatever doesn’t care; or maybe regulatory crackdown; or something) and also intuitively plausible that there won’t be. But it doesn’t feel that plausible that there will be that much of a slowdown on whatever AGI capabilities research is being spurred by the current state of things. That ship seems to have sailed some years ago. In other words, I expect there to continue being lots of investment in “Red” AGI capabilities research.
People asked Eliezer about getting people like Terence Tao to work on alignment.
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
Setting aside the possibility of recruiting security geniuses such as DJB (because MIRI and friends seem uninterested in that, for whatever reason).
It seems to me that adding more high-IQ people to the “set of people thinking about AGI” has been tried in the past, and the effect was to accelerate AI capabilities. What you fundamentally need is dispositional/sociological, not mental horsepower.
You could argue that with reprogenetics, it will be possible to make humans so smart that they’re in a qualitatively different regime from Terence Tao. In which case they could destroy the world during a teenage tantrum. Wisdom doesn’t necessarily scale with IQ. Children don’t necessarily adopt the ideology of their parents. And it’s not ethical to “shut a person off” or read their thoughts like you could do for a rogue AI.
It seems to me that the community is taking the same “Leeroy Jenkins” approach to reprogenetics that it took to AGI, without serious effort at e.g. public red-teaming. Equally catastrophic results seem quite plausible. Feeling blackpilled on this community.
I’m curious what has been happening with young geniuses specifically; do they seem more or less accelerating AI capabilities compared to the older guard?
I think you need both, by a long shot. Humanity is bottlenecked on ideas in so many ways in so many domains. Humanity is also bottlenecked on values / caring about the right things / being wise / etc.
But I would also say that humanity is specifically bottlenecked on having people who are both wise and very smart. We can at least unblock on “very smart”.
I also think that if there were a lot more very smart people, we’d do a better job at supporting them.
That said, education is super important.
Also, I agree that wisdom specifically is important and interesting, and that we should investigate whether / how we could genomically vector for traits like that.
Yes, please send serious effort at public red-teaming. (Also send serious effort at public blue-teaming.) You can rest assured though, approximately no one from the community is working on this.
Here’s my main efforts so far re/ red-teaming: https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts https://berkeleygenomics.org/articles/Potential_perils_of_germline_genomic_engineering.html
Eliezer also said under a post about Superbabies that it is third most important project on the planet and it gets too little attention.
He also said in one of the interviews that he thinks alignment is too hard for unaugmented humans, that we need human intelligence augmentation (probably in form of adult gene therapy with suicidal volunteers, because waiting for babies to grow up is too long), that maybe if you are somewhat smarter than John von Neumann, you get security mindset by default. Though irrc he also pointed out somewhere that one of problems is that getting from IQ 220 to 250 should be much harder than from 70 to 100 or from 100 to 130 (iirc, because here we can just copy what exists).
It’s pretty dubious whether one can safely go to 220, let alone 250. However, I think having lots and lots of [people like your favorite super smart people] would be great.
To clarify, we do not currently have clean access to this data. We don’t have an accurate way to measure intelligence, nor have these tests been applied to a random sample of humans at scale. Strongly select for any current measurement of intelligence and we will also accidentally be selecting for potentially quite deleterious correlated things as well. Just a note, when each of your test --> feedback learning steps takes decades and many non-consenting human lives.
Could you be more specific please? E.g. what is “accurate” here, what is “random”, etc.
For example, IQ is I think in some respects the most solid construct in all of psychometrics. It has “quite high” test-retest correlation (with different numbers depending what we ask), and correlates “highly” with real-world metrics like educational attainment and income.
It’s true that the at-scale tests are “significantly worse”, but they are still “pretty good”. They will get better over time as more and better data is collected, and algorithms for combining datasets get better.
Potentially. This is an area for ongoing investigation. You sound confident though; what leads you to think this? Compare “Assessing Potential Pleiotropic Off-Target Effects” from Herasight’s paper Interpreting Polygenic Prediction of Cognitive Ability: Evidence for Direct, Reliable, and Portable Genetic Effects:
[Note that the authors say these “effect sizes likely reflect upper bounds due to inflation from assortative mating and genetic nurture, and should thus primarily be interpreted for directionality”.]
It could be that there are unmeasured, unknown traits that are increased by the genes in the intelligence PGS. I’ll think more about this, but my guess is that we should expect these effects to be small and fine (e.g. because intelligence is generally uncorrelated or slightly positively correlated with prima-facie fine/good things that we have measured).
I really think the effect that AI safety people’s short timelines have on whether or not reprogenetics research gets done is ~0. Human intelligence enhancement is illegal, and faces a massive social stigma. We could have cloned the smartest humans starting decades ago and we haven’t (at least publicly) because governments and the regulators seem to hate that idea.
We could have been doing selective breeding to foster intelligence since we discovered evolution in the early 1800s. We haven’t done it because of stigma and lack of government interest. Short AI timelines is just irrelevant.
I think IVF wasn’t illegal before it was practiced, but the inventor was definitely persecuted and there was a ton of social stigma against it. These things develop. I’m more interested in
Is reprogenetics actually good? Or more importantly, how would we do it in a beneficial way?
How should society process the idea of reprogenetics?
There’s a very good reason to not clone humans, which is that you stand a quite high risk of producing sick and/or disabled babies due to the significant deviation of an SCNT embryo from the natural epigenetic state of an embryo. This is how pet cloning (e.g. ViaGen) works; especially in the earlier days of pet cloning, the process would look something like:
make hundreds of embryos via SCNT
many don’t grow
try to implant hundreds into dozens of surrogates
most don’t implant
some miscarry
many are born with terrible issues (birth defects / illnesses / dying shortly after birth)
maybe one or a few are in good shape
There’s good reason to not allow human cloning. I don’t think it was motivated by wanting to not have smart people be cloned, though I haven’t checked the original reasoning for these laws, it’s not totally implausible that this would appear there.
Lots of different people have lots of different reasons for not wanting to work on reprogenetics or not thinking it’s a good thing to do. I’ve talked to lots (tens) of people who brought up short AGI timelines as the or a main reason they didn’t think reprogenetics would help.
I think this is basically just false in the context of the post, unless you’re talking about something mostly unrelated.
Although HIA will result in Humanity being better able to tackle AGI, I think it may also result in an adverse effect if we do not get down the problem of Human Alignment correctly. Now, if there are more people working to reduce existential risk than there are people who are working to advance AGI as fast as possible, HIA is helpful. And the more people that are working on existential risk compared to AGI, the more percentage points of existential risk gets reduced per IQ increased. However, we should consider the alternative. If there are currently a bunch of people working to accelerate AGI, and fewer people than that working on existential risk, then HIA has a negative effect on existential risk.
This I expect would be lessened somewhat as people with higher intelligence figure out that AGI might, possibly, kill us all if done wrong. But that lessening assumes everyone is working with an altruist mindset, and it might be the case that despite the altruistic motivations, personal greed/ambition/etc. overcomes it and has them work on AGI anyway. It might also be the case that greater intelligence results in greater rationalization without proper rationalist training, (although research suggests higher IQ leads to a naturally higher level of rational thinking, they are still independent) which would amplify the effects of unwholesome motivations.
The TL;DR here is that if we have more people “against us” (working on AGI) than “with us” (working on existential risk) more intelligence works for “them” and not for “us”.
I still haven’t fully thought this through, so I don’t want to fully defend a position. But see https://tsvibt.blogspot.com/2025/11/hia-and-x-risk-part-1-why-it-helps.html and https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts
Note that the reprogenetics program is extremely achievable with ordinary biological science. There’s a relatively large pool of talented geneticists who are unlikely to have the security mindset necessary to directly work on capable AI, so I don’t think this program will detract human capital from other cause areas relevant to AI survivability. The risk does not extend far past the experimental subjects. It’s low cost with potentially huge upside. And it’s not like having healthier and smarter children is a thing people would be opposed to absent AI risk.
I also agree with the RobertM comment that the substantial impact with short timelines is well-argued. I think it’s worth adding that even if this program is doomed to fail because we get clotheslined by short timelines before it can do anything, it would be more dignified to die having started trying to become capable of solving alignment that it would be to be killed by misalignment without trying anything even conditioned on being correct that there’s predictably not enough time to complete the program. This is not as important as the argument that we should play to our outs, but it makes me feel better.
I agree with this characterization, if we’re taking a bird’s-eye view of science. But to be clear, there are lots of unknowns here, such as how to do full in vitro gametogenesis, how to avoid genome degradation in cultured cells, what happens when you push a polygenic score way far out, how can chromosome selection be implemented, etc. Of course, there are also non-science / non-technological issues (legal, social).
I definitely think there is a strong case for reprogenetics absent AI risk, and that’s the case I mostly want to be making to society. In other words, I think it’s mostly reasonable & good for society to not really account for AI risk when considering whether reprogenetics is a good idea, and instead evaluate it only on other grounds, such as whether it’s good for the kids and whether society can handle that technology healthily. (In part because that consideration somewhat instrumentalizes children. It shouldn’t be categorically forbidden to be motivated to have kids because of some outcome you want them to bring about, but it should be viewed with a lot of skepticism / worry / carefulness, especially when coming from society rather than parents and especially when it’s a potentially overbearingly strong motivation.)
My main issues are the following:
Using terms from AI-2040, this requires the world to choose Plan S instead of Plan A where AGI does arrive, but is disallowed from starting the RSI, bottlenecking the risk on three considerations, only ONE of which is due NOT to a lack of political will.
Human intelligence enhancement could be related to data progress instead of algorithmic progress, as Beren once said about LLMs, or fail to produce scientific effects, especially if we take into account the ongoing education crisis in the USA, which Zvi covered in his Childhood and Education roundups #17 and #18, or Cannell’s case for AlphaGo’s performance being due to having an amount of data comparable with human champions. These champions have spent an unincreasable part of their lives practicing Go, and enhanced humans will also have spent an unincreasable part of their lives studying alignment-related sciences.
(I don’t understand your first point. I argued in the post that I don’t think AGI is that likely to arrive very soon, and even if you do think that, HIA has substantial positive expected benefit.)
Re/ your second point, also not sure I understand. You’re saying that maybe humans with amplified intelligence might not be able to contribute much because their education is poor in general, or specifically because contribution is bottleneck on years of experience working on the alignment problem or adjacent sciences? I mean, data and education are important too. We can & should also support education, and try out things to support geniuses in particular. (But if you’re saying that this implies technological HIA such as reprogenetics wouldn’t work or wouldn’t have much impact, I don’t see how that follows at all.
As far as I understand, the main case against short timelines was in your posts “Do confident short timelines make sense?” (Jul 2025!) and the post made on Jul 2023(!!). I had Claude Sonnet 5 prepare the list of breakthroughs between July 2023 and now:
Claude’s list
Quite a lot happened in this three-year stretch. Here’s the shape of it:
Late 2023 — multimodal goes mainstream Google launched Gemini in December 2023 as a multimodal competitor to GPT-4, integrated initially into Bard and other Google tools. This was part of a broader shift where models stopped being text-only and started natively handling images (and later audio/video) in one architecture, following GPT-4′s earlier multimodal debut.
2024 — reasoning models arrive The single biggest architectural shift of the period came in September 2024, when OpenAI released o1-preview, the first in a new series of “reasoning models” trained specifically for chain-of-thought problem solving, rather than just generating fluent text in one pass. This kicked off what’s often called the “reasoning” aka inference-scaling aka Reinforcement Learning from Verifiable Rewards (RLVR) revolution — models that spend extra compute “thinking” before answering hard problems.
Early 2025 — the DeepSeek shock and open-weight reasoning January 2025 brought DeepSeek-R1, an open model that acquired reasoning capabilities solely through reinforcement learning, which shook markets by matching frontier reasoning performance at a fraction of the training cost — DeepSeek-V3 was more than a technical achievement; it signaled that accessible, high-performing models could thrive outside the traditional big tech ecosystem. A Berkeley team even replicated core concepts of DeepSeek’s R1-Zero model on a budget of just $30, with a 3-billion-parameter model called “TinyZero” trained via reinforcement learning, showing the technique wasn’t exclusive to giant labs. OpenAI responded by pushing further with o3, o3-mini, and o4-mini, and reasoning became a signature feature of models from nearly every other major AI lab.
2024–2025 — the rise of agents Alongside reasoning, agentic AI emerged — systems that don’t just respond to prompts but autonomously plan, execute, and adapt to accomplish complex goals. Rather than just answering, these systems reason through multi-step plans, invoke external tools and APIs, maintain memory across interactions, verify results, and recover from errors with minimal human oversight. Standardized protocols for tool use (like MCP) matured enough that by mid-2026 major frameworks like LangChain and LlamaIndex fully support MCP, making it trivial to add tool use to any model.
2025–2026 — architecture experimentation and efficiency The field diversified past plain transformers. Hybrid architectures blending attention with state-space (Mamba-style) layers became popular for efficiency — this hybrid-architecture trend with alternating attention and alternative layers became a relatively popular development, with Qwen3.6 using Gated DeltaNet layers instead of Mamba-2 layers. NVIDIA’s Nemotron 3 Super was an open, efficient Mixture-of-Experts hybrid Mamba-Transformer model designed for agentic reasoning. Diffusion-based (non-autoregressive) language models also appeared as a genuinely different generation paradigm — models like Seed Diffusion Preview, based on discrete-state diffusion, offering fast inference speed through non-sequential, parallel generation rather than token-by-token decoding.
2026 — rapid-fire frontier releases This year has seen an unusually fast release cadence across labs: January through April all featured at least one frontier-class launch — Google shipped Gemini 3.1 Pro in late February, Anthropic shipped Opus 4.7 in mid-April, and OpenAI shipped GPT-5.5 in April. Meta also pushed back into frontier territory with a model called Muse Spark. Coding and agentic-workflow benchmarks became key battlegrounds: Claude Opus led SWE-bench Pro while GPT-5.5 led Terminal-Bench 2.0, with Claude stronger on cold-start code synthesis and GPT-5.5 stronger on multi-turn agent loops. Efficiency also kept improving — models like DeepSeek V4-Flash offered a 1M-token context window at roughly 50x cheaper input pricing than GPT-5.5. Most recently, Anthropic released Claude Opus 5 in late July 2026, alongside continued releases from Google (Gemini 3.5/3.6 Flash), Alibaba (Qwen3.7/3.8), Moonshot AI (Kimi K3), and others.
A few threads run through all of it: reasoning/test-time compute became a standard model capability rather than a novelty, open-weight models closed much of the gap with closed frontier labs while driving costs down dramatically, context windows grew enormously (into the millions of tokens), and the center of gravity shifted from “chatbot that answers” to “agent that acts” — using tools, maintaining state, and completing multi-step tasks with less supervision.
The case against novel conceptual reasoning seems to have partially lost its juice given that scaling and the innovations described in the collapsed section (which IMHO are closer to education techniques than to architectural breakthroughs. Novel architectures like neuralese have yet to be discovered) gave rise to models as capable as Claude Mythos, Astra and other discoverers of novel theorems and cyber-related exploits, or Claude Opus 5 making a breakthrough in the ARC-AGI-3 non-scaffold.
As for the second point, yes, I would expect conceptual research to be bottlenecked on years of experience working on sciences like alignment or mechinterp (e.g. the AI-2027 Race branch had Agent-4 start with understanding its own cognition by superintelligent mechinterp, then construct Agent-5 with one goal). However, I struggle to understand what experiment could reveal that HIA worked as you describe versus shifting the human’s interests.
I’m not interested in arguing with your LLM. I don’t believe I’ve ever expressed much or any skepticism about theorem proving, ARC-whatever, or computer hacking coming from current AI research.
Well, like, if someone went into theoretical physics, they might produce intellectual progress on the order of [pick your favorite brilliant physicist] or instead [pick your favorite highly motivated but not very successful theoretical physicist].
pre-RSI AGI are likely less aligned than enhanced humans
Are you saying that native intelligence doesn’t matter as much as education? Because I’m very sure that is incorrect; my intuition points strongly to many life outcomes pointing better for a 135 iq person vs a 100 iq person, all else being equal~ (and with the exception of mental illness/depression/burnout, which is a fraction of the population but far from 100%.
I feel very very confused as to why these timelines are so long. And why just scaling up pretraining more won’t just work and why the primary bottleneck isn’t mainly the physical buildout and gpu availability.
My basic case, available in the linked posts in some more detail and breadth, is:
By default, we have “very long” timelines because we don’t have a strong reason to think AGI gets invented at some specific time.
To update strongly on that, we’d have to get strong evidence that we’ve gotten most or all of the ideas involved in making AGI.
People adduce the rise in capabilities as the main reason that they updated to confident short timelines.
But the rise seems largely explicable as ~imitation + a cherry on top. Or in other words, the inference to “we’ve found most of all of the ideas” is not supported by our observations
People adduce the utility of AI coding in AI R&D as a reason to expect fast future progress.
But this doesn’t say much about how far is left; and I’m pretty skeptical of huge (like >10x, say) speedups on the research tasks that most matter for AGI capabilities progress.
So we should somewhat still be in the prior, though with some substantial update towards short timelines; but not way over to like “90% in 10 years” and similar. That’s unjustified.
You write:
Would you please read https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#The__no_blockers__intuition and maybe some surrounding sections?
Another reason I’d support reprogenetics work is because it might resolve one of the most burning questions in our minds, which is whether something like a software-only intelligence explosion is plausible, because there’s one (very major) constraint that applies to human intelligence augmentation that doesn’t apply to AI development, and that is the fact that we can’t increase computational power directly like we can for AI models by even 1 order of magnitude for inference or training, because we’d cook the body (at least not without technologies that is only practical to invent post-industrial explosion.)
This means work on reprogenetics could in theory probe the question indirectly, by measuring how large the returns to algorithmic progress are from a fixed base like the human mind.
We’d need much better measurements, especially measurements about whether or not when diminishing returns to intelligence without compute increases kicks in, and measurements about how well intelligence increases translate to other good metrics, but if we had the correct measurements, we could answer the question of whether software-only intelligence explosions are plausible, and to a first approximation, good AI policies (that aren’t just pure transparency/obviously good stuff) and good AI safety prioritization depend on answering this question.
But have we solved the alignment problem for hyper intelligent humans? Who says they solve the alignment problem instead of wiping us out the same way?
People are not as loyal to the interests of other people as we would like, but we have no idea how to make an AI care even a tiny bit about people in a way that has a high probability of persisting after the AI has become super-humanly capable.
Please see this comment: https://www.lesswrong.com/posts/iQzxxgJXXaAQjq7Jz/faq-isn-t-agi-coming-too-soon-for-reprogenetics-to-help?commentId=PQpbETKdDaD7Ao5Jq
I’ll agree that all children should be raised to be kind, wise, generous, future-looking, joining the global community of souls, and similar. If you’re saying that HIA kids are especially evil or something, I doubt this. If you’re saying that they form a cabal, I also doubt this, as they will be mixed in with everyone, in every way other than the fact that they got some HIA; they will be geographically distributed, ideologically distributed, etc., and also distributed in phenotypes, and also not far outside the human envelope of phenotypes.
How would you prioritize preventing neurodegenerative diseases relative to adult HIA and reprogenetics? Preventing these diseases seems to circumvent a number of the drawbacks you cite for adult HIA as it is:
well-studied
backed by pharma / public opinion
has predictable benefits
Strictly speaking it doesn’t amplify intelligence beyond what humans can do at their best, but still seems extremely valuable as an ill person basically ends up with negative productivity and preserving intellect would allow people to leverage all of the crystallized intelligence they accumulated over their life.
Well, it seems great of course. But reprogenetics, and adult HIA if it worked, would do something that keeping existing people healthy cannot do: lead to many many more geniuses. Imagine for example that there were 10x the number of people who are in the reference class of “the smartest 10,000 people in the world”.
What about the elephant in the room? ‘Eugenics’ isn’t anywhere close to being in the Overton window, and this is obviously ‘eugenics’. It seems clear to me that the PR issue is the most consequential blocker for any substantial development in this general area, so solving it is of primary concern.
I wouldn’t say it’s cleanly “the most consequential blocker” exactly. There are major technical barriers. A key one is full, efficient in vitro gametogenesis. Talent, funding, and legal regulation are also key blockers—though of course, those latter ones are significantly affected by society’s stances toward reprogenetics. Anyway, it’s definitely in the top few blockers, and definitely in the top few priorities to address. My own strategic focus is compelled by that consideration, but also compelled by other considerations (comparative advantage, opportunities, tractability, etc.), so I haven’t focused so much on it directly. Instead for now I think more about the deeper issues, to check and work out the case in favor—i.e. laying some groundwork to hopefully help society process reprogenetics over time.
Some more remarks:
I believe that the object-level case for reprogenetics being beneficial to more or less all individuals, to societies, and to humanity as a whole is strong. I also believe reprogenetics can be pursued in a way that alleviates or avoids most of the potential harms. I think this possibility and intention can be honestly communicated, and ought to be persuasive. I furthermore think it will in fact be persuasive to many people, and that being persuasive to many people (a substantial fraction of society) is enough to make it workable.
In particular, I think there’s a fairly clear distinction in intentions and plans between reprogenetics and eugenics: reprogenetics is pluralist (parents choose on behalf of their future children, to give their children good lives, according to the parents’s various notions of a good life), whereas eugenics is anti-pluralist and subordinates the good of the child to the good of society. See “Genomic emancipation contra eugenics”.
I think the antipathy is somewhat overblown. Polls show plenty of people are interested in using the technology, think it would be a good thing, and so on (especially for disease risk reduction). Bioethics reports and councils on germline engineering often come out partially or largely in favor of reprogenetics, but with conditions (safety, equal access, properly communicated, etc.). There is, though, pressure in academia to downplay the benefits of reprogenetics, upplay the costs, or derogate the sector overall.
I think the antipathy is somewhat second-order. That is, people choose to not work on reprogenetics not because they think it’s bad, but because they think other people think it’s bad. Therefore I expect interventions to be somewhat high-leverage in expectation–the situation is not so stable, e.g. there may be preference cascades in favor of reprogenetics.
Makes sense. I guess what I’m asking is whether you’re aware that anybody else is working on it, or is likely to start soon?
Yeah, but flipping a few elites who would be willing to go on the record is probably necessary to set this off.
Not really! I kinda sorta have my eye on it, but I’m heavily capacity-constrained! I need help!
Polygenic embryo screening companies are pushing the conversation forward, though I often have small- or medium-sized disagreements with how they do that. Also, they are partial and people know that, so it’s not the same. Also they aren’t super incentivized to do public goods, as opposed to promoting their brand / company / product / representatives.
The people involved in polygenic embryo screening companies, and others, have done lots of okay or good writing on the topic. E.g. some relatively brave academics write about it. But I think a more intentional & high-effort strategy would pay off.
It would help, yeah. IDK if it’s necssary; I mean, I think there’s lots of channels that could help.
This is irrelevant! In what way is it better to be replaced by Homo Hypersapiens than to be replaced by AGI? Possibly I could go with that on some really long timescale—e.g. at least hundreds if not thousands of years, but decades? Rate of change matters more than what the change actually is. I often put it like this: If we could have asked our fishy ancestors whether or not they would like to evolve into us, I’m pretty sure that, certainly if the timeframe was a few generations, they would have said no way and would have stopped it if they could. Or, alternatively, consider that, whilst you may be happy to have someone younger and better than you replace you at work when you retire, you don’t want it to happen before then even though your replacement is of the same species.
I hope to give a fuller answer later. Short answers:
Most importantly, reprogenetics won’t be making “a new species”; these kids would be just like other kids. The differences you’d notice would be: statistically noticeably healthier (lower rates of diabetes, heart disease, cancer, etc.); and on average smart or really smart, in the way you’re familiar with (for parents who make that choice). Here’s an image from “Genomic emancipation”, illustrating that HIA kids would be part of the big cloud of humanity:
I disagree a lot with “Rate of change matters more than what the change actually is.”. Humans or HIA humans are way way way easier to align than AGI. Alignment matters a lot.
But yes, I would feel quite worried about making “500 IQ kids”. That seems like a quite alarming rate of change. It’s also extremely infeasible via any current or near future technology (besides some extreme adult HIA stuff, which seems dangerous to me).
The rate of change for realistic HIA via reprogenetics (and most adult routes) is not remotely that big.
Well, I would like to invest in a future that is great & empowered for everyone. This includes parents supporting & empowering their children and handing off the reins; and includes children working to save everyone through breakthrough technology & medicine &c.
Here’s a second-order effect worth considering: Germline modification for intelligence is statutorily prohibited in most countries, the relevant professional societies are currently calling for a 10 year moratorium on the topic, and the public polling is uniformly and staunchly opposed. What if all that is for a good reason? Might some bright young person wanting to work on AI Safety end up having a counterfactually reduced impact if they choose to pursue such a devastatingly disfavored research path?
Just to check, these are two separate points, right? I.e. you’re suggesting:
There are good reasons that the public is largely against reprogenetics for intelligence.
Talent would be wasted on reprogenetics because it’s disfavored by the public, and therefore won’t have much impact.
There are definitely good reasons that the public answers the way they do on surveys. However, I also think that reprogenetics can be done in a way that people would like. I think the disconnect comes from society not having processed this yet. That’s part of why there’s so much altruistic alpha here—reprogenetics is good and there’s very little organized advocacy for it!
If you think reprogenetics isn’t feasible to do in a way that is legibly beneficial to approximately everyone and to humanity and society, I’d be curious to hear your thoughts about that, e.g. what’s most bad about it / what are the consequences that make you suspect it would turn out bad.
Regarding your first sentence: I have to say, I suspect you did not read these things you’re citing. Maybe you found these with an AI search? Do not comment on my posts with citations that you did not actually read to at least verify that the claims you’re citing are even stated there.
The survey you linked seems to discuss laws mainly about gene editing specifically. Reprogenetics includes other genomic vectoring methods—and in fact the ones I’m most interested in are not editing (rather, iterated meiotic selection and chromosome selection). It’s plausible that the bans would be extended or would be interpreted to apply to such technologies, but I don’t know. One of the main justifications for banning heritable reproductive gene editing specifically is that gene editors are in fact unsafe currently. This would apply less, or at least differently, to selection methods.
It also doesn’t seem to mention intelligence or cognitive traits.
I believe that PGT-P, i.e. polygenic screening, is banned in many countries; but this is again also a ban regarding reducing disease risks. And again, there’s been no substantive organized effort to change this, and therefore there could very well be high altruistic alpha here.
The study that you cited is
a sample of only Australians
shows fairly strong support (3.15 on a scale of 1-4) for germline editing to improve health
DOES NOT ASK ABOUT INTELLIGENCE
Here is a better poll: https://www.pewresearch.org/science/2020/12/10/biotechnology-research-viewed-with-caution-globally-but-most-support-gene-editing-for-babies-to-treat-disease/
It includes this:
That comes from here:
Which indeed shows that overall people answer that it’s “misusing technology”. (Note the strange wording; one wouldn’t really “change a baby’s genetic characteristics”.) Note the significant variation by country. Note also the high support for using reprogenetics “to REDUCE THE RISK of a serious disease or condition that could occur over the course of the baby’s lifetime”.