It might be interesting for someone to look into: Have there been large coordinated attempts to wean an industry off of blood money? (Successfully or not.) E.g. blood diamonds, blood gold, blood chocolate, blood cobalt, etc. What can we learn about that task from historical examples? What might transfer to frontier AI research and the surrounding ecosystem? E.g., is it at all possible to get “harvest” companies to be satisfied with some fixed level of model and not pay for advances? I know the answer is “no” but just saying it might be interesting.
TsviBT
Karma: 11,693
Personality is a more difficult issue because
more potential for scientifically unknown effects—it’s a complex trait
more potential bad parental decision-making (e.g. not understanding what very high disagreeableness really means)
more potential for parental or even state misuse (e.g. wanting a kid to be very obedient)
more weirdness regarding consent and human dignity and stuff; I think it’s pretty unproblematic to decrease disease risk and increase healthspan and increase capabilities, and only slightly problematic (due to competition effects and possible health issues) to tweak appearance (though I don’t really care about this one); but I think it’s kinda problematic with personality traits because you’re messing with values in ways you don’t understand; not so problematic that it’s ruled out to tweak traits within the quite-normal human range, and I’m probably ultimately pretty in favor of it, but I’d want more sensitivity on these questions
technically more difficult than IQ or disease because conceptually muddled, hard to measure, and maybe genuinely more genetically complex.
That said, yeah, I’m in favor of working out how to do it well. E.g. I’m interested in understanding and eventually measuring “wisdom” https://www.lesswrong.com/posts/fzKfzXWEBaENJXDGP/what-is-wisdom-1 .
I would be curious what you think about the idea of more permanent economic rifts and also the general economics of gene editing?
As a matter of science and technology, reprogenetics should be inexpensive. I’ve analyzed this area quite a bit (though not focused specifically on eventual cost), see https://berkeleygenomics.org/articles/Methods_for_strong_human_germline_engineering.html . My fairly strong guess is that it’s perfectly feasible to have strong reprogenetics that’s pretty inexpensive (on the order of $5k to $25k for a pretty strongly genomically vectored zygote). From a tech and science perspective, I think I see multiple somewhat-disjunctive ways, each of which is pretty plausibly feasible, and each of which doesn’t seem to have any inputs that can’t be made inexpensive.
(As a comparison point, IVF is expensive—something like $8k to $20k—but my guess is that this is largely because of things like regulatory restrictions (needing an MD to supervise egg retrieval, even though NPs can do it well), drug price lock-in (the drugs are easy to manufacture, so are available cheaply on gray markets), and simply economic friction/overhang (CNY is cheaper basically by deciding to be cheaper and giving away some concierge-ness). None of this solves things for IVF today; I’m just saying, it’s not expensive due to the science and tech costing $20k.)
Assuming that it can be technically inexpensive, that cuts out our work for us: make it be inexpensive, by
Making the tech inexpensive
Thinking not just about the current tech, but investing in the stronger tech (stronger → less expensive https://berkeleygenomics.org/articles/Methods_for_strong_human_germline_engineering.html#strong-gv-and-why-it-matters )
Culture of innovation
Both internal to the field, and pressure / support from society and gvt
Don’t patent and then sit on it or keep it proprietary; instead publish science, or patent and license, or at least provide it as a platform service
Avoiding large other costs
Sane regulation
No monopolies
Subsidies
Might it be smart to make it a public good instead?
I definitely think that
as much as possible, gvt should fund related science to be published openly; this helps drive down prices, enables more competitive last-mile industry (clinics, platforms for biological operations, etc.), and signals a societal value of caring about this and not leaving it up to raw market forces
probably gvt should provide subsidies with some kind of general voucher (however, it should be a general voucher only, not a voucher for specific genomic choices—I don’t want the gvt controlling people’s genomic choices according to some centralized criterion, as this is eugenicsy, cf. https://berkeleygenomics.org/articles/Genomic_emancipation_contra_eugenics.html )
Is that what you mean? I don’t think we can rely on gvt and philanthropic funding to build out a widely-accessible set of clinics / other practical reprogenetics services, so if you meant nationalizing the industry, my guess is no, that would be bad to do.
Hmmm but what if human good not coupled with human wisdom? Maybe more intelligence more power seeking if not carefully implemented?
I think this is just not the case; I’d guess it’s slightly the opposite on average, but in any case, I’ve never heard anyone make an argument for this based on science or statistics. (There could very well be a good such arguments, curious to hear!)
Separately, I’d suggest that humanity is bottlenecked on good ideas—including ideas for how to have good values / behave well / accomplish good things / coordinate on good things / support other people in getting more good. A neutral/average-goodness human, but smart, would I think want to contribute to those problems, and be more able to do so.
Moral mazes lead to psychopaths in control? Maybe non cool humans take control?
This is a significant worry—but my guess is that having lots more really smart people would make the problem get better in the long run. That stuff is already happening. Figuring out how to avoid it is a very difficult unsolved problem, which is thus likely to be heavily bottlenecked on ideas of various kinds (e.g. ideas for governance, for culture, for technology to implement good journalism, etc etc.).
What if like homo deus we get separate Very Smart group of humans and one not so smart?
I agree this would most likely be either somewhat bad or quite bad, probably quite bad (depending on the details), both causally and also indicatorily.
I’ll restrict my discussion to reprogenetics, because I think that’s the main way we get smarter humans. My responses would be “this seems quite unlikely in the short run (like, a few generations)” and “this seems pretty unlikely in the longer run, at least assuming it’s in fact bad” and “there’s a lot we can do to make those outcomes less likely (and less bad)”.
Why unlikely in the short run? Some main reasons:
Uptake of reprogenetics is slow (takes clock time), gradual (proceeds by small steps), and fairly visible (science generally isn’t very secretive; and if something is being deployed at scale, that’s easy to notice, e.g. many clinics offering something or some company getting huge investments). These give everyone more room to cope in general, as discussed above, and in particular gives more time for people to notice emerging inequality and such. So humanity in general gets to learn about the tech and its meaning, institute social and governmental regulation, gain access, understand how to use it, etc.
Likewise, the strength of the technology itself will grow gradually (though I hope not too slowly). Further, even as the newest technology gets stronger, the previous medium strength tech gets more uptake. This means there’s a continuum of people using different levels of strength of the technology.
Parents will most likely have quite a range of how much they want to use reprogenetics for various things. Some will only decrease their future children’s disease risks; some will slightly increase IQ; some might want to increase IQ around as much as is available.
IQ, and even more so for other cognitive capacities and traits, is controlled…
partly by genetic effects we can make use of (currently something in the ballpark of 20% or so);
partly by genetic effects we can’t make use of (very many of which might take a long time to make use of because they have small and/or rare effects, or because they have interactions with other genes and/or the environment);
partly by non-genetic causes (unstructured environment such as randomness in fetal development, structured external environment such as culture, structured internal environment such as free self-creation / decisions about what to be or do).
Thus, we cannot control these traits, in the sense of hitting a narrow target; we can only shift them around. We cannot make the distribution of a future child’s traits be narrow. (This is a bad state of affairs in some ways, but has substantive redeeming qualities, which I’m invoking here: people couldn’t firmly separate even if they wanted to (though this may need quantification to have much force).)
Together, I suspect that the above points create, not separated blobs, but a broad continuum:
As mentioned in the post, there are ceilings to human intelligence amplification that would probably be hit fairly quickly, and probably wouldn’t be able to be bypassed quickly. (Who knows how long, but I’m imagining at least some decades—e.g. BCIs might take on that time scale or longer to impact human general intelligence—reprogenetics is the main strategy that takes advantage of evolution’s knowledge about how to make capable brains, and I’m guessing that knowledge is more than we will do in a couple decades but not insanely much in the grand scheme of things.)
What can we do? Without going into detail on how, I’ll just basically say “greatly increase equality of access through innovation and education, and research cultures to support those things”.
I have a firm rule of single source of truth. For any given post / article, I have 1 (one) file in which I write the post. I generally write in markdown, with a couple small added conveniences. Then if there’s some way that I want to post it (personal blog, google doc, email, LW, https://berkeleygenomics.org/Explore, pdf), I have a script for that type of post. The scripts are somewhat of a mess, but not that bad, and this hullaballoo makes it fairly easy to crosspost and to edit crossposted posts.
This is one class of instances among many other classes of instances of the general problem of desires not aggregating. In the good case, first desire paths form through natural aggregation; then the aggregated desire gets compiled into more permanent / designed / intentional / legible structures. There’s plenty of opportunity to innovate norms, but many norms need participation to be worthwhile, and it’s often not helpful to try to change them just by saying “let’s all do X”. Further, some norms are exclusive with other norms, so there are group decisions to be made, in addition to opportunities for pareto improvements. In such cases, there has to be buy-in. The cognitive effort to get buy-in is substantial, so the desires related to the potential benefits of new norm + buy-in don’t get aggregated; Alice takes potshots that don’t add up with each other, and people can tell that she’s doing that and therefore her efforts aren’t worth investing in. Therefore I propose the norm that everyone should be more open to imagining what-ifs—envisioning possible collective futures—with the understanding that the pie-in-the-sky nature of such envisioning is to a substantial degree determined by the fact that we collectively don’t envision, rather than that we don’t commit or act.
“as smart as a human engineer across all relevant domains”
I dispute that LLMs are like this; I think they and their training have a bunch of performance capability and not much ability to generate those de novo.
gets us to “the best AI humans will ever be able to create” quite a bit quicker than we’d otherwise get there,
Maybe; to some extent I’d expect this to hit various walls, though not sure; Amdahl’s law; and IDK how people get very confident of this.
(actually I asked for Gemini 3.1 pro preview for a summary of a copypasted version and it wasn’t good)
Is there a way you could restate this in terms of one or more propositions that are fairly load-bearing for your confident short timelines? Is it this?
it is pretty likely that huge amounts of superhuman crystalized intelligence can find fluid-intelligence-emulating mechanisms
To mods (@Raemon): this would be an example of a place where a “have an LLM summarize the exchanges in this thread between these two users, with links / quotes” button could be helpful.
(Also, it seems that the option to temporarily switch to LW editor was disabled, so I cannot tag people?)
I’m struggling to get clear on the logical structure of your beliefs. Cf. your comments
Like, it seems like you can maybe get a strategically superhuman AI by relying on a lattice of more-or-less specialized superhuman skills (including superhuman engineering, and superhuman persuasion, and superhuman corporate strategy, and so on), without having much fluid intelligence.
To be clear, it also seems possible to me that we will make superhuman AI agents that don’t have this fluid intelligence special sauce. Those AIs will be adequate to automate almost all human labor, because almost all of human labor is more-or-less routine application of crystalized knowledge. We’ll be living in a radical new world of ~full automation, except for a small number of geniuses who are adding critical insight steps to the new cyborg-process of doing science.[1]
But I will be surprised if we hang out in that regime for very long, before the combined might of humanity’s geniuses augmented their armies of superhumanly capable routine engineers, and enormous computer infrastructure to do massive experiments, can’t hit on a mechanism that replicates the human fluid intelligence special sauce.[2]
and
I’m putting forward a disjunction:
Fluid intelligence isn’t necessary for Strategically Superhuman AI.or
LLM based agents will develop fluid intelligence on the default technological trajectory, via the application of not-very-clever ideas.or
There’s about one or two “breakthrough” ideas missing, that when combined with the existing LLM-agent techniques, will make LLM-agents that can do the fluid intelligence thing (or a substitute for the fluid intelligence thing). Having armies of LLM-agents that can automate engineering and experimentation seems like it should accelerate the discovery of those one or two breakthroughs. Those last two legs of the disjunction are assuming that there are not many pieces left before fluid intelligence is solved, but not making much of a claim about how many pieces we already have. Like, depending on what one means by “pieces”, maybe we have 0 out of 1 (and we’re likely to get that one in the next five years), or maybe we have 95 out of 100 (and we’re likely to get the last five in the next five years).
and
Like, if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 years (though we would still have an emergency, because automating science isn’t a necessary capability for destabilizing or ending the world via any of a number of different pathways).
And
I think I’m saying “crystalized intelligence can, to a large extent, substitute for fluid intelligence”.
Can you please clarify what are main big thingies that you expect to happen fairly confidently within 10 years? Can you clarify what drives your confident beliefs in those thingies happening? E.g. are you confident in some X happening due to being confident that we have fluid intelligence already or are close, or is that not the case?
Maybe you could make up words for the main scenarios and main capabilities / faculties you’re thinking of?
I think basically you said “they generate a novel concept that hadn’t been generated before and also people go on to use that concept in industry/science”, does that sound right?
Yeah, basically. I’m trying to be concrete here, and just saying “their intellectual output could be judged like human intellectual output is judged”.
And wanting an example of thing that’s more like “what’s something that’d make you go ‘okay, this was in fact as smart as a four year old’” (and therefore either the end is nigh, or, we’re about to learn that children in fact did not have nearly all the intelligence juice.”)
It’s a good question but it’s hard because that stuff looks from the outside like mostly pretty easy tasks. The way in which it is not easy is the way in which it is not “a task”. I guess, “very sample efficient learning” would be a concrete thing that 4yos do.
I think I would generally avoid saying that LLMs or current learning programs don’t generate new concepts simpliciter. Plausibly I did, but if so, I’d hopefully be able to claim that it was a typo or elision for space/clarity. What I said here was “good at generating interesting novel concepts on par with humans”. I know perfectly well that LLMs gain concepts (after a fashion) during training and have written about that. I would dispute them using / having concepts in the same relevant ways that humans have them though.
I’m confident that there’s lots of interesting content generally speaking contained in LLMs, gained through training, which is unknown to all humans. (The same could be said of other systems such as AlphaGo, and even old-style Stockfish during runtime if you admit that.)
(Though, apparently, and unlike human experts, the models don’t thereby learn words for those concepts, or have the ability to introspect and put handles on their conceptual representations, any more than I can introspect into how my visual cortex works.)
So like, yeah, they have something kinda related to human concepts in their full power, but not. This fits with my claim that they don’t have much originary general intelligence; they have distilled GI from humans, some more distilled stuff that’s not exactly “knowledge from humans” but is kinda more narrow (like, LLMs know word collocation frequencies like no human does); and some other stuff that’s not very general. I posit.
Thanks.
It’s a central example of a research task that requires insight and developing new, deep, technical concepts, not just patternmatching and reasoning by analogy to similar-seeming past problems.
(I wouldn’t use these words probably, but it’s a fine gesture in the direction.)
But everyone seems to be eliding this distinction between fluid intelligence and crystalized intelligence!
[Acknowledging that you acknowledged the ontology skew] Meh. I don’t think “fluid” vs. “crystallized” is all that important / clear / useful a distinction. It kinda sorta gets at the things, and other people bring it up so I use it. IDK if other people elide that distinction. I’d talk more about general intelligence, though that packs in additional stuff; I’m using something spiritually like Yudkowsky’s definition; something like “cross-domain optimization power divided by inputs”. Also gestured here: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#AGI
Some people don’t seem to notice the difference at all, and think that the crystallized intelligence of the AIs is the same kind of thing as the deep fluid intelligence.
IDK if this is true.
This is a kind of obvious point, but everyone around me seems to be operating on a strategic model that apparently ignores it!
No, I don’t think there’s one specific point I think everyone’s missing, or everyone with short timelines is missing. If there were, I suppose it would be more like “notice that you’re deferring way more than you’re noticing, and that others are too, and that this is creating very bad epistemic tidal forces” or something, but that’s kinda hard to discuss productively. I keep repeating that I don’t understand how you / others got to be so confident in short timelines in part so that it’s clear that IDK what you’re missing even on my perspective. I try to ask people to lay out their reasoning more in order to bring out background assumptions that I can disagree with better, which background assumptions support the [past x years of LLM observations] --> [confident short timelines] update. But that doesn’t work that well / is frustrating.
For example, I think that besides the “crystallizes int could make fluid int” thing, you’re also saying “we have fluid int or are close to it probably”. I’m asking about that and wanting to argue against that (that is, argue against the reasoning I’ve heard so far as not supporting the stated confidence). IDK how to phrase that as a positive statement for a position summary, because I would normally view it as weird / the wrong order of operations to start negating arguments (whatever your specific arguments are) without hearing them first, but I suppose I could preliminarily say “I’m guessing that you made a wrong update from observations to timelines based on a misconstrual of what matters about intelligence and what is the causal structure between cognitive faculties and cognitive performances.”, as described here https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=QBca6vhdeKkjyNLKa and here https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=DDaz5zcETcuyuy5Xx . But again, that feels awkward to guess about given that I don’t understand your reasons for the update.
Kill everyone? I’d be pretty surprised, like 1 in 100 or 200 surprised or something like that.
Generating interesting novel concepts on par with humans? See https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce?commentId=dqbLkADbJQJi6bFtN
Now, would you list something impressive that you do expect an AI to do in the next 1.5 years (that I might not say)?
All the impressive ML results so far have only worked either in a narrow subspace around the training data (e.g. LLMs, still mostly the case even with RL), or in very small worlds (e.g. pure-RL game-players). There has been ~zero progress on fluid/general intelligence. Therefore, extrapolating straight lines on graphs predicts ~zero progress on fluid/general intelligence by doing more of the same kind of thing. The induction on increasing ‘intelligence’ that lots of other people appeal to only works by inappropriate compression.
I largely agree with this, yeah. It would need some probability caveats; I put nontrivial, like O(1-5%), on various scenarios leading to AGI within 10 years—largely the sorts of things people talk about, and generally “maybe I’m just confused and GPT architecture / training plus RLVR and a bit more whatever basically implements a GI seed” or “maybe I’m totally confused about “GI seed” being much of a thing or being ~necessary for world-ending AI”.
I also wouldn’t have quite so tight a categorization of sources of capabilities. Cf. https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=QBca6vhdeKkjyNLKa
It’s still likely that we live in something like the 2011-Yudkowsky world as described in this tweet, with AGI to come from a lot of accumulation of insight. ML successes misleadingly make that world look falsified, if you aren’t tracking what they are and aren’t successes at.
Yeah, something like that. (I feel I have very little handle on how much insight is left, social dynamics around investment in conceptual “blue” capabilities research, etc.; hence very broad timelines. I also don’t much predict “there aren’t other major, impactful, discontinuous milestones before true world-ending AGI”; GPTs seem to be such a thing.)
Like, it’s evidence against the way of thinking that says understanding of intelligence is important. When you say (implicitly) ‘we probably need lots of AGI seedstuff’, I want to say ‘why isn’t the thought process you’re using to say that surprised, and downvoted, by how little stuff we needed to make LLMs?’.
It should probably be slightly directionally downvoted (though I’m not sure which preregistered hypotheses are doing better). But I think not very much, because I think that we did not observe “surprisingly obvious / easy / black-box idea generates lots of generally-shaped capabilities”. Partly that’s because the capabilities aren’t generally-distributed; e.g., gippities aren’t good at generating interesting novel concepts on par with humans, AFAIK. Partly that’s because there’s a great big screening-off explanation for the somewhat-generally-distributed capabilities that gippities do have: they got it from the data. I think we observed “surprisingly obvious / easy / black-box idea suddenly hoovers up lots of generally-shaped capabilities from the generally-shaped performances in the dataset (which we thus learned are surprisingly low-hanging fruit to distill from the data)”. (I do have the sense that there’s some things here that I’m not being clear about in my thinking, or at least in what I’ve written. One thing that I didn’t touch on, but that’s relevant, is that humans seem to exhibit this GI seedstuff, so it at least exists; whether it’s necessary to have that seedstuff to get various concrete consequences of AI is another question.)
if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 ye
Ok great. Can you clarify why you think this? Previously you wrote, in response to “What makes you think we’re close?”:
That the AI agents are already able to do, or are a few METR doublings from being able to do, almost all of the mental work that humans do, weighted by “time spent doing that work.”
Can you clarify / expand? What makes you think the METR results imply we’re close to having algorithmic ideas sufficient to automate scientific discovery?
(Haven’t had a chance to evaluate the post, but just commenting that it seems interesting, and I only just happened to stumble upon it by a random google search, which is probably my fault somehow but I’m just noticing...)