Max H

Karma: 2,516

Most of my posts and comments are about AI and alignment. Posts I’m most proud of, which also provide a good introduction to my worldview:

I also created Forum Karma, and wrote a longer self-introduction here.

PMs and private feedback are always welcome.

NOTE: I am not Max Harms, author of Crystal Society. I’d prefer for now that my LW postings not be attached to my full name when people Google me for other reasons, but you can PM me here or on Discord (m4xed) if you want to know who I am.

Max H 24 Aug 2025 18:42 UTC
3 points
0
in reply to: Buck’s comment on: Buck’s Shortform
What specifically do you think is obviously wrong about the village idiot <-> Einstein gap? This post from 2008 which uses the original chart makes some valid points that hold up well today, and rebuts some real misconceptions that were common at the time.
The original chart doesn’t have any kind of labels or axes, but here are two ways you could plausibly view it as “wrong” in light of recent developments with LLMs:
- Duration: the chart could be read as a claim that the gap between the development of village idiot and Einstein-level AI in wall-clock time would be more like hours or days rather than months or years.
- Size and dimensionality of mind-space below the superintelligence level. The chart could be read as a claim that the size of mindspace between village idiot and Einstein is relatively small, so it’s surprising to Eliezer-200x that there are lots of current AIs landing in between them, and staying there for a while.
I think it’s debatable how much Eliezer was actually making the stronger versions of the claims above circa 2008, and also remains to be seen how wrong they actually are, when applied to actual superintelligence instead of whatever you want to call the AI models of today.
OTOH, here are a couple of ways that the village idiot <-> Einstein post looks prescient:
- Qualitative differences between the current best AI models and second-to-third tier models are small. Most AI models today are all roughly similar to each other in terms of overall architecture and training regime, but there are various tweaks and special sauce that e.g. Opus and GPT-5 have that Llama 4 doesn’t. So you have something like: Llama 4: GPT-5 :: Village idiot : Einstein, which is predicted by:
Maybe Einstein has some minor genetic differences from the village idiot, engine tweaks. But the brain-design-distance between Einstein and the village idiot is nothing remotely like the brain-design-distance between the village idiot and a chimpanzee. A chimp couldn’t tell the difference between Einstein and the village idiot, and our descendants may not see much of a difference either.
(and something like a 4B parameter open-weights model is analogous to the chimpanzee)
Whereas I expect that e.g. Robin Hanson in 2008 would have been quite surprised by the similarity and non-specialization among different models of today.
- Implications for scaling. Here’s a claim on which I think the Eliezer-200x Einstein chart makes a prediction that is likely to outperform other mental models of 2008, as well as various contemporary predictions based on scaling “laws” or things like METR task time horizon graphs:
  
  ”The rough number of resources, in terms of GPUs, energy, wall clock time, lines of Python code, etc. needed to train and run best models today (e.g. o4, GPT-5), are sufficient (or more than sufficient) to train and run a superintelligence (without superhuman / AI-driven levels of optimization / engineering / insight).”
  
  My read of task-time-horizon and scaling law-based models of AI progress is that they more strongly predict that further AI progress will basically require more GPUs. It might be that the first Einstein+ level AGI is in fact developed mostly through scaling, but these models of progress are also more surprised than Eliezer-2008 when it turns out that (ordinary, human-developed) algorithmic improvements and optimizations allow for the training of e.g. a GPT-4-level model with many fewer resources than it took to train the original GPT-4 just a few years ago.

Max H 13 Aug 2025 13:57 UTC
2 points
1
in reply to: the gears to ascension’s comment on: Forum Karma: view stats and find highly-rated comments for any LW user
Thanks for the report, should be fixed now.

The issue was that the LW GraphQL API has changed slightly, apparently. The user query suggested here no longer works, but something like:
```
{
      GetUserBySlug(slug: "max-h") {
          _id
          slug
          displayName
          pageUrl
          postCount
          commentCount
          createdAt
      }
}
```
works fine.

Max H 9 Jul 2025 15:56 UTC
21 points
9
on: Applying right-wing frames to AGI (geo)politics
I prefer (classical / bedrock) liberalism as a frame for confronting societal issues with AGI, and am concerned by the degree to which recent right-wing populism has moved away from those tenets.
Liberalism isn’t perfect, but it’s the only framework I know of that even has a chance of resulting in a stable consensus. Other frames, left or right, have elements of coercion and / or majoritarianism that inevitably lead to legitimacy crises and instability as stakes get higher and disagreements wider.
My understanding is that a common take on both the left and right these days is that, well, liberalism actually hasn’t worked out so great for the masses recently, so everyone is looking for something else. But to me every “something else” on both the left and right just seems worse—Scott Alexander wrote a bunch of essays like 10y ago on various aspects of liberalism and why they’re good, and I’m not aware of any comprehensive rebuttal that includes an actually workable alternative.
Liberalism doesn’t imply that everyone needs to live under liberalism (especially my own preferred version / implementation of it), but it does provide a kind of framework for disagreement and settling differences in a way that is more peaceful and stable than any other proposal I’ve seen.
So for example on protectionism, I think most forms of protectionism (especially economic protectionism) are bad and counterproductive economic policy. But even well-implemented protectionism requires a justification beyond just “it actually is in the national interest to do this”, because it infringes on standard individual rights and freedoms. These freedoms aren’t necessarily absolute, but they’re important enough that it requires strong and ongoing justification for why a government is even allowed to do that kind of thing. AGI might be a pretty strong justification!
But at the least, I think anyone proposing a framework or policy position which deviates from a standard liberal position should acknowledge liberalism as a kind of starting point / default, and be able to say why the tradeoff of any individual freedom or right is worth making, each and every time it is made. (And I do not think right-wing frameworks and their standard bearers are even trying to do this, and that is very bad.)

Max H 30 Jun 2025 4:58 UTC
19 points
19
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
I think it was fine for Nate to delete your comment and block you, and fine for you to repost it as a short form.
But my anecdote is a valid report of the historical consequences of talking with Nate – just as valid as the e/acc co-founder’s tweet.
“just as valid” [where validity here = topical] seems like an overclaim here. And at the time of your comment, Nate had already commented in other threads, which are now linked in a footnote in the OP:
By “cowardice” here I mean the content, not the tone or demeanor. I acknowledge that perceived arrogance and overconfidence can annoy people in communication, and can cause backlash. For more on what I mean by courageous vs cowardly content, see this comment. I also spell out the argument more explicitly in this thread.
So it’s a bit of a stretch to say that any AI safety-related discussion or interpersonal interaction that Nate has ever had in any context is automatically topical.
I also think your description of Nate’s decision to delete your comment as “not … allowing people to read negative truths about his own behavior” is somewhat overwrought. Both of the comment threads you linked were widely read and discussed at the time, and this shortform will probably also get lots of eyeballs and attention.
At the very least, there is an alternate interpretation, which is that the comment really was off-topic in Nate’s view, and given the history between the two of you, he chose to block + delete instead of re-litigating or engaging in a back-and-forth that both of you would probably find unpleasant and unproductive. Maybe it would have been more noble or more wise of him to simply let your comment stand without direct engagement, but that can also feel unpleasant (for Nate or others).

Max H 29 Jun 2025 11:50 UTC
6 points
1
in reply to: Stephen Martin’s comment on: Support for bedrock liberal principles seems to be in pretty bad shape these days
I gave YIMBYism as an example of a policy agenda that would benefit from more widespread support for liberalism, not as something I personally support in all cases.
A liberal argument for NIMBYism could be: people are free to choose the level of density and development that they want within their own communities. But they should generally do so deliberately and through the rule of law, rather than through opposition to individual developments (via a heckler’s veto, discretionary review processes that effectively require developers to lobby local politicians and woo random interest groups, etc.). Existing strict zoning laws are fine in places where they already exist, but new laws and restrictions should be wary of treading on the rights of existing property owners, and of creating more processes that increase discretionary power of local lawmakers and busybodies.

Max H 29 Jun 2025 3:15 UTC
5 points
2
in reply to: Gordon Seidoh Worley’s comment on: Support for bedrock liberal principles seems to be in pretty bad shape these days
Hmm, I’m not so pessimistic. I don’t think the core concepts of liberalism are so complex or unintuitive that the median civically engaged citizen can’t follow along given an amenable background culture.
And lots of policy, political philosophy, culture, big ideas, etc. are driven by elites of some form, not just liberalism. Ideas and culture among elites can change and spread very quickly. I don’t think a liberal renaissance requires “wrestling control” of any particular institutions so much as a cultural shift that is already happening to some degree (it just needs slightly better steering IMO).

Max H 28 Jun 2025 22:35 UTC
2 points
0
in reply to: the gears to ascension’s comment on: Support for bedrock liberal principles seems to be in pretty bad shape these days
I don’t personally feel screwed over, and I suspect many of the people in the coalitions I mentioned feel similarly. I am sympathetic to people who do feel that way, but I am not really asking them to unilaterally honor anything. The only thing in my post that’s a real concrete ask is for people who do already broadly support liberalism, or who have preferred policy agendas that would benefit from liberalism, be more outspoken about their support.

(To clarify, I have been using “liberalism” as a shorthand for “bedrock liberalism”, referring specifically to the principles I listed in the first paragraph—I don’t think everything that everyone usually calls “liberalism” is broadly popular with all the coalitions I listed, but most would at least pay lip service to the specific principles in the OP.)

Max H 28 Jun 2025 22:00 UTC
2 points
0
in reply to: the gears to ascension’s comment on: Support for bedrock liberal principles seems to be in pretty bad shape these days
I don’t really agree with the characterization of recent history as people realizing that “liberalism isn’t working”, and to the degree that I would advocate for any specific policy change, I support a “radical incrementalist” approach. e.g. maybe the endpoint of the ideal concept of property rights is pretty far from wherever we are right now, but to get there we should start with small, incremental changes that respect existing rights and norms as much as possible.
So for example, I think Georgism is a good idea in general, but not a panacea, and a radical and sudden implementation would be illiberal for some of the reasons articulated by @Matthew Barnett here.
I think a more realistic way to phase in Georgism that respects liberal principles would mainly take the form of more efficient property tax regimes—instead of complex rules and constant fights over property tax assessment valuations, there would be hopefully slightly less complex fights over land valuations, with phase-ins that keep the overall tax burden roughly the same. Some property owners with relatively low-value property on higher value land (e.g. an old / low density building in Manhattan) would eventually pay more on the margin, while others with relatively high-value property on lower value land (e.g. a newer / high density building in the exurbs) would pay a bit less. Lots of people in the middle of the property-vs-land value spectrum would pay about the same. But this doesn’t really get at the core philosophical objections you or others might have with current norms around the concept of property ownership in general.

Support for bedrock liberal principles seems to be in pretty bad shape these days

Max H28 Jun 2025 20:37 UTC

32 points

52 comments4 min readLW link

Max H 28 Apr 2025 23:24 UTC
4 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I kind of doubt that leaders at big labs would self-identify as being motivated by anything like Eliezer’s notion of heroic responsibility. If any do self-identify that way though, they’re either doing it wrong or misunderstanding. Eliezer has written tons of stuff about the need to respect deontology and also think about all of the actual consequences of your actions, even (especially when) the stakes are high:
The critical question here is: what happens if the plot successfully places the two of them in an epistemic Cooperation-Defection Dilemma, where rather than the two of them just having different goals, Carissa believes that he is mistaken about what happens...

In this case, Carissa could end up believing that to play ‘Defect’ against him would be to serve even his own goals, better than her Cooperating would serve them. Betraying him might seem like a friendly act, an act of aid.
(https://glowfic.com/replies/1874768#reply-1874768)
If he commits to a drastic action he will estimate that actual victory lies at the end of it, and his desperation and sacrifice will not have figured into that estimation process as positive factors. His deontology is not for sale at the price point of failure.
(https://glowfic.com/replies/1940939#reply-1940939)
Starting an AI lab in order to join a doomed race to superintelligence, and then engaging in a bunch of mundane squabbles for corporate control, seems like exactly the opposite of the sentiment here:
For Albus Dumbledore, as for her, the rule in extremis was to decide what was the right thing to do, and do it no matter the cost to yourself. Even if it meant breaking your bounds, or changing your role, or letting go of your picture of yourself. That was the last resort of Gryffindor.
(https://hpmor.com/chapter/93)
Also, re this example:
It also seemingly justifies or obligates Sam Altman to fight back when the OpenAI board tried to fire him, if he believed the board was interfering with his mission.

In general, it seems perfectly fine and normal for a founder-CEO to fight back against a board ouster—no need to bring heroic responsibility into it. Of course, all parties including the CEO and the board should stick to legal / above-board / ethical means of “fighting back”, but if there’s a genuine disagreement between the board and the CEO on how to best serve shareholder interests (or humanity’s interests, for a non-profit), why wouldn’t both sides vigorously defend their own positions and power?

Perhaps the intended reading of your example is that heroic responsibility would obligate or justify underhanded tactics to win control, when the dispute has existential consequences. But I think that’s a misunderstanding of the actual concept. Ordinary self-confidence and agency obligate you to defend your own interests / beliefs / power, and heroic responsibility says that you’re obligated to win without stepping outside the bounds of deontology or slipping into invalid / motivated reasoning.

Max H 6 Apr 2025 16:36 UTC
12 points
−2
on: Max H’s Shortform
Maybe the recent tariff blowup is actually just a misunderstanding due to bad terminology, and all we need to do is popularize some better terms or definitions. We’re pretty good at that around here, right?
Here’s my proposal: flip the definitions of “trade surplus” and “trade deficit.” This might cause a bit of confusion at first, and a lot of existing textbooks will need updating, but I believe these new definitions capture economic reality more accurately, and will promote clearer thinking and maybe even better policy from certain influential decision-makers, once widely adopted.
New definitions:
- Trade surplus: Country A has a bilateral “trade surplus” with Country B if Country A imports more tangible goods (cars, steel, electronics, etc.) from Country B than it exports back. In other words, Country A ends up with more real, physical items. Country B, meanwhile, ends up with more than it started with of something much less important: fiat currency (flimsy paper money) or 1s and 0s in a digital ledger (probably not even on a blockchain!).
  If you extrapolate this indefinitely in a vacuum, Country A eventually accumulates all of Country B’s tangible goods, while Country B is left with a big pile of paper. Sounds like a pretty sweet deal for Country A if you ask me.
  It’s OK if not everyone follows this explanation or believes it—they can tell it’s the good one because it has “surplus” in the name. Surely everyone wants a surplus.
- Trade deficit: Conversely, Country A has a “trade deficit” if it exports more tangible resources than it imports, and thus ends up with less goods on net. In return, it only receives worthless fiat currency from some country trying to hoard actual stuff for their own people. Terrible deal!
  Again, if you don’t totally follow, that’s OK, just pay attention to the word “deficit”. Everyone knows that deficits are bad and should be avoided.
Under the new definitions, it becomes clear that merely returning to the previous status quo of a few days ago, where the US only “wins” the trade war by several hundred billion dollars, is insufficient for the truly ambitious statesman. Instead, the US government should aggressively mint more fiat currency in order to purchase foreign goods, magnifying our trade surplus and ensuring that in the long run the United States becomes the owner of all tangible global wealth.
Addressing second order concerns: if we’re worried about a collapse in our ability to manufacture key strategic goods at home during a crisis, we can set aside part of the resulting increased surplus to subsidize domestic production in those areas. Some of the extra goods we’re suddenly importing will probably be pretty useful in getting some new factories of our own off the ground. (But of course we shouldn’t turn around and export any of that domestic production to other countries! That would only deplete our trade surplus.)

Max H 4 Mar 2025 6:35 UTC
9 points
−3
on: Self-fulfilling misalignment data might be poisoning our AI models
Describing misaligned AIs as evil feels slightly off. Even “bad goals” makes me think there’s a missing mood somewhere. Separately, describing other peoples’ writing about misalignment this way is kind of straw.
Current AIs mostly can’t take any non-fake responsibility for their actions, even if they’re smart enough to understand them. An AI advising someone to e.g. hire a hitman to kill their husband is a bad outcome if there’s a real depressed person and a real husband who are actually harmed. An AI system would be responsible (descriptively / causally, not normatively) for that harm to the degree that it acts spontaneously and against its human deployers’ wishes, in a way that is differentially dependent on its actual circumstances (e.g. being monitored / in a lab vs. not).
Unlike current AIs, powerful, autonomous, situationally-aware AI could cause harm for strategic reasons or as a side effect of executing large-scale, transformative plans that are indifferent (rather than specifically opposed) to human flourishing. A misaligned AI that wipes out humanity in order to avoid shutdown is a tragedy, but unless the AI is specifically spiteful or punitive in how it goes about that, it seems kind of unfair to call the AI itself evil.

Max H 25 Feb 2025 0:35 UTC
4 points
0
on: Export Surplusses
The original tweets seem at least partially tongue-in-cheek? Trade has lots of benefits that don’t depend on the net balance. If Country A buys $10B of goods from Country B and sells $9B of other goods to country B, that is $19B of positive-sum transactions between individual entities in each country, presumably with all sorts of positive externalities and implications about your economy.
The fact that the net flow is $1B in one direction or the other just doesn’t matter too much. Having a large trade surplus (or large trade deficit) is a proxy for generally doing lots of trading and industry, which will tend to correlate with a lot of other things that made or will make you wealthy. But it would be weird if a country could get rich solely by running a trade surplus, while somehow avoiding reaping any of the other usual benefits of trading. “Paying other countries to discern your peoples’ ability to produce” is plausibly a benefit that you get from a trade surplus even if you try hard to avoid all the others, though.

Max H 6 Feb 2025 1:45 UTC
52 points
18
on: Wired on: “DOGE personnel with admin access to Federal Payment System”
My guess is that the IT and computer security concerns are somewhat exaggerated and probably not actually that big of a deal, nor are they likely to cause any significant or lasting damage on their own. At the very least, I wouldn’t put much stock in what a random anonymous IT person says, especially when those words are filtered through and cherry-picked by a journalist.
These are almost certainly sprawling legacy systems, not a modern enterprise cloud where you can simply have a duly authorized superadmin grant a time-limited ReadOnly IAM permission to * or whatever, along with centralized audit logging and sophisticated change management. There are probably more old-school / manual processes in place, which require going through layers of humans who aren’t inclined to be cooperative or speedy, especially at this particular moment. I think Elon (and Trump) have some justified skepticism of those processes and the people who implemented them.
Still, there’s going to be some kind of audit logging + technical change management controls, and I kind of doubt that any of Elon’s people are going to deliberately sidestep or hide from those, even if they don’t follow all the on-paper procedures and slash some red tape.
And ultimately, even sophisticated technical controls are not a substitute for actual legal authority, which they (apparently / perhaps questionably) have. I’ll be much more concerned if they start violating court orders, even temporarily. e.g. I think it would be very bad (and more plausible than IT malfeasance or negligence) if they are ordered to stop doing whatever by a lower court, but they don’t actually stop, or slow-walk on reversing everything, because they expect to win on appeal to the Supreme Court (even if they’re correct about their appeal prospects). IDK about Elon’s people specifically, but I think ignoring court orders (especially lower courts and temporary injunctions) is a more dangerous form of institutional decay that Trump is likely to usher in, especially since the legislature seems unlikely to offer any real push-back / rebuke.

Max H 5 Feb 2025 1:04 UTC
13 points
5
in reply to: evhub’s comment on: evhub’s Shortform
It seems more elegant (and perhaps less fraught) to have the reference class determination itself be a first class part of the regular CEV process.
For example, start with a rough set of ~all alive humans above a certain development threshold at a particular future moment, and then let the set contract or expand according to their extrapolated volition. Perhaps the set or process they arrive at will be like the one you describe, perhaps not. But I suspect the answer to questions about how much to weight the preferences (or extrapolated CEVs) of distant ancestors and / or “edge cases” like the ones you describe in (b) and (c) wouldn’t be affected too much by the exact starting conditions either way.
Re: the point about hackability and tyranny, humans already have plenty of mundane / naturalistic reasons to seek power / influence / spread of their own particular current values, absent any consideration about manipulating a reference class for a future CEV. Pushing more of the CEV process into the actual CEV itself minimizes the amount of further incentive to do these things specifically for CEV reasons. Whereas, if a particular powerful person or faction doesn’t like your proposed lock-in procedure, they now have (more of) an incentive to take power beforehand to manipulate or change it.

Max H 26 Dec 2024 19:03 UTC
43 points
0
on: How to have Polygenically Screened Children
My wife completed two cycles of IVF this year, and we had the sequence data from the preimplantation genetic testing on the resulting embryos analyzed for polygenic factors by the unnamed startup mentioned in this post.
I can personally confirm that the practical advice in this post is generally excellent.
The basic IVF + testing process is pretty straightforward (if expensive), but navigating the medical bureaucracy can be a hassle once you want to do anything unusual (like using a non-default PGT provider), and many clinics aren’t going to help you with anything to do with polygenic screening, even if they are open to it in principle. So knowing exactly what you want and what you need to ask for is key.
Since this post was written, there have been lots of other developments and related posts in this general area:
And probably many others I am forgetting. But if you’re a prospective parent looking for practical advice on how to navigate the IVF process and take advantage of the latest in genetic screening technology, this post is still the best place to start that I know of. Some of the things in the list above are more speculative, but the technology for selection is basically ready and practical now, and the effect size doesn’t have to be very large for it to beat the status quo of having an embryologist eyeball it.
I think this post is a slam dunk for a +9 and a spot in the LW canon, both for its object-level information and its exemplary embodiment of the virtue of empiricism and instrumental rationality. The rest of this review details my own experience with IVF in the U.S. in 2024.
This section of the original post basically covers it, but to recap, the two main things you’ll want to ask your prospective IVF clinic are:
- Can we use Orchid Labs or Genomic Prediction for PGT?
- Can we implant any healthy embryo of our choosing? (some clinics can have policies against sex selection, etc.)
In my experience, the best time to ask these questions is in-person at your initial consultation; it can be hard to get answers over the phone / in email before you’re at least a prospective patient, since they generally require a doctor or NP to answer.
The good news is, if you get affirmative answers to these questions, you mostly don’t need to worry about whether the clinic or your embryologist is skeptical or even outright hostile to polygenic screening, because you can simply request your sequence data from your PGT provider directly and have it analyzed on your own.
Note: Genomic Prediction offers their own polygenic screening test (LifeView), but if you’re planning to have a third party analyze the sequence data for non-disease traits, you don’t need this. You can just have your IVF clinic order PGT-A tests from GP, and then request your raw sequence data from GP directly once you get the PGT-A results. AFAIK the actual sequencing that GP does is the same regardless of what test you order from them, and they’re happy to share the raw data with you if you ask.
Another thing you’ll want to confirm is whether you can store any aneuploid embryo(s) long-term. Aneuploid embryos are typically considered not viable and most clinics won’t even try to implant them. But they’re worth keeping frozen speculatively, in case future technology allows them to be repaired, etc. Some clinics will have a policy of automatically discarding confirmed-aneuploid embryos unless you make specific arrangements to do something else with them. Usually this will be a question in a big packet of paperwork you’ll have to fill out about what you want to do with the embryos / eggs in various scenarios, e.g. death / divorce / time limit etc. so just make sure to read carefully.
On selecting a good-quality IVF clinic: the live birth metrics in this post are a good starting point, but probably confounded somewhat by the population the clinic serves, and realistically the biggest factor in determining how many embryos you get is going to be your personal health factors and age. My wife is over 30 and in pretty good shape and took a bunch of vitamins before / during the cycles (B12, Omega-3, Myo-Inositol, a prenatal vitamin) and we ended up with 21 eggs retrieved across two cycles, which is right around the expected number for her age.
These attrited down to 10 mature embryos during the fertilization process, 5 of which were screened out as aneuploid via ordinary PGT-A. We had the remaining 5 embryos polygenically screened.
We’re planning to start implanting next year, so I can’t speak to that part of the process personally yet, but overall we’re very happy with the results so far. There was a clear “winner” among the embryos we screened that will be our first choice for implantation, but it’s nice to have all the data we can on all the embryos, and depending on how things go we may end up using more than one of them down the line.
The polygenic screening wasn’t cheap, and given the number of embryos we had to select from, the maximum possible benefit is pretty mild (2.3 bits of selection if we only use one of the 5 and it successfully implants). But given the hassle and expense of IVF itself (not to mention pregnancy and raising a child...) it seems overwhelmingly worth it. We were considering IVF for fertility preservation reasons anyway, so the main question for us was the marginal cost of the extra screening on top.
I’d like to write a longer post with my own takes about having / raising kids on the eve of AI, but for now I’ll just say a few things:
- The choice about whether to have kids is always a personal one, but right now seems like as good a time as any, historically speaking, and takes like these seem crazy wrong to me. Even if you’re thinking purely in terms of the advantage you can give your kids (probably not a good idea to think this way), by far the biggest advantage is being marginally earlier. If you’re interested in having kids but worried about the costs or effectiveness of current IVF / screening methods, consider just having kids the old-fashioned way instead of delaying.
- I have short timelines and expect the default outcome from ASI being developed is swift human extinction, but I still think it’s worth having kids now, at least for me personally. My wife and I had happy childhoods and would have enjoyed being alive even it was only for a short time, and hopefully that’s at least a somewhat heritable trait. And regardless of the endpoint, I expect things to be pretty OK (even great) at least for me and my family, right up until the end, whenever that is. Despite being a “doomer”, I am long the market, and expect to be pretty well-off even if those particular bets don’t pay off and the larger world is somewhat chaotic in the short term.
- 2.3 bits of selection on a single kid realistically isn’t going to make a difference in any kind of “Manhattan project for human intelligence enhancement” and that’s not why we did it. But my sense from having gone through this process is that the barriers to some of the things that Tsvi describes here are more social and financial and scale than technical.

Max H 18 Oct 2024 1:58 UTC
2 points
1
in reply to: Matthew Barnett’s comment on: The Hidden Complexity of Wishes
My main point was that I thought recent progress in LLMs had demonstrated progress at the problem of building such a function, and solving the value identification problem, and that this progress goes beyond the problem of getting an AI to understand or predict human values.
I want to push back on this a bit. I suspect that “demonstrated progress” is doing a lot of work here, and smuggling an assumption that current trends with LLMs will continue and can be extrapolated straightforwardly.
It’s true that LLMs have some nice properties for encapsulating fuzzy and complex concepts like human values, but I wouldn’t actually want to use any current LLMs as a referent or in a rating system like the one you propose, for obvious reasons.
Maybe future LLMs will retain all the nice properties of current LLMs while also solving various issues with jailbreaking, hallucination, robustness, reasoning about edge cases, etc. but declaring victory already (even on a particular and narrow point about value identification) seems premature to me.
Separately, I think some of the nice properties you list don’t actually buy you that much in practice, even if LLM progress does continue straightforwardly.
A lot of the properties you list follow from the fact that LLMs are pure functions of their input (at least with a temperature of 0).
Functional purity is a very nice property, and traditional software that encapsulates complex logic in pure functions is often easier to reason about, debug, and formally verify vs. software that uses lots of global mutable state and / or interacts with the outside world through a complex I/O interface. But when the function in question is 100s of GB of opaque floats, I think it’s a bit of a stretch to call it transparent and legible just because it can be evaluated outside of the IO monad.
Aside from purity, I don’t think your point about an LLM being a “particular function” that can be “hooked up to the AI directly” is doing much work - input() (i.e. asking actual humans) seems just as direct and particular as llm(). If you want your AI system to actually do something in the messy real world, you have to break down the nice theoretical boundary and guarantees you get from functional purity somewhere.
More concretely, given your proposed rating system, simply replace any LLM calls with a call that just asks actual humans to rate a world state given some description, and it seems like you get something that is at least as legible and transparent (in an informal sense) as the LLM version. The main advantage with using an LLM here is that you could potentially get lots of such ratings cheaply and quickly. Replay-ability, determinism and the relative ease of interpretability vs. doing neuroscience on the human raters are also nice, but none of these properties are very reassuring or helpful if the ratings themselves aren’t all that good. (Also, if you’re doing something with such low sample efficiency that you can’t just use actual humans, you’re probably on the wrong track anyway.)

Max H 8 Oct 2024 1:51 UTC
2 points
−7
in reply to: Raemon’s comment on: “Slow” takeoff is a terrible term for “maybe even faster takeoff, actually”
Economic / atomic takeoff

Max H 8 Oct 2024 1:49 UTC
2 points
0
on: “Slow” takeoff is a terrible term for “maybe even faster takeoff, actually”
For specifically discussing the takeoff models in the original Yudkowsky / Christiano discussion, what about:
Economic vs. atomic takeoff
Economic takeoff because Paul’s model implies rapid and transformative economic growth prior to the point at which AIs can just take over completely. Whereas Eliezer’s model is that rapid economic growth prior to takeover is not particularly necessary—a sufficiently capable AI could act quickly or amass resources while keeping a low profile, such that from the perspective of almost all humanity, takeover is extremely sudden.
Note: “atomic” here doesn’t necessarily mean “nanobots”—the goal of the term is to connote that an AI does something physically transformative, e.g. releasing a super virus, hacking / melting all uncontrolled GPUs, constructing a Dyson sphere, etc. A distinguishing feature of Eliezer’s model is that those kinds of things could happen prior to the underlying AI capabilities that enable them having more widespread economic effects.

IIUC, both Eliezer and Paul agree that you get atomic takeoff of some kind eventually, so one of the main disagreements between Paul and Eliezer could be framed as their answer to the question: “Will economic takeoff precede atomic takeoff?” (Paul says probably yes, Eliezer says maybe.)
Separately, an issue I have with smooth / gradual vs. sharp / abrupt (the current top-voted terms) is that they’ve become a bit overloaded and conflated with a bunch of stuff related to recent AI progress, namely scaling laws and incremental / iterative improvements to chatbots and agents. IMO, these aren’t actually closely related nor particularly suggestive of Christiano-style takeoff—if anything it seems more like the opposite:
- Scaling laws and the current pace of algorithmic improvement imply that labs can continue improving the underlying cognitive abilities of AI systems faster than those systems can actually be deployed into the world to generate useful economic growth. e.g. o1 is already “PhD level” in many domains, but doesn’t seem to be on pace to replace a significant amount of human labor or knowledge work before it is obsoleted by Opus 3.5 or whatever.
- Smooth scaling of underlying cognition doesn’t imply smooth takeoff. Predictable, steady improvements on a benchmark via larger models or more compute don’t tell you which point on the graph you get something economically or technologically transformative.

Max H 19 Jun 2024 23:15 UTC
6 points
0
on: My AI Model Delta Compared To Christiano
I’m curious what you think of Paul’s points (2) and (3) here:
- Eliezer often talks about AI systems that are able to easily build nanotech and overpower humans decisively, and describes a vision of a rapidly unfolding doom from a single failure. This is what would happen if you were magically given an extraordinarily powerful AI and then failed to aligned it, but I think it’s very unlikely what will happen in the real world. By the time we have AI systems that can overpower humans decisively with nanotech, we have other AI systems that will either kill humans in more boring ways or else radically advanced the state of human R&D. More generally, the cinematic universe of Eliezer’s stories of doom doesn’t seem to me like it holds together, and I can’t tell if there is a more realistic picture of AI development under the surface.
- One important factor seems to be that Eliezer often imagines scenarios in which AI systems avoid making major technical contributions, or revealing the extent of their capabilities, because they are lying in wait to cause trouble later. But if we are constantly training AI systems to do things that look impressive, then SGD will be aggressively selecting against any AI systems who don’t do impressive-looking stuff. So by the time we have AI systems who can develop molecular nanotech, we will definitely have had systems that did something slightly-less-impressive-looking.
And specifically to what degree you think future AI systems will make “major technical contributions” that are legible to their human overseers before they’re powerful enough to take over completely.
You write:
I expect that, shortly after AIs are able to autonomously develop, analyze and code numerical algorithms better than humans, there’s going to be some pretty big (like, multiple OOMs) progress in AI algorithmic efficiency (even ignoring a likely shift in ML/AI paradigm once AIs start doing the AI research). That’s the sort of thing which leads to a relatively discontinuous takeoff.
But how likely do you think it is that these OOM jumps happen before vs. after a decisive loss of control?
My own take: I think there will probably be enough selection pressure and sophistication in primarily human-driven R&D processes alone to get to uncontrollable AI. Weak AGIs might speed the process along in various ways, but by the time an AI itself can actually drive the research process autonomously (and possibly make discontinuous progress), the AI will already also be capable of escaping or deceiving its operators pretty easily, and deception / escape seems likely to happen first for instrumental reasons.
But my own view isn’t based on the difficulty of verification vs. generation, and I’m not specifically skeptical of bureaucracies / delegation. Doing bad / fake R&D that your overseers can’t reliably check does seem somewhat easier than doing real / good R&D, but not always, and as a strategy seems like it would usually be dominated by “just escape first and do your own thing”.

Max H

Sup­port for bedrock liberal prin­ci­ples seems to be in pretty bad shape these days

Support for bedrock liberal principles seems to be in pretty bad shape these days