Max H

Karma: 3,012

Most of my posts and comments are about AI and alignment. Posts I’m most proud of, which also provide a good introduction to my worldview:

I also created Forum Karma, and wrote a longer self-introduction here.

PMs and private feedback are always welcome.

NOTE: I am not Max Harms, author of Crystal Society. I’d prefer for now that my LW postings not be attached to my full name when people Google me for other reasons, but you can PM me here or on Discord (m4xed) if you want to know who I am.

Max H 26 Feb 2026 15:04 UTC
2 points
0
in reply to: Zack_M_Davis’s comment on: The persona selection model
I don’t think I buy this one. Theorems and scientific phenomena are universal, but the model can only “see” them through the data we give them. The fact that chain-of-thought reasoning improves performance (and that you can intervene on them to change the answer) suggests that reasoning is meaningfully happening “in” the natural language token output even if it’s not perfectly faithful.
Buy it as what? You mentioned data-dependent generalization, I thought because you were using at as an example / reason that alien LLMs would be different. I pointed out in response that a lot of the data is actually the same, in some sense - someone or (some-LLM) who studies chemistry in English will be able to predict the effects of mixing baking soda and vinegar anywhere. Maybe before you get to full understanding, you can get various Sapir-Whorf-like effects based on what language you’re working in (e.g. perhaps LLMs learn chemistry quickly and more accurately in French, or something), but so what? Eventually with enough scale, they all saturate your evals, regardless of what language they initially learned in. My point is that the curriculum and format of the data in an Alien LLM corpus is at least arguably more similar to a human LLM corpus, than either dataset is similar to the format and curriculum in the respective human and alien natural growth processes.
That’s not my understanding; do you have a cite I should look at?
Not really. The thing I was thinking of and maybe mis-remembering or mis-applying, was translation between pairs of languages for which their are few or no direct human translations between the pair. IIUC, for most of the language pairs on Google Translate for which this is true, Google Translate used to work by translating from Language A <-> English <-> Language B, and this didn’t work very well. Nowadays Google Translate uses some kind of LLM and it apparently works much better. I hypothesize that LLMs faculty with translation would extend to alien languages as well; given how much LLMs have improved machine translation and how good LLMs are at deciphering codes and patterns in text generally. But I concede that’s not the same as definitely already being “extremely superhuman” at it, which was what I said in the grandparent.

Max H 25 Feb 2026 6:05 UTC
13 points
2
in reply to: Zack_M_Davis’s comment on: The persona selection model
I think it would help to be more specific about what behaviors you have in mind. The repetition trap that base models get stuck in sometimes seems like a good example. What else?
Pretty much every aspect of being an LLM, aside from (maybe / arguably) the human-like behaviors mentioned in this post, seem like examples of things that Earth LLMs would have in common with Alien LLMs.
When you ask an LLM to generate some code, it will one-shot write an entire file token-by-token, with no syntax errors (and often no logic errors if it’s something simple), without getting fatigued or bored. OTOH when Claude Code operates on an existing codebase, it starts every session Memento-style, re-orienting and / or operating on summaries and readmes as its only form of memory. Neither of these are how any human I know writes or relates to code, or any other form of text.
The phenomenon of data-dependent generalization—e.g., the fact that you can fit a network to randomized labels which don’t generalize, in contrast to how useful classifers do generalize—suggests that the algorithm learned during training depends on the data.
Sure, but all of the scientific knowledge, math, logical reasoning, etc. would be (functionally) almost exactly the same between a human and alien corpus, and that’s probably a huge chunk of where LLM capabilities come from. More planet-specific data would be similar to the degree that aspects of evolved life (culture, biology, sociology) are similar across different planets and evolutionary histories. Which, IDK, there might be some very different kinds of aliens out there, though some things are probably fairly convergent. The point is that any beings that evolved over billions of years in the same universe probably have more in common with each other than entities that they train artificially through a very different process.
That is, it’s not that “transformers” are a fixed sequence-predicting algorithm that can do English and Klingon; rather, the algorithms that transformers learn from being trained to predict English are different from the algorithms they learn from being trained to predict Klingon.
Aren’t LLMs actually extremely superhuman at translation and interpretation tasks, even for languages with few or no samples in training? Like an LLM that is really good at English could start translating and even generating Klingon (or a never-before-seen Alien language) in a much more sample-efficient way than the best human linguists. Skill at predicting English actually does generalize to predicting language in general, at least for other human languages, and probably for alien languages too.

Max H 25 Feb 2026 2:03 UTC
4 points
0
on: The persona selection model
Our experience of AI assistants is that they are astonishingly human-like. By this we don’t just mean that they use natural language. Rather, we mean that their behaviors and apparent psychologies resemble those of humans. As discussed above, AI assistants express emotions and use anthropomorphic language to describe themselves. They at times appear frustrated or panicked and make the sorts of mistakes that frustrated or panicked humans make. More broadly, human concepts and human ways of thinking appear to be the native language in which AI assistants operate.
IDK, I appreciate the caveating and precision and concreteness in the other sections, and there does seem to be non-zero predictive power here, but both “astonishingly” and “human-like” still feel kinda off / over-claiming to me.
I would say it’s more like, AI assistants sometimes exhibit behaviors of evolved + naturally-selected entities, which include humans, but could also include aliens and some animals. I’d still bet that an LLM trained on a corpus of an alien civilization’s text would have more in common with Earth LLMs, both behaviorally and internally, than Earth LLMs have in common with humans or alien LLMs have in common with aliens.
And “astonishingly” depends on your priors I suppose, but even if you don’t consider the training process at all, it doesn’t seem that surprising once you filter for “operates on human language and does something that humans find useful with it in a chatbot form factor”, that you get some “human-like” behaviors in contexts where that’s useful (or at least plausibly sensible / adaptive) given the rough capability and intelligence level of the system.
Roughly-human-level capability + operating on human text + doing something useful to humans with it is a lot of selection, and “human-like” behavior as defined is a pretty wide target, so it just isn’t too surprising (at least in retrospect) that you get some overlap.

Max H 18 Feb 2026 23:20 UTC
20 points
11
in reply to: npostavs’s comment on: You’re an AI Expert – Not an Influencer
For that particular example I think the bigger issue is that some experts were saying things that were failing to model reality or transparently silly in ways that anyone could notice, e.g. saying or implying that the risk of spreading disease depends on the righteousness of your cause.
There’s not really a way to hold actual-nonsense views (privately / smuggling or openly) and not come across as off-putting and nutty to lots of people. If a domain expert does hold merely unpopular or controversial views though, I think it’s better to be very direct and up front about them, e.g. a doctor could say something like:
“Large gatherings of any kind pose some risk of spreading respiratory illnesses, but outdoor gatherings are generally less risky than indoor gatherings. [cite apolitical evidence]

My recommendation as both a domain expert and a concerned citizen is that local governments should use all legal means to cancel public gatherings by default, but should make exceptions on a case-by-case basis when the importance of a gathering outweighs the risk. I believe that protesting for [particular cause] is one such case where the righteousness of the cause outweighs the risk.”
This might still be off-putting to a lot of people depending on the value of [cause], but at least it’s not an assault on basic reasoning.
But leaving out the second paragraph entirely, if it’s what you actually believe, and scrupulously sticking to only statements like the first paragraph mostly doesn’t work, because a Professional is only asked to make statements like the first paragraph in a context where it’s easy to figure out / guess / assume the worst on where they stand on things in the second paragraph, e.g. you are saying it to give public officials cover / justification to implement their own preferred policies on gathering restrictions.
In general, people are very, very good at figuring this kind of thing out and connecting the dots, even if you follow all the advice in this post and carefully never directly reveal your own politics or connect them to your expertise. It’s a very deep / old social instinct that modern media and PR training can’t reliably beat, and one of the lessons of the last decade is that you can quickly shred your credibility by trying.

Max H 18 Feb 2026 5:59 UTC
82 points
25
on: You’re an AI Expert – Not an Influencer
The advice in this post is framed as hard-won domain expertise, backed by empiricism and experience. But I think it’s actually closer to something more like folk wisdom from a milieu that is currently at a low point of cultural power and influence. Ostensibly-neutral (but in reality heavily blue-tribe-coded) “experts” who stay in their lane are not exactly popular or influential at the moment, and to my eye their track record of positive impact on various topics in the past has been mixed at best.
I agree that it’s good to be aware of these dynamics and dichotomy, but that doesn’t mean you have to try actively fitting into them. There are other options besides choosing not to play, and playing by the pre-existing rules.

Max H 18 Feb 2026 1:21 UTC
2 points
0
on: Max H’s Shortform
Medicine as an example of a failure of virtue ethics?
Epistemic status: kinda half-baked / not-confident claim
Context: I have been following @Richard_Ngo’s recent writing about consequentialism and virtue with some interest, though the thoughts below aren’t directly responding to anything in particular that he has written.
I think it’s uncontroversial around here to say that the field of medicine as a whole is under-performing and inadequate relative to what it could be—many people are getting sub-optimal treatment and health outcomes, and lots of questionably-useful research is produced and cited, vs. what’s theoretically / technologically possible given the collective resources spent on healthcare and research. Without getting into the specifics of what the failures and inadequacies of medicine are here though, I think it’s interesting and maybe informative to view them through the lens of a systemic failure of virtue ethics.
By virtue and virtue ethics below, I mean (what I think is) a relatively standard conception of virtue—acting in accordance with principles that are generally regarded as good, noble, pro-social, etc. according to one’s culture, in-group, and beliefs, with room for judicious flexibility based on experience and context, in contrast to deontology or consequentialism.
Richard lists “common-sense virtues like integrity, honor, kindness and dutifulness”, but I think it is fair to take a slightly more expansive view on what can be classified as a virtue in medicine: respect for and universal adherence to established procedure, loss aversion, and safety-ism are generally viewed negatively around these parts, but they are important operating principles in various medical systems. I claim that these are relatively central examples of virtues by the definition above - practitioners who adhere to them are typically regarded as good, virtuous, and following “common-sense” by the general public in both abstract terms and when applied to concrete situations. They’re more narrow than, say, “integrity”, but they’re still general and flexible enough to guide decision-making and evaluation in many different contexts.
And medical professionals and systems mostly do live up to their stated principles, so the problem is not a lack of virtue by individual participants—it’s that the field as a whole has settled on the wrong virtues, with no good mechanism for self-correction. An unusually careful, analytical, and consequentialist thinker might notice in the moment that the virtues of medicine I listed sometimes conflict with deeper and more general virtues of kindness and integrity when put into practice and applied strictly, but I don’t think that happens often enough for virtue-based decision-making to succeed in medicine.

Max H 13 Feb 2026 23:35 UTC
62 points
23
on: Max H’s Shortform
Peter Thiel pointed out that the common folk wisdom in business that you learn more from failure than success is actually wrong—failure is overdetermined and thus uninteresting.
I think you can make an analogous observation about some prosaic alignment research—a lot of it is the study of (intellectually) interesting failures, which means that it can make for a good nerdsnipe, but it’s not necessarily that informative or useful if you’re actually trying to succeed at (or model) doing something truly hard and transformative.
Glitch tokens, the hot mess work, and various things related to jailbreaking, simulators, and hallucinations come to mind as examples of lines of research and discussion that an analogy to business failure predicts won’t end up being centrally relevant to real alignment difficulties. Which is not to say that the authors of these works are claiming that they will be, nor that this kind of work can’t make for effective demonstrations and lessons. But I do think this kind of thing is unlikely to be on the critical path for trying to actually solve or understand some deeper problems.
Another way of framing the observation above is that it is an implication of instrumental convergence: without knowing anything about its internals, we can say confidently that an actually-transformative AI system (aligned or not) will be doing something that is at least roughly coherently consequentialist. There might be some intellectually interesting or even useful lessons to be learned from studying the non-consequentialist / incoherent / weird parts of such a system or its predecessors, but in my frame, these parts (whatever they end up being), are analogous to the failures and missteps of a business venture, which are overdetermined if the business ultimately fails, or irrelevant if it succeeds.

Max H 10 Feb 2026 16:44 UTC
13 points
1
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
hmm, I like the diagnosis of issues with the EA worldview, but I don’t really buy that they’re downstream of issues with consequentialism and utilitarianism itself.
I would say it’s more like: Effective Altruism has historically embraced a certain flavor of utilitarianism and naive consequentialism that attempts to be compatible with pre-existing vibes and (to some degree) mainstream politics. Concretely, EAs are (to their credit) willing to bite some strange bullets and then act on their conclusions, and are also generally pro-market compared to mainstream Democratic politics. But they’re still very, very Blue Tribe-coded culture-wise, and this causes them to deviate from actually-correct versions of consequentialism and utilitarianism in predictable directions.
Or: in my view, “Utilitarianism and non-naive consequentialism with guardrails” is pretty close to correct philosophy for humans; the issue is that the EA worldview systematically selects for the wrong guardrails^[1]. But better ones are available; for example Eliezer wrote this nearly 20 years ago: Ends Don’t Justify Means (Among Humans)
I’d be interested in hearing what kind of criticism you have of the posts in that sequence, and whether your issues with EA are more about a lack of emphasis and embrace of some of those principles, or that the ideas in that sequence are incomplete or even fundamentally mistaken or leading people astray.
1. ^
  ETA: And IMO this systematic selection is downstream of culture, not utilitarianism

Max H 4 Feb 2026 18:20 UTC
7 points
2
on: Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)
I agree that the claims the Anthropic researchers are making here are kind of wacky, but there is a related / not-exactly-steelman argument that has been floating around LW for a while, namely that there is an assumption by many old-school AI alignment people that transformer models will necessarily get more coherent as they get smarter (and larger), when (according to the arguers) that assumption hasn’t been fully justified or empirically been the case so far.
I recall @nostalgebraist’s comment here as an example of this line of discussion that was highly upvoted at the time.
So a generous / benign interpretation of the “Hot mess” work is that is an attempt to empirically investigate this argument and the questions that nostalgabraist and others have posed.
Personally, I continue to think that most of these discussions are kind of missing the point of the original arguments and assumptions that they’re questioning. The actual argument that coherence and agency are deeply and closely related to the ability to usefully and generally plan, execute, and adapt in a sample-efficient way, doesn’t depend on what’s happening in any particular existing AI system or assume anything about how they will work. It might or might not be the case that these properties and abilities will emerge directly in transformer models as they get larger—or they’ll emerge as a result of putting the model in the right kind of harness / embodiment, or as part of some advancement in a post-training process deliberately designed to shape them for coherence, or they’ll emerge in some totally different architecture / paradigm—but exactly how and when that happens isn’t a crux for any of my own beliefs or worldview.
Put another way, “a country of geniuses in a datacenter” had better be pretty good at working together and pursuing complex, long time-horizon goals coherently if they want to actually get anything useful done! Whether and how the citizens of that country contain large transformer models as a key component is maybe an interesting question from a timelines / forecasting perspective or if you want to try building that country right away, but it doesn’t seem particularly relevant to what happens shortly afterwards if you actually succeed.

Max H 29 Jan 2026 4:59 UTC
2 points
−1
in reply to: Kaj_Sotala’s comment on: Dario Amodei – The Adolescence of Technology
I suspect that there might be a crux that’s something like: are future AIs more naturally oriented toward something like consequentialist reasoning or shaped cognition:
I think this is closer to a restatement of your / Dario’s position, rather than a crux. My claim is that it doesn’t matter whether specific future AIs are “naturally” consequentialists or something else, or how many degrees of freedom there are to be or not be a consequential and still get stuff done. Without bringing AI into it at all, we can already know (I claim, but am not really expanding on here), that consequentialism itself is extremely powerful, natural, optimal, etc. and there are some very general and deep lessons that we can learn from this. “There might be a way to build an AI without all that” or even “In practice that won’t happen by default given current training methods, at least for a while” could be true, but it wouldn’t change my position.
But if something like this is where Dario is coming from, then I wouldn’t say that the problem is that he has missed a bit about how the world works. It’s that he has noticed that current AI looks like it’d be based on shaped cognition if extrapolated further,
OK, sure.
and that there hasn’t been a strong argument for why it couldn’t be kept that way relatively straightforwardly.
Right, this is closer to where I disagree. I think there is a strong argument about this that doesn’t have anything to do with “shaped cognition” or even AI in particular.
On the other hand, expertise research finds that trying to do consequentialist reasoning in most established domains is generally error-prone and a mark of novices, and experts have had their cognition shaped to just immediately see the right thing and execute it. And people are generally not very consequentialist about navigating their lives and just do whatever everyone else does, and often this is actually a better idea than trying to figure out everything in your life from first principles.
I would flag this as exactly the wrong kind of lesson / example to learn something interesting about consequentialism—failure and mediocrity are overdetermined; it’s just not that interesting that there are particular contrived examples where some humans fail at applying consequentialism. Some of the best places to look for the deeper lessons and intuitions about consequentialism are environments where there is a lot of cut-throat competition, possibility for outlier success and failure, not artificially constrained or bounded in time or resources, etc.

Max H 27 Jan 2026 18:31 UTC
4 points
−8
in reply to: Kaj_Sotala’s comment on: Dario Amodei – The Adolescence of Technology
You seem to be reading Dario to say “tendencies like instrumental power-seeking won’t emerge at all”.
I am more saying that the when Dario and others dismiss what they call “doomer” arguments as vague / clean theories, ungrounded philosophy, etc. and couch their own position as moderate + epistemically humble, what’s actually happening is Dario himself failing to generalize about how the world works.
We can imagine that some early powerful AIs will also miss those lessons / generalizations, either by chance or because of deliberate choices that the creators make, but if you count on that, or even just say that we can’t really know exactly how it will play out until we build and experiment, you’re relying on your own ignorance and lack of understanding to tell an overly-conjunctive story, even if parts of your story are supported by experiment. That chain of reasoning is invalid, regardless of what is true in principle or practice about the AI systems people actually build.
On Dario’s part I suspect this is at least partly motivated cognition, but for others, one way past this failure mode could be to study and reflect on examples in domains that are (on the surface) unrelated to AI. Unfortunately, having someone else spell out the connections and deep lessons from this kind of study has had mixed results in the past—millions of words have been spilled on LW and other places over the years and it usually devolves into accusations of argument by analogy, reference class tennis, navel-gazing, etc.

Max H 26 Jan 2026 23:39 UTC
31 points
3
on: Dario Amodei – The Adolescence of Technology
This “misaligned power-seeking” is the intellectual basis of predictions that AI will inevitably destroy humanity.
The problem with this pessimistic position is that it mistakes a vague conceptual argument about high-level incentives—one that masks many hidden assumptions—for definitive proof. I think people who don’t build AI systems every day are wildly miscalibrated on how easy it is for clean-sounding stories to end up being wrong, and how difficult it is to predict AI behavior from first principles, especially when it involves reasoning about generalization over millions of environments (which has over and over again proved mysterious and unpredictable). Dealing with the messiness of AI systems for over a decade has made me somewhat skeptical of this overly theoretical mode of thinking.
One of the most important hidden assumptions, and a place where what we see in practice has diverged from the simple theoretical model, is the implicit assumption that AI models are necessarily monomaniacally focused on a single, coherent, narrow goal, and that they pursue that goal in a clean, consequentialist manner. In fact, our researchers have found that AI models are vastly more psychologically complex, as our work on introspection or personas shows.
False / non-sequitur? Instrumental convergence and optimality of power-seeking are facts that describe important facets of reality. They unpack to precise + empirical + useful models of many dynamics in economics, games, markets, biology, computer security, and many adversarial interactions among humans generally.
The fact that these dynamics don’t (according to Dario / Anthropic) make useful predictions about the behavior of current / near-future AI systems, and the fact that current AI systems are not actually all that powerful or dangerous, is not a coincidence. But that isn’t at all a refutation of power-seeking and optimization as convergent behavior of actually-powerful agents! I think people who build AI systems every day are “wildly miscalibrated” on how empirically well-supported and widely applicable these dynamics and methods of thinking are outside their own field.
Dario’s “more moderate and more robust version” of how power-seeking could be a real risk seems like an overly-detailed just-so story about some ways instrumental convergence and power-seeking could emerge in current AI systems, conveniently in ways that Anthropic is mostly set up to catch / address. But the actually-correct argument is more like: if instrumental convergence and power-seeking don’t emerge in some form, then the AI system you end up with won’t actually be sufficiently powerful for what you want to do, regardless of how aligned it is. And even if you do manage to build something powerful enough for whatever you want to do that is aligned and doesn’t converge towards power-seeking, that implies someone else can build a strictly more powerful system which does converge, likely with relative ease compared to the effort you put in to build the non-convergent system. None of this depends on whether the latest version of Claude is psychologically complex or has a nice personality or whatever.

Max H 6 Jan 2026 5:02 UTC
18 points
14
in reply to: Richard_Ngo’s comment on: Toss a bitcoin to your Lightcone – LW + Lighthaven’s 2026 fundraiser
It’s the digital equivalent of printing something in vivid color on a glossy brochure rather than in black and white on A4 paper.
Hmm, maybe, but if that’s what it takes to get people to start consuming text content (instead of video), I’ll take it? By “medium” I was primarily referring to text over video, not that dashboards and polished graphics—I think if Lightcone were to spend time creating e.g. polished political ads or explainer / informational videos of any kind that would be pretty bad, unless they strongly endorsed the content.
I wasn’t super explicit about it in the grandparent, but two load-bearing and perhaps unusual things I believe:
- A large fraction of people, including elites, get basically all of their information and form their opinions and beliefs based on non-text sources: TV news, shortform video, instructional and educational videos, talking and listening to other people, etc.
  (I suspect a lot of LW and adjacent people who tend to consume a lot of text content don’t realize how widespread this is due to typical-minding.)
- This is probably really bad. There are a bunch of topics and discourse that are just totally unsuited for and unproductive when they happen in any medium other than long-form prose. Intellectuals can sometimes have live debates and discussions that others might get some value out of observing, but the source of the value here comes almost entirely from those intellectuals doing a lifetime of reading and writing beforehand as preparation.

Max H 6 Jan 2026 1:53 UTC
6 points
1
in reply to: Richard_Ngo’s comment on: Toss a bitcoin to your Lightcone – LW + Lighthaven’s 2026 fundraiser
I also feel pretty meh about the content of Deciding to Win and (somewhat) AI 2027, but as a medium, I think popularizing widely accessible, written long-form prose as a mode of elite discourse is unambiguously positive and potentially very impactful.
It’s kind of taken for granted around here, but “reliably and deeply engage with thoughtful written word” is absolutely not a bar that most discourse of any stripe meets. If Lightcone changes that even a little, for example by causing policymakers, elected officials, and other kinds of elites to shift a bit of their time from trawling social media and TV news, reacting to vibes and shallow popular sentiment, to engaging more with bloggers and public intellectuals, that seems like it could be worth a lot.
As a medium, a slick one page website seems like a nice middle ground in the design space between books (less accessible / widely shareable, often too long, don’t allow for good back-and-forth / commentary) and blog posts with comment sections (ideal for truly deep engagement, but not as legible / accessible for different reasons). And all of these are much better than TV news, “community engagement”, social media, video, etc.

Max H 29 Dec 2025 4:23 UTC
8 points
5
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
Also worth remembering that (actual) RSI was never a necessary condition for ruin. It seems at least plausible that at some point, human AI researchers on their own will find methods of engineering an AGI to sufficiently superhuman levels, to the point where the AI is smart enough to start developing nanotech and / or socially engineering humans for bootstrapping needs.
So even if labs were carefully monitoring for RSI and trying to avoid it (rather than deliberately engineering for it + frog boiling in the meantime), an AI inclined to take over might find that it doesn’t even need to bother with potentially dicey self-modifications until after it has already secured victory.

Max H 23 Dec 2025 19:00 UTC
15 points
7
on: Why does Eliezer make abrasive public comments?
Meta: any discussion or reaction you observe to abrasiveness and communication style (including the discussion here) is selected for people who are particularly sensitive to and / or feel strongly enough about these things one way or the other to speak up. I think if you don’t account for this, you’ll end up substantially overestimating the impact and EV in either direction.
To be clear, I think this selection effect is not simply, “lots of people like to talk about Eliezer”, which you tried to head off as best you could. If you made a completely generic post about discourse norms, strategic communication, the effects and (un)desirability of abrasiveness and snark, when and how much it is appropriate, etc. it might get less overall attention. But my guess is that it would still attract the usual suspects and object-level viewpoints, in a way that warps the discussion due to selection.
As a concrete example of the kind of effect this selection might have: I find the norms of discourse on the EA Forum somewhat off-putting, and in general I find that thinking strategically about communication (as opposed to simply communicating) feels somewhat icky and not particularly appealing as a conversational subject. From this, you can probably infer how I feel about some of Eliezer’s comments and the responses. But I also don’t usually feel strongly enough about it to remark on these things either way. I suspect I am not atypical, but that my views are underrepresented in discussions like this.

Max H 21 Dec 2025 22:52 UTC
4 points
0
on: What’s the Current Stock Market Bubble?
Another factor is that the absolute size of the boom (or bubble) is somewhat smaller than it appears if you just look at dollar-denominated increases in the value of stocks.
Stocks are denominated in dollars, and the value of a dollar has fallen substantially in real terms in the last couple of years, in large part due to tariffs (or uncertainty created by tariffs) and inflation. These are mostly independent effects from each other and the AI boom, and neither is particularly good for stimulating actual healthy economic growth, but they could soften the effect of any future bubble popping, because there is less actual growth / bubble to pop than what it looks like if you just look at returns in nominal terms.

Max H 4 Dec 2025 0:55 UTC
2 points
0
on: A Critique of Yudkowsky’s Protein Folding Heuristic
because any solver specialized to the evolutionary subset is guaranteed to fail once the promise is removed; outside that tightly curated domain, the mapping reverts to an unbounded and intractable instance class.
There are plenty of expansions you could make to the “evolutionary subset” (some of them trivial, some of them probably interesting) for which no theorem from complexity theory guarantees that the problem of predicting how any particular instance in the superset folds is intractable.
In general, hardness results from complexity theory say very little about the practical limits on problem-solving ability for AI (or humans, or evolution) in the real world, precisely because the “standard abstraction schemes” do not fully capture interesting aspects of the real-world problem domain, and because the results are mainly about classes and limiting behavior rather than any particular instance we care about.
In many hardness and impossibility results, “adversarial / worst-case” are doing nearly all of the work in the proof, but if you’re just trying to build some nanobots you don’t care about that. Or more prosaically, if you want to steal some cryptocurrency, in real life you use a side-channel or 0-day in the implementation (or a wrench attack); you don’t bother trying to factor large numbers.
IMO it is correct to mostly ignore these kinds of things when building your intuition about what a superintelligence is likely or not likely to be able to do, once you understand what the theorems actually say. NP-hardness says, precisely, that “if a problem is NP-hard (and $P \neq N P$ ), that implies that there is no deterministic algorithm anyone (even a superintelligence) can run, which accepts arbitrary instances of the problem and finds a solution in time steps polynomial in the size of the problem instance.”. This statement is precise and formal, but unfortunately it doesn’t mention protein folding, and even the implications that it has for an idealized formal model of protein folding are of limited use when trying to predict what specific proteins AlphaFold-N will / won’t be able to predict correctly.

Max H 27 Nov 2025 16:09 UTC
37 points
25
on: Alignment remains a hard, unsolved problem
if you asked me to pick between the CEV of Claude 3 Opus and that of a median human, I think it’d be a pretty close call (I’d probably pick Claude, but it depends on the details of the setup).
This example seems like it is kind of missing the point of CEV in the first place? If you’re at the point where you can actually pick the CEV of some person or AI, you’ve already solved most or all of your hard problems.
Setting aside that picking a particular entity is already getting away from the original formulation of CEV somewhat, the main reason I see to pick a human over Opus is that a median human very likely has morally-relevant-to-other-humans qualia, in ways that current AIs may not.
I realize this is maybe somewhat tangential to the rest of the post, but I think this sort of disagreement is central to a lot of (IMO misplaced) optimism based on observations of current AIs, and implies an unjustifiably high level of confidence in a theory of mind of AIs, by putting that theory on par with a level of confidence that you can justifiably have in a theory of mind for humans. Elaborating / speculating a bit:
My guess is that you lean towards Opus based on a combination of (a) chatting with it for a while and seeing that it says nice things about humans, animals, AIs, etc. in a way that respects those things’ preferences and shows a generalized caring about sentience and (b) running some experiments on its internals to see that these preferences are deep or robust in some way, under various kinds of perturbations.
But I think what models (or a median / randomly chosen human) say about these things is actually one of the less important considerations. I am not as pessimistic as, say, Wei Dai about how bad humans currently are at philosophy, but neither the median human nor any AI model that I have seen so far can talk sensibly about the philosophy of consciousness, morality, alignment, etc. nor even really come close. So on my view, outputs (both words and actions) of both current AIs and average humans on these topics are less relevant (for CEV purposes) than the underlying generators of those thoughts and actions.
In humans, we have a combination of (a) knowing a lot about evolution and neuroscience and (b) being humans ourselves. Taken together, these two things bridge the gap of a lot of missing or contentious philosophical knowledge—we don’t have to know exactly what qualia are to be pretty confident that other humans have them via introspection + knowing that the generators are (mechanically) very similar. Also, we know that the generators of goodness and sentience in humans generalize well enough, at least from median to >.1%ile humans—for the same reasons (a) and (b) above, we can be pretty confident that the smartest and most good among us feel love, pain, sorrow, etc. in roughly similar ways to everyone else, and being multiple standard deviations (upwards) among humans for smartness and / or goodness (usually) doesn’t cause a person to do crazy / harmful things. I don’t think we have similarly strong evidence about how AIs generalize even up to that point (let alone beyond).

Not sure where / if you disagree with any of this, but either way, the point is that I think that “I would pick Opus over a human” for anything CEV-adjacent implies a lot more confidence in a philosophy of both human and AI minds than is warranted.
In the spirit of making empirical / falsifiable predictions, a thing that would change my view on this is if AI researchers (or AIs themselves) started producing better philosophical insights about consciousness, metaethics, etc. than the best humans did in 2008, where these insights are grounded by their applicability to and experimental predictions about humans and human consciousness (rather than being self-referential / potentially circular insights about AIs themselves). I don’t think Eliezer got everything right about philosophy, morality, consciousness, etc. 15y ago, but I haven’t seen much in the way of public writing or discourse that has improved on things since then, and in many ways the quality of discourse has gotten worse. I think it would be a positive sign (but don’t expect to see it) if AIs were to change that.

Max H 16 Nov 2025 4:04 UTC
6 points
4
on: Relitigating the Race to Build Friendly AI
1. MIRI was rolling their own metaethics (deploying novel or controversial philosophy) which is not a good idea even if alignment turned out to be not that hard in a technical sense.
What specifically is this referring to? The Mere Goodness sequences?
I read your recent post about not rolling your own metaethics as addressed mostly at current AGI or safety researchers who are trying to build or align AIs today. I had thought what you were saying was that those researchers would be better served by stopping what they are doing with AI research, and instead spend their time carefully studying / thinking about / debating / writing about philosophy and metaethics. If someone asked me, I would point to Eliezer’s metaethics sequences (and some of your posts and comments, among others) as a good place to start with that.
I don’t think Eliezer got everything right about philosophy, morality, decision theory, etc. in 2008, but I don’t know of a better / more accessible foundation, and he (and you) definitely got some important and basic ideas right, which are worth accepting and building on (as opposed to endlessly rehashing or recursively going meta on).
Is your view that it was a mistake to even try writing about metaethics while also doing technical alignment research in 2008? Or that the specific way Eliezer wrote those particular sequences is so bad / mistaken / overconfident, that it’s a central example of what you want to caution against with “rolling your own metaethics”? Or merely that Eliezer did not “solve” metaethics sufficiently well, and therefore he (and others) were mistaken to move ahead and / or turn their attention elsewhere? (Either way / regardless, I still don’t really know what you are concretely recommending people do instead, even after reading this thread.)

Max H

Medicine as an example of a failure of virtue ethics?