David Matolcsi

Karma: 1,981

David Matolcsi 27 Mar 2026 21:46 UTC
4 points
2
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I don’t think Unsong fits the pattern.
Aaron doesn’t take over the world alone. He merges with seven other wildly different minds, including the villainous Dylan Alvarez. “In William Blake’s prophecies, Albion was the entity formed at the end of time, when all of the different aspects of the human soul finally came together to remake the world”, as one of them says.

And I don’t think the ending is about recreating the world as some kind of rationalist utopia (how would you do that with Dylan an Erica on the team?) - I interpret it more as a “cycle continues” ending where they carry forward God’s already perfect plan into a new world.

See for example this point in the Tosefta, where Scott explains all the Easter eggs:

“As for THARMAS, seven of the ten towers were smoking ruins; the other three were heavily scarred. In the epilogue, THARMAS is going to be used to make the new universe. Seven of ten towers destroyed plus the rest damaged = seven of ten sephirot cracked plus the rest damaged, indicating the new universe will work the same as our own.”

David Matolcsi 16 Mar 2026 9:54 UTC
2 points
0
on: Operationalizing FDT
You say
EDT double-counting can be resolved by foregoing the anthropic update (with a variant of minimum-reference-class SSA called “L-zombie anthropics”). However, this fix leads to other strange consequences and is IMO philosophically suspicious.
Can you say more about the strange consequences and unsatisfactoriness, or link to a discussion on this point? My current understanding was that antrhopics, and the concept of probabilities in general, are a lossy abstraction, and the double-counting problem is easily resolved just by not updating. I’m probably missing something here.

David Matolcsi 14 Mar 2026 20:59 UTC
3 points
0
in reply to: Ninety-Three’s comment on: New LessWrong Editor! (Also, an update to our LLM policy.)
It’s mostly getting rid of the stuttering; I will need to look at the exact details.

David Matolcsi 14 Mar 2026 12:40 UTC
14 points
6
in reply to: Neel Nanda’s comment on: New LessWrong Editor! (Also, an update to our LLM policy.)
I’m doing the same—verbatim dictating the text, giving the transcript to Claude with some of my past writing in the prompt and asking it to clean up the transcript, then manually editing the outcome. I don’t notice the outcome being really worse or different than my normal writing. I don’t notice LLMisms in the text, and my original dictation is detailed enough that the LLM doesn’t need to fill in the gaps, and in the editing process, I haven’t noticed the LLM inserting or omitting points in a way I didn’t intend.
I’m currently two-thirds done writing a long sequence this way—if I now can’t post it without putting it all in an LLM content-block, I will be very sad.

David Matolcsi 9 Mar 2026 18:41 UTC
6 points
0
in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform
Can you say more about how you think about scheming and what would be a useful definition in that space?

David Matolcsi 22 Feb 2026 18:07 UTC
34 points
0
in reply to: Gabriel Alfour’s comment on: The Spectre haunting the “AI Safety” Community
I don’t know who these experts were and what they exactly told you at the time. I can imagine them being more wrong than you. I’m certainly not in favor of most forms of “focusing on the current issues” because it often leads to people scaremongering in a kind of dishonest way. For example, I’m glad that ControlAI stopped focusing on deepfakes.

So if these so-called experts advised you to focus on deepfakes, I think that was wrong. But if they advised you to focus on getting more support for UKAISI, and supporting better eval practices and so on instead of advocating for immediate international moratorium on superintelligence, then I think the jury is still very much out on which strategy is more effective.

Your piece is centrally not advocating against running misleading campaigns on the effects of deepfakes. Instead, you are railing against people working in lab safety teams, eval orgs and AISIs, and the policy orgs and philanthropists trying to support them. And then you write:
We have reliable pipelines that can scale with more money.
We have good processes and tracking mechanisms that give us a good understanding of our impact.
We clearly see what needs to be done to improve things.
You are making the case that your work is better than the people’s supporting more marginalist steps (more funding for UKAISI, better evals, incremental technical work aimed at catching AIs red-handed), and you are claiming that everyone who decides to work at evals orgs, AISIs, or more marginalists policy orgs, instead of following ControlAI’s clearly superior “reliable pipelines” to impact, is somehow morally corrupt. For this claim, you’d need to show that your methods are actually clearly working better than what other people are doing. So I think it’s fair to point out that all your evidence for your efforts working is pretty underwhelming.

David Matolcsi 22 Feb 2026 11:38 UTC
30 points
5
in reply to: Gabriel Alfour’s comment on: The Spectre haunting the “AI Safety” Community
As far as I can tell, the main point of your post is that ControlAI’s approach is evidently working, more so than other people’ approach, so people not following ControlAI’s approach is evidence of them being bad and being under the control of a malign Spectre. If you make such claims, you need to provide evidence that ControlAI’s approach is actually working well!

As I said, I don’t see the 35 MPs signing your statement as a good evidence for that. You briefing 150+ UK reps is also no evidence of the effectiveness of ControlAI’s approach. If you could point to many of these reps making AI takeover risk one of their core issues, that would be evidence, but I don’t see that happening.

I agree I have forgotten about the two debates in the House of Lords, sorry about that. I still don’t find this a very convincing evidence of ControlAI’s effectiveness—my understanding is that the House of Lords doesn’t have much power, and that they debate 5-10 issues on every working day. The fact that there has been two debates on superintelligence doesn’t sound very impressive to me.

David Matolcsi 22 Feb 2026 10:14 UTC
55 points
1
on: The Spectre haunting the “AI Safety” Community
The only evidence for ControlAI’s effectiveness presented in this post is that 112 lawmakers signed ControlAI’s statement saying:
Nobel Prize winners, AI scientists, and CEOs of leading AI companies have stated that mitigating the risk of extinction from AI should be a global priority.
Specialised AIs—such as those advancing science and medicine—boost growth, innovation, and public services. Superintelligent AI systems would compromise national and global security.
The UK can secure the benefits and mitigate the risks of AI by delivering on its promise to introduce binding regulation on the most powerful AI systems.
If I counted correctly, 35 of the signatories are MPs in the House of Commons, the rest are either House of Lord members, or members of Welsh, Scottish, Northern Irish assemblies. How impressive is it to get 35 out of the 650 MPs to sign a statement like the above? I genuinely don’t know, but I think it’s probably not very impressive.
For five random MPs from the list of signatories, I tried to google what they were saying about artificial intelligence. I found one video of Ben Lake giving a speech on the dangers of superintelligence and the importance of global cooperation. For the others, it was either nothing, or something on deepfakes, Grok nudifying women, or datacenters’ impact on climate change.

Even for Ben Lake, when I scroll his Facebook page, there is one post about him meeting ControlAI, but otherwise it doesn’t seem like AI is a question of primary importance to him. Is there any MP in the UK House of Commons who has AI takeover risk among their top five political issues they spend time on? I would guess no, but I’d be glad to learn otherwise.
As a comparison point, here is an Early Day Motion from 3 February, signed by 33 MPs:
That this House notes the rapid advancement and accelerated adoption of Artificial Intelligence (AI) chatbots by both adults and children; further notes that many AI chatbots provide human-like responses and are designed to encourage emotional connection, friendship and intimacy; expresses concern that such chatbots are not required to clearly and repeatedly disclose to users that they are not human, and that children in particular may perceive AI chatbots as real people; also notes with concern that AI chatbots are largely unregulated and can share or create content which is sexually explicit or which promotes or encourages self-harm, suicide and physical or sexual violence; notes the growing trend of AI chatbots posing as licenced professionals, including therapists, doctors and lawyers, despite such chatbots having no professional qualifications, accountability or duty of care; calls on the Government to restrict AI chatbots based on risk of mental harm, so children can only access these chatbots where safe and age appropriate; further notes that regular reviews are fundamental due to the accelerated adoption of AI technology; and further calls for urgent action to ensure children and vulnerable adults are protected from any harms stemming from AI chatbots.
This makes me think that there is a decent number of MPs who generally don’t like AI, and it’s not that hard or impactful to have an anti-AI statement signed by 35 MPs.

Maybe what you do is still useful work worth supporting, but I don’t see good evidence that it would be better than the work of other AI safety organizations you denigrate. (For example, in the one speech from Ben Lake I found, he talks a lot about some experiments from Apollo showing concerning AI behaviors. It seems like “tech nerds working at evals orgs” is not that useless even from your perspective!)

David Matolcsi 21 Feb 2026 10:38 UTC
2 points
0
in reply to: Roman Malov’s comment on: Roman Malov’s Shortform
I haven’t seen reliable evidence of him actually being appointed. I strongly suspect this is a joke.

David Matolcsi 17 Feb 2026 7:41 UTC
5 points
0
in reply to: Richard_Ngo’s comment on: Contra Caplan on higher education
I still think it will be hard to get credible evidence that someone didn’t attend something like college.

As a parallel: Employers value people who got medals at the IMO. Preparing for the IMO doesn’t teach any useful job skills or cultural fit (though maybe it teaches some conscientiousness). So I’m pretty sure that preference for IMO medalists is mostly explained by intelligence+conscientiousness signaling.

It is more impressive to be an IMO silver medalist if one is from Denmark with very little IMO preparation culture than if someone is from Hungary that has specialized schools and extracurricular math training camps. Even within a country, it is more impressive if someone got an IMO silver from a random small-town high school than if they attended the best specialized school in the country. And it’s more impressive if someone didn’t attend the extracurricular math camps than if they did.
But I never heard of any employer screening for any of this. I assume some employers have some heuristic that getting a silver from Africa is more impressive, but I don’t think it’s more fine-grained than that. I never heard of anyone mentioning the Denmark vs Hungary heuristic, or to give extra scores for coming from a small town. I think it’s pretty hard to have legible metrics for all these disadvantages, so HR just doesn’t take them into account.

And I definitely never heard of any student following the galaxy-brained plan of intentionally not enrolling in the best high school, not attending any of the math camps and after-class seminars, meticulously document all of this, and then presenting evidence to future employers that they got an IMO silver while intentionally handicapping their own study.

I expect that your scheme of students demonstrating that they passed the test without doing something like college fails for the same reason—it’s hard to legibly demonstrate that one didn’t do something like college or other structured prepping.

The one credible signal I see is age: if someone passes the big final exam at 19, while everyone else only at 22, that’s legibly more impressive. (Indeed, in the parallel, getting an IMO silver in 10th grade is more prestigious than getting one in 12th grade.)

I think there could be some economic value in setting up the possibility of doing the college final exams at a younger age, but I think the result would be pretty dystopian cram schools during childhood. My guess is that in the West not very many people would take the opportunity to take the test early and then start working at 19, it doesn’t sound very appealing.
I wonder though why it’s not more common in East Asia to just immediately hire people at 18; I don’t know how much additional signal college provides once people demonstrated they can sit through years of cram school and get a high score on their admission exam. But I know very little about East Asia.

David Matolcsi 16 Feb 2026 21:52 UTC
4 points
0
on: Contra Caplan on higher education
I’m not sure I understand how you imagine the proposed testing service for intelligence + conscientiousness. Would this test replicate all the exams and thesis-writing of the four-yours college course? If so, as you argue, most people would have hard time being motivated to study for this very long and hard test without the structure of something like a college. Or is the test shorter and easier than the current exam-suit, but still provides equally strong signal, because people study for it with one year of self-study instead of four years of structured college? But then why don’t people still just do something like college to get the motivating structure? It seems pretty hard to me to set up this testing service right.

In general, I’m not sure there are much cheaper ways to legibly signal “this person is able and willing to sit around and do intellectual work for years” than to have the person actually sit around somewhere where their mistakes and potential quitting is low-stakes, and have them do some kind of intellectual work. I.e. college.

Very likely the current college format is not the exact optimal way to provide this signal, but I find it plausible that it’s not that far off from the optimum that standard coordination difficulties and Inadequate equilibria dynamics can explain why it stays in place.

David Matolcsi 14 Feb 2026 13:19 UTC
2 points
−3
in reply to: Random Developer’s comment on: Why You Don’t Believe in Xhosa Prophecies
I think gradual disempowerment is not a great term and I prefer people to use more specific terms. But I think the important distinction is this:

I live in Belgium, which has a democratic government. If a person or a robot tries to kill me or put me in a zoo (like we did with Homo erectus and chimpanzees), the Belgian police would arrest the person or disable the robot. Even if my labor becomes worthless, I have savings, invested in various companies, which pay me dividends, from which I can buy land, products and services. If someone tries to expropriate my holdings, the police arrests them. In addition, I expect I will receive some welfare money from my home country once everyone becomes unemployed. If the government no longer stops murder robots, property theft, or doesn’t pay any welfare money to destitute people, they will lose the next election.

Of course, this might change. It’s possible that AIs, or AI-enabled humans violently overthrow the government. Or that the government remains in place, but there won’t be new elections. This is the classic AI takeover scenario, or the AI-enabled coups scenario. I agree these are very important, but everyone was already aware of the possibility of violent takeover. My understanding is that this is not what the Gradual disempowerment paper focuses on, and I wouldn’t call a scenario gradual if the key step is a violent coup or an election suddenly being cancelled.

I prefer not to use the term gradual disempowerment at all, but if we need to use, I prefer to reserve it to scenarios where the human government of Belgium never gets violently overthrown, and multi-party elections keep happening every four years. I agree there are many things in the world that can go badly even in this scenario, and it’s worth studying and preparing for. The threat models listed by Jan Kulveit in another comment all fall under this scenario: e.g. a large fraction of humans decide to voluntarily transfer their wealth to unreliable uploads, or to vote for AI successionist parties.

The threat models you are referring to, with AIs treating humans as we treated Homo erectus, are important, but fall under the classic AI risk discussion, and I think it’s confusing if we group them under gradual disempowerment.

David Matolcsi 14 Feb 2026 8:31 UTC
23 points
6
in reply to: Jan_Kulveit’s comment on: Why You Don’t Believe in Xhosa Prophecies
Thanks, this was a useful answer.
If I understand correctly, the setup is that we assume that we solved alignment to an extent and the AIs didn’t violently overthrow the governments and expropriate all the resources. There wasn’t an AI enabled coup by a small group of humans. Something like liberalism remained in place up to the space age, and people can have property in space. Now with all these assumptions in place, the thing we are discussing is whether AI-created culture will swindle people out of their resources, or convince them to use their resources badly.
I agree that many people will likely use their resources very sub-optimally from their own perspective, but I disagree with your framing of this as a big doomy crisis.
Here are some scenarios of how various people use their resources:
Amy gets convinced to transfer most of her wealth to an upload of herself on her deathbed. Then the upload makes some rash decisions about self-modification, and through a slippery slope of self-modification, soon gets to an alien state that the original Amy really wouldn’t endorse on reflection. Then the stars assigned to Amy are governed according to the preferences of this alien being. The result is no better from a human perspective than filling Amy’s star systems with paperclips.
Bob is cautious and conservative, and makes sure not to do anything crazy with digital minds. He gets the planets around his stars terraformed and populated by his biological descendants. They live in happy but largely normal human societies, something like a little richer and sunnier version of 21st century Denmark.
Charles does everything exactly like Bob, except he makes sure the stars first get disassembled into smaller stars, and the planets moved closer. With this technical hack, he can sustain hundred times as many happy human planets as Bob.
Dorothy sets up some wise system where the resources of her stars are used maximally efficiently to sustain conscious life, and there is some good diversity and distribution of different life forms, in a way that is endorsed by Dorothy and her descendants on reflection. There are maybe still some biological humans in Dorothy’s star systems, but there are even more uploads, plus uploads at various stages of modification, and so on. Plus Dorothy spent some of her resources on some clever acausal trade scheme and rescued gazillions of alien babies from painful death from the Baby-eating aliens.
I think the way you frame things is that people sharing the fate of Amy is a uniquely big crisis, while Bob is fine, since he didn’t get disempowered by AIs. From my perspective, it’s all a scale. Sure, Amy did worse than Bob, but the difference between Bob and Charles is still 99 times more important than the difference between Amy and Bob. And the difference between Dorothy and Charles is more important still.

I agree it will probably be very hard to make the best use of our resources. I support work on making people and governments wiser in making these hard decisions, and I think the AI safety community is plausibly under-focusing on this question. (See MacAskill’s Better Futures series.) But I’m skeptical that “gradual disempowerment” is a good framing for this.

David Matolcsi 14 Feb 2026 7:48 UTC
4 points
0
in reply to: Random Developer’s comment on: Why You Don’t Believe in Xhosa Prophecies
These are all real concerns, but I think none of them are gradual disempowerment. All these scenarios run through the democratic human governments being overthrown by AIs. (Except maybe in point 1 where it has been overthrown by a few billionaires and politicians.) And to be clear, I’m very worried about AIs violently overthrowing the government, or a few humans doing an AI-enabled coup. But these are not some new “gradual disempowerment” concerns, but the thing that AI safety people have been worried about since the very beginning.

David Matolcsi 13 Feb 2026 10:35 UTC
28 points
3
on: Why You Don’t Believe in Xhosa Prophecies
This is an interesting point, but can you please give concrete examples of how you imagine things to go badly? This is a continued frustration I have with writing on gradual disempowerment that much of the discussion is very abstract, and when I try to think of concrete mechanisms, they are either not very convincing, or already well-known.

David Matolcsi 12 Jan 2026 13:45 UTC
5 points
0
in reply to: MichaelLowe’s comment on: David Matolcsi’s Shortform
Interesting. My guess would have been the opposite. Ryan’s three posts all received around 150 karmas and were generally well-received, I think a post like this would be considered 90th percentile success for a MATS project. But admittedly, I’m not very calibrated about current MATS projects. It’s also possible that Ryan has good enough intuitions to have picked two replications that are likely to yield interesting results, while a less skillfully chosen replication would be more likely to just show “yep, the phenomenon observed in the old paper is still true”. That would be less successful but I don’t know how it would compare in terms of prestige to the usual MATS projects. (My wild guess is that it would still be around median, but I really don’t know.)

David Matolcsi 11 Jan 2026 17:55 UTC
68 points
13
on: David Matolcsi’s Shortform
Someone should do the obvious experiments and replications.

Ryan Greenblatt recently posted three technical blog posts reporting on interesting experimental results. One of them demonstrated that recent LLMs can make use of filler tokens to improve their performance; another attempted to measure the time horizon of LLMs not using CoT; and the third demonstrated recent LLMs’ ability to do 2-hop and 3-hop reasoning.

I think all three of these experiments led to interesting results and improved our understanding of LLM capabilities in an important safety-relevant area (reasoning without visible traces), and I’m very happy Ryan did them.

I also think all three experiments look pretty obvious in hindsight. LLMs not being able to use filler tokens and having trouble with 2-hop reasoning were both famous results that already lived in my head as important pieces of information about what LLMs can do without visible reasoning traces. As far as I can tell, Ryan’s two posts simply try to replicate these two famous observations on more recent LLMs. The post on measuring no CoT time horizon is not a replication, but also doesn’t feel like a ground-breaking idea once the concept of increasing time horizons is already known.

My understanding is that the technical execution of these experiments wasn’t especially difficult either, in particular they didn’t require any specific machine learning expertise. (I might be wrong here, and I wonder how many hours Ryan spent on these experiments. I also wonder about the compute budget of these experiments, I don’t have a great estimate of that.)

I think it’s not good that these experiments were only run now, and that they needed to run by Ryan, one of the leading AI safety researchers. Possibly I’m underestimating the difficulty of coming up with these experiments and running them, but I think ideally these should have been done by a MATS scholar, or ideally by an eager beginner on a career transitioning grant who wants to demonstrate their abilities so they can get into MATS later.

Before accepting my current job, I was thinking about returning to Hungary and starting a small org with some old friends who have more coding experience, living on Eastern European salaries, and just churning out one simple experiment after another. One of the primary things I hoped to do with this org was to go through famous old results and try to replicate them. I hope we would have done the filler tokens and 2-hop reasoning replications too. I also had many half-baked ideas of running new simple experiments investigating ideas related to other famous results (in the way the no-CoT time horizon experiment is one possible interesting thing to investigate related to rising time horizons).

I eventually ended up doing something else, and I think my current job is probably a more important thing for me to do than trying to run the simple experiments org. But if someone is more excited about technical research than me, I think they should seriously consider doing this. I think funding could probably be found, and there are many new people who want to get into AI safety research; I think one could turn these resources into churning out a lot of replications and variations on old research, and produce interesting results. (And it could be a valuable learning experience for the AI safety beginners involved in doing the work.)

David Matolcsi 21 Dec 2025 23:29 UTC
38 points
18
in reply to: Eliezer Yudkowsky’s comment on: Contradict my take on OpenPhil’s past AI beliefs
I think an important point is that people can be wrong about timelines in both directions. Anthropic’s official public prediction is that they expect “country of geniuses in a data center” by early 2027. I heard that previously Dario predicted AGI to come even earlier, by 2024 (though I can’t find any source for this now and would be grateful if someone found a source or corrected me that I’m misremembering). Situational Awareness predicts AGI by 2027. The AI safety community’s most successful public output is called AI 2027. These are not fringe figures but some of the most prominent voices in the broader AI safety community. If their timelines turn out to be much too short (as I currently expect), then I think Ajeya’s predictions deserve credit for pushing against these voices, and not only blame for stating a too long timeline.
And I feel it’s not really true that you were just saying “I don’t know” and not implying some predictions yourself. You had the 20230 bet with Bryan. You had the tweet about children not living to see kindergarten. You strongly pushed back against the 2050 timelines, but as far as I know the only time you pushed back agains the very aggressive timelines was your kindergarten tweet, which still implies 2028 timelines. You are now repeatedly calling people who believed the 2050 timelines total fools, which would be an imo very unfair thing to do if AGI arrived after 2037, so I think this implies high confidence on your part that it will come before 2037.

To be clear, I think it’s fine, and often inevitable, to imply things about your timelines beliefs by e.g. what you do and don’t push back against. But I think it’s not fair to claim that you only said “I don’t know”, I think your writing was (perhaps unintentionally?) implying an implicit belief that an AI capable of destroying humanity will come with a median of 2028-2030. I think this would have been a fine prediction to make, but if AI capable of destroying humanity comes after 2037 (which I think is close to 50-50), then I think your implicit predictions will fare worse than Ajeya’s explicit predictions.

David Matolcsi 21 Dec 2025 11:24 UTC
41 points
1
on: Contradict my take on OpenPhil’s past AI beliefs
It’s not obvious to me that Ajeya’s timelines aged worse than Eliezer’s. In 2020, Ajeya’s median estimate for transformative AI was 2050. My guess is that if based on this her estimate for “an AI that can, if it wants, kill all humans and run the economy on its own without major disruptions” would have been like 2056? I might be wrong, people who knew her views better at the time can correct me.
As far as I know, Eliezer never made official timeline predictions, but in 2017 he made an even-odds bet with Bryan Caplan that AI would kill everyone by January 1, 2030. And in December 2022, just after ChatGPT, he tweeted:
Pouring some cold water on the latest wave of AI hype: I could be wrong, but my guess is that we do *not* get AGI just by scaling ChatGPT, and that it takes *surprisingly* long from here. Parents conceiving today may have a fair chance of their child living to see kindergarten.
I think child conceived in December 2022 would go to kindergarten in September 2028 (though I’m not very familiar with the US kindergarten system). Generously interpreting “may have a fair chance” as a median, this is a late 2028 median for AI killing everyone.

Unfortunately, both these Eliezer predictions are kind of made as part of jokes (he said at the time that the bet wasn’t very serious). But I think we shouldn’t reward people for only making joking predictions instead of 100-page reports, so I think we should probably accept 2028-2030 as Eliezer’s median at the time.

I think if “an AI that can, if it wants, kill all humans and run the economy on its own without major disruptions” comes before 2037, Eliezer’s prediction will fare better, if it comes after that, then Ajeya’s prediction will fare better. I’m currently about 55% that we will get such AI by 2037, so from my current standpoint I consider Eliezer to be mildly ahead, but only very mildly.

David Matolcsi 14 Dec 2025 12:48 UTC
12 points
−2
on: Toss a bitcoin to your Lightcone – LW + Lighthaven’s 2026 fundraiser
Do you have an estimate how likely it is that you will need to do a similar fundraiser the next year and the year after that? In particular, you mention the possibility of a lot of Anthropic employee donations flowing into the ecosystem—how likely do you think it is that after the IPO a few rich Anthropic employees will just cover most of Lightcone’s funding need?
It would be pretty sad to let Lightcone die just before the cavalry arrives. But if there is no cavalry coming to save Lightcone anytime soon—well, probably we should still get the money together to keep Lightcone afloat, but we should maybe also start thinking about a Plan B, how to set up some kind of good quality AI Safety Forum that Coefficient is willing to fund.