David Matolcsi

Karma: 2,450

David Matolcsi 29 Jul 2026 20:30 UTC
3 points
0
in reply to: Elias Schmied’s comment on: Infinite ethics and UDASSA
Sure, but normal hypotheses like “I’m a flesh-and-blood human on Earth, a tiny planet 13 billion years after the Big Bang” also take a lot of computation. My current guess is that logarithm of runtime is not that big of a penalty, and the Schmidhuber prior will give you something similar to the Solomonoff-prior. But if I’m wrong about this, and the runtime penalty is stronger than I imagine, that’s even worse news for the Schmidhuber prior because then you need to make up exotic hypotheses about how the world was born yesterday.

David Matolcsi 29 Jul 2026 19:47 UTC
5 points
0
in reply to: Elias Schmied’s comment on: Infinite ethics and UDASSA
Staring at the Schmidhuber paper for a while now, I don’t think the speed prior helps much here. My understanding is that instead of using the length of a program that produces a string, it uses the length of the program plus the logarithm of the runtime of the output string being computed. But I don’t think this helps much: we will be a bit envious of the aliens who lived 6 billion years after the Big Bang (they have 1 bit advantage over us), but we can still easily outcompete them by building a bigger obelisk. And I think world summoning by writing also works: in the thought experiment, the new world’s program code already encodes full brain-uploads and everything, so people can immediately start living happy lives in the new world; pointing to their experiences doesn’t take much more computation the amount of computation that happened in the outside world leading up to the moment of writing down the new world’s program code.
So I don’t think the speed prior fixes any of the problems described here, and meanwhile it seems to make various unintuitive predictions (at least according to Shmidhuber’s paper, I haven’t thought through if these are fixable), so altogether I’m not very excited about it.

David Matolcsi 29 Jul 2026 19:00 UTC
3 points
0
in reply to: Elias Schmied’s comment on: Infinite ethics and UDASSA
Thanks! No, I haven’t really heard of the Schmidhuber speed prior. Do you have a link to your favorite explanation of it?

David Matolcsi 29 Jul 2026 11:29 UTC
2 points
0
in reply to: Wei Dai’s comment on: You can, in fact, bamboozle an unaligned AI into sparing your life
Thanks! I’ve read these, and your writing in general has been an important influence on my thinking on these topics since I wrote this post two years ago.

I wrote up my current position on simulations and related topics in this sequence, largely building on some of your older writing. My understanding is that you are less in favor of the type of arguments presented there as you once were, but I’m still curious what you think of them if you have time to take a look.

David Matolcsi 22 Jul 2026 14:19 UTC
2 points
0
in reply to: 1a3orn’s comment on: AI 2040: Is it Actually a Deal?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”

David Matolcsi 21 Jul 2026 17:24 UTC
2 points
0
in reply to: 1a3orn’s comment on: AI 2040: Is it Actually a Deal?
Yeah, I agree that from a privacy perspective, it is a real loss to lose the open-weigh models. (Though I’m unsure how much I buy the argument that privacy-preserving options are super important even if the vast majority don’t use it. I haven’t thought about this too much.)

I didn’t mean it though that European governments will be the ones setting up the AIs. My understanding is that Plan A envisions AI still being provided by companies, and the number of near-frontier model providers being somewhat large in different jurisdictions, because full research transparency makes it relatively easy to catch up. And given that the companies can’t really compete on the capabilities of their models, they will need a different moat, and I think ensuring privacy is one of the most natural selling points they can offer. Jurisdictions will also be interested in their AI companies being successful, and they might also be interested in privacy for their own reasons (e.g. the example I gave of different European governments wanting to rely on a company’s services without any of the other governments spying on the company). I wouldn’t be surprised if we had companies in the Plan A world that are approximately as trustworthy on privacy as Tinfoil is now.

But most importantly, how realistic do you think the alternative is? This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout. It seems very unlikely to me, because I expect that governments will want to clamp down first on misuse then on autonomous AI criminals.

David Matolcsi 21 Jul 2026 5:37 UTC
4 points
−1
on: AI 2040: Is it Actually a Deal?
I very much agree with you first point (see my comment here), I’m also somewhat sympathetic to your second point, but I’m a bit confused about the third.

Isn’t it already true that the US government can in theory surveil all the conversations you have with closed-source models and with open-source models running on AWS and other large servers?

So is the only difference in Plan A that you lose locally-run open-weight models? I agree that’s a real loss, but my understanding is that currently only very few tech-savvy and unusually privacy-loving people use those. Plus it seems pretty likely that competeitive open-weight development will stop anyway due to economic reasons and normal government regulations.
At least in Plan A there are some official efforts towards privacy-preserving technologies (though you are right that there is very little guarantee that nations don’t secretly pull out of the privacy-preserving rules). And more importantly, multiple countries have their own AI and datacenters, so they can compete for customers by showing stronger verification mechanisms that they are not actually spying on you unless a reliable classifier flags that you are building superintelligence or WMDs. In particular, I can imagine that many small European governments will want to use the European AI for processing state secrets, so the different European governments will set up a pretty reliable multi-party auditing mechanism to verify that none of the others are spying on the conversations of the European AI. If that happens, you can also use the European AI if you care a lot about privacy.

Maybe I’m missing something, but Plan A doesn’t seem worse to me than the default trajectory from a privacy perspective.

David Matolcsi 20 Jul 2026 20:12 UTC
2 points
0
in reply to: Eric Neyman’s comment on: strawberry calm’s Shortform
Fair.

David Matolcsi 20 Jul 2026 16:45 UTC
5 points
2
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
Is this true? The three famous conjectures that I know that AI solved are the unit distance problem (counterexample), the Jacobian conjecture (counterexample) and cycle double cover conjecture (positive proof). This doesn’t feel very lopsided to me. Do you know of other similarly famous conjectures that AIs solved?

I would only bet at 30% on meeting grabby aliens

David Matolcsi16 Jul 2026 20:42 UTC

20 points

0 comments8 min readLW link

David Matolcsi 14 Jul 2026 9:11 UTC
4 points
0
in reply to: dynomight’s comment on: dynomight’s Shortform
I mean, the AI Futures Project is in fact predicting a way bigger crazy reality-shift than anything that happened in our lifetime or, to a first approximation, ever.

The end of work! Extreme abundance! Immortality! The birth of a second, smarter species! Space colonization!

Compared to that, what reality-shifts happened recently? Smart phones?

David Matolcsi 11 Jul 2026 19:50 UTC
2 points
0
in reply to: Thomas Kwa’s comment on: AI 2040: Plan A
I think countries sometimes have honor and follow their commitments even if it’s not locally game theoretically optimal.

David Matolcsi 11 Jul 2026 13:49 UTC
18 points
13
in reply to: Nisan’s comment on: Selective Optimism: a critique of AI 2040
To be fair, I think the fact that both his wife and daughter thought he was in favor of preemptive war is pretty strong evidence that he was talking about it in private conversations.

David Matolcsi 11 Jul 2026 10:38 UTC
22 points
6
in reply to: Daniel Kokotajlo’s comment on: AI 2040: Plan A
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India’s leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it’s smarter than other models, even though it doesn’t directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transparent enough about their research.
The American AISI only has 200 people at this point, because they started out small, and you can’t easily grow an organization more than 2x per year. Also, 50 out of the 200 people joined because they care about the ocean cluster harming the fish. (It is true though that probably it helps a bunch that they have good AI assistants. Maybe the crux is that I don’t believe it that much that the AI assistants will solve the dysfunctionality.)

The American AISI starts a bilateral conversation with the Indian regulators, but by the time they get to anywhere, it looks like the Europeans and two American companies have already probably copied something like what the Indians did. The world clearly didn’t end, and the AIs are just a bit smoother to use now. So no one escalates to a big diplomatic fight, and they quietly accept that the Indians didn’t publish enough details this time. Everyone keeps scaling up the technique with more compute and more data.

A month later, there is a new report which indicates that some interpretability techniques show that some scheming propensities are maybe going up, maybe linked to the new technique. After another month (do you know how slow it is to do anything in a government agency?) the American AISI calls up the President, and tells him to put pressure on American companies and other world leaders to roll back the new technique. Unfortunately, the President doesn’t have much time, because the Australian PM is pressuring him to make sure no one’s AI goes along with human rights violations in an ongoing war in the Middle East. Plus, he needs to speak at an environmentalist conference about the fish. (Also, he is still the President, running the non-AI aspects of the country.)

Now the President needs to call up the other world leaders and make it clear that he is willing to go to war if needed, because this interp scheming score really shouldn’t go up (unlike last time when a bias score was going up, but after a lot of yelling, people decided it’s not worth fighting over). So the President and other world leaders (reminder: Xi is 73, Modi is 75, Trump is 80, Biden left office at 82) get together and discuss the interp results and come to an agreement.

----

I think probably the sheer amount of bureaucracy still slows down algorithmic progress in the companies, and probably the government agencies pushing for safer directions and trying to ban more unsafe directions will have a non-zero correlation with actual safety. But it’s going to be a mess, and I don’t think you can slow down progress more than 2x this way, and the companies and countries that are more willing to skirt the rules will come out ahead more.

In contrast, GPUs, fabs and EUV machines are physical objects. I have much more trust in the government to competently control them and slow progress that way than through governing algorithmic progress.

David Matolcsi 11 Jul 2026 9:04 UTC
4 points
−6
in reply to: elifland’s comment on: AI 2040: Plan A
I think it’s pretty likely that if you start a very serious pause, with compute chain dismantling, then the race doesn’t restart for a very long time. And in the meantime, the world might just become a better place by default. I find it relatively plausible that in the next hundred years, the world would make as much progress towards democracy, stability and peace as Europe made since 1926. If US and China and the other great powers of 2126 will be on as friendly terms as France and Germany are now, then I will feel much better about slowly restarting AI development. If you think that the deal is very likely to break early, or if you don’t believe that the default secular trends in the world point in a good direction, then admittedly this less promising.

David Matolcsi 11 Jul 2026 8:13 UTC
8 points
0
in reply to: Daniel Kokotajlo’s comment on: AI 2040: Plan A
I think I’m just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don’t see who else could attempt a covert project with 100k hidden GPUs.

And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it’s over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.

(It’s also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can’t source new GPUs, no? I don’t remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that’s not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that’s bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn’t have advanced compute an robotics, and will lead to bombing.

My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the compute and the already existing huge robot economy and drone army. The path to victory seems much harder in the world without a lot of compute and robots.

David Matolcsi 11 Jul 2026 7:41 UTC
2 points
0
in reply to: Lukas Finnveden’s comment on: AI 2040: Plan A
Yes, that’s fair. But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won’t really be lifted or progress will be reversed in other ways. (E.g. I don’t expect markets to be enthusiastic about the proposal to put all data centers in Mongolia, under the threat of Chinese bombing.) So I think the initial stock market crash at the announcement of Plan A will still be huge—I would now guess between 30% and 50% as big as in the case of full dismantling of the compute supply chain.

I think one of the main political barriers to starting either the compute dismantling or Plan A is the unpopular initial crash and the corresponding industry outcry, and that will comparably be huge in both cases.

David Matolcsi 10 Jul 2026 15:11 UTC
26 points
2
on: AI 2040: Plan A
Given the level of political will and international coordination in the story, why can’t they just dismantle the compute supply chain?

If I understand correctly, the main argument agains Plan S is that at some point the global pause agreement will break down, and then we will be back where we are right now, and the race restarts again at a break-neck speed.

But what if part of the pause deal is that, both in China and in US allies, we destroy a large chunk of the existing GPUs, destroy the fabs, destroy the cutting-edge EUV machines, destroy the equipment necessary to build the EUV machines and disperse the teams working at all these companies so institutional knowledge is lost?

Once the compute supply chain is dismantled, governments can pay attention that no new cutting-edge chip fabs or necessary equipments are made—something that seems much easier to enforce than the restrictions on dangerous algorithmic progress in Plan A.
My understanding is that this wouldn’t have huge effects outside the AI industry. While there would be a huge stock market crash and it would be expensive to compensate the the affected companies, I’m not sure the financial loss would be more than 2x bigger than the stock market crash coming from all the priced-in growth being stopped in Plan A.

With the compute supply chain dismantled, the pause deal falling apart looks much less worrying. I expect that even if everyone started to try going full-speed when the deal fails, it would still take 10-20 years to rebuild the compute supply chain and train cutting-edge AIs again. During those 10-20 years, the pause deal can be revived again, and generally it seems good to have this longer time.
So I think the biggest question here is whether we expect the world to generally get better or worse during a long pause. I expect that the world is probably going to get better: I find the long-term historical trends encouraging, and I think that if we try to build AGI a hundred years from now, that will likely go better than if we try now. (Especially if we do some genetic engineering in the meantime.)

David Matolcsi 5 Jul 2026 8:30 UTC
2 points
−2
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
I think people’s main threat model these days for loss of control is a model achieving rouge internal deployment, or being able to operate internally without much effective oversight in other ways, or sabotaging the control and alignment of future models; I think literally escaping the datacenter and self-exfiltrating the weights is usually considered a relatively small part of the threat model these days. So I think when you say “risks from internal deployment”, people usually associate that with the AI achieving rouge internal deployment, or sabotaging alignment research for the next models, so it is confusing to people if you refer to risks from internal deployment as distinct from loss of control.

I agree that AI-enable human power concentration is a distinct risk which can be worsened by internal models becoming much more powerful than anything deployed externally.

David Matolcsi 5 Jul 2026 7:43 UTC
4 points
4
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
Did Carlsmith make impressive predictions? Or are you refering to Carl Shulman?

David Matolcsi

I would only bet at 30% on meet­ing grabby aliens

I would only bet at 30% on meeting grabby aliens