David Matolcsi

• David Matolcsi 17 Sep 2026 21:01 UTC

  2 points

  0

  in reply to: Alex_Altair’s comment on: Alex_Al­tair’s Shortform

  I highly doubt that ancient people didn’t have this concept. What about the Ark of the Covenant? Or the Ship of Theseus, a ship maintained through centuries in the Athenian harbor, allegedly since the time of Theseus, the legendary civic hero of Athens?^[1] Sure, these were religious objects, but as far as I can tell basically all ancient culture and nationhood was intertwined with religion, so cultural heritage was usually also religious heritage.
  
  Another interesting example is the Uffington White Horse that has been maintained through millennia. People kept maintaining it even after England converted to Christianity, so this is an example of cultural heritage that is maintained despite no religious connection.

  1. ^

     I was very amused when I first read in Plutarch that this was a literal ship and not just a thought experiment.

• David Matolcsi 15 Sep 2026 7:24 UTC

  7 points

  2

  in reply to: habryka’s comment on: Con­sider how your global gov­er­nance pro­posal is differ­ent from the EU Code of Practice

  That’s not at all how it works though!

  The part of the AI Act we are talking about is enforced by the Commission. The Commission can directly request measures or hand out fines. This is similar to the DMA and DSA laws, which were passed a bit before the AI Act. Under the DSA and DMA laws, Apple, AliExpress, Google, Meta, Temu and X have already been fined in the last one and a half years for sums ranging between 120 million and 890 million. TikTok, AliExpress and X (and maybe others, I’m mostly going by a quick Claude summary here) have also agreed to various changes in response to the Commission’s DMA and DSA requests.

  Companies can and often do attack fining decisions in court, and the adjudication can indeed take a while (though I think usually not six years). But the fine needs to be paid or at least locked down by the company at the time of fining. Later, if the Court decides on the company’s side, the Commission needs to pay back the money with a relatively high interest rate. But this means that even if court cases take a long time, companies still have an incentive to comply with the law, since the fine is paid immediately.

  I would appreciate if you passed on this correction to the people among whom this is “the standard reason” the AI Act can’t work.

  (Note that I’m not a lawyer, I’m not working directly on this question, I’m definitely not speaking for my employer, and I’m just writing things based on public information I and Claude found online.)

• David Matolcsi 14 Sep 2026 15:44 UTC

  4 points

  1

  on: Brock­man says Hug­gingFace in­ci­dent model had not been al­ign­ment-trained; ETA: roon clar­ifies that it was

  At 6:35 in this video:
  https://​​www.youtube.com/​​watch?v=56GuvofZgB4

• David Matolcsi 14 Sep 2026 13:05 UTC

  3 points

  1

  in reply to: Lukas Finnveden’s comment on: Con­sider how your global gov­er­nance pro­posal is differ­ent from the EU Code of Practice

  I think there is some value. I think it’s great that the Code and the AI Office exists. Similar thing with more buy-in from the US and China could have some additional value. But mounting a really big response to things quickly is not an easy business. I would be more positive if people’s ambitious governance proposals included a line saying that “our proposal is different from the Code because the the regulatory body/​bodies in question will have a call with the US President and the Chinese Premier every Monday”. AI 2040 has that, which makes it more credible that their proposal makes a situation an order of magnitude better compared to what we already have. Still, even then, I’m worried that things will move too fast by default, and there will be too much distraction and noise that will drown out the most important signals. I explained that worry in response to AI 2040 in this comment. https://​​www.lesswrong.com/​​posts/​​pFzctpJBat95SrCyC/​​ai-2040-plan-a?commentId=j4QbYtfDYLcNnTroN

• David Matolcsi 14 Sep 2026 12:35 UTC

  5 points

  1

  in reply to: solhando’s comment on: Con­sider how your global gov­er­nance pro­posal is differ­ent from the EU Code of Practice

  I don’t think I should go into details (and I’m not a lawyer), but I’m not sure people should put too much weight into solhando’s interpretation of the AI Act wrt things like the Hugging Face event.

  And yes, having a global body with more explicit support from the governments of US and China would be great, though my guess is that it would be less than an order of magnitude more powerful, because of the reasons I will describe below.

  it can’t be top-down imposed by an unelected group of people

  The AI Act was passed by the European Parliament, which is a directly elected legislative body. The AI Act contains Article 55, requiring providers of the most advanced models to conduct state-of-the-art model evaluations, assess and mitigate systemic risks, keep track of and report information about serious incidents, and ensure adequate cybersecurity of their models and their physical instrastucture.

  The AI Act also foresees that a Code of Practice will be written (Art 56). The authors of the Code are indeed not directly elected, but have been appointed by the Commission, which is accountable to the Parliament.

  As Kaj Sotala also says, I think this is a very normal and legitimate way for a technical regulation to work.

  But yes, I agree that if you want to take much more drastic actions (e.g. if it turns out that at some level of capabilities everyone needs to pause for many years to get actually solid safety cases), then you probably need more legitimacy and more direct buy-in from world leaders. But this is my whole point. The global standards body Dario and Demis envision is also explicitly framed as a technical body, looking at evals and safety arguments and so on. The rulebooks and the enforcement of that global standards body won’t be any more of a “direct democracy” than what the EU Commission is doing—technical regulations just don’t work like that anywhere. If you want more drastic actions with direct and full-on buy-in from world leaders, you have a better chance with a pause than with a technical regulation.

• David Matolcsi 13 Sep 2026 16:42 UTC

  12 points

  12

  in reply to: Jiro’s comment on: Con­sider how your global gov­er­nance pro­posal is differ­ent from the EU Code of Practice

  It’s very normal that if companies want to sell their products in a country, they need to comply with some regulations on their products.

Con­sider how your global gov­er­nance pro­posal is differ­ent from the EU Code of Practice

• David Matolcsi 13 Sep 2026 7:56 UTC

  2 points

  −2

  in reply to: habryka’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  Sorry, fixed.
  
  I think that pointing to one nebulous group as an outgroup is equally bad for increasing tribalism as pointing to two nebulous groups. (Bowing out of the discussion now.)

• David Matolcsi 12 Sep 2026 17:28 UTC

  2 points

  0

  in reply to: Ben Pace’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  Sorry, I agree that Zach’s comment in this context was not particularly productive. I was responding to the “I’m frustrated with this being the second time in recent history where you posted/​reposted a thing that ends up reinforcing tribal boundaries” comment that I felt was trying to establish a general norm which is unevenly enforced.

• David Matolcsi 12 Sep 2026 8:13 UTC

  6 points

  0

  in reply to: habryka’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  Eliezer: 1 2 3 4

  Richard Ngo: (note that these are not necessary low-quality comments, but are still carving out tribes): 1 2

  Ben Pace responding to Wei Dai’s question: here

  Also, I thing John’s comment is pretty tribal and doesn’t include himself in the criticism. If a rare non-liberal in SF said “I hate that liberals and San Franciscans can’t do the Obviously Correct thing and cleanse their streets from crime”, I wouldn’t think that of this as a non-tribal self-critical comment because he is a San Franciscan criticizing San Franciscans.

• David Matolcsi 11 Sep 2026 10:48 UTC

  15 points

  11

  in reply to: habryka’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  I agree that talking about tribes is generally bad, and it would be good if we could go back to doing less of it.
  
  But I think that in the recent past, John Wentworth, Richard Ngo, Eliezer and others have started to ignore this norm, railing against the moral failings of a nebulous outgroup, “the EAs”, and getting wildly upvoted for that with minimal pushback even when their comments are pretty low quality imo.
  
  (My least favorite example is this comment from John, calling a nebulous outgroup immoral and underperforming a five year old, while not providing any specific examples of people lying, or how to navigate the tradeoff between The Obvious Right Thing To Do “spread true important things, don’t strategically hide your views”, and between the view that it’s e.g. bad for Thomas Kwa to go to OpenAI to investigate RSI and publish about it. I think the entire comment is just a naked status attack on the other tribe, with very little justification provided, but it sill got highly upvoted.)
  
  Maybe there is a good reason for John, Richard, Eliezer, etc to start loudly blaming the nebulous other tribe: sometimes coalitions break down and it’s good to name things as they are.
  
  But I think at that point it’s fair for people who feel attacked by these comments to start also acknowledging that tribes are being formed, and talk explicitly about the behavior and track record of different tribes.
  
  In my opinion, it would be good to deescalate, and make fewer comparisons between nebulous groups. But I think it’s not going to work if you only push back when one nebulous group does it, and not when the other.

• David Matolcsi 10 Sep 2026 19:53 UTC

  22 points

  8

  in reply to: Richard_Ngo’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  [Paul is] the closest thing there is to a leader of what I’ll call the “pragmatic AI safety” cluster

  Is that true? I haven’t been in Constellation since a long time, but my impression was that Paul largely stopped commenting on LessWrong and Constellation Slack around the time he joined the US Government. Is he still a particularly influential voice in AI safety? I thought he traded away his voice for USG involvement years ago, and at that point he might as well get involved with OpenAI too. In general, it doesn’t seem that bad policy to me that some thoughtful people should try to specialize in joining powerful institutions and talk sense to them at the cost of their public voice, while others should remain fully independent and try to become honest and unbiased public thought-leaders.

• David Matolcsi 10 Sep 2026 7:09 UTC

  2 points

  0

  in reply to: bpomo’s comment on: bpomo’s Shortform

  Sorry, you are right, I misremembered the claim in Alex’s shortform. I’m editing my comment now.

• David Matolcsi 9 Sep 2026 21:01 UTC

  3 points

  0

  in reply to: Cleo Nardo’s comment on: bpomo’s Shortform

  I agree that Redwood has been historically very good at explaining why they are doing what they are doing. However, I do think that the posts making the case for AI control in particular are getting a bit old, and it would be very good to see updates on them in light of everything that happened in the last two years (e.g. Buck’s recent claim that it’s quite possible that it would have been net negative to implement AI control in the past, because it would have prevented the HF incident).

• David Matolcsi 8 Sep 2026 20:38 UTC

  20 points

  17

  in reply to: roha’s comment on: anaguma’s Shortform

  If they intentionally looked up his work and used in their solution, that would be an absolutely outrageous thing to do. Not just usual recklessness, but actual, clearly malicious actions. A bunch of employees would know. The gain is not that big compared to the outrage if it comes out that they did that. These kind of conspiracies very rarely happen.

• David Matolcsi 8 Sep 2026 20:19 UTC

  5 points

  4

  in reply to: roha’s comment on: anaguma’s Shortform

  Yeah, but I bet it’s not true.

• David Matolcsi 8 Sep 2026 19:38 UTC

  35 points

  5

  in reply to: DW11’s comment on: anaguma’s Shortform

  They didn’t try to remove Alpöge as a coauthor from the Euler result he actually helped solve. They just didn’t want to include him in their generous offer to be a coauthor on the NS result which they didn’t manage to solve before OpenAI got there first.

• David Matolcsi 8 Sep 2026 19:34 UTC

  23 points

  10

  in reply to: anaguma’s comment on: anaguma’s Shortform

  I’m probably missing sonething, but what does Buckmaster mean by “a sequence of announcements say something I know to be false”?

  I’m confused what OpenAI did wrong here. Option 1 sounds very reasonable: Buckmaster and Alpöge publish their actual results (the Euler result) under their own name (no mention of removing Alpöge here), and then OpenAI publishes the NS result. And Option 2 is more generous than that, as they allow Buckmaster to be a coauthor on the NS paper, because the rumor of him getting close is what motivated them to start working. Iiuc, under Option 2, Alpöge would still be an author on the Euler result which they actually produced, he just wouldn’t be included in the generosity of being on the NS paper.

  And I think that probably it doesn’t matter if some of Buckmaster’s older conversations got into the training data, and OpenAI knows that, but they can’t explicitly say that no old conversation made into the training data if Buckmaster didn’t opt out. I somewhat strongly suspect that Buckmaster is paranoid here.

• David Matolcsi 6 Sep 2026 10:18 UTC

  10 points

  1

  in reply to: habryka’s comment on: Alex Mallen’s Shortform

  In past discussions I had with Buck, he seemed to think most of the value would come from just executing control schemes within a lab without any ability to share evidence with the outside world.

  I would be curious if Buck still believes this. In the original Ten people on the inside post, “Build concrete evidence of risk, to increase political will towards reducing misalignment risk” is listed as the first point of what they should do, and so far I understood the primary value of the ten people on the inside to be noticing if things go off the rails and informing the outside world. If that’s not the main strategy, that would be good to know.
  
  In general, it seems that safety people inside and outside the companies are putting more and more effort into monitoring and control, and I would like to understand the current theory of change behind that, as I write in this shortform. Is the mainline plan to discover and disclose more concerning incidents, so the politicians take action? If so, how do we ensure that the politicians will react more to the future incidents than they are reacting to the current ones? Or is the mainline plan to create a technical solution to alignment through Few-shot catastrophe prevention? If so, can people write up more detailed explanation for why they expect that to work? And does it look like OpenAI and Anthropic are now implementing a good version of Few-shot catastrophe prevention from the current incidents? (Not a rhetorical question, I know very little about the few-shot plan and I don’t know how it compares to the reality.)

  Given the amount of effort going into monitoring and control, I think it would be very important to have a post setting out the theory of change in light of the updates we had since the original posts on AI control came out.

• David Matolcsi 5 Sep 2026 22:14 UTC

  6 points

  0

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  I agree it would be good if they gave access to reasoning=None, but I don’t understand them as implying that it’s for safety reasons that they don’t make the reasoning=None version available. That would indeed be complete nonsense, but I don’t think they are saying that.