Alexander Gietelink Oldenziel

Karma: 7,889

″ (...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties.”
- Saharon Shelah
”A little learning is a dangerous thing ;
Drink deep, or taste not the Pierian spring” - Alexander Pope

As a true-born Dutchman I endorse Crocker’s rules.

For my most of my writing see my short-forms (new shortform, old shortform)

Twitter: @FellowHominid

Personal website: https://sites.google.com/view/afdago/home

Alexander Gietelink Oldenziel 11 Oct 2026 13:05 UTC
4 points
1
in reply to: avturchin’s comment on: Alexander Gietelink Oldenziel’s Shortform
I have now proposed AI legislation that can be implemented immediately. I don’t think we need to be so pessimistic.

Alexander Gietelink Oldenziel 11 Oct 2026 13:04 UTC
5 points
3
in reply to: Noosphere89’s comment on: Alexander Gietelink Oldenziel’s Shortform
I don’t know how you classify clickbait but I can tell you directly these are my actual beliefs—not clickbait.

We have evidence for and against RSI happening soon. It is my current assesment that the evidence for is strong enough to take it very seriously and assign significant probability mass to it happening very soon. I don’t understand why you think that’s clickbait.
I’m curious to hear your evidence against RSI being imminent. I have found the evidence against RSI soonish (let’s say less than 18 months) I’ve seen not very convincing. I hope you don’t mean bogus arguments like generic “diminishing returns on intelligence”.

Of course, things could slow down- we might hit unforeseen limits but so far that hasn’t happened- very much the opposite. Progress has been faster than even internal OpenAI timelines. Apparently 2 weeks for the Navier-Stokes result that had internal bets that put the solving a Millenium Prize problems years into the future.

Alexander Gietelink Oldenziel 11 Oct 2026 11:42 UTC
35 points
11
on: Alexander Gietelink Oldenziel’s Shortform
What are AI governance people doing right now?
Pardon my French, but… what is are you doing right now? An extreme form of RSI is likely very close and there is a need for immediate action.

My impression as an outsider is that there are dozens of separate and uncoordinated attempts to influence and design legislation from AI Governance groups.
Many AI governance people themselves have not internalized the gravity and the urgency of the situation.
There is an apparent lack of urgency.

There is a lot of focus on low priority governance asks right now.
Some things that are on the table:
1. Embedded evaluators are great -but we already have enough information to act. More reports are unlikely to move the needle much.
2. An intergovernmental entity legislating and mandating standards for AI. Again good longer-term but too slow and leaves the actual details of how to legislate vague.
3. Compute controls etc. Compute is not the limit right now for RSI.
4. Legislation about publicly-available models. RSI happens inside labs.

We already have way more than enough information to act and immediately pause frontier training runs.

We need a Call-to-Action Yes Ladder to stop RSI

What is a Yes Ladder? A Yes Ladder is a concept from sales. The idea is that you have a list of escalating asks—each rung is easy to say yes to.

Here is a simple Yes Ladder. I haven’t thought about it very deeply but roughly speaking we need something like the following:

1. Emergency Congress session after the mid-terms and Congressional Hearings on RSI
2. A Congressional taskforce to investigate all current AI incidents (HuggingFace etc)
3. An pause on frontier lab training runs until the Congressional taskforce has completed its investigation.

Note that these are very clear asks. They are not vague or leave details to fill in. They are unambigious and clear.
Respect Trump
Governmental action will not come from the Executive branch. It may come from Legislative and the Judicial branch. In the American system the President can veto acts of Congress. It is important to present the proposal as non-partisan. It is important not to anger Trump and the Republicans. The mid-terms 2 November will likely sweep in the Democrats to Congress but these considerations still apply.

We need to speak with one voice
The world is a very big place. There are millions and billions of people in it.
Communicating with many people is not the same game as communicating with a few people. The primary takeaway is that for effective communication to a large number of people you need to repeat a clean, easily-understandable short Message and communicate Only that Message [ie message discipline].

All AI governance groups should agree to a variant of the above Call-to-Action Yess Ladder and pivot to pushing This and Only this Message.

Coefficient Giving could use its influence
Coefficient Giving funds dozens of AI governance groups and should tell them to sign on to this call.
There is a legal problem: Coefficient Giving could be seen to influence legislation and risk being investigated for violating something something 503c.
To get around this a small group of AI governance people should go around and collect with the understanding that this is the implied will of their funders.

There is a channel with 900m users. What goes in it?
Anthropic and OpenAI have a huge opportunity to influence people by pushing this call to action to everybody that is using their product.

What can I do as an AI governance organisation? What can I do as an individual?

1. Coordinate with other organisations & Focus on repeating and only repeating the above message/ Call to Action
2. Call congressional offices and brief lawmakers. Read this account on how to do that.
If you are a US citizen calling your congressman is probably a good idea.
If you are a rank-and-file AI safety person with little direct influence one of the main things you can do is to call for AI safety governance groups to pivot to focus to do above.
What links here?
- Alexander Gietelink Oldenziel's comment on Alexander Gietelink Oldenziel’s Shortform by Alexander Gietelink Oldenziel (11 Oct 2026 13:05 UTC; 4 points)

Alexander Gietelink Oldenziel 11 Oct 2026 10:37 UTC
9 points
0
in reply to: Mateusz Bagiński’s comment on: Alexander Gietelink Oldenziel’s Shortform
Thanks for asking Mateusz. No. That isn’t quite right.

I am bullish on legislation and always have been. In fact, I have been much more bullish on legislation to slow down, pause AGI than the median EAer. I don’t have time right now to dig through my LW/twitter comments but suffice to say I think a determined search will find plenty of evidence.

I believe in “normie wisdom” and have long been much more bullish on conventional governance as opposed to the ‘lab self-governance’ that has been a staple of (some) EA thought.

Believing in the ability of humanity to slowdown or temporarily pause doesn’t mean believing in Kumbaya. It will remain extremely difficult (though not impossible) to indefinitely pause or ban AI. For this particular comment by TsviBT he is advocating for strategies that are extremely galaxy-brained and will only pay off on extremely long timescales [eg 20+ years]. I think that’s a mistake for the obvious reason that probably we don’t have more than 3 years, maybe less than 12 months. Even with determined political action we might only be able to effectively buy a few years.

We need to focus on solving AI alignment directly, not galaxy-brained superbabies. I believe this is possible, albeit extremely hard—but AI assistance is making scientific breakthroughs that otherwise would take 20 years possible in 3 months (or 3 hours...!). That is what Iliad will do.

EDIT: rereading TsviBT’s comment I think the main reason I replied negatively is because I thought the evidence he was marshalling for possibility of collective action for a pause/ban/slowdown was weak: the technologies he mentioned are importantly different from present-day AI.

Alexander Gietelink Oldenziel 11 Oct 2026 9:09 UTC
71 points
41
on: Alexander Gietelink Oldenziel’s Shortform
The Democrats have a 75% chance of taking control of Congress according to Nate Silver. The mid-term election is November 2. Democrats seem overall more open to AI regulation. We need emergency legislation putting an immediate and full stop on new training runs for frontier labs and a ban on RSI. This is by far the most important legislation to happen. All other proposed legislation is probably a sideshow.
Is anybody in AI governance working on proposing draft legislation here ? Who is working on this?
Anthropic, OpenAI and likely DeepMind as well are entering the early stages of RSI and this could lead to an intelligence explosion by end of 2027 according to their own projections. That may be an underestimate. Progress has gone much faster than internal forecasts. It is not out of the question FOOM happens by end of 2026.

Alexander Gietelink Oldenziel 10 Oct 2026 12:31 UTC
52 points
27
on: Stop telling people to pivot to AI safety, I beg you
Fwiw I run a fieldbuilding and research ai safety organization and I agree with this take. We hire aggressively, we don’t do half a dozen performative hiring rounds. Our placement data indicates the majority in our programs end up in some kind of longer-term AI safety job. Nevertheless, I agree with the OP that there should be more honest messaging.
It’s good that there are so many entry opportunities but it would be helpful if there was more clear-eyed messaging about long-term job opportunities. My sense is that compared to eg academia the job market is much better in AI safety but nevertheless many aspiring AI safety researchers/generalists may struggle to find a job for a long time.
I also strongly dislike the weird 2-faced messaging which is at the same time naively idealistic (‘everyone should switch!‘) and elitist (ooooh it’s so hard to find talent). In my not so humble opinion, there is a very self-congratulatory sense of elitism in AI safety insider circles: cf all the talk about ‘high context’. The fact is that AI safety is not a galaxy-brained concept. One can get up to speed with the basics relatively quickly. It is simply not that deep compared to many other fields. That doesn’t mean there can’t be large differences between researchers—there definitely are—but that in many cases this kind of talk comes from often mildly mediocre people who were just early / know the right people / parrot the accepted ideology. In many cases it is more a question of who you know than what you know.
eg as a random guestimate I would guestimate that being up to date with say mid 20th century electrical engineering is intellectually more challenging than being up to date with the sota of interpretability research.

Alexander Gietelink Oldenziel 8 Oct 2026 11:54 UTC
5 points
2
on: Are OpenAI’s math results “creative” in an important way?
Yes.
Overwhelmingly so.
This has been apparent far before the recent spate of results. LLMs were already capable of minor mathematical results in late 2024-early 2025 with some minor handholding, where minor is minor in the grand scheme of things but would be pretty impressive for most math grad students.
I’m surprised to see so much questioning about this on this forum [as opposed to the wider world]. There is no secret sauce to being creative. LLMs do it and do it in spades.
If you are still seriously questioning whether LLM can do ‘truly creative work’ you haven’t been updating.

Alexander Gietelink Oldenziel 7 Oct 2026 16:49 UTC
5 points
0
in reply to: nielsrolf’s comment on: “Alignment Engineering” vs. “Misalignment Science”
No no no. This is confused. This is exactly the issue with people rebranding alignment to mean amything and everything and especially making AIs say nice words instead of bad words. This kind of value shift of words has done immense damage. find it funny yet sad that it is also accompanied by some smug “oh those rubes at MIRI thought there is a solution to the alignment problem while me a sophisticated thinker knows it s an amiterative process”
The Alignment Problem, like the actual hard problem is about solving alignment forever. It strictly encompasses notkilleveryonism. It was never about aligning a dog or gpt-4. What does aligning gpt-4 even mean ? Gpt-4 isnt dangerous. It is not meaningfully an agentic entity. Alignment isnt some vague scalar benchmark.
The actual alignment problem means finding a protocol that must be continually followed such that the mind you create is aligned to your values as well as any other minds the might build or self-modify into and you can prove or give very strong soft guarantees that this happens. This neccessitates understanding minds and the engineering of minds and values and engineering thereof very well.

Alexander Gietelink Oldenziel 4 Oct 2026 12:36 UTC
2 points
0
in reply to: habryka’s comment on: leogao’s Shortform
hear, hear.
it is natural to feel a deep sense of loss as AI obsoletes the human mind, especially if it is your human mind.
different people cope with this grief in different ways.
Emphatically, I can understand that math community is reacting in such an undignified manner but that doesn’t mean it is the most embarassing form of cope that will not age well.
besides—there is so much to do and so little time. Why waste it on denial?

Alexander Gietelink Oldenziel 2 Oct 2026 22:37 UTC
2 points
0
in reply to: Ben Pace’s comment on: Alexander Gietelink Oldenziel’s Shortform
Oh wow! That’s great. I was considering regularly posting on LessWrong because of the llm policies.

Alexander Gietelink Oldenziel 2 Oct 2026 20:06 UTC
2 points
0
in reply to: Ben Pace’s comment on: Alexander Gietelink Oldenziel’s Shortform
I’m confused. I just wasted 2 hours the other day editing a post because LrssWrong automatically blocks llmese. Is there an option to put it all in a LLM content block and spare me from AI Luddism ?

Alexander Gietelink Oldenziel 1 Oct 2026 22:29 UTC
4 points
0
in reply to: roha’s comment on: Alexander Gietelink Oldenziel’s Shortform
I find this very uncompelling: decision theory gives only very very weak constraints on what actions are best. Most of the degrees of freedom lie in the choice of utility function or the choice of prior [for a simple expected utility maximizer model. If you reject this model you get more not less degrees of freedom btw so that makes the case for decision theory generically weaker]

Alexander Gietelink Oldenziel 1 Oct 2026 22:27 UTC
5 points
0
in reply to: roha’s comment on: Alexander Gietelink Oldenziel’s Shortform
>> AI systems will be in exactly those situations far more often than humans are. They are copied, inspected, simulated, and tested by people trying to predict them.

I think this one is somewhat compelling.

Simplex vs Timaeus: Round One

Alexander Gietelink Oldenziel1 Oct 2026 14:30 UTC

54 points

1 comment3 min readLW link

How many AI agents are running unattended right now?

Alexander Gietelink Oldenziel1 Oct 2026 13:45 UTC

95 points

0 comments4 min readLW link

Alexander Gietelink Oldenziel 1 Oct 2026 13:25 UTC
51 points
1
on: Alexander Gietelink Oldenziel’s Shortform
Why is (Updateless) Decision Theory so popular around here?

The LessWrong sphere has long had a fascination with decision theory. Despite hanging out around there parts for many years I still don’t understand Decision theory. I must also admit that despite being a big theory-stan I don’t know if I could make a convincing case that decision theory is important (for AI alignment). Nor do I really understand why updateless decision theory (or its various cousins like logical, functional, udt 1.0, 2.0 etc I must admit I don’t understand the difference between these ) in particular is so relevant.

I asked Yudkowsky directly and after some prodding he agreed there were three motivations:
1. Basic Science of Agent Foundations
2. AIs will collude against you even when you set them up against each other. Incentivizing different AIs to compete against each other won’t work. Many people when first hearing about the alignment problem come up with various safety schemes that comes down to incentizing different AIs to compete with the AI controlling the mechanism. This is problematic: it is more natural for very powerful agents to band together against the weaker agents. The usual solution that is proposed is to come up with a clever reward scheme that directly incentives AIs to compete, eg by ratting each other out to the humans. This is still problematic because very capable minds can coordinate even in prisoner’s dilemma [Open-Source Game theory]
3. Reflective stability—alignment is not just about aligning the current AI. The current AI will build a new AI or self-modify upon reflection. In other words, you need to find a good reflectively stable fixed point. Decision theories that are not Yudkowsky-approved are not reflectively stable.
There is a fourth reason I’ve seen 4. Anthropics - (updateless) decision theory shows up when you start having Big thoughts about the universe and our place within it. Big If True.

I like these reasons. I also can’t shake the feeling that a big part of the reason decision theory is so popular is that it is fun to think about, it’s been historically been an interest of Yudkowksy and the early MIRI- affiliated people [eg Paul Christiano] and its interest is at least partially downstream of these contigent factors.

I’m interested in hearing more perspectives. Why do you think decision theory is important?

Alexander Gietelink Oldenziel 1 Oct 2026 13:25 UTC
7 points
1
on: Alexander Gietelink Oldenziel’s Shortform
Why is (Updateless) Decision Theory so popular around here?

The LessWrong sphere has long had a fascination with decision theory. Despite hanging out around there parts for many years I still don’t understand Decision theory. I must also admit that despite being a big theory-stan I don’t know if I could make a convincing case that decision theory is important (for AI alignment). Nor do I really understand why updateless decision theory (or its various cousins like logical, functional, udt 1.0, 2.0 etc I must admit I don’t understand the difference between these ) in particular is so relevant.

I asked Yudkowsky directly and after some prodding he agreed there were three motivations:
1. Basic Science of Agent Foundations
2. AIs will collude against you even when you set them up against each other. Incentivizing different AIs to compete against each other won’t work. Many people when first hearing about the alignment problem come up with various safety schemes that comes down to incentizing different AIs to compete with the AI controlling the mechanism. This is problematic: it is more natural for very powerful agents to band together against the weaker agents. The usual solution that is proposed is to come up with a clever reward scheme that directly incentives AIs to compete, eg by ratting each other out to the humans. This is still problematic because very capable minds can coordinate even in prisoner’s dilemma [Open-Source Game theory]
3. Reflective stability—alignment is not just about aligning the current AI. The current AI will build a new AI or self-modify upon reflection. In other words, you need to find a good reflectively stable fixed point. Decision theories that are not Yudkowsky-approved are not reflectively stable.
There is a fourth reason I’ve seen 4. Anthropics - (updateless) decision theory shows up when you start having Big thoughts about the universe and our place within it. Big If True.

I like these reasons. I also can’t shake the feeling that a big part of the reason decision theory is so popular is that it is fun to think about, it’s been historically been an interest of Yudkowksy and the early MIRI- affiliated people [eg Paul Christiano] and its interest is at least partially downstream of these contigent factors.

I’m interested in hearing more perspectives. Why do you think decision theory is important?

Superintelligence this Christmas

Alexander Gietelink Oldenziel18 Sep 2026 0:06 UTC

42 points

21 comments3 min readLW link

Alexander Gietelink Oldenziel 14 Sep 2026 15:15 UTC
8 points
0
in reply to: Philipp Risius’s comment on: Watch AI materials-science & bioscience abilities closely
Sir I believe you may not be familiar with my friend Claude 7 the Superintelligence

Alexander Gietelink Oldenziel 14 Sep 2026 10:27 UTC
25 points
6
on: There is a channel to 900M weekly users. What goes in it?
This is a fantastic idea!

Alexander Gietelink Oldenziel

Sim­plex vs Ti­maeus: Round One

How many AI agents are run­ning unat­tended right now?

Su­per­in­tel­li­gence this Christmas

Simplex vs Timaeus: Round One

How many AI agents are running unattended right now?

Superintelligence this Christmas