Ebenezer Dukakis

Karma: 718

Ebenezer Dukakis 1 Dec 2025 2:11 UTC
4 points
0
on: Alignment remains a hard, unsolved problem
Pournelle’s Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people:
- First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration.
- Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc.
The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.
Source.

I’m concerned that this “law” may apply to Anthropic. People devoted to Anthropic as an organization will have more power than people devoted to the goal of creating aligned AI.

I would encourage people at Anthropic to leave a line of retreat and consider the “least convenient possible world” where alignment is too hard. What’s the contingency plan for Anthropic in that scenario?

Next, devise a collective decision-making procedure for activating that contingency plan. For example, maybe the contingency plan should be activated if X% of the technical staff votes to activate it. Perhaps after having a week of discussion first? What would be the trigger to spend a week doing discussion? You can answer these questions and come up with a formal procedure.

If you had both a contingency plan and a formal means to activate it, I would feel a lot better about Anthropic as an organization.

Ebenezer Dukakis 26 Nov 2025 8:22 UTC
13 points
0
in reply to: Seth Herd’s comment on: Gemini 3 Pro Is a Vast Intelligence With No Spine
I wonder if Google is optimizing harder for benchmarks, to try and prop up its stock price against possible deflation of an AI bubble.
It occurs to me that an AI alignment organization should create comprehensive private alignment benchmarks and start releasing the scores. They would have to be constructed in a non-traditional way so they’re less vulnerable to standard goodharting. If these benchmarks become popular with AI users and AI investors, they could be a powerful way to steer AI development in a more responsible direction. By keeping them private, you could make it harder for AI companies to optimize against the benchmarks, and nudge them towards actually solving deeper alignment issues. It would also be a powerful illustration of the point that advanced AI will need to solve unforeseen/out-of-distribution alignment challenges. @Eliezer Yudkowsky

Ebenezer Dukakis 10 Nov 2025 4:25 UTC
2 points
0
on: Problems I’ve Tried to Legibilize
Thanks for making this list!

Having written all this down in one place, it’s hard not to feel some hopelessness that all of these problems can be made legible to the relevant people, even with a maximum plausible effort.

I think that a major focus should be on prioritizing these problems based on how plausible a story you can tell for a catastrophic outcome if the problem remains unsolved, conditional on an AI that is corrigible and aligned in the ordinary sense.

I suppose coming up with such a clear catastrophe story for a problem is more or less the same thing as legibilizing it, which reinforces my point from the previous thread that a priori, it seems likely to me that illegible problems won’t tend to be as important to solve.

The longer a problem has been floating around without anyone generating a clear catastrophe story for it, the greater probability we should assign that it’s a “terminally illegible” problem which just won’t cause a catastrophe if it’s unsolved.

Maybe it would be good to track how much time has been spent attempting to come up with a clear catastrophe story for each problem, so people can get a sense of when diminishing research returns are reached for a given problem? Perhaps researchers who make attempts should leave a comment in this thread indicating how much time they spent trying to generate catastrophe stories for each problem?

Perhaps it’s worth concluding on a point from a discussion between @WillPetillo and myself under the previous post, that a potentially more impactful approach (compared to trying to make illegible problems more legible), is to make key decisionmakers realize that important safety problems illegible to themselves (and even to their advisors) probably exist, therefore it’s very risky to make highly consequential decisions (such as about AI development or deployment) based only on the status of legible safety problems.

I still think the best way to do this is to identify at least one problem which initially seemed esoteric and illegible, and eventually acquired a clear and compelling catastrophe story. Right now this discussion all seems rather hypothetical. From my perspective, the problems on your list seem to fall into two rough categories: legible problems which seem compelling, and super-esoteric problems like “Beyond Astronomical Waste” which don’t need to be solved prior to creation of an aligned AI. Off the top of my head I haven’t noticed a lot of problems moving from one category to the other by my lights? So just speaking for myself, this list hasn’t personally convinced me that esoteric and illegible problems should receive much more scarce resources, although I admit I only took a quick skim.

Ebenezer Dukakis 8 Nov 2025 3:44 UTC
1 point
0
in reply to: TristanTrim’s comment on: Legible vs. Illegible AI Safety Problems

I definitely like the directions you are exploring in and I agree they are improvements over the implicit AGI lab directed concept. That’s a useful thing to keep in mind, but so is what keeps them from being final ideas.

+1

What do you think? Does that make sense at all, or maybe it seems more like a time wasting distraction? I have to admit I’m uncomfortable with the amount I have gotten stuck on the idea that championing this concept is a useful thing for me to be doing.

Glad you’re self-aware about this. I would focus less on championing the concept, and more on treating it as a hypothesis about a research approach which may or may not deliver benefits. I wouldn’t evangelize until you’ve got serious benefits to show, and show those benefits first (with the concept that delivered those benefits as more of a footnote).

Ebenezer Dukakis 7 Nov 2025 5:22 UTC
3 points
0
in reply to: TristanTrim’s comment on: Legible vs. Illegible AI Safety Problems

Your rot13 suggestion has the obvious corruption problem, but also has the problem of public relations for the plan. I doubt it would be popular. However, I like where your head is at.

My suggestion is not supposed to be the final idea. It’s just supposed to be an improvement over what appears to be Wei Dai’s implicit idea, of having philosophers who have some connection to AGI labs solve these philosophical issues, and hardcode solutions in so they can’t be changed.

(Perhaps you could argue that Wei Dai’s implicit idea is better, because there’s only a chance that these philosophers will be listened to, and even then it will be in the distant future. Maybe those conditions keep philosophers honest. But we could replicate those conditions in my scenario as well: Randomly generate 20 different groups of philosophers, then later randomly choose 1 group to act on their conclusions, and only act on their conclusions after a 30-year delay.)

Basically, whatever system we use for deciding on and controlling the corrigible AI becomes the system we are concerned with ensuring the alignment of. It doesn’t really solve the problem, it just backs it up one matryoshka doll around the AI.

I’m not convinced they are the same problem, but I suppose it can’t hurt to check if ideas for the alignment problem might also work for the “morality is scary” problem.

Ebenezer Dukakis 7 Nov 2025 4:34 UTC
1 point
0
in reply to: Jakub Growiec’s comment on: Publishing academic papers on transformative AI is a nightmare
Maybe you could collaborate with Chad Jones or some other heavyweight?

Ebenezer Dukakis 7 Nov 2025 4:33 UTC
3 points
1
in reply to: MattJ’s comment on: Publishing academic papers on transformative AI is a nightmare
Based on the OP it seems there are some issues writing papers about transformative AI in general. I would say it’s best for the journal to feature a variety of perspectives on this topic instead of privileging a particular viewpoint. “The Journal of Existential Risk of AI” risks being perceived as too niche, in my opinion. But “The Journal of Transformative AI” seems likely to acquire interest by default, if AI continues to progress rapidly.

Ebenezer Dukakis 5 Nov 2025 9:37 UTC
3 points
0
in reply to: Wei Dai’s comment on: Legible vs. Illegible AI Safety Problems
Again, thanks for the reply.

Building a corrigible AGI has a lot of advantages. But one disadvantage is the “morality is scary” problem you mention in the linked comment. If there is a way to correct the AGI, who gets to decide when and how to correct it? Even if we get the right answers to all of the philosophical questions you’re talking about, and successfully program them into the AGI, the philosophical “unwashed masses” you fear could exert tremendous public pressure to use the corrigibility functionality and change those right answers into wrong ones.

Since corrigibility is so advantageous (including its ability to let us put off all of your tricky philosophical problems), it seems to me that we should think about the “morality is scary” problem so we can address what appears to be corrigibility’s only major downside. I suspect the “morality is scary” problem is more tractable than you assume. Here is one idea (I did a rot13 so people can think independently before reading my idea): Oevat rirelbar va gur jbeyq hc gb n uvtuyl qrirybcrq fgnaqneq bs yvivat. Qrirybc n grfg juvpu zrnfherf cuvybfbcuvpny pbzcrgrapr. Inyvqngr gur grfg ol rafhevat gung vg pbeerpgyl enax-beqref cuvybfbcuref ol pbzcrgrapr nppbeqvat gb 3eq-cnegl nffrffzragf. Pbaqhpg n tybony gnyrag frnepu sbe cuvybfbcuvpny gnyrag. Pbafgehpg na vibel gbjre sbe gur jvaaref bs gur gnyrag frnepu gb fghql cuvybfbcul naq cbaqre cuvybfbcuvpny dhrfgvbaf juvyr vfbyngrq sebz choyvp cerffher.

Ebenezer Dukakis 5 Nov 2025 8:50 UTC
1 point
0
in reply to: Wei Dai’s comment on: Legible vs. Illegible AI Safety Problems
Thanks for the reply. I’m not a philosopher, but it seems to me that most of these problems could be addressed after an AGI is built, if the AGI is corrigible. Which problems can you make the strongest case for as problems which we can’t put off this way?

Ebenezer Dukakis 5 Nov 2025 8:42 UTC
3 points
2
on: Publishing academic papers on transformative AI is a nightmare
Is starting a new journal devoted to this topic a feasible option?

Ebenezer Dukakis 5 Nov 2025 8:21 UTC
19 points
4
on: Legible vs. Illegible AI Safety Problems
The underlying assumption here (“the halt assumption”) seems to be that big-shot decisionmakers will want to halt AI development if it’s clear that unsolved alignment problems remain.

I’m a little skeptical of the halt assumption. Right now it seems that unsolved alignment problems remain, yet I don’t see big-shots moving to halt AI development. About a week after Grok’s MechaHitler incident, the Pentagon announced a $200 million contract with xAI.

Nonetheless, in a world where the halt assumption is true, the highest-impact action might be a meta approach of “making the notion of illegible problems more legible”. In a world where the halt assumption becomes true (e.g. because the threshold for concern changes), if the existence and importance of illegible problems has been made legible to decisionmakers, that by itself might be enough to stop further development.

So yeah, in service of increasing meta-legibility (of this issue), maybe we could get some actual concrete examples of illegible problems and reasons to think they are important? Because I’m not seeing any concrete examples in your post, or in the comments of this thread. I think I am more persuadable than a typical big-shot decisionmaker, yet my cynical side reads this post and thinks: “Navel-gazers think it’s essential to navel-gaze. News at 11.”

Another angle: Are there concrete examples of AI alignment problems which were once illegible and navel-gazey, which are now legible and obviously important? (Hopefully you won’t say the need for AI alignment itself; I’ve been on board with that for as long as I can remember.)

There is another possible world here, which is that legibility actually correlates pretty well with real-world importance, and the halt assumption is false, and your post is going to redirect scarce AI alignment talent away from urgent problems which matter, and towards fruitless navel-gazing. I’m not claiming this is the world we live in, but it would be good to gather evidence, and concrete examples could help.

I fully support people publishing lists of neglected AI alignment problems. (Why can’t I find a list like that already?) But I suspect many list entries will have been neglected for good reason.

Ebenezer Dukakis 27 Oct 2025 11:01 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform

alignment equivalents to “make a trillion dollars” for capabilities that are easy to verify, strictly imply alignment, and extremely difficult to get any traction on (and with it, a series of weakenings of such a metric that are easier to get traction on but also less-strictly imply alignment).

I expect there’s a fair amount of low-hanging fruit in finding good targets for automated alignment research. E.g. how about an LLM agent which reads 1000s of old LW posts looking for a good target? How about unlearning? How about a version of RLHF where you show an alignment researcher two AI-generated critiques of an alignment plan, and they rate which critique is better?

Ebenezer Dukakis 14 Oct 2025 1:08 UTC
22 points
0
on: The Most Common Bad Argument In These Parts
I believe in Thinking Fast and Slow, Kahneman refers to this fallacy as “What You See Is All There Is” (WYSIATI). And it used to be common for people to talk about “Unknown Unknowns” (things you don’t know, that you also don’t know you don’t know).

Ebenezer Dukakis 12 Sep 2025 5:42 UTC
13 points
2
in reply to: Remmelt’s comment on: MAGA speakers at NatCon were mostly against AI

To me, if someone posts an intense-feeling negatively worded text in response to what other people are doing, it usually signals that there is something they care about that they perceive to be threatened. I’ve found it productive to try relate with that first, before responding. Jumping to enforcing general rules stipulated somewhere in the community, and then implying that the person not following those rules is not harmonious with or does not belong to the community, can get counterproductive.

I’m a bit concerned about a situation where “insiders” always get this sort of contextual benefit-of-the-doubt, and “outsiders” don’t.

Ebenezer Dukakis 12 Sep 2025 3:46 UTC
10 points
0
in reply to: habryka’s comment on: MAGA speakers at NatCon were mostly against AI
Just to clarify here, I have no issue with you thinking the post is bad. That seems besides the point to me. My issue is with you doing much of what you accuse Miller of doing.

Insults: “This post seems completely insane to me, as do people who unquestionable retweet it.”

Aggression: “I cannot believe I have to argue for this… [cursing]...”

Sneering: “Has anyone who liked this actually read this post? How on earth is this convincing to anyone?”

Note also that the discussion around the Bentham post was previously calm and friendly. You walked in and dramatically worsened the discourse quality. By contrast, Geoffrey engages on hot-button political topics where discussion is already very heated.

As a quick and relatively objective measure, with a quick search, out of all 80K Geoffrey Miller tweets, I was only able to find one non-quoted f-bomb (“Fuck the Singularity.”).

Your tweets appear to have a somewhat larger number of them, and they’re often directed at individuals rather than abstract concepts. “Fuck them”, “fuck you”, “fuck [those people]”.

As a matter of simple intellectual honesty, it would be nice if you could acknowledge that you engage in insults and aggressive behavior on Twitter. You might be doing it less than Geoffrey does. You might express it in a different way. But it’s just a question of degree, as far as I can tell. I really don’t think you have much moral high ground here.

You also have far fewer tweets than Geoffrey does (factor of ~16 difference). So it’s not just that you’ve dropped more f-bombs than him; your density of f-bombs appears to be far higher.

Ebenezer Dukakis 9 Sep 2025 2:39 UTC
9 points
3
in reply to: Ebenezer Dukakis’s comment on: MAGA speakers at NatCon were mostly against AI
Keep in mind that US conservatives are liable to be reading this thread, trying to determine whether they want to ally with a group such as yourselves. Conservatives have much more leverage to dictate alliance terms than you do. Note the alliance with the AI art people was apparently already wrecked. Something you might ask yourselves: If you can’t make nice with a guy like me, who shares more of your ideals than either artists or US conservatives do, how do you expect to make nice with US conservatives?

Ebenezer Dukakis 9 Sep 2025 2:12 UTC
9 points
−7
in reply to: Ben Pace’s comment on: MAGA speakers at NatCon were mostly against AI

It’s not a norm of discourse that one cannot state that a position is absurd.

Speaking as someone who makes very little effort to avoid honey consumption, my opinion of Habryka would have dropped much less if he’d said something like: “Sorry, this position is just intuitively absurd to me, and I’m happy to reject it on that basis.” So I don’t think the issue has to do with absurdity per se.

I said I thought he violated “what I’d consider reasonable norms of discourse”. You can see Ben West thought something similar.

I’d estimate that Habryka violated roughly 7 or 8 of the Hacker News commenting guidelines in that discussion.

Your ideas about reasonable discourse can be different from mine, and Ben West’s, and Hacker News’. That’s OK. I was just sharing my opinion.

It’s been a while since I read that discussion. I remember my estimation of Habryka dropped dramatically when I read it. Maybe I can try to reconstruct why in more detail if you want. But contrasting what Habryka wrote with the HN commenting guidelines seems like a reasonable starting point.

And it is a virtue of discourse to show up and argue for one’s stances, as Habryka does throughout that thread!

You’ll notice that Habryka doesn’t provide any concrete example of Geoffrey violating a norm of reasonable discourse in this thread. I did provide a concrete example.

Is it possible that invokation of such “norms” can be a mere figleaf for drawing ingroup/outgroup boundaries in the traditional tribalistic way?

Is it too much to ask that Dear Leadership is held to the same standards, and treated the same way, as everyone else is?

Ebenezer Dukakis 8 Sep 2025 23:54 UTC
3 points
−15
in reply to: habryka’s comment on: MAGA speakers at NatCon were mostly against AI
I’ve seen Eliezer violate what I’d consider norms of reasonable discourse on Twitter. You too.

Ebenezer Dukakis 23 Aug 2025 9:56 UTC
7 points
2
in reply to: sunwillrise’s comment on: Banning Said Achmiz (and broader thoughts on moderation)
Before you quit, maybe we can create a wiki page of people who left, with contact information, to open the door for a refugee forum at some point in the future?

Ebenezer Dukakis 15 Aug 2025 4:19 UTC
5 points
−7
in reply to: Eliezer Yudkowsky’s comment on: The Problem

Of the clever solutions you invented and tested within the survivable regime, 2/3rds of them survive the 6 changes you didn’t see coming, 1/3rd fail. Now you’re dead.

It seems unreasonable to conclude we’re now dead, if 2/3rds of our solutions survived the 6 changes we didn’t see coming.

The success of a single solution should ideally be more of a sufficient condition for success, rather than a necessary condition. (Note this is plausible depending on the nature of the “solutions”. Consider a simple “monitors for bad thoughts” model. If even a single monitor flags bad thoughts, we can instantly pull the plug and evaluate. A malicious AI has to bypass every single monitor to execute malice. If a single monitor works consistently and reliably, that ends up being a sufficient condition for overall prevention of malice.)

If you’re doing this right, your solutions should have a lot of redundancy and uncorrelated failure modes. 2/3rds of them working should ideally be plenty.

[Edit: I notice people disagreevoting this. I’m very interested to learn why you disagree, either in this comment thread or via private message.]