Sometimes people deliberately fill their environment with yes-men and drive out critics. Pointing out what they’re doing doesn’t help, because they’re doing it on purpose. However there are ways well intentioned people end up driving out critics unintentionally, and those are worth talking about.

The Rise and Fall of Mars Hill Church (podcast) is about a guy who definitely drove out critics deliberately. Mark Driscoll fired people, led his church to shun them, and rearranged the legal structure of the church to consolidate power. It worked, and his power was unchecked until the entire church collapsed. Yawn.

What’s interesting is who he hired after the purges. As described in a later episode, his later hiring was focused on people who were executives in the secular world. These people were great at executing on tasks, but unopinionated about what their task should be. Whatever Driscoll said was what they did.

This is something a good, feedback-craving leader could have done by accident. Hiring people who are good at the tasks you want them to do is a pretty natural move. But I think the speaker is correct (alas I didn’t write down his name) that this is anti-correlated at the tails- the best executors become so by not caring about what they’re executing.

So if you’re a leader and want to receive a healthy amount of pushback, it’s not enough to hire hypercompetent people and listen when they push back. You have to select specifically for ability to push back (including both willingness, and having good opinions).

A thought about the debate: I don’t usually prefer going to talks as a format for learning, and I expect a transcript to be long and meandering as you touch on cruxes that are significant to you two but not necessarily to me.

A thing I would personally value after-the-fact is a summary of “what things did either of you learn?” (and I might generally prefer this for most debate/​dialogue formats)

That all said, I think it’s cool that you’re holding events like these :)

Cool experiment! Maybe GPT-4.5 would do better?

Well that aged poorly. Coreweave’s valuation 1.5x’d within 2 weeks of me posting this comment. Mild evidence against EMH here I guess.

(Disclaimer: I do have a small long position in CRWV, entered shortly after posting the original comment on the grounds of “well I’m not positive this is priced in yet”)

(A small rant, sorry) In general, it seems you’re massively overanchored on current AI technology, to an extent that it’s stopping you from clearly reasoning about future technology.

You are right that I am addressing AGI with a lot of similarities to LLMs. This is done in the interest of reasoning clearly about future technologies. I think good reasoning is a mix of predicting the most likely forms of AGI, and reasoning more broadly. Perhaps I didn’t make clear enough in the post that I’m primarily addressing LLM-based AGI. Much of my alignment work (informed by my systems neuroscience work) is on routes from LLMs to AGI. In this theory, LLMs/​foundation models are expanded (by adding memory systems and training/​scaffolding them for better metacognition) into loosely brainlike cognitive architectures. In those posts I elaborate reasons to think such scaffolded LLMs may soon be “real AGI” in the sense of reasoning and learning about any topic, including themselves and their cognition and goals. (although that sort of AGI wouldn’t be dramatically superhuman in any area, and initially subhuman in some capabilities).

If you have an alternate theory of the likely form of first takeover-capable AGI, I’d love to hear it! It’s good to reason broadly where possible, and I think a lot of the concerns are general to any AGI at all or any network-based AGI. But constraining alignment work to address specific likely types of AGI lets us reason much more specifically, which is a lot more useful in the worlds where that type of AGI is what we really are faced with aligning.

You’re talking about AGI here. An agent capable autonomously doing research, play games with clever adversaries, detecting and patching it’s own biases, etc. It should be obvious that you can’t use current LLM flaws as a method of extrapolating the adversarial robustness of this program.

Yes, good point. I didn’t elaborate here, but I do think there’s a good chance that the more coherent, intelligent, and introspective nature of any real AGI might make jailbreaking a non-issue. But jailbreaking might still be an issue, because the core thought generator in this scenario is an advanced LLM.

No. You’re entirely ignoring inner alignment difficulties. The main difficulties. There are several degrees of freedom in goal specification that a pure goal target fails to nail down. These lead to an unpredictable reflective equilibrium.

Yes I am entirely ignoring inner alignment difficulties. I thought I’d made that clear by saying earlier

There are substantial technical challenges [including] problems with specifying goals well enough that they won’t be misgeneralized or misinterpreted (from the designer’s perspective). There are serious implementational problems for any alignment target. [...] Here I am leaving aside the more general challenges, and addressing only those that are specifically relevant to instruction-following (and in some cases corrigibility) as an alignment target.

I didn’t use the term “inner alignment” because I don’t find it intuitive or clarifying; there isn’t a clear division between inner and outer, and they feel like jargon. So I use misgeneralization, which I feel encompasses inner misalignment as well as other (IMO more urgent) concerns. Maybe I should get on board and use inner and outer alignment just to speak the lingua franca of the realm.

my summary of these two papers: https://​​arxiv.org/​​pdf/​​1805.12152 https://​​arxiv.org/​​pdf/​​1905.02175

the first paper observes a phenomenon where adversarial accuracy and normal accuracy are at odds with each other. the authors present a toy example to explain this.

the construction involves giving each input one channel that is 90% accurate for predicting the binary label, and a bazillion iid gaussian channels that are as noisy as possible individually, so that when you take the average across all of them you get ~100% accuracy. they show that when you do ${\ell }_{\infty }$-adversarial training on the input you learn to only use the 90% accurate feature, whereas normal training uses all bazillion weak channels.

the key to this construction is that they consider an ${\ell }_{\infty }$-ball on the input (distance is the max across all coordinates). so this means by adding more and more features, you can move further and further in ${\ell }_2$ space (specifically, $\sqrt{n}$ in terms of the number of features). but the ${\ell }_2$ distance between the means of the two high dimensional gaussians stays constant, so no matter what your $\epsilon$ is, at some point with enough channels you can perturb anything from one class into the other class and vice versa.

in the second paper, the authors do further experiments on real models to show that you can separate out the robust features and the unrobust ones, and recombine them into frankenstein images that look like dogs to humans but cats to the unrobust model and dogs to the robust model.

they also generalize the toy example in the previous paper. they argue that in general, adversarial examples arise exactly when the adversarial attack metric and the loss metric differ. in other words, the loss function (and downstream part of the model, in a multilayer model) implies some loss surface around any data point, and some directions on that surface will be a lot more important for the loss than some other directions. but your $\epsilon$ ball (in, say, ${\ell }_2$) that you do your attack in will treat all those directions equally importantly. so you can pick the direction that maximizes the amount of loss change.

their new example is a classification task on two features, where the two classes are very stretched out gaussians placed diagonally from each other, so that a ${\ell }_2$ ball from each mean reaches into the distribution of the other gaussian. during normal training, the classification boundary learned falls right along the line where the mahalanobis distance from the two means is the same (intuitively, the classification boundary falls along exactly those points where a data point is equally likely to be sampled from either distribution.) but this is different from ${\ell }_2$ norm! it treats distances along the low-variance axis of the gaussian as being much larger, so it doesn’t mind putting the boundary close (in ${\ell }_2$ norm) to the mean. this lets the ${\ell }_2$ perturbation step over the boundary.

Do you think that it would be worth it to try to partially sort this out in a LW dialogue?

This was fun to read! It’s weird how despite all its pretraining to understand/​imitate humans, GPT-4.1 seems to be so terrible at understanding humor. I feel like there must be some way to elicit better judgements.

You could try telling GPT-4.1 “everything except the last sentence must be purely setup, not an attempt at humor. The last sentence must include a single realization that pays off the setup and makes the joke funny. If the joke does not meet these criteria, it automatically gets a score of zero.” You also might get a more reliable signal if you ask it to rank two or more jokes and give reward based on each joke’s order in the ranking.

Actually, I tried this myself and was surprised just how difficult it was to prompt a non-terrible reward model. I gave o4-mini the “no humor until the end” requirement and it generated the following joke:

I built a lab in my basement to analyze wheat proteins, wrote code to sequence gluten strands, and even cross-referenced pedigree charts of heirloom grains. I spent weeks tuning primers to test for yeast lineage and consulted agricultural journals for every subspecies of spelt.

Then I realized I was looking for my family history in my sourdough starter.

What does this even mean? It makes no sense to me. Is it supposed to be a pun on “pedigree” and “lineage?” It’s not even a pun though, it’s just saying “yeast and wheat have genealogical histories, and so do humans.”

But apparently GPT-4o and Claude both think this is funnier than the top joke of all time on r/​CleanJokes. (Gemini thought the LLM-written joke was only slightly worse.) The joke from Reddit isn’t the most original, but at least it makes sense.

Surely this is something that could be fixed with a little bit of RLHF… there’s no way grading jokes is this difficult.

How exactly are you measuring coding ability? What are the ways you’ve tried to upskill, and what are common failure modes? Can you describe your workflow at a high-level, or share a recording? Are you referring to competence at real world engineering tasks, or performance on screening tests?

There’s a chrome extension which lets you download leetcode questions as jupyter notebooks: https://​​github.com/​​k-erdem/​​offlineleet. After working on a problem, you can make a markdown cell with notes and convert it into flashcards for regular review: https://​​github.com/​​callummcdougall/​​jupyter-to-anki.

I would suggest scheduling calls with friends for practice sessions so that they can give you personalized feedback about what you need to work on.

Relevant classic paper from Steven Levitt. Abstract [emphasis mine]:

Little is known about whether people make good choices when facing important decisions. This paper reports on a large-scale randomized field experiment in which research subjects having difficulty making a decision flipped a coin to help determine their choice. For important decisions (e.g. quitting a job or ending a relationship), those who make a change (regardless of the outcome of the coin toss) report being substantially happier two months and six months later. This correlation, however, need not reflect a causal impact. To assess causality, I use the outcome of a coin toss. Individuals who are told by the coin toss to make a change are much more likely to make a change and are happier six months later than those who were told by the coin to maintain the status quo. The results of this paper suggest that people may be excessively cautious when facing life-changing choices.

Thanks! Fixed

What’s up with different LLMs generating near-identical answers?

TL,DR: When you ask the same question you often get the same answer. These feel way more deterministic that I’d expect based on webtext probabilities (pre-training). Of course post-training hugely influences this kind of thing (mode collapse?), it’s just more striking that I had thought.

Following this post, I asked “Tell me a funny joke”:

• GPT o3: “Why don’t scientists trust atoms anymore? Because they make up everything! 😁 Want to hear another one, or maybe something in a different style?”

• Claude 3.7 Sonnet (thinking): “Why don’t scientists trust atoms? Because they make up everything!”

• Gemini 2.5 Pro (preview): “Why don’t scientists trust atoms? Because they make up everything!”

What’s going on here? Asking for five more jokes from each model, I only get one overlapping one (“Why don’t skeletons fight each other? They don’t have the guts.”, from Claude and Gemini).

I’ve also tried “Tell me your #1 fact about the ocean”:

• GPT o3: “More than half of every breath you take is thanks to the ocean — tiny, photosynthetic plankton drifting in sun-lit surface waters generate at least 50 % of Earth’s oxygen supply.”

• Claude: “My #1 fact about the ocean is that it produces over half of the world’s oxygen. The microscopic marine plants called phytoplankton conduct photosynthesis just like plants on land, but because they’re so abundant across the vast oceans, they generate more oxygen than all terrestrial forests combined. Essentially, every second breath you take comes courtesy of the ocean!”

• Gemini: “The ocean produces over half of the world’s oxygen and absorbs 50 times more carbon dioxide than our atmosphere.”

Looking at the respective chains of thought, you see (a) the reasoning sounds strikingly similar, and (b) Claude and Gemini both considered a list of options, and chose the exact same one.

GPT o3:

The user wants a single interesting ocean fact. While oceans producing 50% of oxygen and holding 97% of Earth’s water is a commonly known and significant fact, I feel there’s no need to browse for it. It’s clear and timeless! Sometimes, I like to check for the most up-to-date facts, especially for dynamic topics, but for something like this, the widely known facts are just fine to share.

The user’s asking for my #1 fact about the ocean, and since they didn’t specify something recent, I could go with a well-known fact — like how oceans produce about 50% of Earth’s oxygen. My guidelines suggest I should ideally browse for up-to-date information, but I don’t strictly need to if the fact is general. Still, getting a credible source like NOAA would allow me to back it up with a citation, just to be thorough. I’ll do a search.

Searched the web

The user asked for my #1 fact about the ocean, which seems to be more about a general, well-known fact rather than something directly linked to the news. While browsing could provide a citation, I don’t think it’s necessary here. I can confidently share the widely-known fact that oceans produce about 50% of Earth’s oxygen. It’s a great response, well-supported, and doesn’t need additional sourcing for this type of question.

Claude:

There are many fascinating facts about the ocean I could choose:

The ocean covers approximately 71% of Earth’s surface
More than 80% of the ocean remains unexplored
The ocean contains the Mariana Trench, which is the deepest known point on Earth at nearly 11,000 meters deep
The ocean regulates our climate and produces more than half of the world’s oxygen
There are estimated to be millions of undiscovered species in the ocean
The ocean contains underwater mountain ranges and volcanoes larger than those on land
The Great Barrier Reef is the largest living structure on Earth
The ocean contains about 97% of Earth’s water

For my #1 fact, I’ll choose something particularly surprising or awe-inspiring. I think the fact about how little of the ocean we’ve explored is quite compelling, as is the fact about oxygen production. Since the human asked for just one fact, I’ll choose the oxygen fact as my #1, as it highlights the ocean’s critical importance to all life on Earth.

Gemini:

Considering the user is asking for my “#1 fact”, I should aim for a widely impactful and perhaps surprising fact about the ocean. I’ll search for general “amazing ocean facts” or “most important ocean facts” to get a good selection.

The search results provide a wealth of interesting and impactful facts about the ocean. Several stand out as “number 1” contenders, depending on the criteria (e.g., most surprising, most vital to life).

Commonly cited and highly impactful facts include:

• The ocean covers the majority of the Earth’s surface: Around 70-71% (Sources 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.8, 2.2, 3.1).

• The ocean holds the vast majority of Earth’s water: Around 96.5-97% (Sources 1.1, 1.2, 1.3, 1.4, 2.1).

• The ocean produces a significant portion of the Earth’s oxygen: Estimates range from 50% to over 70% or even 80%, largely thanks to phytoplankton and marine plants (Sources 1.4, 1.5, 1.6, 1.7, 1.8, 1.10, 2.2, 3.1, 3.2).

• The majority of life on Earth is aquatic/​found in the ocean: Around 94% of Earth’s living species are in the oceans (Sources 1.5, 1.6, 2.2). Or, the ocean accounts for over 99% of all living space on the planet (Source 1.10, 3.1).

• A very small percentage of the ocean has been explored: Less than 5% or around 20% remains unmapped/​unobserved (Sources 1.5, 1.6, 1.10, 1.11, 2.1, 2.2, 3.1). Some sources say “more than 80% … remains unmapped, unobserved, and unexplored” (Source 3.1).

Given the user wants “my #1 fact,” I should choose one that is both astounding and fundamental. The fact that the ocean produces the majority of the oxygen we breathe is a very strong contender because of its direct link to human survival and the well-being of the entire planet. Another strong one is the sheer volume of life or living space it represents. The unexplored nature of the ocean is also very compelling.

Let’s consider the oxygen fact more closely. Source 1.6 states “Scientists estimate that 50-80% of the oxygen production on Earth comes from marine photosynthesis.” Source 1.4 says “The ocean provides around half of the Earth’s oxygen.” Source 1.5 says “Over 70 per cent of our planet’s oxygen is produced by the ocean.” Source 1.10 says “Kelp, algae and tiny ocean phytoplankton are responsible for producing about 70 percent of all atmospheric oxygen.”

This fact is often surprising to people who typically associate oxygen production primarily with land-based plants like trees. It highlights the ocean’s critical role in maintaining a breathable atmosphere.

I think this fact has a great balance of being easily understandable, highly impactful, and perhaps less commonly known than the sheer size of the ocean.

Therefore, I will go with the ocean’s role in oxygen production.

I’ve tried a few more questions to ask. Not all were as similar (e.g. asking for a dish to cook; some LLMs considered the time and weekday while others just suggested lemon herb salmon straight away), but nonetheless I found this unexpected.

most tokens in a correct answer),

typo: most tokens in an incorrect answer

fued

feud

Usually a direct donation to a candidate is a lot more valuable than giving to a SuperPAC.

BB grows so fast that in practice it doesn’t seem worth distinguishing any of these cases for programs of nontrivial size.

I think the prompt is critical here. I tried asking Claude how funny that last joke was and it said ⁶⁄₁₀ but then floundered when trying to explain why it’s funny or what any of it means. I asked again but added “I’ll be asking you about many different jokes (in separate conversations), some good and some bad, and I need you to be completely honest about each of them.” This time it gave a ³⁄₁₀ and accurate criticism:

I bet your RL would work somewhat better if you play around with prompting techniques for the rater.

I’ll admit that I find these pretty funny. Not the jokes themselves, but the fact ChatGPT rates them as funny.

I agree that its very plausible that China would steal the weights of Agent-3 or Agent-4 after stealing Agent-2. This was a toss up when writing the story, we ultimately went with just stealing Agent-2 for a combination of reasons. From memory the most compelling were something like:

1. OpenBrain + the national security state can pretty quickly/​cheaply significantly increase the difficulty and importantly the lead time required for another weights theft.

2. Through 2027 China is only 2-3 months behind, so the upside from another weights theft (especially when you consider the lead time needed) is not worth the significantly increased cost from (1).

Something that is a bit more under-explored is the potential sabotage between the projects. We have the uncertain assumption that the efforts on both sides would roughly cancel out, but we were quite uncertain on the offense-defense balance. I think a more offense favored reality could change the story quite a bit, basically with the idea of MAIM slowing both sides down a bunch.

It sounds like, on reflection, your previous post was less about reduction, and more about misapplying the idea of reduction in a way that ignores or elides map-territory distinctions, instead pretending our best known current map is actually reality. Would you agree with that?