Ian McKenzie

Karma: 424

Ian McKenzie 13 Apr 2026 5:05 UTC
8 points
0
on: Ian McKenzie’s Shortform
I find it hard to imagine just how different my entire experience of the world and my place in it would be if I had a different worldview. I’ve never really believed in God, definitely not seriously, but surely everything would feel different if I thought it was all for the best, or part of a great plan, or only temporary, on my way to something more real and important. As opposed to believing that it’s all arbitrary and there is no reason behind existence, no design.

Or if I thought that the best days of humanity were long in the past, and that we’d never again reach the heights achieved by our ancestors. Or I was in the middle ages and thought the Greeks were the best we’d ever do. As opposed to living at the middle of an exponential increase in GDP, and progress is commonplace.

Or I thought that I’d live a normal life, and planning for retirement made sense, or even being able to answer the question of where I want to see myself in five years. As opposed to expecting a singularity to either destroy or transform the world before I’m 40.

How much do these background beliefs control what we can think, or what we can feel?

Ian McKenzie 23 Mar 2026 0:22 UTC
2 points
0
on: Ian McKenzie’s Shortform
It seems like the switch where high-status brands went from making high quality products to featuring their name and logo prominently was good for both producer and consumer: the producer gets free advertising, and the consumer more clearly signals their wealth. I don’t know the history of this, but I’m interested in how this happened – seems like there’s a tough coordination problem, where if one brand/consumer switches then they just look tacky.

(Not all brands have switched, and maybe there’s a new money/old money difference.)

Ian McKenzie 4 Mar 2026 17:50 UTC
2 points
0
in reply to: habryka’s comment on: Responsible Scaling Policy v3
Drake is right, sorry for the confusion. We were not intentionally misleading – we missed a footnote on the announcement when putting together the initial tweet thread that narrowed the claim to just bio rather than CBRN, as discussed in the rest of the announcement. We did later find a vulnerability that allowed us to bypass the filters in the bio setting, reported it, and it was patched. I think that follow-up work took more on the order of 40 person-hours, but was a general method that could extract information in a range of settings. I don’t know how likely it is that there are further such vulnerabilities.

Even if single-query jailbreaking was O(10) hours though, having to send many queries to discover that jailbreak makes it much easier to catch through monitoring.

Ian McKenzie 22 Feb 2026 0:45 UTC
12 points
4
on: Ian McKenzie’s Shortform
Some infectious disease graphs I would like to see but haven’t been able to find:
- Effectiveness of covid and flu vaccines vs time since injection.
- Contagiousness with cold/flu/covid vs time since infection.
- Decrease in infection probability vs distance from infected person.

Ian McKenzie 16 Feb 2026 2:53 UTC
1 point
0
in reply to: Raemon’s comment on: Ian McKenzie’s Shortform

probably most patterns that evolved naturally don’t successfully navigate superintelligence well and I’m not sure it’s the right standard for them.

I’m not sure what you mean by standard, but navigating superintelligence well is something I care a lot about. So it seems like a reasonable thing to criticize a system for, and it would be great if we found a pattern that did navigate it well (even if finding or switching to another one is very hard).

Ian McKenzie 3 Feb 2026 3:57 UTC
6 points
0
in reply to: AlphaAndOmega’s comment on: Ian McKenzie’s Shortform
Under this view you can totally have intermediary metrics, they just look more like “how much does your society avoid tragedies of the commons” rather than “what is the median quality of life”.

To be clear, this post was not intended as a subtle endorsement of communism. I agree with MondSemmel’s point that basically any system which produced slower economic growth would probably do better under this view, if only because AI development is slower.

Ian McKenzie 2 Feb 2026 0:02 UTC
57 points
11
on: Ian McKenzie’s Shortform
Maybe the most important test for a political or economic system is whether it self-destructs. This is in contrast to whether it produces good intermediate outcomes. In particular, if free-market capitalism leads to an uncontrolled intelligence explosion, then it doesn’t matter if it produced better living standards than alternative systems for ~200 years – it still failed at the most important test.

A couple of other ways to put it:
- Would the US economic/political system pass the Great Filter?
- Would Norway do an intelligence explosion?
Under this view, political/economic systems that produce less growth but don’t create the incentives for unbounded competition are preferred. Sadly, for Molochian reasons this seems hard to pull off.

Ian McKenzie 25 Jan 2026 19:30 UTC
6 points
5
on: To be well-calibrated is to be punctual
I like this idea. As I tried to be more organized and less late to things, I implicitly did something like this, and this is a nice framing of that process.

I do think the asymmetry of the consequences distorts the updates I make a little though – since I am trying hard not to be late, I sometimes leave an unreasonable amount of buffer. I was once 45 minutes early to an appointment because I was taking public transport to an unfamiliar part of the city. I find it harder to make an update based on being early, because I don’t know the variance – if I’m late (and I was trying hard not to be), then I clearly underestimated the worst case, but if I’m early then I could have just got lucky.

Ian McKenzie 18 Jan 2026 18:42 UTC
1 point
0
on: If AI alignment is only as hard as building the steam engine, then we likely still die
I think there are multiple ways of interpreting “alignment is as difficult as X”. There’s “the safety issues in building AGI are similar to the safety issues in building X”, but there’s also “solving the safety issues in building AGI takes the same level of total effort as building X”.

I interpreted Chris Olah’s graph as the latter – that the ‘steam engine world’ is a world where solving AI safety takes as much total effort as building the steam engine, agnostic of how that effort is spent. NOT that in those worlds, you solve AI safety issues in the same way that you solve steam engine safety issues.

Put another way, I was imagining the graph as primarily quantitative – you could crudely replace the x-axis with “# person-hours”.

Ian McKenzie 11 Jan 2026 21:00 UTC
24 points
11
on: Ian McKenzie’s Shortform
John Wentworth says:

You will never find a $100 bill on the floor of Grand Central Station at rush hour, because someone would have picked it up already.

Are you really less likely to find $100 in Grand Central Station than finding $100 anywhere else? It’s true that there are many more people who could find it before you, but there are also many more people that could drop $100. If you imagine a 1D version, where everyone walks through either Grand Central Station or a quiet alley along the same line, one after the other, then it seems like you should be equally likely to find $100 in either case – if the person in front of you in the line drops $100.

Ian McKenzie 2 Jan 2026 22:01 UTC
4 points
2
in reply to: Ben Pace’s comment on: The Company Man

There are many ideas in here that I’ve heard said offhand but never really dived into, and there’s something very informative and satisfying about seeing them painted in detail. Similar to the difference between a one-sentence description of a painting, and the actual painting.

It’s satire though, it conveys some vibes but the detail that’s being painted is not an accurate portrayal of the actual detail that exists. Put another way, I think it would be a mistake if you encountered someone in real life and thought “ah yes, that’s ‘the person who feels they have had a meditation/emotional insight and this has gotten them over the intellectual hump of not building killing machines’ from The Company Man, I know how they think”.

Ian McKenzie 16 Dec 2025 21:17 UTC
5 points
0
on: How Colds Spread
I wonder what the driving factor of transmission is before symptoms emerge. If everyone was very careful about following good practices once they were obviously sick (e.g. wearing masks, sanitizing their hands after blowing nose or any contact with face), what would we have to do to prevent the spread? Do fomites become more important if you aren’t coughing or sneezing yet?

Ian McKenzie 11 Jul 2025 2:04 UTC
2 points
0
on: The bitter lesson of misuse detection
Our paper on defense in depth (STACK) found similar results – similarly-sized models with a few-shot prompt significantly outperformed the specialized guard models, even when adjusting for FPR on benign queries.

Ian McKenzie 8 Aug 2023 2:07 UTC
5 points
0
in reply to: Honglu Fan’s comment on: Password-locked models: a stress case for capabilities evaluation
One thing is that even given access to the model weights and the code behind the API, you could not tell if the model was password-locked, whereas you would see the hardcoded verifier. Thus if a lab wanted to hide capabilities they could delete the training data and you would have no way of knowing.

Ian McKenzie 21 Feb 2023 0:12 UTC
3 points
0
in reply to: Gunnar_Zarncke’s comment on: Russell Conjugations list & voting thread
The Wikipedia article has a typo in one of these: it should say “I am sparkling; you are unusually talkative; he is drunk.” (as in the source)

Ian McKenzie 29 Sep 2022 18:08 UTC
1 point
1
in reply to: Dave Orr’s comment on: Inverse Scaling Prize: Round 1 Winners
We did do human validation on the tasks with Surge: redefine-math, quote-repetition, and hindsight-neglect all got 100% agreement, and NeQA got 98% agreement. I agree though that it seems likely many people would do the task ‘wrong’, so maybe the task would be improved by adding clearer instructions.

The situation feels somewhat like model splintering to me: the few-shot examples fit both patterns but the question doesn’t. The larger models are learning the incorrect generalization.

I think it’s important to note that LMs learning to respond in the same way as the average internet user is in some sense expected but can still be an example of inverse scaling – we would like our models to be smarter than that.

Ian McKenzie 16 Apr 2022 17:18 UTC
1 point
0
in reply to: NunoSempere’s comment on: A Quick Guide to Confronting Doom
I think 75% is 1:3 rather than 1:2.

Ian McKenzie 9 Apr 2022 20:57 UTC
1 point
0
in reply to: Jonathan Paulson’s comment on: Why Instrumental Goals are not a big AI Safety Problem
A couple of things that come to mind:
- You’ll run into ELK-like difficulties when trying to train B: maybe it looks to the human that the off switch is still accessible but in reality you would not be able to activate it for some reason—A steers into a Faraday cage or something more clever.
- (In response to your reply to acylhalide) Assuming you solve the training difficulties, B has to be at least as good at modelling the world as A: it needs to be able to predict the consequences of action sequences from A to see if the off switch is accessible.
- (In response to your reply to Evan) I don’t necessarily expect A+B+R to be as capable as A+R: adding B introduces a huge shift to the environment that A has to deal with. If A always disabled the off switch in regular training, and now is unable to execute action sequences that involve that, then it may not be able to act effectively.

Ian McKenzie 9 Aug 2020 13:52 UTC
1 point
0
on: Can we expect more value from AI alignment than from an ASI with the goal of running alternate trajectories of our universe?
Is your suggestion to run this system as a source of value, simulating lives for their own sake rather than to improve the quality of life of sentient beings in our universe? Our history (and present) aren’t exactly utopian, and I don’t see any real reason to believe that slight variations on it would lead to anything happier.

I think we can expect to achieve a lot more than that from a properly aligned AGI. There is so much suffering that could be alleviated right now with proper coordination, as a lower bound on how much better it could be than just effectively running copies of our timeline but at lower resolution.

Ian McKenzie 7 Aug 2020 9:41 UTC
1 point
0
on: My take on CHAI’s research agenda in under 1500 words
Is the “Going Beyond Agents” section part of CHAI’s research agenda, or your take on further challenges from an embedded agency perspective?