MichaelDickens

• MichaelDickens 17 Jun 2026 12:20 UTC

  5 points

  1

  in reply to: rahulxyz’s comment on: A fron­tier AI com­pany should shut down

  There is virtue in saying what should be done, even if it has minimal chance of actually happening.

• MichaelDickens 17 Jun 2026 12:20 UTC

  2 points

  0

  in reply to: denkenberger’s comment on: A fron­tier AI com­pany should shut down

  That would save the AI companies money, but would not really get them more research done.

  If the supply curve shifts right, quantity supplied increases.

  As for whether it is much harder to enforce if there are automated AI researchers, I don’t know.

  If we already have an autonomous AI researcher, there’s a good chance the researcher takes off to ASI while policy-makers are still trying to figure out what’s doing on. Originally I said it was harder to enforce but really I think the concern is that it makes AI development go a lot faster.

• MichaelDickens 17 Jun 2026 3:53 UTC

  2 points

  0

  in reply to: denkenberger’s comment on: A fron­tier AI com­pany should shut down

  pausing further development once they get an automated AI researcher

  I’m not sure I get this...are you talking about a scenario where AI companies develop a viable automated AI researcher and then nobody uses that automated researcher? After developing autonomous AI research seems like a difficult time to pause because a pause is way harder to enforce now.

• MichaelDickens 16 Jun 2026 17:06 UTC

  9 points

  3

  in reply to: Yair Halberstadt’s comment on: A fron­tier AI com­pany should shut down

  Minimal chance that any AI company shuts down (at least under current circumstances). But I figured it was worth publicly acknowledging that an AI company should shut down, even though I expect they won’t.

• MichaelDickens 16 Jun 2026 2:06 UTC

  8 points

  1

  in reply to: Gretta Duleba’s comment on: Gretta Duleba’s Shortform

  Writing down other people’s thoughts is an underrated activity!

• MichaelDickens 16 Jun 2026 0:14 UTC

  2 points

  0

  in reply to: Raemon’s comment on: A fron­tier AI com­pany should shut down

  Yeah the obvious reason to predict more success on LW is that the post is implicitly pessimistic about AI companies’ ability to solve alignment. The reason I expected less success is that the post is making a simplified, arguably naive* argument where I could’ve made many caveats, or said a lot more about the real-world complexities of what I’m proposing, but I left those bits out because ultimately I didn’t think they were important enough. LW users (including me) tend to write comments pointing out those things I didn’t talk about.

  In retrospect I suppose it’s unsurprising that this post was well-received on LW. It might just be risk-aversion combined with the fact that I’m not very good at predicting which posts LW users will like.

  *in the colloquial sense of “not understanding how the world works”, not the mathematician’s sense

• MichaelDickens 15 Jun 2026 23:41 UTC

  3 points

  1

  in reply to: Jiro’s comment on: Use the Try Harder, Luke

  (Yes, I know this is an old post.)

  One of the great things about LessWrong is you can still comment on old posts, even from decades ago!

• MichaelDickens 15 Jun 2026 22:51 UTC

  2 points

  0

  in reply to: Thane Ruthenis’s comment on: Leg­ible vs. Illeg­ible AI Safety Problems

  To add to this:

  I would speculate that there are more than a hundred factorization algorithms that are both more efficient than the general number field sieve and an equal or shorter inferential distance away, but that haven’t been found. If I’m right, then it’s unsurprising that we found GNFS by stumbling around in a high-dimensional search space.

• MichaelDickens 15 Jun 2026 22:39 UTC

  4 points

  0

  on: A fron­tier AI com­pany should shut down

  This is the latest of several occasions in which I mentally predicted that a post would be well-received on the EA Forum and poorly received on LessWrong, and then it ended up being the other way around.

• MichaelDickens 15 Jun 2026 22:28 UTC

  2 points

  0

  in reply to: Brendan Long’s comment on: A fron­tier AI com­pany should shut down

  I’d expect the US government to demand the weights, GPUs, and all research for national security reasons

  I think there’s a pretty good chance this wouldn’t happen (maybe ⁵⁰⁄₅₀ but I haven’t really thought about it). The US government is mostly not AGI-pilled. Things might be different a year from now.

  Also, even if they wanted to, I think the government would have a hard time putting together an org that’s as good at AI development as OpenAI, Anthropic, or Google.

• MichaelDickens 15 Jun 2026 18:16 UTC

  24 points

  12

  in reply to: Richard_Kennaway’s comment on: A fron­tier AI com­pany should shut down

  A related thesis you could have is: “if you’re a frontier AI company and you’re IPO’ing soon, you should put safeguards in place to give yourself the option to shut down the company if things look too dangerous.”

  There was news recently about how apparently Elon Musk organized the SpaceX governance in such a way that shareholders aren’t allowed to sue him. If he can pull that off, I bet AI companies can find a way to create similar safety protections, if they’re sufficiently motivated.

A fron­tier AI com­pany should shut down

• MichaelDickens 15 Jun 2026 14:22 UTC

  4 points

  0

  in reply to: Gordon Seidoh Worley’s comment on: How to Suffer Less

  I feel like that is not the mechanism in my case? When I’m doing something fun, I’m not secretly experiencing boredom but unaware of it. I’m just not experiencing boredom. Meditation causes me to be bored, rather than revealing the boredom that was already there.

• MichaelDickens 14 Jun 2026 4:56 UTC

  2 points

  0

  in reply to: aysja’s comment on: An­thropic’s Core Views on AI Safety

  If I put on the “we need empirical feedback from neural nets to make progress on alignment” hat, along with my “prudence” hat, I’m thinking things more like, “okay let’s stop scaling now, and just work really hard on figuring out how exactly capabilities emerged between e.g., GPT-3 and GPT-4. Like, what exactly can we predict about GPT-4 based on GPT-3? Can we break down surprising and abrupt less-scary capabilities into understandable parts, and generalize from that to more-scary capabilities?” Basically, I’m hoping for a bunch more proof of concept that Anthropic is capable of understanding and controlling current systems, before they scale blindly. If they can’t do it now, why should I expect they’ll be able to do it then?

  Looking back in 2026, this strikes me as a sad missed opportunity. We’re at a point where the next capabilities jump will probably not get us anything as easily describable as “gain the ability to multiply 3-digit numbers”. Instead it will be something fuzzy and hard to operationalize.

  There was a window where we could do empirical work in the genre of “figure out how exactly model capabilities improve with successive generations”, but that work is much harder to do now—it might not even be possible at all.

• MichaelDickens 14 Jun 2026 0:35 UTC

  4 points

  −1

  on: MichaelDick­ens’s Shortform

  Harebrained alignment idea: LLMs can’t be trusted to assist with alignment research because it’s too easy to get them to say what you want to hear (e.g. make you think you’ve solved a safety problem when you haven’t). Therefore, AI companies should train a distinct LLM that doesn’t go through RLHF or any other process by which it’s reinforced based on how much people like its responses.

  This doesn’t fix the problem that next-token-generators are architecturally better at generating plausible-sounding statements than true statements, but it does help with sycophancy etc.

  Another related idea is to tune an LLM really hard toward criticism: make it work as hard as it can to come up with reasons why you’re wrong.

• MichaelDickens 14 Jun 2026 0:27 UTC

  8 points

  0

  on: How to Suffer Less

  I used to meditate daily, but I stopped because it was a source of suffering. Specifically, it was boring. When my meditation time was coming up, I found myself not wanting to do it; while meditating, I frequently found myself wishing I was doing something else instead. And I never noticed any significant positive effects. I experienced a mild reduction in anxiety, but not enough to outweigh the suffering of the meditation itself.

• MichaelDickens 14 Jun 2026 0:09 UTC

  2 points

  0

  in reply to: Petropolitan’s comment on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  Yeah, it’s a judgment call as to whether trying to solve alignment empirically is more or less doomed than trying to coordinate to not build ASI. I don’t have a clear argument either way; the best I’ve come up with is a list of heuristic reasons why I believe the “empirical alignment” approach is more doomed, which I wrote here.

• MichaelDickens 12 Jun 2026 17:18 UTC

  11 points

  15

  in reply to: Petropolitan’s comment on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  I can’t speak for people who actually work on theoretical alignment, but my perspective is:

  • Yes, developing theory without the ability to empirically test your theories is really hard and does not have a good historical track record.

  • To do empirical work on aligning ASI, we have to build the thing that kills us, which means we die.

  The seeming impossibility of theoretical alignment work isn’t a good argument that we should do empirical work instead. The two options are: we do the thing that’s really hard and probably won’t work, or we do the thing that kills us. I prefer the former.

• MichaelDickens 11 Jun 2026 0:56 UTC

  4 points

  0

  in reply to: MichaelDickens’s comment on: Akash’s Shortform

  I should clarify that there is a gap between “Anthropic says they are going to conduct research on how to implement a pause” and “Anthropic is actively conducting research on how to implement a pause”, and I am not confident that the former leads to the latter.

How valuable are weak AI safety reg­u­la­tions?