MichaelDickens

Karma: 2,842

MichaelDickens 19 Jun 2026 16:34 UTC
2 points
0
in reply to: Salmonus_Kim’s comment on: A frontier AI company should shut down
Is the idea that AI companies wouldn’t be able to make money if everyone used local AI instead, and they’d be forced to close down?

If the idea is that some people can use local AI to cut down AI companies’ revenue, that might be net positive, although I don’t think it would ever make a big enough dent to force AI companies to shut down. I don’t do it personally but if people want to, I say sure, go for it.

The probability of a superintelligence developing on a personal device is quite low

That’s true, but the other side of the coin is that if a local AI is dumber, then it’s also going to be less useful.

if such a model remains offline, the potential for catastrophic threats is largely mitigated

I don’t think that’s realistic. Back in the day there was a lot of talk about how AI will be safe because we will keep it in a box with no internet access. Then as soon as AI companies built AI that could usefully use the internet, they hooked it up to the internet. Having internet access is just way too useful to expect people not to do it.

MichaelDickens 18 Jun 2026 3:01 UTC
4 points
0
on: Lock-In Risk Needs More Researchers. Here’s Where to Start

Here Are The Pathways

I think this chart misses the most important (non-extinction) pathway, which is that an AI is aligned with the wrong set of values and permanently fixes us on a highly suboptimal path. The level of suboptimality could range from “good but insufficiently flourishing” to a moral catastrophe, e.g. life is good for (post-)humans but other sentient beings are permanently tortured for our mild convenience—something like factory farming, but on a bigger scale and lasting until heat death of the universe.

“AI-driven power concentration’ also feels like a weird category. Power concentration isn’t inherently bad if the ASI in charge has good values. Power diffusion can lead to catastrophic outcomes as well, e.g. competing ASIs torture simulations of their principals as leverage. Although that seems distinct from “lock-in risk” so it probably doesn’t belong on your chart.

MichaelDickens 18 Jun 2026 1:19 UTC
10 points
0
on: Gears for political races

As mentioned above, I estimate most canvassers get 0.1 to 0.3 votes per hour[5] (that’s after adjusting for if we’re counterfactual or not, for people telling us what we want to hear, for volunteers reporting overly rosy numbers, and a general prior towards small effect sizes).

Can you say more about how you came up with this estimate? i.e. what exact calculation did you use?

MichaelDickens 17 Jun 2026 12:20 UTC
5 points
1
in reply to: rahulxyz’s comment on: A frontier AI company should shut down
There is virtue in saying what should be done, even if it has minimal chance of actually happening.

MichaelDickens 17 Jun 2026 12:20 UTC
2 points
0
in reply to: denkenberger’s comment on: A frontier AI company should shut down

That would save the AI companies money, but would not really get them more research done.

If the supply curve shifts right, quantity supplied increases.

As for whether it is much harder to enforce if there are automated AI researchers, I don’t know.

If we already have an autonomous AI researcher, there’s a good chance the researcher takes off to ASI while policy-makers are still trying to figure out what’s doing on. Originally I said it was harder to enforce but really I think the concern is that it makes AI development go a lot faster.

MichaelDickens 17 Jun 2026 3:53 UTC
2 points
0
in reply to: denkenberger’s comment on: A frontier AI company should shut down

pausing further development once they get an automated AI researcher

I’m not sure I get this...are you talking about a scenario where AI companies develop a viable automated AI researcher and then nobody uses that automated researcher? After developing autonomous AI research seems like a difficult time to pause because a pause is way harder to enforce now.

MichaelDickens 16 Jun 2026 17:06 UTC
9 points
3
in reply to: Yair Halberstadt’s comment on: A frontier AI company should shut down
Minimal chance that any AI company shuts down (at least under current circumstances). But I figured it was worth publicly acknowledging that an AI company should shut down, even though I expect they won’t.

MichaelDickens 16 Jun 2026 2:06 UTC
10 points
1
in reply to: Gretta Duleba’s comment on: Gretta Duleba’s Shortform
Writing down other people’s thoughts is an underrated activity!

MichaelDickens 16 Jun 2026 0:14 UTC
2 points
0
in reply to: Raemon’s comment on: A frontier AI company should shut down
Yeah the obvious reason to predict more success on LW is that the post is implicitly pessimistic about AI companies’ ability to solve alignment. The reason I expected less success is that the post is making a simplified, arguably naive* argument where I could’ve made many caveats, or said a lot more about the real-world complexities of what I’m proposing, but I left those bits out because ultimately I didn’t think they were important enough. LW users (including me) tend to write comments pointing out those things I didn’t talk about.

In retrospect I suppose it’s unsurprising that this post was well-received on LW. It might just be risk-aversion combined with the fact that I’m not very good at predicting which posts LW users will like.

*in the colloquial sense of “not understanding how the world works”, not the mathematician’s sense

MichaelDickens 15 Jun 2026 23:41 UTC
3 points
1
in reply to: Jiro’s comment on: Use the Try Harder, Luke

(Yes, I know this is an old post.)

One of the great things about LessWrong is you can still comment on old posts, even from decades ago!

MichaelDickens 15 Jun 2026 22:51 UTC
2 points
0
in reply to: Thane Ruthenis’s comment on: Legible vs. Illegible AI Safety Problems
To add to this:

I would speculate that there are more than a hundred factorization algorithms that are both more efficient than the general number field sieve and an equal or shorter inferential distance away, but that haven’t been found. If I’m right, then it’s unsurprising that we found GNFS by stumbling around in a high-dimensional search space.

MichaelDickens 15 Jun 2026 22:39 UTC
4 points
0
on: A frontier AI company should shut down
This is the latest of several occasions in which I mentally predicted that a post would be well-received on the EA Forum and poorly received on LessWrong, and then it ended up being the other way around.

MichaelDickens 15 Jun 2026 22:28 UTC
2 points
0
in reply to: Brendan Long’s comment on: A frontier AI company should shut down

I’d expect the US government to demand the weights, GPUs, and all research for national security reasons

I think there’s a pretty good chance this wouldn’t happen (maybe ⁵⁰⁄₅₀ but I haven’t really thought about it). The US government is mostly not AGI-pilled. Things might be different a year from now.

Also, even if they wanted to, I think the government would have a hard time putting together an org that’s as good at AI development as OpenAI, Anthropic, or Google.

MichaelDickens 15 Jun 2026 18:16 UTC
25 points
12
in reply to: Richard_Kennaway’s comment on: A frontier AI company should shut down
A related thesis you could have is: “if you’re a frontier AI company and you’re IPO’ing soon, you should put safeguards in place to give yourself the option to shut down the company if things look too dangerous.”

There was news recently about how apparently Elon Musk organized the SpaceX governance in such a way that shareholders aren’t allowed to sue him. If he can pull that off, I bet AI companies can find a way to create similar safety protections, if they’re sufficiently motivated.

MichaelDickens 15 Jun 2026 14:22 UTC
4 points
0
in reply to: Gordon Seidoh Worley’s comment on: How to Suffer Less
I feel like that is not the mechanism in my case? When I’m doing something fun, I’m not secretly experiencing boredom but unaware of it. I’m just not experiencing boredom. Meditation causes me to be bored, rather than revealing the boredom that was already there.

MichaelDickens 14 Jun 2026 4:56 UTC
2 points
0
in reply to: aysja’s comment on: Anthropic’s Core Views on AI Safety

If I put on the “we need empirical feedback from neural nets to make progress on alignment” hat, along with my “prudence” hat, I’m thinking things more like, “okay let’s stop scaling now, and just work really hard on figuring out how exactly capabilities emerged between e.g., GPT-3 and GPT-4. Like, what exactly can we predict about GPT-4 based on GPT-3? Can we break down surprising and abrupt less-scary capabilities into understandable parts, and generalize from that to more-scary capabilities?” Basically, I’m hoping for a bunch more proof of concept that Anthropic is capable of understanding and controlling current systems, before they scale blindly. If they can’t do it now, why should I expect they’ll be able to do it then?

Looking back in 2026, this strikes me as a sad missed opportunity. We’re at a point where the next capabilities jump will probably not get us anything as easily describable as “gain the ability to multiply 3-digit numbers”. Instead it will be something fuzzy and hard to operationalize.

There was a window where we could do empirical work in the genre of “figure out how exactly model capabilities improve with successive generations”, but that work is much harder to do now—it might not even be possible at all.

MichaelDickens 14 Jun 2026 0:35 UTC
4 points
−1
on: MichaelDickens’s Shortform
Harebrained alignment idea: LLMs can’t be trusted to assist with alignment research because it’s too easy to get them to say what you want to hear (e.g. make you think you’ve solved a safety problem when you haven’t). Therefore, AI companies should train a distinct LLM that doesn’t go through RLHF or any other process by which it’s reinforced based on how much people like its responses.

This doesn’t fix the problem that next-token-generators are architecturally better at generating plausible-sounding statements than true statements, but it does help with sycophancy etc.

Another related idea is to tune an LLM really hard toward criticism: make it work as hard as it can to come up with reasons why you’re wrong.

MichaelDickens 14 Jun 2026 0:27 UTC
8 points
0
on: How to Suffer Less
I used to meditate daily, but I stopped because it was a source of suffering. Specifically, it was boring. When my meditation time was coming up, I found myself not wanting to do it; while meditating, I frequently found myself wishing I was doing something else instead. And I never noticed any significant positive effects. I experienced a mild reduction in anxiety, but not enough to outweigh the suffering of the meditation itself.

MichaelDickens 14 Jun 2026 0:09 UTC
2 points
0
in reply to: Petropolitan’s comment on: PSA: Almost nobody is directly working on superintelligent alignment
Yeah, it’s a judgment call as to whether trying to solve alignment empirically is more or less doomed than trying to coordinate to not build ASI. I don’t have a clear argument either way; the best I’ve come up with is a list of heuristic reasons why I believe the “empirical alignment” approach is more doomed, which I wrote here.

MichaelDickens 12 Jun 2026 17:18 UTC
11 points
15
in reply to: Petropolitan’s comment on: PSA: Almost nobody is directly working on superintelligent alignment
I can’t speak for people who actually work on theoretical alignment, but my perspective is:
- Yes, developing theory without the ability to empirically test your theories is really hard and does not have a good historical track record.
- To do empirical work on aligning ASI, we have to build the thing that kills us, which means we die.
The seeming impossibility of theoretical alignment work isn’t a good argument that we should do empirical work instead. The two options are: we do the thing that’s really hard and probably won’t work, or we do the thing that kills us. I prefer the former.