MichaelDickens

Why would AI com­pa­nies use hu­man-level AI to do al­ign­ment re­search?

• MichaelDickensApr 25, 2025, 5:54 PM

  −1 points

  −2

  in reply to: Shankar Sivarajan’s comment on: MichaelDick­ens’s Shortform

  I would not describe it as heroic. I think it’s approximately morally equivalent to choosing an 80% chance of making all Americans immortal (but not non-Americans) and a 20% chance of killing everyone in the world.

  This is not a perfect analogy because the philosophical arguments for discounting future generations are stronger than the arguments for discounting non-Americans.

  (Also my P(doom) is higher than 20%, that’s just an example)

• MichaelDickensApr 25, 2025, 4:18 PM

  6 points

  1

  in reply to: brambleboy’s comment on: MichaelDick­ens’s Shortform

  The argument people make is that LLMs improve the productivity of people’s safety research so it’s worth paying. That kinda makes sense. But I do think “don’t give money to the people doing bad things” is a strong heuristic.

  I’m a pretty big believer in utilitarianism but I also think people should be more wary of consequentialist justifications for doing bad things. Eliezer talks about this in Ends Don’t Justify Means (Among Humans), he’s also written some (IMO stronger) arguments elsewhere but I don’t recall where.

  Basically, if I had a nickel for every time someone made a consequentialist argument for why doing a bad thing was net positive, and then it turned out to be net negative, I’d be rich enough to diversify EA funding away from Good Ventures.

  ---

  I have previously paid for LLM subscriptions (I don’t have any currently) but I think I was not giving enough consideration to the “ends don’t justify means among humans” principle, so I will not buy any subscriptions in the future.

• MichaelDickensApr 25, 2025, 4:03 PM

  5 points

  2

  in reply to: Neel Nanda’s comment on: MichaelDick­ens’s Shortform

  I don’t think “not really caring” necessarily means someone is being deceptive. I hadn’t really thought through the terminology before I wrote my original post, but I would maybe define 3 categories:

  1. claims to care about x-risk, but is being insincere

  2. genuinely cares about x-risk, but also cares about other things (making money etc.), so they take actions that fit their non-x-risk motivations and then come up with rationalizations for why those actions are good for x-risk

  3. genuinely cares about x-risk, and has pure motivations, but sometimes make mistakes and end up increasing x-risk

  I would consider #1 and #2 to be “not really caring”. #3 really cares. But from the outside it can be hard to tell the difference between the three. (And in fact, from the inside, it’s hard to tell whether you’re a #2 or a #3.)

  On a more personal note, I think in the past I was too credulous about ascribing pure motivations to people when I had disagreements with them, when in fact the reason for the disagreement was that I care about x-risk and they’re either insincere or rationalizing. My original post is something I think Michael!2018 would benefit from reading.

• MichaelDickensApr 25, 2025, 12:18 AM

  10 points

  10

  in reply to: Neel Nanda’s comment on: MichaelDick­ens’s Shortform

  What probability do you put that, if Anthropic had really tried, they could have meaningfully coordinated with Openai and Google? Mine is pretty low

  Not GP but I’d guess maybe 10%. Seems worth it to try. IMO what they should do is hire a team of top negotiators to work full-time on making deals with other AI companies to coordinate and slow down the race.

  ETA: What I’m really trying to say is I’m concerned Anthropic (or some other company) would put in a half-assed effort to cooperate and then give up, when what they should do is Try Harder. “Hire a team to work on it full time” is one idea for what Trying Harder might look like.

What AI safety plans are there?

• MichaelDickensApr 23, 2025, 7:07 PM

  2 points

  1

  in reply to: Cole Wyeth’s comment on: MichaelDick­ens’s Shortform

  Your comment about 1e-6 p-doom is not right because we face many other X-risks that developing AGI would reduce.

  Ah you’re right, I wasn’t thinking about that. (Well I don’t think it’s obvious that an aligned AGI would reduce other x-risks, but my guess is it probably would.)

• MichaelDickensApr 23, 2025, 6:11 PM

  118 points

  35

  on: MichaelDick­ens’s Shortform

  I find it hard to trust that AI safety people really care about AI safety.

  • DeepMind, OpenAI, Anthropic, and SSI were all founded in the name of safety. Instead they have greatly increased danger. And at least OpenAI and Anthropic have been caught lying about their motivations:

    • OpenAI: claiming concern about hardware overhang and then trying to massively scale up hardware; promising compute to superalignment team and then not giving it; telling board that model passed safety testing when it hadn’t; too many more to list.

    • Anthropic: promising (in a mealy-mouthed technically-not-lying sort of way) not to push the frontier, and then pushing the frontier; trying (and succeeding) to weaken SB-1047; lying about their connection to EA (that’s not related to x-risk but it’s related to trustworthiness).

  • For whatever reason, I had the general impression that Epoch is about reducing x-risk (and I was not the only one with that impression) but:

    • Epoch is not about reducing x-risk, and they were explicit about this but I didn’t learn it until this week

    • its FrontierMath benchmark was funded by OpenAI and OpenAI allegedly has access to the benchmark (see comment on why this is bad)

    • some of their researchers left to start another build-AGI startup (I’m not sure how badly this reflects on Epoch as an org but at minimum it means donors were funding people who would go on to work on capabilities)

    • Director Jaime Sevilla believes “violent AI takeover” is not a serious concern, and “I also selfishly care about AI development happening fast enough that my parents, friends and myself could benefit from it, and I am willing to accept a certain but not unbounded amount of risk from speeding up development”, and “on net I support faster development of AI, so we can benefit earlier from it” which is a very hard position to justify (unjustified even on P(doom) = 1e-6, unless you assign ~zero value to people who are not yet born)

  • I feel bad picking on Epoch/​Jaime because they are being unusually forthcoming about their motivations, in a way that exposes them to criticism. This is noble of them; I expect most orgs not to be this noble.

    • When some other org does something that looks an awful lot like it’s accelerating capabilities, and they make some argument about how it’s good for safety, I can’t help but wonder if they secretly believe the same things as Epoch and are not being forthright about their motivations

    • My rough guess is for every transparent org like Epoch, there are 3+ orgs that are pretending to care about x-risk but actually don’t

  • Whenever some new report comes out about AI capabilities, like the METR task duration projection, people talk about how “exciting” it is[1]. There is a missing mood here. I don’t know what’s going on inside the heads of x-risk people such that they see new evidence on the potentially imminent demise of humanity and they find it “exciting”. But whatever mental process results in this choice of words, I don’t trust that it will also result in them taking actions that reduce x-risk.

  • Many AI safety people currently or formerly worked at AI companies. They stand to make money from accelerating AI capabilities. The same is true of grantmakers

    • I briefly looked thru some grantmakers and I see financial COIs at Open Philanthropy, Survival and Flourishing Fund, and Manifund; but none at Long-Term Future Fund

  • A different sort of conflict of interest: many AI safety researchers have an ML background and enjoy doing ML. Unsurprisingly, they often arrive at the belief that doing ML research is the best way to make AI safe. This ML research often involves making AIs better at stuff. Pausing AI development (or imposing significant restrictions) would mean they don’t get to do ML research anymore. If they oppose a pause/​slowdown, is that for ethical reasons, or is it because it would interfere with their careers?

  • At a moderate P(doom), say under 25%, from a selfish perspective it makes sense to accelerate AI if it increases the chance that you get to live forever, even if it increases your risk of dying. I have heard from some people that this is their motivation. I am appalled at the level of selfishness required to seek immortality at the cost of risking all of humanity. And I’m sure most people who hold this position know it’s appalling, so they keep it secret and publicly give rationalizations for why accelerating AI is actually the right thing to do. In a way, I admire people who are open about their selfishness, and I expect they are the minority.

  But you should know that you might not be able to trust me either.

  • I have some moral uncertainty but my best guess is that future people are just as valuable as present-day people. You might think this leads me to put too much priority on reducing x-risk relative to helping currently-alive people. You might think it’s unethical that I’m willing to delay AGI (potentially hurting currently-alive people) to reduce x-risk.

  • I care a lot about non-human animals, and I believe it’s possible in principle to trade off human welfare against animal welfare. (Although if anything, I think that should make me care less about x-risk, not more.)

  • ETA: I am pretty pessimistic about AI companies’ plans for aligning ASI. My weakly held belief is that if companies follow their current plans, there’s a 2 in 3 chance of a catastrophic outcome. (My unconditional P(doom) is lower than that.) You might believe this makes me too pessimistic about certain kinds of strategies.

  (edit: removed an inaccurate statement)

  [1] ETA: I saw several examples of this on Twitter. Went back and looked and I couldn’t find the examples I recall seeing. IIRC they were mainly quote-tweets, not direct replies, and I don’t know how to find quote-tweets (the search function was unhelpful).

  What links here?

  • AI #114: Liars, Sy­co­phants and Cheaters by Zvi (May 1, 2025, 2:00 PM; 40 points)

• MichaelDickensApr 22, 2025, 4:55 AM

  2 points

  0

  in reply to: tlevin’s comment on: tlevin’s Shortform

  I’ve been doing this for about 10 years. This January I needed to get some new socks but my brand was discontinued so I decided to buy a few different brands and compare them. I will take this opportunity to write a public sock review.

  1. CS CELERSPORT Ankle Athletic Running Socks Low Cut Sports Tab Socks (the black version of the brand you linked): I did not like the wrinkle in the back, and the texture was a bit weird. ⁴⁄₅.

  2. Hanes Men’s Max Cushioned Ankle Socks: Cozy and nice texture, but they made my feet too hot. I might buy these if I lived somewhere colder. ⁴⁄₅.

  3. Hanes Men’s Socks, X-Temp Cushioned No Show Socks: Nice texture, and not too hot. A little tight on the toes which makes it harder for me to wiggle them. These are the ones I decided to go with. 4.5/​5.

• MichaelDickensApr 22, 2025, 1:51 AM

  1 point

  1

  on: Ex­ist­ing Safety Frame­works Im­ply Un­rea­son­able Confidence

  Finally, none of the frameworks make provisions for putting probabilities on anything. The framework itself needn’t include the hard numbers—indeed, such numbers might best be continually updated—but a risk management framework should point to a process that outputs well-calibrated risk estimates for known and unknown pathways before and after mitigation.

  AI companies should publish numerical predictions, open Metaculus questions for each of their predictions, and provide large monetary prizes for predictors.

• MichaelDickensApr 21, 2025, 5:11 PM

  1 point

  0

  in reply to: snewman’s comment on: AI 2027 is a Bet Against Am­dahl’s Law

  Let’s say out of those 200 activities, (for simplicity) 199 would take humans 1 year, and one takes 100 years. If a researcher AI is only half as good as humans at some of the 199 tasks, but 100x better at the human-bottleneck task, then AI can do in 2 years what humans can do in 100.

• MichaelDickensApr 21, 2025, 2:42 PM

  4 points

  1

  on: AI 2027 is a Bet Against Am­dahl’s Law

  Couldn’t the Amdahl’s Law argument work in the opposite direction (i.e. even shorter timelines)?

  Suppose AI R&D conducted by humans would take 100 years to achieve ASI. By Amdahl’s Law, there is likely some critical aspect of research that humans are particularly bad at, that causes the research to take a long time. An SAR might be good at the things humans are bad at (in a way that can’t be fixed by humans + AI working together—human + calculator is much better at arithmetic than human alone, but human + AlphaGo isn’t better at Go). So SAR might be able to get ASI in considerably less than 100 human-equivalent-years.

  It seems to me that, a priori, we should expect Amdahl’s Law to affect humans and SARs to the same degree, so it shouldn’t change our time estimate. Unless there is some specific reason to believe that human researchers are less vulnerable to Amdahl’s Law; I don’t know enough to say whether that’s true.

• MichaelDickensApr 19, 2025, 10:08 PM

  15 points

  33

  in reply to: Tenoke’s comment on: Why Should I As­sume CCP AGI is Worse Than USG AGI?

  1. I strongly suspect that a Trump-controlled AGI would not respect democracy.

  2. I strongly suspect that an Altman-controlled AGI would not respect democracy.

  3. I have my doubts about the other heads of AI companies.

• MichaelDickensApr 19, 2025, 4:26 AM

  1 point

  0

  in reply to: Viliam’s comment on: Three Months In, Eval­u­at­ing Three Ra­tion­al­ist Cases for Trump

  I think some Trump-adjacent people made this mistake (e.g. Peter Thiel maybe), but I don’t see how any of the three people in OP were making it? None of them were trying to rise to positions of power in the Trump administration as far as I know.

• MichaelDickensApr 19, 2025, 4:23 AM

  5 points

  2

  in reply to: Shankar Sivarajan’s comment on: Three Months In, Eval­u­at­ing Three Ra­tion­al­ist Cases for Trump

  That’s a good map; it changed my mind on how seriously to take these sorts of rankings. Do you know the original source? On reverse image search I just found a bunch of reddit posts with hosted copies of the image.

• MichaelDickensApr 19, 2025, 4:11 AM

  8 points

  0

  in reply to: Thane Ruthenis’s comment on: Train­ing AGI in Se­cret would be Un­safe and Unethical

  I think there is a fourth option (although it’s not likely to happen):

  1. Indefinitely pause AI development.

  2. Figure out a robust way to do preference agglomeration.

  3. Encode #2 into law.

  4. Resume AI development (after solving all other safety problems too, of course).

  I was going to say step 2 is “draw the rest of the owl” but really this plan has multiple “draw the rest of the owl” steps.

• MichaelDickensApr 19, 2025, 3:26 AM

  3 points

  0

  on: An AI Race With China Can Be Bet­ter Than Not Racing

  This is a good concept. I built a similar Squiggle model a few weeks ago* (although it’s still a rough draft), I hadn’t realized you’d beaten me to it. So I guess you won the race to build an arms race model? :P

  If I’m reading this right, it looks like the model assumes that if the US doesn’t race, then China gets TAI first with 100% probability. That seems wrong to me. Race dynamics mean that when you go faster, the other party also goes faster. If the US slows down, there’s a good chance China also slows down.

  Also, regarding specific values, the model’s average P(doom) values are:

  • 10% if race + US wins

  • 20% if race + China wins

  • 15% if no race + China wins

  That doesn’t sound right to me. Racing is very bad for safety and right now the US leaders are not going a good job, so I think P(doom | no race & China wins) is less than P(doom | race & US wins). Although I think this is pretty debatable.

  *My model found that racing was bad and I had to really contort the parameter values to reverse that result. I haven’t thought much about the model construction so there could be unfair built-in assumptions.

• MichaelDickensApr 19, 2025, 1:05 AM

  1 point

  0

  in reply to: Yonatan Cale’s comment on: Plan­ning for Ex­treme AI Risks

  I think the right way to self-destruct isn’t to shut down entirely. It’s to spend all your remaining assets on safety (whether that be lobbying for regulations, or research, or whatever). This would greatly increase the total amount of money spent on safety efforts so it might help quite a lot.

  I do believe shutting down does have a decent chance, although not a comfortingly large one, of scaring government and/​or other AI companies into taking the risks seriously.

• MichaelDickensApr 18, 2025, 10:58 PM

  6 points

  2

  in reply to: MichaelDickens’s comment on: What Makes an AI Startup “Net Pos­i­tive” for Safety?

  I think the statement in the parent comment is too general. What I should have said is that every generalist frontier AI company has been net negative. Narrow AI companies that provide useful services and have ~zero chance of accelerating AGI are probably net positive.

• MichaelDickensApr 18, 2025, 9:47 PM

  16 points

  18

  on: What Makes an AI Startup “Net Pos­i­tive” for Safety?

  I thought, “AI safety conscious people will tend to give up too easily while trying to conceive of a net positive alignment startup,”

  This seems like a fairly important premise to your position but I don’t think it’s true. Many safety-conscious people have started for-profit companies. As far as I can tell, every single one of those companies has been net negative. Safety-conscious people are starting too many companies, not too few.

  I’m not confident that there are no net-positive AI startup ideas. But I’m confident that for the median randomly-chosen idea that someone thinks is net positive, it’s actually net negative.