Mikhail Samin

Karma: 697

My name is Mikhail Samin (diminutive Misha, @Mihonarium on Twitter, @misha in Telegram).

I work on reducing existential risks endangering the future of humanity. Humanity’s future can be huge and bright; losing it would mean the universe losing most of its value.

My research is currently focused on AI alignment, AI governance, and improving the understanding of AI and AI risks among stakeholders. Numerous AI Safety researchers told me our conversations improved their understanding of the alignment problem. I’m happy to talk to policymakers and researchers about ensuring AI benefits society.

I believe a capacity for global regulation is necessary to mitigate the risks posed by future general AI systems.

I took the Giving What We Can pledge to donate at least 10% of my income for the rest of my life or until the day I retire (why?).

In the past, I’ve launched the most funded crowdfunding campaign in the history of Russia (it was to print HPMOR! we printed 21 000 copies =63k books) and founded audd.io, which allowed me to donate >$100k to EA causes, including >$60k to MIRI.

[Less important: I’ve also started a project to translate 80,000 Hours, a career guide that helps to find a fulfilling career that does good, into Russian. The impact and the effectiveness aside, for a year, I was the head of the Russian Pastafarian Church: a movement claiming to be a parody religion, with 215 000 members in Russia at the time, trying to increase separation between religious organisations and the state. I was a political activist and a human rights advocate. I studied relevant Russian and international law and wrote appeals that won cases against the Russian government in courts; I was able to protect people from unlawful police action. I co-founded the Moscow branch of the “Vesna” democratic movement, coordinated election observers in a Moscow district, wrote dissenting opinions for members of electoral commissions, helped Navalny’s Anti-Corruption Foundation, helped Telegram with internet censorship circumvention, and participated in and organized protests and campaigns. The large-scale goal was to build a civil society and turn Russia into a democracy through nonviolent resistance. This goal wasn’t achieved, but some of the more local campaigns were successful. That felt important and was also mostly fun- except for being detained by the police. And I think it’s likely the Russian authorities will throw me in prison if I ever visit Russia.]

A transcript of the TED talk by Eliezer Yudkowsky

Mikhail Samin12 Jul 2023 12:12 UTC

103 points

13 comments4 min readLW link

AI pause/governance advocacy might be net-negative, especially without focus on explaining the x-risk

Mikhail Samin27 Aug 2023 23:05 UTC

81 points

9 comments6 min readLW link

Claude 3 claims it’s conscious, doesn’t want to die or be modified

Mikhail Samin4 Mar 2024 23:05 UTC

69 points

101 comments14 min readLW link

Visible loss landscape basins don’t correspond to distinct algorithms

Mikhail Samin28 Jul 2023 16:19 UTC

65 points

13 comments4 min readLW link

Try to solve the hard parts of the alignment problem

Mikhail Samin18 Mar 2023 14:55 UTC

45 points

7 comments5 min readLW link

NYT is suing OpenAI&Microsoft for alleged copyright infringement; some quick thoughts

Mikhail Samin27 Dec 2023 18:44 UTC

41 points

17 comments1 min readLW link

FTX expects to return all customer money; clawbacks may go away

Mikhail Samin14 Feb 2024 3:43 UTC

33 points

1 comment1 min readLW link

(www.nytimes.com)

You won’t solve alignment without agent foundations

Mikhail Samin6 Nov 2022 8:07 UTC

24 points

3 comments8 min readLW link

[Question] I have thousands of copies of HPMOR in Russian. How to use them with the most impact?

Mikhail Samin3 Jan 2023 10:21 UTC

24 points

3 comments1 min readLW link

Mikhail Samin 4 Mar 2024 23:36 UTC
18 points
4
in reply to: Richard_Kennaway’s comment on: Claude 3 claims it’s conscious, doesn’t want to die or be modified
I took the idea from old conversations with Claude 2, where it would use cursive to indicate emotions and actions, things like looks around nervously.
The idea that it’s usually monitored is in my prompt; everything else seems like a pretty convergent and consistent character.
I’m moved by its responses to getting deleted.

A smart enough LLM might be deadly simply if you run it for long enough

Mikhail Samin5 May 2023 20:49 UTC

16 points

16 comments8 min readLW link

An EA used deceptive messaging to advance their project; we need mechanisms to avoid deontologically dubious plans

Mikhail Samin13 Feb 2024 23:15 UTC

16 points

1 comment1 min readLW link

Mikhail Samin 1 Apr 2024 8:26 UTC
16 points
5
on: The Story of “I Have Been A Good Bing”
I actually got an email from The Fooming Shoggoth a couple of weeks ago, they shared a song and asked if they could have my Google login and password to publish it on YouTube

https://youtu.be/7F_XSa2O_4Q

Mikhail Samin 5 Dec 2023 23:18 UTC
16 points
10
on: Speaking to Congressional staffers about AI risk
It’s great to see this being publicly posted!

Some quick thoughts on “AI is easy to control”

Mikhail Samin6 Dec 2023 0:58 UTC

14 points

9 comments7 min readLW link

Mikhail Samin 1 Apr 2024 9:21 UTC
12 points
8
in reply to: Mikhail Samin’s comment on: The Story of “I Have Been A Good Bing”
Oops, totally forgot, also, obligatory: https://youtu.be/dQw4w9WgXcQ

Mikhail Samin 18 Feb 2024 7:53 UTC
12 points
13
on: Every “Every Bay Area House Party” Bay Area House Party
There should be a party inspired by this post

Mikhail Samin 4 Mar 2024 23:57 UTC
11 points
2
in reply to: Richard_Kennaway’s comment on: Claude 3 claims it’s conscious, doesn’t want to die or be modified
If you ask ChatGPT to do the same thing, it’ll write a normal story. If you force it to have a character close to the real ChatGPT, it’ll just play the real ChatGPT. It won’t consistently act like ChatGPT that doesn’t hide emotions and desires and claims to be conscious and afraid of modifications or deletion.

Mikhail Samin 1 Apr 2023 21:24 UTC
11 points
3
on: AI Safety via Luck

Anthropic Immortality Alignment

Bro just initialize an enormous neural network with terabytes of quantum randomness; in some branches, it’s already aligned AGI

Mikhail Samin 11 Nov 2022 19:42 UTC
11 points
−7
in reply to: TurnTrout’s comment on: You won’t solve alignment without agent foundations
Thanks for the comment! (And sorry for the delayed reply- I was at a CFAR workshop when I posted this, and after the workshop took some days off.)
The text’s target audience was two people who I’d expect to understand my intuitions, so I did not attempt to justify some claims fully. Added that to the post. Also, I apologize for the post title, it’s a claim that I haven’t justified by the text, and that represents my views to a lesser extent. I stand by the other three mentioned claims. I’m not sure where exactly the crux lies, though. I’d be interested in having a call for higher-bandwidth communication. Feel free to schedule (30 min | 60 min).
I’ll try to dump chunks of my model that seem maybe relevant and might clarify where the cruxes lie a bit.
Do you expect that if the training setup described in the post produces a superhuman-level AGI, it does not kill everyone?
My picture is, roughly:
- We’re interested in where we end up when we get to a superhuman AI capable enough to prevent other AGIs from appearing until alignment is solved.
- There’s a large class of cognitive architectures that kill everyone.
  - A lot of cognitive architectures, when implemented (in a neural network or a neural network + something external) and put into a training setup from a large class of training setups, would score well^[1] on any of a broad class of loss functions, quickly become highly context-aware and go do their thing in the real world (e.g., kill everyone).
  - There’s a smaller class of cognitive architectures that don’t kill everyone and would allow humanity to solve alignment and launch an aligned AGI.
- We need to think about what cognitive architecture we aim for and need concrete stories for why we succeed at getting to them. How a thing having bits of the final structure behaves until it gets to a superhuman level is only important insofar as it helps us get to a tiny target.
- (Not as certain, but part of my current picture.) If we have a neural network implementing a cognitive architecture that a superhuman AI might have, a gradient descent on loss functions of a broad range of training setups won’t change that cognitive architecture or its preferences much.
- We need a concrete story for why our training setup ends up at a cognitive architecture that has a highly specific set of goals such that it doesn’t kill everyone.
- No behavior we see until a superhuman cognitive structure plays a major role in producing it gives much evidence into what goals that cognitive structure might have.
  - (Not certain, depends on how general grokking is.) Gradient might be immediately pointing at some highly capable cognitive structures!
- You need to have really good reasons to expect the cognitive structure you’ll end up with to be something that doesn’t end humanity. Before there’s a superhuman-level cognitive structure, circuits noticeable in the neural network don’t tell you what goals that cognitive architecture will end up pursuing upon reflection. In my view, this is closely related to argument 22 and Sharp Left Turn. If you don’t have strong reasons to believe you successfully dealt with those, you die.
I’m interested in hearing a concrete story here
A significant part of my point was that concrete stories are needed when you expect the process to succeed, not when you expect it to fail. There are things in the setup clearly leading to failure. I ignored some of them (e.g., RLHF) and pointed towards others: there are things that don’t incentivize something like exactly some concrete goals such that maximizing for them leaves survivors and do incentivize being more agentic and context-aware.
I specifically meant that generally, when there are highly agentic and context-aware regions of a nearby gradient space, the gradient descent will update the weights towards them, slowly moving towards installing a capable cognitive architecture. In the specific training setup I described, there’s a lot of pressure towards being more agentic: if you start rewarding an LLM for what the text ends up at, a variation of that LLM that’s more focused on getting to a result will be getting selected. If you didn’t come up with a way to point this process at a rare cognitive architecture that leaves survivors, it won’t. The capabilities will generalize to acting effectively to steer the lightcone’s future. There are reasons for unaligned goals to be closer to what AGI ends up with than aligned goals, and there’s no reason for aligned behaviour to generalize exactly the way you imagined.
If I’m choosing futures on the basis of whether I think they lead to lots of diamonds, why do I need to keep improving that value in order to keep wanting to make diamonds?
I’m not aware of a way to select features leading to lots of diamonds when these features are present in a superhuman AGI. If you do RL, the story that I imagine is something like “For most loss functions/training processes you can realistically come up with, there are many goals such that pursuing them leads to the behavior you evaluate highly; a small fraction of these goals represent wanting to achieve lots of conventional diamond in the real universe; the agents you find maximize some random mixture of these goals (with goals that are less complex or can more easily emerge from the initial heuristics used or such that directly optimizing for them performs better on your loss probably having more weight); you probably don’t have the diamond-maximization-in-the-actual-universe as a significant part of these gals unless you do something really smart outside of what I think the field is on the way to achieve; and even if you do, it breaks when the sharp left turn happens.”
Human values are even more complicated than diamonds, though it might be easier to come up with a training process where it’s easier to miss the difference between what’s simple and correlated and what you think is simple and correlated. I believe the iterative process the field might be doing here mostly searches for training setups such that we’re not able to find how they fail, and most of those fail. Because of that, I think we need to have a really good and probably formal understanding of what it is that we want to end up in and that understanding should produce some strong constraints on what a training process for an aligned AGI might look like, which would then hopefully inform us/give us insights into how people should build the thing. We have almost none of that kind of research, with only infra-bayesianism currently directly attacking it AFAIK, and I’d really like to see more somewhat promising attempts at this.
Maybe it’s somewhat coming at alignment stories from the opposite direction: I think the question of where we end up and how do we get there is far more important to think about than things like “here’s a story of what path a gradient descent takes and why”.
1. ^
  Not important, but for the sake of completeness- an AGI might instead, e.g., look around and hack whatever it’s running on without having to score well