Mikhail Samin

Won­der about the hard parts of the al­ign­ment problem

• Mikhail Samin 16 Mar 2023 11:10 UTC

  6 points

  1

  in reply to: baturinsky’s comment on: Won­der about the hard parts of the al­ign­ment problem

  I don’t expect everyone to disregard the danger; I do expect most people building capable AI systems to continue to hide hard problems. Hiding the hard problems is much easier than solving them, but I guess produces plausible-sounding solutions just as well.

  Roughly human level humans don’t contribute significantly to AI alignment research and can’t be pivotally used. So I don’t think you think that a roughly human level AI system can contribute significantly to AI alignment research. Maybe you (as many seem to) think that if someone runs not-that-superhuman language models with clever prompt engendering, fine-tuning, and systems around, than the whole system can solve alignment or be pivotally used, and the point of the post is that the whole system is superhuman, not roughly human-level, if it’s capable enough to solve alignment or be pibotally used, and you need to direct the whole system somewhere, and unless you made the whole system optimize for something you actually want, it probably kills you before it solves alignment.

• Mikhail Samin 13 Feb 2023 20:20 UTC

  1 point

  0

  in reply to: paulfchristiano’s comment on: Re­view of AI Align­ment Progress

  In RLHF, the gradient descent will steer the model towards being more agentic about the entire output (and, speculatively, more context-aware), because that’s the best way to produce a token on a high-reward trajectory. The lowest loss is achievable with a superintelligence that thinks about a sequence of tokens that would be best at hacking human brains (or a model of human brains) and implements it token by token.

  That seems quite different from a GPT that focuses entirely on predicting the current token and isn’t incentivized to care about the tokens that come after the current one outside of what the consequentialists writing (and selecting, good point) the text would be caring about. At the lowest loss, the GPT doesn’t use much more optimization power than what the consequentialists plausibly had/​used.

  I have an intuition about a pretty clear difference here (and have been quite sure that RL breaks myopia for a while now) and am surprised by your expectation for myopia to be preserved. RLHF means optimizing the entire trajectory to get a high reward, with every choice of every token. I’m not sure where the disagreement comes from; I predict that if you imagine fine-tuning a transformer with RL on a game where humans always make the same suboptimal move, but they don’t see it, you would expect the model, when it becomes smarter and understands the game well enough, to start picking instead a new move that leads to better results, with the actions selected for what results they produce in the end. It feels almost tautological: if the model sees a way to achieve a better long-term outcome, it will do that to score better. The model will be steered towards achieving predictably better outcomes in the end. The fact that it’s a transformer that individually picks every token doesn’t mean that RL won’t make it focus on achieving a higher score. Why would the game being about human feedback prevent a GPT from becoming agentic and using more and more of its capabilities to achieve longer-term goals?

  (I haven’t read the “conditioning generative models“ sequence, will probably read it. Thank you)

• Mikhail Samin 13 Feb 2023 12:44 UTC

  2 points

  −9

  in reply to: paulfchristiano’s comment on: Re­view of AI Align­ment Progress

  I’m surprised. It feels to me that there’s an obvious difference between predicting one token of text from a dataset and trying to output a token in a sequence with some objective about the entire sequence.

  RLHF models optimize for the entire output to be rated highly, not just for the next token, so (if they’re smart enough) they perform better if they think what current tokens will make it easier for the entire output to be rated highly (instead of outputting just one current token that a human would like).

• Mikhail Samin 7 Feb 2023 15:30 UTC

  1 point

  0

  on: Mikhail Samin’s Shortform

  [RETRACTED after Scott Aaronson’s reply by email]

  I’m surprised by Scott Aaronson’s approach to alignment. He has mentioned in a talk that a research field needs to have at least one of two: experiments or a rigorous mathematical theory, and so he’s focusing on the experiments that are possible to do with the current AI systems.

  The alignment problem is centered around optimization producing powerful consequentialist agents appearing when you’re searching in spaces with capable agents. The dynamics at the level of superhuman general agents are not something you get to experiment with (more than once); and we do indeed need a rigorous mathematical theory that would describe the space and point at parts of it that are agents aligned with us.

  [removed]

  I’m disappointed that, currently, only Infra-Bayesianism tries to achieve that^[1], that I don’t see dozens of other research directions trying to have a rigorous mathematical theory that would provide desiderata for AGI training setups, and that even actual scientists entering the field [removed].

  1. ^

     Infra-Bayesianism is an approach that tries to describe agents in a way that would closely resemble the behaviour of AGIs, starting with a way you can model them having probabilities about the world in a computable way that solves non-realizability in RL (short explanation, a sequence with equations and proofs) and making decisions in a way that optimization processes would select for, and continuing with a formal theory of naturalized induction and, finally, a proposal for alignment protocol.

     To be clear, I don’t expect Infra-Bayesianism to produce an answer to what loss functions should be used to train an aligned AGI in the time that we have remaining; but I’d expect that if there were a hundred research directions like that, trying to come up with a rigorous mathematical theory that successfully attacks the problem, with thousands of people working on them, some would succeed.

Mikhail Samin’s Shortform

• Mikhail Samin 2 Feb 2023 16:01 UTC

  1 point

  0

  on: What I mean by “al­ign­ment is in large part about mak­ing cog­ni­tion aimable at all”

  Oh. I was thinking the diamond maximizer problem is about making the AI care about the specific goal of maximizing diamond, not about making the AI have some consistent goal slot instead of lots of cognitive spaghetti code. I think it’s simple to make a highly agentic AI with some specific goals it’s really good at maximizing (if you have infinite compute and don’t care about what these goals actually are; I have no idea how to point that at diamond). Should I write a description somewhere?

[Question] I have thou­sands of copies of HPMOR in Rus­sian. How to use them with the most im­pact?

• Mikhail Samin 11 Nov 2022 19:42 UTC

  11 points

  −7

  in reply to: TurnTrout’s comment on: You won’t solve al­ign­ment with­out agent foundations

  Thanks for the comment! (And sorry for the delayed reply- I was at a CFAR workshop when I posted this, and after the workshop took some days off.)

  The text’s target audience was two people who I’d expect to understand my intuitions, so I did not attempt to justify some claims fully. Added that to the post. Also, I apologize for the post title, it’s a claim that I haven’t justified by the text, and that represents my views to a lesser extent. I stand by the other three mentioned claims. I’m not sure where exactly the crux lies, though. I’d be interested in having a call for higher-bandwidth communication. Feel free to schedule (30 min | 60 min).

  I’ll try to dump chunks of my model that seem maybe relevant and might clarify where the cruxes lie a bit.

  Do you expect that if the training setup described in the post produces a superhuman-level AGI, it does not kill everyone?

  My picture is, roughly:

  • We’re interested in where we end up when we get to a superhuman AI capable enough to prevent other AGIs from appearing until alignment is solved.

  • There’s a large class of cognitive architectures that kill everyone.

    • A lot of cognitive architectures, when implemented (in a neural network or a neural network + something external) and put into a training setup from a large class of training setups, would score well^[1] on any of a broad class of loss functions, quickly become highly context-aware and go do their thing in the real world (e.g., kill everyone).

    • There’s a smaller class of cognitive architectures that don’t kill everyone and would allow humanity to solve alignment and launch an aligned AGI.

  • We need to think about what cognitive architecture we aim for and need concrete stories for why we succeed at getting to them. How a thing having bits of the final structure behaves until it gets to a superhuman level is only important insofar as it helps us get to a tiny target.

  • (Not as certain, but part of my current picture.) If we have a neural network implementing a cognitive architecture that a superhuman AI might have, a gradient descent on loss functions of a broad range of training setups won’t change that cognitive architecture or its preferences much.

  • We need a concrete story for why our training setup ends up at a cognitive architecture that has a highly specific set of goals such that it doesn’t kill everyone.

  • No behavior we see until a superhuman cognitive structure plays a major role in producing it gives much evidence into what goals that cognitive structure might have.

    • (Not certain, depends on how general grokking is.) Gradient might be immediately pointing at some highly capable cognitive structures!

  • You need to have really good reasons to expect the cognitive structure you’ll end up with to be something that doesn’t end humanity. Before there’s a superhuman-level cognitive structure, circuits noticeable in the neural network don’t tell you what goals that cognitive architecture will end up pursuing upon reflection. In my view, this is closely related to argument 22 and Sharp Left Turn. If you don’t have strong reasons to believe you successfully dealt with those, you die.

  I’m interested in hearing a concrete story here

  A significant part of my point was that concrete stories are needed when you expect the process to succeed, not when you expect it to fail. There are things in the setup clearly leading to failure. I ignored some of them (e.g., RLHF) and pointed towards others: there are things that don’t incentivize something like exactly some concrete goals such that maximizing for them leaves survivors and do incentivize being more agentic and context-aware.

  I specifically meant that generally, when there are highly agentic and context-aware regions of a nearby gradient space, the gradient descent will update the weights towards them, slowly moving towards installing a capable cognitive architecture. In the specific training setup I described, there’s a lot of pressure towards being more agentic: if you start rewarding an LLM for what the text ends up at, a variation of that LLM that’s more focused on getting to a result will be getting selected. If you didn’t come up with a way to point this process at a rare cognitive architecture that leaves survivors, it won’t. The capabilities will generalize to acting effectively to steer the lightcone’s future. There are reasons for unaligned goals to be closer to what AGI ends up with than aligned goals, and there’s no reason for aligned behaviour to generalize exactly the way you imagined.

  If I’m choosing futures on the basis of whether I think they lead to lots of diamonds, why do I need to keep improving that value in order to keep wanting to make diamonds?

  I’m not aware of a way to select features leading to lots of diamonds when these features are present in a superhuman AGI. If you do RL, the story that I imagine is something like “For most loss functions/​training processes you can realistically come up with, there are many goals such that pursuing them leads to the behavior you evaluate highly; a small fraction of these goals represent wanting to achieve lots of conventional diamond in the real universe; the agents you find maximize some random mixture of these goals (with goals that are less complex or can more easily emerge from the initial heuristics used or such that directly optimizing for them performs better on your loss probably having more weight); you probably don’t have the diamond-maximization-in-the-actual-universe as a significant part of these gals unless you do something really smart outside of what I think the field is on the way to achieve; and even if you do, it breaks when the sharp left turn happens.”

  Human values are even more complicated than diamonds, though it might be easier to come up with a training process where it’s easier to miss the difference between what’s simple and correlated and what you think is simple and correlated. I believe the iterative process the field might be doing here mostly searches for training setups such that we’re not able to find how they fail, and most of those fail. Because of that, I think we need to have a really good and probably formal understanding of what it is that we want to end up in and that understanding should produce some strong constraints on what a training process for an aligned AGI might look like, which would then hopefully inform us/​give us insights into how people should build the thing. We have almost none of that kind of research, with only infra-bayesianism currently directly attacking it AFAIK, and I’d really like to see more somewhat promising attempts at this.

  Maybe it’s somewhat coming at alignment stories from the opposite direction: I think the question of where we end up and how do we get there is far more important to think about than things like “here’s a story of what path a gradient descent takes and why”.

  1. ^

     Not important, but for the sake of completeness- an AGI might instead, e.g., look around and hack whatever it’s running on without having to score well

You won’t solve al­ign­ment with­out agent foundations

• Mikhail Samin 2 Oct 2021 21:37 UTC

  5 points

  on: Whole Brain Emu­la­tion: No Progress on C. el­gans After 10 Years

  There’s a scan of 1 mm^3 of a human brain, 1.4 petabytes with hundred(s?) of millions of synapses

  ‪https://​​ai.googleblog.com/​​2021/​​06/​​a-browsable-petascale-reconstruction-of.html‬