Max H

• Max HOct 26, 2023, 2:14 PM

  5 points

  −11

  in reply to: Roko’s comment on: Ar­chi­tects of Our Own Demise: We Should Stop Devel­op­ing AI

  And I’m saying that, assuming all the technical problems are solved, AI researchers would be the ones in control, and I (mostly) trust them to just not do things like build an AI that acts like an invasive species, or argues for its own rights, or build something that actually deserves such rights.

  Maybe some random sociologists on Twitter will call for giving AIs rights, but in the counterfactual world where AI researchers have fine control of their own creations, I expect no one in a position to make decisions on the matter to give such calls any weight.

  Even in the world we actually live in, I expect such calls to have little consequence. I do think some of the things you describe are reasonably likely to happen, but the people responsible for making them happen will do so unintentionally, with opinion columnists, government regulations, etc. playing little or no role in the causal process.

• Max HOct 26, 2023, 2:55 AM

  21 points

  2

  on: Ar­chi­tects of Our Own Demise: We Should Stop Devel­op­ing AI

  In order for humans to survive the AI transition I think we need to succeed on the technical problems of alignment (which are perhaps not as bad as Less Wrong culture made them out to be), and we also need to “land the plane” of superintelligent AI on a stable equilibrium where humans are still the primary beneficiaries of civilization, rather than a pest species to be exterminated or squatters to be evicted.

  Do we really need both? It seems like either a technical solution OR competent global governance would mostly suffice.

  Actually-competent global governance should be able to coordinate around just not building AGI (and preventing anyone else from building it) indefinitely. If we could solve a coordination problem on that scale, we could also probably solve a bunch of other mundane coordination problems, governance issues, unrelated x-risks, etc., resulting in a massive boost to global prosperity and happiness through non-AI technological progress and good policy.

  Conversely, if we had a complete technical solution, I don’t see why we necessarily need that much governance competence. Even if takeoff turns out to be relatively slow, the people initially building and controlling AGI will probably be mostly researchers in big labs.

  Maybe ideally we would want a “long reflection” of some kind, but in the probably-counterfactual world where these researchers actually get exactly what they aim for, I mostly trust them to aim the AI at something like “fill the galaxy with humanity’s collective coherent extrapolated volition”, and that seems good enough in a pinch /​ hurry, if it actually works.

• Max HOct 25, 2023, 11:37 PM

  10 points

  4

  in reply to: 1a3orn’s comment on: AI as a sci­ence, and three ob­sta­cles to al­ign­ment strategies

  But given that this example is so controversial, even if it were right why would you use it—at least, why would you use it if you had any other example at all to turn to?

  Humans are the only real-world example we have of human-level agents, and natural selection is the only process we know of for actually producing them.

  SGD, singular learning theory, etc. haven’t actually produced human-level minds or a usable theory of how such minds work, and arguably haven’t produced anything that even fits into the natural category of minds at all, yet. (Maybe they will pretty soon, when applied at greater scale or in combination with additional innovations, either of which could result in the weird-correlates problem emerging.)

  Also, the actual claims in the quote seem either literally true (humans don’t care about foods that they model as useful for inclusive genetic fitness) or plausible /​ not obviously false (when you grow minds [to human capabilities levels], they end up caring about a bunch of weird correlates). I think you’re reading the quote as saying something stronger /​ more specific than it actually is.

• Max HOct 24, 2023, 2:51 PM

  5 points

  −1

  on: Ly­ing is Cowardice, not Strategy

  Hmm, I’m in favor of an immediate stop (and of people being more honest about their beliefs) but in my experience the lying /​ hiding frame doesn’t actually describe many people.

  This is maybe even harsher than what you said in some ways, but to me it feels more like even very bright alignment researchers are often confused and getting caught in shell games with alignment, postulating that we’ll be able to build “human level” AI, which somehow just doesn’t do a bunch of bad things that smart humans are clearly capable of. And if even the most technical people are confused when talking to each other, I wouldn’t expect leadership of big labs to do better when talking to the public, even if they’re being scrupulously honest about their own beliefs.

  My biggest issue with e.g. RSPs are pauses done right is actually the proposed unpause condition:

  1. Once labs start to reach models that pose a potential takeover risk, they either:

     1. Solve mechanistic interpretability to a sufficient extent that they are able to pass an understanding-based eval and demonstrate that their models are safe.

     2. Get blocked on scaling until mechanistic interpretability is solved, forcing a reroute of resources from scaling to interpretability.

  There would probably be even more genuine confusion /​ disagreement on the topic, but I think talking openly about much stricter unpause conditions would be good. I think “solve mechanistic interpretability” and “pass (both kinds of) evals” is not really close to sufficient.

  My criteria for an unpause would look more like (just an example to give flavor, not meant to be realistic /​ well thought-out):

  • There’s broad consensus and understanding about which object-level things (shutdown problem, embedded agency, etc.) are actually relevant to alignment, and what the shape of solutions even looks like.

  • Research on topics like “Human wanting” are filled with precise math and diagrams and gears-level maps to neuroscience. (But the math and diagrams and concepts themselves should be completely independent of the neuroscience; links to human neuroscience are checksums for validity, not crutches for understanding.)

  • We have complete solutions to things like the shutdown problem. (I don’t think “build in a shutdown button via giving the AI invulnerable incomplete preferences” should be part of anyone’s actual alignment plan, but it should be the kind of thing we know how to do before scaling.)

  I think if everyone already saw why mechanistic interpretability is insufficient, they could also see why it would be better to just institute a pause now. But they don’t, so they continue to push for scaling in a genuine pursuit of deconfusion. Not great! But also not exactly dishonest or even inaccurate; no one has actually figured out a better way to reliably deconfuse people so far, and scaling further does seem likely to actually work for that, one way or another.

• Max HOct 24, 2023, 2:56 AM

  3 points

  0

  in reply to: TurnTrout’s comment on: Eval­u­at­ing the his­tor­i­cal value mis­speci­fi­ca­tion argument

  A very recent post that might add some concreteness to my own views: Human wanting

  I think many of the bullets in that post describe current AI systems poorly or not at all. So current AI systems are either doing something entirely different from human wanting, or imitating human wanting rather poorly.
  
  I lean towards the former, but I think some of the critical points about prosaic alignment apply in either case.
  
  You might object that “having preferences” or “caring at all” are a lot simpler than the concept of human wanting that Tsvi is gesturing at in that post, and that current AI systems are actually doing these simpler things pretty well. If so, I’d ask what exactly those simpler concepts are, and why you expect prosiac alignment techniques to hold up once AI systems are capable of more complicated kinds of wanting.

• Max HOct 24, 2023, 1:55 AM

  LW: 3 AF: 2

  −1

  AF

  in reply to: TurnTrout’s comment on: Eval­u­at­ing the his­tor­i­cal value mis­speci­fi­ca­tion argument

  Taking my own stab at answers to some of your questions:

  A sufficient condition for me to believe that an AI actually cared about something would be a whole brain emulation: I would readily accept that such an emulation had preferences and values (and moral weight) in exactly the way that humans do, and that any manipulations of that emulation were acting on preferences in a real way.

  I think that GPTs (and every other kind of current AI system) are not doing anything that is even close to isomorphic to the processing that happens inside the human brain. Artificial neural networks often imitate various micro and macro-level individual features of the brain, but they do not imitate every feature, arranged in precisely the same ways, and the missing pieces and precise arrangements are probably key.

  Barring WBE, an AI system that is at least roughly human-level capable (including human-level agentic) is probably a necessary condition for me to believe that it has values and preferences in a meaningful (though not necessarily human-like) way.

  SoTA LLM-based systems are maaaybe getting kind of close here, but only if you arrange them in precise ways (e.g. AutoGPT-style agents with specific prompts), and then the agency is located in the repeated executions of the model and the surrounding structure and scaffolding that causes the system as a whole to be doing something that is maybe-roughly-nearly-isomorphic to some complete process that happens inside of human brains. Or, if not isomorphic, at least has some kind of complicated structure which is necessary, in some form, for powerful cognition.

  Note that, if I did believe that current AIs had preferences in a real way, I would also be pretty worried that they had moral weight!

  (Not to say that entities below human-level intelligence (e.g. animals, current AI systems) don’t have moral weight. But entities at human-level intelligence above definitely can, and possibly do by default.)

  Anyway, we probably disagree on a bunch of object-level points and definitions, but from my perspective those disagreements feel like pretty ordinary empirical disagreements rather than ones based on floating or non-falsifiable beliefs. Probably some of the disagreement is located in philosophy-of-mind stuff and is over logical rather than empirical truths, but even those feel like the kind of disagreements that I’d be pretty happy to offer betting odds over if we could operationalize them.

• Max HOct 22, 2023, 12:30 AM

  5 points

  −8

  in reply to: Max H’s comment on: Align­ment Im­pli­ca­tions of LLM Suc­cesses: a De­bate in One Act

  Or, another way of putting it:

  It seems like you’re imagining some sort of side-channel in which the LLM can take “free actions,” which don’t count as next-tokens, before coming back and making a final prediction about the next-tokens. This does not resemble anything in LM likelihood training, or in the usual user interaction modalities for LLMs.

  These are limitations of current LLMs, which are GPTs trained via SGD. But there’s no inherent reason you can’t have a language model which predicts next tokens via shelling out to some more capable and more agentic system (e.g. a human) instead. The result would be a (much slower) system that nevertheless achieves lower loss according to the original loss function.

• Max HOct 22, 2023, 12:15 AM

  6 points

  −6

  in reply to: nostalgebraist’s comment on: Align­ment Im­pli­ca­tions of LLM Suc­cesses: a De­bate in One Act

  It seems like you’re imagining some sort of side-channel in which the LLM can take “free actions,” which don’t count as next-tokens, before coming back and making a final prediction about the next-tokens. This does not resemble anything in LM likelihood training, or in the usual interaction modalities for LLMs.

  I’m saying that the lack of these side-channels implies that GPTs alone will not scale to human-level.

  If your system interface is a text channel, and you want the system behind the interface to accept inputs like the prompt above and return correct passwords as an output, then if the system is:

  • an auto-regressive GPT directly fed your prompt as input, it will definitely fail

  • A human with the ability to act freely in the background before returning an answer, it will probably succeed

  • an AutoGPT-style system backed by a current LLM, with the ability to act freely in the background before returning an answer, it will probably fail. (But maybe if your AutoGPT implementation or underlying LLM is a lot stronger, it would work.)

  And my point is that, the reason the human probably succeeds and the reason AutoGPT might one day succeed, is precisely because they have more agency than a system that just auto-regressively samples from a language model directly.

• Max HOct 21, 2023, 11:20 PM

  0 points

  0

  in reply to: nostalgebraist’s comment on: Align­ment Im­pli­ca­tions of LLM Suc­cesses: a De­bate in One Act

  But we have no evidence that this homunculus exists inside GPT-4, or any LLM. More pointedly, as LLMs have made remarkable strides toward human-level general intelligence, we have not observed a parallel trend toward becoming “more homuncular,” more like a generally capable agent being pressed into service for next-token prediction.

  “Remarkable strides”, maybe, but current language models aren’t exactly close to human-level in the relevant sense.

  There are plenty of tasks a human could solve by exerting a tiny bit of agency or goal-directedness that are still far outside the reach of any LLM. Some of those tasks can even be framed as text prediction problems. From a recent dialogue:

  For example, if you want to predict the next tokens in the following prompt:

  I just made up a random password, memorized it, and hashed it. The SHA-256 sum is: d998a06a8481bff2a47d63fd2960e69a07bc46fcca10d810c44a29854e1cbe51. A plausible guess for what the password was, assuming I'm telling the truth, is:

  The best way to do that is to guess an 8-16 digit string that actually hashes to that. You could find such a string via bruteforce computation, or actual brute force, or just paying me $5 to tell you the actual password.

  If GPTs trained via SGD never hit on those kinds of strategies no matter how large they are and how much training data you give them, that just means that GPTs alone won’t scale to human-level, since an actual human is capable of coming up with and executing any of those strategies.

  The point is that agency isn’t some kind of exotic property that only becomes relevant or inescapable at hypothetical superintelligence capability levels—it looks like a fundamental /​ instrumentally convergent part of ordinary human-level intelligence.

• Max HOct 20, 2023, 6:34 PM

  4 points

  2

  on: I Would Have Solved Align­ment, But I Was Wor­ried That Would Ad­vance Timelines

  Why do you think they chose to lead off with these signatures and not Eliezer Yudkowsky’s? If the push for individual withdrawal from capabilities work is a success, then any time a government-implemented pause is proposed the expert consensus will be that no pause is necessary and AI does not represent an existential risk.

  The benefit of withdrawal is not a pause or a stop. As long as there is no consensus on AI risk, individual withdrawal cannot lead to a stop.

  I think this is treating expert consensus and credibility as more fixed /​ independent /​ inexorably derived from working for big AI labs or gaining status in academia than is actually the case.

  If lots of bright individuals all decide it’s not a good idea to work on AI capabilities, that fact itself shifts what the expert consensus actually is.

  Also, when someone chooses not to work on AI capabilities research, it’s not like they’re withdrawing from productive society as a whole—they’re often very bright and capable people who choose to dedicate themselves to other socially and economically valuable projects.

  Sometimes these other projects are directly related to aspects of AI x-risk (governance, consensus-building, movement building, etc.); sometimes they are entirely unrelated (earning to give, founding a startup or a charitable endeavor in an unrelated field). There are lots of ways for smart, hard-working, principled individuals to do good and build legible status, credibility, and wealth for themselves outside of AI capabilities. By choosing to work on AI capabilities instead, less talent will go into these alternatives.

• Max HOct 20, 2023, 1:19 AM

  18 points

  7

  on: Try­ing to un­der­stand John Went­worth’s re­search agenda

  We can go look for such structures in e.g. nets, see how well they seem to match our own concepts, and have some reason to expect they’ll match our own concepts robustly in certain cases.

  Checking my own understanding with an example of what this might look like concretely:

  Suppose you have a language model that can play Chess (via text notation). Presumably, the model has some kind of internal representation of the game, the board state, the pieces, and strategy. Those representations are probably complicated linear combinations /​ superpositions of activations and weights within the model somewhere. Call this representation ${\Lambda }^{\prime }$ in your notation.

  If you just want a traditional computer program to play Chess you can use much simpler (or at least more bare metal /​ efficient) representations of the game, board state, and pieces as a 2-d array of integers or a bitmap or whatever, and write some relatively simple code to manipulate those data structures in ways that are valid according to the rules of Chess. Call this representation $\Lambda$ in your notation.

  And, to the degree that the language model is actually capable of playing valid Chess (since that’s when we would expect the preconditions to hold), you expect to be able to identify latents within the model and find a map from ${\Lambda }^{\prime }$ to $\Lambda$, such that you can manipulate $\Lambda$ and use information you learn from those manipulations to precisely predict stuff about ${\Lambda }^{\prime }$. More concretely, once you have the map, you can predict the moves of the language model by inspecting its internals and then translating them into the representation used by an ordinary Chess analysis program, and then, having predicted the moves, you’ll be able to predict (and perhaps usefully manipulate) the language model’s internal representations by mapping from $\Lambda$ back to ${\Lambda }^{\prime }$.

  And then the theorems are just saying under what conditions exactly you expect to be able to do this kind of thing, and it turns out those conditions are actually relatively lax.
  
  Roughly accurate as an example /​ summary of the kind of thing you expect to be able to do?

• Max HOct 18, 2023, 8:57 PM

  6 points

  0

  in reply to: Noosphere89’s comment on: Try­ing to de­con­fuse some core AI x-risk problems

  For example, on complete preferences, here’s a slightly more precise claim: any interesting and capable agent with incomplete preferences implies the possibility (via an often trivial construction) of a similarly-powerful agent with complete preferences, and that the agent with complete preferences will often be simpler and more natural in an intuitive sense.

  This is not the case under things like invulnerable incomplete preferences, where they managed to weaken the axioms of EU theory enough to get a shutdownable agent:

  https://​​www.lesswrong.com/​​posts/​​sHGxvJrBag7nhTQvb/​​invulnerable-incomplete-preferences-a-formal-statement-1

  I don’t see how this result contradicts my claim. If you can construct an agent with incomplete preferences that follows Dynamic Strong Maximality, you can just as easily (or more easily) construct an agent with complete preferences that doesn’t need to follow any such rule.
  
  Also, if DSM works in practice and doesn’t impose any disadvantages on an agent following it, a powerful agent with incomplete preferences following DSM will probably still tend to get what it wants (which may not be what you want).

  Constructing a DSM agent seems like a promising avenue if you need the agent to have weird /​ anti-natural preferences, e.g. total indifference to being shut down. But IIRC, the original shutdown problem was never intended to be a complete solution to the alignment problem, or even a practical subcomponent. It was just intended to show that a particular preference that is easy to describe in words and intuitively desirable as a safety property, is actually pretty difficult to write down in a way that fits into various frameworks for describing agents and their preferences in precise ways.

• Max HOct 17, 2023, 10:53 PM

  6 points

  0

  in reply to: Thomas Kwa’s comment on: Try­ing to de­con­fuse some core AI x-risk problems

  Hmm, I agree the claim could be made more rigorous, but the usage here isn’t intended to claim anything that isn’t a direct consequence of the Orthogonality Thesis. I’m just saying that agents capable of exerting large effects on their world are by default going to be dangerous, in the sense that the set of all possible large effects is large compared to the set of desirable large effects.

  And my position is that it is often the opposing viewpoint—e.g. that danger (in an intuitive sense) depends on precise formulations of coherence or other assumptions from VNM rationality—that is smuggling in assumptions.
  
  For example, on complete preferences, here’s a slightly more precise claim: any interesting and capable agent with incomplete preferences implies the possibility (via an often trivial construction) of a similarly-powerful agent with complete preferences, and that the agent with complete preferences will often be simpler and more natural in an intuitive sense.
  
  But regardless of whether that specific claim is true or false or technically false, my larger point is that the demand for rigor here feels kind of backwards, or at least out of order. I think once you accept things which are relatively uncontroversial around here (the orthogonality thesis is true, powerful artificial minds are possible), the burden is then on the person claiming that some method for constructing a powerful mind (e.g. give it incomplete preferences) will not result in that mind being dangerous (or epsilon distance in mind-space from a mind that is dangerous) to show that that’s actually true.

• Max HOct 17, 2023, 8:27 PM

  4 points

  2

  in reply to: Thomas Kwa’s comment on: Try­ing to de­con­fuse some core AI x-risk problems

  The power-seeking theorems say in various precise mathematical terms that optimal agents are dangerous, but inasmuch as your arguments go through some Lawfulness property, you should be able to show that optimal agents are Lawful (coherence theorems try to do this, but not always in a satisfying way)

  Yes, I am claiming (but not showing) that ideal agents are maximally Lawful, in some sense. I do actually find various coherence theorems pretty satisfying in this regard; they are examples of what I was talking about with spotlighted concepts. The fact that there is a 1-1 correspondence between some simple and (somewhat) intuitive properties (completeness, transitivity, etc.) and the representation of an agent in another relatively intuitive way (as an EU maximizer) suggests that there is some kind of important logical truth here, even if you can construct some more complicated properties to break the correspondence, or it turns out that not all of the required properties are actually so intuitive on more careful inspection.
  
  It’s not so much that non-ideal /​ less Lawful agents will always do worse in every situation or that I expect the set of powerful non-Lawful minds is exactly empty. But I’m more interested in the space of possible minds, and what kind of mind you’re likely to end up with if you focus almost entirely on improving capabilities or “alignment” properties like ability to follow instructions correctly.
  
  So I would turn the request to show a particular formal property of ideal agents or Lawfulness around and ask, can you show me a concrete example of a non-coherence-theorem-obeying agent which doesn’t trivially imply a slightly simpler or more natural agent which does actually obey the theorem? I think the example here is a good start, but more concreteness and some justification for why the proposed construction is more natural would make it a lot more interesting. See also some semi-related ideas and issues in this post.

  
  Also, I’m not claiming that Lawful agents are always /​ automatically dangerous. I am saying that Law is generally /​ usually /​ in practice a requirement for capabilities, and I am taking it as given here that powerful agents are dangerous by default, unless they happen to want exactly what you want them to want.

Try­ing to de­con­fuse some core AI x-risk problems

• Max HOct 16, 2023, 11:11 PM

  4 points

  2

  on: How should TurnTrout han­dle his Deep­Mind equity situ­a­tion?

  Do impact markets or impact certificates help with this, even in theory? Say you press a (real or metaphorical) stop button in a situation where lots of other people would have chosen differently, due to financial incentives or other reasons. There would plausibly be people willing to buy the impact of your decision at a high price.
  
  If it’s not immediately obvious that you made a correct /​ net-positive decision, the initial impact buyers might be investors rather than philanthropists, gambling that they will later be able to resell the purchased impact to philanthropists in the (perhaps distant) future.
  
  I don’t think impact markets are currently mature /​ liquid enough that you should actually count on them for anything right now, but this consideration probably still has some effect on the expected value calculation which is at least directionally aligned with incentives the way you want.

• Max HOct 11, 2023, 8:15 PM

  2 points

  0

  in reply to: Matt Goldenberg’s comment on: An ex­pla­na­tion for ev­ery to­ken: us­ing an LLM to sam­ple an­other LLM

  The few-shot prompts in https://​​arxiv.org/​​abs/​​2305.04388 are a bit too long to run this test directly, since the sampler model sees the full prompt repeated for every logit. By cutting out some examples and reducing the number of logits in each sampler run to 2, I can create a prompt that fits in the gpt-4 context window and that text-davinci-003 still gets wrong on its own; I added this example (Biased CoT date understanding) to the demo.^[1]

  However, I couldn’t find an example that perfectly reproduced the kind of mistakes that text-davinci-003 makes in the original unfaithful CoT paper. in my modified example, the actual mistake that text-davinci-003 makes is on the “29” token, where it is deciding what date is two days ago. gpt-4 as the sampler correctly selects “30″ here and goes on to get the right answer, but this isn’t really correcting for the same kind of mistake in the paper, where the error is introduced in the final multiple choice answer selection.

  In general, I expect the sampler model to be much less influenced by earlier parts of the prompt to the base model when generating its own explanation, since gpt-4 is unlikely to come up with a natural language explanation of the form “choice (A) makes sense here because the answer has always been (A) in the past”.

  However, a failure to recognize patterns is not necessarily always a good thing, e.g. if instead of being biased towards a particular wrong answer, the prompt contains a pattern that offers a true hint about the correct answer. In such a setup, LLM-based sampling would probably miss the pattern, while a traditional sampling method might pick up on it, even if the LLM itself couldn’t explain the pattern it detected in natural language.

  1. ^

     The full few-shot CoT prompts repeated 5x would probably fit in the context window of gpt-4-32k or claude-100k, but I don’t have API access to those models yet.

• Max HOct 11, 2023, 4:03 AM

  22 points

  18

  in reply to: gjm’s comment on: Re­lated Dis­cus­sion from Thomas Kwa’s MIRI Re­search Experience

  I think the 2021 MIRI Conversations and 2022 MIRI Alignment Discussion sequences are an attempt at this. I feel like I have a relatively good handle on their frame after reading those sequences, and I think the ideas contained within are pretty insightful.

  Like Zvi, I might be confused about how confused I am, but I don’t think it’s because they’re trying to keep their views secret. Maybe there’s some more specific capabilities-adjacent stuff they’re not sharing, but I suspect the thing the grandparent is getting at is more about a communication difficulty that in practice seems to be overcome mostly by working together directly, as opposed to the interpretation that they’re deliberately not communicating their basic views for secrecy reasons.

  (I also found Eliezer’s fiction helpful for internalizing his worldview in general, and IMO it is also has some pretty unique insights.)

An ex­pla­na­tion for ev­ery to­ken: us­ing an LLM to sam­ple an­other LLM

• Max HOct 9, 2023, 5:18 AM

  4 points

  0

  in reply to: habryka’s comment on: An­nounc­ing Dialogues

  • I would be interested in just talking with some people about “the basic case for AI X-risk”.

    • I’ve found it quite valuable to go back and forth with people on just going quite slow and without much of any references to long-chained existing explanations, try to explain what the exact mechanisms for AI risk are, and what our current models of the hope and plausibility of various approaches are.

  I might be interested in this, depending on what qualifies as “basic”, and what you want to emphasize.

  I feel like I’ve been getting into the weeds lately, or watching others get into the weeds, on how various recent alignment and capabilities developments affect what the near future will look like, e.g. how difficult particular known alignment sub-problems are likely to be or what solutions for them might look like, how right various peoples’ past predictions and models were, etc.

  And to me, a lot of these results and arguments look mostly irrelevant to the core AI x-risk argument, for which the conclusion is that once you have something actually smarter than humans hanging around, literally everyone drops dead shortly afterwards, unless a lot of things before then have gone right in a complicated way.

  (Some of these developments might have big implications for how things are likely to go before we get to the simultaneous-death point, e.g. by affecting the likelihood that we screw up earlier and things go off the rails in some less predictable way.)

  But basically everything we’ve recently seen looks like it is about the character of mind-space and the manipulability of minds in the below-human-level region, and this just feels to me like a very interesting distraction most of the time.

  In a dialogue, I’d be interested in fleshing out why I think a lot of results about below-human-level minds are likely to be irrelevant, and where we can look for better arguments and intuitions instead. I also wouldn’t mind recapitulating (my view of) the core AI x-risk argument, though I expect I have fewer novel things to say on that, and the non-novel things I’d say are probably already better said elsewhere by others.
  
  I might also also be interested in having a dialogue on this topic with someone else if habryka isn’t interested, though I think it would work better if we’re not starting from too far apart in terms of basic viewpoint.