Two existing suggestions for how to avoid existential risk naturally fall out of this framing.
Go all the way to the left (even further than the picture implies) by giving the AI no output channels whatsoever. This is Microscope AI.
Go all the way to the bottom and avoid all agent-like systems, but allow autonomous systems like self-driving cars. This is (as I understand it) Comprehensive AI Services (CAIS).
I recently realized that I’ve been confused about an extremely basic concept: the difference between an Oracle and an autonomous agent.
This feels obvious in some sense. But actually, you can ‘get’ to any AI system via output behavior + robotics. If you can answer arbitrary questions, you can also answer the question ‘what’s the next move in this MDP’, or less abstractly, ‘what’s the next steering action of the imaginary wheel’ (for a self-driving car). And the difference can’t be ‘an autonomous agent has a robotic component’.
The essential difference seems to be that the former system only uses its output channels whenever it is probed, whereas the second uses them autonomously. But I don’t ever hear people make this distinction. I think part of the reason why I hadn’t internalized this as an axis before is that there is the agent vs. nonagent thing, but actually, those are orthogonal to each other. We clearly can have any of the four combinations of {agent, nonagent} {autonomous, non-autonomous}.[1]
It’s a pretty bad sign that I don’t know without looking at the definition whether ‘tool AI’ refers to the entire bottom half or just the bottom-left quadrant. With looking, it seems to be just the latter.
What led me to this was thinking about Corrigibility. I think it is applicable to the entire top half, all agent-like systems, but it feels like a stronger requirement for the top right, autonomous agents. If you have an oracle, then corrigibility seems to reduce to ‘don’t try to influence user’s behavior through your questions’.
When I look at this, I am convinced by the arguments that we probably can’t just build Tool AI, but I super want the most powerful systems of the future be non-autonomous. That just seems to be way safer without sacrificing a lot of performance. I think because of this, I’ve been thinking of IDA as trying to build non-autonomous systems (basically oracles), even though the sequence pretty clearly seems to have autonomous systems in mind.[2] On the other hand, Debate seems to be primarily aimed at non-autonomous systems, which (if true) is an interesting difference.
So is all of this just news to me, and actually everyone is aware of this distinction?
There’s an interesting corollary of semi-decidable languages that sounds like the kind of cool fact you would teach in class, but somehow I’ve never heard or read it anywhere.
A semi-decidable language is a set over a finite alphabet such that there exists a Turing machine such that, for any , if you run on input , then [if it halts after finitely many steps and outputs ‘1’, whereas if , it does something else (typically, it runs forever)].
The halting problem is semi-decidable. I.e., the language of all bit codes of Turing Machines that (on empty input) eventually halt is semi-decidable. However, for any , there is a limit, call it , on how long Turing Machines with bit code of length at most can run, if they don’t run forever.[1] So, if you could compute an upper-bound on , you could solve the halting problem by building a TM that
Computes the upper bound
Simulates the TM encoded by for steps
Halts; outputs 1 if the TM halted and 0 otherwise
Since that would contradict the fact that is not fully decidable, it follows that it’s impossible to compute an upper bound. This means that the function not only is uncomputable, but it grows faster than any computable function.
An identical construction works for any other semi-decidable language, which means that any semi-decidable language determines a function that grows faster than any computable function. Which seems completely insane since is computable .
This just follows from the fact that there are only finitely many such Turing Machines, and a finite subset of them that eventually halt, so if halts after steps, then the limit function is defined by . ↩︎
Arguably, if there is something truly wrong with the list, I should have an issue with it.
This is non-obvious. It seems like you are extrapolating from yourself to everyone else. In my model, how much you would mind being on such a list is largely determined by how much social anxiety you generally feel. I would very much mind being on that list, even if I felt like it was justified.
Knowing the existence of the list (again, even if it were justified) would also make me uneasy to talk to curi.
This post does not explicitly mention the difference between strong and weak HCH, but it links to meta-execution, which is trying to implement strong HCH.
In strong HCH, subagents don’t disappear after having been consulted once. Instead, whenever a human consults a subtree, she can remember the ID of the subtree and choose to consult it again later. (She can even send messages that contain pointers to other agents, but my concern already arises if we don’t include that change.)
This difference seems like an extremely big deal. In particular, it makes it unclear why we would even need a human in the process. If is already stronger than , then the system may not benefit from having manage anything. Instead, could just repeatedly query the same copy of to make progress on the input question.
In this case, there is no longer a reason to model this process via Factored Cognition. In IDA, only the highest level of Factored Cognition happens explicitly (the human may actually divide a question), and every other step happens implicitly (we ‘speculate’ that the model uses FC because it imitates what Is doing). So if the highest level doesn’t use Factored Cognition anymore, then the entire scheme doesn’t use Factored Cognition anymore.
We’re left with agent overseeing agent , where is allowed to think for much longer than .
This might also be a scheme that works, but insofar as dumping Factored Cognition altogether is undesirable, it seems like meta-execution needs a mechanism to discourage this behavior. Since is a human, I guess we could just tell her not to do this? But if we do, in fact, want to discourage this behavior, it seems like the change introduces a spectrum where we trade off security for capability, rather than making a straight-forward improvement to the scheme.
Is there something I’m not seeing? Would appreciate input on this.
That makes me feel a bit like the student who thinks they can debate the professor after having researched the field for 5 minutes.
But it would actually be a really good sign if the area has become more accessible.
The motivation seems trivial to me, which might be part of the problem. Most training procedures are obviously outer-misaligned, so if we have one that may plausibly be outer-aligned (and might plausibly scale), that seems like an obvious reason to take it seriously. I’ve felt like I totally got that once I first understood what IDA is trying to do.
Does it make sense what experience I had reading the post?
It does, but it still leaves me with the problem that it doesn’t seem to be connected to the remaining sequence. IDA isn’t about how we take an already trained system with human-level performance and use it for good things; it’s about how we train a system from the ground up.
The real problem may be that I expect a post like this to closely tie into the actual scheme, so when it doesn’t I take it as evidence that I’ve misunderstood something. What this post talks about may just not be intended to be [the problem the remaining sequence is trying to solve].
My high-level model for how the first AGI systems will be built is a black box search, possible plus transparency tools. (This might already be wrong since you’ve said in a previous post that you think black box searches are unlikely.) In that context, Corrigibility seems to be a property of the system itself, not of the objective. In other words, it’s about Inner Alignment not Outer Alignment.
In this context, it seems extremely difficult to build Corrigibility into the system. However, if I’ve understood the last section of this post correctly, the point of Corrigibility is not that we do something to insert this property into an otherwise non-corrigible training process or system, but that certain approaches lead to corrigible agents by default, and this is an argument for why we might expect the resulting systems to be aligned.
I presently don’t understand why stock IDA (i.e., training system to imitate the performance of system ) would lead to corrigible systems. If the reason is in the details, this may not mean much since I don’t know the details. If the reason is that the generic idea leads to act based agents since Distillation is implemented by something like initiation learning, this seems unconvincing because it’s about the training signal and not the model’s true objective. If the reason is that the overseer is so powerful that it can also solve inner alignment concerns, then I get idea much better and am uncertain what to think. (Although, that would seem to imply that transparency tools are an extremely important component of making this work.)
I generally think of stock IDA as applied to question-answering only, leading to an Oracle. (This might also be wrong). In that case, I think Corrigibility basically means that the Oracle doesn’t try to influence the user’s preferences through its answers? The objection that my brain spits out here is that being myopic sounds like a property that is also sufficient and easier to verify.
It seems to me like there are two separate ideas in this post.
One is HCH itself. Actually, HCH (defined as an infinitely large tree of humans) is a family of schemes with two parameters, so it’s really
where is a human, and the amount of time each human has to think. This is impossible to implement since we don’t have infinitely many copies of the same human—and also because the scheme requires that, every time a human consults a subtree, we freeze time for that human until the subtree has computed the answer. But it’s useful insofar as it can be approximated.
Whether or not there exists any human such that, if we set to an hour, has superintelligent performance on a question-answering task is unclear and relies one some version of the Factored Cognition hypothesis.
Separately, the schemes are about implementations, but they still define targets that can only be approximated, not literally implemented.
Taking just the first one—if is a prediction algorithm, then each for can be defined recursively. Namely, we set to be ‘s learned prediction of ‘s output, and to be ‘s learned prediction of ’s output. Each step requires a training process. We could then set , but this is not literally achievable since it requires infinitely many training steps.
That all makes sense.
Alignment for prosaic AGI in particular will be less important if we don’t actually develop prosaic AGI, but I think that this is a very big problem:
There’s a ‘don’t’ missing here I believe (otherwise I’m confused by the sentence).
Given that you liked it, can you explain to me why? After having already read the sequence (much of it twice), I’m pretty confused about the structure in the first section. I don’t see the point of introducing the Steering Problem at all. It’s similar to Intent Alignment, but not exactly the same (since it makes stronger assumptions about the nature of the AI) -- and the rest of the sequence (and IDA in general) seems to be trying to solve intent alignment, not the steering problem. It’s listed under ‘motivation’ but I don’t really get aspect, either. I don’t know how I’m supposed to connect it to the rest of the sequence.
Common wisdom says that someone accusing you of especially hurts if, deep down, you know that is true. This is confusing because the general pattern I observe is closer to the opposite. At the same time, I don’t think common wisdom is totally without a basis here.
My model to unify both is that someone accusing you of hurts proportionally to how much hearing that you do upsets you.[1] And of course, one reason that it might upset you is that it’s not true. But a separate reason is that you’ve made an effort to delude yourself about it. If you’re a selfish person but spend a lot of effort pretending that you’re not selfish at all, you super don’t want to hear that you’re actually selfish.
Under this model, if someone gets very upset, it might be that that deep down they know the accusation is true, and they’ve tried to pretend it’s not, but it might also be that the accusation is super duper not true, and they’re upset precisely because it’s so outrageous.
Proportional just means it’s one multiplicative factor, though. I think it also matters how high-status you perceive the other person to be. ↩︎
There’s no simple simple math to decide what point in the ZOPA we settle on.
Actually, you could model this as a bargaining problem. The only issue that the basic theory models only one resource that is to be distributed among two people, whereas now we have two (coconuts and bananas). This means it requires both of us to be honest about our preferences for how we weight bananas vs coconuts. Given that, bargaining theory requires that we define
A feasibility set , a closed subset of that is often assumed to be convex, the elements of which are interpreted as agreements. is often assumed to be convex because, for any two feasible outcomes, a convex combination (a weighted average) of them is typically also feasible.
A disagreement, or threat, point , where and are the respective payoffs to player 1 and player 2, which they are guaranteed to receive if they cannot come to a mutual agreement.
The disagreement point equals the pair of {coconut plus banana utility for (me,you) according to our honest preferences} we get when we don’t trade. The feasibility set consists of all possible pairs of of {coconut plus banana utility for (me,you) according to our honest preferences} when we trade according to any one exchange rate.[1] So point in the ZOPA corresponds to one point in the space.[2] Given this, there’s a unique mathematical solution that follows from some basic properties.
If both parties accept this, this would solve a part of the problem. Not all, since we are still incentivized to lie about our preferences.
I think it is a really bad idea to disincentivise medium effort comments that point out problems with important studies. I don’t want it to be a requirement that you know statistics before you get to question a study.
Thanks, that’s helpful.
Is there a consensus on how much to take if you haven’t measured your blood levels?
Since we’re comparing the mean of several numbers, and there are many factors other than vitamin D influencing those numbers, I think it’s a priori not surprising that there are exceptions.
My problem with this post is that you seem to be applying a standard that basically asks,
“Are there formal arguments demonstrating that coherence goal-directedness?”
Whereas the question that I would ask is,
“Does coherence goal-directedness?”
This paragraph, for example
There’s a final issue with the whole setup of an agent traversing states: in the real world, and in examples like non-transitive travel, we never actually end up in quite the same state we started in. Perhaps we’ve gotten sunburned along the journey. Perhaps we spent a few minutes editing our next blog post. At the very least, we’re now slightly older, and we have new memories, and the sun’s position has changed a little. And so, just like with definition 2, no series of choices can ever demonstrate incoherent revealed preferences in the sense of definition 1, since every choice actually made is between a different set of possible states. (At the very least, they differ in the agent’s memories of which path it took to get there.⁴ And note that outcomes which are identical except for slight differences in memories should sometimes be treated in very different ways, since having even a few bits of additional information from exploration can be incredibly advantageous.)
Seems quite important when discussing the first question, but almost entirely irrelevant for the second question. It may be difficult to formalize the MDP such that it captures this, but obviously we don’t expect a superintelligent AI to be predictably stupid in the way Eliezer lines out, and this is true we can formalize it or not.
Speaking just about the latter question, I mainly see the ‘states don’t capture everything we care about’ argument as relevant:
When we do so, we find that it has several shortcomings—in particular, it rules out some preferences which seem to be reasonable and natural ones. For example, suppose you want to write a book which is so timeless that at least one person reads it every year for the next thousand years. [...]
But even with this one—it’s an argument that [the smallest possible class of which we can say that it will capture everything the agent cares about] will be larger than the set of states. That does not mean that it will be so large that the implication of the VNM theorem stops being scary. We know that it won’t be as large as [state-trajectories] because that would allow AI’s that are visibly stupid, so it’ll be some intermediate class. Is that large enough for the implication of the VNM to be substantially less meaningful? My honest best guess is still ‘probably not’.
I see. That all makes sense. Kind of interesting; there’s not really much we appear to disagree on, I just took your post as making a stronger claim than you’re really making.
Richard Ngo did consider this line of argument, see Coherent behaviour in the real world is an incoherent concept.
Thanks—that gets at exactly what I was talking about. If I have more to say, I’ll do it under that post.
Syntax guide here. Alternatively, click on ‘FAQ’ in the left panel and CTRL+F for ‘Markdown’; that’s how I found it.