rvnnt

• rvnnt 3 Jul 2020 15:21 UTC

  1 point

  in reply to: Rohin Shah’s comment on: [AN #102]: Meta learn­ing by GPT-3, and a list of full pro­pos­als for AI alignment

  The space of all possible algorithms one could run on three-digit-addition-strings like “218+375” seems rather vast. Could it be that what GPT3 is doing is something like

  • generating a large bunch of candidate algorithms, and

  • estimating the likelihoods of those algorithms given the examples, and

  • doing something like a noisy/​weak Bayesian update, and

  • executing one of the higher-posterior algorithms, or some “fuzzy combination” of them?

  Obviously this is just wild, vague speculation; but to me it intuitively seems like it would at least sort of answer your question. What do you think? (Could GPT3 be doing something like the above?)

  (To a human, it might feel like [the actual algorithm for addition] is a glaringly obvious candidate. But, on something like a noisy simplicity prior over all possible string-manipulations-algorithms, [the actual algorithm for addition] maybe starts looking like just one of the more conspicuous needles in a haystack?)

• rvnnt 19 Sep 2021 6:46 UTC

  1 point

  in reply to: ike’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  Thanks for the response. I hadn’t heard of SIA before. After a bit of searching, I’m guessing you’re referring to the Self-Indication Assumption.(?)

  SIA, intuitions about it:

  Looks like there’s a lot of stuff to read, under SIA (+ SSA).

  My current impression is that SIA is indeed confused (using a confused ontology/​Map). But given how little I know of SIA, I’m not super confident in that assessment (maybe I’m just misunderstanding what people mean by SIA).

  Maybe if I find the time, I’ll read up on SIA, and write a post about why/​how I think it’s confused. (I’m currently guessing it’d come down to almost the same things I’d write in the long version of this post—about how people end up with confused intuitions about nonexistent sampling processes inserting nonexistent “I/​me” ghosts into some brains but not others.)

  If you could share links/​pointers to the “strong intuitions /​ arguments many people have for SIA” you mentioned, I’d be curious to take a look at them.

  Bets and paradoxes:

  I don’t understant what you mean by {running into paradoxes if I insist the probability is ⁵⁰⁄₅₀ and each agent is given a 1:3 odds bet}. If we’re talking about the bet as described in Eliezer’s original post, then the (a priori) expected utility of accepting the bet would be 0.5*(18 − 23) + 0.5(2 − 18*3) = −20, so I would not want to accept that bet, either before or after seeing green, no? I’m guessing you’re referring to some different bet. Could you describe in more detail what bet you had in mind, or how a paradox arises?

• rvnnt 20 Sep 2021 16:34 UTC

  1 point

  in reply to: ike’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  It seems to me that, also with the bet you describe, there is no paradox/​inconsistency.

  To make sure we’re talking about the same thing: The bet I’m considering is:

  each agent in each room is separately/​independently given the option to bet that the coin came up heads. If the agent says “yes, heads”, and coin=1, that one agent wins 1 utilon. If the agent says “yes, heads”, but coin=0, that one agent loses 3 utilons.

  One likely source of confusion that I see here is: If one thinks about {what the agent cares about} in terms of “I”, “me”, “this agent”, or other such concepts which correspond poorly to the Territory (in this kind of dilemma/​situation).

  To properly deconfuse that, I recommend Tabooing “this”, “I”, “me”, “you”, etc. (A trick I found useful for that was to consider variations of the original dilemma, where the agents additionally have numbers tattooed on them; either numbers from 1 to 20, or random UUIDs, or etc.; either visible to the agent or not. Then one can formulate the agent’s utility function in terms of “agent with number N tattooed on it”, instead of e.g. “instances of me”.)

  For brevity, below I do use “this” and “I” and etc. Hopefully enough of the idea still comes through to be useful.

  If what the agent cares about is something like “utilons gained in total, by computations/​agents that are similar to the original agent”, then:

  • Before the experiment: The agent would want agents in green rooms to accept the bet, and agents in red rooms to reject the bet.

  • Upon waking up in a green room: The agent has received no information which would allow it to distinguish between coin-flip-outcomes, and its probability for coin=1 is still ⁵⁰⁄₅₀. I.e., the agent is in practically the same situation as before the experiment, and so its answer is still the same: accept the bet. (And conversely if in a red room.)

  The above seems consistent/​paradox-free to me.(?)

  If what the agent cares about is something like “utilons gained, by this particular blob of atoms, and the temporal sort-of-continuation of it, as usually understood by e.g. humans”, then:

  • Before the experiment: The original description of the dilemma leaves unclear what happens to the original blob of atoms, but here I’ll assume that the original blob-of-atoms is destroyed or somehow split 20 ways. In that case, before the experiment, the agent would not care at all how the other copy-agents bet. They’re not the same blob-of-atoms, after all. The pre-experiment agent’s preferences w.r.t. betting are undefined.

  • Upon waking up in a green room: the agent’s probability distribution over coin=1 is still ⁵⁰⁄₅₀. And it is a new blob-of-atoms, not the original pre-experiment blob-of-atoms. It doesn’t care at all how other blobs-of-atoms fare. It rejects the bet.

  To the extent that there’s an inconsistency between the pre-experiment agent and in-room agents, I think it’s due to them being different agents with different utility functions. So it doesn’t seem accurate to say that either agent’s individual preferences are inconsistent?

  A likely objection I anticipate to the above is something like:

  “{The total number of utilons (summed over agents) gained by agents using the above probability-update- and decision-algorithm} is less than {the total number of utilons gained by agents that would update their probability for coin=1 to 90% upon seeing green}.”

  To which I think Blob-of-atoms #N would respond:

  “Yes, but Blob #N does not care that Other-blobs-of-atoms-using-the-same-reasoning-algorithm tend to fare poorly. Blob #N isn’t trying to cooperate with near-copies of itself. So what if other blobs-of-atoms don’t gain utility? That argument doesn’t really weigh against Blob #N’s reasoning-algorithm, does it?”

  Does this make sense to you? If not, what seems (most) wrong/​confused?

• rvnnt 20 Sep 2021 16:41 UTC

  1 point

  in reply to: JBlack’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  I’m not sure what you mean, by me “rejecting decision theories”? Maybe my most recent reply to ike helps clarify what kinds of decisions I think the agents “ought” to take?

• rvnnt 21 Sep 2021 7:37 UTC

  1 point

  in reply to: rvnnt’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  Update: Upon considering the situation where

  • each blob-of-atoms cares only about itself (not about similar/​copied agents)

  • and the original blob-of-atoms is inserted (e.g. uniformly at random) into one of the rooms (along with 19 copy-agents)

  it seems that there is in fact a temporal inconsistency: Before the experiment, original-blob would want all agents (including original-blob) in green rooms to accept the bet, but upon waking up and observing green, original-blob would reject the bet. Will update the post to reflect this.

• rvnnt 21 Sep 2021 7:41 UTC

  1 point

  in reply to: ike’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  If you’re ok with time inconsistent probabilities then you can be dutch booked.

  Sure. Has some part of what I’ve written given the impression that I think time-inconsistent probabilities (or preferences) are OK?

  I think of identity in terms of expectations. [...]

  I want to give a thumbs-up to the policy of sharing ways-of-thinking-about-stuff. (Albeit that I think I see how that particular way of thinking about this stuff is probably confused. I’m still suggesting Tabooing “I”, “me”, “you”, “[me] waking up in …”, etc.) Thanks.

  It’s not clear how the utility function in your first case says to accept the bet given that [...]

  True, that part of what I wrote glossed over a large bunch of details (which may well be hiding confusion on my part). To try to quickly unpack that a bit:

  • In the given scenario, each agent cares about all similar agents.

  • Pretending to be a Solomonoff inductor, and updating on all available information/​observations—without mapping low-level observations into confused nonsense like “I/​me is observing X”—an agent in a green room ends up with p(coin=1) = 0.5.

  • The agent’s model of reality includes a model of {the agent itself, minus the agent’s model of itself (to avoid infinite recursion)}.

  • Looking at that model from a bird’s-eye-view, the agent searches for an action $a$ that would maximize $$\sum _{w\in W}\text{utility received by xeroxed agents in the version of}w\text{where this agent outputs}a$$, where $W$ is the set of “possible” worlds. (I.e.$W$ is the set of worlds that are consistent with what has been observed thus far.) (We’re not bothering to weight the summed terms by $p\left(w\right)$ because here all $w$ are equiprobable.)

  • According to the agent’s model, all in-room-agents are running the same decision-algorithm, and thus all agents observing the same color output the same decision. This constrains what $W$ can contain. In particular, it only contains worlds $w$ where if this agent is outputting $a$, then also all other agents (in rooms of the same color) are also outputting $a$.

  • The agent’s available actions are “accept bet” and “decline bet”. When the agent considers those worlds where it (and thus, all other agents-in-green) outputs “accept bet”, it calculates the total utility gained by xeroxed agents to be higher, than in those worlds where it output “decline bet”.

  • The agent outputs “accept bet”.

  If the above is not “maximizing utility”, then I’m confused about what (you mean by) “maximizing utility”. Did this clarify anything?

  My version of the bet shouldn’t depend on if you care about other agents or not, because the bet doesn’t affect other agents.

  It’s true that (if the rooms are appropriately sealed off from each other) the blobs-of-atoms in different rooms cannot causally affect each other. But given knowledge that all agents are exact copies of each other, the set of “possible” worlds is constrained to contain only {worlds where all agents (in rooms of the same color) output the same decision}. (I’m thinking very loosely in terms of something like Solomonoff induction here.) Thus it seems to me that {operating/​deciding as if agents in other rooms “could” decide something different from each other} is like operating with the wrong set of “possible” worlds; i.e. like doing something wrong relative to Solomonoff induction, and/​or having an incorrect model of reality.

  Maybe: try Tabooing the word “affect”?

• rvnnt 21 Sep 2021 7:47 UTC

  1 point

  in reply to: Measure’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  I’m confused: Where have I said that probabilities are undefined?

  I did say that, if the pre-experiment agent-blob-of-atoms cares only about itself, and not the in-room-agents, then its preferences w.r.t. how the in-room-agents bet are undefined. Because its utility function was (by definition) independent of what happens to the in-room-agents. But I don’t think I’ve implied that any probabilities are undefined.

  Did this help clarify things?

• rvnnt 21 Sep 2021 16:37 UTC

  1 point

  in reply to: ike’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  Thanks for the suggestions. Clearly there’s still a lot of potentially fruitful disagreement here, some of it possibly mineable for insights; but I’m going to put this stuff on the shelf for now. Anyway, thanks.

• rvnnt 21 Sep 2021 16:39 UTC

  1 point

  in reply to: JBlack’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  You appeared to be rejecting any meaning for probability of events such as “I am a copy” [...]

  If I can translate “I am a copy” to e.g. “an agent that is currently having observations/​thoughts x,y,z is a copy” (or to something else that does not depend on (seemingly-to-me ill-defined) things like “me”), then I do think that probabilities can and should be assignable to those kinds of events. I guess my post/​writing was even less clear than I thought.

  The fact remains that the indistinguishable copies still need to make decisions, may [...]

  I think I’m entirely in agreement with that paragraph.

  Declaring that all utilities must be summed or averaged or whatever [...]

  I don’t understand where that paragraph is coming from. (I’m guessing it’s coming from my writing being much less clear or much more prone to misinterpretation than I thought. Feel free to not explain where that paragraph came from.)

• rvnnt 22 Sep 2021 16:14 UTC

  1 point

  in reply to: JBlack’s comment on: Out­law­ing An­throp­ics: Dis­solv­ing the Dilemma

  Ah, the example I gave above was not very good. To clarify:

  If I can translate things like “I am a copy” to {propositions defined entirely in terms of non-magical things}, then I think it should be possible to assign probabilities to them.

  Like, imagine “possible” worlds $w$ are Turing machines, or cellular automata, or some other kind of well defined mathematical object. Then, for any computable function $f$ over worlds, I think that

  • it should be possible to assign probabilities to things like $f\left(w\right)=42$, or $f\left(w\right)\le 1$, or whatever

  • and the above kinds of things are (probably?) the only kinds of things for which probabilities even are “well defined”.

  (I currently wouldn’t be able to give a rigorous definition of what “well defined” means in the above; need to think about that.)

  If you can come up with events/​propositions that

  • can not (even in principle) be reduced to the $f\left(w\right)=x$ form above,

  • but which also would be necessary to assign probabilities to, in order to be able to make decisions,

  then I’d be interested to see them!

• rvnnt 20 Apr 2022 16:26 UTC

  1 point

  on: Align­ment and Deep Learning

  I found this post to be interesting. I feel it might be waving at something important. Thanks for writing it.

  I’m confused by a bunch of things here, notably point (4.) in the five-point list at the end of the post. I think it’s doing a lot of work, and likely hiding some difficult problems. Below are some notes and questions. If OP (or anyone else) feels like answering the questions, or pointing out where I’m wrong/​confused, that’d be really neat.

  (i)

  If the Predictor is sufficiently intelligent to model human-level Users, it would not be safe to train/​run.(?) If, on the other hand, the Predictor is too dumb to model human-level Users, then it’s useless for learning “human values”.

  If so, then: The Predictor would need to be kept dumb during training (before it has learned human values); e.g. by limiting the number of parameters in the Predictor model. But also, when deployed, it would need to be able to gain further capabilities—at least up till “human level”.

  And yet, when deployed in the real world, the Predictor would lose access to the reward function built into the training environment.

  Thus, questions:

  • How does the Predictor learn/​gain capabilities during deployment? How is it scored/​rewarded?

  • How, concretely, do we safely move from the training regime to the deployment regime? (Use different-sized models in training and deployment, and somehow (how?) transfer the smaller (trained) model’s capabilities to the larger one, and then… ???)

  (ii)

  If the Predictor’s reward function is of type UserState → Reward, then the Predictor would likely end up learning to deceive/​hack the User. I’m assuming the Predictor’s reward function is of type EnvironmentState → Reward.(?)

  But then: It seems the main benefit of the proposed scheme would come from training the Predictor to interact with other agents in order to learn their goals. (If not, then how does this scheme differ meaningfully from one where the Predictor-agent is alone in an environment, without a User-agent?)

  If that’s so, then: The usefulness of this approach would depend critically on things like

  • the interface /​ “interaction channels” between User and Predictor

  • the User’s utility function being very hard to learn without interacting with the User

  • the User’s utility function also depending on the User’s internal state

  Is that correct?

• rvnnt 20 Apr 2022 16:28 UTC

  1 point

  on: Align­ment and Deep Learning

  IIUC, OP is proposing that the Predictor agent be directly incentivized (trained) to predict/​model the User.

  I think it’s important to note that “predict the User” is very different from “optimize for the User’s values”. I think that, if one were to optimize really hard for “predict the User”, one would end up doing things like simulating the User in all kinds of situations, most of which would be pretty horrible for the User (assuming the User’s preferences/​values were anywhere near as fragile/​selective as human values).

  I think training an AGI to predict/​understand human values would likely end in something a lot worse than death. I think “predict/​understand the User” should probably never be directly incentivized at all. (We’d want the AI to do that only for instrumental reasons, in its attempts to optimize for the user’s values.)

  (I suspect lots of people, when imagining “simulating lots of situations in order to understand User’s values”, end up sampling those imagined situations from something like “the set of situations I’ve experienced”. But our experiences are very strongly biased towards desirable situations—we spend most of our time optimizing our lives, after all. Given that human values are rather fragile/​selective, I suspect the vast majority of all possible experiences/​situations would rank very very low according to “human values”; no adversarial selection needed: most actually random experiences are probably pretty unpleasant.)

  Given the above, I’m confused as to why one would propose directly incentivizing “predict the User”. Do others disagree with the risks, or think the risks are worth the benefits, or something else?

• rvnnt 28 Apr 2022 16:02 UTC

  1 point

  in reply to: Aiyen’s comment on: Align­ment and Deep Learning

  I still feel like there’s a big hole in this scheme. Maybe I’m just not getting something. Here’s a summary of my current model of this scheme, and what I find confusing:

  • We train an agent (the Predictor) in a large number of different environments, with a large number of different utility/​reward functions. (And some of those reward functions may or may not be easier to learn by interacting with certain parts of the environment which we humans think of as “the User agent”.)

  • Presumably/​hopefully, this leads to the Predictor learning things like caution/​low impact, or heuristics like “seek to non-destructively observe other agent-like things in the environment”. (And if so, that could be very useful!)

  • But: We cannot use humans in the training environments. And: We cannot train the Predictor to superhuman levels in the training environments (for the obvious reason that it wouldn’t be aligned).

  • When we put the Predictor in the real world (to learn human values), it could no longer be scored the same way as in the training environments—because the real world does not have a built-in reward function we could plug into the Predictor.

  • Thus: When deployed, the Predictor would be sub-human, and also not have access to a reward function.

  • And so, question: How do we train the Predictor to learn human values, and become highly capable, once it is in the real world? What do we use as a reward signal?

  The closest thing to an answer I found was this:

  And even if it’s object level model of human values was static, it could still continue learning about the world with that model providing the scoring needed.

  But IIUC this is suggesting that we could/​should build a powerful AI with an imperfect/​incorrect and static (fixed/​unchangeable) model $V$ of human values, and then let that AI bootstrap to superintelligence, with $V$ as its reward function? But that seems like an obviously terrible idea, so maybe I’ve misunderstood something? What?

• rvnnt 28 Apr 2022 16:04 UTC

  1 point

  in reply to: Aiyen’s comment on: Align­ment and Deep Learning

  Yep, I understood that you intended for the Predictor to also/​primarily be scored on how well it fulfills the User’s values.

  I’m modeling our disagreement something like this:

  • Aiyen: It could be a good idea to directly incentivize a powerful AI to learn to predict humans, so long as one also directly incentivizes it to optimize for humans values.

  • rvnnt: Directly incentivizing a powerful AI to learn to predict humans would likely lead to the AI allocating at least some fraction of its (eventually vast) resources to e.g. simulating humans experiencing horrible things. Thus it would probably be a very bad idea to directly incentivize a powerful AI to learn to predict humans, even if one also incentivizes it to optimize for human values.

  Does that seem roughly correct to you? (If yes, I’m curious how you’d guarantee that the Predictor does not end up allocating lots of resources to some kind of mindcrime?)

• rvnnt 18 May 2022 10:39 UTC

  3 points

  on: Why I’m Op­ti­mistic About Near-Term AI Risk

  I find the story about {lots of harmless, specialized transformer-based models being developed} to be plausible. I would not be surprised if many tech companies were to follow something like that path.

  However, I also think that the conclusion—viz., it being unlikely that any AI will pose an x-risk in the next 10-20 years—is probably wrong.

  The main reason I think that is something like the following:

  In order for AI to pose an x-risk, it is enough that even one research lab is a bit too incautious/​stupid/​mistakenly-optimistic and “successfully” proceeds with developing AGI capabilities. Thus, the proposition that {AI will not pose an x-risk within N years} seems to require that $$\forall \text{ResearchLab}\in \left\{\text{all research labs that will exist over next N years}\right\}\forall \text{Techniques}\subseteq \{\text{ML techniques available in next N years}\text{ResearchLab}\text{can/will NOT develop dangerous AGI with}\text{Techniques}$$

  And the above is basically a large conjunction over many research labs and as-yet-unknown future ML technologies. I think it is unlikely to be true. Reasons why I think it is unlikely to be true:

  • It seems plausible to me that highly capable, autonomous (dangerous) AGI could be built using some appropriate combination of already existing techniques + More Compute.

  • Even if it weren’t/​isn’t possible to build dangerous AGI with existing techniques, a lot of new techniques can be developed in 10 years.

  • There are many AI research labs in existence. Even if most of them were to pursue only narrow/​satisfactory AI, what are the odds that not one of them pursues (dangerous) autonomous AGI?

  • I’m under the impression that investment in A(G)I capabilities research is increasing pretty fast; lots of smart people are moving into the field. So, the (near) future will contain even more research labs and sources of potentially dangerous new techniques.

  • 10 years is a really long time. Like, 10 years ago it was 2012, deep learning was barely starting to be a thing, the first Q-learning-based Atari-playing model (DQN) hadn’t even been released yet, etc. A lot of progress has happened from 2012 to 2022. And the amount of progress will presumably be (much) greater in the next 10 years. I feel like I have almost no clue what the future will look like in 10-20 years.

  • We (or at least I) still don’t even know any convincing story of how to align autonomous AGI to “human values” (whatever those even are). (Let alone having practical, working alignment techniques.)

  Given the above, I was surprised by the apparent level of confidence given to the proposition that “AI is unlikely to pose an existential risk in the next 10-20 years”. I wonder where OP disagrees with the above reasoning?

  Regarding {acting quickly} vs {movement-building, recruiting people into AI safety, investing in infrastructure, etc.}: I think it’s probably obvious, but maybe bears pointing out, that when choosing strategies, one should consider not only {the probability of various timelines} but also {the expected utility of executing various strategies under various timelines}.

  (For example, if timelines are very short, then I doubt even my best available {short-timelines strategy} has any real hope of working, but might still involve burning a lot of resources. Thus: it probably makes sense for me to execute {medium-to-long timelines strategy}, even if I assign high probability to short timelines? This may or may not generalize to other people working on alignment.)

  What links here?

  • If no near-term al­ign­ment strat­egy, re­search should aim for the long-term by harsimony (9 Jun 2022 19:10 UTC; 7 points)

• rvnnt 8 Jul 2022 9:36 UTC

  3 points

  2

  on: Prin­ci­ples for Align­ment/​Agency Projects

  I think these are very good principles. Thank you for writing this post, John.

  Thoughts on actually going about Tackling the Hamming Problems in real life:

  My model of humans (and of doing research) says that in order for a human to actually successfully work on the Hard Parts, they probably need to enjoy doing so on a System 1 level. (Otherwise, it’ll probably be an uphill battle against subconscious flinches away from Hard Parts, procrastinating with easier problems, etc.)

  I’ve personally found these essays to be insightful and helpful for (i.a.) training my S1 to enjoy steering towards Hard Parts. I’m guessing they could be helpful to many other people too.

• rvnnt 12 Sep 2022 11:37 UTC

  1 point

  on: Briefly think­ing through some analogs of debate

  As a Debater’s capabilities increase, I expect it to become more able to convince a human of both true propositions and also of false propositions. Particularly when the propositions in question are about complex (real-world) things. And in order for Debate to be useful, I think it would indeed have to be able to handle very complex propositions like “running so-and-so AI software would be unsafe”. For such a proposition $P$, on the limit of Debater capabilities, I think a Debater would have roughly as easy a time convincing a human of $\neg P$ as of $P$. Hence: As Debater capabilities increase, if the judge is human and the questions being debated are complex, I’d tentatively expect the Debaters’ arguments to mostly be determined by something other than “what is true”.

  I.e., the approximate opposite of

  “in the limit of argumentative prowess, the optimal debate strategy converges to making valid arguments for true conclusions.”

• rvnnt 14 Sep 2022 10:33 UTC

  2 points

  0

  on: Try­ing to find the un­der­ly­ing struc­ture of com­pu­ta­tional systems

  Regarding if-branching in arithmetic: If I understand correctly, you’re looking for a way to express

  if a then (b+c) else (b-c)

  using only arithmetic? If so: Under some assumptions about the types of a, b, and c, the above can be expressed simply as

  a*(b+c) + (1-a)*(b-c).

  Another perspective: Instead of trying to figure out {how a boolean input to an IF-node could change the function in another node}, maybe think of a single curried function node of type Bool -> X -> X -> Y. Partially applying it to different values of type Bool will then give you different functions of type X -> X -> Y.

  (Was that helpful?)

• rvnnt 28 Oct 2022 10:43 UTC

  0 points

  −3

  on: How Risky Is Trick-or-Treat­ing?

  What about (e.g.) risks from kids ingesting candy sourced from multiple unknown people (any one of whom could poison a whole lotta kids with decent chances of not getting caught)? Maybe tell kids to only accept cash?

• rvnnt 3 Nov 2022 12:43 UTC

  2 points

  0

  in reply to: DPiepgrass’s comment on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  In order for a Tool/​Oracle to be highly capable/​useful and domain-general, I think it would need to perform some kind of more or less open-ended search or optimization. So the boundary between “Tool”, “Oracle”, and “Sovereign” (etc.) AI seems pretty blurry to me. It might be very difficult in practice to be sure that (e.g.) some powerful “tool” AI doesn’t end up pursuing instrumentally convergent goals (like acquiring resources for itself). Also, when (an Oracle or Tool is) facing a difficult problem and searching over a rich enough space of solutions, something like “consequentialist agents” seem to be a convergent thing to stumble upon and subsequently implement/​execute.

  Suggested reading: https://​​www.lesswrong.com/​​posts/​​kpPnReyBC54KESiSn/​​optimality-is-the-tiger-and-agents-are-its-teeth