ricraz
Karma: 1,452 (LW), 53 (AF)
NewTop
Thanks for the detailed comments! I only have time to engage with a few of them:
Most of this is underdefined, and that’s unsettling at least in some (but not necessarily all) cases, and if we want to make it less underdefined, the notion of ‘one ethics’ has to give.
I’m not that wedded to ‘one ethics’, more like ‘one process for producing moral judgements’. But note that if we allow arbitrariness of scope, then ‘one process’ can be a piecewise function which uses one subprocess in some cases and another in others.
I find myself having similarly strong meta-level intuitions about wanting to do something that is “non-arbitrary” and in relevant ways “simple/elegant”. …motivationally it feels like this intuition is importantly connected to what makes it easy for me to go “all-in“ for my ethical/altruistic beliefs.
I agree that these intuitions are very strong, and they are closely connected to motivational systems. But so are some object-level intuitions like “suffering is bad”, and so the relevant question is what you’d do if it were a choice between that and simplicity. I’m not sure your arguments distinguish one from the other in that context.
one can maybe avoid to feel this uncomfortable feeling of uncertainty by deferring to idealized reflection. But it’s not obvious that this lastingly solves the underlying problem
Another way of phrasing this point: reflection is almost always good for figuring out what’s the best thing to do, but it’s not a good way to define what’s the best thing to do.
For the record, this is probably my key objection to preference utilitarianism, but I didn’t want to dive into the details in the post above (for a very long post about such things, see here).
From Rohin’s post, a quote which I also endorse:
You could argue that while [building AIs with really weird utility functions] is possible in principle, no one would ever build such an agent. I wholeheartedly agree, but note that this is now an argument based on particular empirical facts about humans (or perhaps agent-building processes more generally).
And if you’re going to argue based on particular empirical facts about what goals we expect, then I don’t think that doing so via coherence arguments helps very much.
This seems pretty false to me.
I agree that this problem is not a particularly important one, and explicitly discard it a few sentences later. I hadn’t considered your objection though, and will need to think more about it.
(Side note: I’m pretty annoyed with all the use of “there’s no coherence theorem for X” in this post.)
Mind explaining why? Is this more a stylistic preference, or do you think most of them are wrong/irrelevant?
the “further out” your goal is and the more that your actions are for instrumental value, the more it should look like world 1 in which agents are valuing abstract properties of world states, and the less we should observe preferences over trajectories to reach said states.
Also true if you make world states temporally extended.
If I had to define it using your taxonomy, then yes. However, it’s also trying to do something broader. For example, it’s intended to be persuasive to people who don’t think of meta-ethics in terms of preferences and rationality at all. (The original intended audience was the EA forum, not LW).
Edit: on further reflection, your list is more comprehensive than I thought it was, and maybe the people I mentioned above actually would be on it even if they wouldn’t describe themselves that way.
Another edit: maybe the people who are missing from your list are those who would agree that morality has normative force but deny that rationality does (except insofar as it makes you more moral), or at least are much more concerned with the former than the latter. E.g. you could say that morality is a categorical imperative but rationality is only a hypothetical imperative.
Arguments for moral indefinability
Coherent behaviour in the real world is an incoherent concept
There are some interesting insights about the overall viewpoint behind this book, but gosh the tone of this post is vicious. I totally understand frustration with stupidity in fiction, and I’ve written such screeds in my time too. But I think it’s well worth moderating the impulse to do so in cases like this where the characters whose absolute stupidity you’re bemoaning map onto the outgroup in so many ways.
Agreed, except that the behaviour described could also just be procrastination.
I don’t think it depends on how much A and B, because the “expected amount” is not a special point. In this context, the update that I made personally was “There are more shifts than I thought there were, therefore there’s probably more of A and B than I thought there was, therefore I should weakly update against AI safety being important.” Maybe (to make A and B more concrete) there being more shifts than I thought downgrades my opinion of the original arguments from “absolutely incredible” to “very very good”, which slightly downgrades my confidence that AI safety is important.
As a separate issue, conditional on the field being very important, I might expect the original arguments to be very very good, or I might expect them to be very good, or something else. But I don’t see how that expectation can prevent a change from “absolutely exceptional” to “very very good” from downgrading my confidence.
Vote counting bug?
Apologies if this felt like it was targeted specifically at you and other early AI safety advocates, I have nothing but the greatest respect for your work. I’ll rewrite to clarify my intended meaning, which is more an attempt to evaluate the field as a whole. This is obviously a very vaguely-defined task, but let me take a stab at fleshing out some changes over the past decade:
1. There’s now much more concern about argument 2, the target loading problem (as well as inner optimisers, insofar as they’re distinct).
2. There’s now less focus on recursive self-improvement as a key reason why AI will be dangerous, and more focus on what happens when hardware scales up. Relatedly, I think a greater percentage of safety researchers believe that there’ll be a slow takeoff than used to be the case.
3. Argument 3 (prosaic AI alignment) is now considered more important and more tractable.
4. There’s now been significant criticism of coherence arguments as a reason to believe that AGI will pursue long-term goals in an insatiable maximising fashion.
I may be wrong about these shifts—I’m speaking as a newcomer to the field who has a very limited perspective on how it’s evolved over time. If so, I’d be happy to be corrected. If they have in fact occurred, here are some possible (non-exclusive) reasons why:
A. None of the proponents of the original arguments have changed their minds about the importance of those arguments, but new people came into the field because of those arguments, then disagreed with them and formulated new perspectives.
B. Some of the proponents of the original arguments have changed their minds significantly.
C. The proponents of the original arguments were misinterpreted, or overemphasised some of their beliefs at the expense of others, and actually these shifts are just a change in emphasis.
I think none of these options reflect badly on anyone involved (getting everything exactly right the first time is an absurdly high standard), but I think A and B would be weak evidence against the importance of AI safety (assuming you’ve already conditioned on the size of the field, etc). I also think that it’s great when individual people change their minds about things, and definitely don’t want to criticise that. But if the field as a whole does so (whatever that means), the dynamics of such a shift are worth examination.
I don’t have strong beliefs about the relative importance of A, B and C, although I would be rather surprised if any one of them were primarily responsible for all the shifts I mentioned above.
I endorse ESRogs’ answer. If the world were a singleton under the control of a few particularly benevolent and wise humans, with an AGI that obeys the intention of practical commands (in a somewhat naive way, say, so it’d be unable to help them figure out ethics) then I think argument 5 would no longer apply, but argument 4 would. Or, more generally: argument 5 is about how humans might behave badly under current situations and governmental structures in the short term, but makes no claim that this will be a systemic problem in the long term (we could probably solve it using a singleton + mass surveillance); argument 4 is about how we don’t know of any governmental(/psychological?) structures which are very likely to work well in the long term.
Having said that, your ideas were the main (but not sole) inspiration for argument 4, so if this isn’t what you intended, then I may need to rethink its inclusion.
Disentangling arguments for the importance of AI safety
Nice overview :) One point: the introductory sentences don’t seem to match the content.
It is clear to most AI safety researchers that the idea of “human values” is underdefined, and this concept should be additionally formalized before it can be used in (mostly mathematical) models of AI alignment.
In particular, I don’t interpret most of the researchers you listed as claiming that “[human values] should be formalized”. I think that’s a significantly stronger claim than, for example, the claim that we should try to understand human values better.
Are you claiming that price per computation would drop in absolute terms, or compared with the world in which Moore’s law continued? The first one seems unobjectionable, the default state of everything is for prices to fall since there’ll be innovation in other parts of the supply chain. The second one seems false. Basic counter-argument: if it were true, why don’t people produce chips from a decade ago which are cheaper per amount of computation than the ones being produced today?
1. You wouldn’t have to do R&D, you could just copy old chip designs.
2. You wouldn’t have to keep upgrading your chip fabs, you could use old ones.
3. People could just keep collecting your old chips without getting rid of them.
4. Patents on old chip designs have already expired.
AI services can totally be (approximately) VNM rational—for a bounded utility function.
Suppose an AI service realises that it is able to seize many more resources with which to fulfil its bounded utility function. Would it do so? If no, then it’s not rational with respect to that utility function. If yes, then it seems rather unsafe, and I’m not sure how it fits Eric’s criterion of using “bounded resources”.
Note that CAIS is suggesting that we should use a different prior: the prior based on “how have previous advances in technology come about”. I find this to be stronger evidence than how evolution got to general intelligence.
I agree with Eric’s claim that R&D automation will speed up AI progress. The point of disagreement is more like: when we have AI technology that’s able to do basically all human cognitive tasks (which for want of a better term I’ll call AGI, as an umbrella term to include both CAIS and agent AGI), what will it look like? It’s true that no past technologies have looked like unified agent AGIs—but no past technologies have also looked like distributed systems capable of accomplishing all human tasks either. So it seems like the evolution prior is still the most relevant one.
“Humans think in terms of individuals with goals, and so even if there’s an equally good approach to AGI which doesn’t conceive of it as a single goal-directed agent, researchers will be biased against it.”
I’m curious how strong an objection you think this is. I find it weak; in practice most of the researchers I know think much more concretely about the systems they implement than “agent with a goal”, and these are researchers who work on deep RL. And in the history of AI, there were many things to be done besides “agent with a goal”; expert systems/GOFAI seems like the canonical counterexample.
I think the whole paradigm of RL is an example of a bias towards thinking about agents with goals, and that as those agents become more powerful, it becomes easier to anthropomorphise them (OpenAI Five being one example where it’s hard not to think of it as a group of agents with goals). I would withdraw my objection if, for example, most AI researchers took the prospect of AGI from supervised learning as seriously as AGI from RL.
A clear counterargument is that some companies will have AI CEOs, and they will outcompete the others, and so we’ll quickly transition to the world where all companies have AI CEOs. I think this is not that important—having a human in the loop need not slow down everything by a huge margin, since most of the cognitive work is done by the AI advisor, and the human just needs to check that it makes sense (perhaps assisted by other AI services).
I claim that this sense of “in the loop” is irrelevant, because it’s equivalent to the AI doing its own thing while the human holds a finger over the stop button. I.e. the AI will be equivalent to current CEOs, the humans will be equivalent to current boards of directors.
To the extent that you are using this to argue that “the AI advisor will be much more like an agent optimising for an open-ended goal than Eric claims”, I agree that the AI advisor will look like it is “being a very good CEO”. I’m not sure I agree that it will look like an agent optimizing for an open-ended goal, though I’m confused about this.
I think of CEOs as basically the most maximiser-like humans. They have pretty clear metrics which they care about (even if it’s not just share price, “company success” is a clear metric by human standards), they are able to take actions that are as broad in scope as basically any actions humans can take (expand to new countries, influence politics, totally change the lives of millions of employees), and almost all of the labour is cognitive, so “advising” is basically as hard as “doing” (modulo human interactions). To do well they need to think “outside the box” of stimulus and response, and deal with worldwide trends and arbitrarily unusual situations (has a hurricane just hit your factory? do you need to hire mercenaries to defend your supply chains?) Most of them have some moral constraints, but also there’s a higher percentage of psychopaths than any other role, and it’s plausible that we’d have no idea whether an AI doing well as a CEO actually “cares about” these sorts of bounds or is just (temporarily) constrained by public opinion in the same way as the psychopaths.
The main point of CAIS is that services aren’t long-term goal-oriented; I agree that if services end up being long-term goal-oriented they become dangerous.
I then mentioned that to build systems which implement arbitrary tasks, you may need to be operating over arbitrarily long time horizons. But probably this also comes down to how decomposable such things are.
If you go via the CAIS route you definitely want to prevent unbounded AGI maximizers from being created until you are sure of their safety or that you can control them. (I know you addressed that in the previous point, but I’m pretty sure that no one is arguing to focus on CAIS conditional on AGI agents existing and being more powerful than CAIS, so it feels like you’re attacking a strawman.)
People are arguing for a focus on CAIS without (to my mind) compelling arguments for why we won’t have AGI agents eventually, so I don’t think this is a strawman.
Given a sufficiently long delay, we could use CAIS to build global systems that can control any new AGIs, in the same way that government currently controls most people.
This depends on having pretty powerful CAIS and very good global coordination, both of which I think of as unlikely (especially given that in a world where CAIS occurs and isn’t very dangerous, people will probably think that AI safety advocates were wrong about there being existential risk). I’m curious how likely you think this is though? If agent AGIs are 10x as dangerous, and the probability that we eventually build them is more than 10%, then agent AGIs are the bigger threat.
I also am not sure why you think that AGI agents will optimize harder for self-improvement.
Because they have long-term convergent instrumental goals, and CAIS doesn’t. CAIS only “cares” about self-improvement to the extent that humans are instructing it to do so, but humans are cautious and slow. Also because even if building AGI out of task-specific strongly-constrained modules is faster at first, it seems unlikely that it’s anywhere near the optimal architecture for self-improvement.
Compared to what? If the alternative is “a vastly superintelligent AGI agent that is acting within what is effectively the society of 2019”, then I think CAIS is a better model. I’m guessing that you have something else in mind though.
It’s something like “the first half of CAIS comes true, but the services never get good enough to actually be comprehensive/general. Meanwhile fundamental research on agent AGI occurs roughly in parallel, and eventually overtakes CAIS.” As a vague picture, imagine a world in which we’ve applied powerful supervised learning to all industries, and applied RL to all tasks which are either as constrained and well-defined as games, or as cognitively easy as most physical labour, but still don’t have AI which can independently do the most complex cognitive tasks (Turing tests, fundamental research, etc).
You’re right, this is a rather mealy-mouthed claim. I’ve edited it to read as follows:
the empirical claim that we’ll develop AI services which can replace humans at most cognitively difficult jobs significantly before we develop any single strongly superhuman AGI
This would be false if doing well at human jobs requires capabilities that are near AGI. I do expect a phase transition—roughly speaking I expect progress in automation to mostly require more data and engineering, and progress towards AGI to require algorithmic advances and a cognition-first approach. But the thing I’m trying to endorse in the post is a weaker claim which I think Eric would agree with.
AGI is … something that approximates an expected utility maximizer.
This seems like a trait which AGIs might have, but not a part of how they should be defined. I think Eric would say that the first AI system which can carry out all the tasks we would expect an AGI to be capable of won’t actually approximate an expected utility maximiser, and I consider it an open empirical question whether or not he’s right.
Many risk-reducing services (especially ones that can address human safety problems) seem to require high-level general reasoning abilities, whereas many risk-increasing services can just be technical problem solvers or other kinds of narrow intelligences or optimizers, so CAIS is actually quite unsafe, and hard to make safe, whereas AGI / goal-directed agents are by default highly unsafe, but with appropriate advances in safety research can perhaps be made safe.
Yeah, good point. I guess that my last couple of sentences were pretty shallowly-analysed, and I’ll retract them and add a more measured conclusion.
Agreed that “clarification” is confusing. What about “exploration”?