Because picking a successor is like picking a policy, and risk aversion over policies can give different results than risk aversion over actions.
I was already thinking the AI would be risk averse over whole policies and the aggregate value of their future, not locally/greedily/separately for individual actions and individual unaggregated rewards.
Thanks! This makes sense.
I agree model-free RL wouldn’t necessarily inherit the risk aversion, although I’d guess there’s still a decent chance it would, because that seems like the most natural and simple way to generalize the structure of the rewards.
Why would hardcoded model-based RL probably self-modify or build successors this way, though? To deter/prevent threats from being made in the first place or even followed through on? But, does this actually deter or prevent our threats when evaluating the plan ahead of time, with the original preferences? We’d still want to shut it and any successors down if we found out (whenever we do find out, or it starts trying to take over), and it should be averse to that increased risk ahead of time when evaluating the plan.
Risk aversion wouldn’t help humanity much if we build unaligned AGI anyhow. The least risky plans from the AI’s perspective are still gonna be bad for humans.
I think there are (at least) two ways to reduce this risk:
Temporal discounting. The AI wants to ensure its own longevity, but is really focused on the very near term, just making it through the next day or hour, or whatever, so increasing the risk of being caught and shut down now by doing something sneaky looks bad even if it increases the expected longevity significantly, because it’s discounting the future so much. It will be more incentivized to do whatever people appear to want it to do ~now (regardless of impacts on the future), or else risk being shut down sooner.
Difference-making risk aversion, i.e. being risk averse with respect to the difference with inaction (or some default safe action).[1] This makes inaction look relatively more attractive. (In this case, I think the agent can’t be represented by a single consistent utility function over time, so I wonder if self-modification or successor risks would be higher, to ensure consistency.)
And you could fix this to be insensitive to butterfly effects, by comparing quantile functions as random variables instead.
Could we be sure that a bare utility maximizer doesn’t modify itself into a mugging-proof version? I think we can. Such modification drastically decreases expected utility.
Maybe for positive muggings, when the mugger is offering to make the world much better than otherwise. But it might self-modify to not give into threats to discourage threats.
You could use a bounded utility function, with sufficiently quickly decreasing marginal returns. Or use difference-making risk aversion or difference-making ambiguity aversion.
Maybe also just aversion to Pascal’s mugging itself, but then the utility maximizer needs to be good enough at recognizing Pascal’s muggings.
FWIW, I’m not sure if that’s true relative to the average person, and I’d guess non-consequentialist philosophers are more averse to biting bullets than the average EA and maybe rationalist.
I think your scenario only illustrates a problem with outer alignment (picking the right objective function), and I think it’s possible to state an objective, that if it could be implemented sufficiently accurately and we could guarantee the AI followed it (inner alignment), would not result in a dystopia like this. If you think the model would do well at inner alignment if we fixed the problems with outer alignment, then it seems like a very promising direction and this would be worth pointing out and emphasizing.
I think the right direction is modelling how humans now (at the time before taking the action), without coercion or manipulation, would judge the future outcome if properly informed about its contents, especially how humans and other moral patients are affected (including what happened along the way, e.g. violations of consent and killing). I don’t think you need the coherency of coherent extrapolated volition, because you can still capture people finding this future substantially worse than some possible futures, including the ones where the AI doesn’t intervene, by some important lights, and just set a difference-making ambiguity averse objective. Or, maybe require it not to do substantially worse by any important light, if that’s feasible: we would allow the model to flexibly represent the lights by which humans judge outcomes where doing badly in one means doing badly overall. Then it would be incentivized to focus on acts that seem robustly better to humans.
I think an AI that actually followed such an objective properly would not, by the lights of whichever humans whose judgements it’s predicting, increase the risk of dystopia through its actions (although building it may be s-risky, in case of risks of minimization). Maybe it would cause us to slow moral progress and lock in the status quo, though. If the AI is smart enough, it can understand “how humans now (at the time before taking the action), without coercion or manipulation, would judge the future outcome if properly informed about its contents”, but it can still be hard to point the AI at that even if it did understand it.
Another approach I can imagine is to split up the rewards into periods, discount them temporally, check for approval and disapproval signals in each period, and make it very costly relative to anything else to miss one approval or receive a disapproval. I describe this more here and here. As JBlack pointed out in the comments of the second post, there’s incentive to hack the signal. However, as long as attempts to do so are risky enough by the lights of the AI and the AI is sufficiently averse to losing approval or getting disapproval and the risk of either is high enough, it wouldn’t do it. And of course, there’s still the problem of inner alignment; maybe it doesn’t even end up caring about approval/disapproval in the way our objective says it should out of distribution.
I would say it seems like it’s almost there, but it also seems to me to already have some fluid intelligence, and that might be why it seems close. If it doesn’t have fluid intelligence, then my intuition that it’s close may not be very reliable.
(Your reply is in response to a comment I deleted, because I thought it was basically a duplicate of this one, but I’d be happy if you’d leave your reply up, so we can continue the conversation.)
See here, starting from “consider a scheme like the following”. In short: should be possible, but seems non-trivially difficult.
That seems like a high bar to me for testing for any fluid intelligence, though, and the vast majority of humans would do about as bad or worse (but possibly because of far worse crystallized intelligence). Similarly, in your post, “No scientific breakthroughs, no economy-upturning startup pitches, certainly no mind-hacking memes.”
I would say to look at it based on definitions and existing tests of fluid intelligence. These are about finding patterns and relationships between unfamiliar objects and any possible rules relating to them, applying those rules and/or inference rules with those identified patterns and relationships, and doing so more or less efficiently. More fluid intelligence means noticing patterns earlier, taking more useful steps and fewer useless steps.
Some ideas for questions:
Invent new games or puzzles, and ask it to achieve certain things from a given state.
Invent new mathematical structures (e.g. new operations on known objects, or new abstract algebraic structures based on their axioms) and ask the LLM to reason about them and prove theorems (that weren’t too hard to prove yourself or for someone else to prove).
Ask it to do hardness proofs (like NP-hardness proofs), either between two new problems, or just with one problem (e.g. ChatGPT proved a novel problem was NP-hard here).
Maybe other new discrete math problems.
EDIT: New IMO and Putnam problems.
My impression is that there are few cross-applicable techniques in these areas, and the ones that exist often don’t get you very far to solving problems. To do NP-hardness proofs, you need to identify patterns and relationships between two problems. The idea of using “gadgets” is way too general and hides all of the hard work, which is finding the right gadget to use and how to use it. EDIT: For IMO and Putnam problems, there are some common tools, too, but if just simple pattern matching for those was all it took, math undergrads would generally be good at them, and they’re not, so it probably does take considerable fluid intelligence.
I guess one possibility is that an LLM can try a huge number of steps and combinations of steps before generating the next token, possibly looking ahead multiple steps internally before picking one. Maybe it could solve hard problems this way without fluid intelligence.
I think many EAs/rationalists shouldn’t find this to be worse for humans than life today on the views they apparently endorse, because each human looks better off under standard approaches to intrapersonal aggregation: they get more pleasure, less suffering, more preference satisfaction (or we can imagine some kind of manipulation to achieve this), but at the cost of some important frustrated preferences.
Suppose that some technology requires 10 components to get to work. Over the last decades, you’ve seen people gradually figure out how to build each of these components, one by one. Now you’re looking at the state of the industry, and see that we know how to build 9 of them. Do you feel that the technology is still a long time away, because we’ve made “zero progress” towards figuring out that last component?
This seems pretty underspecified, so I don’t know, but I wouldn’t be very confident it’s close:
Am I supposed to assume the difficulty of the last component should reflect the difficulty of the previous ones?
I’m guessing you’re assuming the pace of building components hasn’t been decreasing significantly. I’d probably grant you this, based on my impression of progress in AI, although it could depend on what specific components you have in mind.
What if the last component is actually made up of many components?
I agree with the rest of your comment, but it doesn’t really give me much reason to believe it’s close, rather than just closer than before/otherwise.
If they have zero fluid intelligence now, couldn’t it be that building fluid intelligence is actually very hard and we’re probably a long way off, maybe decades? It sounds like we’ve made almost no progress on this, despite whatever work people have been doing.
There could still be a decent probability of AGI coming soon, and that could be enough to warrant acting urgently (or so could non-AGI, e.g. more task-specific AI used to engineer pathogens).
For starters, suppose the AI straps lots of humans into beds, giving them endless morphine and heroin IV drips, and the humans get into such a state of delirium that they repeatedly praise and thank the AI for continuing to keep the heroin drip turned on.
This dystopian situation would be, to the AI, absolute ecstasy—much like the heroin to those poor humans.
This seems to require some pretty important normative claims that seem controversial in EA and the rationality community. Based on your description, it seems like the humans come to approve of this (desire/prefer this) more than they approved of their lives before (or we could imagine similar scenarios where this is the case), and gain more pleasure from it, and you could have their approval (by assumption) outweigh the violation of the preferences for this not to happen. So, if you’re a welfarist consequentialist and a hedonist or a desire/preference theorist, and unless an individual’s future preferences count much less, this just seems better for those humans than what normal life has been like lately.
Some ways out seem to be:
Maybe certain preference-affecting views, or discounting future preferences, or antifrustrationism (basically negative preference utilitarianism), or something in these directions
Counting preferences more or less based on their specific contents, e.g. wanting to take heroin is a preference that counts less in your calculus
Non-hedonist and non-preferential/desire-based welfare (possibly in addition to hedonistic and preferential/desire-based welfare), e.g. objective goods/bads
Non-welfarist consequentialist values, i.e. valuing outcomes for reasons other than how they matter to individuals’ welfare
Non-consequentialism, e.g. constraints on violating preferences or consent, or not getting affirmative consent
Actually, whatever preferences they had before and were violated were stronger than the ones they have now and are satisfied
Actually, we should think much bigger; maybe we should optimize with artificial consciousness instead and do that a lot more.
If it’s something like 1 or 5, it should instead (or also?) model what the humans already want, and try to get that to happen.
On this take, especially with your skepticism of LLM fluid intelligence and generality, is there much reason to expect AGI to be coming any time soon? Will it require design breakthroughs?
I’m not sure this is the most useful way to think about it, either, because it includes the possibility that we didn’t solve the Riemann hypothesis first just because we weren’t really interested in it, not because of any kind of inherent difficulty to the problem or our suitability to solving it earlier. I think you’d want to consider:
alternative histories where solving the Riemann hypothesis was a (or the) main goal for humanity, and
alternative histories where world takeover was a (or the) main goal for humanity (our own actual history might be close enough)
and ask if we solve the Riemann hypothesis at earlier average times in worlds like 1 than we take over the world in worlds like 2.
We might also be able to imagine species that could take over the world but seem to have no hope of ever solving the Riemann hypothesis, and I think we want to distinguish that from just happening to not solve it first. Depending on what you mean by “taking over the world”, other animals may have done so before us, too, e.g. arthropods. Or even plants or other forms of life more or before any group of animals, even all animals combined.
Even without ensuring inner alignment, is it possible to reliably train the preferences of an AGI to be more risk averse, to be bounded and to discount the future more? For example, just by using rewards in RL with those properties and even if the AGI misgeneralizes the objective or the objective is not outer aligned, the AGI might still internalize the intended risk aversion and discounting. How likely is it to do so? Or, can we feasibly hardcode risk aversion, bounded preferences and discounting into today’s models without reducing capabilities too much?
My guess is that such AGIs would be safer. Given that there are some risks to the AGI that the AGI will be caught and shut down if it tries to take over, takeover attempts should be relatively less attractive. But could it make much difference?
How feasible is it to build a powerful AGI with little agency but that’s good at answering questions accurately?
Could we solve alignment with one, e.g. getting it to do research, or using it to verify the outputs of other AGIs?
We’d want to pick something to
have badness per unit of resources (or opportunity cost) only moderately higher than any actually bad thing according to the surrogate,
scale like actually bad things according to the surrogate, and
be extraordinarily unlikely to occur otherwise.
Maybe something like doing some very specific computations, or building very specific objects.
I think the acausal economy would look aggressively space expansionist/resource-exploitative (those are the ones that will acquire and therefore control the most resources; others will self-select out or be out-competed) and, if you’re pessimistic about alignment, with some Goodharted human(-like) values from failed alignment (and possibly some bad human-like values). The Goodharting may go disproportionately in directions that are more resource-efficient and allow faster resource acquisition and use and successful takeover (against their creators and other AI). We may want to cooperate most with those using their resources disproportionately for artificial minds or for which there’s the least opportunity cost to do so (say because they’re focusing on building more hardware that could support digital minds).
Harmful, maybe? Not all harms involve destruction (physical or relationships, etc.).
I see a few comments here on fortified foods. I think the vitamin D and iron is usually in less bioavailable forms (D2 and non-heme iron) in fortified plant-based foods than in animal products, and I don’t know if % daily value on labels account for that. I take a multivitamin with apparently 100% of the daily value for both, and it wasn’t enough based on my bloodwork and lightheadedness/blackouts (few of the other vegans I know got lightheaded; I had been giving blood every 2 months, but even after I stopped, the multivitamin alone didn’t seem to be enough), so I take separate supplements for each on top of the multivitamin. I was also eating nuts, legumes, and kale or spinach for iron most days.
Looking at my soy milk (Silk), per cup, the label says it has only 2 micrograms of vitamin D (D2) and 10% of the daily value. Some of my other vegan dairy products have no vitamin D. There’s no vitamin D in Beyond Burgers, but there is a decent amount of (non-heme) iron per patty (5.5 mg, 31% DV). Similarly for my Yves veggie chicken. I’d be surprised if most vegans not taking supplements with vitamin D even reach 100% DV according to the labels (whether they account for bioavailbility or not) through diet alone.
I live in Canada, though, so vitamin D is harder to get from the sun during the winter.