If I knew it was going to happen in 2035 to 2039, I think I would mostly back up and try to improve the overall quality of US governance, and/or work to get competent candidates for the 2034 presidential election.
Do you mean the quality of US governance with respect to AI or in general? In the latter case, I’m curious what your concrete plans would be, since this is usually considered a difficult and not very neglected (but still very important!) area.
I think when Olah says that solving alignment may be as “easy” as the steam engine, he’s basically envisioning current training + eval techniques (or similar techniques equivalently difficult to the steam engine) scaling all the way to superintelligence. (This is my interpretation; I might be wrong here.) For instance, maybe inducing corrigibility in ASI turns out to be not that difficult, such that the “first critical try” framework does not really apply, and takeoff is slow enough that model organisms/evals work means we can test our alignment methods and have them reasonably generalize to real world scenarios. Disagreeing with this view just means that “alignment” is harder than the steam engine scenario.
I realize I’m being a little pedantic here, but on the “joke” calculation: the problem here is that is a binary function depending on whether utilitarians join or not, right? For instance, let be the effective safety premium from safety-minded utilitarians joining (the value being negative as joining presumably accelerates the company), and suppose that each utilitarian joining leads to acceleration. Then a rational utilitarian would demand premium, which is not negligible.
Going back to the joke calculation, it implies that the bottleneck to preventing defection is coordination: utilitarians acting together would not join for value as it is against their interests, but individually they have zero counterfactual impact, so they all join. In the real world, coordination is plausibly relevant, but it seems like the more relevant problem is just the object-level beliefs of those joining: how impactful is your work versus how much it accelerates capabilities, along with vague non-utilitarian desires like status that bias people towards joining the big labs.
I recommend https://agentfoundations.study/, and much of https://www.aisafety.com/stay-informed,
Currently these two links include the commas so they redirect to 404 pages
We (or at least a majority of humans) do still have inner desires to have kids, though; they just get balanced out by other considerations, mostly creature comforts/not wanting to deal with the hassle of kids. But yeah, evolution did not foresee birth control, so that’s a substantial misgeneralization.
We are still a very successful species overall according to IGF, but birth rates continue to decline, which is why I made my last point about inner alignment possibly drifting farther and farther away the stronger the inner optimizer (e.g. human culture) becomes.
I saw that Katja Grace has said something similar here; I’m just putting my own spin on the idea.
The relevance of the evolutionary analogy for inner alignment has been long discussed in this community, but one observation that seems to not be mentioned is that humans are still… pretty good at inclusive genetic fitness? Even in way-out-of-distribution environments like modern society, we still have strong desires to eat food, stay alive, find mates and reproduce (although the last one has relatively decreased recently; IGF hasn’t totally generalized). We don’t monomanically optimize for IGF, but we (and probably future NN-based AIs) don’t monomanically optimize for anything, and our motivational circuits still do a pretty good job at keeping our species alive. So… why should we expect future AIs to catastrophically fail (i.e. be completely non-inner aligned with what we wanted it to do) at doing the actions we rewarded in RL training, which should be a much stronger outer optimizer than evolution?
Some possible objections:
“Human values are more fragile than IGF, so it’s much easier to catastrophically fail on human values”
Is this true? Is it really easier to misgeneralize on human values than on IGF? Maybe, but we have a lot of animal skulls on the road that say otherwise
More relevantly, modern LLMs have already learned human values pretty well, so the difficulty of enacting said values shouldn’t matter as much if the concepts already exist in the weights (I’m less sure about this)
“Optimizing a generally intelligent, situationally aware agent presents unique challenges compared to evolution because of scheming, gradient hacking, wireheading, etc.”
Sure! This definitely seems like a problem. However, by the time the AI gains the capabilities needed for scheming, its inner alignment may have to be absolutely terrible for catastrophic effects to occur once out of training, as otherwise we end up in the “mostly fine” state that evolution stumbled into with humans.
“AIs could drift off over time in the same way that humans seem to be currently with evolution”
Yep, this also seems like a problem. Hopefully general capabilities allow a value-aligned AI to strategically preserve its values over time. We could also continually optimize our AIs; gradient descent hopefully never becomes ~billions of times weaker than any inner optimizer like evolution is versus human culture.
The thing is, there exists lots of popular movies about rogue AIs taking over the world -- 2001, Terminator, etc etc—so the concept should already exist in popular culture. The roadblocks seem to be:
The threat somehow doesn’t seem as tangible or threatening as, for example, ISIS developing a bioweapon or the CCP permanently dominating the world. One explanation is that the reference class for “enemy does bad things with new technology” or other near-term threat models contains lots of examples throughout history, whereas “species smarter than humans” contains none. Related:
The threat doesn’t seem realistic, i.e. people (even those who want to accelerate towards AGI) have long timelines. Hypothetically, if you truly “feel the AGI” and understand that we’re close to building something smarter than us in every way, the idea that we should make sure it does what we want should be intuitive. I don’t know if making people “feel the AGI” is a smart PR strategy, but nevertheless this does seem to still be a barrier to the public taking AGI risk seriously.
In this case, the starving person presumably has to press the button or else starve to death, and thus has no bargaining power. The other person only has to offer the bare minimum beyond what the starving person needs to survive, and the starving person must take the deal. In Econ 101 (assuming away monopolies, information asymmetry, etc.), exploited workers do have bargaining power by being able to work for other companies, hence why companies can’t just do stupid, spiteful actions in the long term.
It might be relevant to note that the meaningfulness of this coherence definition depends on the chosen environment. For instance, in an deterministic forest MDP where an agent at a state can never return to for any and there is only one path between any two states, suppose we have a deterministic policy and let , , etc. Then for the zero-current-payoff Bellman equations, we only need that for any successor from , for any successor from , etc. We can achieve this easily by, for example, letting all values except be near-zero; since is a successor of iff (as otherwise there would be a cycle), this fits our criterion. Thus, every is coherent in this environment. (I haven’t done the explicit math here, but I suspect that this also works for non-deterministic and non-stochastic MDPs.)
Importantly, using the common definition of language models in an RL setting where each state represents a sequence of tokens and each action adds a token to the end of a sequence of length to produce a sequence of length , the environment is a deterministic forest, as there is only one way to “go between” two sequences (if one is a prefix of the other, choose the remaining tokens in order). Thus, any language model is coherent, which seems unsatisfying. We could try using a different environment, but this risks losing stochasticity (as the output logits of an LM is determined by its input sequence) and gets complicated pretty quickly (use natural abstractions/world model as states?).
Right, I think this somewhat corresponds to the “how long it takes a policy to reach a stable loop” (the “distance to loop” metric), which we used in our experiments.
What did you use your coherence definition for?
Yeah, it’s a mishmash of all the runs, including on-policy and RL runs.
I wouldn’t expect this to be true for LR=5e-5 (where, of course, we see many instances of transfer), at least based on our IFEval box plot. I do agree that some of the high-LR runs are worse than ideal, though. If it’s helpful, I think I’ll add a graphic showing us trying to train back in IFEval capabilities while preserving the alternate behavior rate, similar to Olympiads (it’s pretty low-cost to do). I think the capability degradation box plot clearly shows that transfer is quite doable without the hypothesis you mention, however.