I’m interested in doing in-depth dialogues to find cruxes. Message me if you are interested in doing this.
I do alignment research, mostly stuff that is vaguely agent foundations. Currently doing independent alignment research on ontology identification. Formerly on Vivek’s team at MIRI.
While I agree with this as a philosophical stance, it seems clear that, at least in humans, facts and values are in practice entangled.
I agree humans absorb (terminal) values from people around them. But this property isn’t something I want in a powerful AI. I think it’s clearly possible to design an agent that doesn’t have the “absorbs terminal values” property, do you agree?
Even if the terminal value doesn’t change, there are facts that could result in behavior that, to a reasonable outside observer, seems to reflect a conflicting value.
Yeah! I see this as a different problem from the value binding problem, but just as important. We can split it into two cases:
The new beliefs (that lead to bad actions) are false.[1] To avoid this happening we need to do a good job designing the epistemics of the AI. It’ll be impossible to avoid misleading false beliefs with certainty, but I expect there to be statistical learning type results that say that it’s unlikely and becomes more unlikely with more observation and thinking.
The new beliefs (that lead to bad actions) are true, and unknown to the human AI designers. (e.g. we’re in a simulation and the gods of the simulation have set things up such that the best thing for the AI to do looks evil from the human perspective. The AI is acting in our interest here. Maybe out of caution we want to design the AI values such that it wants to shut down in circumstances this extreme, just in case there’s been an epistemic problem and it’s actually case 1.
I’m assuming a correspondence theory of truth, where beliefs can be said to be true or false without reference to actions or values. This is often a crux with people that are into active inference or shard theory.
I think you’re mostly right about the problem but the conclusion doesn’t follow.
First a nitpick: If you find out you’re being manipulated, your terminal values shouldn’t change (unless your mind is broken somehow, or not reflectively stable).
But there’s a similar issue: You could discover that all your previous observations were fed to you by a malicious demon, and nothing that you previously cared about actually exists. So your values don’t bind to anything in the new world you find yourself in.
In that situation, how do we want an AI to act? There’s a few options, but doing nothing seems like a good default. Constructing the value binding algorithm such that this is the resulting behaviour doesn’t seem that hard, but it might not be trivial.
Does this engage with what you’re saying?
Yeah makes sense. I don’t want to make it harder to write stuff though. The contrast does make the shortform rhetorically better and that is good. With these comments as context, it doesn’t seem super necessary to edit it.
Thus, the original argument (the “Value Misspecification Argument”) is wrong and the people who believed it should at least stop believing it).
That post is confused about what MIRI ever believed, and john’s comment does a really good job of explaining why. I’m guessing you put that first paragraph in for for rhetorical contrast, rather than thinking it’s a true summary of any particular person’s past beliefs, but I think doing this is corrosive to group epistemics. It’s really handy to be able to keep track of what people believed and whether they updated on new evidence, and this process is damaged when people misrepresent what other people previously believed.
I don’t currently understand why you’re focusing so much on synergistic information, rather than just using redundancy to do the whole abstraction hierarchy.
I think I see why you’ve gone that way, because examples with synergistic information aren’t able to be broken down cleanly. I agree that this is annoying, but it can just be ignored (like in Condensation, which just becomes more of an approximate result in those cases).
Another approach is to add another mechanism for breaking up the model, other than information independence. For example, you could allow causal cross-links in synergistic cases. If that makes sense? Z should be thought of as causally downstream of X and Y, and if you allow this link to be encoded in your model, then you get to break up X,Y,Z without increasing the complexity of the model overall.
So under the Standard Model, future states do pin down the entire past.
Good point, I was thinking in terms of toy markov chains rather than real physics when I said that.
would it produce any qualitatively different conclusions here?
It would change the conclusions a lot if the initial conditions were unstructured noise, and the laws of physics were very simple, because then the AMSS would just contain the laws of physics and there’d be no compression benefit from multi-level structure.
Interesting. Elaborate?
It requires something like an IID assumption, so it’s fairly useless for your purposes. But once we’ve got that assumption, then we can bound |empirical error—generalisation error| with a function of hypothesis complexity and the number of data points (Chapter 7.2 in Understanding Machine Learning). So we can say simpler hypotheses will continue to work roughly as well as they worked on the training data, without any reference to whether they are true or not, or any philosophical justification of the prior.
The IID assumption really sucks though, I really want there to be some weaker assumption that lets us conclude a similar thing. And intuitively this should be a thing that exists, because in science and in real life, we constantly learn to use simple approximate rules-of-thumb, but we justify them in a similar way. “The rule has a good track record and I don’t see any reason it won’t work in this next specific case”.
There’s a somewhat more Bayesian way of doing the same thing, where you do logical induction over whether the “true underlying hypothesis” implies the rule-of-thumb you’ve observed. But I think if you pull apart how the logical induction is working there, it has to be doing something like the frequentist thing.
The idea that there’s a simple state in the future, that still pins down the entire past, seems possible but weird. Most of the time when events evolve into a simple state, it’s because information is destroyed. This isn’t really a counter-argument, it’s just trying to put into words what feels odd.
One thing that’s confusing to me: Why K-complexity of the low-level history? Why not, for example, Algorithmic Minimal Sufficient Statistic, which doesn’t count the uniform noise? Or memory-bounded K-complexity, which might also favour multi-level descriptions.
I think I prefer frequentist justifications for complexity priors, because they explain why it works even on small parts of the universe.
But this explanation doesn’t sit well with me, because under this kind of prior, the fundamental laws of physics being so simple is really surprising.
Then, we could exploit it to compress the description of the full-fidelity/lowest-level history.
I don’t think this works if the lowest level laws of physics are very very simple. The laws of physics at the lowest level + initial conditions are sufficient to roll out the whole history, so (in K-complexity) there’s no benefit to adding descriptions of the higher levels.
Maybe if lots of noise is constantly being injected into the universe, this would change things. Because then the noise counts as part of the initial conditions. So the K-complexity of the universe-history is large, but high-level structure is common anyway because it’s more robust to that noise?
What’s the anthropic prior? Is this something discussed elsewhere? What is it adding over the Solomonoff prior?
I haven’t finished working through your post yet, but you might be interested in Condensation, which seems quite similar.
Can’t you pirate the ebook? Seems fine if you’ve already paid for it.
I think John’s anti-control post is the best I know of
I’ve been meaning to write a post like this for a while. I think your conclusion is right and all of your arguments are true, but I don’t think most of it engages with a strong version of the superalignment argument, so I don’t think a proponent of superalignment-like plans would find this convincing.
When discussing the quality of a plan, we should assume that a company (and/or government) is really trying to make the plan work and has some lead time. If they do, then the second and fourth arguments can be dismissed.
Stopping self-improvement by restricting actions and very heavily supervising seems maybe doable (if risky[1]) for near-human-level AI (unless the capabilities profile is uneven in an inconvenient way), so the first argument is also unconvincing.
I think 3 is a strong argument, and I think it’s crazy how bad the proposals I’ve read are (about what research the AIs should do).
I think the strongest argument is that it’ll take a lot of researcher-labour to distinguish good from bad research, so it overall doesn’t speed up research much. This is because our only way to train for good research is to train for research that is very difficult to distinguish from good research, and it takes a lot of labour to avoid being misled by good-looking-but-not-quite-good research. (You do make this argument in several places, but in less detail).
Proponents often say that verification is easier than generation, and this is sometimes true but not true enough for most kinds of research. Verification is also time consuming, and more so the better the AI is at sweeping errors under the rug. And they are already really good at sweeping errors under the rug, I find it really annoying to have an LLM help me with proofs because of how often it makes hard-to-spot mistakes. Often it feels like it slows me down overall.
So you need a lot of researcher-labour for the training signal, and you also need a lot to verify any alignment progress produced. (On top of this, the signal is gonna be super noisy and researchers will disagree about whether research-outputs have merit). Since skilled-researcher-labour will be the bottleneck, it’s not clear that the plan actually accelerates research progress that much. It probably does a bit, but I don’t know how to work out how much, and intuitively it seems unlikely to be more than 3x. It’s plausible it slows it down on net, if mistakes sometimes get past the researchers.
In my experience, this argument about the quantity of labour involved seems to occasionally convince people who believed in superalignment.
Especially risky given that other countries will likely try to steal the AGI.
You’re right, I edited it.
That makes sense about local validity.
The way that the analogy interacts with other assumptions seems crucial. I don’t mean to insult Will, if it helps I also think there are a bunch of strawmen in IABIED. But I think most readers whose attention was drawn to the following quote would understand that the evolution analogy needs to be combined with the other things listed there to conclude that alignment is very difficult.
“If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation. But when some of the problems stay out of sight? When some complications inevitably go unforeseen? When the AIs are grown rather than crafted, and no one understands what’s going on inside of them?”
..
If that’s what the argument is about, it’s really important for the authors to explain how that argument connects to the broader notion of training used in practice, and I don’t remember this happening. I don’t remember them talking carefully about the still broader question of “what happens when you do get to examine results and intermediate results and make adjustments based on observations?”
Neither do I, but this doesn’t seem really important for a non-researcher audience. Conditional on the claims that weird goal errors are difficult to understand by examining behaviour, and interventions to patch weird goal errors often don’t generalise well. If you buy those claims, then it’s easy to extrapolate what happens when you examine results and make adjustments based on those.
I’m not trying to debate or gotcha. I agree that if I tried to do adversarial nitpicking at IABIED I could make it sound equally bad. I found Will’s review convincing, in the sense that it intuitively snapped me into the worldview where the evolutionary analogy isn’t a good argument. I spent the day thinking about it, and I wrote out my own steelman of it that extrapolated details, and re-evaluated whether I thought the original argument was valid, and decided that yeah it still was. This exercise was partially motivated by you saying that your complaints were similar in another comment.
Then I went through and found the important differences between my steelman-will-beliefs and my actual beliefs, the places where I thought it was locally making a mistake, and wrote them down, and then turned that into this shortform. I framed it as misrepresenting after re-reading chapter 4 to check how my argument matched up. Maybe this was a bad way to write it up. It definitely feels like he’s doing the opposite of steelmanning, not particularly trying to convey a good version of the argument in the book, or understand the coherent worldview that produced it.
But it’s an honest guess that this is a thing Will is missing (how the evolution analogy should be scoped, and how the other premises are separate from it and also necessary). The guess was constructed without knowing Will or reading much of his other writing, so I admit it’s pretty likely to be wrong, but if so maybe someone will explain how.
But either way, I figured it’s particularly worth publishing this particular part of the things I wrote today because of how often I hear people misunderstand the evolution analogy.
Sorry I intended single training run to refer to purely running SGD on a training set. As opposed to humans examining the result or intermediate result and making adjustments based on their observations. So at least 2 & 3, as they definitely involve human intervention.
I think Will MacAskill’s summary of the argument made in Chapter 4 of IABIED is inaccurate, and his criticisms don’t engage with the book version. Here’s how he summarises the argument:
The evolution analogy:
Illustrative quote: “To extend the [evolution] analogy to AI: [...] The link between what the AI was trained for and what it ends up caring about would be complicated, unpredictable to engineers in advance, and possibly not predictable in principle.”
Evolution is a fine analogy for ML if you want to give a layperson the gist of how AI training works, or to make the point that, off-distribution, you don’t automatically get what you trained for. It’s a bad analogy if you want to give a sense of alignment difficulty.
Y&S argue:
Humans inventing and consuming sucralose (or birth control) would have been unpredictable from evolution’s perspective, and is misaligned with the goal of maximising inclusive genetic fitness.
The goals of a superintelligence will be similarly unpredictable, and similarly misaligned.
On my reading, the argument goes more like this:[1]
The analogy to evolution (and a series of AI examples) is used to argue that there is a complicated relationship between training environment and preferences (in a single training run!), and that we don’t have a good understanding of that relationship. The book uses “complications” to refer to weird effects in the link between training environment and resulting preferences.
From this, the book concludes: Alignment is non-trivial, and shouldn’t be attempted by fools.[2]
Then it adds some premises:[3]
It’s difficult to spot some complications.
It’s difficult to patch some complications.
The book doesn’t explicitly draw a conclusion just from purely the above premises, but I imagine this is sufficient to conclude some intermediate level of alignment difficulty, depending on the difficulty of spotting and patching the most troublesome complications.
The book adds:[4]
Some complications may only show up behaviourally after a lot of deliberation and learning, making them extremely difficult to detect and extremely difficult to patch.
Some complications only show up after reflection and self-correction.
Some complications only show up after AIs have built new AIs.
From this it concludes: It’s unrealistically difficult to remove all complications, and shouldn’t be attempted with anything like current levels of understanding.[5]
So MacAskill’s summary of the argument is inaccurate. It removes all of the supporting structure that makes the argument work, and pretends that the analogy was used by itself to support the strong conclusion.
He goes on to criticise the analogy by pointing at (true) dis-analogies between evolution and the entire process of building an AI:
But our training of AI is different to human evolution in ways that systematically point against reasons for pessimism.
The most basic disanalogy is that evolution wasn’t trying, in any meaningful sense, to produce beings that maximise inclusive genetic fitness in off-distribution environments. But we will be doing the equivalent of that!
That alone makes the evolution analogy of limited value. But there are some more substantive differences. Unlike evolution, AI developers can:
See the behaviour of the AI in a very wide range of diverse environments, including carefully curated and adversarially-selected environments.
Give more fine-grained and directed shaping of individual minds throughout training, rather than having to pick among different randomly-generated genomes that produce minds.
Use interpretability tools to peer inside the minds they are creating to better understand them (in at least limited, partial ways).
Choose to train away from minds that have a general desire to replicate or grow in power (unlike for evolution, where such desires are very intensely rewarded).
(And more!)
So these dis-analogies don’t directly engage with the argument. If they were directly engaging with the first part of the argument, they would be about the predictability of a single training run, rather than the total AI research process.
The dis-analogies could be reinterpreted as engaging with the other premises: 1 and 3 could be interpreted as claims that we do have (potentially yet to be invented) methods of detecting complications, and 2 and 4 could be interpreted as claims that we similarly have methods to patch complications. But the examples of unintended LLM weirdness given throughout the book are enough to show that at least current techniques aren’t working well for detection or patching of weirdness.
More importantly, the examples of particularly difficult complications entirely ignored, despite their necessity for the conclusion.
I’ve reordered it for logical clarity.
From the book: “If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation.”
These are mentioned in the book (e.g. in footnote 2), and the first is briefly supported with examples. The second isn’t supported in this chapter.
In the paragraphs following “So far, we’ve only touched on the sorts of complications that would arise in the preferences trained directly into an AI.”
“Problems like this are why we say that if anyone builds it, everyone dies.”
1 vote
Overall karma indicates overall quality.
0 votes
Agreement karma indicates agreement, separate from overall quality.
Okay. I think this anthropic theory makes a falsifiable prediction (in principle). The infinite precision real numbers could be algorithmically simple, or they could be unstructured. The theory predicts that they are not algorithmically simple. If it were the case that they were algorithmically simple, we could run a solomonoff inductor on the macrostates and it would recover the full microstates (and this would probably be simpler than the abstraction-based compression).