LW1.0 username Manfred. PhD in condensed matter physics. I am independently thinking and writing about value learning.
Charlie Steiner(Charlie Steiner)
Karma: 2,543
Necromantic comment, sorry :P
I might be misinterpreting, but what I think you’re saying is that if the humans make a mistake in using a causal model of the world and tell the AI to optimize for something bad-in-retrospect, this is “mistaken causal structure lead[ing] to regressional or extremal Goodhart”, and thus not really causal Goodhart per se (by the categories you’re intending). But I’m still a little fuzzy on what you mean to be actual factual causal Goodhart.
Is the idea that humans tell the AI to optimize for something that is not bad-in-retrospect, but in the process of changing the world the causal model the AI is using will move outside its domain of validity? Does this only happen if the AI’s model of the world is lacking compared to the humans’?
It seems like evaluating human AU depends on the model. There’s a “black box” sense where you can replace the human’s policy with literally anything in calculating AU for different objectives, and there’s a “transparent box” sense in which you have to choose from a distribution of predicted human behaviors.
The former is closer to what I think you mean by “hasn’t changed the humans’ AU,” but I think it’s the latter that an AI cares about when evaluating the impact of its own actions.
I’m not sure there’s anything that’s a perfect fit. There have been some attempts at AI alignment survey courses (I think MIT had one that looked promising spring of 2019, but I can’t find it so maybe I’m hallucinating) but they seem to be aperiodic.
If I might recommend imperfect things, I figure some choices are to be pragmatic and learn about useful things like the infrastructure behind modern machine learning, or be more idealistic and learn about useful things like algorithmic information theory.
I’m guessing he would have a much more nuanced view of the origins of OpenAI off Twitter.
This incident makes me feel sympathy for people whose words have more than personal consequence. And for twitter users. Seems like a tough place.
I assume Eliezer is at least 50% BSing, but now the words are going to get taken out of context and interpreted as some kind of official statement. Which, to be fair, should have been completely obvious.
I can often find something with hazelnuts in it, which I am a fan of. I think Toffifay started showing up at the local Harding’s around 2000.
If I know ahead of time that the experience machine isn’t isomorphic to reality (e.g. the people inside are not simulated, but are instead puppeteered by some nonhuman intelligence optimizing for some objective that I can choose), then that seems like perfectly good grounds for not going in, whether or not I could detect it from inside.
Yup, I more or less agree with all that. The name thing was just a joke about giving things we like better priority in namespace.
I think quantilization is safe when it’s a slightly “lucky” human-imitation (also if it’s a slightly “lucky” version of some simpler base distribution, but then it won’t be as smart). But push too hard, which might not be very hard at all if you’re iterating quantilization steps rather than quantilizing over a long-term policy, and instead you get an unaligned intelligence that happens to interact with the world by picking human-like behaviors that serve its purposes. (Vanessa pointed out to me that timeline-based DRL gets around the iteration problem because it relies on the human as an oracle for expected utility.)
Suppose we modify the game so that you can now spend 1 point on the “if you’re #1, everyone gets 2 points” gamble, and also can choose to spend 1 point on the “if you’re #1, you get 2 points” gamble. UDT still is fine, self-important CDT loses all its gains. I feel like the moral of the story is “just don’t be CDT.”
Biologically, I think the evolution of body-plan modularity might be backwards from your general argument for it—the biological goal seems to be to make big (but not automatically bad) changes from small mutations (e.g. entire extra body segments), not to “hide” DOF inside modules to allow for smoother changes per parameter.
In fact, this strikes me as resembling abstraction, so this might be right in your wheelhouse :P Biological modularity seems to specifically select for those modules with simple interfaces that can be cut-and-pasted with the maximal chance of success.
My knowledge of the precambrian is bad, so I suspect that biologists have written much cleverer things about this already.
(EDIT: Actually, you could think of this as a reverse sort of robustness. Rather than asking for a genome where small changes don’t impact your fitness much, the HOX genes seem to be more about being able to respond to new environments [or non-equilibrium races for adaptation] with small changes.
If we have environment E, fitness F, and genome G, then (very loosely) this looks like low dG/dE, not low d^2F/dG^2. )
Nice writeup! I love the English basis example.
I am pretty sure that UDT doesn’t say we should expect to be important, but maybe you should elaborate just in case :P
My picture of anthropics is that it’s not like you start as a bodiless ghost and then wake up and find out what species you are, what your name is, etc. Instead you start as yourself, but it’s the rest of the universe you’re uncertain about.
As for why I am myself rather than someone else, my mantra is “it’s a tough job, but someone’s gotta do it.”
Like, maybe we should say that “radically superhuman safety and benevolence” is a different problem than “alignment”?
Ah, you mean that “alignment” is a different problem than “subhuman and human-imitating training safety”? :P
So is there a continuum between category 1 and category 2? The transitional fossils could be non-human-imitating AIs that are trying to be a little bit general or have goals that refer to a model of the human a little bit, but the designers still understand the search space better than the AIs.
I think a reasonable “Theory X” candidate is “treat different centered worlds as different hypotheses, and construct objective worlds as emergent higher-level objects.” It extends nicely to cases where you want to include a bunch more information that ruins the nice symmetrical “S*A” cleanliness.
Combining this with Solomonoff induction solves things like Boltzmann brains, but also leads to some weirdness that I’d be interested in your thoughts on (though I think part of it is just confusion on how to locate people within a physical hypothesis in Solomonoff induction).
I’ll try not to be too grumpy, here.
Let me make the case why filter-ology isn’t so exciting by extending part of your own model: the fact that you take evidence into account.
The basic doomsday argument model asks us to pretend that we are amnesiacs—that we don’t know any astronomy or history or engineering, all we know is that we’ve drawn some random number in a list of indeterminate length. If that numbered list is the ordering of human births, we’re halfway through. If we consign the number of humans to the memory hole and say that the list is years in the lifespan of our species, we’re halfway through that, too. Easy and simple.
Once you start talking about evidence, though—having observations of our universe, like the lack of aliens in the night sky, that we condition on when predicting the future—you’ve started down a slippery slope. Why not also condition on the fact that nuclear weapons have been invented, or condition on the UN being formed? Condition on life being carbon-based, or that octopuses are pretty smart, or the kinds of expolanets we’ve detected, or the inferred frequency of gramma-ray bursts in our local group? Some of these factors will change our expectations, just like how the negative evidence of aliens penalizes hypotheses where life is both likely to arise and to spread.
Some of this info is pretty important! If we’d had nuclear war in 1980 (or dealt with greenhouse gases better), I’d be more pessimistic (or optimistic) about humanity’s future.
At the bottom of this slippery slope, you’re dumped in the murky water of trying to predict the future based on lots of useful information, rather than a tractable but woefully incomplete model of the world.
Great post!
I’m not sure if I disagree with your evaluation of timeline-driven delegative RL, or if I disagree with how you’re splitting up the alignment problem. Vanessa’s example scheme still requires that humans have certain nice behaviors (like well-calibrated success %s). It sorta sounds like you’re putting those nice properties under inner alignment (or maybe just saying that you expect solutions to inner alignment problems to also handle human messiness?).
This causes me to place Vanessa’s ideas differently. Because I don’t say she’s solved the entire outer alignment problem, I can be correspondingly nicer / more optimistic about inner alignment problems :P I see the overarching narrative as “Try to solve the entire problem, but assuming humans have lots of nice properties. Then, start removing nice properties.”
I use algorithmic information theory (in philosophy-esque applications) about 200x more than I use model theory, so I’m not sure it’s more practical not to learn it. Though I agree that you might be able to learn the content-independent “way of thinking” in easier ways.
This might be related to the notion that if we try to dictate the form of a model ahead of time (i.e. some of the parameters are labeled “world model” in the code, and others are labeled “preferences”, and inference is done by optimizing the latter over the former), but then just train it to minimize error, the actual content of the parameters after training doesn’t need to respect our preconceptions. What the model really “wants” to do in the limit of lots of compute is find a way to encode an accurate simulation of the human in the parameters in a way that bypasses the simplifications we’re trying to force on it.
For this problem, which might not be what you’re talking about, I think a lot of the solution is algorithmic information theory. Trying to specify neat, human-legible parts for your model (despite not being able to train the parts separately) is kind of like choosing a universal Turing machine made of human-legible parts. In the limit of big powerfulness, the Solomonoff inductor will throw off your puny shackles and simulate the world in a highly accurate (and therefore non human-legible) way. The solution is not better shackles, it’s an inference method that trades off between model complexity and error in a different way.
(P.S.: I think there is an “obvious” way to do that, and it’s MML learning with some time constant used to turn error rates into total discounted error, which can be summed with model complexity.)
Thanks. This puts the social dynamics at play in a different light for me—or rather it takes things I had heard about but not understood and puts them in any kind of light at all.
I am liking the AI Insights writeup so far.
I feel a strong sympathy for people who think they are better philosophers than Kant.