LW1.0 username Manfred. Day job is condensed matter physics, hobby is thinking I know how to assign anthropic probabilities.
Charlie Steiner(Charlie Steiner)
Karma: 2,072
I don’t think we disagree too much, but what does “play the right functional role” mean, since my desires are not merely about what brain-state I want to have, but are about the real world? If I have a simple thermostat where a simple bimetallic spring opens or closes a switch, I can’t talk about the real-world approximate goals of the thermostat until I know whether the switch goes to the heater or to the air conditioner. And if I had two such thermostats, I would need the connections to the external world to figure out if they were consistent or inconsistent.
In short, the important functional role that my desires play does not just take place intra-cranially, they function in interaction with my environment. If you were a new superintelligence, and the first thing you found was a wireheaded human, you might conclude that humans value having pleasurable brain states. If the first thing you found were humans in their ancestral environment, you might conclude that they value nutritious foods or producing healthy babies. The brains are basically the same, but the outside world they’re hooked up to is different.
So from the premises of functionalism, we get a sort of holism.
Thanks for the pointer to this interesting talk.
I do wish he’d backed up his optimism about exponential growth with a bit more inside view.
I feel like we can approximately split the full alignment problem into two parts: low stakes and handling catastrophes.
Insert joke about how I can split physics research into two parts: low stakes and handling catastrophes.
I’m a little curious about whether assuming fixed low stakes accidentally favors training regimes that have the real-world drawback of raising the stakes.
But overall I think this is a really interesting way of reframing the “what do we do if we succeed?” question. There is one way it might be misleading, which is I think that we’re left with much more of the problem of generalizing beyond the training domain that it first appears: even though the AI gets to equilibrate to new domains safely and therefore never has to take big leaps of generalization, the training signal itself has to do all the work of generalization that the trained model gets to avoid!
Very interesting! More interesting to me than the last time I looked through your proposal, both because of some small changes I think you’ve made but primarily because I’m a lot more amenable to this “genre” than I was.
I’d like to encourage a shift in perspective from having to read preferences from the brain, to being able to infer human preferences from all sorts of human-related data. This is related to another shift from trying to use preferences to predict human behavior in perfect detail, to being content to merely predict “human-scale” facts about humans using an agential model.
These two shifts are related by the conceptual change from thinking about the human preferences as “in the human,” thus being inextricably linked to understanding humans on a microscopic level, to thinking about human preferences as “in our model of the human”—as being components that need to be understood as elements of an intentional-stance story we’re telling about the world.
This of course isn’t to say that brains have no mutual information with values. But rather than having two separate steps in your plan like “first, figure out human values” and “later, fit those human values into the AI’s model of the world,” I wonder if you’ve explored how it could work for the AI to try to figure out human values while simultaneously locating them within a way (or ways) of modeling the world.
Thanks for fixing the formatting!
We’re pretty sure dark matter is just stuff that doesn’t interact much except through gravity (see https://www.lesswrong.com/posts/rNFzvii8LtCL5joJo/dark-matters ). And we think we know how gravity behaves with quantum mechanics when gravity is weak (though we have experiments set to test what we think we know, see https://www.newscientist.com/article/free-fall-experiment-test-gravity-quantum-force/ ).
Congrats!
I’ll procrastinate from thesis-writing to fill out the form :)
Wow, I’m really sorry for my bad reading comprehension.
Anyhow, I’m skeptical that scientist AI part 2 would end up doing the right thing (regardless of our ability to interpret it). I’m curious if you think this could be settled without building a superintelligent AI of uncertain goals, or if you’d really want to see the “full scale” test.
This runs headfirst into the problem of radical translation (which in AI is called “AI interpretability.” Only slightly joking.)
Inside our Scientist AI it’s not going to say “murder is bad,” it’s going to say “feature 1000 1101 1111 1101 is connected to feature 0000 1110 1110 1101.” At at first you might think this isn’t so bad, after all, AI interpretability is a flourishing field, let’s just look at some examples and visualizations and try to figure out what these things are. But there’s no guarantee that these features correspond neatly to their closest single-world English equivalent, especially after you try to generalize them to new situations. See also Kaj’s posts on concept safety. Nor are we guaranteed uniqueness—the Scientist AI doesn’t have to form one single feature for “murder” that has no neighbors, there might be a couple hundred near-equivalents that are difficult for us to tease apart.
Edit: reading comprehension is hard, see Michele’s reply.
Well, you sure weren’t falsely advertising :)
I think you’ve got SR wrong, obv. Maybe because you’re not imagining changes of reference frame to mix time and space coordinates for far-away places? Here’s a lorentz transform gif, showing how events (spacetime points) change when you change velocity : https://i.stack.imgur.com/O4OnL.gif The points move along these hyperbolas. So “length-contracting” your way to alpha centauri looks like moving a point that is in your future lightcone but very far away, until it’s directly in your future direction (you’ll go straight there in free-fall).
The reason you can’t use this to set an absolute reference frame is because the lorentz transformation effect moves things in both time and space, in a way that means the “present distance” (in the way that you think is the present in your reference frame) will be smallest precisely when you are stationary relative to it (i.e. when it’s a vertical line in a spacetime diagram). This means that if there are two different stars out there moving at different velocities, the longest distance to them will be measured at different velocities.
Look at that lorentz transform gif again, but try to pick out diagonal lines of points with your eye! Diagonal lines are moving objects. Watch what happens to all sorts of diagonal lines, particularly their “present distance” along the x-axis.
The other thing I’d argue against here is drawing too much from the copernican principle. Not thinking our scale is special doesn’t have to mean we have strong evidence that all scales are the same—we could be equally virtuous copernicans just be being uncertain about the laws of physics at different scales in a symmetrical way. Then if we do experiments and see differences at different scales, we can update that uncertainty to learn about the world.
I think the most likely is lack of funding for intermediate steps. But I think if it happens, it’s not going to be coordinated action motivated by AGI safety concerns, but another “AI winter” motivated by short-termism about profit margins.
On the anthropics question: No, Mr. Bond, we should expect ourselves to die. If you’re about to get shot in the head, you don’t “expect the gun to jam”, you just expect to probably die (and therefore take actions like buying life insurance or a bulletproof helmet based on that belief). Anthropic reasoning about the future is tricky precisely because it’s tempting to neglect this possibility.
I’m curious how you’d estimate the relative importance of these factors. Myself, I think one of them really outweighs the others.
I think you can steelman Ben Goertzel-style worries about near-term amoral applications of AI being bad “formative influences” on AGI, but mostly under a continuous takeoff model of the world. If AGI is a continuous development of earlier systems, then maybe it shares some datasets and learned models with earlier AI projects, and definitely it shares the broader ecosystems of tools, dataset-gathering methodologies, model-evaluating paradigms, and institutional knowledge on the part of the developers. If the ecosystem in which this thing “grows up” is one that has previously been optimized for marketing, or military applications, or what have you, this is going to have ramifications in how the first AGI projects are designed and what they get exposed to. The more continuous you think the development is going to be, the more this can be intervened on by trying to make sure that AI is pro-social even in the short term.
The amount that I care about this problem is proportional to the chance that I’ll survive to have it.
Pausing reading at “Paul’s simple argument” to jot this down: The expected values are identical when you’re conditioning on all the parent nodes of the action (i.e. you have full knowledge of your own decision-making process, in your decision-making process). But if you can’t do that, then it seems like EDT goes nuts—e.g. if there’s a button you won’t want to press, and you’re not conditioning on your own brain activity, then EDT might evaluate the expected utility of pressing the button by assuming you have a harmful seizure that makes you hit the button, since it just looks for the most likely route, irrespective of utility. This seems like it might be related to problems of embodied cognition—not conditioning on all parent nodes of the action leads the EDT algorithm to treat the seizure as “just part of the plan,” when in fact it’s a breakdown of the cognition implementing EDT in the first place.
After reading more of the post, what seems to be going on is exploiting the difference between the output of the decision algorithm and what gets counted as an action. Sticking with the seizure example—in the Dutch book scenario, the CDT agent happily buys “seizure insurance conditional on A=a.” Then when it’s making its choice to not press the button, it notices that it gets the highest utility from “sell back the seizure insurance and don’t press the button,” so it tries to do that—but some small percentage of the time it still has the seizure before accidentally pressing the button. I’m not sure that a decision theory that models the distinction between its choice and the “actual action” sells back the seizure insurance there. In fact, we could probably fix the problem entirely within CDT by modeling the decision algorithm output as the “actual action” and the intervening seizure a stochastic part of the environment.
I think in the real world, I am actually accumulating evidence against magic faster than I am trying to commit elaborate suicide.
I initially thought you were going to debate Beth Barnes.
Also, thanks for the episode :) It was definitely interesting, although I still don’t have a good handle on why some people are optimistic that there aren’t classes of arguments humans will “fall for” irrespective of their truth value.
One generalization I am also interested in is to learn not merely abstract objects within a big model, but entire self-contained abstract levels of description, together with actions and state transitions that move you between abstract states. E.g. not merely detecting that “the grocery store” is a sealed box ripe for abstraction, but that “go to the grocery store” is a valid action within a simplified world-model with nice properties.
This might be significantly more challenging to say something interesting about, because it depends not just on the world but on how the agent interacts with the world.
I agree with faul sname, ADifferentAnonymous, shminux, etc. If every single person in the world had to play russian roulette (1 bullet and 5 empty chambers), and the firing pin was broken on exactly one gun in the whole world, everyone except the person with the broken gun would be dead after about 125 trigger pulls.
So if I remember being forced to pull the trigger 1000 times, and I’m still alive, it’s vastly more likely that I’m the one human with the broken gun, or that I’m hallucinating, or something else, rather than me just getting lucky. Note that if you think you might be hallucinating, and you happen to be holding a gun, I recommend putting it down and going for a nap, not pulling the trigger in any way. But for the sake of argument we might suppose the only allowed hypotheses are “working gun” and “broken gun.”
Sure, if there are miraculous survivors, then they will erroneously think that they have the broken gun, in much the same way that if you flipped a coin 1000 times and just so happened to get all heads, you might start to think you had an unfair coin. We should not expect to be able to save this person. They are just doomed.
It’s like poker. I don’t know if you’ve played poker, but you probably know that the basic idea is to make bets that you have the best hand. If you have 4 of a kind, that’s an amazing hand, and you should be happy to make big bets. But it’s still possible for your opponent to have a royal flush. If that’s the case, you’re doomed, and in fact when the opponent has a royal flush, 4 of a kind is almost the worst hand possible! It makes you think you can bet all your money when in fact you’re about to lose it all. It’s precisely the fact that four of a kind is a good hand almost all the time that makes it especially bad that remaining tiny amount of the time.
The person who plays russian roulette and wins 1000 times with a working gun is just that poor sap who has four of a kind into a royal flush.
(P.S.: My post is half explanation of how I would calculate the answer, and half bullet-biting on an unusual anthropic problem. The method has a short summary: just have a probabilistic model of the world and then condition on the existence of yourself (with all your memories, faculties, etc). This gives you the right conditional probability distribution over the world. The complications are because this model has to be a fancy directed graph that has “logical nodes” corresponding to the output of your own decision-making procedure, like in TDT. )
No, I’m definitely being more descriptivist than causal-ist here. The point I want to get at is on a different axis.
Suppose you were Laplace’s demon, and had perfect knowledge of a human’s brain (it’s not strictly necessary to pretend determinism, but it sure makes the argument simpler). You would have no need to track the human’s “wants” or “beliefs,” you would just predict based on the laws of physics. Not only could you do a better job than some human psychologist on human-scale tasks (like predicting in advance which button the human will press), you would be making information-dense predictions about the microphysical state of the human’s brain the would just be totally beyond a model of humans coarse-grained to the level of psychology rather than physics.
So when you say “External reference is earned by reasoning in such a way that attributing content like ‘the cause of this and that sensory state …’ is a better explanation”, I totally agree, but I want to emphasize: better explanation for whom? If we somehow built Laplace’s demon, what I’d want to tell it is something like “model me according to my own standards for intentionality.”