An alien planet contains joy and suffering in a ratio that makes them exactly cancel out according to your morality. You are exactly ambivalent about the alien planet blowing up. The alien planet can’t be changed by your actions, so you don’t need to cancel plans to go there and reduce the suffering when you find out that the planet blew up. Say that they existed long ago. In general we are setting up the situation so that the planet blowing up doesn’t change your expected utility, or the best action for you to take. We set this up by a pile of contrivances. This still feels impactful.
Donald Hobson
Karma: 620
Imagine a planet with aliens living on it. Some of those aliens are having what we would consider morally valuable experiences. Some are suffering a lot. Suppose we now find that their planet has been vaporized. By tuning the relative amounts of joy and suffering, we can make it so that the vaporization is exactly neutral under our morality. This feels like a big deal, even if the aliens were in an alternate reality that we could watch but not observe.
Our intuitive feeling of impact is a proxy for how much something effects our values and our ability to achive them. You can set up contrived situations where an event doesn’t actually effect our ability to achive our values, but still triggers the proxy.
Would the technical definition that you are looking for be value of information. Feeling something to be impactful means that a bunch of mental heuristics think it has a large value of info?
“Either my curtains are red, or they are blue.” Would be a false dilemma that doesn’t fit any category. You can make a false dilemma out of any pair of non mutually exclusive predicates, there is no need for them to refer to values or actions.
If we stop doing something that almost all first world humans are doing (say 1 billion people), then our impact will be about a billionth of the size of the problem. Given the size of impact that an effective altruist can hope to have, this tells us why non actions don’t have super high utilities in comparison. If there were 100 000 effective altruists (probably an overestimate), This would mean that all effective altruists refraining from doing X, would make the problem 0.01% better. Both how hard it is to refrain, and the impact if you manage it depend on the problem size, all pollution vs plastic straws. Assuming that this change took only 0.01% of the effective altruists time. (10 seconds per day, 4 of which you are asleep for). Clearly this change has to be something as small as avoiding plastic straws, of smaller. Assume linearity in work and reward, the normal assumption being diminishing returns. This makes the payoff equivalent to all effective altruists working full time on solving the problem, and solving it.
Technically, you need to evaluate the marginal value of one more effective altruist. If it was vitally important that someone worked on AI, but you have far more people than you need to do that, and the rest are twiddling their thumbs, get them reusing straws (Actually get them looking for other cause areas, reusing straws only makes sense if you are confidant that no other priority causes exist)
Suppose omega came to you and said that if you started a compostable straw buisness, there was an 0.001% chance of success, by which omega means solving the problem without any externalities. (The straws are the same price, just as easy to use, don’t taste funny ect.) Otherwise, the buisness will waste all your time and do nothing.
If this doesn’t seem like a promising opportunity for effective altruism, don’t bother with reusable straws either. In general the chance of success is 1/( Number of people using plastic straws X Proportion of time wasted avoiding them )
Fair enough, I think that satisfies my critique.
A full consideration of proving too much requires that we have uncertainty both over what arguments are valid, and over the real world. The uncertainty about what arguments are valid, along with our inability to consider all possible arguments makes this type of reasoning work. If you see a particular type of argument in favor of conclusion X, and you disagree with conclusion X, then that gives you evidence against that type of argument.
This is used in moral arguments too. Consider the argument that touching someone really gently isn’t wrong. And if it isn’t wrong to touch someone with force F, then it isn’t wrong to touch them with force F+0.001 Newtons. Therefore, by induction, it isn’t wrong to punch people as hard as you like.
Now consider the argument that 1 grain of sand isn’t a heap. If you put a grain of sand down somewhere that there isn’t already a heap of sand, you don’t get a heap. Therefore by induction, no amount of sand is a heap.
If you were unsure about the morality of punching people, but knew that heaps of sand existed, then seeing the first argument would make you update towards “punching people is ok”. When you then see the second argument, you update to “inductive arguments don’t work in the real world.” and reverse the previous update about punching people.
Seeing an argument for a conclusion that you don’t believe can make you reduce your credence on other statements supported by similar arguments.
I disagree with insight 5. I think that even if you uploaded the worlds best computer security experts and gave them 1000 years to redesign every piece of software and hardware, then you threw all existing computers in the trash and rolled out their design. Even with lots of paranoia, a system designed to the goal of making things as hard for an ASI as possible, and not trading off any security for usability, compatability or performance, (while still making a system significantly more useful than having no computers) this wouldn’t stop an ASI.
If you took an advanced future computer containing an ASI back in time to the 1940′s, before there were any other computers at all, it would still be able to take over the world. There are enough people that can be persuaded, and enough inventions that any fool can put together out of household materials. At worst, it would find its way to a top secrete government bunker, where it would spend its time breaking enigma and designing superweapons, until it could build enough robots and compute. The government scientists just know that the last 10 times they followed its instructions, they ended up with brilliant weapons, and the AI has fed them some story about it being sent back in time to help them win the war.
Hacking through the internet might be the path of least resistance for an ASI, but other routes to power exist.
Regarding example 4. Believing something because a really smart person believes it is not a bad heuristic, as long as you aren’t cherry-picking the really smart person. prepriors, then everyone says, you had the misfortune of being born wrong, I’m lucky enough to be born right. If you were transported to an alternate reality, where half the population thought 2+2=4, and half thought 2+2=7, would you become uncertain, or would you just think that the 2+2=7 population were wrong?
The argument about believing in Cthulhu because that was how you were raised proving too much, itself proves too much.
Regarding example 4. Believing something because a really smart person believes it is not a bad heuristic, as long as you aren’t cherry-picking the really smart person. If you have data about many smart people, taking the average is an even better heuristic. As is focusing on the smart people that are experts in some relevant field. The usefulness of this heuristic also depends on your definition of ‘smart’. There are a few people with a high IQ, a powerful brain capable of thinking and remembering well, but who have very poor epistemology, and are creationists or Scientologists. Many definitions of smart would rule out these people, requiring some rationality skills of some sort. This makes the smart people heuristic even better.
The problem with the maths is that it does not correlate ‘values’ with any real world observable. You give all objects a property, you say that that property is distributed by simplicity priors. You have not yet specified how these ‘values’ things relate to any real world phenomenon in any way. Under this model, you could never see any evidence that humans don’t ‘value’ maximizing paperclips.
To solve this, we need to understand what values are. The values of a human are much like the filenames on a hard disk. If you run a quantum field theory simulation, you don’t have to think about either, you can make your predictions directly. If you want to make approximate predictions about how a human will behave, you can think in terms of values and get somewhat useful predictions. If you want to predict approximately how a computer system will behave, instead of simulating every transistor, you can think in terms folders and files.
I can substitute words in the ‘proof’ that humans don’t have values, and get a proof that computers don’t have files. It works the same way, you turn your uncertainty in the relation between the exact and the approximate into a confidence that the two are uncorrelated. Making a somewhat naive and not formally specified assumption along the lines of, “the real action taken optimizes human values better than most possible actions” will get you a meaningful but not perfect definition of ‘values’. You still need to say exactly what a “possible action” is.
Making a somewhat naive and not formally specified assumption along the lines of, “the files are what you see when you click on the file viewer” will get you a meaningful but not perfect definition of ‘files’. You still need to say exactly what a “click” is. And how you translate a pattern of photons into a ‘file’.
We see that if you were running a quantum simulation of the universe, then getting values out of a virtual human is the same type of problem as getting files off a virtual computer.
Intuition Dump. Safety through this mechanism seems to me like aiming a rocket at mars, and accidently hitting the moon. There might well be a region where P(doom|superintelligence) is lower, but lower in the sense of 90% is lower than 99.9%. Suppose we have the clearest, most stereotypical case of wireheading as I understand it. A mesa optimizer with a detailed model of its own workings and the terminal goal of maximizing the flow of current in some wire. (During training, current in the reward signal wire reliably correlated with reward.)
The plan that maximizes this flow long term is to take over the universe and store all the energy, to gradually turn into electricity. If the agent has access to its own internals before it really understands that it is an optimizer, or is thinking about the long term future of the universe, it might manage to brick its self. If the agent is sufficiently myopic, is not considering time travel or acausal trade, and has the choice of running high voltage current through itself now, or slowly taking over the world, it might choose the former.
Note that both of these look like hitting a fairly small target in design, and lab enviroment space. The mesa optimizer might have some other terminal goal. Suppose the prototype AI has the opportunity to write arbitrary code to an external computer, and an understanding of AI design before it has self modification access. The AI creates a subagent that cares about the amount of current in a wire in the first AI, the subagent can optimize this without destroying itself. Even if the first agent then bricks itself, we have a AI that will dig the fried circuit boards out the trashcan, and throw all the cosmic commons into protecting and powering them.
In conclusion, this is not a safe part of agentspace, just a part that’s slightly less guaranteed to kill you. I would say it was of little to no strategic importance. Especially if you think that all humans making AI will be reasonably on the same page regarding safety, scenarios where AI alignment is nearly solved, and the first people to ASI barely know the field exists are unlikely. If the first ASI self destructs for reasons like this, we have all the peaces for making superintelligence, and people with no sense safety are trying to make one. I would expect another attempt a few weeks later to doom us. (Unless the first AI bricked itself in a sufficiently spectacular manor, like hacking into nukes to create a giant EMP in its circuits. That might get enough people seeing danger to have everyone stop.)
The proposition graph is non standard as far as I know. The syntax tree is kind of standard, but a bit unusual. You might want to use them if I show how to use them for logical counterfactuals. (Which I haven’t finished yet)
I think that our research is at a sufficiently early stage that most technical work could contribute to most success stories. We are still mostly understanding the rules of the game and building the building blocks. I would say that we work on AI safety in general until we find anything that can be used at all. (There is some current work on things like satisficers that seem less relevant to sovereigns. I am not discouraging working on areas that seem more likely to help some success stories, just saying that those areas seem rare.)
The bound isn’t always achievable if you don’t know Y. Suppose with uniform over the 6 possible outcomes. You find out that (without loss of generality). You must construct a function st . Because as far as you know, could be 2 or 3, and you have to be able to construct from and . But then we just take the most common output of , and that tells us so .
(If you had some other procedure for calculating X from S and Y, then you can make a new function that does that, and call it S, so an arbitrary function from X to Y+{err} is the most general form of S.
Logical Counterfactuals and Proposition graphs, Part 3
Logical Counterfactuals and Proposition graphs, Part 2
No, what I am saying is that humans judge things to be more different when the difference will have important real world consequences in the future. Consider two cases, one where the water will be tipped into the pool later, and the other where the water will be tipped into a nuclear reactor, which will explode if the salt isn’t quite right.
There need not be any difference in the bucket or water whatsoever. While the current bucket states look the same, there is a noticeable macro-state difference between nuclear reactor exploding and not exploding, in a way that there isn’t a macrostate difference between marginally different eddy currents in the pool. I was specifying a weird info theoretic definition of significance that made this work, but just saying that the more energy is involved, the more significant works too. Nowhere are we referring to human judgement, we are referring to hypothetical future consequences.
Actually the rule, your action and its reversal should not make a difference worth tracking in its world model, would work ok here. (Assuming sensible Value of info). The rule that it shouldn’t knowably affect large amounts of energy is good too. So for example it can shuffle an already well shuffled pack of cards, even if the order of those cards will have some huge effect. It can act freely without worrying about weather chaos effects, the chance of it causing a hurricane counterbalanced by the chance of stopping one. But if it figures out how to twitch its elbow in just the right way to cause a hurricane, it can’t do that. This robot won’t tip the nuclear bucket, for much the same reason. It also can’t make a nanobot that would grey goo earth, or hack into nukes to explode them. All these actions effect a large amount of energy in a predictable direction.
Basically all formal proofs assume that the hardware is perfect. Rowhammer.
In the case of the industrial process, you could consider the action less reversible because while the difference in the water is small, the difference in what happens after that is larger. (Ie industrial part working or failing.). This means that at some point within the knock on effects of tipping over the carefully salt balanced bucket, there needs to be an effect that counts as “significant”. However, there must not be an effect that counts as significant in the case where it’s a normal swimming pool, and someone will throw the bucket into the pool soon anyway. Lets suppose water with a slightly different salt content will make a nuclear reactor blow up. (And no humans will notice the robot tipping out and refilling the bucket, so the counterfactual on the robots behavior actually contains an explosion.)
Suppose you shake a box of sand. With almost no information, the best you can do to describe the situation is state the mass of sand, shaking speed and a few other average quantities. With a moderate amount of info, you can track the position and speed of each sand grain, with lots of info, you can track each atom.
There is a sense in which average mass and velocity of the sand, or even position of every grain, is a better measure than md5 hash of position of atom 12345. It confines the probability distribution for the near future to a small, non convoluted section of nearby configuration space.
Suppose we have a perfectly Newtonian solar system, containing a few massive objects and many small ones.
We start our description at time 0. If we say how much energy is in the system, then this defines a non convoluted subset of configuration space. The subset stays just as non convoluted under time evolution. Thus total energy is a perfect descriptive measure. If we state the position and velocity of the few massive objects, and the averages for any large dust clouds, then we can approximately track our info forward in time, for a while. Liouvilles theorem says that configuration space volume is conserved, and ours is. However, our configuration space volume is slowly growing more messy and convoluted. This makes the descriptive measure good but not perfect. If we have several almost disconnected systems, the energy in each one would also be a good descriptive measure. If we store the velocity of a bunch of random dust specks, we have much less predictive capability. The subset of configuration space soon becomes warped and twisted until its convex hull, or epsilon ball expansion cover most of the space. This makes velocities of random dust specs a worse descriptive measure. Suppose we take the md5 hash of every objects position, rounded to the nearest nanometer, in some coordinate system and concatenated together. this forms a very bad descriptive measure. After only a second of time evolution, this becomes a subset of configuration space that can only be defined in terms of what it was a second ago, or by a giant arbitrary list.
Suppose the robot has one hour to do its action, and then an hour to reverse it. We measure how well the robot reversed its original action by looking at all good descriptive measures, and seeing the difference in these descriptive measures from what they would have been had the robot done nothing.
We can then call an action reversible if there would exist an action that would reverse it.
Note that the crystal structure of a rock now tells us its crystal structure next year, so can be part of a quite good measure. However the phase of the moon will tell you everything from the energy production of tidal power stations to the migration pattern of moths. If you want to make good (non convoluted) predictions about these things, you can’t miss it out. Thus almost all good descriptive measures will contain this important fact.
A reversible action is any action taken in the first hour such that there exists an action that approximately reverses it that the robot could take in the second hour. (The robot need not actually take the reverse action, maybe a human could press a reverse button.)
Functionality of nuclear power stations, and level of radiation in atmosphere are also contained in many good descriptive measures. Hence the robot should tip the bucket if it won’t blow up a reactor.
(This is a rough sketch of the algorithm with missing details, but it does seem to have broken the problem down into non value laden parts. I would be unsurprised to find out that there is something in the space of techniques pointed to that works, but also unsurprised to find that none do.)
What if I make each time period in the ”...” one nanosecond shorter than the previous.
You must believe that there is some length of time, t>most of a day, such that everyone in the world being tortured for t-1 nanosecond is better than one person being tortured for t.
Suppose there was a strong clustering effect in human psychology, such that less than a week of torture left peoples minds in one state, and more than a week left them broken. I would still expect the possibility of some intermediate cases on the borderlines. Things as messy as human psychology, I would expect there to not be a perfectly sharp black and white cutoff. If we zoom in enough, we find that the space of possible quantum wavefunctions is continuous.
There is a sense in which specs and torture feel incomparable, but I don’t think this is your sense of incomparability, to me it feels like moral uncertainty about which huge number of specs to pick. I would also say that “Don’t torture anyone” and “don’t commit attrocities based on convoluted arguments” a good ethical injunction. If you think that your own reasoning processes are not very reliable, and you think philosophical thought experiments rarely happen in real life, then implementing the general rule “If I think I should torture someone, go to nearest psych ward” is a good idea. However I would want a perfectly rational AI which never made mistakes to choose torture.
Viewed from the outside, in the logical counterfactual where the agent crosses, PA can prove its own consistency, and so is inconsistent. There is a model of PA in which “PA proves False”. Having counterfactualed away all the other models, these are the ones left. Logical counterfactualing on any statement that can’t be proved or disproved by a theory should produce the same result as adding it as an axiom. Ie logical counterfactualing ZF on choice should produce ZFC.
The only unreasonableness here comes from the agent’s worst case optimizing behaviour. This agent is excessively cautious. A logical induction agent, with PA as a deductive process will assign some prob P strictly between 0 and 1 to “PA is consistent”. Depending on which version of logical induction you run, and how much you want to cross the bridge, crossing might be worth it. (the troll is still blowing up the bridge iff PA proves False)
A logical counterfactual where you don’t cross the bridge is basically a counterfactual world where your design of logical induction assigns lower prob to “PA is consistent”. In this world it doesn’t cross and gets zero.
The alternative is a logical factual where it expects +ve util.
So if we make logical induction like crossing enough, and not mind getting blown up much, it crosses the bridge. Lets reverse this. An agent really doesn’t want blown up.
In the counterfactual world where it crosses, logical induction assigns more prob to “PA is consistant”. The expected utility procedure has to use its real probability distribution, not ask the counterfactual agent for its expected util.
I am not sure what happens after this, I think you still need to think about what you do in impossible worlds. Still working it out.
Firstly, some topics are just easier if they can be presented the right format. Geometry will be easier in a format that allows diagrams, compared to an audiobook. Secondly, formats, like websites are often Schelling points, not many serious scientists publish their work as a series of gifs on twitter. Most scientists know this, and so don’t look for a series of gifs on twitter. Then there are affordances, a video of a maths lecture on Youtube, (there are quite a few uni lectures that have been filmed and put on Youtube) might be informative, but have links to lolcats all down the side. If the medium distracts you, you will learn less.
There is also a sense of making use of the medium. Take the medium of videogame. In principle a video game can display an arbitrary pattern of pixels on a screen. However, suppose that the pattern of pixels most suited to learning some topic looks like pages of text. There is no point making a videogame, just to reimplement a document reader in it. So all the people trying to make serious learning resources use text documents. Any video game that is made is full of interactive widgets that aren’t that useful for learning, and is usually targeted at those with too little attention span to read much text.
A lot of the time, I would recommend going for any format in which the information you want is explained by someone who knows the subject and is good at teaching. If the subject is obscure, go for anywhere that you can find the info at all. If distraction is a big problem for you, download the Youtube videos that you intend to watch, unplug your router and then watch them.