Wei_Dai2

Karma: 169

Wei_Dai2 23 Feb 2009 16:25 UTC
0 points
on: Fairness vs. Goodness
This is fascinating. JW plays C in the last round, even though AA just played D in the next-to-last round. What explains that? Maybe JW’s belief in his own heroic story is strong enough to make him sacrifice his self-interest?

Theoretically, of course, utility functions are invariant up to affine transformation, so a utility’s absolute sign is not meaningful. But this is not always a good metaphor for real life.

So you’re suggesting that real life has some additional structure which is not representable in ordinary game theory formalism? Can you think of an extension to game theory which can represent it? (Mathematically, not just metaphorically.)

Wei_Dai2 7 Feb 2009 22:43 UTC
5 points
on: Epilogue: Atonement (8/8)
Robin, what is your favorite piece of academic philosophy that argues about values?

Nicholas, our own universe may have an infinite volume, and it’s only the speed of light that limits the size of the observable universe. Given that infinite universes are not considered implausible, and starlines are not considered implausible (at least as a fictional device), I find it surprising that you consider starlines that randomly connect a region of size 2^(10^20) to be implausible.
Starlines have to have an average distance of something, right? Why not 2^(10^20)?

Wei_Dai2 7 Feb 2009 2:55 UTC
16 points
on: Epilogue: Atonement (8/8)
Nicholas, suppose Eliezer’s fictional universe contains a total of 2^(10^20) star systems, and each starline connects two randomly selected star systems. With a 20 hour doubling speed, the Superhappies, starting with one ship, can explore 2^(t365²⁴⁄₂₀) random star systems after t years. Let’s say the humans are expanding at the same pace. How long will it take, before humans and Superhappies will meet again?

According to the birthday paradox, they will likely meet after each having explored about sqrt(2^(10^20)) = 2^(510^19) star systems, which will take 510^19/(365*24/20) or approximately 10^17 years to accomplish. That should be enough time to get over our attachment to “bodily pain, embarrassment, and romantic troubles”, I imagine.

Wei_Dai2 6 Feb 2009 19:54 UTC
11 points
on: Epilogue: Atonement (8/8)
But the tech in the story massively favors the defense, to the point that a defender who is already prepared to fracture his starline network if attacked is almost impossible to conquer (you’d need to advance faster than the defender can send warnings of your attack while maintaining perfect control over every system you’ve captured). So an armed society would have a good chance of being able to cut itself off from even massively superior aliens, while pacifists are vulnerable to surprise attacks from even fairly inferior ones.

I agree, and that’s why in my ending humans conquer the Babyeaters only after we develop a defense against the supernova weapon. The fact that the humans can see the defensive potential of this weapon, but the Babyeaters and the Superhappies can’t, is a big flaw in the story. The humans sacrificed billions in order to allow the Superhappies to conquer the Babyeaters, but that makes sense only if the Babyeaters can’t figure out the same defense that the humans used. Why not?

Also, the Superhappies’ approach to negotiation made no game theoretic sense. What they did was, offer a deal to the other side. If they don’t accept, impose the deal on them anyway by force. If they do accept, trust that they will carry out the deal without try to cheat. Given these incentives, why would anyone facing a Superhappy in negotiation not accept and then cheat? I don’t see any plausible way in which this morality/negotiation strategy could have become a common one in Superhappy society.

Lastly, I note that the Epilogue of the original ending could be named Atonement as well. After being modified by the Superhappies (like how the Confessor was “rescued”?), the humans would now be atoning for having forced their children suffer pain. What does this symmetry tell us, if anything?

Wei_Dai2 5 Feb 2009 22:48 UTC
3 points
on: True Ending: Sacrificial Fire (7/8)
So, what about the fact that all of humanity now knows about the supernova weapon? How is it going to survive the next few months?

Wei_Dai2 4 Feb 2009 10:25 UTC
0 points
on: Three Worlds Decide (5/8)
In case it wasn’t clear, the premise of my ending is that the Ship’s Confessor really was a violent thief and drug dealer from the 21th century, but his “rescue” was only partially successful. He became more rational, but only pretended to accept what became the dominant human morality of this future, patiently biding time his whole life for an opportunity like this.

Wei_Dai2 4 Feb 2009 10:03 UTC
0 points
on: Three Worlds Decide (5/8)
The Ship’s Confessor uses the distraction to anesthetizes everyone except the pilot. He needs the pilot to take command of the starship and to pilot it. The ship stays to observe which star the Superhappy ship came from, then takes off for the nearest Babyeater world. They let the Babyeaters know what happened, and tell them to supernova the star that Superhappies came from at all costs.

When everyone wakes up, the Ship’s Confessor convinces the entire crew to erase their memory of the true Alderson’s Coupling Constant, ostensibly for the good of humanity. He pretends to do so himself, but doesn’t. After the ship returns to human space, he uses his accumulated salary to build a series of hidden doomsday devices around every human colony, and becomes the dictator of humanity through blackmail. Everyone is forced to adopt an utility function of his choosing as their own. With every resource of humanity devoted to the subject, scientists under his direction eventually discover a defense against the supernova weapon, and soon after that, the Babyeaters are conquered, enslaved, and farmed for their crystal brains. Those brains, when extracted and networked in large arrays, turn out to be the cheapest and most efficient computing substrate in the universe. These advantages provide humanity with such a strong competitive edge, that it never again faces an alien that is its match, at least militarily.

Before the universe ends in a big crunch, the Confessed (humanity’s eventual name) goes on to colonize more than (10^9)^(10^9) star systems, and to meet and conquer almost as many alien species, but the Superhappy people are never seen again. Their fate becomes one of the most traded futures in the known universe, but those bets will have to remain forever unsettled.

Wei_Dai2 1 Feb 2009 22:52 UTC
14 points
on: The Allais Paradox
Eliezer, I see from this example that the Axiom of Independence is related to the notion of dynamic consistency. But, the logical implication goes only one way. That is, the Axiom of Independence implies dynamic consistency, but not vice versa. If we were to replace the Axiom of Independence with some sort of Axiom of Dynamic Consistency, we would no longer be able to derive expected utility theory. (Similarly with dutch book/money pump arguments, there are many ways to avoid them besides being an expected utility maximizer.)

I’m afraid that the Axiom of Independence cannot really be justified as a basic principle of rationality. Von Neumann and Morgenstern probably came up with it because it was mathematically necessary to derive Expected Utility Theory, then they and others tried to justify it afterward because Expected Utility turned out to be such an elegant and useful idea. Has anyone seen Independence proposed as a principle of rationality prior to the invention of Expected Utility Theory?
What links here?

Wei_Dai2 1 Feb 2009 3:06 UTC
0 points
on: Value is Fragile
To expand on my categorization of values a bit more, it seems clear to me that at least some human value do not deserved to be forever etched into the utility function of a singleton. Those caused by idiosyncratic environmental characteristics like taste for salt and sugar, for example. To me, these are simply accidents of history, and I wouldn’t hesitate (too much) to modify them away in myself, perhaps to be replaced by more interesting and exotic tastes.

What about reproduction? It’s a value that my genes programmed into me for their own purposes, so why should I be obligated to stick with it forever?

Or consider boredom. Eventually I may become so powerful that I can easily find the globally optimal course of action for any set of goals I might have, and notice that the optimal course of action often involves repetition of some kind. Why should I retain my desire not to do the same thing over and over again, which was programmed into me by evolution back when minds had a tendency to get stuck in local optimums?

And once I finally came to that realization, I felt less ashamed of values that seemed ‘provincial’ - but that’s another matter.

Eliezer, I wonder if this actually has more to do with your current belief that rationality equals expected utility maximization. For an expected utility maximizer, there is no distinction between ‘provincial’ and ‘universal’ values, and certainly no reason to ever feel ashamed of one’s values. One just optimizes according to whatever values one happens to have. But as I argued before, human beings are not expected utility maximizers, and I don’t see why we should try to emulate them, especially this aspect.

Wei_Dai2 1 Feb 2009 1:33 UTC
0 points
on: Value is Fragile
Tim and Tyrrell, do you know the axiomatic derivation of expected utility theory? If you haven’t read http://cepa.newschool.edu/het/essays/uncert/vnmaxioms.htm or something equivalent, please read it first.

Yes, if you change the spaces of states and choices, maybe you can encode every possible agent as an utility function, not just those satisfying certain axioms of “rationality” (which I put in quotes because I don’t necessarily agree with them), but that would be to miss the entire point of expected utility theory, which is that it is supposed to be a theory of rationality, and is supposed to rule out irrational preferences. That means using state and choice spaces where those axiomatic constraints have real world meaning.

Wei_Dai2 31 Jan 2009 1:24 UTC
6 points
on: Value is Fragile
A utility function is like a program in a Turing-complete language. If the behaviour can be computed at all, it can be computed by a utility function.

Tim, I’ve seen you state this before, but it’s simply wrong. A utility function is not like a Turing-complete language. It imposes rather strong constraints on possible behavior.

Consider a program which when given the choices (A,B) outputs A. If you reset it and give it choices (B,C) it outputs B. If you reset it again and give it choices (C,A) it outputs C. The behavior of this program cannot be reproduced by a utility function.

Here’s another example: When given (A,B) a program outputs “indifferent”. When given (equal chance of A or B, A, B) it outputs “equal chance of A or B”. This is also not allowed by EU maximization.

Wei_Dai2 30 Jan 2009 21:03 UTC
1 point
on: The Baby-Eating Aliens (1/8)
If the aliens’ wetware (er, crystalware) is so efficient that their children are already sentient when they are still tiny relative to adults, why don’t the adults have bigger brains and be much more intelligent than humans? Given that they also place high values on science and rationality, had invented agriculture long before humans did, and haven’t fought any destructive wars recently, it makes no sense that they have a lower level of technology than humans at this point.

Other than that, I think the story is not implausible. The basic lesson here is the same as in Robin’s upload scenarios: when sentience is really cheap, no one will be valued (much) just for being sentient. If we want people to be valued just for being sentient, either the wetware/crystalware/hardware can’t be too efficient, or we need to impose some kind of artificial scarcity on sentience.

Wei_Dai2 30 Jan 2009 19:08 UTC
0 points
on: Value is Fragile
Maybe we don’t mean the same thing by boredom?

I’m using Eliezer’s definition: a desire not to do the same thing over and over again. For a creature with roughly human-level brain power, doing the same thing over and over again likely means it’s stuck in a local optimum of some sort.

Genome equivalents which don’t generate terminally valued individual identity in the minds they descrive should outperform those that do.

I don’t understand this. Please elaborate.

Why not just direct expected utility? Pain and pleasure are easy to find but don’t work nearly as well.

I suppose you mean why not value external referents directly instead of indirectly through pain and pleasure. As long as wireheading isn’t possible, I don’t see why the latter wouldn’t work just as well as the former in many cases. Also, the ability to directly value external referents depends on a complex cognitive structure to assess external states, which may be more vulnerable in some situations to external manipulation (i.e. unfriendly persuasion or parasitic memes) than hard-wired pain and pleasure, although the reverse is probably true in other situations. It seems likely that evolution would come up with both.

Define sexual. Most sexual creatures are too simple to value the first two. Most plausible posthumans aren’t sexual in a traditional sense.

I mean reproduction where more than one party contributes genetic material and/or parental resources. Even simple sexual creatures probably have some notion of beauty and/or status to help attract/select mates, but for the simplest perhaps “instinct” would be a better word than “value”.

- likely values for intelligent creatures with sexual reproduction (music, art, literature, humor)

Disagree.

These all help signal fitness and attract mates. Certainly not all intelligent creatures with sexual reproduction will value exactly music, art, literature, and humor, but it seems likely they will have values that perform the equivalent functions.

Wei_Dai2 30 Jan 2009 9:04 UTC
4 points
1
on: Value is Fragile
We can sort the values evolution gave us into the following categories (not necessarily exhaustive). Note that only the first category of values is likely to be preserved without special effort, if Eliezer is right and our future is dominated by singleton FOOM scenarios. But many other values are likely to survive naturally in alternative futures.
- likely values for all intelligent beings and optimization processes (power, resources)
- likely values for creatures with roughly human-level brain power (boredom, knowledge)
- likely values for all creatures under evolutionary competition (reproduction, survival, family/clan/tribe)
- likely values for creatures under evolutionary competition who cannot copy their minds (individual identity, fear of personal death)
- likely values for creatures under evolutionary competition who cannot wirehead (pain, pleasure)
- likely values for creatures with sexual reproduction (beauty, status, sex)
- likely values for intelligent creatures with sexual reproduction (music, art, literature, humor)
- likely values for intelligent creatures who cannot directly prove their beliefs (honesty, reputation, piety)
- values caused by idiosyncratic environmental characteristics (salt, sugar)
- values caused by random genetic/memetic drift and co-evolution (Mozart, Britney Spears, female breasts, devotion to specific religions)
The above probably isn’t controversial, rather the disagreement is mainly on the following:
- the probabilities of various future scenarios
- which values, if any, can be preserved using approaches such as FAI
- which values, if any, we should try to preserve
I agree with Roko that Eliezer has made his case in an impressive fashion, but it seems that many of us are still not convinced on these three key points.

Take the last one. I agree with those who say that human values do not form a consistent and coherent whole. Another way of saying this is that human beings are not expected utility maximizers, not as individuals and certainly not as societies. Nor do most of us desire to become expected utility maximizers. Even amongst the readership of this blog, where one might logically expect to find the world’s largest collection of EU-maximizer wannabes, few have expressed this desire. But there is no principled way to derive an utility function from something that is not an expected utility maximizer!

Is there any justification for trying to create an expected utility maximizer that will forever have power over everyone else, whose utility function is derived using a more or less arbitrary method from the incoherent values of those who happen to live in the present? That is, besides the argument that it is the only feasible alternative to a null future. Many of us are not convinced of this, neither the “only” nor the “feasible”.

Wei_Dai2 12 Jan 2009 20:35 UTC
7 points
on: Eutopia is Scary
Why make science a secret, instead of inventing new worlds with new science for people to explore? Have you heard of “Theorycraft”? It’s science applied to the World of Warcraft, and for some, Theorycraft is as much fun as the game it’s based on.

Is there something special about the science of base-level reality that makes it especially fun to explore and discover? I think the answer is yes, but only if it hasn’t already been explored and then covered up again and made into a game. It’s the difference between a natural and an artificial challenge.

Wei_Dai2 20 Dec 2008 4:28 UTC
15 points
on: Complex Novelty
One day we’ll discover the means to quickly communicate insights from one individual to another, say by directly copying and integrating the relevant neural circuitry. Then, in order for an insight to be Fun, it will have to be novel to transhumanity, not just the person learning or discovering it. Learning something the fast efficient way will not be Fun because there’s not true effort. Pretending that the new way doesn’t exist, and learning the old-fashioned way, will not be Fun because there’s not true victory.

I’m not sure there are enough natural problems in the universe to supply the whole of transhumanity with an adequate quantity of potential insights. “Natural” meaning not invented for the sole purpose of providing an artificial challenge. Personally, I can’t see how solving the n-th random irrelevant mathematical problem is any better than lathing the n-th table leg.

Wei_Dai2 16 Dec 2008 21:29 UTC
2 points
on: Not Taking Over the World
Anna Salamon wrote: Is it any safer to think ourselves about how to extend our adaptation-executer preferences than to program an AI to figure out what conclusions we would come to, if we did think a long time?

First, I don’t know that “think about how to extend our adaptation-executer preferences” is the right thing to do. It’s not clear why we should extend our adaptation-executer preferences, especially given the difficulties involved. I’d backtrack to “think about what we should want”.

Putting that aside, the reason that I prefer we do it ourselves is that we don’t know how to get an AI to do something like this, except through opaque methods that can’t be understood or debugged. I imagine the programmer telling the AI “Stop, I think that’s a bug.” and the AI responding with “How would you know?”

g wrote: Wei Dai, singleton-to-competition is perfectly possible, if the singleton decides it would like company.

In that case the singleton might invent a game called “Competition”, with rules decided by itself. Anti-prediction says that it’s pretty unlikely those rules would happen to coincide with the rules of base-level reality, so base-level reality would still be controlled by the singleton.

Wei_Dai2 16 Dec 2008 9:24 UTC
3 points
on: Not Taking Over the World
Robin wrote: Having to have an answer now when it seems an likely problem is very expensive.

(I think you meant to write “unlikely” here instead of “likely”.)

Robin, what is your probability that eventually humanity will evolve into a singleton (i.e., not necessarily through Eliezer’s FOOM scenario)? It seems to me that competition is likely to be unstable, whereas a singleton by definition is. Competition can evolve into a singleton, but not vice versa. Given that negentropy increases as mass squared, most competitors have to remain in the center, and the possibility of a singleton emerging there can’t ever be completely closed off. BTW, a singleton might emerge from voluntary mergers, not just one competitor “winning” and “taking over”.

Another reason to try to answer now, instead of later, is that coming up with a good answer would persuade more people to work towards a singleton, so it’s not just a matter of planning for a contingency.

Wei_Dai2 16 Dec 2008 8:18 UTC
2 points
on: Not Taking Over the World
An expected utility maximizer would know exactly what to do with unlimited power. Why do we have to think so hard about it? The obvious answer is that we are adaptation executioners, not utility maximizers, and we don’t have an adaptation for dealing with unlimited power. We could try to extrapolate an utility function from our adaptations, but given that those adaptations deal only with a limited set of circumstances, we’ll end up with an infinite set of possible utility functions for each person. What to do?

James D. Miller: But about 100 people die every minute!

Peter Norvig: Refusing to act is like refusing to allow time to pass.

What about acting to stop time? Preserve Earth at 0 kelvin. Gather all matter/energy/negentropy in the rest of the universe into secure storage. Then you have as much time as you want to think.

Wei_Dai2 14 Dec 2008 20:45 UTC
1 point
on: What I Think, If Not Why
Maybe we don’t need to preserve all of the incompressible idiosyncrasies in human morality. Considering that individuals in the post-Singularity world will have many orders of magnitude more power than they do today, what really matter are the values that best scale with power. Anything that scales logarithmically for example will be lost in the noise compared to values that scale linearly. Even if we can’t understand all of human morality, maybe we will be able to understand the most important parts.

Just throwing away parts of one’s utility function seems bad. That can’t be optimal right? Well, as Peter de Blanc pointed out, it can be if there is no feasible alternative that improves expected utility under the original function. We should be willing to lose our unimportant values to avoid or reduce even a small probability of losing the most important ones. With CEV, we’re supposed to implement it with the help of an AI that’s not already Friendly, and if we don’t get it exactly right on the first try, we can’t preserve even our most important values. Given that we don’t know how to safely get an unFriendly AI to do anything, much less something this complicated, the probability of failure seems quite large.