Just this guy, you know?
Dagon
Karma: 11,725
How trustworthy is the randomizer?
I’d pick B in both situations if it seemed likely that the offer were trustworthy. But in many cases, I’d give some chance of foul play, and it’s FAR easier for an opponent to weasel out of paying if there’s an apparently-random part of the wager. Someone says “I’ll pay you $24k”, it’s reasonably clear. They say “I’ll pay you $27k unless these dice roll snake eyes” and I’m going to expect much worse odds than 35⁄36 that I’ll actually get paid.
So for 1A > 1B, this may be based on expectation of cheating. For 2A < 2B, both choices are roughly equally amenable to cheating, so you may as well maximize your expectation.
It seems likely that this kind of thinking is unconscious in most people, and therefore gets applied in situations where it’s not relevant (like where you CAN actually trust the probabilities). But it’s not automatically irrational.
The opposite is worth pointing out as well. Decisions that seem easy because they’re small, but are repeated many times may add up to far more important than the difficult, rare ones.
I like this illustration, as it addresses TWO common misunderstandings. Recognizing that the payoff is in incomparable utilities is good. Even better is reinforcing that there can never be further iterations. None of the standard visualizations prevent people from extending to multiple interactions.
And it makes it clear that (D,D) is the only rational (i.e. WINNING) outcome.
Fortunately, most of our dilemmas repeated ones, in which (C,C) is possible.
I THINK rational agents will defect 100 times in a row, or 100 million times in a row for this specified problem. But I think this problem is impossible. In all cases there will be uncertainty about your opponent/partner—you won’t know its utility function perfectly, and you won’t know how perfectly it’s implemented. Heck, you don’t know your OWN utility function perfectly, and you know darn well you’re implemented somewhat accidentally. Also, there are few real cases where you know precisely when there will be no further games that can be affected by the current choice.
In cases of uncertainty on these topics, cooperation can be rational. Something on the order of tit-for-tat with an additional chance of defecting or forgiving that’s based on expectation of game ending with this iteration might be right.
Somehow I’d never thought of this as a rationalist’s dilemma, but rather a determinism vs free will illustration. I still see it that way. You cannot both believe you have a choice AND that Omega has perfect prediction.
The only “rational” (in all senses of the word) response I support is: shut up and multiply. Estimate the chance that he has predicted wrong, and if that gives you +expected value, take both boxes. I phrase this as advice, but in fact I mean it as prediction of rational behavior.
Does this imply that YOU would one-box Newcomb’s offer with Clippy as Omega? And that you think at least some Clippies would take just one box with you as Omega?
For the problem as stated, what probability would you assign to Clippy’s Cooperation (on both the one-shot or fixed-iteration, if they’re different).
This seems a dumb semantic mistake, not a deep truth. You’re confusing “going to” as a prediction and “going to” as a statement of intent. You might prefer the word “intend” if that’s what you mean. And however you phrase it, there is uncertainty in both your chance of success, and limits to the amount of effort and risk you’ll undertake to accomplish this particular mission.
Infinity screws up a whole lot of this essay. Large-but-finite is way way harder, as all the “excuses”, as you call them, become real choices again. You have to figure out whether to monitor for potential conflicts, including whether to allow others to take whatever path you took to such power. Necessity is back in the game.
I suspect I’d seriously consider just tiling the universe with happy faces (very complex ones, but still probably not what the rest of y’all think you want). At least it would be pleasant, and nobody would complain.
This is a silly line of argument. You can’t hold identity constant and change the circumstances very much.
If I were given unlimited (or even just many orders of magnitude more than I now have) power, I would no longer be me. I’d be some creature with far more predictive and reflective accuracy, and this power would so radically change my expectations and beliefs that it’s ludicrous to think that the desires and actions of the powerful agent would have any relationship to what I predict I’d do.
I give high weight (95%+) that this is true for all humans, including Robin and Elizer.
There is no evidence in impossible predictions based on flawed identity concepts.
Oh, I keep meaning to ask: Elizer, do you think FAI is achievable without first getting FNI (friendly natural intelligence)? If we can’t understand and manipulate humans well enough to be sure they won’t destroy the world (or create an AI that does so...), how can we understand and manipulate an AI that’s more powerful than a human?
It’s getting close to the time when you have to justify your bias against universe-tiling.
If you truly believe that happiness/fun/interest/love/utils is the measurement to maximize, and you believe in shutting up and multiplying, then converting all matter to orgasmium sounds right, as a first approximation. You’d want self-improving orgasmium, so it can choose to replace itself with something that can enjoy even more fully and efficiently, of course.
Heh, if I could believe in a limited creator-god, I’d be tempted to think humans might be seed orgasmium. Our job is to get better at it and fill the universe with ourselves.
Why are you biased toward the status quo for this human desire for “meaning” or “intensity” (both of which boil down to “emotional motivation”). The vast majority of terminal goals that I can imagine can be better pursued if fewer people (in the wide sense; people = sentient actors) are struggling to have an effect on the universe because they’re more afraid of meaninglessness than of doing harm.
These don’t seem very universal to me. I think a whole lot of people would choose to live in a world that violates some or all, especially if they’re also allowed to change themselves to enjoy it more.
Rule 23 seems especially strange. Isn’t modification of cognitive ability and structure what the singularity is all about?
What if I want a wonderful and non-mysterious universe? Your current argument seems to be that there’s no such thing. I don’t follow why this is so. “Fun” (defined as desire for novelty) may be the simplest way to build a strategy of exploration, but it’s not obvious that it’s the only one, is it?
A series on “theory of motivation” that explores other options besides novelty and fun as prime directors of optimization processes that can improve the universe (in their and maybe even our eyes).
I’m with Doug S. on this one. None of the “leaders” who be in a position to create such plans are in any way motivated to develop or publish such things, or to follow them after the fact.
For those that ARE interested and motivated to make such things, a better term would be “prediction” than “plan”. And we’re very bad at such predictions, as we always seem to underestimate the shortsightness of every participant, from government to private industry.
I find this direction of inquiry fascinating. Access to source code is not all that interesting, but simulations (especially when you introduce uncertainty (both error and intentional deception)) are a closer analog to human decisions.
Depending on what it meant by the question, either “I’m a pattern of information encoded in my brain” or “I’m a side-effect of processing that my brain does for it’s own reasons” are the one-sentence descriptions I’d use.
I identify with the meat that currently contains/creates/executes me, but not perfectly—I don’t expect that replacing the meat with a sufficiently-similar replacement would alter the experience of being me.
Identity, of course is a continuum, not a binary measure. A different brain with the same patterns and inputs would be so much like me that I’d call them the same. But really it might be no more similar than future me and past me in the “same” body.
Are you looking for things that cannot be used for signaling, or things that have primary motivations other than signaling, or something else? A theory of partial motivations seems doomed, unless you’ve first solved the problem of scalar (as opposed to ordinal) choices.
I don’t think there’s ANYTHING which qualifies as action and has no signaling component. Any biological function carries a bit of data about health, and any mental function says something about motivation or ability.
Even things normally done in private (shitting, inner dialog) are signals. The fact that we keep them private is an indication of following social norms.
If we’re just trying to ignore signaling as a motivation, and look for activities which we’re motivated to do in addition to their signaling value, I’d propose looking at things with immediate survival value, and things that don’t involve cognitive choice. “Removing your hand from a hot stove” is something I don’t think is primarily done for signaling.
The more signaling matters, the less I can trust such reasoning, as it usually does not acknowledge the signaling influences.
I like this motivation, but I think you’re going down a dead end looking for non-signaling-but-reasoned-and-interesting choices. I don’t think there’s such a thing, as signaling is part of our motivational makeup, and factors into even things that other people cannot know.
I expect much more success attacking the second half of the sentence. Knowing that signaling is a core component of preferences, train your decision apparatus to acknowledge it, and take it into account. Note that this applies not to just signaling, but for all motivations that some parts of you wish you didn’t have, or didn’t weigh so heavily.
Do we get feedback on the modifications the chronophone makes to our statements? I think I would find it useful in identifying my own biases to see how it translates statements I think are probably true into things I think probably aren’t.