# Exterminating life is rational

I don’t mean that de­cid­ing to ex­ter­mi­nate life is ra­tio­nal. But if, as a so­ciety of ra­tio­nal agents, we each max­i­mize our ex­pected util­ity, this may in­evitably lead to our ex­ter­mi­nat­ing life, or at least in­tel­li­gent life.

Ed Regis re­ports on p 216 of “Great Mambo Chicken and the Tran­sHu­man Con­di­tion,” (Pen­guin Books, Lon­don, 1992):

Ed­ward Tel­ler had thought about it, the chance that the atomic ex­plo­sion would light up the sur­round­ing air and that this con­fla­gra­tion would then prop­a­gate it­self around the world. Some of the bomb mak­ers had even calcu­lated the nu­mer­i­cal odds of this ac­tu­ally hap­pen­ing, com­ing up with the figure of three chances in a mil­lion they’d in­cin­er­ate the Earth. Nev­er­the­less, they went ahead and ex­ploded the bomb.

Was this a bad de­ci­sion? Well, con­sider the ex­pected value to the peo­ple in­volved. Without the bomb, there was a much, much greater than 31,000,000 chance that ei­ther a) they would be kil­led in the war, or b) they would be ruled by Nazis or the Ja­panese. The loss to them if they ig­nited the at­mo­sphere would be an­other 30 or so years of life. The loss to them if they lost the war and/​or were kil­led by their en­e­mies would also be an­other 30 or so years of life. The loss in be­ing con­quered would also be large. Easy de­ci­sion, re­ally.

Sup­pose that, once a cen­tury, some party in a con­flict chooses to use some tech­nique to help win the con­flict that has a p=3/​1,000,000 chance of elimi­nat­ing life as we know it. Then our ex­pected sur­vival time is 100 times the sum from n=1 to in­finity of np(1-p)n-1. If I’ve done my math right, that’s ≈ 33,777,000 years.

This sup­po­si­tion seems rea­son­able to me. There is a bal­ance be­tween offen­sive and defen­sive ca­pa­bil­ity that shifts as tech­nol­ogy de­vel­ops. If tech­nol­ogy keeps chang­ing, it is in­evitable that, much of the time, a tech­nol­ogy will provide the abil­ity to de­stroy all life be­fore the counter-tech­nol­ogy to defend against it has been de­vel­oped. In the near fu­ture, biolog­i­cal weapons will be more able to wipe out life than we are able to defend against them. We may then de­velop the abil­ity to defend against biolog­i­cal at­tacks; we may then be safe un­til the next dan­ger­ous tech­nol­ogy.

If you be­lieve in ac­cel­er­at­ing change, then the num­ber of im­por­tant events in a given time in­ter­val in­creases ex­po­nen­tially, or, equiv­a­lently, the time in­ter­vals that should be con­sid­ered equiv­a­lent op­por­tu­ni­ties for im­por­tant events shorten ex­po­nen­tially. The 34M years re­main­ing to life is then in sub­jec­tive time, and must be mapped into re­al­time. If we sup­pose the sub­jec­tive/​real time ra­tio dou­bles ev­ery 100 years, this gives life an ex­pected sur­vival time of 2000 more re­al­time years. If we in­stead use Ray Kurzweil’s figure of about 2 years, this gives life about 40 re­main­ing re­al­time years. (I don’t recom­mend Ray’s figure. I’m just giv­ing it for those who do.)

Please un­der­stand that I am not yet an­other “prophet” be­moan­ing the fool­ish­ness of hu­man­ity. Just the op­po­site: I’m say­ing this is not some­thing we will out­grow. If any­thing, be­com­ing more ra­tio­nal only makes our doom more cer­tain. For the agents who must ac­tu­ally make these de­ci­sions, it would be ir­ra­tional not to take these risks. The fact that this level of risk-tol­er­ance will in­evitably lead to the snuf­fing out of all life does not make the ex­pected util­ity of these risks nega­tive for the agents in­volved.

I can think of only a few ways that ra­tio­nal­ilty can not in­evitably ex­ter­mi­nate all life in the cos­molog­i­cally (even ge­olog­i­cally) near fu­ture:

• We can out­run the dan­ger: We can spread life to other planets, and to other so­lar sys­tems, and to other galax­ies, faster than we can spread de­struc­tion.

• Tech­nol­ogy will not con­tinue to de­velop, but will sta­bi­lize in a state in which all defen­sive tech­nolo­gies provide ab­solute, 100%, fail-safe pro­tec­tion against all offen­sive tech­nolo­gies.

• Peo­ple will stop hav­ing con­flicts.

• Ra­tional agents in­cor­po­rate the benefits to oth­ers into their util­ity func­tions.

• Ra­tional agents with long lifes­pans will pro­tect the fu­ture for them­selves.

• Utility func­tions will change so that it is no longer ra­tio­nal for de­ci­sion-mak­ers to take tiny chances of de­stroy­ing life for any amount of util­ity gains.

• In­de­pen­dent agents will cease to ex­ist, or to be free (the Sin­gle­ton sce­nario).

Let’s look at these one by one:

## We can out­run the dan­ger.

We will colonize other planets; but we may also figure out how to make the Sun go nova on de­mand. We will colonize other star sys­tems; but we may also figure out how to liber­ate much of the en­ergy in the black hole at the cen­ter of our galaxy in a gi­ant ex­plo­sion that will move out­ward at near the speed of light.

One prob­lem with this idea is that apoc­a­lypses are cor­re­lated; one may trig­ger an­other. A dis­ease may spread to an­other planet. The choice to use a planet-bust­ing bomb on one planet may lead to its re­tal­i­a­tory use on an­other planet. It’s not clear whether spread­ing out and in­creas­ing in pop­u­la­tion ac­tu­ally makes life more safe. If you think in the other di­rec­tion, a smaller hu­man pop­u­la­tion (say ten mil­lion) stuck here on Earth would be safer from hu­man-in­sti­gated dis­asters.

But nei­ther of those are my fi­nal ob­jec­tion. More im­por­tant is that our com­pres­sion of sub­jec­tive time can be ex­po­nen­tial, while our abil­ity to es­cape from ever-broader swaths of de­struc­tion is limited by light­speed.

## Tech­nol­ogy will sta­bi­lize in a safe state.

Maybe tech­nol­ogy will sta­bi­lize, and we’ll run out of things to dis­cover. If that were to hap­pen, I would ex­pect that con­flicts would in­crease, be­cause peo­ple would get bored. As I men­tioned in an­other thread, one good ex­pla­na­tion for the in­ces­sant and coun­ter­pro­duc­tive wars in the mid­dle ages—a rea­son some of the ac­tors them­selves gave in their writ­ings—is that the no­bil­ity were bored. They did not have the con­cept of progress; they were just look­ing for some­thing to give them pur­pose while wait­ing for Je­sus to re­turn.

But that’s not my fi­nal re­jec­tion. The big prob­lem is that by “safe”, I mean re­ally, re­ally safe. We’re talk­ing about bring­ing ex­is­ten­tial threats to chances less than 1 in a mil­lion per cen­tury. I don’t know of any defen­sive tech­nol­ogy that can guaran­tee a less than 1 in a mil­lion failure rate.

## Peo­ple will stop hav­ing con­flicts.

That’s a nice thought. A lot of peo­ple—maybe the ma­jor­ity of peo­ple—be­lieve that we are in­evitably pro­gress­ing along a path to less vi­o­lence and greater peace.

They thought that just be­fore World War I. But that’s not my fi­nal re­jec­tion. Evolu­tion­ary ar­gu­ments are a more pow­er­ful rea­son to be­lieve that peo­ple will con­tinue to have con­flicts. Those that avoid con­flict will be out-com­peted by those that do not.

But that’s not my fi­nal re­jec­tion ei­ther. The big­ger prob­lem is that this isn’t some­thing that arises only in con­flicts. All we need are de­sires. We’re will­ing to tol­er­ate risk to in­crease our util­ity. For in­stance, we’re will­ing to take some un­known, but clearly greater than one in a mil­lion chance, of the col­lapse of much of civ­i­liza­tion due to cli­mate warm­ing. In re­turn for this risk, we can en­joy a bet­ter lifestyle now.

Also, we haven’t burned all physics text­books along with all physi­cists. Yet I’m con­fi­dent there is at least a one in a mil­lion chance that, in the next 100 years, some physi­cist will figure out a way to re­duce the earth to pow­der, if not to crack space­time it­self and undo the en­tire uni­verse. (In fact, I’d guess the chance is nearer to 1 in 10.)1 We take this ex­is­ten­tial risk in re­turn for a con­tinued flow of benefits such as bet­ter graph­ics in Halo 3 and smaller iPods. And it’s rea­son­able for us to do this, be­cause an im­prove­ment in util­ity of 1% over an agent’s lifes­pan is, to that agent, ex­actly bal­anced by a 1% chance of de­stroy­ing the Uni­verse.

The Wikipe­dia en­try on Large Had­con Col­lider risk says, “In the book Our Fi­nal Cen­tury: Will the Hu­man Race Sur­vive the Twenty-first Cen­tury?, English cos­mol­o­gist and as­tro­physi­cist Martin Rees calcu­lated an up­per limit of 1 in 50 mil­lion for the prob­a­bil­ity that the Large Hadron Col­lider will pro­duce a global catas­tro­phe or black hole.” The more au­thor­i­ta­tive “Re­view of the Safety of LHC Col­li­sions” by the LHC Safety Assess­ment Group con­cluded that there was at most a 1 in 1031 chance of de­stroy­ing the Earth.

The LHC con­clu­sions are crim­i­nally low. Their ev­i­dence was this: “Na­ture has already con­ducted the LHC ex­per­i­men­tal pro­gramme about one billion times via the col­li­sions of cos­mic rays with the Sun—and the Sun still ex­ists.” There fol­lowed a cou­ple of sen­tences of hand­wav­ing to the effect that if any other stars had turned to black holes due to col­li­sions with cos­mic rays, we would know it—ap­par­ently due to our flawless abil­ity to de­tect black holes and as­cer­tain what caused them—and there­fore we can mul­ti­ply this figure by the num­ber of stars in the uni­verse.

I be­lieve there is much more than a one-in-a-billion chance that our un­der­stand­ing in one of the steps used in ar­riv­ing at these figures is in­cor­rect. Based on my ex­pe­rience with peer-re­viewed pa­pers, there’s at least a one-in-ten chance that there’s a ba­sic ar­ith­metic er­ror in their pa­per that no one has no­ticed yet. I’m think­ing more like one-in-a-mil­lion, once you cor­rect for the an­thropic prin­ci­ple and for the chance that there is a mis­take in the ar­gu­ment. (That’s based on a be­lief that pri­ors for any­thing likely enough that smart peo­ple even thought of the pos­si­bil­ity should be larger than one in a billion, un­less they were speci­fi­cally try­ing to think of ex­am­ples of low-prob­a­bil­ity pos­si­bil­ities such as all of the air molecules in the room mov­ing to one side.)

The Trinity test was done for the sake of win­ning World War II. But the LHC was turned on for… well, no prac­ti­cal ad­van­tage that I’ve heard of yet. It seems that we are will­ing to tol­er­ate one-in-a-mil­lion chances of de­stroy­ing the Earth for very lit­tle benefit. And this is ra­tio­nal, since the LHC will prob­a­bly im­prove our lives by more than one part in a mil­lion.

## Ra­tional agents in­cor­po­rate the benefits to oth­ers into their util­ity func­tions.

“But,” you say, “I wouldn’t risk a 1% chance of de­stroy­ing the uni­verse for a 1% in­crease in my util­ity!”

Well… yes, you would, if you’re a ra­tio­nal ex­pec­ta­tion max­i­mizer. It’s pos­si­ble that you would take a much higher risk, if your util­ity is at risk of go­ing nega­tive; it’s not pos­si­ble that you would not ac­cept a .999% risk, un­less you are not max­i­miz­ing ex­pected value, or you as­sign the null state af­ter uni­verse-de­struc­tion nega­tive util­ity. (This seems difficult, but is worth ex­plor­ing.) If you still think that you wouldn’t, it’s prob­a­bly be­cause you’re think­ing a 1% in­crease in your util­ity means some­thing like a 1% in­crease in the plea­sure you ex­pe­rience. It doesn’t. It’s a 1% in­crease in your util­ity. If you fac­tor the rest of your uni­verse into your util­ity func­tion, then it’s already in there.

The US na­tional debt should be enough to con­vince you that peo­ple act in their self-in­ter­est. Even the most moral peo­ple—in fact, es­pe­cially the “most moral” peo­ple—do not in­cor­po­rate the benefits to oth­ers, es­pe­cially fu­ture oth­ers, into their util­ity func­tions. If we did that, we would en­gage in mas­sive eu­gen­ics pro­grams. But eu­gen­ics is con­sid­ered the great­est im­moral­ity.

But maybe they’re just not as ra­tio­nal as you. Maybe you re­ally are a ra­tio­nal saint who con­sid­ers your own plea­sure no more im­por­tant than the plea­sure of ev­ery­one else on Earth. Maybe you have never, ever bought any­thing for your­self that did not bring you as much benefit as the same amount of money would if spent to re­pair cleft palates or dis­tribute vac­cines or mosquito nets or wa­ter pumps in Africa. Maybe it’s re­ally true that, if you met the girl of your dreams and she loved you, and you won the lot­tery, put out an album that went plat­inum, and got pub­lished in Science, all in the same week, it would make an im­per­cep­ti­ble change in your util­ity ver­sus if ev­ery­one you knew died, Bernie Mad­off spent all your money, and you were un­fairly con­victed of mur­der and di­ag­nosed with can­cer.

It doesn’t mat­ter. Be­cause you would be adding up ev­ery­one else’s util­ity, and ev­ery­one else is get­ting that 1% ex­tra util­ity from the bet­ter graph­ics cards and the smaller iPods.

But that will stop you from risk­ing at­mo­spheric ig­ni­tion to defeat the Nazis, right? Be­cause you’ll in­cor­po­rate them into your util­ity func­tion? Well, that is a sub­set of the claim “Peo­ple will stop hav­ing con­flicts.” See above.

And even if you some­how worked around all these ar­gu­ments, evolu­tion, again, thwarts you.2 Even if you don’t agree that ra­tio­nal agents are self­ish, your un­selfish agents will be out-com­peted by self­ish agents. The claim that ra­tio­nal agents are not self­ish im­plies that ra­tio­nal agents are un­fit.

## Ra­tional agents with long lifes­pans will pro­tect the fu­ture for them­selves.

The most fa­mil­iar idea here is that, if peo­ple ex­pect to live for mil­lions of years, they will be “wiser” and take fewer risks with that time. But the flip side is that they also have more time to lose. If they’re de­cid­ing whether to risk ig­nit­ing the at­mo­sphere in or­der to lower the risk of be­ing kil­led by Nazis, lifes­pan can­cels out of the equa­tion.

Also, if they live a mil­lion times longer than us, they’re go­ing to get a mil­lion times the benefit of those nicer iPods. They may be less will­ing to take an ex­is­ten­tial risk for some­thing that will benefit them only tem­porar­ily. But benefits have a way of in­creas­ing, not de­creas­ing, over time. The dis­cov­ery of the law of grav­ity and of the in­visi­ble hand benefit us in the 21st cen­tury more than they did the peo­ple of the 17th cen­tury.

But that’s not my fi­nal re­jec­tion. More im­por­tant is time-dis­count­ing. Agents will time-dis­count, prob­a­bly ex­po­nen­tially, due to un­cer­tainty. If you con­sid­ered benefits to the fu­ture with­out ex­po­nen­tial time-dis­count­ing, the benefits to oth­ers and to fu­ture gen­er­a­tions would out­weigh any benefits to your­self so much that in many cases you wouldn’t even waste time try­ing to figure out what you wanted. And, since fu­ture gen­er­a­tions will be able to get more util­ity out of the same re­sources, we’d all be obliged to kill our­selves, un­less we rea­son­ably think that we are con­tribut­ing to the de­vel­op­ment of that ca­pa­bil­ity.

Time dis­count­ing is always (so far) ex­po­nen­tial, be­cause non-asymp­totic func­tions don’t make sense. I sup­posed you could use a tri­gono­met­ric func­tion in­stead for time dis­count­ing, but I don’t think it would help.

Could a con­tinued ex­po­nen­tial pop­u­la­tion ex­plo­sion out­weigh ex­po­nen­tial time-dis­count­ing? Well, you can’t have a con­tinued ex­po­nen­tial pop­u­la­tion ex­plo­sion, be­cause of the speed of light and the Planck con­stant. (I leave the de­tails as an ex­er­cise to the reader.)

Also, even if you had no time-dis­count­ing, I think that a ra­tio­nal agent must do iden­tity-dis­count­ing. You can’t stay you for­ever. If you change, the fu­ture you will be less like you, and weigh less strongly in your util­ity func­tion. Ob­jec­tions to this gen­er­ally as­sume that it makes sense to trace your iden­tity by fol­low­ing your phys­i­cal body. Phys­i­cal bod­ies will not have a 1-1 cor­re­spon­dence with per­son­al­ities for more than an­other cen­tury or two, so just for­get that idea. And if you don’t change, well, what’s the point of liv­ing?

Evolu­tion­ary ar­gu­ments may help us with self-dis­count­ing. Evolu­tion­ary forces en­courage agents to em­pha­size con­ti­nu­ity or an­ces­try over re­sem­blance in an agent’s self­ness func­tion. The ma­jor vari­able is re­pro­duc­tion rate over lifes­pan. This ap­plies to genes or memes. But they can’t help us with time-dis­count­ing.

I think there may be a way to make this one work. I just haven’t thought of it yet.

## A benev­olent sin­gle­ton will save us all.

This case takes more anal­y­sis than I am will­ing to do right now. My short an­swer is that I place a very low ex­pected util­ity on sin­gle­ton sce­nar­ios. I would al­most rather have the uni­verse eat, drink, and be merry for 34 mil­lion years, and then die.

I’m not ready to place my faith in a sin­gle­ton. I want to work out what is wrong with the rest of this ar­gu­ment, and how we can sur­vive with­out a sin­gle­ton.

(Please don’t con­clude from my ar­gu­ments that you should go out and cre­ate a sin­gle­ton. Creat­ing a sin­gle­ton is hard to undo. It should be deferred nearly as long as pos­si­ble. Maybe we don’t have 34 mil­lion years, but this es­say doesn’t give you any rea­son not to wait a few thou­sand years at least.)

## In conclusion

I think that the figures I’ve given here are con­ser­va­tive. I ex­pect ex­is­ten­tial risk to be much greater than 31,000,000 per cen­tury. I ex­pect there will con­tinue to be ex­ter­nal­ities that cause sub­op­ti­mal be­hav­ior, so that the ac­tual risk will be greater even than the already-suffi­cient risk that ra­tio­nal agents would choose. I ex­pect pop­u­la­tion and tech­nol­ogy to con­tinue to in­crease, and ex­is­ten­tial risk to be pro­por­tional to pop­u­la­tion times tech­nol­ogy. Ex­is­ten­tial risk will very pos­si­bly in­crease ex­po­nen­tially, on top of the sub­jec­tive-time ex­po­nen­tial.

Our great­est chance for sur­vival is that there’s some other pos­si­bil­ity I haven’t thought of yet. Per­haps some of you will.

1 If you ar­gue that the laws of physics may turn out to make this im­pos­si­ble, you don’t un­der­stand what “prob­a­bil­ity” means.

2 Evolu­tion­ary dy­nam­ics, the speed of light, and the Planck con­stant are the three great en­ablers and pre­ven­ters of pos­si­ble fu­tures, which en­able us to make pre­dic­tions farther into the fu­ture and with greater con­fi­dence than seem in­tu­itively rea­son­able.

• Here’s a pos­si­ble prob­lem with my anal­y­sis:

Sup­pose Omega or one of its ilk says to you, “Here’s a game we can play. I have an in­finitely large deck of cards here. Half of them have a star on them, and one-tenth of them have a skull on them. Every time you draw a card with a star, I’ll dou­ble your util­ity for the rest of your life. If you draw a card with a skull, I’ll kill you.”

How many cards do you draw?

I’m pretty sure that some­one who be­lieves in many wor­lds will keep draw­ing cards un­til they die. But even if you don’t be­lieve in many wor­lds, I think you do the same thing, un­less you are not max­i­miz­ing ex­pected util­ity. (Un­less chance is quan­tized so that there is a min­i­mum pos­si­ble pos­si­bil­ity. I don’t think that would help much any­way.)

So this whole post may boil down to “max­i­miz­ing ex­pected util­ity” not ac­tu­ally be­ing the right thing to do. Also see my ear­lier, equally un­pop­u­lar post on why ex­pec­ta­tion max­i­miza­tion im­plies av­er­age util­i­tar­i­anism. If you agree that av­er­age util­i­tar­i­anism seems wrong, that’s an­other piece of ev­i­dence that max­i­miz­ing ex­pected util­ity is wrong.

• “Every time you draw a card with a star, I’ll dou­ble your util­ity for the rest of your life. If you draw a card with a skull, I’ll kill you.”

Sorry if this ques­tion has already been an­swered (I’ve read the com­ments but prob­a­bly didn’t catch all of it), but...

I have a prob­lem with “dou­ble your util­ity for the rest of your life”. Are we talk­ing about utilons per sec­ond? Or do you mean “dou­ble the util­ity of your life”, or just “dou­ble your util­ity”? How does dy­ing a cou­ple of min­utes later af­fect your util­ity? Do you get the en­tire (now dou­bled) util­ity for those few min­utes? Do you get pro rata util­ity for those few min­utes di­vided by your ex­pected lifes­pan?

Re­lated to this is the ques­tion of the util­ity penalty of dy­ing. If your util­ity func­tion in­cludes benefits for other peo­ple, then your best bet is to draw cards un­til you die, be­cause the benefits to the rest of the uni­verse will mas­sively out­weigh the in­evita­bil­ity of your death.

If, on the other hand, death sets your util­ity to zero (pre­sum­ably be­cause your util­ity func­tion is strictly only a func­tion of your own ex­pe­riences), then… yeah. If Omega re­ally can dou­ble your util­ity ev­ery time you win, then I guess you keep draw­ing un­til you die. It’s an ab­surd (but math­e­mat­i­cally plau­si­ble) situ­a­tion, so the ab­surd (but math­e­mat­i­cally plau­si­ble) an­swer is cor­rect. I guess.

• Re­for­mu­la­tion to weed out un­in­ter­est­ing ob­jec­tions: Omega knows ex­pected util­ity ac­cord­ing to your prefer­ence if you go on with­out its in­ter­ven­tion U1 and util­ity if it kills you U0U1.

My an­swer: even in a de­ter­minis­tic world, I take the lot­tery as many times as Omega has to offer, know­ing that the prob­a­bil­ity of death tends to cer­tainty as I go on. This ex­am­ple is only in­valid for money be­cause of diminish­ing re­turns. If you re­ally do pos­sess the abil­ity to dou­ble util­ity, low prob­a­bil­ity of pos­i­tive out­come gets squashed by high util­ity of that out­come.

• There’s an ex­cel­lent pa­per by Peter le Blanc in­di­cat­ing that un­der rea­son­able as­sump­tions, if you util­ity func­tion is un­bounded, then you can’t com­pute finite ex­pected util­ities. So if Omega can dou­ble your util­ity an un­limited num­ber of times, you have other prob­lems that crip­ple you in the ab­sence of in­volve­ment from Omega. Dou­bling your util­ity should be a math­e­mat­i­cal im­pos­si­bil­ity at some point.

That de­mol­ishes “Shut up and Mul­ti­ply”, IMO.

SIAI ap­par­ently paid Peter to pro­duce that. It should get more at­ten­tion here.

• So if Omega can dou­ble your util­ity an un­limited num­ber of times

This was not as­sumed, I even ex­plic­itly said things like “I take the lot­tery as many times as Omega has to offer” and “If you re­ally do pos­sess the abil­ity to dou­ble util­ity”. To the ex­tent dou­bling of util­ity is ac­tu­ally pro­vided (and no more), we should take the lot­tery.

• Also, if your util­ity func­tion’s scope is not limited to per­cep­tion-se­quences, Peter’s re­sult doesn’t di­rectly ap­ply. If your util­ity func­tion is lin­ear in ac­tual, rather than per­ceived, pa­per­clips, Omega might be able to offer you the deal in­finitely many times.

• Also, if your util­ity func­tion’s scope is not limited to per­cep­tion-se­quences, Peter’s re­sult doesn’t di­rectly ap­ply.

How can you act upon a util­ity func­tion if you can­not eval­u­ate it? The util­ity func­tion needs in­puts de­scribing your situ­a­tion. The only available in­puts are your per­cep­tions.

• The util­ity func­tion needs in­puts de­scribing your situ­a­tion. The only available in­puts are your per­cep­tions.

Not so. There’s also log­i­cal knowl­edge and log­i­cal de­ci­sion-mak­ing where noth­ing ever changes and no new ob­ser­va­tions ever ar­rive, but the game still can be in­finitely long, and con­tain all the es­sen­tial parts, such as learn­ing of new facts and de­ter­mi­na­tion of new de­ci­sions.

(This is of course not rele­vant to Peter’s model, but if you want to look at the un­der­ly­ing ques­tions, then these strange con­struc­tions ap­ply.)

• Does my en­tire post boil down to this seem­ing para­dox?

(Yes, I as­sume Omega can ac­tu­ally dou­ble util­ity.)

The use of U1 and U0 is need­lessly con­fus­ing. And it changes the game, be­cause now, U0 is a util­ity as­so­ci­ated with a sin­gle draw, and the anal­y­sis of do­ing re­peated draws will give differ­ent an­swers. There’s also too much change in go­ing from “you die” to “you get util­ity U0″. There’s some se­man­tic trick­i­ness there.

• Pretty much. And I should men­tion at this point that ex­per­i­ments show that, con­trary to in­struc­tions, sub­jects nearly always in­ter­pret util­ity as hav­ing diminish­ing marginal util­ity.

• Well, that leaves me even less op­ti­mistic than be­fore. As long as it’s just me say­ing, “We have op­tions A, B, and C, but I don’t think any of them work,” there are a thou­sand pos­si­ble ways I could turn out to be wrong. But if it re­duces to a math prob­lem, and we can’t figure out a way around that math prob­lem, hope is harder.

• Can util­ity go ar­bi­trar­ily high? There are diminish­ing re­turns on al­most ev­ery kind of good thing. I have difficulty imag­in­ing life with util­ity or­ders of mag­ni­tude higher than what we have now. In­finitely long youth might be worth a lot, but even that is only so many dou­blings due to dis­count­ing.

I’m cu­ri­ous why it’s get­ting down­voted with­out re­ply. Re­lated thread here. How high do you think “util­ity” can go?

• I would guess you’re be­ing down­voted by some­one who is frus­trated not by you so much as by all the other peo­ple be­fore you who keep bring­ing up diminish­ing re­turns even though the con­cept of “util­ity” was in­vented to get around that ob­jec­tion.

“Utility” is what you have af­ter you’ve fac­tored in diminish­ing re­turns.

We do have difficulty imag­in­ing or­ders of mag­ni­tude higher util­ity. That doesn’t mean it’s non­sen­si­cal. I think I have or­ders of mag­ni­tude higher util­ity than a microbe, and that the microbe can’t un­der­stand that. One rea­son we de­velop math­e­mat­i­cal mod­els is that they let us work with things that we don’t in­tu­itively un­der­stand.

If you say “Utility can’t go that high”, you’re also re­ject­ing util­ity max­i­miza­tion. Just in a differ­ent way.

• Noth­ing about util­ity max­i­miza­tion model says util­ity func­tion is un­bounded—the only math­e­mat­i­cal as­sump­tions for a well be­haved util­ity func­tion are U’(x) >= 0, U″(x) ⇐ 0.

If the func­tion is let’s say U(x) = 1 − 1/​(1+x), U’(x) = (x+1)^-2, then it’s a prop­erly be­hav­ing util­ity func­tion, yet it never even reaches 1.

And util­ity max­i­miza­tion is just a model that breaks eas­ily—it can be use­ful for hu­mans to some limited ex­tent, but we know hu­mans break it all the time. Try­ing to imag­ine util­ities or­ders of mag­ni­tude higher than cur­rent gets it way past its break­ing point.

• Noth­ing about util­ity max­i­miza­tion model says util­ity func­tion is unbounded

Yep.

the only math­e­mat­i­cal as­sump­tions for a well be­haved util­ity func­tion are U’(x) >= 0, U″(x) ⇐ 0

Utility func­tions aren’t nec­es­sar­ily over do­mains that al­low their deriva­tives to be scalar, or even mean­ingful (my no­tional u.f., over 4D world-his­to­ries or some­thing similar, sure isn’t). Even if one is, or if you’re hold­ing fixed all but one (real-val­ued) of the pa­ram­e­ters, this is far too strong a con­straint for non-patholog­i­cal be­hav­ior. E.g., most peo­ple’s (no­tional) util­ity is pre­sum­ably strictly de­creas­ing in the num­ber of times they’re hit with a base­ball bat, and non-mono­tonic in the amount of salt on their food.

• Can util­ity go ar­bi­trar­ily high?

We could have a con­test, where each con­tes­tant tries to de­scribe a sce­nario that has the largest util­ity to a judge. I bet that af­ter a few rounds of this, we’ll con­verge on some sce­nario of max­i­mum util­ity, no mat­ter who the judge is.

Does this show that util­ity can’t go ar­bi­trar­ily high?

ETA: The above per­haps only shows the difficulty of not get­ting stuck in a lo­cal max­i­mum. Maybe a bet­ter ar­gu­ment is that a hu­man mind can only con­sider a finite sub­set of con­figu­ra­tion space. The point in that sub­set with the largest util­ity must be the max­i­mum util­ity for that mind.

• Sorry for com­ing late to this party. ;)

Much of this dis­cus­sion seems to me to rest on a similar con­fu­sion to that ev­i­denced in “Ex­pec­ta­tion max­i­miza­tion im­plies av­er­age util­i­tar­i­anism”.

As I just pointed out again, the vNM ax­ioms merely im­ply that “ra­tio­nal” de­ci­sions can be rep­re­sented as max­imis­ing the ex­pec­ta­tion of some func­tion map­ping world his­to­ries into the re­als. This func­tion is con­ven­tion­ally called a util­ity func­tion. In this sense of “util­ity func­tion”, your prefer­ences over gam­bles de­ter­mine your util­ity (up to an af­fine trans­form), so when Omega says “I’ll dou­ble your util­ity” this is just a very round­about (and rather odd) way of say­ing some­thing like “I will do some­thing suffi­ciently good that it will in­duce you to ac­cept my offer”.* Given stan­dard as­sump­tions about Omega, this pretty ob­vi­ously means that you ac­cept the offer.

The con­fu­sion seems to arise be­cause there are other map­pings from world his­to­ries into the re­als that are also con­ven­tion­ally called util­ity func­tions, but which have noth­ing in par­tic­u­lar to do with the vNM util­ity func­tion. When we read “I’ll dou­ble your util­ity” I think we in­tu­itively parse the phrase as refer­ring to one of these other util­ity func­tions, which is when prob­lems start to en­sue.

Max­imis­ing ex­pected vNM util­ity is the right thing to do. But “max­imise ex­pected vNM util­ity” is not es­pe­cially use­ful ad­vice, be­cause we have no ac­cess to our vNM util­ity func­tion un­less we already know our prefer­ences (or can rea­son­ably ex­trap­o­late them from prefer­ences we do have ac­cess to). Max­imis­ing ex­pected utilons is not nec­es­sar­ily the right thing to do. You can max­i­mize any (po­ten­tially bounded!) pos­i­tive mono­tonic trans­form of utilons and you’ll still be “ra­tio­nal”.

* There are sets of “ra­tio­nal” prefer­ences for which such a state­ment could never be true (your prefer­ences could be rep­re­sented by a bounded util­ity func­tion where dou­bling would go above the bound). If you had such prefer­ences and Omega pos­sessed the usual Omega-prop­er­ties, then she would never claim to be able to dou­ble your util­ity: ergo the hy­po­thet­i­cal im­plic­itly rules out such prefer­ences.

NB: I’m aware that I’m fudg­ing a cou­ple of things here, but they don’t af­fect the point, and un­fudg­ing them seemed likely to be more con­fus­ing than helpful.

• so when Omega says “I’ll dou­ble your util­ity” this is just a very round­about (and rather odd) way of say­ing some­thing like “I will do some­thing suffi­ciently good that it will in­duce you to ac­cept my offer”

It’s not that easy. As hu­mans are not for­mally ra­tio­nal, the prob­lem is about whether to bite this par­tic­u­lar bul­let, show­ing a form that fol­low­ing the de­ci­sion pro­ce­dure could take and ask­ing if it’s a good idea to adopt a de­ci­sion pro­ce­dure that forces such de­ci­sions. If you already ac­cept the de­ci­sion pro­ce­dure, of course the prob­lem be­comes triv­ial.

• Which de­ci­sion pro­ce­dure are you talk­ing about? Max­imis­ing ex­pected vNM util­ity and max­i­miz­ing (e.g.) ex­pected utilons are quite differ­ent pro­ce­dures—which was ba­si­cally my point.

The former doesn’t force such de­ci­sions at all. That’s pre­cisely why I said that it’s not use­ful ad­vice: all it says is that you should take the gam­ble if you pre­fer to take the gam­ble.* (More­over, if you did not pre­fer to take the gam­ble, the hy­po­thet­i­cal dou­bling of vNM util­ity could never hap­pen, so the set up already as­sumes you pre­fer the gam­ble. This seems to make the hy­po­thet­i­cal not es­pe­cially use­ful ei­ther.)

On the other hand “max­i­mize ex­pected utilons” does provide con­crete ad­vice. It’s just that (AFAIK) there’s no rea­son to listen to that ad­vice un­less you’re risk-neu­tral over utilons. If you were suffi­ciently risk averse over utilons then a 50% chance of dou­bling them might not in­duce you to take the gam­ble, and noth­ing in the vNM ax­ioms would say that you’re be­hav­ing ir­ra­tionally. The re­ally in­ter­est­ing ques­tion then be­comes whether there are other good rea­sons to have par­tic­u­lar risk prefer­ences with re­spect to utilons, but it’s a ques­tion I’ve never heard a par­tic­u­larly good an­swer to.

* At least pro­vided do­ing so would not re­sult in an in­con­sis­tency in your prefer­ences. [ETA: Ac­tu­ally, if your prefer­ences are in­con­sis­tent, then they won’t have a vNM util­ity rep­re­sen­ta­tion, and Omega’s claim that she will dou­ble your vNM util­ity can’t ac­tu­ally mean any­thing. The set-up there­fore seems to im­ply that you prefer­ences are nec­es­sar­ily con­sis­tent. There sure seem to be a lot of sur­rep­ti­tious as­sump­tions built in here!]

• Which de­ci­sion pro­ce­dure are you talk­ing about? Max­imis­ing ex­pected vNM util­ity and max­i­miz­ing (e.g.) ex­pected utilons are quite differ­ent pro­ce­dures—which was ba­si­cally my point.

[...] you should take the gam­ble if you pre­fer to take the gamble

The “pre­fer” here isn’t im­me­di­ate. Peo­ple have (in­ter­nal) ar­gu­ments about what should be done in what situ­a­tions pre­cisely be­cause they don’t know what they re­ally pre­fer. There is an easy an­swer to go with the whim, but that’s not prefer­ence peo­ple care about, and so we de­liber­ate.

When all con­fu­sion is defeated, and the prefer­ence is laid out ex­plic­itly, as a de­ci­sion pro­ce­dure that just crunches num­bers and pro­duces a de­ci­sion, that is by con­struc­tion ex­actly the most prefer­able ac­tion, there is noth­ing to ar­gue about. Ar­gu­ment is not a part of this form of de­ci­sion pro­ce­dure.

In real life, ar­gu­ment is an im­por­tant part of any de­ci­sion pro­ce­dure, and it is the means by which we could se­lect a de­ci­sion pro­ce­dure that doesn’t in­volve ar­gu­ment. You look at the pos­si­ble solu­tions pro­duced by many tools, and judge which of them to im­ple­ment. This makes the de­ci­sion pro­ce­dure differ­ent from the first kind.

One of the tools you con­sider may be a “util­ity max­i­miza­tion” thingy. You can’t say that it’s by defi­ni­tion the right de­ci­sion pro­ce­dure, as first you have to ac­cept it as such through ar­gu­ment. And this ap­plies not only to the par­tic­u­lar choice of prior and util­ity, but also to the al­gorithm it­self, to the pos­si­bil­ity of rep­re­sent­ing your true prefer­ence in this form.

The “utilons” of the post linked above look differ­ent from the vN-M ex­pected util­ity be­cause their dis­cus­sion in­volved ar­gu­ment, in­for­mal steps. This doesn’t pre­clude the topic the ar­gu­ment is about, the “utilons”, from be­ing ex­actly the same (ex­pected) util­ity val­ues, ap­prox­i­mated to suit more in­for­mal dis­cus­sion. The differ­ence is that the in­for­mal part of de­ci­sion-mak­ing is con­sid­ered as part of de­ci­sion pro­ce­dure in that post, un­like what hap­pens with the for­mal tool it­self (that is dis­cussed there in­for­mally).

By con­sid­er­ing the dou­ble-my-util­ity thought ex­per­i­ment, the fol­low­ing ques­tion can be con­sid­ered: as­sum­ing that the best pos­si­ble util­ity+prior are cho­sen within the ex­pected util­ity max­i­miza­tion frame­work, do the de­ci­sions gen­er­ated by the re­sult­ing pro­ce­dure look satis­fac­tory? That is, is this form of de­ci­sion pro­ce­dure ad­e­quate, as an ul­ti­mate solu­tion, for all situ­a­tions? The an­swer can be “no”, which would mean that ex­pected util­ity max­i­miza­tion isn’t a way to go, or that you’d need to ap­ply it differ­ently to the prob­lem.

• I’m strug­gling to figure out whether we’re ac­tu­ally dis­agree­ing about any­thing here, and if so, what it is. I agree with most of what you’ve said, but can’t quite see how it con­nects to the point I’m try­ing to make. It seems like we’re some­how man­ag­ing to talk past each other, but un­for­tu­nately I can’t tell whether I’m miss­ing your point, you’re miss­ing mine, or some­thing else en­tirely. Let’s try again… let me know if/​when you think I’m go­ing off the rails here.

If I un­der­stand you cor­rectly, you want to eval­u­ate a par­tic­u­lar de­ci­sion pro­ce­dure “max­i­mize ex­pected util­ity” (MEU) by see­ing whether the re­sults it gives in this situ­a­tion seem cor­rect. (Is that right?)

My point was that the re­sult given by MEU, and the ev­i­dence that this can provide, both de­pend cru­cially on what you mean by util­ity.

One pos­si­bil­ity is that by util­ity, you mean vNM util­ity. In this case, MEU clearly says you should ac­cept the offer. As a re­sult, it’s tempt­ing to say that if you think ac­cept­ing the offer would be a bad idea, then this pro­vides ev­i­dence against MEU (or equiv­a­lently, since the vNM ax­ioms im­ply MEU, that you think it’s ok to vi­o­late the vNM ax­ioms). The prob­lem is that if you vi­o­late the vNM ax­ioms, your choices will have no vNM util­ity rep­re­sen­ta­tion, and Omega couldn’t pos­si­bly promise to dou­ble your vNM util­ity, be­cause there’s no such thing. So for the hy­po­thet­i­cal to make sense at all, we have to as­sume that your prefer­ences con­form to the vNM ax­ioms. More­over, be­cause the vNM ax­ioms nec­es­sar­ily im­ply MEU, the hy­po­thet­i­cal also as­sumes MEU, and it there­fore can’t provide ev­i­dence ei­ther for or against it.*

If the hy­po­thet­i­cal is go­ing to be use­ful, then util­ity needs to mean some­thing other than vNM util­ity. It could mean he­dons, it could mean valu­tilons,** it could mean some­thing else. I do think that re­sponses to the hy­po­thet­i­cal in these cases can provide use­ful ev­i­dence about the value of de­ci­sion pro­ce­dures such as “max­i­mize ex­pected he­dons” (MEH) or “max­i­mize ex­pected valu­tilons” (MEV). My point on this score was sim­ply that there is no par­tic­u­lar rea­son to think that ei­ther MEH or MEV were likely to be an op­ti­mal de­ci­sion pro­ce­dure to be­gin with. They’re cer­tainly not im­plied by the vNM ax­ioms, which re­quire only that you should max­imise the ex­pec­ta­tion of some (pos­i­tive) mono­tonic trans­form of he­dons or valu­tilons or what­ever.*** [ETA: As a spe­cific ex­am­ple, if you de­cide to max­i­mize the ex­pec­ta­tion of a bounded con­cave func­tion of he­dons/​valu­tilons, then even if he­dons/​valu­tilons are un­bounded, you’ll at some point stop tak­ing bets to dou­ble your he­dons/​valu­tilons, but still be an ex­pected vNM util­ity max­i­mizer.]

Does that make sense?

* This also means that if you think MEU gives the “wrong” an­swer in this case, you’ve got­ten con­fused somewe­here—most likely about what it means to dou­ble vNM util­ity.

** I define these here as the out­put of a func­tion that maps a spe­cific, cer­tain, world his­tory (no gam­bles!) into the re­als ac­cord­ing to how well that par­tic­u­lar world his­tory mea­sures up against my val­ues. (Apolo­gies for the pro­lifer­a­tion of ter­minol­ogy—I’m try­ing to guard against the pos­si­bil­ity that we’re us­ing “utilons” to mean differ­ent things with­out in­ad­ver­tently end­ing up in a messy defi­ni­tional ar­gu­ment. ;))

*** A corol­lary of this is that re­ject­ing MEH or MEV does not con­sti­tute ev­i­dence against the vNM ax­ioms.

• You are plac­ing on a test the fol­low­ing well-defined tool: ex­pected util­ity max­i­mizer with a prior and “util­ity” func­tion, that eval­u­ates the events on the world. By “util­ity” func­tion here I mean just some func­tion, so you can drop the word “util­ity”. Even if peo­ple can’t rep­re­sent their prefer­ence as ex­pected some-func­tion max­i­miza­tion, such tool could still be con­structed. The ques­tion is whether such a tool can be made that always agrees with hu­man prefer­ence.

An easy ques­tion is what hap­pens when you use “he­dons” or some­thing else equally in­ad­e­quate in the role of util­ity func­tion: the tool starts to make de­ci­sions with which we dis­agree. Case closed. But maybe there are other set­tings un­der which the tool is in perfect agree­ment with hu­man judg­ment (af­ter re­flec­tion).

Utility-dou­bling thought ex­per­i­ment com­pares what is bet­ter ac­cord­ing to the judg­ment of the tool (to take the card) with what is bet­ter ac­cord­ing to the judg­ment of a per­son (maybe not take the card). As the tool’s de­ci­sion in this thought ex­per­i­ment is made in­var­i­ant on the tool’s set­tings (“util­ity” and prior), show­ing that the tool’s de­ci­sion is wrong ac­cord­ing to a per­son’t prefer­ence (af­ter “care­ful” re­flec­tion), proves that there is no way to set up “util­ity” and prior so that the “util­ity” max­i­miza­tion tool rep­re­sents that per­son’s prefer­ence.

• As the tool’s de­ci­sion in this thought ex­per­i­ment is made in­var­i­ant on the tool’s set­tings (“util­ity” and prior), show­ing that the tool’s de­ci­sion is wrong ac­cord­ing to a per­son’s prefer­ence (af­ter “care­ful” re­flec­tion), proves that there is no way to set up “util­ity”

My ar­gu­ment is that, if Omega is offer­ing to dou­ble vNM util­ity, the set-up of the thought ex­per­i­ment rules out the pos­si­bil­ity that the de­ci­sion could be wrong ac­cord­ing to a per­son’s con­sid­ered prefer­ence (be­cause the claim to be dou­bling vNM util­ity em­bod­ies an as­sump­tion about what a per­son’s con­sid­ered prefer­ence is). AFAICT, the thought ex­per­i­ment then amounts to ask­ing: “If I should max­i­mize ex­pected util­ity, should I max­i­mize ex­pected util­ity?” Re­gard­less of whether I should ac­tu­ally max­i­mize ex­pected util­ity or not, the cor­rect an­swer to this ques­tion is still “yes”. But the thought ex­per­i­ment is com­pletely un­in­for­ma­tive.

Do you un­der­stand my ar­gu­ment for this con­clu­sion? (Fourth para of my pre­vi­ous com­ment.) If you do, can you point out where you think it goes astray? If you don’t, could you tell me what part you don’t un­der­stand so I can try to clar­ify my think­ing?

On the other hand, if Omega is offer­ing to dou­ble some­thing other than vNM util­ity (he­dons/​valu­tilons/​what­ever) then I don’t think we have any dis­agree­ment. (Do we? Do you dis­agree with any­thing I said in para 5 of my pre­vi­ous com­ment?)

My point is just that the thought ex­per­i­ment is un­der­speci­fied un­less we’re clear about what the dou­bling ap­plies to, and that peo­ple some­times seem to shift back and forth be­tween differ­ent mean­ings.

• What you just said seems cor­rect.

What was origi­nally at is­sue is whether we should act in ways that will even­tu­ally de­stroy our­selves.

I think the big-pic­ture con­clu­sion from what you just wrote is that, if we see that we’re act­ing in ways that will prob­a­bly ex­ter­mi­nate life in short or­der, that doesn’t nec­es­sar­ily mean it’s the wrong thing to do.

How­ever, in our cir­cum­stances, time dis­count­ing and “iden­tity dis­count­ing” en­courage us to start en­joy­ing and doom­ing our­selves now; whereas it would prob­a­bly be bet­ter to spread life to a few other galax­ies first, and then en­joy our­selves.

(I ad­mit that my use of the word “bet­ter” is prob­le­matic.)

• if we see that we’re act­ing in ways that will prob­a­bly ex­ter­mi­nate life in short or­der, that doesn’t nec­es­sar­ily mean it’s the wrong thing to do.

Well, I don’t dis­agree with this, but I would still agree with it if you sub­sti­tuted “right” for “wrong”, so it doesn’t seem like much of a con­clu­sion. ;)

• it doesn’t seem like much of a con­clu­sion.

Mov­ing back to­ward your ig­no­rance prior on a topic can still in­crease your log-score if the hy­poth­e­sis was con­cen­trat­ing prob­a­bil­ity mass in the wrong ar­eas (failing to con­cen­trate a sub­stan­tial amount in a right area).

• You ar­gue that the thought ex­per­i­ment is triv­ial and doesn’t solve any prob­lems. In my com­ments above I de­scribed a spe­cific setup that shows how to use (in­ter­pret) the thought ex­per­i­ment to po­ten­tially ob­tain non-triv­ial re­sults.

• I ar­gue that the thought ex­per­i­ment is am­bigu­ous, and that for a cer­tain defi­ni­tion of util­ity (vNM util­ity), it is triv­ial and doesn’t solve any prob­lems. For this defi­ni­tion of util­ity I ar­gue that your ex­am­ple doesn’t work. You do not ap­pear to have en­gaged with this ar­gu­ment, de­spite re­peated re­quests to point out ei­ther where it goes wrong, or where it is un­clear. If it goes wrong, I want to know why, but this con­ver­sa­tion isn’t re­ally helping.

For other defi­ni­tions of util­ity, I do not, and have never claimed that the thought ex­per­i­ment is triv­ial. In fact, I think it is very in­ter­est­ing.

• I ar­gue that the thought ex­per­i­ment is am­bigu­ous, and that for a cer­tain defi­ni­tion of util­ity (vNM util­ity), it is triv­ial and doesn’t solve any prob­lems. For this defi­ni­tion of util­ity I ar­gue that your ex­am­ple doesn’t work.

If by “your ex­am­ple” you re­fer to the setup de­scribed in this com­ment, I don’t un­der­stand what you are say­ing here. I don’t use any “defi­ni­tion of util­ity”, it’s just a pa­ram­e­ter of the tool.

• it’s just a pa­ram­e­ter of the tool.

It’s also an en­tity in the prob­lem set-up. When Omega says “I’ll dou­ble your util­ity”, what is she offer­ing to dou­ble? Without defin­ing this, the prob­lem isn’t well-speci­fied.

• Cer­tainly, you need to re­solve any un­der­speci­fi­ca­tion. There are ways to do this use­fully (or not).

• Agreed. My point is sim­ply that one par­tic­u­lar (tempt­ing) way of re­solv­ing the un­der­speci­fi­ca­tion is non-use­ful. ;)

• It seems like you are as­sum­ing that the only effect of dy­ing is that it brings your util­ity to 0. I agree that af­ter you are dead your util­ity is 0, but be­fore you are dead you have to die, and I think that is a strongly nega­tive util­ity event. When I pic­ture my util­ity play­ing this game, I think that if I start with X, then I draw a start and have 2X. Then I draw a skull, I look at the skull, my util­ity drops to −10000X as I shit my pants and beg omega to let me live, and then he kills me and my util­ity is 0.

I don’t know how much sense that makes math­e­mat­i­cally. But it cer­tainly feels to me like fear of death makes dy­ing a more nega­tive event than just a drop to util­ity 0.

• The skull cards are elec­tro­cuted, and will kill you in­stantly and painlessly as soon as you touch them.

(Be care­ful to touch only the cards you take.)

• As­sum­ing the util­ity in­crease holds my re­main­ing lifes­pan con­stant, I’d draw a card ev­ery few years (if al­lowed). I don’t claim to max­i­mize “ex­pected in­te­gral of hap­piness over time” by do­ing so (sub­sti­tute util­ity for hap­piness if you like; but per­haps util­ity should be for­ward-look­ing and in­clude ex­pected hap­piness over time as just one of my val­ues?). Of course, by sup­pos­ing my util­ity can be dou­bled, I’ll never be fully satis­fied.

• The “jus­tified ex­pec­ta­tion of pleas­ant sur­prises”, as some­one or other said.

• I’d won­dered why no­body brought up MWI and an­thropic prob­a­bil­ities yet.

As for this, it re­minds me of a Dutch book ar­gu­ment Eliezer dis­cussed some time ago. His ar­gu­ment was that in cases where some kind of in­finity is on the table, aiming to satis­fice rather than op­ti­mize can be the bet­ter strat­egy.

In my case (as­sum­ing I’m quite con­fi­dent in Many-Wor­lds), I might de­cide to take a card or two, go off and en­joy my­self for a week, come back and take an­other card or two, et cetera.

• Many wor­lds have noth­ing to do with val­idity of suici­dal de­ci­sions. If you have an an­swer that max­i­mizes ex­pected util­ity but gives al­most-cer­tain prob­a­bil­ity of to­tal failure, you still take it in a de­ter­minis­tic world. There is no magic by which de­ter­minis­tic world de­clares that the de­ci­sion-the­o­retic calcu­la­tion is in­valid in this par­tic­u­lar case, while many-wor­lds lets it be.

• I think you’re right. Would you agree that this is a prob­lem with fol­low­ing the policy of max­i­miz­ing ex­pected util­ity? Or would you keep draw­ing cards?

• This is a var­i­ant on the St. Peters­burg para­dox, in­nit? My preferred re­s­olu­tion is to as­sert that any re­al­iz­able util­ity func­tion is bounded.

• Thanks for the link—this is an­other form of the same para­dox or­th­nor­mal linked to, yes. The Wikipe­dia page pro­poses nu­mer­ous “solu­tions”, but most of them just dodge the ques­tion by tak­ing ad­van­tage of the fact that the para­dox was posed us­ing “ducats” in­stead of “util­ity”. It seems like the no­tion of “util­ity” was in­vented in re­sponse to this para­dox. If you pose it again us­ing the word “util­ity”, these “solu­tions” fail. The only pos­si­bly work­able solu­tion offered on that Wikipe­dia page is:

Re­jec­tion of math­e­mat­i­cal expectation

Var­i­ous au­thors, in­clud­ing Jean le Rond d’Alem­bert and John May­nard Keynes, have re­jected max­i­miza­tion of ex­pec­ta­tion (even of util­ity) as a proper rule of con­duct. Keynes, in par­tic­u­lar, in­sisted that the rel­a­tive risk of an al­ter­na­tive could be suffi­ciently high to re­ject it even were its ex­pec­ta­tion enor­mous.

• The page notes the re­for­mu­la­tion in terms of util­ity, which it terms “su­per St. Peters­berg para­dox”. (It doesn’t have its own sec­tion, or I’d have linked di­rectly to that.) I agree that there doesn’t seem to be a work­able solu­tion—my last re­fuge was just de­stroyed by Vladimir Nesov.

• I agree that there doesn’t seem to be a work­able solu­tion—my last re­fuge was just de­stroyed by Vladimir Nesov.

I’m afraid I don’t un­der­stand the difficulty here. Let’s as­sume that Omega can ac­cess any point in con­figu­ra­tion space and make that the re­al­ity. Then ei­ther (A) at some point it runs out of things with which to en­tice you to draw an­other card, in which case your util­ity func­tion is bounded or (B) it never runs out of such things, in which case your util­ity func­tion in un­bounded.

Why is this so para­dox­i­cal again?

• If it’s not para­dox­i­cal, how many cards would you draw?

• I guess no more than 10 cards. That’s based on not be­ing able to imag­ine a sce­nario such that I’d pre­fer .999 prob­a­bil­ity of death + .001 prob­a­bil­ity of sce­nario to the sta­tus quo. But it’s just a guess be­cause Omega might have bet­ter imag­i­na­tion that I do, or un­der­stand my util­ity func­tion bet­ter than I do.

• Omega offers you the heal­ing of all the rest of Real­ity; ev­ery other sen­tient be­ing will be pre­served at what would oth­er­wise be death and al­lowed to live and grow for­ever, and all un­bear­able suffer­ing not already in your causal past will be pre­vented. You alone will die.

You wouldn’t take a trust­wor­thy 0.001 prob­a­bil­ity of that re­ward and a 0.999 prob­a­bil­ity of death, over the sta­tus quo? I would go for it so fast that there’d be speed lines on my quarks.

Really, this whole de­bate is just about peo­ple be­ing told “X utilons” and in­ter­pret­ing util­ity as hav­ing diminish­ing marginal util­ity—I don’t see any rea­son to sup­pose there’s more to it than that.

• You alone will die.

There’s no rea­son for Omega to kill me in the win­ning out­come...

You wouldn’t take a trust­wor­thy 0.001 prob­a­bil­ity of that re­ward and a 0.999 prob­a­bil­ity of death, over the sta­tus quo?

Well, I’m not as al­tru­is­tic as you are. But there must be some pos­i­tive X such that even you wouldn’t take a trust­wor­thy X prob­a­bil­ity of that re­ward and a 1-X prob­a­bil­ity of death, over the sta­tus quo, right? Sup­pose you’ve drawn enough cards to win this prize, what new prize can Omega offer you to en­tice you to draw an­other card?

• There’s no rea­son for Omega to kill me in the win­ning out­come...

Omega’s a bas­tard. So what?

Well, I’m not as al­tru­is­tic as you are.

WHAT? Are you hon­estly sure you’re THAT not as al­tru­is­tic as I am?

But there must be some pos­i­tive X such that even you wouldn’t take a trust­wor­thy X prob­a­bil­ity of that re­ward and a 1-X prob­a­bil­ity of death, over the sta­tus quo, right?

There’s the prob­lem of whether the sce­nario I de­scribed which in­volves a “for­ever” and “over all space” ac­tu­ally has in­finite util­ity com­pared to in­cre­ments in my own life which even if I would oth­er­wise live for­ever would be over an in­finites­i­mal frac­tion of all space, but if we fix that with a rather smaller prize that I would still ac­cept, then yes of course.

Sup­pose you’ve drawn enough cards to win this prize, what new prize can Omega offer you to en­tice you to draw an­other card?

Heal this Real­ity plus an­other three?

• Omega’s a bas­tard. So what?

That’s fine, I just didn’t know if that de­tail had some im­pli­ca­tion that I was miss­ing.

WHAT? Are you hon­estly sure you’re THAT not as al­tru­is­tic as I am?

Yes, I’m pretty sure, al­though I leave open the pos­si­bil­ity that I may en­counter an ar­gu­ment in the fu­ture that would per­suade me to change my mind. My un­der­stand­ing is that most peo­ple have prefer­ences like mine, so I’m sur­prised that you’re so sur­prised.

It seems that I had missed the ear­lier posts on bounded vs. un­bounded util­ity func­tions. I’ll fol­low up there to avoid re­tread­ing old ground.

• Yes, I’m pretty sure, al­though I leave open the pos­si­bil­ity that I may en­counter an ar­gu­ment in the fu­ture that would per­suade me to change my mind. My un­der­stand­ing is that most peo­ple have prefer­ences like mine, so I’m sur­prised that you’re so sur­prised.

I’m shocked, and I hadn’t thought that most peo­ple had prefer­ences like yours—at least would not ver­bally ex­press such prefer­ences; their “real” prefer­ences be­ing a whole sep­a­rate moral is­sue be­yond that. I would have thought that it would be mainly psy­chopaths, the Rand-dam­aged, and a few un­for­tu­nate moral philoso­phers with mis­taken metaethics, who would de­cline that offer.

I guess I would fol­low up with these ques­tions: (1) When you see some­one else hurt­ing, or at­tend a friend’s funeral, do you feel sad; (2) are you more viscer­ally afraid of your own death than the strength of that emo­tion, if com­par­ing two sin­gle cases; (3) do you de­cline to mul­ti­ply out of a de­liber­ate be­lief that all events af­ter your own death ought to have zero util­ity to you, even if they feel sad when you think about them now; or (4) do you just gen­er­ally want to leave the in­tu­itive judg­ment (2) with its in­nate lack of mul­ti­pli­ca­tion undis­turbed?

Or if I’m ask­ing the wrong ques­tions here, then what is go­ing on? I would ex­pect most hu­mans to in­stinc­tively feel that their whole tribe, to say noth­ing of the en­tire rest of re­al­ity, was worth some­thing; and I would ex­pect a ra­tio­nal­ist to un­der­stand that if their own life does not liter­ally have lex­i­co­graphic pri­or­ity (i.e., lives of oth­ers have in­finites­i­mal=0 value in the util­ity func­tion) then the mul­ti­pli­ca­tion fac­tor here is over­whelming; and I would also ex­pect you, Wei Dai, to not mis­tak­enly be­lieve that you were ra­tio­nally forced to be lex­i­co­graph­i­cally self­ish re­gard­less of your feel­ings… so I’m re­ally not clear on what could be go­ing on here.

I guess my most im­por­tant ques­tion would be: Do you feel that way, or are you de­cid­ing that way? In the former case, I might just need to make a movie show­ing one in­di­vi­d­ual af­ter an­other be­ing healed, and af­ter you’d seen enough of them, you would agree—the visceral emo­tional force hav­ing be­come great enough. In the lat­ter case I’m not sure what’s go­ing on.

PS again: Would you ac­cept a 60% prob­a­bil­ity of death in ex­change for heal­ing the rest of re­al­ity?

• I guess I would fol­low up with these ques­tions: (1) When you see some­one else hurt­ing, or at­tend a friend’s funeral, do you feel sad; (2) are you more viscer­ally afraid of your own death than the strength of that emo­tion, if com­par­ing two sin­gle cases; (3) do you de­cline to mul­ti­ply out of a de­liber­ate be­lief that all events af­ter your own death ought to have zero util­ity to you, even if they feel sad when you think about them now; or (4) do you just gen­er­ally want to leave the in­tu­itive judg­ment (2) with its in­nate lack of mul­ti­pli­ca­tion undis­turbed?

1: Yes. 2: Yes. 3: No. 4: I see a num­ber of rea­sons not to do straight mul­ti­pli­ca­tion:

• Straight mul­ti­pli­ca­tion leads to an ab­surd de­gree of un­con­cern for one­self, given that the num­ber of po­ten­tial peo­ple is as­tro­nom­i­cal. It means, for ex­am­ple, that you can’t watch a movie for en­joy­ment, un­less that some­how in­creases your pro­duc­tivity for sav­ing the world. (In the least con­ve­nient world, watch­ing movies uses up time with­out in­creas­ing pro­duc­tivity.)

• No one has pro­posed a form of util­i­tar­i­anism that is free from para­doxes (e.g., the Repug­nant Con­clu­sion).

• My cur­rent po­si­tion re­sem­bles the “Prox­im­ity ar­gu­ment” from Re­vis­it­ing tor­ture vs. dust specks:

Prox­im­ity ar­gu­ment: don’t ask me to value strangers equally to friends and rel­a­tives. If each ad­di­tional per­son mat­ters 1% less than the pre­vi­ous one, then even an in­finite num­ber of peo­ple get­ting dust specks in their eyes adds up to a finite and not es­pe­cially large amount of suffer­ing.

This agrees with my in­tu­itive judg­ment and also seems to have rel­a­tively few philo­soph­i­cal prob­lems, com­pared to valu­ing ev­ery­one equally with­out any kind of dis­count­ing.

I guess my most im­por­tant ques­tion would be: Do you feel that way, or are you de­cid­ing that way?

My last bul­let above already an­swered this, but I’ll re­peat for clar­ifi­ca­tion: it’s both.

PS again: Would you ac­cept a 60% prob­a­bil­ity of death in ex­change for heal­ing the rest of re­al­ity?

This should be clear from my an­swers above as well, but yes.

• Oh, ’ello. Glad to see some­body still re­mem­bers the prox­im­ity ar­gu­ment. But it’s adapted to our world where you gen­er­ally can­not kill a mil­lion dis­tant peo­ple to make one close rel­a­tive happy. If we move to a world where Omegas reg­u­larly ask peo­ple difficult ques­tions, a lot of peo­ple adopt­ing prox­im­ity rea­son­ing will cause a huge tragedy of the com­mons.

About Eliezer’s ques­tion, I’d ex­change my life for a re­li­able 0.001 chance of heal­ing re­al­ity, be­cause I can’t imag­ine liv­ing mean­ingfully af­ter be­ing offered such a wa­ger and re­fus­ing it. Can’t imag­ine how I’d look other LW users in the eye, that’s for sure.

• Can’t imag­ine how I’d look other LW users in the eye, that’s for sure.

I pub­li­cly re­jected the offer, and don’t feel like a pariah here. I won­der what is the ac­tual de­gree of al­tru­ism among LW users. Should we set up a poll and gather some ev­i­dence?

• Co­op­er­a­tion is a differ­ent con­sid­er­a­tion from prefer­ence. You can pre­fer only to keep your own “body” in cer­tain dy­nam­ics, no mat­ter what hap­pens to the rest of the world, and still benefit the most from, roughly speak­ing, helping other agents. Which can in­clude oc­ca­sional self-sac­ri­fice a la coun­ter­fac­tual mug­ging.

• No one has pro­posed a form of util­i­tar­i­anism that is free from para­doxes (e.g., the Repug­nant Con­clu­sion).

I’d be in­ter­ested to know what you think of Crit­i­cal-Level Utili­tar­i­anism and Pop­u­la­tion-Rel­a­tive Bet­ter­ness as ways of avoid­ing the re­pug­nant con­clu­sion and other prob­lems.

• So does your an­swer change once you’ve drawn 10 cards and are still al­ive?

• No, if my guess is cor­rect, then some time be­fore I’m offered the 11th card, Omega will say “I can’t dou­ble your util­ity again” or equiv­a­lently, “There is no prize I can offer you such that you’d pre­fer a .5 prob­a­bil­ity of it to keep­ing what you have.”

• After fur­ther thought, I see that case (B) can be quite para­dox­i­cal. Con­sider Eliezer’s util­ity func­tion, which is sup­pos­edly un­bounded as a func­tion of how many years he lives. In other words, Omega can in­crease Eliezer’s util­ity with­out bound just by giv­ing him in­creas­ingly longer lives. Ex­pected util­ity max­i­miza­tion then dic­tates that he keeps draw­ing cards one af­ter an­other, even though he knows that by do­ing so, with prob­a­bil­ity 1 he won’t live to en­joy his re­wards.

• When you go to in­finity, you’d need to define ad­di­tional math­e­mat­i­cal struc­ture that an­swers your ques­tion. You can’t just con­clude that the cor­rect course of ac­tion is to keep draw­ing cards for eter­nity, do­ing noth­ing else. Even if at each mo­ment the right ac­tion is to draw one more card, when you con­sider the over­all strat­egy, the strat­egy of draw­ing cards for all time may be a wrong strat­egy.

For ex­am­ple, con­sider the fol­low­ing prefer­ence on in­finite strings. A string has util­ity 0, un­less it has the form 11111.....11112222...., that is a finite num­ber of 1 fol­lowed by in­finite num­ber of 2, in which case its util­ity is the num­ber of 1s. Clearly, a string of this form with one more 1 has higher util­ity than a string with­out, and so a string with one more 1 should be preferred. But a string con­sist­ing only of 1s doesn’t have the non-zero-util­ity form, be­cause it doesn’t have the tail of in­finite num­ber of 2s. It’s a fal­lacy to fol­low an in­cre­men­tal ar­gu­ment to in­finity. In­stead, one must fol­low a one-step ar­gu­ment that con­sid­ers the in­finite ob­jects as whole.

• What you say sounds rea­son­able, but I’m not sure how I can ap­ply it in this ex­am­ple. Can you elab­o­rate?

Con­sider Eliezer’s choice of strate­gies at the be­gin­ning of the game. He can ei­ther stop af­ter draw­ing n cards for some in­te­ger n, or draw an in­finite num­ber of cards. First, (sup­pos­ing it takes 10 sec­onds to draw a card)

EU(draw an in­finite num­ber of cards) = 12 U(live 10 sec­onds) + 14 U(live 20 sec­onds) + 18 U(live 30 sec­onds) …

which ob­vi­ously con­verges to a small num­ber. On the other hand, EU(stop af­ter n+1 cards) > EU(stop af­ter n cards) for all n. So what should he do?

• This ex­poses a hole in the prob­lem state­ment: what does the Omega’s prize mea­sure? We de­ter­mined that U0 is the coun­ter­fac­tual where Omega kills you, U1 is the coun­ter­fac­tual where it does noth­ing, but what is U2=U1+3*(U1-U0)? This seems to be the ex­pected util­ity of the event where you draw the lucky card, in which case this event con­tains, in par­tic­u­lar, your fu­ture de­ci­sions to con­tinue draw­ing cards. But if it’s so, it places a limit on how your util­ity can be im­proved fur­ther dur­ing the lat­ter rounds, since if your util­ity con­tinues to in­crease, it con­tra­dicts the state­ment in the first round that your util­ity is go­ing to be only U2, and no more. Utility can’t change, as each util­ity is a val­u­a­tion of a spe­cific event in the sam­ple space.

So, the al­ter­na­tive for­mu­la­tion that re­moves this con­tra­dic­tion is for Omega to only as­sert that the ex­pected util­ity given that you re­ceive a lucky card is no less than U2. In this case the right strat­egy seems to be con­tinue draw­ing cards in­definitely, since the util­ity you re­ceive could be in some­thing other than your own life, now spent draw­ing cards only.

This how­ever seems to sidestep the is­sue. What if the only util­ity you see is in the fu­ture ac­tions you do, which don’t in­clude pick­ing cards, and you can’t in­ter­leave cards with other ac­tions, that is you must al­lot a given amount of time to pick­ing cards.

You can re­cast the prob­lem of choos­ing each of the in­finite num­ber of de­ci­sions (or one among all available in some sense in­finite se­quences of de­ci­sions) to the prob­lem of choos­ing a finite “seed” strat­egy for mak­ing de­ci­sions. Say, only a finite num­ber of strate­gies is available, for ex­am­ple only what fits in the mem­ory of the com­puter that starts the en­ter­prise, that could since the start of the ex­per­i­ment be ex­panded, but the first ver­sion has a speci­fied limit. In this case, the right pro­gram is as close to Busy Beaver is you can get, that is you draw cards as long as pos­si­ble, but only finitely long, and af­ter that you stop and go on to en­joy the ac­tual life.

• Why are you treat­ing time as in­finite? Surely it’s finite, just tak­ing un­bounded val­ues?

Even if at each mo­ment the right ac­tion is to draw one more card, when you con­sider the over­all strat­egy, the strat­egy of draw­ing cards for all time may be a wrong strat­egy.

But you’re not asked to de­cide a strat­egy for all of time. You can change your de­ci­sion at ev­ery round freely.

• But you’re not asked to de­cide a strat­egy for all of time. You can change your de­ci­sion at ev­ery round freely.

You can’t change any fixed thing, you can only de­ter­mine it. Change is a time­ful con­cept. Change ap­pears when you com­pare now and to­mor­row, not when you com­pare the same thing with it­self. You can’t change the past, and you can’t change the fu­ture. What you can change about the fu­ture is your plan for the fu­ture, or your knowl­edge: as the time goes on, your idea about a fact in the now be­comes a differ­ent idea to­mor­row.

When you “change” your strat­egy, what you are re­ally do­ing is chang­ing your mind about what you’re plan­ning. The ques­tion you are try­ing to an­swer is what to ac­tu­ally do, what de­ci­sions to im­ple­ment at each point. A strat­egy for all time is a gen­er­a­tor of de­ci­sions at each given mo­ment, an al­gorithm that runs and out­puts a stream of de­ci­sions. If you know some­thing about each par­tic­u­lar de­ci­sion, you can make a gen­eral state­ment about the whole stream. If you know that each next de­ci­sion is go­ing to be “ac­cept” as op­posed to “de­cline”, you can prove that the re­sult­ing stream is equiv­a­lent to an in­finite stream that only an­swers “ac­cept”, at all steps. And at the end, you have a pro­cess, the con­se­quences of your de­ci­sion-mak­ing al­gorithm con­sist in all of the de­ci­sions. You can’t change that con­se­quence, as the con­se­quence is what ac­tu­ally hap­pens, if you changed your mind about mak­ing a par­tic­u­lar de­ci­sion along the way, the effect of that change is already fac­tored in in the re­sult­ing stream of ac­tions.

The con­se­quen­tial­ist prefer­ence is go­ing to com­pare the effect of the whole in­finite stream of po­ten­tial de­ci­sions, and un­til you know about the finite­ness of the fu­ture, the state space is go­ing to con­tain el­e­ments cor­re­spond­ing to the in­finite de­ci­sion traces. In this state space, there is an in­finite stream cor­re­spond­ing to one de­cid­ing to con­tinue pick­ing cards for eter­nity.

• Thanks, I un­der­stand now.

• Whoa.

Is there some­thing I can take that would help me un­der­stand that bet­ter?

• Does Omega’s util­ity dou­bling cover the con­tents of the as-yet-un­touched deck? It seems to me that it’d be pretty spiffy re: my util­ity func­tion for the deck to have a re­duced chance of kil­ling me.

• At first I thought this was pretty funny, but even if you were jok­ing, it may ac­tu­ally map to the ex­tinc­tion prob­lem, since each new tech­nol­ogy has a chance of mak­ing ex­tinc­tion less likely, as well. As an ex­am­ple, nu­clear tech­nol­ogy had some prob­a­bil­ity of kil­ling ev­ery­one, but also some prob­a­bil­ity of mak­ing Orion ships pos­si­ble, al­low­ing di­as­pora.

• While I’m gam­ing the sys­tem, my life­time util­ity func­tion (if I have one) could prob­a­bly be dou­bled by giv­ing me a rea­son­able suite of su­per­pow­ers, some of which would let me iden­tify the rest of the cards in the deck (X-ray vi­sion, pre­cog pow­ers, etc.) or be pro­tected from what­ever mechanism the skull cards use to kill me (im­mu­nity to elec­tric­ity or just straight-up in­vuln­er­a­bil­ity). Is it a stipu­la­tion of the sce­nario that noth­ing Omega does to tweak the util­ity func­tion upon draw­ing a star af­fects the risks of draw­ing from the deck, di­rectly or in­di­rectly?

• It should be, es­pe­cially since the ex­is­ten­tial-risk prob­lems that we’re try­ing to model aren’t known to come with su­per­pow­ers or other such es­cape hatches.

• Yeesh. I’m chang­ing my mind again tonight. My only ex­cuse is that I’m sick, so I’m not think­ing as straight as I might.

I was origi­nally think­ing that Vladimir Nesov’s re­for­mu­la­tion showed that I would always ac­cept Omega’s wa­ger. But now I see that at some point U1+3*(U1-U0) must ex­ceed any up­per bound (as­sum­ing I sur­vive that long).

Given U1 (util­ity of re­fus­ing ini­tial wa­ger), U0 (util­ity of death), U_max, and U_n (util­ity of re­fus­ing wa­ger n as­sum­ing you sur­vive that long), it might be pos­si­ble that there is a se­quence of wa­gers that (i) offer pos­i­tive ex­pected util­ity at each step; (ii) asymp­tot­i­cally ap­proach the up­per bound if you sur­vive; and (iii) have a prob­a­bil­ity of sur­vival ap­proach­ing zero. I con­fess I’m in no state to cope with the math nec­es­sary to give such a se­quence or dis­prove its ex­is­tence.

• There is no such se­quence. Proof:

In or­der for wa­ger n to be non­nega­tive ex­pected util­ity, P(death)*U_0 + (1-P(death))*U_(n+1) >= U_n. Equiv­a­lently, P(death this time | sur­vived un­til n) ⇐ (U_(n+1)-U_n) /​ (U_(n+1)-U0).

As­sume the worst case, equal­ity. Then the cu­mu­la­tive prob­a­bil­ity of sur­vival de­creases by ex­actly the same fac­tor as your util­ity (con­di­tioned on sur­vival) in­creases. This is sim­ple mul­ti­pli­ca­tion, so it’s true of a se­quence of bor­der­line wa­gers too.

With a bounded util­ity func­tion, the worst se­quence of wa­gers you’ll ac­cept in to­tal is P(death) ⇐ (U_max-U0)/​(U1-U0). Which is ex­actly what you’d ex­pect.

• When there’s an in­finite num­ber of wa­gers, there can be a dis­tinc­tion be­tween ac­cept­ing the whole se­quence at one go and ac­cept­ing each wa­ger one af­ter an­other. (There’s a para­dox as­so­ci­ated with this dis­tinc­tion, but I for­get what it’s called.) Your sec­ond-last sen­tence seems to be a con­clu­sion about ac­cept­ing the whole se­quence at one go, but I’m wor­ried about ac­cept­ing each wa­ger one af­ter an­other. Is the dis­tinc­tion im­por­tant here?

• there can be a dis­tinc­tion be­tween ac­cept­ing the whole se­quence at one go and ac­cept­ing each wa­ger one af­ter an­other.

Are you think­ing of the Rie­mann se­ries the­o­rem? That doesn’t ap­ply when the pay­off ma­trix for each bet is the same (and finite).

• No, it was this thing. I just couldn’t ar­tic­u­late it.

• A bounded util­ity func­tion prob­a­bly gets you out of all prob­lems along those lines.

Cer­tainly it’s good in the par­tic­u­lar case: your ex­pected util­ity (in the ap­pro­pri­ate sense) is an in­creas­ing func­tion of bets you ac­cept and in­creas­ing se­quences don’t have con­ver­gence is­sues.

• How would you bound your util­ity func­tion? Just pick some ar­bi­trary con­verg­ing func­tion f, and set util­ity’ = f(util­ity)? That seems ar­bi­trary. I sus­pect it might also make the­o­rems about ex­pec­ta­tion max­i­miza­tion break down.

• No, I’m not ad­vo­cat­ing chang­ing util­ity func­tions. I’m just say­ing that if your util­ity func­tion is bounded, you don’t have ei­ther of these prob­lems with in­finity. You don’t have the con­ver­gence prob­lem nor the origi­nal prob­lem of prob­a­bil­ity of the good out­come go­ing to zero. Of course, you still have the re­sult that you keep mak­ing bets till your util­ity is maxed out with very low prob­a­bil­ity, which both­ers some peo­ple.

• How would it help if this se­quence ex­isted?

• If the se­quence ex­ists, then the para­dox* per­sists even in the face of bounded util­ity func­tions. (Or pos­si­bly it already per­sists, as Vladimir Nesov ar­gued and you agreed, but my cold-virus-ad­dled wits aren’t sharp enough to see it.)

* The para­dox is that each wa­ger has pos­i­tive ex­pected util­ity, but ac­cept­ing all wa­gers leads to death al­most surely.

• Ah. So you don’t want the se­quence to ex­ist.

• In the sense that if it ex­ists, then it’s a bul­let I will bite.

• Why is re­jec­tion of math­e­mat­i­cal ex­pec­ta­tion an un­work­able solu­tion?

This isn’t the only sce­nario where straight ex­pec­ta­tion is prob­le­matic. Pas­cal’s Mug­ging, time­less de­ci­sion the­ory, and max­i­miza­tion of ex­pected growth rate come to mind. That makes four.

In my opinion, LWers should not give ex­pected util­ity max­i­miza­tion the same ax­io­matic sta­tus that they award con­se­quen­tial­ism. Is this worth a top level post?

• This is ex­actly my take on it also.

There is a model which is stan­dard in eco­nomics which say “peo­ple max­i­mize ex­pected util­ity; risk averse­ness arises be­cause util­ity func­tions are con­cave”. This has always struck me as ex­tremely fishy, for two rea­sons: (a) it gives rise to para­doxes like this, and (b) it doesn’t at all match what mak­ing a choice feels like for me: if some­one offers me a risky bet, I feel in­clined to re­ject it be­cause it is risky, not be­cause I have done some ex­ten­sive in­te­gra­tion of my util­ity func­tion over all pos­si­ble out­comes. So it seems a much safer as­sump­tion to just as­sume that peo­ple’s prefer­ences are a func­tion from prob­a­bil­ity dis­tri­bu­tions of out­comes, rather than mak­ing the more re­stric­tive as­sump­tion that that func­tion has to arise as an in­te­gral over util­ities of in­di­vi­d­ual out­comes.

So why is the “ex­pected util­ity” model so pop­u­lar? A cou­ple of months ago I came across a blog-post which pro­vides one clue: it pointed out that stan­dard zero-sum game the­ory works when play­ers max­i­mize ex­pected util­ity, but does not work if they have prefer­ences about prob­a­bil­ity dis­tri­bu­tions of out­comes (since then in­tro­duc­ing mixed strate­gies won’t work).

So an economist who wants to ap­ply game the­ory will be in­clined to as­sume that ac­tors are max­i­miz­ing ex­pected util­ity; but we LWers shouldn’t nec­es­sar­ily.

• There is a model which is stan­dard in eco­nomics which say “peo­ple max­i­mize ex­pected util­ity; risk averse­ness arises be­cause util­ity func­tions are con­vex”.

Do you mean con­cave?

A cou­ple of months ago I came across a blog-post which pro­vides one clue: it pointed out that stan­dard zero-sum game the­ory works when play­ers max­i­mize ex­pected util­ity, but does not work if they have prefer­ences about prob­a­bil­ity dis­tri­bu­tions of out­comes (since then in­tro­duc­ing mixed strate­gies won’t work).

Tech­ni­cally speak­ing, isn’t max­i­miz­ing ex­pected util­ity a spe­cial case of hav­ing prefer­ences about prob­a­bil­ity dis­tri­bu­tions about out­comes? So maybe you should in­stead say “does not work el­e­gantly if they have ar­bi­trary prefer­ences about prob­a­bil­ity dis­tri­bu­tions.”

This is what I tend to do when I’m hav­ing con­ver­sa­tions in real life; let’s see how it works on­line :-)

• Do you mean con­cave?

Yes, thanks. I’ve fixed it.

• What does it mean, tech­ni­cally, to have a prefer­ence “about” prob­a­bil­ity dis­tri­bu­tions?

• I think I and John Maxwell IV mean the same thing, but here is the way I would phrase it. Sup­pose some­one is offer­ing me the pick a ticket for one of a range of differ­ent lot­ter­ies. Each lot­tery offers the same set of prizes, but de­pend­ing on which lot­tery I par­ti­ci­pate in, the prob­a­bil­ity of win­ning them is differ­ent.

I am an agent, and we as­sume I have a prefer­ence or­der on the lot­ter­ies—e.g. which ticket I want the most, which ticket I want the least, and which tick­ets I am in­differ­ent be­tween. The ac­tion that will be ra­tio­nal for me to take de­pends on which ticket I want.

I am say­ing that a gen­eral the­ory of ra­tio­nal ac­tion should deal with ar­bi­trary prefer­ence or­ders for the tick­ets. The more stan­dard the­ory re­stricts at­ten­tion to prefer­ence or­ders that arise from first as­sign­ing a util­ity value to each prize and then com­put­ing the ex­pected util­ity for each ticket.

• Let’s define an “ex­per­i­ment” as some­thing that ran­domly changes an agent’s util­ity based on some prob­a­bil­ity den­sity func­tion. An agent’s “de­sire” for a given ex­per­i­ment is the amount of util­ity Y such that the agent is in­differ­ent be­tween the ex­per­i­ment oc­cur­ring and hav­ing their util­ity changed by Y.

From Pfft we see that economists as­sume that for any given agent and any given ex­per­i­ment, the agent’s de­sire for the ex­per­i­ment is equal to $\int_{-\infty}^{\infty}xf\(x\$dx), where x is an amount of util­ity and f(x) gives the prob­a­bil­ity that the ex­per­i­ment’s out­come will be chang­ing the agent’s util­ity by x. In other words, economists as­sume that agents de­sire ex­per­i­ments ac­cord­ing to their ex­pec­ta­tion, which is not nec­es­sar­ily a good as­sump­tion.

• Hmm… I hope you in­ter­pret your own words so that what you write comes out cor­rect, your lan­guage is im­pre­cise and at first I didn’t see a way to read what you wrote that made sense.

When I reread your com­ment to which I asked my ques­tion with this new per­spec­tive, the ques­tion dis­ap­peared. By “prefer­ence about prob­a­bil­ity dis­tri­bu­tions” you sim­ply mean prefer­ence over events, that doesn’t nec­es­sar­ily satisfy ex­pected util­ity ax­ioms.

ETA: Note that in this case, there isn’t nec­es­sar­ily a way of as­sign­ing (sub­jec­tive) prob­a­bil­ities, as sub­jec­tive prob­a­bil­ities fol­low from prefer­ences, but only if the prefer­ences are of the right form. Thus, say­ing that those not-ex­pected-util­ity prefer­ences are over prob­a­bil­ity dis­tri­bu­tions is more con­cep­tu­ally prob­le­matic than say­ing that they are over events. If you don’t use prob­a­bil­ities in the de­ci­sion al­gorithm, prob­a­bil­ities don’t mean any­thing.

• Hmm… I hope you in­ter­pret your own words so that what you write comes out cor­rect, your lan­guage is im­pre­cise and at first I didn’t see a way to read what you wrote that made sense.

I am ea­ger to im­prove. Please give spe­cific sug­ges­tions.

By “prefer­ence about prob­a­bil­ity dis­tri­bu­tions” you sim­ply mean prefer­ence over events, that doesn’t nec­es­sar­ily satisfy ex­pected util­ity ax­ioms.

Right.

Note that in this case, there isn’t nec­es­sar­ily a way of as­sign­ing (sub­jec­tive) prob­a­bil­ities, as sub­jec­tive prob­a­bil­ities fol­low from prefer­ences, but only if the prefer­ences are of the right form.

Hm? I thought sub­jec­tive prob­a­bil­ities fol­lowed from prior prob­a­bil­ities and ob­served ev­i­dence and stuff. What do prefer­ences have to do with them?

Thus, say­ing that those not-ex­pected-util­ity prefer­ences are over prob­a­bil­ity dis­tri­bu­tions is more con­cep­tu­ally prob­le­matic than say­ing that they are over events.

Are you us­ing my tech­ni­cal defi­ni­tion of event or the stan­dard defi­ni­tion?

Prob­a­bly I should not have re­defined “event”; I now see that my use is non­stan­dard. Hope­fully I can clar­ify things. Let’s say I am go­ing to roll a die and give you a num­ber of dol­lars equal to the num­ber of spots on the face left point­ing up­ward. Ac­cord­ing to my (poorly cho­sen) use of the word “event”, the pro­cess of rol­ling the die is an “event”. Ac­cord­ing to what I sus­pect the stan­dard defi­ni­tion is, the die land­ing with 4 spots face up would be an “event”. To clear things up, I sug­gest that we re­fer to the rol­ling of the die as an “ex­per­i­ment”, and 4 spots land­ing face up as an “out­come”. I’m go­ing to rewrite my com­ment with this new ter­minol­ogy. I’m also re­plac­ing “value” with “de­sire”, for what it’s worth.

If you don’t use prob­a­bil­ities in the de­ci­sion al­gorithm, prob­a­bil­ities don’t mean any­thing.

The way I want to eval­u­ate the de­sir­a­bil­ity of an ex­per­i­ment is more com­pli­cated than sim­ply com­put­ing its ex­pected value. But I still use prob­a­bil­ities. I would not give Pas­cal’s mug­ger any money. I would think very care­fully about an ex­per­i­ment that had a 99% prob­a­bil­ity of get­ting me kil­led and a 1% prob­a­bil­ity of gen­er­at­ing 101 times as much util­ity as I ex­pect to gen­er­ate in my life­time, whereas a perfect ex­pected util­ity max­i­mizer would take this deal in an in­stant. Etc.

• Roughly speak­ing, event is a set of al­ter­na­tive pos­si­bil­ities. So, the whole roll of a die is an event (set of all pos­si­ble out­comes of a roll), as well as spe­cific out­comes (sets that con­tain a sin­gle out­come). See prob­a­bil­ity space for a more de­tailed defi­ni­tion.

One way of defin­ing prior and util­ity is just by first tak­ing a prefer­ence over the events of sam­ple space, and then choos­ing any pair prior+util­ity such that ex­pected util­ity calcu­lated from them in­duces the same or­der on events. Of course, the origi­nal or­der on events has to be “nice” in some sense for it to be pos­si­ble to find prior+util­ity that have this prop­erty.

Any ob­ser­va­tions and up­dat­ing con­sist in choos­ing what events you work with. Once prior is fixed, it never changes.

(Of course, you should read up on the sub­ject in greater de­tail than I hint at.)

• One way of defin­ing prior and util­ity is just by first tak­ing a prefer­ence over the events of sam­ple space, and then choos­ing any pair prior+util­ity such that ex­pected util­ity calcu­lated from them in­duces the same or­der on events.

Um, isn’t that ob­vi­ously wrong? It sounds like your are sug­gest­ing that we say “I like play­ing black­jack bet­ter than play­ing the lot­tery, so I should choose a prior prob­a­bil­ity of win­ning each and a util­ity as­so­ci­ated with win­ning each so that that prefer­ence will re­main con­sis­tent when I switch from ‘prefer­ence mode’ to ‘util­i­tar­ian mode’.” Wouldn’t it be bet­ter to choose the util­ities of win­ning based on the prizes they give? And choose the pri­ors for each based on study­ing the his­tory of each game care­fully?

Any ob­ser­va­tions and up­dat­ing con­sist in choos­ing what events you work with. Once prior is fixed, it never changes.

Events are sets of out­comes, right? It sounds like you are sug­gest­ing that peo­ple up­date their prob­a­bil­ities by reshuffling which out­comes go with which events. Aren’t events just a layer of for­mal­ity over out­comes? Isn’t real learn­ing what hap­pens when you change your es­ti­ma­tions of the prob­a­bil­ities of out­comes, not when you re­clas­sify them?

It al­most seems to me as if we are talk­ing past each other… I think I need a bet­ter back­ground on this stuff. Can you recom­mend any books that ex­plain prob­a­bil­ity for the lay­man? I already read a large sec­tion of one, but ap­par­ently it wasn’t very good...

Although I do think there is a chance you are wrong. I see you mix­ing up out­come-de­sir­a­bil­ity es­ti­mates with chance-of-out­come es­ti­mates, which seems ob­vi­ously bad.

• If you don’t want the choice of prefer­ence to turn out bad for you, choose good prefer­ence ;-) There is no free­dom in choos­ing your prefer­ence, as the “choice” is it­self a de­ci­sion-con­cept, defined in terms of prefer­ence, and can’t be a party to the defi­ni­tion of prefer­ence. When you are speak­ing of a par­tic­u­lar choice of prefer­ence be­ing bad or fool­ish, you are judg­ing this choice from the refer­ence frame of some other prefer­ence, while with prefer­ence as foun­da­tion of de­ci­sion-mak­ing, you can’t go through this step. It re­ally is that ar­bi­trary. See also: Pri­ors as Math­e­mat­i­cal Ob­jects, Prob­a­bil­ity is Sub­jec­tively Ob­jec­tive.

You are con­fus­ing prob­a­bil­ity space and its prior (the fun­da­men­tal struc­ture that bind the rest to­gether) with ran­dom vari­ables and their prob­a­bil­ity dis­tri­bu­tions (things that are based on prob­a­bil­ity space and that “in­ter­act” with each other through the defi­ni­tion in terms of the com­mon prob­a­bil­ity space, re­stricted to com­mon events). In­for­mally, when you up­date a ran­dom vari­able given ev­i­dence (event) X, it means that you re­calcu­late the prob­a­bil­ity dis­tri­bu­tion of that vari­able only based on the re­main­ing el­e­ments of the prob­a­bil­ity space within event X. Since this can of­ten be done us­ing other prob­a­bil­ity dis­tri­bu­tions of var­i­ous vari­ables ly­ing around, you don’t always see the prob­a­bil­ity space ex­plic­itly.

• Why is re­jec­tion of math­e­mat­i­cal ex­pec­ta­tion an un­work­able solu­tion?

Well, re­jec­tion’s not a solu­tion per se un­til you pick some­thing jus­tifi­able to re­place it with.

I’d be in­ter­ested in a top-level post on the sub­ject.

• If this con­di­tion makes a differ­ence to you, your an­swer must also be to take as many cards as Omega has to offer.

• I don’t fol­low.

(My as­ser­tion im­plies that Omega can­not dou­ble my util­ity in­definitely, so it’s in­con­sis­tent with the prob­lem as given.)

• You’ll just have to con­struct a less con­ve­nient pos­si­ble world where Omega has merely trillion cards and not an in­finite amount of them, and an­swer the ques­tion about tak­ing a trillion cards, which, if you ac­cept the lot­tery all the way, leaves you with 2 to the trillionth power odds of dy­ing. Find my re­for­mu­la­tion of the topic prob­lem here.

• Agreed.

• Gotcha. Nice re­for­mu­la­tion.

• His ar­gu­ment was that in cases where some kind of in­finity is on the table, aiming to satis­fice rather than op­ti­mize can be the bet­ter strat­egy.

Can we ap­ply that to de­ci­sions about very-long-term-but-not-in­finitely-long times and very-small-but-not-in­finitely-small risks?

Hmm… it ap­pears not. So I don’t think that helps us.

Where did you get the term “satis­fice”? I just read that dutch-book post, and while Eliezer points out the flaw in de­mand­ing that the Bayesian take the in­finite bet, I didn’t see the word ‘satis­fic­ing’ in their any­where.

• Huh, I must have “re­mem­bered” that term into the post. What I mean is more suc­cinctly put in this com­ment.

Can we ap­ply that to de­ci­sions about very-long-term-but-not-in­finitely-long times and very-small-but-not-in­finitely-small risks?

Hmm… it ap­pears not. So I don’t think that helps us.

This ques­tion still con­fuses me, though; if it’s a rea­son­able strat­egy to stop at N in the in­finite case, but not a rea­son­able strat­egy to stop at N if there are only N^^^N iter­a­tions… some­thing about it dis­turbs me, and I’m not sure that Eliezer’s an­swer is ac­tu­ally a good patch for the St. Peters­burg Para­dox.

• It’s an old AI term mean­ing roughly “find a solu­tion that isn’t (likely) op­ti­mal, but good enough for some pur­pose, with­out too much effort”. It im­plies that ei­ther your com­puter is too slow for it to be eco­nom­i­cal to find the true op­ti­mum un­der your mod­els, or that you’re too dumb to come up with the right mod­els, thus the pop­u­lar­ity of the idea in AI re­search.

You can be im­pressed if some­one starts with a crite­ria for what “good enough” means, and then comes up with a method they can prove meets the crite­ria. Other­wise it’s spin.

• I’m more used to it as a psy­chol­ogy (or be­hav­ior econ) term for a spe­cific, psy­cholog­i­cally re­al­is­tic, form of bounded ra­tio­nal­ity. In par­tic­u­lar, I’m used to it be­ing nega­tive! (that is, a heuris­tic which of­ten de­gen­er­ates pro­duces a bias)

• But even if you don’t be­lieve in many wor­lds, I think you do the same thing, un­less you are not max­i­miz­ing ex­pected util­ity. (Un­less chance is quan­tized so that there is a min­i­mum pos­si­ble pos­si­bil­ity. I don’t think that would help much any­way.)

Or un­less your util­ity func­tion is bounded above, and the util­ity you as­sign to the sta­tus quo is more than the av­er­age of the util­ity of dy­ing straight away and the up­per bound of your util­ity func­tion, in which case Omega couldn’t pos­si­bly dou­ble your util­ity. (In­deed, I can’t think of any X right now such that I’d pre­fer {50% X, 10% I die right now, 40% busi­ness as usual} to {100% busi­ness as usual}.)

• I don’t be­lieve in quan­tifyable util­ity (and thus not in dou­bled util­ity) so I take no cards. But yeah, that looks like a way to make util­i­tar­ian equiv­a­lent to suici­dal.

• If I draw cards un­til I die, my ex­pected util­ity is pos­i­tive in­finity. Though I will al­most surely die and end up with util­ity 0, it is log­i­cally pos­si­ble that I will never die, and end up with a util­ity of pos­i­tive in­finity. In this case, 10 + 0(pos­i­tive in­finity) = pos­i­tive in­finity.

The next para­graph re­quires that you as­sume our ini­tial util­ity is 1.

If you want, warp the prob­lem into an iso­mor­phic prob­lem where the prob­a­bil­ities are differ­ent and all util­ities are finite. (Isn’t it cool how you can do that?) In the origi­nal prob­lem, there’s always a 56 chance of util­ity dou­bling and a 16 chance of it go­ing to 12. (Be­ing dead isn’t THAT bad, I guess.) Let’s say that where your util­ity func­tion was U(w), it is now f(U(w)), where f(x) = 1 − 1/​(2 + log_2 x). In this case, the util­ities 12, 1, 2, 4, 8, 16, . . . be­come 0, 12, 23, 34, 45, 56, . . . . So, your ini­tial util­ity is 12, and Omega will ei­ther lower your util­ity to 0 or raise it by ap­ply­ing the func­tion U’ = U/​(U + 1). Your ex­pected util­ity af­ter draw­ing once was pre­vi­ously U’ = 53U + 12; it’s now… okay, my math-stamina has run out. But if you calcu­late ex­pected util­ity, and then calcu­late the prob­a­bil­ity that re­sults in that ex­pected util­ity, I’m bet­ting that you’ll end up with a 12 prob­a­bil­ity of ever* dy­ing.

(The above para­graph sur­round­ing a nut: any uni­verse can be in­ter­preted as one where the prob­a­bil­ities are differ­ent and the util­ity func­tion has been changed to match… of­ten, prob­a­bly.)

• This is com­pletely off topic (and maybe I’m just not get­ting the joke) but does Many Wor­lds nec­es­sar­ily im­ply many hu­man wor­lds? Star Trek tropes aside, I was un­der the im­pres­sion that Many Wor­lds only mat­tered to gluons and Shrod­inger’s Cat—that us macro crea­tures are pretty much screwed.

...

You were jok­ing, weren’t you? I like jokes.

• “Many wor­lds” here is short­hand for “ev­ery time some event hap­pens that has more than one pos­si­ble out­come, for ev­ery pos­si­ble out­come, there is (or comes into be­ing) a world in which that was the out­come.”

As far as the truth or falsity of Many Wor­lds mat­ter­ing to us—I don’t think it can mat­ter, if you max­i­mize ex­pected util­ity (over the many wor­lds).

• That is not what Many Wolds says. It is only about quan­tum out­comes, not “pos­si­ble” out­comes.

• Dou­ble your util­ity for the rest of your life com­pared to what? If you draw cards un­til you die, that sounds like it just means you have twice as much fun draw­ing cards as you would have with­out help. I guess that could be lots of fun if you’re the kind of per­son who gets a rush off of Rus­sian roulette un­der nor­mal cir­cum­stances, but if you’re not, you’d prob­a­bly be bet­ter off flip­ping off Omega and watch­ing some TV.

What if your util­ity would have been nega­tive? Doesn’t dou­bling it make it twice as bad?

• Good point. Bet­ter not draw a card if you have nega­tive util­ity.

Just trust that Omega can dou­ble your util­ity, for the sake of ar­gu­ment. If you stop be­fore you die, you get all those dou­blings of util­ity for the rest of your life.

I’d cer­tainly draw one card. But would I stop draw­ing cards?

Think­ing about this in com­mon­sense terms is mis­lead­ing, be­cause we can’t imag­ine the differ­ence be­tween 8x util­ity and 16x util­ity. But we have a math­e­mat­i­cal the­ory about ra­tio­nal­ity. Just ap­ply that, and you find the re­sults seem un­satis­fac­tory.

• Think­ing about this in com­mon­sense terms is mis­lead­ing, be­cause we can’t imag­ine the differ­ence be­tween 8x util­ity and 16x utility

I can’t even imag­ine dou­bling my util­ity once, if we’re only talk­ing about self­ish prefer­ences. If I un­der­stand vNM util­ity cor­rectly, then a dou­bling of my per­sonal util­ity is a situ­a­tion which I’d be will­ing to ac­cept a 50% chance of death in or­der to achieve (as­sum­ing that my util­ity is scaled so that U(dead) = 0, and with­out set­ting a con­stant level, we can’t talk about dou­bling util­ity). Given my life at the mo­ment (apart­ment with mort­gage, two chron­i­cally ill girlfriends, de­cent job with un­pleas­antly long com­mute, mod­er­ate phys­i­cal and men­tal health), and think­ing about the best pos­si­ble life I could have (vol­cano lair, cat­girls), I wouldn’t be will­ing to take that bet. In­tu­ition has already failed me on this one. If Omega can re­ally de­liver on his promise, then ei­ther he’s offer­ing a lifestyle liter­ally be­yond my wildest dreams, or he’s let­ting me in­clude my prefer­ences for other peo­ple in my util­ity func­tion, in which case I’ll prob­a­bly have cured can­cer by the tenth draw or so, and I’ll run into the same break­down of in­tu­ition af­ter about sev­enty draws, by which time ev­ery­one else in the world should have their own vol­cano lairs and cat­girls.

With the prob­lem as stated, any finite num­ber of draws is the ra­tio­nal choice, be­cause the pro­posed util­ity of N draws out­weighs the risk of death, no mat­ter how high N is. The prob­a­bil­ity of death is always less than 1 for a finite num­ber of draws. I don’t think that con­sid­er­ing the limit as N ap­proaches in­finity is valid, be­cause ev­ery time you have to de­cide whether or not to draw a card, you’ve only drawn a finite num­ber of cards so far. Cer­tainty of death also oc­curs in the same limit as in­finite util­ity, and in­finite util­ity has its own prob­lems, as dis­cussed el­se­where in this thread. It might also leave you open to Pas­cal’s Scam—give me \$5 and I’ll give you in­finite util­ity!

But we have a math­e­mat­i­cal the­ory about ra­tio­nal­ity. Just ap­ply that, and you find the re­sults seem un­satis­fac­tory.

I agree—to keep draw­ing un­til you draw a skull seems wrong. How­ever, to say that some­thing “seems un­satis­fac­tory” is a state­ment of in­tu­ition, not math­e­mat­ics. Our in­tu­ition can’t weigh the value of ex­po­nen­tially in­creas­ing util­ity against the cost of an ex­po­nen­tion­ally diminish­ing chance of sur­vival, so it’s no won­der that the math­e­mat­i­cally de­rived an­swer doesn’t sit well with in­tu­ition.

• This is known as the Deck of Many Things in the D&D com­mu­nity. What you do is make com­mon­ers each draw a card and rob those that sur­vive.

Edit: Haha, dis­re­gard that, I suck cocks.

Edit, edit: Se­ri­ously though, I as­sign minus in­finity to my death. Thus I never know­ingly en­dan­ger my­self. Thus I draw no cards. I also round tiny prob­a­bil­ities down to zero so I can go out­side de­spite the risk of me­te­ors.

• And even if you some­how worked around all these ar­gu­ments, evolu­tion, again, thwarts you. Even if you don’t agree that ra­tio­nal agents are self­ish, your un­selfish agents will be out-com­peted by self­ish agents. The claim that ra­tio­nal agents are not self­ish im­plies that ra­tio­nal agents are un­fit.

This is not how evolu­tion works. Evolu­tion cares about how many of your offspring sur­vive. Selfish­ness need not be con­ducive to this. Also, evolu­tion can’t re­ally thwart you. You’re done evolv­ing; you can check it off your to-do list.

It’s en­tirely plau­si­ble that be­ing un­selfish is adap­tive; from a per­sonal (non-gene, i.e. the per­spec­tive we ac­tu­ally have) per­spec­tive, hav­ing chil­dren is ex­tremely un­selfish.

Selfish­ness and un­selfish­ness are ara­tional. Ra­tion­al­ity is about max­i­miz­ing the out­put of your util­ity func­tion (in this con­text). Selfish­ness is about what that util­ity func­tion ac­tu­ally is.

• Hon­estly, isn’t this nit­pick­ing? It’s true that Lord Aza­toth stopped se­lect­ing for genes in our species ten thou­sand years ago, but when that game stopped work­ing for him he switched to mak­ing our memes com­pete against ea­chother (in any sane world we’d be hav­ing this con­ver­sa­tion in Chi­nese, and my mother’s ‘Scot­tish’ sur­name wouldn’t be Nordic).

You’re ab­solutely right, and he did sim­plify this por­tion, but it doesn’t un­der­mine the weight of his ar­gu­ment any more than my say­ing “I’m not sex­ist, I’m a fully evolved male!” is ren­dered ir­rele­vant by the fact that cur­rent so­cial mores have lit­tle to noth­ing to do with evolu­tion­ary biol­ogy.

It’s one thing to cor­rect Phil’s state­ment, or offer a sug­gested re­word­ing that would im­prove the strength of the point he was try­ing to make, but if feels as if you’re pin point­ing this one poor choice of word­ing, and us­ing it to im­ply that the en­tire premise is flawed.

• as if you’re pin point­ing this one poor choice of word­ing, and us­ing it to im­ply that the en­tire premise is flawed.

Ar­gu­men­tum ad evolu­tionum is both com­mon enough and hor­ribly wrong enough that I would not call it “nit­pick­ing.” The claim that un­selfish agents will be out­com­peted by self­ish agents is com­plex, con­text-de­pen­dent, and re­quires sup­port. The idea that there will some­how be an equil­ibrium in which un­selfish agents get crowded out seems ab­surd, and this is what “evolu­tion” seems in­tended to evoke, be­cause evolu­tion is (in sig­nifi­cant part) about com­pet­i­tively crowd­ing out the sub-op­ti­mal.

He also makes a much big­ger mis­take, and I should have ad­dressed that in greater de­tail. Utility curves are ara­tional, and term “self­ish” gets con­fused way more than it should. It seems clear from con­text that he means it he­do­nis­ti­cally, i.e. my own he­do­nis­tic ex­pe­rience is my only con­cern if I’m self­ish; I don’t care about what other peo­ple want or think. If my ac­tual util­ity curve in­volves other peo­ple’s util­ity, or it in­volves max­i­miz­ing the num­ber of pa­per clips in ex­is­tence, there is ab­solutely no rea­son to be­lieve I could bet­ter ac­com­plish goals if I were “self­ish” by this defi­ni­tion.

Utility curves are strictly ara­tional. A ra­tio­nal pa­per­clip max­i­mizer is an en­tirely pos­si­ble be­ing. Any state­ment of the kind “Ra­tional agents are/​are not self­ish” is a type er­ror; self­ish­ness is en­tirely or­thog­o­nal to ra­tio­nal­ity.

• It seems clear from con­text that he means it he­do­nis­ti­cally, i.e. my own he­do­nis­tic ex­pe­rience is my only con­cern if I’m self­ish; I don’t care about what other peo­ple want or think.

In­stead of try­ing to in­ter­pret the con­text, you should be­lieve that I mean what I say liter­ally. I re­peat:

If you still think that you wouldn’t, it’s prob­a­bly be­cause you’re think­ing a 1% in­crease in your util­ity means some­thing like a 1% in­crease in the plea­sure you ex­pe­rience. It doesn’t. It’s a 1% in­crease in your util­ity. If you fac­tor the rest of your uni­verse into your util­ity func­tion, then it’s already in there.

In fact, I have already ex­plained my us­age of the word “self­ish” to you in this same con­text, re­peat­edly, in a differ­ent post.

Psy­chohis­to­rian wrote:

Utility curves are strictly ara­tional. A ra­tio­nal pa­per­clip max­i­mizer is an en­tirely pos­si­ble be­ing. Any state­ment of the kind “Ra­tional agents are/​are not self­ish” is a type er­ror; self­ish­ness is en­tirely or­thog­o­nal to ra­tio­nal­ity.

I quote my­self again:

If you act in the in­ter­est of oth­ers be­cause it’s in your self-in­ter­est, you’re self­ish. Ra­tional “agents” are “self­ish”, by defi­ni­tion, be­cause they try to max­i­mize their util­ity func­tions. An “un­selfish” agent would be one try­ing to also max­i­mize some­one else’s util­ity func­tion. That agent would ei­ther not be “ra­tio­nal”, be­cause it was not max­i­miz­ing its utiltity func­tion; or it would not be an “agent”, be­cause agent­hood is found at the level of the util­ity func­tion.

• Of course, you have already shown that you choose to pre­tend I am us­ing the word “self­ish” in the col­lo­quial sense which I have re­peat­edly ex­plic­itly said is not the sense I am us­ing it in, in this post and in oth­ers, so this isn’t go­ing to help.

If it isn’t work­ing, why don’t you try some­thing differ­ent?

• (I deleted that para­graph.)

Do you have an idea for some­thing else to try?

• I don’t think it’s re­ally a nec­es­sary dis­tinc­tion; the idea of an un­selfish util­ity max­i­mizer doesn’t quite make sense, be­cause util­ity is defined so neb­u­lously that pretty much ev­ery­one has to seek max­i­miz­ing their util­ity.

• the idea of an un­selfish util­ity max­i­mizer doesn’t quite make sense

You’re right that it doesn’t make sense, which is why some peo­ple as­sume I mean some­thing else when I say “self­ish”. But a lot of com­menters do seem to be­lieve in un­selfish util­ity max­i­miz­ers, which is why I keep us­ing the word.

• Avoid­ing morally charged words. If pos­si­ble shy far far away from ANY pat­tern that peo­ple can au­to­mat­i­cally match against with sys­tem 2 so that sys­tem 1 stays en­gaged.
My ar­ti­cle here http://​​www.forbes.com/​​2009/​​06/​​22/​​sin­gu­lar­ity-robots-com­put­ers-opinions-con­trib­u­tors-ar­tifi­cial-in­tel­li­gence-09_land.html is an at­tempt to do this.

• If pos­si­ble shy far far away from ANY pat­tern that peo­ple can au­to­mat­i­cally match against with sys­tem 2 so that sys­tem 1 stays en­gaged.

Do you mean “sys­tem 1 … sys­tem 2”?

• Ra­tional agents in­cor­po­rate the benefits to oth­ers into their util­ity func­tions.

as a sec­tion header may have thrown me off there.

That aside, I do un­der­stand what you’re say­ing, and I did no­tice the origi­nal con­trast be­tween the 1%/​1%. Though I’d note it doesn’t fol­low that a ra­tio­nal agent would be will­ing to take a 1% chance of de­stroy­ing the uni­verse in ex­change for a 1% in­crease in his util­ity func­tion; the uni­verse be­ing de­stroyed would prob­a­bly out­put a nega­tive, i.e. greater than 100% loss, so that’s not an even bet.

The whole ara­tional point is my mis­take; the whole para­graph:

But maybe they’re just not as ra­tio­nal as you...

reads very much like it is us­ing self­ish in the strict rather than holis­tic util­ity sense, and that was what I was fo­cus­ing on in this re­sponse. I was fo­cus­ing speci­fi­cally on that sec­tion and did not reread the whole post, so I got the wrong idea. My point on evolu­tion re­mains, and the nega­tive-util­ity ar­gu­ment still makes the +1% for 1% chance of de­struc­tion ar­gu­ment fail. But this doesn’t mat­ter much, since one can hardly sup­pose all agents in charge of mak­ing such de­ci­sions will be perfectly ra­tio­nal.

• and the nega­tive-util­ity ar­gu­ment still makes the +1% for 1% chance of de­struc­tion ar­gu­ment fail

That’s why what I wrote in that sec­tion was:

it’s not pos­si­ble that you would not ac­cept a .999% risk, un­less you are not max­i­miz­ing ex­pected value, or you as­sign the null state af­ter uni­verse-de­struc­tion nega­tive util­ity.

You wrote:

But this doesn’t mat­ter much, since one can hardly sup­pose all agents in charge of mak­ing such de­ci­sions will be perfectly ra­tio­nal.

I am sup­pos­ing that. That’s why it’s in the ti­tle of the post. I don’t mean that I am cer­tain that is how things will turn out to be. I mean that this post says that ra­tio­nal be­hav­ior leads to these con­se­quences. If that means that the only way to avoid the de­struc­tion of life is to cul­ti­vate a par­tic­u­lar bias, then that’s the im­pli­ca­tion.

• Evolu­tion cares about how many of your offspring sur­vive. Selfish­ness need not be con­ducive to this.

Selec­tion, act­ing on the in­di­vi­d­ual, se­lects for those in­di­vi­d­u­als who act in ways that cause their own offspring to sur­vive more. That is what I mean by self­ish­ness. Selfish genes. Selfish memes.

Once peo­ple no longer die, se­lec­tion will not have so much to do with death and re­pro­duc­tion, but with the ac­cu­mu­la­tion of re­sources. Think about that, and it will be­come more clear that that will se­lect di­rectly for self­ish­ness in the con­ven­tional sense.

• Although your con­clu­sions are very de­press­ing, it seems I must ac­cept them. The other com­menters’ re­luc­tance to agree puz­zles me.

• I find the anal­y­sis largely con­vinc­ing as well, and fur­ther feel a 3/​1M chance per cen­tury of ex­is­ten­tial dis­aster is ex­tremely con­ser­va­tive. But I also don’t find the idea of a sin­gle­ton de­press­ing. Bostrom sug­gests the idea of a sin­gle­ton be­ing a world demo­cratic gov­ern­ment or a benev­olent su­per­in­tel­li­gent ma­chine, which Eliezer’s CEV seems able to re­al­ize, at least with my ini­tial un­der­stand­ing. It even seems pos­si­ble that sin­gle­tons such as that might dis­solve them­selves if that’s what was de­sired (<-se­ri­ous hand­wav­ing), but I ad­mit that a sin­gle­ton has such po­ten­tial for stay­ing power that it’s prob­a­bly best to as­sume it’s “for­ever”.

With my views on the varied risks we face, the unique po­ten­tial of a sin­gle­ton to solve many of them, and with a per­sonal es­ti­mate of .5 prob­a­bil­ity of sur­viv­ing this cen­tury at best, a sin­gle­ton seems worth look­ing into. Its a huge dan­ger it­self, but I think we ought to in­ves­ti­gate the best ways to make a “safe” sin­gle­ton at the same time as look­ing for ways to avoid risk with­out one, not wait­ing un­til we are sure we ab­solutely need one.

I re­al­ize this was not the fo­cus of the post and so apol­o­gize if it’s too off-topic. I wanted to draw more at­ten­tion to it as a po­ten­tial solu­tion, though I don’t mean to with­draw at­ten­tion from the post’s cen­tral is­sues.

• What are the ex­is­ten­tial risks for a multi-galaxy su­per-civ­i­liza­tion? Or even a multi-stel­lar civ­i­liza­tion ex­pand­ing out­ward at some frac­tion of light speed? I don’t see how life can be ex­ter­mi­nated once it has spread that far. “liber­ate much of the en­ergy in the black hole at the cen­ter of our galaxy in a gi­ant ex­plo­sion” does not make sense, since a black hole is not con­sid­ered a store of en­ergy that can be liber­ated.

If you are spec­u­lat­ing about new physics that haven’t been dis­cov­ered yet, then “sub­jec­tive-time ex­po­nen­tial” and risk per cen­tury seems ir­rele­vant (we can just as­sume that all of physics will be dis­cov­ered sooner or later), and a more per­ti­nent ques­tion might be how much of physics are as yet undis­cov­ered, and what is the like­li­hood that some new physics will al­low a galaxy/​uni­verse kil­ler to be built.

I ar­gue that the amount of physics left to be dis­cov­ered is finite, and there­fore the like­li­hood that a galaxy/​uni­verse kil­ler can be built in the fu­ture does not ap­proach ar­bi­trar­ily close to 1 as time goes to in­finity.

• Speak­ing of new physics, there was the dis­cov­ery that stars are other suns rather than tiny holes in ce­les­tial sphere… and in the fu­ture there’s the pos­si­bil­ity of dis­cov­er­ing prac­ti­cally at­tain­able in­ter­stel­lar travel. Dis­cov­er­ies in physics can have differ­ent effects.

And if we’re to talk of limitless new and amaz­ing physics, there may be su­per­bombs, and there may be in­finite sub­jec­tive time within finite vol­ume of space­time, or some­thing of that sort.

• I don’t see how life can be ex­ter­mi­nated once it has spread that far.

You may be right. It takes a long time to be­come a multi-galaxy su­per-civ­i­liza­tion. IIRC our galaxy is 100,000 light-years across, and the near­est galaxy is about 2 mil­lion light-years away. We might make it in time. It de­pends a lot on how far time-com­pres­sion goes, and on how cor­re­lated apoc­a­lypses are.

“liber­ate much of the en­ergy in the black hole at the cen­ter of our galaxy in a gi­ant ex­plo­sion” does not make sense, since a black hole is not con­sid­ered a store of en­ergy that can be liber­ated.

Wrong. Google ‘black hole ex­plo­sions’.

I ar­gue that the amount of physics left to be dis­cov­ered is finite, and there­fore the like­li­hood that a galaxy/​uni­verse kil­ler can be built in the fu­ture does not ap­proach ar­bi­trar­ily close to 1 as time goes to in­finity.

That’s my hope as well.

• Wrong. Google ‘black hole ex­plo­sions’.

None of the re­sults in­di­cate a pos­si­bil­ity that the “en­ergy in the black hole at the cen­ter of our galaxy” can be liber­ated in a gi­ant ex­plo­sion.

The first re­sult is a 1974 pa­per by Stephen Hawk­ings pre­dict­ing that black holes emit black-body ra­di­a­tion at a tem­per­a­ture in­verse in its mass. For large black holes this tem­per­ate is close to ab­solute zero, mak­ing them more use­ful as en­tropy dumps than en­ergy sources.

On the other hand, if you could si­mul­ta­neously con­vert a lot of or­di­nary mat­ter into nu­mer­ous tiny black holes, they would all in­stantly evap­o­rate and have the effect of a sin­gle great ex­plo­sion, so that’s one risk to be wor­ried about.

• None of the re­sults in­di­cate a pos­si­bil­ity that the “en­ergy in the black hole at the cen­ter of our galaxy” can be liber­ated in a gi­ant ex­plo­sion.

You’re right about that. But they do in­di­cate that the en­ergy in smaller black holes can be liber­ated in gi­ant ex­plo­sions. And they in­di­cate that black holes could be used as an en­ergy sources. So when you said, “a black hole is not con­sid­ered a store of en­ergy that can be liber­ated,” that was wrong; or at least it was wrong if you meant “a black hole is not con­sid­ered a store of en­ergy.” And that was what I said was wrong.

• Why are you con­tin­u­ing to talk about this one par­tic­u­lar hy­po­thet­i­cal risk?

• I asked for a list of pos­si­ble risks, and no­body has given any other an­swer...

• Still, the ques­tion of whether one par­tic­u­lar risk is real has al­most no bear­ing on the to­tal ex­is­ten­tial risk.

• That’s only true if there are lots of differ­ent ex­is­ten­tial risks be­sides this par­tic­u­lar one. The fact that no one has an­swered my ques­tion with a list of such risks seems to ar­gue against that. I also ar­gued ear­lier that the amount of physics left to be dis­cov­ered is finite, so the num­ber of such risks is finite.

More gen­er­ally, I guess it boils down to cog­ni­tive strate­gies. I like to start from spe­cific ex­am­ples, build in­tu­itions, find similar­i­ties, then pro­ceed to gen­er­al­ize. I pro­gram like this too. If I have to write two pro­ce­dures that I know will end up shar­ing a lot of code, I will write one com­plete pro­ce­dure first, then fac­tor out the com­mon code as I write the sec­ond one, in­stead of writ­ing the com­mon func­tion first. I sup­pose this seems like a waste of time to some­one used to work­ing di­rectly on the gen­eral/​ab­stract is­sue.

• Well, you know my spe­cific ex­am­ple of a risk. Even if you know all about physics, that is the rules of the game, you can still lose to an op­po­nent that can figure out a win­ning strat­egy.

Ex­am­ples are good when you can con­fi­dently say some­thing about them, and their very ex­is­tence was in ques­tion. But there are so many ways to sidestep mere phys­i­cal threat that it doesn’t seem a good choice. An ex­plo­sion is just some­thing that hap­pens to the lo­cal re­gion, in a lawful phys­i­cal way. You could cook some dy­namic re­dun­dancy to pre­serve com­pu­ta­tion in case of an ex­plo­sion.

• if you met the girl of your dreams and she loved you

cough

• How about, “the thing of your dreams”?

• What’s that Pre­cious? f you found The Pre­cious and loved it. Much tastier.

• Yeah, I should always have to say “man|woman he|she”. Be­cause my writ­ing just isn’t wordy enough yet.

How about “the thing of your dreams”?

• Thing? That’s the best you can do? “Per­son”.

• I’m shocked at your in­sen­si­tivity to­wards lovers of an­i­mals and inan­i­mate ob­jects.

• You can love them, but they can’t love you in the same way a per­son can. (Ob­vi­ously an­i­mals can feel love un­der any rea­son­able defi­ni­tion, but they can’t act on that love in the same way a per­son can).

But re­ally this com­ment is to note that, as I write, Ali­corn got 3 karma for helpfully point­ing out (yet) an(other) in­stance of harm­ful bias, while PhilGoetz got 4 karma for a rel­a­tively flip an­swer. (Not that his an­swer is stupid, just that I think we could all come up with such clever quib­bles in re­sponse to most of what any­body said ever, and this would clearly not be pro­duc­tive over­all, there­fore I’d ar­gue that the quib­bles are mainly used to sig­nal “I’m not re­ally tak­ing you se­ri­ously”.)

• For what it’s worth, I down­voted Ali­corn’s com­ment when it was at 7 be­cause I didn’t want to see yet an­other gen­der war at the top of a po­ten­tially in­ter­est­ing com­ment page. Hon­estly, at this point I wish she would stop do­ing what she’s do­ing: it’s more painful to my per­cep­tion of LW than any hid­den gen­der bias Phil may have over­looked in the post.

• I was aware that I was writ­ing a gen­der-bi­ased de­scrip­tion when I wrote the origi­nal post. I de­cided to write it from one gen­der’s point of view, and trust the reader to in­ter­pret it in­tel­li­gently. I find gen­der-neu­tral text is usu­ally stilted and dis­tract­ing.

• What­ever gen­der-neu­tral lan­guage may usu­ally be like, in this case I don’t think the cor­rec­tion to ‘per­son’ is very stilted or dis­tract­ing, IMO. (Would be even bet­ter to re­place ‘he or she or it’ with ‘they’, but I re­al­ize some peo­ple dis­like this style.) There were also other pos­si­ble mod­ifi­ca­tions to the text.

As­sum­ing you agree (since you changed your post) - part of the prob­lem is that even in a case where a good solu­tion was rel­a­tively eas­ily available, you didn’t look for it, even though you knew your phras­ing might be offen­sive to some read­ers (or dis­tract­ing or what­ever you choose to call it). This im­plies, to those read­ers who are dis­tracted by your phras­ing and spend a few sec­onds think­ing about the is­sue, that you didn’t bother not to give offense. And that’s what (some of them may be) re­ally offended at, I think. Con­tin­u­ing this, your com­ment im­plies that any reader who takes offense is be­hav­ing “un­in­tel­li­gently”.

While you say,

I de­cided to write it from one gen­der’s point of view, and trust the reader to in­ter­pret it in­tel­li­gently.

What you mean is, you trust read­ers of the “wrong” gen­der to in­ter­pret. Read­ers who are “like you” in this as­pect, which ought to be com­pletely ir­rele­vant to the dis­cus­sion at hand, don’t need to in­ter­pret any­thing at all. And gen­der-bi­ased text dis­tracts the “wrong” read­ers a lot more than most gen­der-neu­tral text dis­tracts you or most other read­ers.

While “in­tel­li­gently” means here “what­ever I meant even if the text I write doesn’t ex­press it well”. I wish this kind of com­mu­ni­ca­tion worked. But it doesn’t. When peo­ple re­peat­edly tell you that some not-quite-literal turn of phrase you’re us­ing is mis­in­ter­preted com­pared to what you mean, I think you should stop us­ing it.

• Dude. I could have used “per­son”, but would be left with a “he or she”. Stilted.

As­sum­ing you agree (since you changed your post)

I didn’t re­al­ize that chang­ing my post so as not to offend some­one im­plies I agree with them. I will change it back.

you didn’t look for it

I didn’t? Funny, I thought I did. But I guess you know bet­ter.

What you mean is, you trust read­ers of the “wrong” gen­der to in­ter­pret.

Maybe I am a bet­ter source on what I mean than your mal­i­cious imag­i­na­tion.

If you look over pre­vi­ous things I’ve writ­ten, you’ll see that some­times I say “he”, and some­times I say “she”. I have been con­scious of ev­ery sin­gle time I wrote “he” or “she” prob­a­bly since be­fore you were born. But I write one post, over 3000 words long, in which I have ex­actly one case of gen­dered speech, and the coin flip comes up so that I write “he” in­stead of she, and you’re all over me for be­ing an in­sen­si­tive sex­ist pig.

If all that my 20+ years of care­fully writ­ing gen­der-bal­anced text has done is to en­courage peo­ple like you feel en­ti­tled to lec­ture me from your moral high horse on any oc­ca­sion when I don’t mea­sure up com­pletely to your stan­dards, then I’m done be­ing gen­der-neu­tral. Ap­par­ently it just makes things worse.

I’m sorry that I origi­nally replied flip­pantly. This whole ex­change wouldn’t have hap­pened if I’d just quietly changed the text.

Let’s hear from other read­ers. 2 read­ers are offended by non-gen­der-neu­tral lan­guage. If any of you think that au­thors should be al­lowed to use gen­dered lan­guage, let him or her speak, or for­ever hold his or her peace.

• I find gen­der-neu­tral lan­guage nit-pick­ing off-putting. This thread per­suades me that LessWrong is a waste of my time and I should stay away.

• That is valid logic if you’re look­ing for plea­sure from LessWrong. It is not valid if you are in­ter­ested in be­ing less wrong.

• If all that my 20+ years of care­fully writ­ing gen­der-bal­anced text has done is to en­courage peo­ple like you feel en­ti­tled to lec­ture me from your moral high horse on any oc­ca­sion when I don’t mea­sure up com­pletely to your stan­dards, then I’m done be­ing gen­der-neu­tral.

Does that mean that you are only gen­der neu­tral be­cause you like ap­proval from the gen­der neu­tral­ity cops, and if they stop ap­prov­ing of you, you have no rea­son to con­tinue to pur­sue/​im­prove your decades-long policy of try­ing to do the right thing?

• It means that be­ing gen­der neu­tral has en­couraged you to feel like you have the right to tell other peo­ple how to write, and to look down on any­one who uses the word “he”.

• I would not have ob­jected to your use of a male-spe­cific phrase if you had not writ­ten in the sec­ond per­son. I’d be will­ing to take your word for it that your choice was ran­dom and I wouldn’t care—if it were about some hy­po­thet­i­cal per­son who was male. It was about a “you” ad­dressed in the post, and I, as a reader, was there­fore ex­cluded.

• I can un­der­stand that a lit­tle bet­ter.

I’d like to delete this con­ver­sa­tion from Less Wrong. I’d rather have done this by email. No­body else seems to be read­ing it any­way. You can reach me at @ya­hoo.

In my ex­pe­rience, dis­agree­ments get more heated when done in pub­lic posts than in pri­vate emails.

• I don’t like to delete things that have gone on for this long. In the fu­ture, you could PM peo­ple who make com­ments you’d like to re­ply to but think may de­velop into “heated dis­agree­ments”. But if no one else is read­ing it, then some of the votes on the com­ments are un­ac­counted for.

• lrn 2 they

• Maybe I am a bet­ter source on what I mean than your mal­i­cious imag­i­na­tion.

My com­ment was pre­cisely about the fact that peo­ple can mi­s­un­der­stand what you ac­tu­ally mean be­cause your words are open to an­other in­ter­pre­ta­tion.

I hope my imag­i­na­tion isn’t par­tic­u­larly mal­i­cious (though as befits this site I won’t as­sume such a thing). I in­tended to com­ment not about your ac­tual mean­ing but about the way oth­ers, like Ali­corn, ap­pear to per­ceive it.

As for the part about “you trust read­ers of the “wrong” gen­der to in­ter­pret”, I’m sure you didn’t mean to think about only some read­ers; in fact you didn’t think about only some of the read­ers. I was talk­ing about the sep­a­rate fact that het­ero-male read­ers wouldn’t need to in­ter­pret your words in any but the literal way.

Please, let oth­ers com­ment. Even if there’s no con­sen­sus it’s bet­ter to reach a sta­tus quo to avoid hash­ing this out again ev­ery few days. (Go­ing by what I’ve read in LW be­fore I started com­ment­ing.)

• My com­ment was pre­cisely about the fact that peo­ple can mi­s­un­der­stand what you ac­tu­ally mean be­cause your words are open to an­other in­ter­pre­ta­tion.

No; your com­ment claimed to know what I think and what I mean.

in fact you didn’t think about only some of the read­ers.

You’re do­ing it again! And you’re wrong, again.

• First I said that you said, or your words im­plied, that you thought only about some of the read­ers. And you said I was wrong:

What you mean is, you trust read­ers of the “wrong” gen­der to in­ter­pret. Maybe I am a bet­ter source on what I mean than your mal­i­cious imag­i­na­tion.

Then I said, OK, I be­lieve you, you did think about all of the read­ers. And you say I’m wrong again:

in fact you didn’t think about only some of the read­ers. You’re do­ing it again! And you’re wrong, again.

Now I’m just con­fused. Pos­si­bly it’s my mis­take/​mi­s­un­der­stand­ing.

• Then I said, OK, I be­lieve you, you did think about all of the read­ers. And you say I’m wrong again:

Sorry. I parsed your sen­tence to mean some­thing else.

This is my karmic pay­back for things I said to Eliezer.

• Hooray thread necro­mancy!

One op­tion is to always use fe­male-gen­dered lan­guage. Then women won’t feel slighted by male-priv­ileg­ing lan­guage, and men will al­most en­tirely not care or feel slighted.

• I voted this com­ment down be­cause it wasn’t good enough to jus­tify dig­ging up an old flame war.

• What is it some folks have against ad­di­tions to “old” dis­cus­sions?

If you don’t want to re-join, don’t. The only nega­tive effect that is ap­par­ent to me seems to be a pid­dlingly small amount of screen real es­tate un­der “Re­cent Com­ments” for a short pe­riod of time.

Also, I didn’t ac­tu­ally con­tribute any flam­ing (no tear­ing down of any­one else’s sug­ges­tions or be­hav­iors). Only an at­tempt at a con­struc­tive solu­tion to a re­cur­rent prob­lem that no one else seems to have sug­gested yet.

• I agree that adding to old dis­cus­sions isn’t in it­self bad, and that you didn’t con­tribute to any flam­ing. What both­ers me is there’s a chance that oth­ers will read the thread and feel the need to re­spond, and then things might bal­loon.

• I guess you fear other folks wast­ing their time via flam­ing each other more than I do.

I would hope we get to worry less about that sort of thing on this site.

Upvoted. Thanks for the ex­pla­na­tion.

• If you don’t want to re-join, don’t. The only nega­tive effect that is ap­par­ent to me seems to be a pid­dlingly small amount of screen real es­tate un­der “Re­cent Com­ments” for a short pe­riod of time.

So if I post a “Make Money Fast” ad ev­ery day, that is OK be­cause the only nega­tive effeect is a pid­dlingly small amount of screen real es­tate for a short pe­riod of time ev­ery day?

• Depends. Can I do it from home, part time?

• Yes. And you can be your own boss and set your own hours, too.

• You don’t see a dis­tinc­tion be­tween at­tempts at con­struc­tive post­ing and spam? Or you just felt like be­ing snarky?

• Yes, I see a differ­ence. And agree with you that GGGGGP was an at­tempt at con­struc­tive post­ing.

When I saw your ar­gu­ment of the sec­ond para­graph of GGGP, I be­came wor­ried that it would en­courage peo­ple to lower their post­ing stan­dards and con­se­quently over time drive busy thought­ful read­ers away. And the first re­fu­ta­tion of your ar­gu­ment that oc­cured to me was to point out that the ar­gu­ment could be used to jus­tify spam as well as to jus­tify your com­ment.

What I ne­glected to no­tice is that the tone of my com­ment and the fact that I im­plic­itly com­pare you to a spam­mer had a high prob­a­bil­ity of mak­ing you con­clude that I do not wel­come you here. That’s not true, and please for­give my clum­si­ness.

• Bad idea, don­wn­voted.

Men shouldn’t be con­sid­ered obli­gated not to care any more than women should.

• It’s not a prefer­ence or a ‘should’ on my part; it’s a fact about the vast ma­jor­ity of men. They won’t feel slighted. (I didn’t down­vote you.)

• I’m not down­vot­ing you for adding the com­ment to an old thread, I’m down­vot­ing you for the “hooray thread necro­mancy” sen­tence which com­pletely dis­tracted from any­thing ac­tu­ally mean­ingful you had to say in your com­ment and turned the whole sub­thread into a dis­cus­sion of whether it’s okay to add com­ments to old thread.

Yes, it’s okay to add com­ments to old threads. If your com­ment has util­ity enough to be part of the thread, then it’ll have util­ity enough for peo­ple reread­ing the old thread af­ter a few years too. Op­po­si­tion to such is the product of the me­chan­ics of other fo­rums where re­vived old threads get boosted up thus drown­ing the newer ones—this isn’t the case here, so it doesn’t ap­ply.

But it’s NOT OKAY to waste space pat­ting your­self on the back about how you added a com­ment to an old thread. That causes dis­trac­tion and di­su­til­ity, as this sub­thread clearly proves.

• The first time I posted on an old thread, the re­ply (from CronoDAS) was “Hooray thread necro­mancy” in sar­casm tags. So I was just pre-emp­tively rec­og­niz­ing that some peo­ple seem to think post­ing on old threads is a ac­tion not to be taken. Not pat­ting my­self on the back.

• Bravo! Thank you :)

• Your au­di­ence con­sists mostly of peo­ple other than you. You may write solely for your own prefer­ences with­out an­noy­ing any­one when the venue is your di­ary.

• The figure of 31,000,000 for the prob­a­bil­ity of the trinity nuke de­stroy­ing the world is al­most cer­tainly too low. Con­sider that, sub­jec­tively, the sci­en­tists should have as­signed at least a 1 in 1000 prob­a­bil­ity that they’d made a mis­take in their calcu­la­tion of safety. Prob­a­bly more like 1 in 100, con­sid­er­ing that the tech­nol­ogy was en­tirely new. In fact, the first se­ri­ous mis­take in a phys­i­cal calcu­la­tion that re­sulted in an ac­tual dis­aster in­volv­ing a nuke was Cas­tle Bravo, which oc­curred prob­a­bly only 50-150 deto­na­tions af­ter trinity. Since then, we have have Ch­er­nobyl, which is ar­guably some­what differ­ent, but still 10,000 dead, and ap­par­ently set off by sci­en­tists who thought what they were do­ing was safe.

Another way to look at it is to ask your­self, if 100 similar in­ci­dents oc­curred (that is, in­stances of sci­en­tists de­vel­op­ing a new and very de­struc­tive tech­nol­ogy in wartime, and wor­ry­ing that it just might blow the whole world up but that it’s prob­a­bly OK), how many in­stances of “fail” would you ex­pect?

Look­ing at it this way, even 1 in 100 is too op­ti­mistic. The dom­i­nant failure mode is that the sci­en­tists fail to grasp a cru­cial con­sid­er­a­tion, like try­ing to build a mil­i­tary su­per-in­tel­li­gence with­out un­der­stand­ing the need for friendly AI, which I sus­pect oc­curs with prob­a­bil­ity 1 in 8.

In fact, now that I think about it, the dom­i­nant failure mode is that the over­all pro­ject lead­er­ship fails to listen to those sci­en­tists who say they might have found a cru­cial con­sid­er­a­tion, if there is one.

• If you don’t be­lieve black holes can ever be used as weapons, here’s an ar­ti­cle about a star 8000 light years away that some as­tronomers worry may harm life on Earth (to what ex­tent it doesn’t say).

• But de­cid­ing to use util­ity func­tions that will [...] seems to be ra­tio­nal.

You can’t de­cide your util­ity func­tion. It’s a given. You can only make de­ci­sions based on prefer­ence that, prob­a­bly, can be rep­re­sented in part as a util­ity func­tion. De­cid­ing to use a par­tic­u­lar util­ity func­tion (that doesn’t hap­pen to be ex­actly the one rep­re­sent­ing your own prefer­ence) con­sti­tutes throw­ing away your hu­man­ity and re­plac­ing it with what­ever the new util­ity func­tion says.

• Okay. Sub­sti­tute “Max­i­miz­ing your ex­pected util­ity” in­stead.

• In­ci­den­tally, I once met Brian Mo­ri­arty at a party John Romero threw where I em­barassed my­self in front of or offended all of my child­hood heroes, one af­ter an­other, end­ing with Steve Woz­niak.

I was talk­ing to him about trends in text ad­ven­tures, and said, “One great thing is that IF au­thors have got­ten away from the idea that ev­ery game has to be about sav­ing the world.”

He said some­thing like, “Well, I hap­pen to think that sav­ing the world is not such a bad thing,” and went off in a bit of a huff. And then I re­mem­bered that he was the au­thor of Trinity, which was about the Trinity test and sav­ing the world from nu­clear holo­caust. (And was a re­ally good game, BTW.)

• The loss to them if they ig­nited the at­mo­sphere would be an­other 30 or so years of life. The loss to them if they lost the war and/​or were kil­led by their en­e­mies would also be an­other 30 or so years of life.

This as­sumes they don’t care about their chil­dren and grand­chil­dren af­ter their death.

• ...and the chil­dren and grand­chil­dren of their en­e­mies, and the mil­lions of cur­rently-liv­ing peo­ple not in­volved in the war, and etc.

• This whole point may re­flect col­lec­tive con­fu­sion sur­round­ing the term “util­ity.”

I do not presently have a co­effi­cient in my util­ity func­tion at­tached to John Doe, who is a me­chanic in Des Moines (I’m as­sum­ing). I know noth­ing about him, and what­ever hap­pens to him does not af­fect my ex­pe­rience of hap­piness in the slight­est. I wish him well, but it would make lit­tle sense to say he is re­flected in my util­ity func­tion. I would agree that, ce­teris paribus, the bet­ter off he is, the bet­ter, but (par­tic­u­larly since I won’t know it), this doesn’t re­ally weigh in my ex­pe­rience of life.

On the other hand, if you asked me if I would rather him die if meant I got a thou­sand dol­lars, I’d have to turn down the offer. I care about his util­ity in the ab­stract, even if it doesn’t ac­tu­ally af­fect my hap­piness.

There’s a rele­vant dis­tinc­tion be­tween ab­stract col­lec­tive util­ity and per­son­ally ex­pe­rienced util­ity. The hu­man mind is not pow­er­ful enough to com­pre­hend true, com­plete, ab­stract util­ity, and if it was you’d prob­a­bly be­come ter­mi­nally de­pressed. One can be­lieve in the im­por­tance of max­i­miz­ing ab­stract util­ity while not ac­tu­ally ex­pe­rienc­ing it. When Omega offers to dou­ble our util­ity, we think that means some­thing we ex­pe­rience, and we don’t ex­pe­rience the ab­stract util­ity of the en­tire planet. I be­lieve that this dis­tinc­tion gets con­fused, lead­ing to this post feel­ing con­trary to in­tu­ition.

On which note, we re­ally don’t know what to­tal util­ity looks like—it’s too com­plex. So the world gets de­stroyed or to­tal util­ity gets dou­bled 50-50 bet is not evaluable by the in­di­vi­d­ual be­cause we don’t know how to eval­u­ate the di­su­til­ity of the world be­ing de­stroyed, other than we’d rather not risk it.

This is all made that much more painful by the fact that rea­son alone can­not say which is prefer­able, the scratch­ing of my finger or the de­struc­tion of the world. I think Hume may have beaten us to this.

• When Omega offers to dou­ble our util­ity, we think that means some­thing we ex­pe­rience, and we don’t ex­pe­rience the ab­stract util­ity of the en­tire planet. I be­lieve that this dis­tinc­tion gets con­fused, lead­ing to this post feel­ing con­trary to in­tu­ition.

Ac­tu­ally, if you think of the para­dox be­low in these terms, as be­ing one where you’re offered vague, un­mea­surable re­wards, it ceases to be a para­dox. It’s only a para­dox be­cause we’ve ab­stracted those con­fus­ing is­sues away.

It is a puz­zle that is meant to get at the ques­tion of whether our math­e­mat­i­cal mod­els of ra­tio­nal­ity are cor­rect. If you’re not talk­ing about math­e­mat­i­cal mod­els, you’re hav­ing a differ­ent con­ver­sa­tion.

• You can put mon­e­tary value on hu­man­ity, just as you can on a per­son’s life.

• If hu­man­ity goes away, who will col­lect the sav­ings?

• I see what he’s say­ing, but there’s some­thing wrong with it. If you put a mon­e­tary value on a life, it means that you could in­crease util­ity by trad­ing that life for more than that much money, be­cause you could do things with that money that would in­crease other peo­ple’s util­ity enough to make up for the life. But once you’ve traded the last life away, you can’t use the money.

• Robert A. Hein­lein: An ex­tinct species has no moral be­havi­our. ~An ad­dress to a West­point grad­u­at­ing class

• On “in­cor­po­rat­ing the benefits to oth­ers into their util­ity func­tions”, you hint at a sharp di­chotomy be­tween Scrooges and Saints—peo­ple who act en­tirely in their own self-in­ter­est, and peo­ple who act in ev­ery­one’s in­ter­est (be­cause that is the na­ture of their own self-in­ter­est). But most hu­mans are not at these poles—most of us act in the in­ter­est of at least sev­eral peo­ple. Mir­ror­ing (un­der­stand­ing) is par­tially a learned trait, but ac­tu­ally car­ing about other peo­ple who you mir­ror is emo­tion­ally “ba­sic”. By this I mean it’s en­tirely in our self-in­ter­est to act in the in­ter­est of some oth­ers. That was to par­tially ad­dress your “un­selfish agents will be out-com­peted by self­ish agents” claim. False di­chotomy.

You haven’t given a con­vinc­ing ar­gu­ment that peo­ple will stop hav­ing Life-And-Death con­flicts.

On the LHC, it sounds like you’re ar­gu­ing for a more pre­cau­tion­ary ap­proach to sci­ence. What would be ac­cept­able con­di­tions in your book for turn­ing on the LHC? (Also, that par­tic­u­lar ex­per­i­ment re­ceived so much pub­lic­ity, the num­bers were un­doubt­edly checked hun­dreds of times).

(I’ve got to get to work, and I’m sure other posters will ad­dress the rele­vant points and prob­lems be­fore I get a chance.)

• By this I mean it’s en­tirely in our self-in­ter­est to act in the in­ter­est of some oth­ers. That was to par­tially ad­dress your “un­selfish agents will be out-com­peted by self­ish agents” claim. False di­chotomy.

It’s not a false di­chotomy. If you act in the in­ter­est of oth­ers be­cause it’s in your self-in­ter­est, you’re self­ish. Ra­tional “agents” are “self­ish”, by defi­ni­tion, be­cause they try to max­i­mize their util­ity func­tions. An “un­selfish” agent would be one try­ing to also max­i­mize some­one else’s util­ity func­tion. That agent would ei­ther not be “ra­tio­nal”, be­cause it was not max­i­miz­ing its utiltity func­tion; or it would not be an “agent”, be­cause agent­hood is found at the level of the util­ity func­tion. I tried to make this point in an­other thread, and lost like 20 karma points do­ing so. But it’s still right. I re­quest any­one down-vot­ing this com­ment to provide some al­ter­na­tive in­ter­pre­ta­tion un­der which a ra­tio­nal agent is not self­ish.

EDIT: A great ex­am­ple of what I mean by “agent­hood is found at the level of the util­ity func­tion” is that you shouldn’t con­sider an ant an agent.

The whole point of the es­say is to try to find some way for it to be in ev­ery­one’s self-in­ter­est to act in ways that will pre­vent us from tak­ing small risks of ex­ter­mi­nat­ing life. And I failed to find any such way. So you see, the en­tire es­say is pred­i­cated on the point that you’re mak­ing.

You haven’t given a con­vinc­ing ar­gu­ment that peo­ple will stop hav­ing Life-And-Death con­flicts.

Do you mean, I haven’t given a con­vinc­ing ar­gu­ment that peo­ple will not stop hav­ing Life-And-Death con­flicts?

On the LHC, it sounds like you’re ar­gu­ing for a more pre­cau­tion­ary ap­proach to sci­ence.

Not ac­tu­ally. The next hun­dred thou­sand years are a spe­cial case.

• I think we’re agree­ing on the first point—any ra­tio­nal agent is self­ish. But then there’s no such thing as an un­selfish agent, right? Also, no need to use the term self­ish, if it’s im­plicit in ra­tio­nal agent. If un­selfish agents don’t ex­ist, it’s easy to out-com­pete them!

“try­ing to also max­i­mize some­one else’s util­ity func­tion… would not be an ‘agent’, be­cause agent­hood is found at the level of the util­ity func­tion.” What do you mean by this? I read this as say­ing that a util­ity func­tion which is di­rectly de­pen­dent on an­other’s util­ity is not a util­ity func­tion. In other words, any­one who cares about an­other, and takes di­rect plea­sure from an­other’s wellbe­ing, is not an agent. If that’s what you mean, then most hu­mans aren’t agents. Other­wise, I’m not un­der­stand­ing.

On Life-And-Death con­flicts, yes, that’s what I meant. You haven’t given any such ar­gu­ment!

On the LHC, why are the next hun­dred thou­sand years a spe­cial case? And again, un­der what con­di­tions should the LHC run?

• Also, no need to use the term self­ish, if it’s im­plicit in ra­tio­nal agent.

Right—now I re­mem­ber, we’ve gone over this be­fore. I think it is im­plicit in ra­tio­nal agent; but a lot of peo­ple for­get this, as ev­i­denced by the many re­sponses that say some­thing like, “But it’s of­ten in an agent’s self-in­ter­est to act in the in­ter­est of oth­ers!”

If you think about why they’re say­ing this in protest to my say­ing that a ra­tio­nal agent is self­ish, it can’t be be­cause they are le­gi­t­i­mately try­ing to point out that a self­ish agent will some­times act in ways that benefit oth­ers. That would be an un­in­ter­est­ing and un­in­for­ma­tive point. No, I think the only thing they can mean is that they be­lieve that de­ci­sion the­ory is some­thing like the In­visi­ble Hand, and will mag­i­cally re­sult in an equil­ibrium where ev­ery­body is nice to each other, and so the agents re­ally aren’t self­ish at all.

So I use the word “self­ish” to em­pha­size that, yes, these agents re­ally pur­sue their own util­ity.

• (Well, “we” haven’t—I’m pretty new on these fo­rums, and missed that dis­agree­ment!)

You still haven’t ad­dressed any of my com­plaints with your ar­gu­ment. I never men­tioned any­thing about time-dis­count­ing—it looked like you saw your sec­ond-to-last propo­si­tion to be the only one with merit, so I was to­tally ad­dress­ing two that you dis­missed.

In my first point, now that we are clear on defi­ni­tions, I meant that you 1) im­plied a di­chotomy be­tween agents whose util­ity func­tions are en­tirely in­de­pen­dent of other peo­ple’s, and those whose util­ity func­tions are very heav­ily de­pen­dent (Scrooges and Saints). You then made the state­ment “un­selfish agents will be out-com­peted by self­ish agents.” Since we agree that there’s no such thing as an un­selfish agent, you prob­a­bly meant “peo­ple who care a lot about ev­ery­one will be out com­peted by peo­ple who care about no­body but them­selves” (self­ish ra­tio­nal agents with highly de­pen­dent vs. highly in­de­pen­dent util­ity func­tions). This is a false di­chotomy be­cause most peo­ple don’t fall into ei­ther ex­treme, but have a util­ity func­tion that de­pends on some oth­ers, but not ev­ery­one and not to an equal de­gree.

And my two ques­tions still stand, on con­flict and the LHC.

(In­ter­est­ing post, by the way!)

• you prob­a­bly meant “peo­ple who care a lot about ev­ery­one will be out com­peted by peo­ple who care about no­body but them­selves.”

No, I didn’t mean that. This is, I think, the 5th time I’ve de­nied say­ing this on Less Wrong. I’ve got to find a way of say­ing this more clearly. I was ar­gu­ing against peo­ple who think that ra­tio­nal agents are not “self­ish” in the sense that I’ve de­scribed el­se­where in these com­ments. If it helps, I’m us­ing the word “self­ish” in a way so that an agent could con­sciously de­sire strongly to help other peo­ple, but still be “self­ish”.

On life-and-death con­flicts, I did give such an ar­gu­ment, but very briefly:

Evolu­tion­ary ar­gu­ments are a more pow­er­ful rea­son to be­lieve that peo­ple will con­tinue to have con­flicts. Those that avoid con­flict will be out-com­peted by those that do not.

I re­al­ize this isn’t enough for some­one who isn’t already fa­mil­iar with the full ar­gu­ment, but it’s af­ter mid­night and I’m go­ing to bed.

On the LHC, why are the next hun­dred thou­sand years a spe­cial case? And again, un­der what con­di­tions should the LHC run?

The next 100,000 years are a spe­cial case be­cause we may learn most of what we will learn over the next billion years in the next 100,000 years. Dur­ing this pe­riod, the risks of some­thing like run­ning the LHC are prob­a­bly out­weighed by how much the knowl­edge ac­quired as a re­sult will help us es­ti­mate fu­ture risks, and figure out a solu­tion to the prob­lem.

• My con­fu­sion isn’t com­ing from the term self­ish, but from the term un­selfish agent. You clearly sug­gested that such a thing ex­ists in the quoted state­ment, and I have no idea what this crea­ture is.

On life-and-death con­flicts, sorry if I’m in­quiring on some­thing widely known by ev­ery­one else, but I wouldn’t mind a link or elab­o­ra­tion if you find the time. I agree that peo­ple will have con­flicts both as a re­sult of hu­man na­ture and of finite re­sources, but I don’t see why con­flicts must always be deadly.

Dur­ing this pe­riod, the risks of some­thing like run­ning the LHC are prob­a­bly out­weighed by how much the knowl­edge ac­quired as a re­sult will help us es­ti­mate fu­ture risks, and figure out a solu­tion to the prob­lem.

You just said the op­po­site of what you said in your origi­nal post here, that the LHC was turned on for no prac­ti­cal ad­van­tage.

• My con­fu­sion isn’t com­ing from the term self­ish, but from the term un­selfish agent. You clearly sug­gested that such a thing ex­ists in the quoted state­ment, and I have no idea what this crea­ture is.

I wrote, “Even if you don’t agree that ra­tio­nal agents are self­ish, your un­selfish agents will be out-com­peted by self­ish agents.” The “un­selfish agent” is a hy­po­thet­i­cal that I don’t be­lieve in, but that the imag­i­nary per­son I’m ar­gu­ing with be­lieves in; and I’m say­ing, “Even if there were such an agent, it wouldn’t be com­pet­i­tive.”

My ar­gu­ment was not very clear. I wouldn’t worry too much over that point.

You just said the op­po­site of what you said in your origi­nal post here, that the LHC was turned on for no prac­ti­cal ad­van­tage.

No; I said, “no prac­ti­cal ad­van­tage that I’ve heard of yet.” First, the word “prac­ti­cal” means “put into prac­tice”, so that learn­ing more the­ory doesn’t count as prac­ti­cal. Se­cond, “that I’ve heard of yet” was a qual­ifier be­cause I sup­pose that some prac­ti­cal ad­van­tage might re­sult from the LHC, but we might not know yet what that will be.

• If “self­ish” (as you use it) is a word that ap­plies to ev­ery agent with­out sig­nifi­cant ex­cep­tion, why would you ever need to use the word? Why not just say “agent”? It seems re­dun­dant, like say­ing “warm-blooded mam­mal” or some­thing.

• Yes, it’s re­dun­dant. I ex­plained why I used it nonethe­less in the great-great-great-grand­par­ent of the com­ment you just made. Sum­mary: You might say “warm-blooded mam­mal” if you were talk­ing with peo­ple who be­lieved in cold-blooded mam­mals.

• Some­one who be­lieves in cold-blooded mam­mals is ei­ther mi­sus­ing the term “mam­mal” or the term “cold-blooded” or both, and I don’t think I’d re­fer to “cold-blooded mam­mals” with­out ad­dress­ing the ques­tion of where that mi­s­un­der­stand­ing is. If peo­ple don’t un­der­stand you when you say “self­ish” (be­cause I think you are us­ing an un­pop­u­lar defi­ni­tion, if noth­ing else) why don’t you leave it out or try an­other word? If I was talk­ing to some­one who in­sisted that mam­mals were cold-blooded be­cause they thought “warm” was syn­ony­mous with “wa­ter boils at this tem­per­a­ture” or some­thing, I’d prob­a­bly first try to cor­rect them—which you seem to have at­tempted for “self­ish” with mixed re­sults—and then give up and switch to “en­dother­mic”.

• Sounds like good ad­vice.

• I read this as say­ing that a util­ity func­tion which is di­rectly de­pen­dent on an­other’s util­ity is not a util­ity func­tion.

No; I meant that each agent has a util­ity func­tion, and tries to max­i­mize that util­ity func­tion.

If we can find an evolu­tion­ar­ily-sta­ble cog­ni­tive makeup for an agent that al­lows it to have a util­ity func­tion that weighs the con­se­quences in the dis­tant fu­ture equally with the con­se­quences to the pre­sent, then we may be saved. In other words, we need to elimi­nate time-dis­count­ing.

One thing I didn’t ex­plain clearly, is that it may be that un­cer­tainty alone pro­vides a large enough time-dis­count­ing to make uni­verse-death in­evitable. Be­cause you’re more and more un­cer­tain what the im­pact of a de­ci­sion will be the farther you look into the fu­ture, you weigh that im­pact less and less the farther into the fu­ture you go.

But maybe this is not in­evitably the right thing to do, if you can find a way to pre­dict fu­ture im­pacts that is un­cer­tain, but also un­bi­ased!

EDIT: No. Un­bi­ased doesn’t cut it.

On life-and-death con­flicts, I did give such an ar­gu­ment, but very briefly:

Evolu­tion­ary ar­gu­ments are a more pow­er­ful rea­son to be­lieve that peo­ple will con­tinue to have con­flicts. Those that avoid con­flict will be out-com­peted by those that do not.

I re­al­ize this isn’t enough for some­one who isn’t already fa­mil­iar with the full ar­gu­ment, but it’s af­ter mid­night and I’m go­ing to bed.

On the LHC, why are the next hun­dred thou­sand years a spe­cial case? And again, un­der what con­di­tions should the LHC run?

The next 100,000 years are a spe­cial case be­cause we may learn most of what we will learn over the next billion years in the next 100,000 years. Dur­ing this pe­riod, the risks of some­thing like run­ning the LHC are prob­a­bly out­weighed by how much the knowl­edge ac­quired as a re­sult will help us es­ti­mate fu­ture risks, and figure out a solu­tion to the prob­lem.

• I was di­rected here from FIMFic­tion.

Be­cause of https://​​en.wikipe­dia.org/​​wiki/​​Sur­vivor­ship_bias we re­ally can’t know what the odds are of do­ing some­thing that ends up wiping out all life on the planet; noth­ing we have tried thus far has even come close, or even re­ally had the ca­pa­bil­ity of do­ing so. Even global ther­monu­clear war, ter­rible as it would be, wouldn’t end all life on Earth, and in­deed prob­a­bly wouldn’t even man­age to end hu­man civ­i­liza­tion (though it would be de­cid­edly un­pleas­ant and hun­dreds of mil­lions of peo­ple would die).

Some peo­ple thought that the nu­clear bomb would ig­nite the at­mo­sphere… but a lot of peo­ple didn’t, ei­ther, and that three in a mil­lion chance… I don’t even know how they got at it, but it sounds like a typ­i­cal wild guess to me. How would you even ar­rive at that figure? In­deed, there is good rea­son to be­lieve that the at­mo­sphere may well have ex­pe­rienced such events be­fore, in the form of im­pact events; this is why we knew, for in­stance, that the LHC was safe—we had ex­pe­rienced con­sid­er­ably more en­er­getic events pre­vi­ously. Some peo­ple claimed it might de­stroy the uni­verse, but the odds were ac­tu­ally 0 - it sim­ply lacked the abil­ity to do so, be­cause if it was go­ing to cause a vac­uum col­lapse the uni­verse would have already been de­stroyed by such an event el­se­where. Mean­while, the physics of small black holes means that they’re not a threat—they would de­cay al­most in­stantly, and would lack the grav­ity nec­es­sary to cause any real prob­lems. And thus far, if we ac­tu­ally look at what we’ve got, the re­al­ity is that ev­ery­thing we have tried has had p=0 of de­stroy­ing civ­i­liza­tion in re­al­ity (that is the uni­verse we -ac­tu­ally- live in), mean­ing that that p = 3 x 10^-6 was ac­tu­ally hope­lessly pes­simistic. Just be­cause some­one can as­sign ar­bi­trary odds to some­thing doesn’t mean that they’re right. In fact, it usu­ally means that they’re bul­lshit­ting.

Re­mem­ber NASA mak­ing up its odds of an in­di­vi­d­ual bolt failing at be­ing one in a 10^8? That’s the sort of made up num­ber we’re look­ing at here.

And that’s the sort of made up num­ber I always see in these situ­a­tions; peo­ple sim­ply come up with stuff, then pre­tend to jus­tify it with math when in re­al­ity it is just a guess. Statis­tics used as a lamp­post; for sup­port, not illu­mi­na­tion.

And this is the biggest prob­lem with all ex­is­ten­tial threats—the great­est ex­is­ten­tial threat to hu­man­ity is, in all prob­a­bil­ity, be­ing smacked by a large me­te­orite, which is some­thing we KNOW, for cer­tain, hap­pens ev­ery once in a while. And if we de­tected that early enough, we could ac­tu­ally pre­vent such an event from hap­pen­ing.

Every­thing else is pretty much en­tirely made up guess­work, based on faulty as­sump­tions, or very pos­si­bly both.

Of the “hu­mans kill us all” sce­nar­ios, the most likely is some hor­rible highly trans­mis­si­ble ge­net­i­cally en­g­ineered dis­ease which was de­liber­ately spread by mad­men in­tent on global de­struc­tion. Here, there are tons of bar­ri­ers; the first, and per­haps largest bar­rier is the fact that crazy peo­ple have trou­ble do­ing this sort of thing; it re­quires a level of or­ga­ni­za­tion which tends to be be­yond them. Se­condly, it re­quires knowl­edge we lack, and which in­deed, once we ob­tain it, may or may not make con­tain­ing the out­break of such a dis­ease rel­a­tively triv­ial—you speak of offense be­ing eas­ier than defense, but in the end, a lot of tech­nolog­i­cal sys­tems are eas­ier to break than they are to make, and un­der­stand­ing how to make some­thing like this may well re­quire us to un­der­stand how to break it in the pro­cess (and in­deed, may well be de­rived from us figur­ing out how to break it). Thirdly, we ac­tu­ally already have mea­sures which re­quire no tech­nol­ogy at all—quaran­tines—which could stop such a thing from wiping out too many peo­ple. Even if you did it in a bunch of places si­mul­ta­neously, you’d still prob­a­bly fail to wipe out hu­man­ity with it just be­cause there are too many peo­ple, too spread out, to ac­tu­ally suc­ceed. And fourth, you’d prob­a­bly need to test it, and that would put you at enor­mous risk of dis­cov­ery. I have my doubts about this sce­nario, but it is by far the like­list sort of tech­nolog­i­cal dis­aster.

Of course, if we have sen­tient non-hu­man in­tel­li­gences, they’d likely be im­mune to such non­sense. And given our im­prove­ments in au­toma­tion con­trol­ling plague-swept ar­eas is prob­a­bly go­ing to only get eas­ier over time; why use sol­diers who can po­ten­tially get in­fected when we can pa­trol with drones?

• Every­thing else is way fur­ther down the totem pole.

Peo­ple talk about the grey goo sce­nario, but I ac­tu­ally think that is quite silly be­cause there is already grey goo all over the planet in the form of life. There are ab­solutely enor­mous amounts of bac­te­ria and viruses and fungi and ev­ery­thing else all around us, and given the enor­mous ad­van­tage which would be con­ferred by be­ing a grey goo from an evolu­tion­ary stand­point, we would ex­pect the en­tire planet to have already been cov­ered in the stuff—prob­a­bly re­peat­edly. The fact that we see so much di­ver­sity—the fact that noth­ing CAN do this, de­spite enor­mous evolu­tion­ary in­cen­tive TO do this—sug­gests that grey goo sce­nar­ios are ei­ther im­pos­si­ble or in­cred­ibly un­likely. And that’s ig­nor­ing the ther­mo­dy­namic is­sues which would al­most cer­tainly pre­vent such a sce­nario from oc­cur­ring as well, given the ne­ces­sity of re­shap­ing what­ever ma­te­rial into the self-repli­cat­ing ma­te­rial, which would surely take more en­ergy than is pre­sent in the ma­te­rial to be­gin with.

Physics ex­per­i­ments gone wrong have similar prob­lems—we’ve seen su­per­novas. The en­ergy re­leased by a su­per­nova is just vastly be­yond what any sort of plane­tary civ­i­liza­tion is likely ca­pa­ble of do­ing. And see­ing as su­per­novas don’t de­stroy ev­ery­thing, it is vastly un­likely that what­ever WE do will do the same. There are enor­mously en­er­getic events in the uni­verse, and the uni­verse it­self is rea­son­ably sta­ble—it seems un­lik­ley that our fee­ble, mere plane­tary en­ergy lev­els are go­ing to do any bet­ter in the “de­stroy ev­ery­thing” de­part­ment. And even be­fore that, there was the Big Bang, and the uni­verse came to ex­ist out of that whole mess. We have the Sun, and me­te­oritic im­pact events, both of which are very pow­er­ful in­deed. And yet, we don’t see ex­otic, earth-shat­ter­ing physics com­ing into play there in un­ex­pected ways. Ex­tremely high en­ergy den­si­ties are not likely to prop­a­gate—they’re likely to dis­si­pate. And we see this in the uni­verse, and in the laws of ther­mo­dy­nam­ics.

It is very easy to IMAGINE a su­per­weapon that an­nihilates ev­ery­thing. But ac­tu­ally build­ing one? Hav­ing one have re­al­is­tic physics? That’s an­other mat­ter en­tirely. In­deed, we have very strong ev­i­dence against it: surely, in­tel­li­gent life has arisen el­se­where in the uni­verse, and we would see galax­ies be­ing an­nihilated by high-end weaponry. We don’t see this hap­pen­ing. Thus we can as­sume with a pretty high level of con­fi­dence that such weapons do not ex­ist or can­not be cre­ated with­out an im­plau­si­ble amount of work.

The difficult physics of in­ter­stel­lar travel is not to be de­nied, ei­ther—the best we can do with pre­sent physics is nu­clear pulse propul­sion, which is per­haps 10% of c and has enor­mous lo­gis­ti­cal is­sues. Any­thing FTL re­quires ex­otic physics which we don’t have any idea of how to cre­ate, and which may well de­scribe situ­a­tions which are not phys­i­cally plau­si­ble—that is to say, the num­bers may work, but there may well be no way to get there, the same as how there’s no par­tic­u­lar rea­son go­ing faster than c is im­pos­si­ble, but you can’t ever even REACH c, so the fact that there is a “safe space” ac­cord­ing to the math on the other side is mean­ingless. Without FTL, in­ter­stel­lar travel is far too slow for such dis­asters to re­ally prop­a­gate them­selves across the galaxy—any sort of plague would die out on the planet it was cre­ated on, and even WITH FTL, it is still rather un­likely that you could eas­ily spread some­thing like that. Only if cheap FTL travel ex­isted would spread­ing the plague be all that vi­able… but with cheap FTL travel, ev­ery­one else can flee it that much more eas­ily.

My con­clu­sion from all of this is that these sorts of es­ti­mates are less “es­ti­mates” and more “wild guesses which we pre­tend have some mean­ing, and which we throw around a lot of fancy math to con­vince our­selves and oth­ers that we have some idea what we’re talk­ing about”. And that es­ti­mates like one in three mil­lion, or one in ten, are wild over­es­ti­mates—and in­deed, aren’t based on any logic any more sound than the guy on the daily show who said that it would ei­ther hap­pen, or it wouldn’t, a 50% chance.

We have ex­tremely strong ev­i­dence against galac­tic and uni­ver­sal an­nihila­tion, and there are ex­tremely good rea­sons to be­lieve that even plane­tary level an­nihila­tion sce­nar­ios are un­likely due to the sheer amount of en­ergy in­volved. You’re look­ing at bio­cides and large rocks be­ing di­verted from their or­bits to hit planets, nei­ther of which are re­ally triv­ial things to do.

It is ba­si­cally a case of http://​​tvtropes.org/​​pmwiki/​​pmwiki.php/​​Main/​​Sci­fiWrit­er­sHaveNoSenseOfS­cale, ex­cept ap­plied in a much more pes­simistic man­ner.

The only re­ally GOOD ar­gu­ment we have for life­time limited civ­i­liza­tions is the url=https://​​en.wikipe­dia.org/​​wiki/​​Fermi_para­dox—that is to say, where are all the bloody aliens? Un­for­tu­nately, the Fermi Para­dox is a some­what weak ar­gu­ment pri­mar­ily be­cause we have ab­solutely no idea what­so­ever which side of the Great Filter we are on. That be­ing said, if prac­ti­cal FTL travel ex­ists, I would ex­pect that to pretty much en­sure that any civ­i­liza­tion which in­vented it would likely sim­ply never die be­cause of how easy it would be to spread out, mak­ing de­stroy­ing them all vastly more difficult. The galaxy would prob­a­bly end up colonized and re­colonized re­gard­less of how much peo­ple fought against it.

Without FTL travel, galac­tic coloniza­tion is pos­si­ble, but it may be im­prac­ti­cal from an eco­nomic stand­point; there is lit­tle benefit to the home planet of hav­ing ad­di­tional planets colonized—in­for­ma­tion is the only thing you could ex­pect to re­ally trade over in­ter­stel­lar dis­tances, and even that is ques­tion­able given that lo­cals will likely try to de­velop tech­nol­ogy lo­cally and beat you to mar­ket, so un­less hab­it­able sys­tems are very close to­gether du­pli­ca­tion of effort seems ex­tremely likely. En­ter­tain­ment would thus be the largest benefit—games, nov­els, movies and such­like. This MIGHT mean that coloniza­tion is un­likely, which would be an­other ex­plaina­tion… but even there, that as­sumes that they wouldn’t want to ex­plore for the sake of do­ing so.

Of course, it is also pos­si­ble we’re already on the other side of the Great Filter, and the rea­son we don’t see any other in­tel­li­gent civ­i­liza­tions coloniz­ing our galaxy is be­cause there aren’t any, or the ones which have ex­isted de­stroyed them­selves ear­lier in their his­tory or were in­ca­pable of pro­gress­ing to the level we reached due to lack of in­tel­li­gence, lack of re­sources, eter­nal, un­end­ing war­fare which pre­vented progress, or some­thing else.

This is why push­ing for hav­ing a mul­ti­plane­tary civ­i­liza­tion is, I think, a good thing; if we hit the point where we had 4-5 ex­tra­so­lar colonies, I think it would be pretty solid ev­i­dence in fa­vor of be­ing be­yond the Great Filter. Given the dearth of ev­i­dence for in­ter­stel­lar dis­asters cre­ated by in­tel­li­gent civ­i­liza­tions, I think that it is likely that our main con­cern about de­stroy­ing our­selves comes un­til the point where we ex­pand.

But I digress.

It isn’t im­pos­si­ble that we will de­stroy our­selves (af­ter all, the Fermi Para­dox does offer some weak ev­i­dence for it), but I will say that I find any sort of claims of num­bers for the like­li­hood of do­ing so in­cred­ibly sus­pect, as they are very likely to be made up. And given that we have no ev­i­dence of civ­i­liza­tions be­ing ca­pa­ble of gen­er­at­ing galaxy-wide dis­asters, it seems likely that what­ever dis­asters ex­ist are plane­tary scale at best. And our lack of any sort of plau­si­ble sce­nar­ios even for that hurts even that ar­gu­ment. The only real ev­i­dence we have against our civ­i­liza­tion ex­ist­ing in­definitely is the Fermi Para­dox, but it has its own flaws. We may de­stroy our­selves. But un­til we find other civ­i­liza­tions, you are fool­ing your­self if you think you aren’t just mak­ing up num­bers. Any­thing which de­stroys us out­side of an im­pact event is likely some­thing we can­not pre­dict.

• Peo­ple talk about the grey goo sce­nario, but I ac­tu­ally think that is quite silly be­cause there is already grey goo all over the planet in the form of life” … ” noth­ing CAN do this, be­cause noth­ing HAS done it.”

The grey goo sce­nario isn’t re­ally very silly. We seem to have had a green goo sce­nario around 1.5 to 2 billion years ago that kil­led off many or most crit­ters around due to re­lease of deadly deadly oxy­gen; if the bac­te­rial ecosys­tem were com­pletely sta­ble against goo sce­nar­ios this wouldn’t have hap­pened. We have had mini goo sce­nar­ios when for ex­am­ple micro­biota pretty well adapted to one species made the jump to an­other and oops, started re­pro­duc­ing rapidly and kil­ling off their new host species rapidly, e.g. Yersi­nia pestis. Just be­cause we haven’t seen a more om­nivous goo sweep over the eco­sphere re­cently …, …other than Homo sapi­ens, which is ac­tu­ally a pretty good ex­am­ple of a grey goo—think of the species as a crude mesoscale uni­ver­sal as­sem­bler, which is spread­ing pretty fast and kil­ling off other species at a good clip and chew­ing up re­sources quite rapidly… … doesn’t mean it couldn’t hap­pen at the microscale also. Ask the anaer­obes if you can find them, they are hid­ing pretty well still af­ter the chloro­phyll in­ci­dent.

Since the down­side is pretty far down, I don’t think com­pla­cency is called for. A rea­son­able cau­tion be­fore de­ploy­ing some­thing that could per­haps eat ev­ery­one and ev­ery­thing in sight seems pru­dent.

Re­mem­ber that the planet spent al­most 4 billion years more or less cov­ered in var­i­ous kind of goo be­fore the Pre­cam­brian Ex­plo­sion. We know /​very lit­tle/​ of the true his­tory of life in all that time; there could have been many, many, many apoc­a­lyp­tic type sce­nar­ios where a new goo was de­ployed that spread over the planet and ate al­most ev­ery­thing, then ei­ther died wal­low­ing in its own cra­pu­lence or formed the base layer for a new sort of evolu­tion.

Mul­ti­cel­lu­lar life could have started to evolve /​thou­sands of times/​ only to be wiped out by goo. If mul­ti­cel­lu­lars only rarely got as far as bones or shells, and were more vuln­er­a­ble to be­ing wiped out by a goo-plo­sion than sin­gle cel­led crit­ters that could re­build their pop­u­la­tion from a few sur­viv­ing pock­ets or spores, how would we even know? Maybe it took billions of years for the Great War Of Goo to end in a Great Com­pro­mise that al­lowed meso­scopic life to be­gin to evolve, maybe there were great dis­tributed net­works of bac­te­rial and viral bio­chem­i­cal com­put­ing en­g­ines that de­vel­oped in­tel­li­gence far be­yond our own and even­tu­ally de­vel­oped al­tru­ism and peace, de­cid­ing to let mul­ti­cel­lu­lar life de­velop.

Or we eu­kary­otes are the stupid run­away “wet” tech­nol­ogy grey goo of prior prokary­ote/​viral in­tel­li­gent net­works, and we /​de­stroyed/​ their net­works and in­tel­li­gence with our run­away re­pro­duc­tion. Maybe the rea­son we don’t see dis­asters like forests and cities dis­solv­ing in swarms of An­dromeda-Strain like uni­ver­sal gob­blers is that safe­guards against that were ei­ther en­g­ineered in, or out­lawed, long ago. Or, more con­ven­tion­ally, evolved.

What we /​do/​ think we know about the his­tory of life is that the Earth evolved sin­gle cel­led life or in­her­ited it via pansper­mia etc. within about half a billion years of the Earth’s co­a­les­cence, then some com­bi­na­tion of goo more or less dom­i­nated the Earth’s sur­face te roost (as far as biol­ogy goes) for over three billion years, esp if you count colonies like stro­ma­to­lites as gooey. In the mid­dle of this long pe­riod was at least one thing that looked like a goo apoc­a­lypse that re­made the Earth profoundly enough that the traces are very ob­vi­ous (e.g. huge beds of iron ore). But there could have been many more mass ex­tinc­tions we know of.

Then less than a billion years ago some­thing changed profoundly and mul­ti­cel­lu­lars started to flour­ish. This era is less than a sixth of the span of life on earth. So… five sixths, goo dom­i­nated world, one sixth, non goo dom­i­nated world, is the short his­tory here. This does not fill me with con­fi­dence that our world is very sta­ble against a new kind of goo based on non-wet, non-bio­chem­i­cal as­sem­blers.

I do think we are pretty likely not to de­ploy grey goo, though. Not be­cause hu­mans are not idiots—I am an idiot, and it’s the kind of mis­take I would make, and I’m demon­stra­bly above av­er­age by many mea­sures of in­tel­li­gence. It’s just that I think Eliezer and oth­ers will de­ploy a pre-nan­otech Friendly AI be­fore we get to the grey goo tip­ping point, and that it will be smart enough, al­tru­is­tic enough, and ca­pa­ble enough to pre­vent hu­man­ity from bletch­ing the planet as badly as the green microbes did back in the day :)

• You are start­ing from the premise that gray goo sce­nar­ios are likely, and try­ing to ra­tio­nal­ize your be­lief.

Yes, we can be clever and think of hu­mans as green goo—the ul­ti­mate in green goo, re­ally. That isn’t what we’re talk­ing about and you know it—yes, in­tel­li­gent life can spread out ev­ery­where, that isn’t what we’re wor­ried about. We’re wor­ried about un­in­tel­li­gent things wiping out in­tel­li­gent things.

The great oxy­gena­tion event is not ac­tu­ally an ex­am­ple of a green goo type sce­nario, though it is an in­ter­est­ing thing to con­sider—I’m not sure if there even is a gen­er­al­ized term for that kind of sce­nario, as it was es­sen­tially slow at­mo­spheric poi­son­ing. It would be more of a gen­er­al­ized bio­cide type sce­nario—the cyanobac­te­ria which caused the great oxy­gena­tion event cre­ated some­thing which was in­ci­den­tally toxic to other things, but it was purely in­ci­den­tal, had noth­ing to do with their own ac­tion, prob­a­bly didn’t even benefit most of them di­rectly (that is to say, the tox­i­c­ity of the oxy­gen they pro­duced prob­a­bly didn’t help them per­son­ally), and what ac­tu­ally took over af­ter­wards were things which were rather differ­ent from what came be­fore, many of which were not de­scended from said cyanobac­te­ria.

It was a ma­jor at­mo­spheric change, and is (the­o­ret­i­cally) a dan­ger, though I’m not sure how much of an ac­tual dan­ger it is in the real world—we saw the at­mo­sphere shift to an oxy­gen-dom­i­nated one, but I’m not sure how you’d do it again, as I’m not sure there’s some­thing else which can be freed en-mass which is toxic—bet­ter oxy­gena­tors than oxy­gen are hard to come by, and by their very na­ture are rather difficult to liber­ate from an en­ergy bal­ance stand­point. It seems likely that our at­mo­sphere is oxy­gen-based and not, say, chlo­rine or fluorine based for a rea­son aris­ing from the physics of liber­at­ing said chem­i­cals from chem­i­cal com­pounds.

As far as re­peated green goo sce­nar­ios prior to 600Mya—I think that’s pretty un­likely, hon­estly. Look­ing at micro­bial di­ver­sity and micro­bial genomes, we see that the do­mains of life are ridicu­lously an­cient, and that di­ver­sity goes back an enor­mously long dis­tance in time. It seems very un­likely that re­peated green goo type sce­nar­ios would spare the amount of di­ver­sity we ac­tu­ally see in the real world. Eukary­otic life arose 1.6-2.1Bya, and as far as mul­ti­cel­lu­lar life goes, we’ve ev­i­dence of cyanobac­te­ria which showed signs of mul­ti­cel­lu­lar­ity 3Bya.

That’s a long, long time, and it seems un­likely that re­peated gray goo sce­nar­ios are what kept life sim­ple. It seems more likely that what kept life sim­ple was the fact that com­plex­ity is hard—in­deed, I sus­pect the big ad­vance­ment was ac­tu­ally ma­jor ad­vance­ments in mod­u­lar­ity of life. The more mod­u­lar life be­comes, the eas­ier it is to evolve quickly and adapt to new cir­cum­stances, but mod­u­lar­ity from non-mod­u­lar­ity is some­thing which is pretty tough to sort out. Once things did sort it out, though, we saw a mas­sive ex­plo­sion in di­ver­sity. Evolv­ing to be bet­ter at evolv­ing is a good strat­egy for con­tin­u­ing to ex­ist, and I sus­pect that com­plex mul­ti­cel­luar life only came to ex­ist when stuff got to the point where this could hap­pen.

If we saw re­peated green goo sce­nar­ios, we’d ex­pect the var­i­ous branches of life to be pretty shal­low—even if some di­ver­sity sur­vived, we’d ex­pect each di­verse group to show a ma­jor bot­tle­neck back at when­ever the last green goo oc­curred. But that’s not what we ac­taully see. Fungi and an­i­mals di­verged about 1.5 Bya, for in­stance, and other eu­kary­otic di­ver­sity oc­curred even prior to that. An­i­mals have been di­verg­ing for 1.2 billion years.

It seems un­likely, then, that there have been any green goo sce­nar­ios in a very, very long time, if in­deed they ever did oc­cur. In­deed, it seems likely that life evolved to pre­vent said sce­nar­ios, and did so suc­cess­fully, as none have oc­curred in a very, very, very long time.

Pestilence is not even close to green goo. Yes, in­tro­duc­ing a new dis­ease into a new species can be very nasty, but it al­most never ac­tu­ally is, as most of the time, it just doesn’t work at all. Even amongst the same species, Smal­lpox and other old-world dis­eases wiped out the Na­tive Amer­i­cans, but Na­tive Amer­i­can dis­eases were not nearly so dev­as­tat­ing to the old-wor­lders.

Most things which try to jump the species bar­rier have a great deal of difficulty in do­ing so, and even when they suc­cess­fully do so, their viru­lence ends up drop­ping over time be­cause be­ing ridicu­lously fatal is ac­tu­ally bad for their own con­tinued prop­a­gataion. And hu­mans have be­come in­creas­ingly bet­ter at stop­ping this sort of thing. I did note en­g­ineered plagues as the most likely tech­nolog­i­cal threat, but com­par­ing them to gray goo sce­nar­ios is very silly—pathogens are enor­mously eas­ier to con­trol. The trou­ble with stuff like gray goo is that it just keeps spread­ing, but with a pathogen, it re­quires a host—there are all sorts of bar­ri­ers in place to pathogens, and ev­ery­thing is evolved to be able to deal with pathogens be­cause they some­times have to deal with even new ones, and things which are more likely to sur­vive ex­po­sure to novel pathogens are more likely to pass on their genes in the long term.

With re­gards to “in­tel­li­gent viral net­works”—this is just silly. Life on earth is NOT the re­sult of in­tel­li­gence. You can tell this from our genomes. There are no signs of en­g­ineer­ing ANYWHERE in us; no signs of in­tel­li­gent de­sign.

• The gray goo is pred­i­cated on the sort of think­ing com­mon in bad scifi.

Ba­si­cally, in scifi the nan­otech self repli­ca­tors which eat ev­ery­thing in their path are cre­ated in one step. As op­posed to re­al­is­tic de­pic­tion of tech­nolog­i­cal progress where the first nan­otech repli­ca­tors have to sit in a batch of spe­cial nu­tri­ents and be microwaved, or oth­er­wise pro­vided en­ergy, while be­ing kept perfectly ster­ile (to keep bac­te­ria from eat­ing your nan­otech). Then it’d get grad­u­ally im­proved in great many steps and find many uses rang­ing from can­cer cure to dish­wash­ers, with cor­re­spond­ing de­vel­op­ment in goo con­trol meth­ods. You don’t want your dish­washer goo eat­ing your bread.

The lev­els of metabolic effi­ciency and sheer uni­ver­sal­ity re­quired for the gray goo to be able to eat ev­ery­thing in it’s path (and that’s stuff which hasn’t got­ten eaten nat­u­rally), re­quire mul­ti­tude of break­throughs on top of an in­cred­ibly ad­vanced nan­otech­nol­ogy and nano-man­u­fac­tur­ing ca­pac­ity within ar­tifi­cial en­vi­ron­ments.

How does such an ad­vanced civ­i­liza­tion fight the gray goo? I can’t know what would be the best method, but a goo equiv­a­lent of bac­te­riophage is go­ing to be a lot, lot less com­pli­cated than the goo it­self (as the goo has to be able to me­tab­o­lize a va­ri­ety of foods effi­ciently).

• Please add some­thing like this to the RW nan­otech ar­ti­cle!

• In­deed, we have very strong ev­i­dence against it: surely, in­tel­li­gent life has arisen el­se­where in the uni­verse, and we would see galax­ies be­ing an­nihilated by high-end weaponry.

That’s a bad ar­gu­ment. We don’t know for sure that in­tel­li­gent life has arisen. The fact that we don’t see events like that can sim­ply mean that we are the first.

• That’s a pretty weak ar­gu­ment due to the medi­ocrity prin­ci­ple and the sheer scale of the uni­verse; while we cer­tainly don’t know the val­ues for all parts of the Drake Equa­tion, we have a pretty good idea, at this point, that Earth-like planets are prob­a­bly pretty com­mon, and given that abio­ge­n­e­sis oc­curred very rapidly on Earth, that is weak ev­i­dence that abio­ge­n­e­sis isn’t hard in an ab­solute sense.

Most likely, the Great Filter lies some­where in the lat­ter half of the equa­tion—com­plex, mul­ti­cel­lu­lar life, in­tel­li­gent life, civ­i­liza­tion, or the rapid de­struc­tion thereof. But even as­sum­ing that in­tel­li­gent life only oc­curs in one galaxy out of ev­ery thou­sand, which is in­cred­ibly un­likely, that would still give us many op­por­tu­ni­ties to ob­serve galac­tic de­struc­tion.

It is the­o­ret­i­cally pos­si­ble that we’re the only life in the Uni­verse, but that is in­cred­ibly un­likely; most Uni­verses in which life ex­ists will have life ex­ist in more than one place.

• given that abio­ge­n­e­sis oc­curred very rapidly on Earth, that is weak ev­i­dence that abio­ge­n­e­sis isn’t hard in an ab­solute sense.

We don’t even know that it oc­curred on earth at all. It might have oc­curred el­se­where in our galaxy and trav­eled to earth via as­ter­oids.

most Uni­verses in which life ex­ists will have life ex­ist in more than one place.

Why? I don’t see any rea­son why that should be the case. If you take for ex­am­ple posts that in­ter­net fo­rum users write most of the time most users who write posts only write one post.

• We don’t even know that it oc­curred on earth at all. It might have oc­curred el­se­where in our galaxy and trav­eled to earth via as­ter­oids.

That would make it more likely that there’s life on other planets, not less likely.

• Most planets and stars in the uni­verse are not in our galaxy. If our galaxy has a bit of uni­cel­lu­lar life be­cause some very rare event hap­pened and is the only galaxy with life, that fits to a uni­verse where we are the only in­tel­li­gent species.

• It looks like you ac­ci­den­tally sub­mit­ted your com­ment be­fore finish­ing it (or there’s a mis­for­mat­ted link or some­thing).

• I cor­rected it.

• After read­ing through all of the com­ments, I think I may have failed to ad­dress your cen­tral point here.

Your cen­tral point seems to be “a ra­tio­nal agent should take a risk that might re­sult in uni­ver­sal de­struc­tion in ex­change for in­creased util­ity”.

The prob­lem here is I’m not sure that this is even a mean­ingful ar­gu­ment to be­gin with. Ob­vi­ously uni­ver­sal de­struc­tion is ex­tremely bad, but the prob­lem is that util­ity prob­a­bly in­cludes all life NOT be­ing ex­tin­guished. Or, in other words, this isn’t nec­es­sar­ily a mean­ingful calcu­la­tion if we as­sume that the al­ter­na­tive makes it more likely that uni­ver­sal an­nihila­tion will oc­cur.

Say the Nazis gain an ex­ces­sive amount of power. What hap­pens then? Well, there’s the risk that they make some sort of plague to cleanse hu­man­ity, screw it up, and wipe ev­ery­one out. That sce­nario seems MORE likely in a Nazi-run world than one which isn’t. And—let’s face it—chances are the Nazis will try and de­velop nu­clear weapons, too, so at best you only bought a few years. And if the wrong peo­ple de­velop them first, you’re in a lot of trou­ble. So the fact of the mat­ter is that the risk is go­ing to be taken re­gard­less, which fur­ther diminishes the loss of util­ity you could ex­pect from uni­ver­sal an­nihila­tion—sooner or later, some­one is go­ing to do it, and if it isn’t you, then it will be some­one else who gains what­ever benefits there are from it.

The higher util­ity situ­a­tion likely de­creases the fu­ture odds of uni­ver­sal an­nihila­tion, mean­ing that, in other words, it is en­tirely ra­tio­nal to take that risk sim­ply be­cause the odds of de­stroy­ing the world NOW are less than the odds of the world be­ing de­stroyed fur­ther on down the line by some­one else if you don’t make this de­ci­sion, es­pe­cially if you can be rea­son­ably cer­tain some­one else is go­ing to try it out any­way. And given the odds are in­cred­ibly low, it is a lot less mean­ingful of a choice to be­gin with.

• In­ci­den­tally, re­gard­ing some other things in here:

[quote]They thought that just be­fore World War I. But that’s not my fi­nal re­jec­tion. Evolu­tion­ary ar­gu­ments are a more pow­er­ful rea­son to be­lieve that peo­ple will con­tinue to have con­flicts. Those that avoid con­flict will be out-com­peted by those that do not.[/​quote]

There’s ac­tu­ally a pretty good counter-ar­gu­ment to this, namely the fact that cap­i­tal is vastly eas­ier to de­stroy than it is to cre­ate, and that, thusly, an area which avoids con­flict has an enor­mous ad­van­tage over one that doesn’t be­cause it main­tains more of its cap­i­tal. As cap­i­tal be­comes in­creas­ingly im­por­tant, con­flict—at least, vi­o­lent, cap­i­tal-de­stroy­ing con­flict—be­comes mas­sively less benefi­cial to the per­pe­tra­tor of said con­flict, dou­bly so when they ac­tu­ally also likely benefit from the cap­i­tal con­tained in other na­tions as well due to trade.

And that’s ig­nor­ing the fact that we’ve already sort of en­g­ineered a global sce­nario where “The West” (the US, Canada, Ja­pan, South Korea, Taiwan, Aus­tralia, New Zealand, and Western Europe, creep­ing now as far east as Poland) never at­tack each other, and slowly make ev­ery­one else in the world more like them. It is group se­lec­tion of a sort, and it seems to be work­ing pretty well. Th­ese coun­tries defend their cap­i­tal, and each oth­ers’ cap­i­tal, benefit from each oth­ers’ cap­i­tal, and en­gage soley in non-vi­o­lent con­flict with each other. If you threaten them, they crush you and make you more like them; even if you don’t, they work to cor­rupt you to make you more like them. In­deed, even places like China are slowly be­ing cor­rupted to be more like the West.

The more that sort of thing hap­pens, the less likely vi­o­lent con­flict be­comes be­cause it is sim­ply less benefi­cial, and in­deed, there is even some ev­i­dence to sug­gest we are be­ing se­lected for docil­ity—in “the West” we’ve seen crime rates and homi­cide rates de­cline for 20+ years now.

As a fi­nal, ran­dom aside:

My fa­vorite thing about the Trinity test was the sci­en­tist who was tak­ing side bets on the an­nihila­tion of the en­tire state of New Mex­ico, right in front of the gov­er­nor of said state, who I’m sure was ab­solutely hor­rified.

• the fact that cap­i­tal is vastly eas­ier to de­stroy than it is to create

Cap­i­tal is also eas­ier to cap­ture than it is to cre­ate. Your ar­gu­ment looks like say­ing that it’s bet­ter to avoid wars than to lose them. Well, yeah. But what about win­ning wars?

we’ve already sort of en­g­ineered a global sce­nario where “The West” … never at­tack each other

In which mean­ing are you us­ing the word “never”? :-D

• The prob­lem is that asym­met­ric war­fare, which is the best way to win a war, is the worst way to ac­quire cap­i­tal. Cruise mis­siles and drones are ex­cel­lent for win­ning with­out any risk at all, but they’re not good for ac­tu­ally keep­ing the cap­i­tal you are try­ing to take in­tact.

Spy­ing, sub­ver­sion, and pur­chas­ing are far cheaper, safer, and more effec­tive means of cap­tur­ing cap­i­tal than vi­o­lence.

As far as “never” goes—the last time any two “Western” coun­tries were at war was World War II, which was more or less when the “West” came to be in the first place. It isn’t the longest of time spans, but over time armed con­flict in Europe has greatly diminished and been pushed fur­ther and fur­ther east.

• The prob­lem is that asym­met­ric war­fare, which is the best way to win a war, is the worst way to ac­quire cap­i­tal.

The best way to win a war is to have an over­whelming ad­van­tage. That sort is situ­a­tion is much bet­ter de­scribed by the word “lop­sided”. Asym­met­ric war­fare is some­thing differ­ent.

Ex­am­ple: Iraqi in­va­sion of Kuwait.

Spy­ing, sub­ver­sion, and pur­chas­ing are far cheaper, safer, and more effec­tive means of cap­tur­ing cap­i­tal than vi­o­lence.

Spy­ing can cap­ture tech­nol­ogy, but tech­nol­ogy is not the same thing as cap­i­tal. Nei­ther sub­ver­sion nor pur­chas­ing are “means of cap­tur­ing cap­i­tal” at all. Sub­ver­sion de­stroys cap­i­tal and pur­chases are ex­changes of as­sets.

As far as “never” goes—the last time any two “Western” coun­tries were at war was World War II, which was more or less when the “West” came to be in the first place.

That’s an un­usual idea of the West. It looks to me like it was cus­tom-made to fit your the­sis.

Can you provide a defi­ni­tion? One suffi­ciently pre­cise to be able to al­lo­cate coun­tries like Poland, Is­rael, Chile, Bri­tish Vir­gin Is­lands, Es­to­nia, etc. to ei­ther “West” or “not-West”.

• Depends on the cap­i­tal. Doesn’t work too well for in­fras­truc­ture and hu­man cap­i­tal, and the west has plenty of those any­way. What the west is in­se­cure about is en­ergy,and it seems that a com­bi­na­tion of diplo­macy, threat and proxy war­fare is a more effi­cient way to keep it flow­ing than all out cap­ture.

• Doesn’t work too well for in­fras­truc­ture and hu­man capital

Depends on the hu­man cap­i­tal. Look at the his­tory of the US space pro­gram :-/​

is a more effi­cient way to keep it flowing

At the mo­ment. I’m wary of evolu­tion­ary ar­gu­ments based on a few decades worth of data.

• The ex­am­ple of von Braun and co crossed my mind. But that was some­thing of a side effect. Fight­ing a war speci­fi­cally to cap­ture a small­ish num­bers of smart peo­ple is frought with risks.

• Op­por­tunis­tic seizure of cap­i­tal is to be ex­pected in a war fought for any pur­pose.

• In­ci­den­tally, you can block­quote para­graphs by putting > in front of them, and you can find other help by click­ing the “Show Help” but­ton to the bot­tom right of the text box. (I have no clue why it’s all the way over there; it makes it way less visi­ble.)

There’s ac­tu­ally a pretty good counter-ar­gu­ment to this, namely the fact that cap­i­tal is vastly eas­ier to de­stroy than it is to cre­ate, and that, thusly, an area which avoids con­flict has an enor­mous ad­van­tage over one that doesn’t be­cause it main­tains more of its cap­i­tal.

But, the more con­flict avoidant the agents in an area, the more there is to gain from be­ing an agent that seeks con­flict.

• The more con­flict avoidant the agents in an area, the more there is to gain from be­ing an agent that seeks con­flict.

This is only true if the con­flict avoidance is in­nate and is not in­stead a form of re­cip­ro­cal al­tru­ism.

Re­cip­ro­cal al­tru­ism is an ESS where pure al­tru­ism is not be­cause you can­not take ad­van­tage of it in this way; if you be­come bel­liger­ent, then ev­ery­one else turns on you and you lose. Thus, it is never to your ad­van­tage to be­come bel­liger­ent.

• Agreed. The word ‘avoid’ and the group se­lec­tion-y ar­gu­ment made me think it was a good idea to raise that ob­jec­tion and make sure we were dis­cussing re­cip­ro­cal paci­fists, not pure paci­fists.

• I don’t even know how they got at it, but it sounds like a typ­i­cal wild guess to me. How would you even ar­rive at that figure?

Here is a con­tem­po­rary pa­per dis­cussing the risk, which doesn’t seem to come up with the 3e-6 num­ber, and here are some of Ham­ming’s re­flec­tions. An ex­cerpt from the sec­ond link:

Shortly be­fore the first field test (you re­al­ize that no small scale ex­per­i­ment can be done—ei­ther you have a crit­i­cal mass or you do not), a man asked me to check some ar­ith­metic he had done, and I agreed, think­ing to fob it off on some sub­or­di­nate. When I asked what it was, he said, “It is the prob­a­bil­ity that the test bomb will ig­nite the whole at­mo­sphere.” I de­cided I would check it my­self!

Comp­ton claims (in an in­ter­view with Pearl Buck I can­not eas­ily find on­line) that 3e-6 was ac­tu­ally the de­ci­sion crite­rion (if it was higher than that, they were go­ing to shut down the pro­ject as more dan­ger­ous than the Nazis), and the es­ti­mate came in at lower, and so they went ahead with the pro­ject.

In mod­ern re­ac­tors, they try to come up with a failure prob­a­bil­ity by putting dis­tri­bu­tions on un­known vari­ables dur­ing po­ten­tial events, simu­lat­ing those events, and then figur­ing out what por­tion of the joint in­put dis­tri­bu­tion will lead to a catas­trophic failure. One could do the same with un­known pa­ram­e­ters like the cross-sec­tion of ni­tro­gen at var­i­ous tem­per­a­tures; “this is what we think it could be, and we only need to be wor­ried if it’s over here.”

• Ap­par­ently I don’t know how to use this sys­tem prop­erly.

• In say­ing “our com­pres­sion of sub­jec­tive time can be ex­po­nen­tial”, do you ac­tu­ally mean that the com­pres­sion rate may keep grow­ing ex­po­nen­tially as a func­tion of real time?

• Com­pres­sion at­tained can be an ex­po­nen­tial func­tion of time. That’s not the same as say­ing that com­pres­sion rate can grow ex­po­nen­tially. I mean, if it’s a “rate”, it already ex­presses how com­pres­sion grows, so “the com­pres­sion rate grows ex­po­nen­tially” means “the first deriva­tive of com­pres­sion grows ex­po­nen­tially”.

Any­way, com­pres­sion can’t keep in­creas­ing in­definitely, due to the Planck con­stant. Mike Vas­sar once did some back-of-en­velope-calcu­la­tions show­ing that we have sur­pris­ingly few or­ders of mag­ni­tude to go be­fore we hit it in terms of com­pu­ta­tional power—less than 20 or­ders of mag­ni­tude, IIRC. But two or­ders of mag­ni­tude is enough to kill us, in this sce­nario. Ba­si­cally, it will take us some­thing like 5 mil­lion years to reach an­other galaxy, at which point you might con­sider life safe. If we get just 2 or­ders of mag­ni­tude out of sub­jec­tive time com­pres­sion that’s like 500 mil­lion years, and our sur­vival to that point seems du­bi­ous.

(I went back to clar­ify this be­cause I re­al­ized peo­ple don’t usu­ally think of 20 or­ders of mag­ni­tude as “sur­pris­ingly few”.)

• By “com­pres­sion rate” I meant “com­pres­sion ra­tio”. Sorry for the con­fu­sion. But you know that if some­thing grows ex­po­nen­tially, all of its nth deriva­tives do also, right?

I did know that the ac­tual uni­verse prob­a­bly has some phys­i­cal limits in how you can shrink a com­pu­ta­tion in space and/​or time, thus my ques­tion. Ac­tu­ally, I thought you might have done the not-math “ex­po­nen­tial” as a way of say­ing “A LOT!!!”

• Okay, it is the same thing as say­ing that com­pres­sion rate can grow ex­po­nen­tially.

I meant ex­po­nen­tial. I don’t know if I be­lieve it’s ex­po­nen­tial, but al­most all other fu­tur­ists say that things are speed­ing up (time is com­press­ing) ex­po­nen­tially.

• “if your util­ity is at risk of go­ing nega­tive; it’s not pos­si­ble that you would not ac­cept a .999% risk”. I as­sume you mean that if my util­ity is around 0, and things are trend­ing to­ward worse, I should be happy to ac­cept a 99.9% chance of de­stroy­ing the uni­verse (as­sum­ing I’m the .1% pos­si­bil­ity gives me some im­prove­ment).

“Is life barely worth liv­ing? Buy a lot­tery ticket, and if you don’t win, kill your­self—you win ei­ther way!”—prob­a­bly not the best mar­ket­ing cam­paign for the state-run lot­tery.

• “if your util­ity is at risk of go­ing nega­tive; it’s not pos­si­ble that you would not ac­cept a .999% risk”

Look at where the semi­colon is. You’ve com­bined the end of one clause with the be­gin­ning of a differ­ent clause.

“Is life barely worth liv­ing? Buy a lot­tery ticket, and if you don’t win, kill your­self—you win ei­ther way!”—prob­a­bly not the best mar­ket­ing cam­paign for the state-run lot­tery.

• Here’s an­other pos­si­ble ob­jec­tion:

Much of time-dis­count­ing is due to un­cer­tainty. Be­cause you’re more and more un­cer­tain what the im­pact of a de­ci­sion will be the farther you look into the fu­ture, you weigh that im­pact less and less the farther into the fu­ture you go.

But if you can find a way to pre­dict fu­ture im­pacts that is un­bi­ased, then you don’t need to time-dis­count due to un­cer­tainty! Um… do you?

No, I think you still do, be­cause your prob­a­bil­ity dis­tri­bu­tion’s var­i­ance in­creases the farther you look into the fu­ture. Rats.

• It seems like this post ex­hibits a great deal of omis­sion bias. Re­fus­ing to make ra­tio­nal trade-offs with ex­is­ten­tial risk doesn’t make the risk go away.

• You seem to be say­ing that we’re doomed if we do, and doomed if we don’t.

So, take on all these risks, be­cause life is go­ing to be ex­tin­guished any­way?

I don’t think that’s what you mean. I think, if you mean any­thing co­her­ent, you mean that… no, I can’t figure out what you might mean.

If you choose not to take a risk, it makes that risk go away. If you mean that you’re go­ing to get hit by some other risk that you didn’t think of any­way, you are show­ing lit­tle faith in in­tel­li­gence. As we learn more, we be­come aware of more and more risks. There can’t be an in­finite num­ber of ex­is­ten­tial risks wait­ing for us, or we wouldn’t be here. There­fore, we can ex­pect to even­tu­ally an­ti­ci­pate most ex­is­ten­tial risks, and deal with them.

• I’m as­sum­ing that avoid­ing risks in­volves a trade­off ver­sus other forms of hu­man welfare, and I don’t see why a strat­egy that makes hu­man­ity worse off but longer lived is nec­es­sar­ily prefer­able to one that makes hu­man­ity bet­ter off but shorter lived.

And yes, we’re “doomed” in the sense that, as far as I un­der­stand, an in­finitely long ex­is­tence isn’t an available op­tion.

• Re: “If you be­lieve in ac­cel­er­at­ing change, then the num­ber of im­por­tant events in a given time in­ter­val in­creases ex­po­nen­tially, or, equiv­a­lently, the time in­ter­vals that should be con­sid­ered equiv­a­lent op­por­tu­ni­ties for im­por­tant events shorten ex­po­nen­tially.”

Uh—that doesn’t go on for­ever. Any more than rats with a grain pile al­lows growth for­ever. Your state­ment takes the idea of ex­po­nen­tial change into an ut­terly ridicu­lous realm.

• You’re right. It still has a large im­pact, though. Even if we get only 3 more dou­blings, it re­duces the time available by a fac­tor of 8.

The near­est other galaxy is 2 mil­lion light years away. If we get 6 dou­blings, that’s 128 mil­lion sub­jec­tive light years. That’s a wor­ri­some amount.

The near­est other star is 4.24 light years away. If we get 20 dou­blings, that’s over 4 mil­lion sub­jec­tive light years away. Also a wor­ri­some amount.

• The ob­ser­va­tion bears on this state­ment:

“More im­por­tant is that our com­pres­sion of sub­jec­tive time can be ex­po­nen­tial, while our abil­ity to es­cape from ever-broader swaths of de­struc­tion is limited by light­speed.”

Even­tu­ally, com­pres­sion in sub­jec­tive time stops, while ex­pan­sion con­tinues.

• Yes, that’s right. If you can sur­vive long enough to get to that point.

• We’re talk­ing about bring­ing ex­is­ten­tial threats to chances less than 1 in a mil­lion per cen­tury. I don’t know of any defen­sive tech­nol­ogy that can guaran­tee a less than 1 in a mil­lion failure rate.

Un­der your the­ory of 3/​1M/​Cen­tury, you’d only need to do bet­ter than a 13 failure rate to lower chances to 1/​1M/​C. A 13 failure rate seems rather plau­si­ble. If the defense had a 1/​1M failure rate, you’d have a 31,000,000,000,000 chance of erad­i­ca­tion per cen­tury.

• As­sume that there is at least one at­tack per cen­tury, and a suc­cess­ful at­tack will end life. There­fore, you need a failure rate less than 1 in a mil­lion to sur­vive a mil­lion cen­turies.

• This as­sumes that a suc­cess­ful at­tack will end life with P~=1 and a suc­cess­ful at­tack will oc­cur once per cen­tury, which seems, to put it mildly, ex­ces­sive.

As I un­der­stood your origi­nal as­sump­tion, each cen­tury sees one event with P=3/​1M of de­struc­tion, in­de­pen­dent of any in­ter­ven­tion. If an in­ter­ven­tion has a 13 failure rate, and you in­ter­vene ev­ery time, this would re­duce your chance of an­nihila­tion/​cen­tury to 1/​1M, which is your goal.

It’s quite pos­si­ble we’re think­ing of differ­ent things when we say failure rate; I mean the failure rate of the defen­sive mea­sure; I think you mean the failure rate as in the pure odds the world blows up.

• I wasn’t us­ing the Trinity case when I wrote that part. This part as­sumes that we will de­velop some tech­nol­ogy X ca­pa­ble of de­stroy­ing life, and that we’ll also de­velop tech­nol­ogy to defend against it. Then say each cen­tury sees 1 at­tack us­ing tech­nol­ogy X that will de­stroy life if it suc­ceeds. (This may be, for in­stance, from a crazy or re­li­gious or just very very an­gry per­son.) You need to defend suc­cess­fully ev­ery time. It’s ac­tu­ally much worse, be­cause there will prob­a­bly be more than one Tech­nol­ogy X.

If you think about ex­ist­ing equil­ibria be­tween at­tack­ers and defen­ders, such as spam­mers vs. spam filters, it seems un­likely that, once tech­nol­ogy has stopped de­vel­op­ing, ev­ery dan­ger­ous tech­nol­ogy X will have such a highly-effec­tive defense Y against it. The pri­ors, I would think, would be that (av­er­aged over pos­si­ble wor­lds) you would have some­thing more like a 50% chance of stop­ping any given at­tack.

• Every or­gainism you see has is the re­sult of an un­bro­ken chain of non-ex­tinc­tion that stretches back some 4 billion years. The rate of com­plete failure for liv­ing sys­tems is not known—but it ap­pears to have been ex­tremely low so far.

• Time com­pres­sion did not start re­cently. (Well, it did, once you ac­count for time com­pres­sion.)

• Bac­te­ria have limited tech­nolog­i­cal ca­pa­bil­ities.

• Your ar­gu­ment as­sumes that the time-hori­zon of ra­tio­nal util­ity max­imisers never reaches fur­ther than their next de­ci­sion. If I only get one shot to in­crease my ex­pected util­ity by 1%, and I’m ra­tio­nal, yes, I’ll take any odds bet­ter than 99:1 in favour on an all or noth­ing bet. That is a highly con­trived sce­nario: it is al­most always pos­si­ble to stake less than your en­tire util­ity on an out­come, in which case you gen­er­ally should in or­der to re­duce risk-of-ruin and thus in­crease long-term ex­pected util­ity.

Fur­ther, the risks of not us­ing nu­clear weapons in the Se­cond World War were noth­ing like those you gave. Ja­pan was in no dan­ger of oc­cu­py­ing the United States at the time the de­ci­sion to ini­ti­ate the Trinity test was made; as for Ger­many, it had already sur­ren­dered! The an­ti­ci­pated effects of not us­ing the Bomb were rather that of los­ing per­haps sev­eral hun­dred thou­sand sol­diers in an in­va­sion of Ja­pan, and of course the large eco­nomic cost of pro­long­ing the war. As for the calcu­lated risk of at­mo­spheric ig­ni­tion, the calcu­lated risk at the time of test­ing was even lower than the 300000:1 you stated (al­though Fermi offered evens on the morn­ing of the Trinity test).

• That is a highly con­trived sce­nario: it is al­most always pos­si­ble to stake less than your en­tire util­ity on an out­come, in which case you gen­er­ally should in or­der to re­duce risk-of-ruin and thus in­crease long-term ex­pected util­ity.

It is de­rived from real-life ex­pe­riences, which I listed. Yes, it is al­most always pos­si­ble to stake less than your en­tire util­ity. Al­most. Hence the “once per cen­tury”, in­stead of “billions of times per day”.

Fur­ther, the risks of not us­ing nu­clear weapons in the Se­cond World War were noth­ing like those you gave. Ja­pan was in no dan­ger of oc­cu­py­ing the United States at the time the de­ci­sion to ini­ti­ate the Trinity test was made; as for Ger­many, it had already sur­ren­dered! The an­ti­ci­pated effects of not us­ing the Bomb were rather that of los­ing per­haps sev­eral hun­dred thou­sand sol­diers in an in­va­sion of Ja­pan, and of course the large eco­nomic cost of pro­long­ing the war.

I think that’s right. Do you re­al­ize you are only mak­ing my case stronger, by show­ing that the de­ci­sion was made by some­what-ra­tio­nal peo­ple for even fewer benefits?

If you ac­cept that it would be ra­tio­nal to perform the Trinity test in a hy­po­thet­i­cal world in which the Ger­mans and Ja­panese were win­ning the war, and in which it had a 3 in 1 mil­lion chance of de­stroy­ing life on Earth, then I have made my point. Ar­gu­ing about what the risks and benefits truly were his­tor­i­cally is ir­rele­vant. It doesn’t re­ally mat­ter what ac­tual hu­mans did in an ac­tual situ­a­tion, be­cause we already agree that hu­mans are ir­ra­tional. What mat­ters is whether we can find con­di­tions un­der which perfectly ra­tio­nal be­ings, faced with the situ­a­tion as I posed it, would choose not to con­duct the Trinity test.

As for the calcu­lated risk of at­mo­spheric ig­ni­tion, the calcu­lated risk at the time of test­ing was even lower than the 300000:1 you stated

So what were they?

• I meant that low ex­is­ten­tial risk wa­gers are al­most always available, re­gard­less of the pres­ence or ab­sence of high ex­is­ten­tial risk wa­gers, and I claimed that those low risk wa­gers are prefer­able even when the cost of not tak­ing the high risk wa­ger is very high, pro­vided that is that you have a long time hori­zon. The only time you should take a high ex­is­ten­tial risk wa­ger is when your long-term fu­ture util­ity will be per­ma­nently and sub­stan­tially de­creased by your not do­ing so. That doesn’t ap­ply to your ex­am­ple of the first nu­clear test, as the al­ter­na­tive would not lead to the night­mare in­va­sion sce­nario, but rather to a bloody but re­cov­er­able mess. So you haven’t proven the ra­tio­nal­ity of test­ing nukes (al­though only in that sce­nario, as you point out).

If you ac­cept that it would be ra­tio­nal to perform the Trinity test in a hy­po­thet­i­cal world in which the Ger­mans and Ja­panese were win­ning the war, and in which it had a 3 in 1 mil­lion chance of de­stroy­ing life on Earth, then I have made my point.

It ac­tu­ally still de­pends on the time hori­zon and the util­ity of a sur­ren­der or truce. I sup­pose a 30 year cut-off could be short enough to make it ra­tio­nal. Of course if be­ing over­run by Nazis is as­sumed to lead to eter­nal hellish con­di­tions...

What mat­ters is whether we can find con­di­tions un­der which perfectly ra­tio­nal be­ings, faced with the situ­a­tion as I posed it, would choose not to con­duct the Trinity test.

I re­ally don’t think we can. If the util­ity of not-test­ing is zero at all times af­ter T, the util­ity of de­stroy­ing the Earth is zero at all times af­ter T, and the util­ity of win­ning the war is greater than or equal to 0 at all times af­ter T, then what­ever your time hori­zon you will test if you are an ex­pected util­ity max­i­mizer. Where I dis­agree is that there ex­ists a base rate of any­where like once per cen­tury for tak­ing a 3 in a mil­lion chance of de­stroy­ing the world in any sce­nar­ios where those sorts of figures don’t hold. I also dis­agree that ra­tio­nal agents will con­front each other with such choices any­thing like so of­ten as once per cen­tury.

As for the calcu­lated risk of at­mo­spheric ig­ni­tion, the calcu­lated risk at the time of test­ing was even lower than the 300000:1 you stated

So what were they?

They were zero! Proved to be phys­i­cally im­pos­si­ble, and then no-one se­ri­ously ques­tioned the val­idity of the calcu­la­tions. As for the best-pos­si­ble es­ti­mate, it must have been higher, of course.

• Where I dis­agree is that there ex­ists a base rate of any­where like once per cen­tury for tak­ing a 3 in a mil­lion chance of de­stroy­ing the world in any sce­nar­ios where those sorts of figures don’t hold. I also dis­agree that ra­tio­nal agents will con­front each other with such choices any­thing like so of­ten as once per cen­tury.

If you were to wind the clock back to 1940, and restart World War 2, is there less than a 10% chance of ar­riv­ing in the night­mare sce­nario? If not, doesn’t this im­ply the base rate is at least once per thou­sand years?

• It ac­tu­ally still de­pends on the time hori­zon. I sup­pose a 30 year cut-off could be short enough to make it ra­tio­nal. Of course if be­ing over­run by Nazis is as­sumed to lead to eter­nal hellish con­di­tions...

The time-hori­zon is very im­por­tant. One of my points is that I don’t see how a ra­tio­nal agent could have a time-hori­zon on the scale of the life of the uni­verse.

• I think that you’re his­tor­i­cally cor­rect, but it’s enough to posit a hy­po­thet­i­cal World War 2 in which the al­ter­na­tive was the night­mare in­va­sion sce­nario, and show that it would then be ra­tio­nal to con­duct the Trinity test.

• I’ve ed­ited my com­ment in re­sponse to the hy­po­thet­i­cal.

• al­though Fermi offered evens on the morn­ing of the Trinity test

I bet I know which side of the bet Fermi was will­ing to take.

• It would be ra­tio­nal to offer any odds for that bet.

• Se­conded re­gard­ing the stakes in WW2. The sci­en­tists weren’t on the front lines ei­ther, so it’s highly doubt­ful they would have been kil­led.

• As­sum­ing ra­tio­nal agents with a rea­son­able level of al­tru­ism (by which I mean, in­cor­po­rat­ing the needs of other peo­ple and fu­ture gen­er­a­tions into their own util­ity func­tions, to a similar de­gree to what we con­sider “de­cent peo­ple” to do to­day)...

If such a per­son figures that get­ting rid of the Nazis or the Daleks or who­ever the threat of the day is, is worth a tiny risk of bring­ing about the end of the world, and their rea­son­ing is com­pletely ra­tio­nal and valid and al­trustic (I won’t say “un­selfish” for rea­sons dis­cussed el­se­where in this thread) and far-sighted (not dis­count­ing fu­ture gen­er­a­tions too much)...

… then they’re right, aren’t they?

If the guys be­hind the Trinity test weighed the nega­tive util­ity of the Axis tak­ing over the world, pre­sum­ably with the end re­sult of boots stamp­ing on hu­man faces for­ever, and de­ter­mined that the 31,000,000 chance of end­ing all hu­man life was worth pre­vent­ing this fu­ture from com­ing to pass, then couldn’t Queen Vic­to­ria perform the same calcu­la­tions, and con­clude “Good heav­ens. Nazis, you say? Spread­ing their hor­rible fas­cism in my em­pire? Never! I do hope those plucky Amer­i­cans man­age to build their bomb in time. Tiny chance of de­stroy­ing the world? Bet­ter they take that risk than let fas­cism rule the world, I say!”

If the util­ity calcu­la­tions performed re­gard­ing the Trinity test were ra­tio­nal, al­trustic and rea­son­ably far-sighted, then they would have been equally valid if performed at any other time in his­tory. If we ap­ply a fu­ture dis­count­ing fac­tor of e^-kt, then that fac­tor would ap­ply equally to all el­e­ments in the util­ity calcu­la­tion. If the net util­ity of the test were pos­i­tive in 1945, then it should have been pos­i­tive at all points in his­tory be­fore then. If Pres­i­dent Tru­man (ra­tio­nally, al­trusti­cally, far-sight­edly) ap­proved of the test, then so should Queen Vic­to­ria, Julius Cae­sar and Ham­murabi have, given suffi­cient in­for­ma­tion. Either the util­ity calcu­la­tions for the test were right, or they weren’t.

If they were right, then the prob­lem stops be­ing “Oh no, fu­ture gen­er­a­tions are go­ing to de­stroy the world even if they’re sen­si­ble and al­tru­is­tic!”, and starts be­ing “Oh no, a hor­rible regime might take over the world! Let’s hope some­one cre­ates a su­per­weapon to stop them, and damn the risk!”

If they were wrong, then the as­sump­tion that the ones perform­ing the calcu­la­tion were ra­tio­nal, al­trustic and far-sighted is wrong. Tak­ing these one by one:

1) The world might be de­stroyed by some­one mak­ing an ir­ra­tional de­ci­sion. No sur­prises there. All we can do is strive to raise the gen­eral level of ra­tio­nal­ity in the world, at least among peo­ple with the power to de­stroy the world.

2) The world might be de­stroyed by some­one with only his own in­ter­ests at heart. So ba­si­cally we might get stuck with Dr Evil. We can’t do a lot about that ei­ther.

3) The world might be de­stroyed by some­one act­ing ra­tio­nally and al­trusti­cally for his own gen­er­a­tion, but who dis­counts fu­ture gen­er­a­tions too much (i.e. his value of k in the dis­count­ing fac­tor is much larger than ours). This seems to be the crux of the prob­lem. What is the “proper” value of k? It should prob­a­bly de­pend on how much longer hu­mans are go­ing to be around, for rea­sons un­re­lated to the ques­tion at hand. If the world re­ally is go­ing to end in 2012, then ev­ery dol­lar spent on pre­vent­ing global warm­ing should have been spent on alle­vi­at­ing short-term suffer­ing all over the world, and the proper value for k is very large. If we re­ally are go­ing to be here for mil­lions of years, then we should be ex­cep­tion­ally care­ful with ev­ery re­source (both ma­te­rial and ne­gen­tropy-based) we con­sume, and k should be very small. Without this knowl­edge, of course, it’s very difficult to de­ter­mine what k should be.

That may be the way to avoid a well-mean­ing sci­en­tist wiping out all hu­man life—find out how much longer we have as a species, and then cam­paign that ev­ery­one should live their lives ac­cord­ingly. Then, the only ex­is­ten­tial risks that would be im­ple­mented are the ones that are ac­tu­ally, se­ri­ously, truly, in­con­tro­vert­ibly, prov­ably worth it.

• You’ve sidestepped my ar­gu­ment, which is that just the ex­is­ten­tial risks that are worth it are enough to guaran­tee de­stroy­ing the uni­verse in a cos­molog­i­cally short time.

• Re: “If tech­nol­ogy keeps chang­ing, it is in­evitable that, much of the time, a tech­nol­ogy will provide the abil­ity to de­stroy all life be­fore the counter-tech­nol­ogy to defend against it has been de­vel­oped.”

Un­sup­ported hy­poth­e­sis. As life spreads out in the uni­verse, it gets harder and harder to de­stroy all of it—while the tech­nol­ogy of de­struc­tion will sta­bil­ise.

• Un­sup­ported hy­poth­e­sis. As life spreads out in the uni­verse, it gets harder and harder to de­stroy all of it—while the tech­nol­ogy of de­struc­tion will sta­bil­ise.

Did you read the es­say be­fore post­ing? I have a sec­tion on life spread­ing out in the uni­verse, and a sec­tion on whether the tech­nol­ogy of de­struc­tion can sta­bi­lize.

• Yes, I read the es­say. It doesn’t make it any more in­evitable. It isn’t in­evitable at all—rather this is spec­u­la­tion on your part, and un­sub­stan­ti­ated spec­u­la­tion that I can see no sen­si­ble ba­sis for.

• It would be more helpful if you ex­plained why each of the many rea­sons I gave are in­sen­si­ble.

• When ar­gu­ing about the fu­ture, the imag­in­able is not all there is. You es­sen­tially gave sev­eral imag­in­able fu­tures (some in which risks con­tinue to arise, and oth­ers in which they do not) and did some hand­wav­ing about which class you con­sid­ered likely to be larger. There are three ways to dis­pute this: to dis­pute your hand­wav­ing (eg, you con­sider com­pres­sion of sub­jec­tive time to be a con­clu­sive ar­gu­ment, as if this is in­evitable), to pro­pose not-con­sid­ered classes of fu­ture (eg, tech­nol­ogy con­tinues to in­crease, but some im­mutable law of the uni­verse means that there are only a finite num­ber of apoc­a­lyp­tic tech­nolo­gies), or to main­tain that there are large classes of fu­ture which can­not pos­si­bly be imag­ined be­cause they do not clearly fall into any cat­e­gories such as we are likely to define in the pre­sent. If you use the lat­ter dis­pute, ar­gu­ing about prob­a­bil­ity is just ar­gu­ing about which un­in­for­ma­tive prior to use.

• I’m not pre­tend­ing this is an air­tight case. If you pre­vi­ously as­sumed that ex­is­ten­tial threats con­verge to zero as ra­tio­nal­ity in­creases; or that ra­tio­nal­ity is always the best policy; or that ra­tio­nal­ity means ex­pec­ta­tion max­i­miza­tion; and now you ques­tion one of those things; then you’ve got­ten some­thing out of it.

• ho­mung sug­gests that there may be im­mutable laws of the uni­verse that mean there are only a finite num­ber of apoc­a­lyp­tic tech­nolo­gies. Note that even if the prob­a­bil­ity of such tech­nolog­i­cal limits is small, in or­der for Phil’s ar­gu­ment to work, ei­ther that prob­a­bil­ity would have to be in­finites­i­mal, or some of the dooms­day de­vices have to con­tinue to be threat­en­ing af­ter the var­i­ous at­tack/​defense strate­gies reach a very ma­ture level of de­vel­op­ment. All of the prob­a­bil­ities look finite to me.

• No; that prob­a­bil­ity about a prop­erty of the uni­verse is a one-shot trial. It only has to be false once, out of one trial.

• So your the­sis is not that ra­tio­nal­ity dooms civ­i­liza­tion, but only that as far as we know, it might. I get it now.

• You talk as if you have pre­sented a cred­ible case—but you re­ally haven’t.

In­stead there is a fan­tasy about mak­ing black holes ex­plode (refer­ences, please!) an­other fan­tasy about sub­jec­tive time com­pres­sion out­strip­ping ex­pan­sion—and a story about dis­asters trig­ger­ing other dis­asters—which is true, but falls a long way short of a cred­ible ar­gu­ment that civil­i­sa­tion is likely to be wiped out once it has spread out a bit.

• In­stead there is a fan­tasy about mak­ing black holes ex­plode (refer­ences, please!)

You have me there. We have not yet suc­cess­fully deto­nated a black hole.

Small black holes are ex­pected to even­tu­ally ex­plode. Large black holes are ex­pected to take longer than the ex­pected life of the uni­verse to evap­o­rate to that point.

Any­way, I’m not a physi­cist. It’s just a hand­wavy ex­am­ple that maybe there is some tech­nol­ogy with so­lar-scale or galaxy-scale de­struc­tive power. When all the hu­mans lived on one is­land, they didn’t imag­ine they could one day de­stroy the Earth.

• Any­way, I’m not a physi­cist. It’s just a hand­wavy ex­am­ple that maybe there is some tech­nol­ogy with so­lar-scale or galaxy-scale de­struc­tive power.

Then the ex­am­ple is pointless. A weapon pow­er­ful enough to cause ex­tinc­tion galaxy wide is a very big if. It’s un­likely there would be, sim­ply be­cause of the mas­sive dis­tances be­tween stars.

Also, if you base your ar­gu­ment (or part of it, any­ways) on such an event, it is equally fair to state “if not”. And in the case of “if not” (which I imag­ine to be highly more likely), the ar­gu­ment must end there.

There­for, it is likely to as­sume that yes, we could out­run our own de­struc­tive ten­den­cies.

When all the hu­mans lived on one is­land, they didn’t imag­ine they could one day de­stroy the Earth.

At that point in our evolu­tion we had no firm grasp on what “world” even meant, let alone a ba­sic un­der­stand­ing of scale. Now, we do. We also have a ba­sic un­der­stand­ing of the uni­verse, and a method to in­crease our un­der­stand­ing (Abil­ity to pos­tu­late the­o­ries, run ex­per­i­ments and col­lect ev­i­dence). When all hu­mans (most likely an an­ces­tor) were con­tained in one ge­o­graphic co­or­di­nate, none of these things even ex­isted as con­cepts. There are a few more prob­lems with this com­par­i­son, but I’ll leave them alone for now, as it does noth­ing to bring them out.

• I wasn’t ask­ing for refer­ences sup­port­ing the idea that we had deto­nated a black hole. It’s an in­cred­ible weapon, which seems to have a low prob­a­bil­ity of ex­ist­ing—based on what we know about physics. The black hole at the cen­ter of our galaxy is not go­ing to go away any time soon.

Bizarre fu­ture spec­u­la­tions which defy the known laws of physics don’t add much to your case.