# Bey­ond Astro­nom­ical Waste

Faced with the as­tro­nom­ical amount of un­claimed and un­used re­sources in our uni­verse, one’s first re­ac­tion is prob­ably won­der­ment and an­ti­cip­a­tion, but a second re­ac­tion may be dis­ap­point­ment that our uni­verse isn’t even lar­ger or con­tains even more re­sources (such as the abil­ity to sup­port 3^^^3 hu­man life­times or per­haps to per­form an in­fin­ite amount of com­pu­ta­tion). In a pre­vi­ous post I sug­ges­ted that the po­ten­tial amount of as­tro­nom­ical waste in our uni­verse seems small enough that a total util­it­arian (or the total util­it­ari­an­ism part of someone’s moral un­cer­tainty) might reason that since one should have made a deal to trade away power/​re­sources/​in­flu­ence in this uni­verse for power/​re­sources/​in­flu­ence in uni­verses with much lar­ger amounts of avail­able re­sources, it would be ra­tional to be­have as if this deal was ac­tu­ally made. But for vari­ous reas­ons a total util­it­arian may not buy that ar­gu­ment, in which case an­other line of thought is to look for things to care about bey­ond the po­ten­tial as­tro­nom­ical waste in our uni­verse, in other words to ex­plore pos­sible sources of ex­pec­ted value that may be much greater than what can be gained by just cre­at­ing worth­while lives in this uni­verse.

One ex­ample of this is the pos­sib­il­ity of es­cap­ing, or be­ing de­lib­er­ately up­lif­ted from, a sim­u­la­tion that we’re in, into a much big­ger or richer base uni­verse. Or more gen­er­ally, the pos­sib­il­ity of con­trolling, through our de­cisions, the out­comes of uni­verses with much greater com­pu­ta­tional re­sources than the one we’re ap­par­ently in. It seems likely that un­der an as­sump­tion such as Teg­mark’s Mathem­at­ical Uni­verse Hy­po­thesis, there are many sim­u­la­tions of our uni­verse run­ning all over the mul­ti­verse, in­clud­ing in uni­verses that are much richer than ours in com­pu­ta­tional re­sources. If such sim­u­la­tions ex­ist, it also seems likely that we can leave some of them, for ex­ample through one of these mech­an­isms:

1. Ex­ploit­ing a flaw in the soft­ware or hard­ware of the com­puter that is run­ning our sim­u­la­tion (in­clud­ing “nat­ural sim­u­la­tions” where a very large uni­verse hap­pens to con­tain a sim­u­la­tion of ours without any­one in­tend­ing this).

2. Ex­ploit­ing a flaw in the psy­cho­logy of agents run­ning the sim­u­la­tion.

3. Al­tru­ism (or other moral/​axi­olo­gical con­sid­er­a­tions) on the part of the sim­u­lat­ors.

4. Other in­stru­mental reas­ons for the sim­u­lat­ors to let out sim­u­lated be­ings, such as want­ing someone to talk to or play with. (Paul Chris­ti­ano’s re­cent When is un­aligned AI mor­ally valu­able? con­tains an ex­ample of this, how­ever the idea there only lets us es­cape to an­other uni­verse sim­ilar to this one.)

(Be­ing run as a sim­u­la­tion in an­other uni­verse isn’t ne­ces­sar­ily the only way to con­trol what hap­pens in that uni­verse. Another pos­sib­il­ity is if uni­verses with halt­ing or­acles ex­ist (which is im­plied by Teg­mark’s MUH since they ex­ist as math­em­at­ical struc­tures in the arith­met­ical hier­archy), some of their or­acle quer­ies may be ques­tions whose an­swers can be con­trolled by our de­cisions, in which case we can con­trol what hap­pens in those uni­verses without be­ing sim­u­lated by them (in the sense of be­ing run step by step in a com­puter). Another ex­ample is that su­per­in­tel­li­gent be­ings may be able to reason about what our de­cisions are without hav­ing to run a step by step sim­u­la­tion of us, even without ac­cess to a halt­ing or­acle.)

The gen­eral idea here is for a su­per­in­tel­li­gence des­cend­ing from us to (after de­term­in­ing that this is an ad­vis­able course of ac­tion) use some frac­tion of the re­sources of this uni­verse to reason about or search (com­pu­ta­tion­ally) for much big­ger/​richer uni­verses that are run­ning us as sim­u­la­tions or can oth­er­wise be con­trolled by us, and then de­term­ine what we need to do to max­im­ize the ex­pec­ted value of the con­sequences of our ac­tions on the base uni­verses, per­haps through one or more of the above lis­ted mech­an­isms.

### Practical Implications

Real­iz­ing this kind of ex­ist­en­tial hope seems to re­quire a higher level of philo­soph­ical soph­ist­ic­a­tion than just pre­vent­ing as­tro­nom­ical waste in our own uni­verse. Com­pared to that prob­lem, here we have more ques­tions of a philo­soph­ical nature, for which no em­pir­ical feed­back seems pos­sible. It seems very easy to make a mis­take some­where along the chain of reas­on­ing and waste a more-than-as­tro­nom­ical amount of po­ten­tial value, for ex­ample by fail­ing to real­ize the pos­sib­il­ity of af­fect­ing big­ger uni­verses through our ac­tions, in­cor­rectly cal­cu­lat­ing the ex­pec­ted value of such a strategy, fail­ing to solve the dis­tri­bu­tional/​on­to­lo­gical shift prob­lem of how to value strange and un­fa­mil­iar pro­cesses or en­tit­ies in other uni­verses, fail­ing to fig­ure out the cor­rect or op­timal way to es­cape into or oth­er­wise in­flu­ence lar­ger uni­verses, etc.

The total util­it­arian in me is thus very con­cerned about try­ing to pre­serve and im­prove the col­lect­ive philo­soph­ical com­pet­ence of our civil­iz­a­tion, such that when it be­comes pos­sible to pur­sue strategies like ones lis­ted above, we’ll be able to make the right de­cisions. The best op­por­tun­ity to do this that I can fore­see is the ad­vent of ad­vanced AI, which is an­other reason I want to push for AIs that are not just value aligned with us, but also have philo­soph­ical com­pet­ence that scales with their other in­tel­lec­tual abil­it­ies, so they can help cor­rect the philo­soph­ical er­rors of their hu­man users (in­stead of merely de­fer­ring to them), thereby greatly im­prov­ing our col­lect­ive philo­soph­ical com­pet­ence.

### Anti­cip­ated Questions

How is this idea re­lated to Nick Bostrom’s Sim­u­la­tion Ar­gu­ment? Nick’s ar­gu­ment fo­cuses on the pos­sib­il­ity of post-hu­mans (pre­sum­ably liv­ing in a uni­verse sim­ilar to ours but just at a later date) sim­u­lat­ing us as their an­cest­ors. It does not seem to con­sider that we may be run­ning as sim­u­la­tions in much lar­ger/​richer uni­verses, or that this may be a source of great po­ten­tial value.

Isn’t this a form of Pas­cal’s Mug­ging? I’m not sure. It could be that when we fig­ure out how to solve Pas­cal’s Mug­ging it will be­come clear that we shouldn’t try to leave our sim­u­la­tion for reas­ons sim­ilar to why we shouldn’t pay the mug­ger. However the ana­logy doesn’t seem so tight that I think this is highly likely. Also, note that the ar­gu­ment here isn’t that we should do the equi­val­ent of “pay the mug­ger” but rather that we should try to bring ourselves into a po­s­i­tion where we can defin­it­ively fig­ure out what the right thing to do is.

• Be­ing run as a sim­u­la­tion in an­other uni­verse isn’t ne­ces­sar­ily the only way to con­trol what hap­pens in that uni­verse.

Had you seen Mul­ti­verse-wide co­oper­a­tion via cor­rel­ated de­cision-mak­ing, btw? (some­what sim­ilar to acausal trade but dif­fers in that it’s based on the agents be­ing sim­ilar to each other rather than mod­el­ing each other)

• Yes, I saw the MSR pa­per when it came out, but I think it’s covered as a spe­cial case of acausal trade. The wiki art­icle on acausal trade (which I be­lieve was writ­ten by Joshua Fox sev­eral years be­fore the MSR pa­per was pub­lished) men­tions it as item 2 un­der “Pre­dic­tion mech­an­isms”.

• If there’s some kind of meas­ure of “ob­server weight” over the whole math­em­at­ical uni­verse, we might be already much lar­ger than 1/​3^^^3 of it, so the total util­it­arian can only gain so much. And even if there’s no meas­ure, I’m not sure my total util­it­ari­an­ism would scale lin­early to such num­bers. But I’m very con­fused about all this.

We can di­vide our cre­dence in total util­it­ari­an­ism into “bounded total util­it­ari­an­ism” (in­clud­ing meas­ure-based) and “un­boun­ded total util­it­ari­an­ism”. Condi­tional on bounded total util­it­ari­an­ism, I don’t see a reason to think that po­ten­tial value gained from con­trolling lar­ger/​richer uni­verses couldn’t be at least sev­eral or­ders of mag­nitude lar­ger than from what hap­pens in this uni­verse. (Maybe this is true for some forms of bounded total util­it­ari­an­ism with par­tic­u­larly low bounds, but shouldn’t be true for all of them.) Condi­tional on un­boun­ded total util­it­ari­an­ism, things are even more con­fus­ing as it’s not clear how un­boun­ded total util­it­ari­an­ism can form­ally work, but in­form­ally it seems that if un­boun­ded total util­it­ari­an­ism can work it very likely say that try­ing to con­trol lar­ger/​richer uni­verses is the right thing to do.

Over­all it seems like a fairly safe con­clu­sion that the part of you that is at­trac­ted by the idea of pre­vent­ing as­tro­nom­ical waste (or a large frac­tion of that part of you) prob­ably shouldn’t stop at just pre­vent­ing as­tro­nom­ical waste in this uni­verse.

• Yeah, I agree the gain can be or­ders of mag­nitude lar­ger than this uni­verse. Only ob­ject­ing to the use of 3^^^3 as a meta­phor, be­cause I’m not sure we can care that strongly.

My in­stinct says we can’t care about any­thing much big­ger than an ex­po­nen­tial. That’s also use­ful for pre­vent­ing Pas­cal’s mug­gings, be­cause I can re­peatedly flip a coin and ask the mug­ger to in­flu­ence the out­come, thus re­du­cing their cred­ib­il­ity ex­po­nen­tially with time. But maybe that’s too con­veni­ent.

• I don’t un­der­stand why you think that the ex­pect­a­tion should be or­ders of mag­nitude lar­ger for other uni­verses. The model “like util­it­ari­an­ism, but with an up­per bound on # of people” seems kind of wacky, maybe it gets a seat in the moral par­lia­ment but I don’t think it’s the dom­in­ant force for caring about as­tro­nom­ical waste. For non-count­ing-meas­ure util­it­ari­an­ism, I don’t see either why the mod­els con­cerned about as­tro­nom­ical waste would as­sign lar­ger uni­verses an over­whelm­ing share of our caring-meas­ure.

It also feels to me like you are 2-en­vel­op­ing wrong if you end up with a 100x ra­tio here. (I.e., if you have 10% prob­ab­il­ity on a model where there two are equal, I don’t think you should end up with 100x.)

Over­all it seems like a fairly safe con­clu­sion that the part of you that is at­trac­ted by the idea of pre­vent­ing as­tro­nom­ical waste (or a large frac­tion of that part of you) prob­ably shouldn’t stop at just pre­vent­ing as­tro­nom­ical waste in this uni­verse.

I you put 50% on a the­ory that cares over­whelm­ingly about in­fin­ite uni­verses and 50% on a the­ory that cares about all uni­verses, the thing to do is prob­ably still to pre­vent as­tro­nom­ical waste in this uni­verse, so that we can later en­gage in trade or spend the re­sources ex­plor­ing whatever angles of at­tack seem use­ful. Maybe this is the kind of thing you have in mind, but it’s a not­able spe­cial case be­cause it seems to re­com­mend the same short-term be­ha­vior.

try­ing to pre­serve and im­prove the col­lect­ive philo­soph­ical com­pet­ence of our civil­iz­a­tion, such that when it be­comes pos­sible to pur­sue strategies like ones lis­ted above, we’ll be able to make the right de­cisions.

I agree that if we don’t even­tu­ally reach philo­soph­ical ma­tur­ity (or end up on an ap­prox­im­ately op­timal philo­soph­ical tra­ject­ory) then we won’t cap­ture most of the value in the uni­verse. It seems like that con­clu­sion doesn’t really de­pend on in­fin­ite uni­verses though (e.g. a util­it­arian might be sim­il­arly con­cerned about dis­cov­er­ing how to op­tim­ally or­gan­ize mat­ter), un­less you think this is the main way our pref­er­ences might not be eas­ily sa­ti­able.

The best op­por­tun­ity to do this that I can fore­see is the ad­vent of ad­vanced AI, which is an­other reason I want to push for AIs that are not just value aligned with us, but also have philo­soph­ical com­pet­ence that scales with their other in­tel­lec­tual abil­it­ies, so they can help cor­rect the philo­soph­ical er­rors of their hu­man users (in­stead of merely de­fer­ring to them), thereby greatly im­prov­ing our col­lect­ive philo­soph­ical com­pet­ence.

This doesn’t seem re­lated to re­cent dis­cus­sions about philo­soph­ical com­pet­ence and AI, since itis about wha we want AI to do even­tu­ally rather than what you want to do in the 21 cen­tury (I’m not sure if it was sup­posed to be re­lated).

• For non-count­ing-meas­ure util­it­ari­an­ism, I don’t see either why the mod­els con­cerned about as­tro­nom­ical waste would as­sign lar­ger uni­verses an over­whelm­ing share of our caring-meas­ure.

I guess with meas­ure-based util­it­ari­an­ism, it’s more about dens­ity of po­ten­tially valu­able things within the uni­verse than size. If our uni­verse only sup­ports 10^120 avail­able op­er­a­tions, most of it (>99%) is go­ing to be devoid of value un­der many eth­ic­ally plaus­ible ways of dis­trib­ut­ing caring-meas­ure over the space-time re­gions within a uni­verse.

I agree that if we don’t even­tu­ally reach philo­soph­ical ma­tur­ity (or end up on an ap­prox­im­ately op­timal philo­soph­ical tra­ject­ory) then we won’t cap­ture most of the value in the uni­verse. It seems like that con­clu­sion doesn’t really de­pend on in­fin­ite uni­verses though (e.g. a util­it­arian might be sim­il­arly con­cerned about dis­cov­er­ing how to op­tim­ally or­gan­ize mat­ter),

Some people seem to think there’s a good chance that our cur­rent level of philo­soph­ical un­der­stand­ing is enough to cap­ture most of the value in this uni­verse. (For ex­ample, if we im­ple­ment a uni­verse-wide sim­u­la­tion de­signed ac­cord­ing to Eliezer’s Fun The­ory, or if we just wipe out all suf­fer­ing.) Oth­ers may think that we don’t cur­rently have enough un­der­stand­ing to do that, but we can reach that level of un­der­stand­ing “by de­fault”. My ar­gu­ment here is that both of these seem less likely if the goal is in­stead to cap­ture value from lar­ger/​richer uni­verses, and that gives more im­petus to try­ing to im­prove our philo­soph­ical com­pet­ence.

un­less you think this is the main way our pref­er­ences might not be eas­ily sa­ti­able.

Not sure what you mean by this.

This doesn’t seem re­lated to re­cent dis­cus­sions about philo­soph­ical com­pet­ence and AI, since it is about what we want AI to do even­tu­ally rather than what you want to do in the 21 cen­tury (I’m not sure if it was sup­posed to be re­lated).

They’re not sup­posed to be re­lated ex­cept in so far as they’re both ar­gu­ments for want­ing AI to be able to help hu­mans cor­rect their philo­soph­ical mis­takes in­stead of just de­fer­ring to hu­mans.

• I guess with meas­ure-based util­it­ari­an­ism, it’s more about dens­ity of po­ten­tially valu­able things within the uni­verse than size. If our uni­verse only sup­ports 10^120 avail­able op­er­a­tions, most of it (>99%) is go­ing to be devoid of value un­der many eth­ic­ally plaus­ible ways of dis­trib­ut­ing caring-meas­ure over the space-time re­gions within a uni­verse.

I agree, but if you have a broad dis­tri­bu­tion over mix­tures then you’ll be in­clud­ing many that don’t use lit­eral loc­a­tions and those will dom­in­ate for “sparse” uni­verses.

I can see eas­ily how you’d get a mod­est factor fa­vor­ing other uni­verses over as­tro­nom­ical waste in this uni­verse, but as your meas­ure/​un­cer­tainty gets broader (or you have a broader dis­tri­bu­tion over trad­ing part­ners) the ra­tio seems to shrink to­wards 1 and I don’t feel like “or­ders of mag­nitude” is that plaus­ible.

Some people seem to think there’s a good chance that our cur­rent level of philo­soph­ical un­der­stand­ing is enough to cap­ture most of the value in this uni­verse. (For ex­ample, if we im­ple­ment a uni­verse-wide sim­u­la­tion de­signed ac­cord­ing to Eliezer’s Fun The­ory, or if we just wipe out all suf­fer­ing.) Oth­ers may think that we don’t cur­rently have enough un­der­stand­ing to do that, but we can reach that level of un­der­stand­ing “by de­fault”. My ar­gu­ment here is that both of these seem less likely if the goal is in­stead to cap­ture value from lar­ger/​richer uni­verses, and that gives more im­petus to try­ing to im­prove our philo­soph­ical com­pet­ence.

I agree this is a fur­ther ar­gu­ment for need­ing more philo­soph­ical com­pet­ence. I per­son­ally feel like that po­s­i­tion is already pretty solid but I ac­know­ledge that it’s not a uni­ver­sal po­s­i­tion even amongst EAs.

They’re not sup­posed to be re­lated ex­cept in so far as they’re both ar­gu­ments for want­ing AI to be able to help hu­mans cor­rect their philo­soph­ical mis­takes in­stead of just de­fer­ring to hu­mans.

“De­fer to hu­mans” could mean many dif­fer­ent things. This is an ar­gu­ment against AI forever de­fer­ring to hu­mans in their cur­rent form /​ with their cur­rent know­ledge. When I talk about “de­fer to hu­mans” I’m usu­ally talk­ing about an AI de­fer­ring to hu­mans who are ex­pli­citly al­lowed to de­lib­er­ate/​learn/​self-modify if that’s what they choose to do (or, per­haps more im­port­antly, to con­struct a new AI with greater philo­soph­ical com­pet­ence and put it in charge).

I un­der­stand that some people might ad­voc­ate for a stronger form of “de­fer to hu­mans” and it’s fine to re­spond to them, but wanted to make sure there wasn’t a mis­un­der­stand­ing. (Also I don’t feel there are very many ad­voc­ates for the stronger form, I think the bulk of the AI com­munity ima­gines our AI de­fer­ring to us but us be­ing free to design bet­ter AIs later.)

• I agree, but if you have a broad dis­tri­bu­tion over mix­tures then you’ll be in­clud­ing many that don’t use lit­eral loc­a­tions and those will dom­in­ate for “sparse” uni­verses.

I cur­rently think that each way of dis­trib­ut­ing caring-meas­ure over a uni­verse should be a sep­ar­ate mem­ber of moral par­lia­ment, given a weight equal to its eth­ical plaus­ib­il­ity, in­stead of hav­ing just one mem­ber with some sort of uni­ver­sal dis­tri­bu­tion. So there ought to be a sub­stan­tial co­ali­tion in one’s moral par­lia­ment that think con­trolling big­ger/​richer uni­verses is po­ten­tially or­ders of mag­nitude more valu­able.

Another in­tu­ition pump here is to con­sider a thought ex­per­i­ment where you think there’s 5050 chance that our uni­verse sup­ports either 10^120 op­er­a­tions or 10^(10^120) op­er­a­tions (and con­trolling other uni­verses isn’t pos­sible). Isn’t there some large co­ali­tion of total util­it­ari­ans in your moral par­lia­ment who would be at least 100x hap­pier to find out that the uni­verse sup­ports 10^(10^120) op­er­a­tions (and be will­ing to bet/​trade ac­cord­ingly)?

When I talk about “de­fer to hu­mans” I’m usu­ally talk­ing about an AI de­fer­ring to hu­mans who are ex­pli­citly al­lowed to de­lib­er­ate/​​learn/​​self-modify if that’s what they choose to do (or, per­haps more im­port­antly, to con­struct a new AI with greater philo­soph­ical com­pet­ence and put it in charge).

Yeah I didn’t make this clear, but my worry here is that most hu­mans won’t choose to “de­lib­er­ate/​​learn/​​self-modify” in a way that leads to philo­soph­ical ma­tur­ity (or con­struct a new AI with greater philo­soph­ical com­pet­ence and put it in charge), if you ini­tially give them an AI that has great in­tel­lec­tual abil­it­ies in most areas but de­fers to hu­mans on philo­soph­ical mat­ters. One pos­sib­il­ity is that be­cause hu­mans don’t have value func­tions that are ro­bust against dis­tri­bu­tional shifts, they’ll (with the help of their AIs) end up do­ing an ad­versarial at­tack against their own value func­tions and not be able to re­cover from that. If they some­how avoid that, they may still get stuck at some level of philo­soph­ical com­pet­ence that is less than what’s needed to cap­ture value from big­ger/​richer uni­verses, and never feel a need to put a new philo­soph­ic­ally com­pet­ent AI in charge. It seems to me that the best way to avoid both of these out­comes (as well as pos­sible near-term moral cata­strophes such as cre­at­ing a lot of suf­fer­ing that can’t be bal­anced out later) is to make sure that the first ad­vanced AIs are highly or scal­ably com­pet­ent in philo­sophy. (I un­der­stand you prob­ably dis­agree with “get­ting stuck” even with re­gard to cap­tur­ing value from big­ger/​richer uni­verses, you’re not very con­cerned about near term moral cata­strophes, and I’m not sure what your think­ing on the un­re­cov­er­able self-at­tack thing is.)

• Another in­tu­ition pump here is to con­sider a thought ex­per­i­ment where you think there’s 5050 chance that our uni­verse sup­ports either 10^120 op­er­a­tions or 10^(10^120) op­er­a­tions (and con­trolling other uni­verses isn’t pos­sible). Isn’t there some large co­ali­tion of total util­it­ari­ans in your moral par­lia­ment who would be at least 100x hap­pier to find out that the uni­verse sup­ports 10^(10^120) op­er­a­tions (and be will­ing to bet/​trade ac­cord­ingly)?

I totally agree that there are mem­bers of the par­lia­ment who would as­sign much higher value on other uni­verses than on our uni­verse.

I’m say­ing that there is also a sig­ni­fic­ant con­tin­gent that cares about our uni­verse, so the people who care about other uni­verses aren’t go­ing to dom­in­ate.

(And on top of that, all of the con­tin­gents are roughly just try­ing to max­im­ize the “mar­ket value” of what we get, so for the most part we need to reason about an even more spread out dis­tri­bu­tion.)

Yeah I didn’t make this clear, but my worry here is that most hu­mans won’t choose to “de­lib­er­ate/​​learn/​​self-modify” in a way that leads to philo­soph­ical ma­tur­ity (or con­struct a new AI with greater philo­soph­ical com­pet­ence and put it in charge), if you ini­tially give them an AI that has great in­tel­lec­tual abil­it­ies in most areas but de­fers to hu­mans on philo­soph­ical mat­ters.

There are tons of ways you could get people to do some­thing they won’t choose to do. I don’t know if “give them an AI that doesn’t de­fer to them about philo­sophy” is more nat­ural than e.g. “give them an AI that doesn’t de­fer to them about how they should de­lib­er­ate/​learn/​self-modify.”

• I’m say­ing that there is also a sig­ni­fic­ant con­tin­gent that cares about our uni­verse, so the people who care about other uni­verses aren’t go­ing to dom­in­ate.

I don’t think I’m get­ting your point here. Per­son­ally it seems safe to say that >80% of the con­tin­gent of my moral par­lia­ment that cares about as­tro­nom­ical waste would say that if our uni­verse was cap­able of 10^(10^120) op­er­a­tions it would be at least 100x as valu­able as if was cap­able of only 10^120 op­er­a­tions. Are your num­bers dif­fer­ent from this? In any case, what im­plic­a­tions are you sug­gest­ing based on “no dom­in­a­tion”?

(And on top of that, all of the con­tin­gents are roughly just try­ing to max­im­ize the “mar­ket value” of what we get, so for the most part we need to reason about an even more spread out dis­tri­bu­tion.)

I don’t un­der­stand this part at all. Please elab­or­ate?

There are tons of ways you could get people to do some­thing they won’t choose to do.

I did pre­face my con­clu­sion with “The best op­por­tun­ity to do this that I can fore­see”, so if you have other ideas about what someone like me ought to do, I’d cer­tainly wel­come them.

I don’t know if “give them an AI that doesn’t de­fer to them about philo­sophy” is more nat­ural than e.g. “give them an AI that doesn’t de­fer to them about how they should de­lib­er­ate/​​learn/​​self-modify.”

Isn’t “how they should de­lib­er­ate/​​learn/​​self-modify” it­self a dif­fi­cult philo­soph­ical prob­lem (in the field of meta-philo­sophy)? If it’s some­how easier or safer to “give them an AI that doesn’t de­fer to them about how they should de­lib­er­ate/​​learn/​​self-modify” than to “give them an AI that doesn’t de­fer to them about philo­sophy” then I’m all for that but it doesn’t seem like a very dif­fer­ent idea from mine.

• I don’t think I’m get­ting your point here. Per­son­ally it seems safe to say that >80% of the con­tin­gent of my moral par­lia­ment that cares about as­tro­nom­ical waste would say that if our uni­verse was cap­able of 10^(10^120) op­er­a­tions it would be at least 100x as valu­able as if was cap­able of only 10^120 op­er­a­tions. Are your num­bers dif­fer­ent from this? In any case, what im­plic­a­tions are you sug­gest­ing based on “no dom­in­a­tion”?

I might have given 50% or 60% in­stead of >80%.

I don’t un­der­stand how you would get sig­ni­fic­ant con­clu­sions out of this without big mul­ti­pli­ers. Yes, there are some par­ti­cipants in your par­lia­ment who care more about worlds other than this one. Those worlds ap­pear to be sig­ni­fic­antly harder to in­flu­ence (by means other than trade), so this doesn’t seem to have a huge ef­fect on what you ought to do in this world. (As­sum­ing that we are able to make trades that we ob­vi­ously would have wanted to make be­hind the veil of ig­nor­ance.)

In par­tic­u­lar, if your ra­tio between the value of big and small uni­verses was only 5x, then that would only have a 5x mul­ti­plier on the value of the in­ter­ven­tions you list in the OP. Given that many of them look very tiny, I as­sumed you were ima­gin­ing a much lar­ger mul­ti­plier. (So­mething that looks very tiny may end up be­ing a huge deal, but once we are already wrong by many or­ders of mag­nitude it doesn’t feel like the last 5x has a huge im­pact.)

I don’t un­der­stand this part at all. Please elab­or­ate?

We will have con­trol over as­tro­nom­ical re­sources in our uni­verse. We can then acaus­ally trade that away for in­flu­ence over the kinds of uni­verses we care about in­flu­en­cing. At equi­lib­rium, ig­nor­ing mar­ket fail­ures and fric­tion, how much you value get­ting con­trol over as­tro­nom­ical re­sources doesn’t de­pend on which kinds of as­tro­nom­ical re­sources you in par­tic­u­lar ter­min­ally value. Every­one in­stru­ment­ally uses the same util­ity func­tion, given by the mar­ket-clear­ing prices of dif­fer­ent kinds of as­tro­nom­ical re­sources. In par­tic­u­lar, the op­timal ra­tio between (say) he­don­ism and tak­ing-over-the-uni­verse de­pends on the mar­ket price of the uni­verse you live in, not on how much you in par­tic­u­lar value the uni­verse you live in. This is ex­actly ana­log­ous to say­ing: the op­timal tradeoff between work and leis­ure de­pends only the mar­ket price of the out­put of your work (ig­nor­ing fric­tion and mar­ket fail­ures), not on how much you in par­tic­u­lar value the out­put of your work.

So the up­shot is that in­stead of us­ing your moral par­lia­ment to set prices, you want to be us­ing a broader dis­tri­bu­tion over all of the people who con­trol as­tro­nom­ical re­sources (weighted by the mar­ket prices of their re­sources). Our pref­er­ences are still evid­ence about what oth­ers want, but this just tends to make the dis­tri­bu­tion more spread out (and there­fore cuts against e.g. caring much less about col­on­iz­ing small uni­verses).

Isn’t “how they should de­lib­er­ate/​​learn/​​self-modify” it­self a dif­fi­cult philo­soph­ical prob­lem (in the field of meta-philo­sophy)? If it’s some­how easier or safer to “give them an AI that doesn’t de­fer to them about how they should de­lib­er­ate/​​learn/​​self-modify” than to “give them an AI that doesn’t de­fer to them about philo­sophy” then I’m all for that but it doesn’t seem like a very dif­fer­ent idea from mine.r

I still don’t really get your po­s­i­tion, and es­pe­cially why you think:

It seems to me that the best way to avoid both of these out­comes [...] is to make sure that the first ad­vanced AIs are highly or scal­ably com­pet­ent in philo­sophy.

I do un­der­stand why you think it’s an im­port­ant way to avoid philo­soph­ical er­rors in the short-term, in that case I just don’t see why you think that such prob­lems are im­port­ant re­l­at­ive to other factors that af­fect the qual­ity of the fu­ture.

This seems to come up a lot in our dis­cus­sions. It would be use­ful if you could make a clear state­ment of why you think this prob­lem (which I un­der­stand as: “en­sure early AI is highly philo­soph­ic­ally com­pet­ent” or per­haps “dif­fer­en­tial philo­soph­ical pro­gress,” set­ting aside the ap­plic­a­tion of philo­soph­ical com­pet­ence to what-I’m-call­ing-align­ment) is im­port­ant, ideally with some kind of quant­it­at­ive pic­ture of how im­port­ant you think it is. If you ex­pect to write that up at some point then I’ll just pause un­til then.

• I don’t un­der­stand how you would get sig­ni­fic­ant con­clu­sions out of this without big mul­ti­pli­ers. Yes, there are some par­ti­cipants in your par­lia­ment who care more about worlds other than this one. Those worlds ap­pear to be sig­ni­fic­antly harder to in­flu­ence (by means other than trade), so this doesn’t seem to have a huge ef­fect on what you ought to do in this world. (As­sum­ing that we are able to make trades that we ob­vi­ously would have wanted to make be­hind the veil of ig­nor­ance.)

Wait, you are as­sum­ing a baseline/​de­fault out­come where acausal trade takes place, and com­par­ing other in­ter­ven­tions to that? My baseline for com­par­ison is in­stead (as stated in the OP) “what can be gained by just cre­at­ing worth­while lives in this uni­verse”. My reas­ons for this are (1) I (and likely oth­ers who might read this) don’t think acausal trade is much more likely to work than the other items on my list and (2) the main in­ten­ded audi­ence for this post is people who have real­ized the im­port­ance of in­flu­en­cing the far fu­ture but not aware of (or have ser­i­ously con­sidered) the pos­sib­il­ity of in­flu­en­cing other uni­verses through things like acausal trade and other items on my list. Even the most soph­ist­ic­ated thinkers in EA seem to fall into this cat­egory, e.g., people like Will MacAskill, Toby Ord, and Nick Beck­stead, un­less they’ve privately con­sidered the pos­sib­il­ity and chose not to talk about it in pub­lic, in which case it still seems safe to as­sume that most people in EA think “cre­at­ing worth­while lives in this uni­verse” is the most good that can be ac­com­plished.

In par­tic­u­lar, if your ra­tio between the value of big and small uni­verses was only 5x, then that would only have a 5x mul­ti­plier on the value of the in­ter­ven­tions you list in the OP. Given that many of them look very tiny, I as­sumed you were ima­gin­ing a much lar­ger mul­ti­plier. (So­mething that looks very tiny may end up be­ing a huge deal, but once we are already wrong by many or­ders of mag­nitude it doesn’t feel like the last 5x has a huge im­pact.)

I don’t un­der­stand where “5x” comes from or why that’s the rel­ev­ant mul­ti­plier in­stead of 100x.

It would be use­ful if you could make a clear state­ment of why you think this prob­lem is important

I’ll think about this, but I think I’d be more mo­tiv­ated to at­tempt this (and maybe also have a bet­ter idea of what I need to do) if other people also spoke up and told me that they couldn’t un­der­stand my past at­tempts to ex­plain this (in­clud­ing what I wrote in the OP and pre­vi­ous com­ments in this thread).

• If there’s some kind of meas­ure of “ob­server weight” over the whole math­em­at­ical uni­verse, we might be already much lar­ger than 1/​3^^^3 of it, so the total util­it­arian can only gain so much.

Could you provide some in­tu­ition for this? Naively, I’d ex­pect our “ob­server meas­ure” over the space of math­em­at­ical struc­tures to be 0.

• I cur­ated this post be­cause it crys­tal­lised an im­port­ant point re­gard­ing op­tim­ising the long-term fu­ture, that I’ve not seen any­one write down suc­cinctly be­fore (with ref­er­ence to the rel­ev­ant tech­nical con­cepts, while still be­ing short and read­able).

• With quantum branch­ing, our uni­verse could have some num­ber like a goo­golplex of stuff, maybe more. And philo­soph­ic­ally, you’re wor­ried about the dif­fer­ence between that and 3^^^3? I get that there’s a big gap there but I’d guess it’s one that we’re defin­i­tion­ally un­able to do use­ful moral reas­on­ing about.

• I feel like scope in­sens­it­iv­ity is some­thing to worry about here. I’d be really happy to learn that hu­man­ity will man­age to take good care of our cos­mic en­dow­ment but my hap­pi­ness wouldn’t scale prop­erly with the amount of value at stake if I learned we took good care of a su­per-cos­mic en­dow­ment. I think that’s the res­ult of my in­ab­il­ity to grasp the quant­it­ies in­volved rather than a true re­flec­tion of my ex­tra­pol­ated val­ues, how­ever.

My con­cern is more that reas­on­ing about en­tit­ies in sim­pler uni­verses cap­able of con­duct­ing acausal trades with us will turn out to be totally in­tract­able (as will the other pro­posed es­cape meth­ods), but since I’m very un­cer­tain about that I think it’s def­in­itely worth fur­ther in­vest­ig­a­tion. I’m also not con­vinced Teg­mark’s MUH is true in the first place, but this post is mak­ing me want to do more read­ing on the ar­gu­ments in fa­vor & op­posed. It looks like there was a Ra­tion­ally Speak­ing epis­ode about it?

• When you’re faced with num­bers like 3^^^3, scope in­sens­it­iv­ity is the cor­rect re­sponse. A goo­golplex is already enough to hold every pos­sible con­fig­ur­a­tion of Life as we know it. “Ham­let, but with ex­tra com­mas in these three places, per­formed by in­tel­li­gent star­fish” is in there some­where in over a goo­gol dif­fer­ent vari­et­ies. What, then, does 3^^^3 add ex­cept more cop­ies of the same?

• Noth­ing, if your defin­i­tion of a copy is suf­fi­ciently gen­eral :-)

Am I un­der­stand­ing you right that you be­lieve in some­thing like a com­pu­ta­tional the­ory of iden­tity and think there’s some sort of bound on how com­plex some­thing we’d at­trib­ute moral pa­tient­hood or in­ter­est­ing­ness to can get? I agree with the former, but don’t see much reason for be­liev­ing the lat­ter.

• I have no idea if there is such a bound. I will never have any idea if there is such a bound, and I sus­pect that neither will any en­tity in this uni­verse. Given that fact, I’d rather make the as­sump­tion that doesn’t turn me stu­pid when Pas­cal’s Wager comes up.

• I just real­ised that the prob­lem of the lim­ited size of the Uni­verse is iso­morphic to the prob­lem of how to sur­vive the end of the uni­verse, which I ana­lysed here, but the es­cape routs de­scribed by OP are dif­fer­ent, and more rely on acausal trade and sim­u­la­tion hack­ing than on physic ma­nip­u­la­tion.

• What of ex­po­nen­tial total util­it­ari­an­ism? That’s a total util­it­ari­an­ism that mul­ti­plies the total util­ity by the ex­po­nen­tial of the pop­u­la­tion. It may be very un­likely, but as pop­u­la­tion grows, it will even­tu­ally come to dom­in­ate.

That’s why I think moral the­or­ies should be nor­m­al­ised in­de­pend­ently, to pre­vent the su­per-pop­u­la­tion ones from win­ning just by de­fault.

• That’s why I think moral the­or­ies should be nor­m­al­ised in­de­pend­ently, to pre­vent the su­per-pop­u­la­tion ones from win­ning just by de­fault.

I’m as­sum­ing this as well. Did I give a dif­fer­ent im­pres­sion in the post? If so I’ll try to cla­rify.

• Normally when I nor­m­al­ise, I use the ex­pec­ted max­imum of the util­ity func­tion if we just max­im­ised it and noth­ing else: ht­tps://​​www.less­wrong.com/​​posts/​​hBJCMWELaW6Mx­inYW/​​in­ter­the­or­etic-util­ity-comparison

There­fore if total util­it­ari­an­ism is not heav­ily weighted, it will likely re­main un­im­port­ant; your phras­ing “or someone whose moral un­cer­tainty in­cludes total util­it­ari­an­ism” sug­ges­ted to me that you thought total util­it­ari­an­ism would be im­port­ant even if as­signed a low weight, which sug­ges­ted that it was not be­ing nor­m­al­ised.

• your phras­ing “or someone whose moral un­cer­tainty in­cludes total util­it­ari­an­ism” sug­ges­ted to me that you thought total util­it­ari­an­ism would be im­port­ant even if as­signed a low weight, which sug­ges­ted that it was not be­ing nor­m­al­ised.

Ok, I didn’t mean that. What I meant was that if your moral un­cer­tainty in­cludes total util­it­ari­an­ism, then the total util­it­arian part should reason as fol­lows. Would it be clearer /​ clear enough if I re­placed “or someone whose moral un­cer­tainty in­cludes total util­it­ari­an­ism” with “or the total util­it­ari­an­ism part of someone’s moral un­cer­tainty”?

• I think that would be clearer, yes.

• Thanks, I’ve made that edit.

• MUH doesn’t im­ply the ex­ist­ence of halt­ing or­acles. Indeed, the Com­put­able Uni­verse Hy­po­thesis is sup­posed to be an ex­ten­sion of the Mathem­at­ical Uni­verse Hy­po­thesis, but CUH says that halt­ing or­acles do not ex­ist.

• There may be sev­eral con­fu­sions hap­pen­ing here. First I’ve been us­ing MUH to mean “ul­ti­mate en­semble the­ory” (i.e., the idea that the Level IV mul­ti­verse of all math­em­at­ical struc­tures ex­ists), be­cause Wiki­pe­dia says MUH is “also known as the ul­ti­mate en­semble the­ory”. But Teg­mark cur­rently defines MUH as “Our ex­ternal phys­ical real­ity is a math­em­at­ical struc­ture” which seems to be talk­ing just about our par­tic­u­lar uni­verse and not say­ing that all math­em­at­ical struc­tures ex­ist. Se­cond if by “MUH doesn’t im­ply the ex­ist­ence of halt­ing or­acles” you mean that MUH doesn’t ne­ces­sar­ily im­ply the ex­ist­ence of halt­ing or­acles in our uni­verse, then I agree. What I meant in the OP is that the ul­ti­mate en­semble the­ory im­plies that uni­verses con­tain­ing halt­ing or­acles ex­ist in the Level IV mul­ti­verse.

Hope­fully that cla­ri­fies things?

• Isn’t all this massively de­pend­ent on how your util­ity $U$ scales with the total num­ber $N$ of well-spent com­pu­ta­tions (e.g. one-bit com­putes)?

That is, I’m ask­ing for a gut feel­ing here: What are your re­l­at­ive util­it­ies for $10^{100}$, $10^{110}$, $10^{120}$, $10^{130}$ uni­verses?

Say, $U(0)=0$, $U(10^100)=1$ (gauge fix­ing); in­stant pain-free end-of-uni­verse is zero util­ity, and a suc­cess­ful col­on­iz­a­tion of the en­tire uni­verse with a sub­op­timal black hole-farm­ing near heat-death is unit util­ity.

Now, per definitionem, the util­ity $U(N)$ of a $N$-com­pu­ta­tion out­come is the in­verse of the prob­ab­il­ity $p$ at which you be­come in­dif­fer­ent to the fol­low­ing gamble: Im­me­di­ate end-of-the-world at prob­ab­il­ity $(1-p)$ vs an up­grade of com­pu­ta­tional world-size to $N$ at prop­ab­il­ity $p$.

I would per­son­ally guess that $U(10^{130})< 2$; i.e. this up­grade would prob­ably not be worth a 50% risk of ex­tinc­tion. This is massively sub­lin­ear scal­ing.

• is avail­able by press­ing CTR+4/​CMD+4 in­stead of us­ing ‘\$’

• I am not sure how one can talk about the ob­served uni­verse and the num­ber 3^^^3 in the same sen­tence, given that the max­imum in­form­a­tional con­tent is roughly 10^120 qubits, the rest is out­side the cos­mo­lo­gical ho­ri­zon. Al­tern­at­ively, if we talk about the sim­u­la­tion ar­gu­ment, then the ex­pres­sion “prac­tical im­plic­a­tions” seems out of place.

• I am not sure how one can talk about the ob­served uni­verse and the num­ber 3^^^3 in the same sen­tence, given that the max­imum in­form­a­tional con­tent is roughly 10^120 qubits, the rest is out­side the cos­mo­lo­gical ho­ri­zon.

Where in the post do you see it sug­ges­ted that our uni­verse is cap­able of con­tain­ing 3^^^3 of any­thing?

Al­tern­at­ively, if we talk about the sim­u­la­tion ar­gu­ment, then the ex­pres­sion “prac­tical im­plic­a­tions” seems out of place.

How so?