# AlexMennen

Karma: 3,334
Page 1
• Ok, I see what you mean about in­de­pen­dence of ir­rele­vant al­ter­na­tives only be­ing a real co­her­ence con­di­tion when the prob­a­bil­ities are ob­jec­tive (or oth­er­wise known to be equal be­cause they come from the same source, even if there isn’t an ob­jec­tive way of say­ing what their com­mon prob­a­bil­ity is).

But I dis­agree that this makes VNM only ap­pli­ca­ble to set­tings in which all sources of un­cer­tainty have ob­jec­tively cor­rect prob­a­bil­ities. As I said in my pre­vi­ous com­ment, you only need there to ex­ist some source of ob­jec­tive prob­a­bil­ities, and you can then use prefer­ences over lot­ter­ies in­volv­ing ob­jec­tive prob­a­bil­ities and prefer­ences over re­lated lot­ter­ies in­volv­ing other sources of un­cer­tainty to de­ter­mine what prob­a­bil­ity the agent must as­sign for those other sources of un­cer­tainty.

Re: the differ­ence be­tween VNM and Bayesian ex­pected util­ity max­i­miza­tion, I take it from the word “Bayesian” that the way you’re sup­posed to choose be­tween ac­tions does in­volve first com­ing up with prob­a­bil­ities of each out­come re­sult­ing from each ac­tion, and from “ex­pected util­ity max­i­miza­tion”, that these prob­a­bil­ities are to be used in ex­actly the way the VNM the­o­rem says they should be. Since the VNM the­o­rem does not make any as­sump­tions about where the prob­a­bil­ities came from, these still sound es­sen­tially the same, ex­cept with Bayesian ex­pected util­ity max­i­miza­tion be­ing framed to em­pha­size that you have to get the prob­a­bil­ities some­how first.

• I think you’re un­der­es­ti­mat­ing VNM here.

only two of those four are rele­vant to co­her­ence. The main prob­lem is that the ax­ioms rele­vant to co­her­ence (acyclic­ity and com­plete­ness) do not say any­thing at all about probability

It seems to me that the in­de­pen­dence ax­iom is a co­her­ence con­di­tion, un­less I mi­s­un­der­stand what you mean by co­her­ence?

cor­rectly point out prob­lems with VNM

I’m cu­ri­ous what prob­lems you have in mind, since I don’t think VNM has prob­lems that don’t ap­ply to similar co­her­ence the­o­rems.

VNM util­ity stipu­lates that agents have prefer­ences over “lot­ter­ies” with known, ob­jec­tive prob­a­bil­ities of each out­come. The prob­a­bil­ities are as­sumed to be ob­jec­tively known from the start. The Bayesian co­her­ence the­o­rems do not as­sume prob­a­bil­ities from the start; they de­rive prob­a­bil­ities from the co­her­ence crite­ria, and those prob­a­bil­ities are spe­cific to the agent.

One can con­struct lot­ter­ies with prob­a­bil­ities that are pretty well un­der­stood (e.g. flip­ping coins that we have ac­cu­mu­lated a lot of ev­i­dence are fair), and you can re­strict at­ten­tion to lot­ter­ies only in­volv­ing un­cer­tainty com­ing from such sources. One may then get prob­a­bil­ities for other, less well-un­der­stood sources of un­cer­tainty by com­par­ing prefer­ences in­volv­ing such un­cer­tainty to prefer­ences in­volv­ing easy-to-quan­tify un­cer­tainty (e.g. if A is preferred to B, and you’re in­differ­ent be­tween 60%A+40%B and “A if X, B if not-X”, then you as­sign prob­a­bil­ity 60% to X. Per­haps not quite as philo­soph­i­cally satis­fy­ing as de­riv­ing prob­a­bil­ities from scratch, but this doesn’t seem like a fatal flaw in VNM to me.

I do not ex­pect agent-like sys­tems in the wild to be pushed to­ward VNM ex­pected util­ity max­i­miza­tion. I ex­pect them to be pushed to­ward Bayesian ex­pected util­ity max­i­miza­tion.

I un­der­stood those as be­ing syn­onyms. What’s the differ­ence?

• I do, how­ever, be­lieve that the sin­gle step co­op­er­ate-defect game which they use to come up with their fac­tors seems like a very sim­ple model for what will be a very com­plex sys­tem of in­ter­ac­tions. For ex­am­ple, AI de­vel­op­ment will take place over time, and it is likely that the same com­pa­nies will con­tinue to in­ter­act with one an­other. Iter­ated games have very differ­ent dy­nam­ics, and I hope that fu­ture work will ex­plore how this would af­fect their cur­rent recom­men­da­tions, and whether it would yield new ap­proaches to in­cen­tiviz­ing co­op­er­a­tion.

It may be difficult for com­pa­nies to get ac­cu­rate in­for­ma­tion about how care­ful their com­peti­tors are be­ing about AI safety. An iter­ated game in which play­ers never learn what the other play­ers did on pre­vi­ous rounds is the same as a one-shot game. This points to a sixth fac­tor that in­creases chance of co­op­er­a­tion on safety: high trans­parency, so that com­pa­nies may ver­ify their com­peti­tors’ co­op­er­a­tion on safety. This is closely re­lated to high trust.

• I ob­ject to the fram­ing of the bomb sce­nario on the grounds that low prob­a­bil­ities of high stakes are a source of cog­ni­tive bias that trip peo­ple up for rea­sons hav­ing noth­ing to do with FDT. Con­sider the fol­low­ing de­ci­sion prob­lem: “There is a but­ton. If you press the but­ton, you will be given $100. Also, press­ing the but­ton has a very small (one in a trillion trillion) chance of caus­ing you to burn to death.” Most peo­ple would not touch that but­ton. Us­ing the same pay­offs and prob­a­bil­ies in a sce­nario to challenge FDT thus ex­ploits cog­ni­tive bias to make FDT look bad. A bet­ter sce­nario would be to re­place the bomb with some­thing that will fine you$1000 (and, if you want, also in­crease the chance of of er­ror).

But then, it seems to me, that FDT has lost much of its ini­tial mo­ti­va­tion: the case for one-box­ing in New­comb’s prob­lem didn’t seem to stem from whether the Pre­dic­tor was run­ning a simu­la­tion of me, or just us­ing some other way to pre­dict what I’d do.

I think the cru­cial differ­ence here is how eas­ily you can cause the pre­dic­tor to be wrong. In the case where the pre­dic­tor simu­lates you, if you two-box, then the pre­dic­tor ex­pects you to two-box. In the case where the pre­dic­tor uses your na­tion­al­ity to pre­dict your be­hav­ior, Scots usu­ally one-box, and you’re Scot­tish, if you two-box, then the pre­dic­tor will still ex­pect you to one-box be­cause you’re Scot­tish.

But now sup­pose that the path­way by which S causes there to be money in the opaque box or not is that an­other agent looks at S...

I didn’t think that was sup­posed to mat­ter at all? I haven’t ac­tu­ally read the FDT pa­per, and have mostly just been op­er­at­ing un­der the as­sump­tion that FDT is ba­si­cally the same as UDT, but UDT didn’t build in any de­pen­dency on ex­ter­nal agents, and I hadn’t heard about any such de­pen­dency be­ing in­tro­duced in FDT; it would sur­prise me if it did.

• 16 Sep 2019 2:15 UTC
LW: 4 AF: 2
AF
I don’t know if I’m a simu­la­tion or a real per­son.

A pos­si­ble re­sponse to this ar­gu­ment is that the pre­dic­tor may be able to ac­cu­rately pre­dict the agent with­out ex­plic­itly simu­lat­ing them. A pos­si­ble counter-re­sponse to this is to posit that any suffi­ciently ac­cu­rate model of a con­scious agent is nec­es­sar­ily con­scious it­self, whether the model takes the form of an ex­plicit simu­la­tion or not.

• I think the coun­ter­fac­tu­als used by the agent are the cor­rect coun­ter­fac­tu­als for some­one else to use while rea­son­ing about the agent from the out­side, but not the cor­rect coun­ter­fac­tu­als for the agent to use while de­cid­ing what to do. After all, know­ing the agent’s source code, if you see it start to cross the bridge, it is cor­rect to in­fer that it’s rea­son­ing is in­con­sis­tent, and you should ex­pect to see the troll blow up the bridge. But while de­cid­ing what to do, the agent should be able to rea­son about purely causal effects of its coun­ter­fac­tual be­hav­ior, screen­ing out other log­i­cal im­pli­ca­tions.

Also, coun­ter­fac­tu­als which pre­dict that the bridge blows up seem to be say­ing that the agent can con­trol whether PA is con­sis­tent or in­con­sis­tent.

Disagree that that’s what’s hap­pen­ing. The link be­tween the con­sis­tency of the rea­son­ing sys­tem and the be­hav­ior of the agent is be­cause the con­sis­tency of the rea­son­ing sys­tem con­trols the agent’s be­hav­ior, rather than the other way around. Since the agent is se­lect­ing out­comes based on their con­se­quences, it does make sense to speak of the agent choos­ing ac­tions to some ex­tent, but I think speak­ing of log­i­cal im­pli­ca­tions of the agent’s ac­tions on the con­sis­tency of for­mal sys­tems as “con­trol­ling” the con­sis­tency of the for­mal sys­tem seems like an in­ap­pro­pri­ate at­tri­bu­tion of agency to me.

• I sup­pose why that’s not why we’re min­i­miz­ing de­ter­mi­nant, but rather frobe­nius norm.

Yes, al­though an­other rea­son is that the de­ter­mi­nant is only defined if the in­put and out­put spaces have the same di­men­sion, which they typ­i­cally don’t.

• First, a vec­tor can be seen as a list of num­bers, and a ma­trix can be seen as an or­dered list of vec­tors. An or­dered list of ma­tri­ces is… a ten­sor of or­der 3. Well not ex­actly. Ap­par­ently some peo­ple are ac­tu­ally dis­ap­pointed with the term ten­sor be­cause a ten­sor means some­thing very spe­cific in math­e­mat­ics already and isn’t just an or­dered list of ma­tri­ces. But what­ever, that’s the term we’re us­ing for this blog post at least.

It’s true that ten­sors are some­thing more spe­cific than mul­ti­di­men­sional ar­rays of num­bers, but Ja­co­bi­ans of func­tions be­tween ten­sor spaces (that be­ing what you’re us­ing the mul­ti­di­men­sional ar­rays for here) are, in fact, ten­sors.

• What this means is for the Ja­co­bian is that the de­ter­mi­nant tells us how much space is be­ing squished or ex­panded in the neigh­bor­hood around a point. If the out­put space is be­ing ex­panded a lot at some in­put point, then this means that the neu­ral net­work is a bit un­sta­ble at that re­gion, since minor al­ter­a­tions in the in­put could cause huge dis­tor­tions in the out­put. By con­trast, if the de­ter­mi­nant is small, then some small change to the in­put will hardly make a differ­ence to the out­put.

This isn’t quite true; the de­ter­mi­nant be­ing small is con­sis­tent with small changes in in­put mak­ing ar­bi­trar­ily large changes in out­put, just so long as small changes in in­put in a differ­ent di­rec­tion make suffi­ciently small changes in out­put.

The frobe­nius norm is noth­ing com­pli­cated, and is re­ally just a way of de­scribing that we square all of the el­e­ments in the ma­trix, take the sum, and then take the square root of this sum.

An al­ter­na­tive defi­ni­tion of the frobe­nius norm bet­ter high­lights its con­nec­tion to the mo­ti­va­tion of reg­u­lariz­ing the Ja­co­bian frobe­nius in terms of limit­ing the ex­tent to which small changes in in­put can cause large changes in out­put: The frobe­nius norm of a ma­trix J is the root-mean-square of |J(x)| over all unit vec­tors x.

• “Con­trol­ling which Everett branch you end up in” is the wrong way to think about de­ci­sions, even if many-wor­lds is true. Brains don’t ap­pear to rely much on quan­tum ran­dom­ness, so if you make a cer­tain de­ci­sion, that prob­a­bly means that the over­whelming ma­jor­ity of iden­ti­cal copies of you make the same de­ci­sion. You aren’t con­trol­ling which copy you are; you’re con­trol­ling what all of the copies do. And even if quan­tum ran­dom­ness does end of mat­ter­ing in de­ci­sions, so that a non-triv­ial pro­por­tion of copies of you make differ­ent de­ci­sions from each other, then you would still pre­sum­ably want a high pro­por­tion of them to make good de­ci­sions; you can do your part to bring that about by mak­ing good de­ci­sions your­self.

• Con­sider read­ing a real physi­cist’s take on the is­sue

This seems phrased to sug­gest that her view is “the real physi­cist view” on the mul­ti­verse. You could also read what Max Teg­mark or David Deutsch, for in­stance, have to say about mul­ti­verse hy­pothe­ses and get a “real physi­cist’s” view from them.

Also, she doesn’t ac­tu­ally say much in that blog post. She points out that when she says that mul­ti­verse hy­pothe­ses are un­scien­tific, she doesn’t mean that they’re false, so this doesn’t seem es­pe­cially use­ful to some­one who wants to know whether there ac­tu­ally is a mul­ti­verse, or is in­ter­ested in the con­se­quences thereof. She says “there is no rea­son to think we live in such mul­ti­verses to be­gin with”, but pro­po­nents of mul­ti­verse hy­pothe­ses have given rea­sons to sup­port their views, which she doesn’t ad­dress.

• #1 (at the end) sounds like com­plex­ity the­ory.

Some of what von Neu­mann says makes it sound like he’s in­ter­ested in a math­e­mat­i­cal foun­da­tion for ana­log com­put­ing, which I think has been done by now.

• On sev­eral oc­ca­sions, the au­thors em­pha­size how the in­tu­itive na­ture of “effec­tive com­putabil­ity” ren­ders fu­tile any at­tempt to for­mal­ize the the­sis. How­ever, I’m rather in­ter­ested in for­mal­iz­ing in­tu­itive con­cepts and there­fore won­dered why this hasn’t been at­tempted.

For­mal­iz­ing the in­tu­itive no­tion of effec­tive com­putabil­ity was ex­actly what Tur­ing was try­ing to do when he in­tro­duced Tur­ing ma­chines, and Tur­ing’s the­sis claims that his at­tempt was suc­cess­ful. If you come up with a new for­mal­iza­tion of effec­tive com­putabil­ity and prove it equiv­a­lent to Tur­ing com­putabil­ity, then in or­der to use this as a proof of Tur­ing’s the­sis, you would need to ar­gue that your new for­mal­iza­tion is cor­rect. But such an ar­gu­ment would in­evitably be in­for­mal, since it links a for­mal con­cept to an in­for­mal con­cept, and there already have been in­for­mal ar­gu­ments for Tur­ing’s the­sis, so I don’t think there is any­thing re­ally fun­da­men­tal to be gained from this.

• Con­sider the halt­ing set; … is not enu­mer­able /​ com­putable.

Here, we should be care­ful with how we in­ter­pret “in­for­ma­tion”. After all, coNP-com­plete prob­lems are triv­ially Cook re­ducible to their NP-com­plete coun­ter­parts (e.g., query the or­a­cle and then negate the out­put), but many be­lieve that there isn’t a cor­re­spond­ing Karp re­duc­tion (where we do a polyno­mial amount of com­pu­ta­tion be­fore query­ing the or­a­cle and re­turn­ing its an­swer). Since we aren’t con­sid­er­ing com­plex­ity but in­stead whether it’s enu­mer­able at all, com­ple­men­ta­tion is fine.

You’re us­ing the word “enu­mer­able” in a non­stan­dard way here, which might in­di­cate that you’ve missed some­thing (and if not, then per­haps at least this will be use­ful for some­one else read­ing this). “Enu­mer­able” is not usu­ally used as a syn­onym for com­putable. A set is com­putable if there is a pro­gram that de­ter­mines whether or not its in­put is in the set. But a set is enu­mer­able if there is a pro­gram that halts if its in­put is in the set, and does not halt oth­er­wise. Every com­putable set is enu­mer­able (since you can just use the out­put of the com­pu­ta­tion to de­cide whether or not to halt). But the halt­ing set is an ex­am­ple of a set that is enu­mer­able but not com­putable (it is enu­mer­able be­cause you can just run the pro­gram coded by your in­put, and halt if/​when it halts). Enu­mer­able sets are not closed un­der com­ple­men­ta­tion; in fact, an enu­mer­able set whose com­ple­ment is enu­mer­able is com­putable (be­cause you can run the pro­grams for the set and its com­ple­ment in par­allel on the same in­put; even­tu­ally one of them will halt, which will tell you whether or not the in­put is in the set).

The dis­tinc­tion be­tween Cook and Karp re­duc­tions re­mains mean­ingful when “polyno­mial-time” is re­placed by “Tur­ing com­putable” in the defi­ni­tions. Any set that an enu­mer­able set is Tur­ing-Karp re­ducible to is also enu­mer­able, but an enu­mer­able set is Tur­ing-Cook re­ducible to its com­ple­ment.

The rea­son “enu­mer­able” is used for this con­cept is that a set is enu­mer­able iff there is a pro­gram com­put­ing a se­quence that enu­mer­ates ev­ery el­e­ment of the set. Given a pro­gram that halts on ex­actly the el­e­ments of a given set, you can con­struct an enu­mer­a­tion of the set by run­ning your pro­gram on ev­ery in­put in par­allel, and adding an el­e­ment to the end of your se­quence when­ever the pro­gram halts on that in­put. Con­versely, given an enu­mer­a­tion of a set, you can con­struct a pro­gram that halts on el­e­ments of the set by go­ing through the se­quence and halt­ing when­ever you find your in­put.

• I don’t fol­low the anal­ogy to 1/​x be­ing a par­tial func­tion that you’re get­ting at.

Maybe a bet­ter way to ex­plain what I’m get­ting at is that it’s re­ally the same is­sue that I pointed out for the two-en­velopes prob­lem, where you know the amount of money in each en­velope is finite, but the uniform dis­tri­bu­tion up to an in­finite sur­real would sug­gest that the prob­a­bil­ity that the amount of money is finite is in­finites­i­mal. Sup­pose you say that the size of the ray is an in­finite sur­real num­ber . The size of the por­tion of this ray that is dis­tance at least from is when is a pos­i­tive real, so pre­sum­ably you would also want this to be so for sur­real . But us­ing, say, , ev­ery point in is within dis­tance of , but this rule would say that the mea­sure of the por­tion of the ray that is farther than from is ; that is, al­most all of the mea­sure of is con­cen­trated on the empty set.

• The lat­ter. It doesn’t even make sense to speak of max­i­miz­ing the ex­pec­ta­tion of an un­bounded util­ity func­tion, be­cause un­bounded func­tions don’t even have ex­pec­ta­tions with re­spect to all prob­a­bil­ity dis­tri­bu­tions.

There is a way out of this that you could take, which is to only in­sist that the util­ity func­tion has to have an ex­pec­ta­tion with re­spect to prob­a­bil­ity dis­tri­bu­tions in some re­stricted class, if you know your op­tions are all go­ing to be from that re­stricted class. I don’t find this very satis­fy­ing, but it works. And it offers its own solu­tion to Pas­cal’s mug­ging, by in­sist­ing that any out­come whose util­ity is on the scale of 3^^^3 has prior prob­a­bil­ity on the scale of 1/​(3^^^3) or lower.

• It’s a bad bul­let to bite. Its sym­me­tries are es­sen­tial to what makes Eu­clidean space in­ter­est­ing.

And here’s an­other one: are you not both­ered by the lack of countable ad­di­tivity? Sup­pose you say that the vol­ume of Eu­clidean space is some sur­real num­ber . Eu­clidean space is the union of an in­creas­ing se­quence of balls. The vol­umes of these balls are all finite, in par­tic­u­lar, less than , so how can you jus­tify say­ing that their union has vol­ume greater than ?

• Why? Plain se­quences are a perfectly nat­u­ral ob­ject of study. I’ll echo gjm’s crit­i­cism that you seem to be try­ing to “re­solve” para­doxes by chang­ing the defi­ni­tions of the words peo­ple use so that they re­fer to un­nat­u­ral con­cepts that have been ger­ry­man­dered to fit your solu­tion, while re­fus­ing to talk about the nat­u­ral con­cepts that peo­ple ac­tu­ally care about.

I don’t think think your pro­posal is a good one for in­dexed se­quences ei­ther. It is pretty weird that shift­ing the in­dices of your se­quence over by 1 could change the size of the se­quence.

• There are mea­surable sets whose vol­umes will not be pre­served if you try to mea­sure them with sur­real num­bers. For ex­am­ple, con­sider . Say its mea­sure is some in­finite sur­real num­ber . The vol­ume-pre­serv­ing left-shift op­er­a­tion sends to , which has mea­sure , since has mea­sure . You can do es­sen­tially the same thing in higher di­men­sions, and the shift op­er­a­tion in two di­men­sions () can be ex­pressed as the com­po­si­tion of two ro­ta­tions, so ro­ta­tions can’t be vol­ume-pre­serv­ing ei­ther. And since differ­ent ro­ta­tions will have to fail to pre­serve vol­umes in differ­ent ways, this will break sym­me­tries of the plane.

I wouldn’t say that vol­ume-pre­serv­ing trans­for­ma­tions fail to pre­serve vol­ume on non-mea­surable sets, just that non-mea­surable sets don’t even have mea­sures that could be pre­served or not pre­served. Failing to pre­serve mea­sures of sets that you have as­signed mea­sures to is en­tirely differ­ent. Non-mea­surable sets also don’t arise in math­e­mat­i­cal prac­tice; half-spaces do. I’m also skep­ti­cal of the ex­is­tence of non-mea­surable sets, but the non-ex­is­tence of non-mea­surable sets is a far bolder claim than any­thing else I’ve said here.