Vladimir_Nesov(Vladimir Nesov)

• Vladimir_Nesov 22 Nov 2020 14:58 UTC

  6 points

  in reply to: abramdemski’s comment on: UDT might not pay a Coun­ter­fac­tual Mugger

  CDT and EDT are also sen­si­tive to their prior. The differ­ence is that it’s a more fa­mil­iar rou­tine to define their prior by ideal­iza­tion of the situ­a­tion be­ing con­sid­ered with­out get­ting out of scope, thus en­sur­ing that we re­main close to in­for­mal ex­pec­ta­tions. When build­ing a tractable model for UDT, it similarly makes sense to spec­ify its prior with­out al­low­ing re­trac­tion of knowl­edge of the situ­a­tion and es­cap­ing to con­sid­er­a­tion of all pos­si­ble situ­a­tions (turn­ing the prior into a model of all pos­si­ble situ­a­tions rather than just of this one situ­a­tion).

  In the case of CDT and EDT, es­cap­ing the bounds of the situ­a­tion looks like a re­friger­a­tor fal­ling from the sky on the ex­per­i­men­tal ap­para­tus. In the case of UDT, it looks like a fund­ing agency re­fus­ing to fund the ex­per­i­ment be­cause its re­sults wouldn’t be poli­ti­cally ac­cept­able, un­less it’s mas­saged to look right, and the agents within the ex­per­i­ment un­der­stand that (and have no sci­en­tific in­tegrity). I think it’s similarly un­rea­son­able for both kinds of de­tails to be in­cluded in mod­els, and it’s similarly pos­si­ble for them to oc­cur in re­al­ity.

• Vladimir_Nesov 22 Nov 2020 14:30 UTC

  2 points

  in reply to: winwonce’s comment on: UDT might not pay a Coun­ter­fac­tual Mugger

  In the first ap­prox­i­ma­tion, the point is not that coun­ter­fac­tual mug­ging (or any other thought ex­per­i­ment) is ac­tu­ally defined in a cer­tain way, but how it should be re­defined in or­der to make it pos­si­ble to nav­i­gate the is­sue. Un­less Nomegas are out­lawed, it’s not pos­si­ble to do any calcu­la­tions, there­fore they are out­lawed. Not be­cause they were already ex­plic­itly out­lawed or were col­lo­quially un­der­stood to be out­lawed.

  But when we look at this more care­fully, the as­sump­tion is not ac­tu­ally needed. If non­speci­fied Nomegas are al­lowed, the dis­tri­bu­tion of their pos­si­ble in­cen­tives is all over the place, so they al­most cer­tainly can­cel out in the ex­pected util­ity of al­ter­na­tive pre­com­mit­ments. The real prob­lem is not with in­tro­duc­tion of Nomegas, but with man­ag­ing to in­clude the pos­si­bil­ities in­volv­ing Omega in the calcu­la­tions (as op­posed to dis­card­ing them as par­tic­u­lar Nomegas), tak­ing into ac­count the set­ting that’s not yet de­scribed at the point where pre­com­mit­ment should be made.

  In coun­ter­fac­tual mug­ging, there is no phys­i­cal time when the agent is in the state of knowl­edge where the rele­vant pre­com­mit­ment can be made (that’s the whole point). In­stead, we can con­struct a hy­po­thet­i­cal state of knowl­edge that has up­dated on the de­scrip­tion of the thought ex­per­i­ment, but hasn’t up­dated on the fact of how the coin toss turned out. The agent never holds this state of knowl­edge as a de­scrip­tion of all that’s ac­tu­ally known. Why re­tract knowl­edge of the coin toss, in­stead of re­tract­ing knowl­edge of the thought ex­per­i­ment? No rea­son, UDT strives to re­tract all knowl­edge and make a com­pletely gen­eral pre­com­mit­ment to all even­tu­al­ities. But in this set­ting, re­tract­ing knowl­edge of the coin toss while re­tain­ing knowl­edge of Omega cre­ates a tractable de­ci­sion prob­lem, thus UDT that no­tices the pos­si­bil­ity will make a pre­com­mit­ment. Similarly, it should pre­com­mit to not pay­ing Omega in a situ­a­tion where a Nomega pun­ish­ing for pay­ing up $100 to Omega (as de­scribed in this post) is known to op­er­ate. But only when it’s known to be there, not when it’s not known to be there.

• Vladimir_Nesov 22 Nov 2020 13:07 UTC

  2 points

  in reply to: winwonce’s comment on: UDT might not pay a Coun­ter­fac­tual Mugger

  I think I see what you mean. The situ­a­tion where you’d make a pre­com­mit­ment, which is de­scribed by the same state of knowl­edge that UDT makes its de­ci­sion un­der, oc­curs be­fore the set­ting of the thought ex­per­i­ment is made clear. Thus it’s not yet clear what kinds of Nomegas can show up with their par­tic­u­lar in­cen­tives, and the pre­com­mit­ment can’t rely on their ab­sense. With some sort of risk-averse sta­tus-quo-an­chored at­ti­tude it seems like “not pre­comit­ting” is there­fore gen­er­ally prefer­able.

  But op­ti­miza­tion of ex­pected util­ity doesn’t work like that. You have the es­ti­mates for pos­si­ble de­ci­sions, and pick the op­tion that’s es­ti­mated to be the best available. Whether it’s the sta­tus quo (“not pre­com­mit­ting”) or not has no bear­ing on the de­ci­sion un­less it’s ex­pressed as a change in the es­i­mate of ex­pected util­ity that makes it lower or greater than ex­pected util­ity of the al­ter­na­tive de­ci­sions. Thus when a thought ex­per­i­ment talks about pre­com­mit­ments or any UDT de­ci­sions, bring­ing in ar­bi­trary Nomegas is a prob­lem be­cause it makes the ex­pected util­ity of pre­com­mit­ments similarly ar­bi­trary, and it’s these ex­pected util­ities that de­ter­mine de­ci­sions. (Whether to make some pre­com­mit­ment or not is it­self a de­ci­sion.) The ob­vi­ous way of mak­ing it pos­si­ble to perform the calcu­la­tion of ex­pected util­ities of pre­com­mit­ments is to make the as­sump­tion of ab­sense of Nomegas, or more gen­er­ally to con­struct the set­tings of pre­com­mit­ments based only on what’s already in the thought ex­per­i­ment.

  (Mis­takes in ex­pected val­ues in the post are a tiny bit rele­vant (one of the val­ues is still wrong af­ter the cor­rec­tion), as they vaguely sig­nal lack of re­li­able knowl­edge of what ex­pected util­ity is, al­though the is­sue here is mostly in­for­mal and cor­rect calcu­la­tion won’t by it­self make things clear. Gen­eral ex­pe­rience with math­e­mat­i­cal proofs might be closer to be­ing helpful, as the is­sue is that the ac­tual al­gorithms be­ing dis­cussed screen off a lot of in­for­mal con­sid­er­a­tions such as whether not mak­ing pre­com­mit­ments is the sta­tus quo.)

• Vladimir_Nesov 22 Nov 2020 8:15 UTC

  5 points

  on: UDT might not pay a Coun­ter­fac­tual Mugger

  This proves too much. For any thought ex­per­i­ment, if you are al­lowed to in­tro­duce a gen­er­al­ized Nomega, you can force any con­clu­sion for an agent that cares about coun­ter­fac­tu­als or has a chance to make a global pre­com­mit­ment. What if Nomega pays $50000 when you do pre­com­mit to pay $100 to Omega? Or when you Defect in Pri­soner’s Dilemma? Or when you Co­op­er­ate in Pri­soner’s Dilemma? This shows that if Nomega is al­lowed to be in­tro­duced, none of the usual de­ci­sion the­ory thought ex­per­i­ments can be use­fully con­sid­ered (with a the­ory like UDT). Thus, it’s a rea­son­able as­sump­tion that Nomega shouldn’t be al­lowed to be in­tro­duced.

  (Btw, the ex­pected val­ues you list in the table are off. Don’t know what’s up with that.)

• Vladimir_Nesov 21 Nov 2020 0:29 UTC

  2 points

  in reply to: abramdemski’s comment on: Sig­nal­ling & Si­mu­lacra Level 3

  I don’t see the dis­tinc­tion be­tween con­no­ta­tion and de­no­ta­tion as an effec­tive way of carv­ing this mud­dle. The prob­lem with sig­nal­ing the­ory of mean­ing is that it ex­plains com­mu­ni­ca­tion of mean­ing as com­mu­ni­ca­tion of truth, mix­ing up these differ­ent things. But mean­ing is of­ten com­mu­ni­cated us­ing the whole palette of tools also used for sig­nal­ing truth. In par­tic­u­lar, com­mu­ni­ca­tion of mean­ing (that is the kinds of things used to con­struct mod­els and hy­pothe­ses) with ut­ter­ances that look like vague rea­son­ing by as­so­ci­a­tion shouldn’t in it­self make it more difficult to rea­son lawfully and clearly about that mean­ing.

  So the method I’m propos­ing is to con­sider any ut­ter­ance in ei­ther ca­pac­ity in turn, with sep­a­rate ques­tions like “Which idea is this draw­ing at­ten­tion to?” and “What weight is im­plied for rele­vant as­ser­tions about this idea?” But at this point I’m not sure what the es­sen­tial difficulty is that re­mains, be­cause I don’t per­ceive the mo­ti­va­tion for the post clearly enough.

• Vladimir_Nesov 20 Nov 2020 22:11 UTC

  4 points

  in reply to: ike’s comment on: Why is there a “clogged drain­pipe” effect in idea gen­er­a­tion?

  Not writ­ing an idea down also helps you nat­u­rally think about it in the back­ground. So it’s more of a trade-off than a gen­er­ally use­ful heuris­tic.

• Vladimir_Nesov 14 Nov 2020 21:52 UTC

  3 points

  in reply to: Slider’s comment on: My Con­fu­sion about Mo­ral Philos­o­phy

  “I have a neat proof. As­sume that A”, “I don’t be­lieve A”, “Well you won’t be­lieve my proof then”

  The proof won’t be a con­vinc­ing ar­gu­ment for agree­ing with its con­clu­sion. But the proof it­self can be checked with­out be­lief in A, and if it checks out, this state of af­fairs can be de­scribed as be­lief in the proof.

• Vladimir_Nesov 14 Nov 2020 20:58 UTC

  4 points

  on: Sig­nal­ling & Si­mu­lacra Level 3

  Com­mu­ni­ca­tion of mean­ing, sig­nal­ing of truth. I’m not sure what es­sen­tial difficulty re­mains if we merely make sure to dis­t­in­guish be­tween com­mu­ni­cat­ing ideas (which in this role are to be made clear, not yet com­pared against the world), and pro­vid­ing ev­i­dence for their rele­vance to the dis­cus­sion or for their cor­re­spon­dence-to-the-world truth. Fire­flies won’t be able to com­mu­ni­cate ideas, only sig­nal truths, so this anal­y­sis doesn’t nat­u­rally ap­ply to them. But lan­guage can com­mu­ni­cate ideas with­out im­pli­ca­tion of their truth, and at this point sig­nal­ing helps if truth is to be ex­tracted some­what di­rectly from other ac­tors and not eval­u­ated in other ways.

  we as­sume the re­mark is rele­vant to the conversation

  For ex­am­ple, in this case the as­sump­tion is part of how mean­ing is guessed, but is in gen­eral un­re­lated to how its truth (or truth of its rele­vance) is to be eval­u­ated. The in­ter­min­gling of the two as­pects of com­mu­ni­ca­tion is mostly short­hand, it can be teased apart.

• Vladimir_Nesov 11 Nov 2020 20:18 UTC

  4 points

  in reply to: Gyrodiot’s comment on: The (Unoffi­cial) Less Wrong Com­ment Challenge

  I don’t be­lieve we’re in that con­text, hence my com­ment.

  It’s im­por­tant to prac­tice habits when they are use­less, or else you end up un­prac­ticed when you need them. So I dis­agree that a (cheap) habit not be­ing use­ful in some case is a rea­son to dis­re­gard lack of its use in that case.

• Vladimir_Nesov 11 Nov 2020 19:47 UTC

  5 points

  in reply to: Gyrodiot’s comment on: The (Unoffi­cial) Less Wrong Com­ment Challenge

  That’s cer­tainly the in­tended mean­ing, but the state­ment it­self is in fact heavy with qual­ifiers. This fact doesn’t vary with good­will.

  I think the first of the qual­ifiers, “If what you say is true, I’ll act on what you say,” is ac­tu­ally a harm­ful an­tipat­tern, since it com­mits to ac­tion on the as­sump­tion that the state­ment is true while pre­tend­ing to com­mit to ac­tion in case the state­ment is true, thus re­serv­ing doubt about the state­ment. It would be healthier to ei­ther com­mit to ac­tion on the as­sump­tion of doubt, or to re­fuse to com­mit be­cause of doubt.

• Vladimir_Nesov 11 Nov 2020 19:39 UTC

  10 points

  in reply to: adamShimi’s comment on: The (Unoffi­cial) Less Wrong Com­ment Challenge

  I don’t think that hav­ing feed­back, even if only some­thing like “I like this post”/​”I dis­liked this post”/​”I had trou­ble to care”, is noise. For me, this is in­cred­ibly im­por­tant, and ev­ery per­son I talked with on this sub­ject agreed.

  Whether peo­ple agree is beside the point. Is this ac­tu­ally true? To figure it out, it’s nec­es­sary to be more clear on what the state­ment means. In what way, speci­fi­cally, is this kind of feed­back im­por­tant? If some­one says “I like your post”, it’s very hard to learn any­thing in par­tic­u­lar from that. If it’s cus­tom­ary to say things like that, some peo­ple would just say them out of mis­placed gen­eral nice­ness, and there will be even less mean­ing in the ut­ter­ance. (Also, there are up­votes/​down­votes to in­di­cate pre­cisely that kind of feed­back.) Some peo­ple like hear­ing that some­one likes their posts, but that’s differ­ent from feed­back be­ing in­struc­tive. Noise can be pleas­ant with­out be­ing en­light­en­ing. Fi­nally, it can be mo­ti­vat­ing to hear that your work is ap­pre­ci­ated.

  It seems that it’s false that this kind of feed­back is im­por­tant for learn­ing things, but true that it’s im­por­tant for mo­ti­va­tion of some au­thors. Th­ese are differ­ent claims that shouldn’t be mixed up. When you say “ev­ery per­son I talked with on this sub­ject agreed”, do you (or they) know what ex­actly they agree with?

  I don’t think it will ac­tu­ally push read­ers away, be­cause they are in to­tal con­trol of their level of com­mit­ment.

  Un­for­tu­nately, norms don’t like nu­ance. This post is very far from ig­nit­ing a norm, but if hy­po­thet­i­cally what it sug­gests bears fruit, it will be in the form of a norm to com­ment more, and that norm will end up pun­ish­ing defec­tors re­gard­less of whether that was an in­tended fea­ture of the norm or not.

• Vladimir_Nesov 11 Nov 2020 19:08 UTC

  −7 points

  in reply to: D0TheMath’s comment on: The (Unoffi­cial) Less Wrong Com­ment Challenge

  I can try out com­ment­ing more if re­ceiv­ing any feed­back is as im­por­tant as you say.

  That’s a most weasely state­ment! If you don’t know if it’s as im­por­tant as they say, can you ac­tu­ally try com­ment­ing more or not? And if you can try, will you? And if you do try, will you suc­ceed?

• Vladimir_Nesov 11 Nov 2020 18:58 UTC

  2 points

  on: The (Unoffi­cial) Less Wrong Com­ment Challenge

  This would in­crease the level of noise. A good change along these lines is for peo­ple who have some­thing to say to speak up more fre­quently (as op­posed to ev­ery­one speak­ing up more fre­quently, re­gard­less of whether they have some­thing non­triv­ial to say). But even that is po­ten­tially haz­ardous, since a norm sug­gest­ing an obli­ga­tion to voice your thoughts makes read­ing more costly, pos­si­bly pushes read­ers away.

• Vladimir_Nesov 10 Nov 2020 15:29 UTC

  LW: 7 AF: 4

  AF

  in reply to: adamShimi’s comment on: Why You Should Care About Goal-Directedness

  Yeah, that was sloppy of the ar­ti­cle. In con­text, the quote makes a bit of sense, and the qual­ifier “in ev­ery de­tail” does use­ful work (though I don’t see how to make the ar­gu­ment clear just by defin­ing what these words mean), but with­out con­text it’s in­valid.

• Vladimir_Nesov 10 Nov 2020 14:41 UTC

  LW: 2 AF: 1

  AF

  in reply to: adamShimi’s comment on: Why You Should Care About Goal-Directedness

  Hav­ing an ex­act model of the world that con­tains the agent doesn’t re­quire any ex­plicit self-refer­ences or refer­ences to the agent. For ex­am­ple, if there are two pro­grams whose be­hav­ior is equiv­a­lent, A and A’, and the agent cor­rectly thinks of it­self as A, then it can also know the world to be a pro­gram W(A’) with some subex­pres­sions A’, but with­out subex­pres­sion A. To see the con­se­quences of its ac­tions in this world, it would be use­ful for the agent to figure out that A is equiv­a­lent to A’, but it is not nec­es­sary that this is known to the agent from the start, so any self-refer­ence in this set­ting is im­plicit. Also, A’ can’t have W(A’) as a subex­pres­sion, for rea­sons that do ad­mit an ex­pla­na­tion given in the quote that started this thread, but at the same time A can have W(A’) as a subex­pres­sion. What is smaller here, the world or the agent?

  (What’s naive Gödelian self-refer­ence? I don’t re­call this term, and googling didn’t help.)

  Deal­ing with self-refer­ence in defi­ni­tions of agents and wor­lds does not re­quire (or even par­tic­u­larly recom­mend) non-re­al­iz­abil­ity. I don’t think it’s an is­sue spe­cific to em­bed­ded agents, prob­a­bly all puz­zles that fall within this scope can be stud­ied while re­quiring the world to be a finite pro­gram. It might be a good idea to look for other set­tings, but it’s not forced by the prob­lem state­ment.

  non-re­al­iz­abil­ity (the im­pos­si­bil­ity of the agent to con­tain an ex­act model of the world, be­cause it is in­side the world and thus smaller)

  Be­ing in­side the world does not make it im­pos­si­ble for the agent to con­tain the ex­act model of the world, does not re­quire non-re­al­iz­abil­ity in its rea­son­ing about the world. This is the same er­ror as in the origi­nal quote. In what way are quines not an in­tu­itive coun­terex­am­ple to this rea­son­ing? Speci­fi­cally, the er­ror is in say­ing “and thus smaller”. What does “smaller” mean, and how does be­ing a part in­ter­act with it? Parts are not nec­es­sar­ily smaller than the whole, they can well be larger. Ex­act de­scrip­tions of wor­lds and agents are not just finite ex­pres­sions, they are at least equiv­alence classes of ex­pres­sions that be­have in the same way, and el­e­ments of those equiv­alence classes can have vastly differ­ent syn­tac­tic size.

  (Of course in some set­tings there are rea­sons for non-re­al­iz­abil­ity to be nec­es­sary or to not be a prob­lem.)

• Vladimir_Nesov 10 Nov 2020 10:43 UTC

  LW: 2 AF: 1

  AF

  in reply to: adamShimi’s comment on: Why You Should Care About Goal-Directedness

  The quote sounds like an ar­gu­ment for non-ex­is­tence of quines or of the con­text in which things like the di­ag­o­nal­iza­tion lemma are for­mu­lated. I think it ob­vi­ously sounds like this, so rais­ing non­spe­cific con­cern in my com­ment above should’ve been enough to draw at­ten­tion to this is­sue. It’s also not a prob­lem Agent Foun­da­tions ex­plores, but it’s pre­sented as such. Given your back­ground and effort put into the post this in­ter­pre­ta­tion of the quote seems un­likely (which is why I didn’t ini­tially clar­ify, to give you the first move). So I’m con­fused. Every­thing is con­fus­ing here, in­clud­ing your com­ment above not tak­ing the cue, pos­i­tive vot­ing on it, and nega­tive vot­ing on my com­ment. Maybe the in­tended mean­ings of “model” and “be­ing ex­act” and “rep­re­sen­ta­tion” are such that the ar­gu­ment makes sense and be­comes re­lated to Agent Foun­da­tions?

• Vladimir_Nesov 9 Nov 2020 21:36 UTC

  LW: -4 AF: -3

  AF

  on: Why You Should Care About Goal-Directedness

  Trou­ble comes from self-refer­ence: since the agent is part of the world, so is its model, and thus a perfect model would need to rep­re­sent it­self, and this rep­re­sen­ta­tion would need to rep­re­sent it­self, ad in­fini­tum. So the model can­not be ex­act.

  ???

• Vladimir_Nesov 5 Nov 2020 12:28 UTC

  2 points

  in reply to: ChristianKl’s comment on: Non Polemic: How do you per­son­ally deal with “ir­ra­tional” peo­ple?

  Ex­actly, that’s what makes the ques­tion as you for­mu­lated it funny. It’s not a ques­tion, or even a re­quest. It’s a non-ne­go­tiable de­mand. If you don’t con­cede, the whole deal is off. Yet not con­ced­ing is of­ten the only rea­son­able thing to do, so it’s a de­mand to be un­rea­son­able mas­querad­ing as a ques­tion, be­cause don’t be rude.

• Vladimir_Nesov 4 Nov 2020 18:03 UTC

  2 points

  in reply to: johnswentworth’s comment on: Con­fu­ci­anism in AI Alignment

  Scien­tists do­ing ba­sic re­search also mostly aren’t mo­ti­vated by the hope that it will some­day lead to prac­ti­cal ap­pli­ca­tions. When there is con­fu­sion or un­cer­tainty about a salient phe­nomenon that can be clar­ified with fur­ther re­search, that is enough. In­ci­den­tally, it is vir­tu­ous and sig­nals tribal alle­giance to that field of re­search. Some of the re­searchers are go­ing to be mo­ti­vated by that.

• Vladimir_Nesov 4 Nov 2020 17:32 UTC

  2 points

  in reply to: johnswentworth’s comment on: Con­fu­ci­anism in AI Alignment

  This plays the same role as ba­sic re­search, ideas that can be de­vel­oped but haven’t found even an inkling of their po­ten­tial prac­ti­cal ap­pli­ca­tions. An er­ror would be think­ing that they are go­ing to be im­me­di­ately use­ful, but that shouldn’t be a strong ar­gu­ment against de­vel­op­ing them, and there should be no cer­tainty that their very in­di­rect use won’t end up cru­cial at some point in the dis­tant fu­ture.