Who are you and how is it that we don’t we know each other yet?

I’d be glad to add a few. I’ll have to do some think­ing but one im­me­di­ately popped to mind so I’ll add it now. Good luck with the pro­ject. Sounds in­ter­est­ing.

re­lated com­ment: https://​fo­rum.effec­tivealtru­ism.org/​posts/​F7hZ8co3L82nTdX4f/​do-eas-un­der­es­ti­mate-op­por­tu­ni­ties-to-cre­ate-many-small#XGSQX45NkAN9qSB9A

Which I sup­pose could be termed “in­finitely con­fused”, but that feels like a mix­ing of lev­els. You’re not con­fused about a given prob­a­bil­ity, you’re con­fused about how prob­a­bil­ity works.

Or al­ter­na­tively, it’s a clever turn of phrase: “in­finitely con­fused” as in con­fused about in­fini­ties.

This is a good point, but per­haps pre­com­mit­ting to give in/​not give in vs. pre­com­mit­ting to black­mail/​not black­mail is a si­mul­ta­neous choice.

“If I have an ac­tion that I can take that would help me but hurt you and I ask you for some com­pen­sa­tion for re­frain­ing from tak­ing this ac­tion, then this is more like a value trade than a black­mail”—Maybe. What about if an ac­tion gives you 1 util­ity, but costs me a 100 and you de­mand 90. That sounds a lot like black­mail!

Us­ing ep­silons can in prin­ci­ple al­low you to up­date. How­ever, the situ­a­tion seems slightly worse than jim­ran­domh de­scribes. It looks like you need P(E|h), or the prob­a­bil­ity if H is false, in or­der to get a pre­cise an­swer. Also, the miss­ing info that jim men­tioned is already enough in prin­ci­ple to let the fi­nal an­swer be any prob­a­bil­ity what­so­ever.

If we use log odds (the frame­work in which we could liter­ally start with “in­finite cer­tainty”) then the an­swer could be any­where on the real num­ber line. We have in­finite (or at least un­bounded) con­fu­sion un­til we make our as­sump­tions more pre­cise.

This math is ex­actly why we say a ra­tio­nal agent can never as­sign a perfect 1 or 0 to any prob­a­bil­ity es­ti­mate. Do­ing so in a uni­verse which then pre­sents you with coun­terev­i­dence means you’re not ra­tio­nal.

Which I sup­pose could be termed “in­finitely con­fused”, but that feels like a mix­ing of lev­els. You’re not con­fused about a given prob­a­bil­ity, you’re con­fused about how prob­a­bil­ity works.

In prac­tice, when a well-cal­ibrated per­son says 100% or 0%, they’re round­ing off from some un­speci­fied-pre­ci­sion es­ti­mate like 99.9% or 0.000000000001.

This map­ping does not match any ac­tual de­ci­sions in black­mail. First, it’s not a si­mul­ta­neous choice, it’s a branch­ing multi-turn de­ci­sion tree. Se­cond, there are more than 2 ac­tions available at var­i­ous stages. Either of these would make pris­oner’s dilemma anal­y­sis sus­pect, to­gether it be­comes much more like multi-street multi-bet poker than like PD.

The “vic­tim” first makes choices (or is born into a situ­a­tion) sus­cep­ti­ble to black­mail. The black­mailer learns of this, and has at least 3 choices: pub­lish the in­for­ma­tion, threaten to pub­lish, or bury the in­for­ma­tion. The “vic­tim” in the threaten-to-pub­lish (black­mail) case offers in­cen­tives (which may be the same as the re­quested fee, or may not) to bury rather than pub­lish, and the black­mailer chooses which ac­tion to take. Even leav­ing out true defec­tion cases (ac­cept the money and pub­lish any­way, or kil­ling the black­mailer), this is a fairly com­plex pay­out tree, and the cor­rect choices are spe­cific to the situ­a­tion. In fact, since parts of the pay­out tree are un­known to one or both play­ers, it’s likely that mixed strate­gies come into play, to pre­vent ex­ploita­tion of the un­knowns.

$P\left(E\right)=P\left(E|H\right)P\left(H\right)+P(E|!H)P(!H)=0\times 1+P(E|!H)\times 0$

In other words, the agent as­signed zero prob­a­bil­ity to an event, and then it hap­pened.

I see. so -

If P(H) = 1.0 - ϵ1

And P(E|H) = 0 + ϵ2

Then it equals “in­finite con­fu­sion”.

Am i cor­rect?

and also, when you use ep­silons, does it mean you get out of the “dogma” of 100%? or you still can’t up­date down from it?

And what i did in my post may just be an­other ex­am­ple of why you don’t put an ac­tual 1.0 in your prior, cause then even if you get ev­i­dence of the same strength in the other di­rec­tion, that would de­mand that you di­vide zero by zero. right?

This is a great post! Thank you in par­tic­u­lar for the Stephen Boyd recom­men­da­tion.

~~~~

In terms of frame­works and ML, you might be in­ter­ested in this blog post an­nounc­ing the re­lease of the DiffEqFlux.jl pack­age in the Ju­lia pro­gram­ming lan­guage. I was re­minded of this be­cause it seems to be an ex­am­ple of merg­ing already-ma­ture frame­works, speci­fi­cally a ma­chine learn­ing frame­work (Flux) and a differ­en­tial equa­tion frame­work (Differ­en­tialEqua­tions.jl).

A re­cent pa­per called Neu­ral Or­di­nary Differ­en­tial Equa­tions ex­plored the prob­lem of pa­ram­e­ter­iz­ing a term in an ODE in­stead of spec­i­fy­ing a se­quence of lay­ers, us­ing ML tech­niques. The benefit of the com­bined frame­work is that it al­lows us to in­cor­po­rate what we already know about the struc­ture of the prob­lem via the ODE, and ap­ply neu­ral net­works to the parts we don’t.

I’m not deeply fa­mil­iar with ei­ther frame­work, but I have been fol­low­ing Differ­en­tialEqua­tions.jl for a while be­cause I had hoped it would let me tackle a few prob­lems I have had my eye on.

~~~~

I feel like Geo­met­ric Alge­bra is point­ing in a similar di­rec­tion to what you de­scribe. It is not di­rectly an ap­plied frame­work, but rather a kind of ur-frame­work for teach­ing and de­vel­op­ing other frame­works on top of it. The idea is that they make ge­o­met­ric ob­jects el­e­ments of the alge­bra, which re­ally boils down to mak­ing the math match the struc­ture of space and in this way pre­serve ge­o­met­ric in­tu­itions through­out. Its ad­vo­cates hope it will serve as a unified frame­work for physics and en­g­ineer­ing ap­pli­ca­tions. The pri­mary the­o­rist is David Hestenes, and his ar­gu­ment for why this was nec­es­sary is found here; there are sur­veys/​tu­to­ri­als from other peo­ple who have done work in the field here and here.

Geo­met­ric Alge­bra and its ex­ten­sion Geo­met­ric Calcu­lus are both still pretty firmly in the ‘re­or­ga­niz­ing math’ kind of en­deavor, with lots of pri­or­ity put on prov­ing re­sults and de­vel­op­ing al­gorithms. But there are a few ar­eas where they are grow­ing into an ap­plied frame­work of the kind you de­scribe, speci­fi­cally robotics, com­puter graph­ics, and com­puter vi­sion.

It was through the robotics ap­pli­ca­tions that I found it; they promised an in­tu­itive ex­pla­na­tion of quater­nions, which oth­er­wise were just this scary way to calcu­late mo­tion.

The lat­ter; it could be any­thing, and by say­ing the prob­a­bil­ities were 1.0 and 0.0, the origi­nal prob­lem de­scrip­tion left out the in­for­ma­tion that would de­ter­mine it.

Thanks for the an­swer! i was some­what amused to see that it ends up be­ing a zero di­vided by zero.

Does the ra­tio be­tween 1ep­silon over 2ep­silon be­ing un­defined means that it’s ar­bi­trar­ily close to half (since 1 over two is half, but that wouldn’t be ex­actly it)? or means that we get the same prob­lem i speci­fied in the ques­tion, where it could be any­thing from (al­most) 0 to (al­most) 1 and we have no idea what ex­actly?

Hi Iwan,

I per­son­ally find this sub­ject (prag­ma­tism gen­er­ally) in­ter­est­ing, but oth­ers here might not. If you’re go­ing to link to an ex­ter­nal source, could you please write a post de­tailing what is be­ing ex­plained in the ex­ter­nal source and why peo­ple here might find it rele­vant plus why they should care about it? I strong-down­voted your post be­cause sim­ply link-post­ing with no con­text or ex­pla­na­tion about why the topic is rele­vant/​why peo­ple should care, is al­most always a non-use­ful thing for peo­ple, and if one is go­ing to post things here, they ought to be use­ful for the com­mu­nity.

Cheers

If you do out the alge­bra, you get that P(H|E) in­volves di­vid­ing zero by zero:

There are two ways to look at this at a higher level. The first is that the alge­bra doesn’t re­ally ap­ply in the first place, be­cause this is a do­main er­ror: 0 and 1 aren’t prob­a­bil­ities, in the same way that the string “hello” and the color blue aren’t.

The sec­ond way to look at it is that when we say $P\left(H\right)=1.0$ and $P\left(E|H\right)=0$, what we re­ally meant was that $P\left(H\right)=1.0-{ϵ}_1$ and $P\left(E|H\right)=0+{ϵ}_2$; that is, they aren’t pre­cisely one and zero, but they differ from one and zero by an un­speci­fied, very small amount. (In­finites­i­mals are like in­fini­ties; $ϵ$ is ar­bi­trar­ily-close-to-zero in the same sense that an in­finity is ar­bi­trar­ily-large). Un­der this in­ter­pre­ta­tion, we don’t have a con­tra­dic­tion, but we do have an un­der­speci­fied prob­lem, since we need the ra­tio $\frac{{ϵ}_1}{{ϵ}_2}$ and haven’t speci­fied it.

• AI sys­tems end up con­trol­led by a group of hu­mans rep­re­sent­ing a small range of hu­man val­ues (ie. an ide­olog­i­cal or re­li­gious group that im­poses val­ues on ev­ery­one else). While not caused only by AI de­sign, it is pos­si­ble that de­sign de­ci­sions could im­pact the like­li­hood of this sce­nario (ie. at what point are val­ues loaded into the sys­tem/​how many peo­ple’s val­ues are loaded into the sys­tem), and is rele­vant for over­all strat­egy.

• Failure to learn how to deal with al­ign­ment in the many-hu­mans, many-AIs case even if sin­gle-hu­man, sin­gle-AI al­ign­ment is solved (which I think An­drew Critch has talked about). For ex­am­ple, AIs ne­go­ti­at­ing on be­half of hu­mans take the stance de­scribed in https://​arxiv.org/​abs/​1711.00363 of agree­ing to split con­trol of the fu­ture ac­cord­ing to which hu­man’s pri­ors are most ac­cu­rate (on po­ten­tially ir­rele­vant is­sues) if this isn’t what hu­mans ac­tu­ally want.

Is this fu­ture AI catas­tro­phe? Or is this just a de­scrip­tion of cur­rent events be­ing a gen­eral grad­ual col­lapse?

This seems like what is hap­pen­ing now, and has been for a while. Ex­ist­ing ML sys­tems are clearly mak­ing Type-I prob­lems, already quite bad be­fore ML was a thing at all, much worse, to the ex­tent that I don’t see much abil­ity left of our civ­i­liza­tion to get any­thing that can’t be mea­sured in a short term feed­back loop—even in spaces like this, ap­peals to non-mea­surable or non-ex­plicit con­cerns are a near-im­pos­si­ble sell.

Part II prob­lems are not yet com­ing from ML sys­tems, ex­actly, But we cer­tainly have al­gorithms that are effec­tively op­ti­mized and se­lected for the abil­ity to gain in­fluence; the al­gorithm gains in­fluence, which causes peo­ple to care about it and feed into it, caus­ing it to get more. If we get less di­rect in the metaphor we get the same thing with memet­ics, cul­ture, life strate­gies, cor­po­ra­tions, me­dia prop­er­ties and so on. The em­pha­sis on choos­ing win­ners, be­ing ‘on the right side of his­tory’, sup­port­ing those who are good at get­ting sup­port. OP notes that this hap­pens in non-ML situ­a­tions ex­plic­itly, and there’s no clear di­vid­ing line in any case.

So if there is an­other the­ory that says, this has already hap­pened, what would one do next?

On the “a day in hell can­not be out­weighed” ques­tion, do you have any anal­y­sis of that in­tu­ition? Are you as­sum­ing that you’ll re­mem­ber that day and be bro­ken by it, or is there some other nega­tive value you’re putting on it. Do you eval­u­ate “a day in hell, 1000 years in heaven, then ter­mi­na­tion” differ­ently from “1000 years in heaven, then a day in hell, then ter­mi­na­tion”? How about “a day in hell, mind-re­set to prior state, then 1000 years in heaven”?

The rea­son I ask is that I’m try­ing to un­der­stand what’s be­ing eval­u­ated. Are you com­par­ing value of in­stan­ta­neous ex­pe­rience in­te­grated over time, or are you com­par­ing value of effect that ex­pe­riences have on your iden­tity?