# crabman

Karma: 227
Page 1
• Ok. It’s just that when I learned that, we didn’t even talk about dual spaces in lin­ear alge­braic sense, we worked just fine in .

• I won­der, what do you think about the chap­ter about dual spaces, dual maps, an­nihila­tor, etc.? To me it seemed not too con­nected with ev­ery­thing else, and that’s bad. If I re­mem­ber cor­rectly, the au­thor uses du­al­ity just to prove a few re­sults and then throws du­al­ity away and never uses it again. Also in real life (nu­mer­i­cal lin­ear alge­bra, ma­chine learn­ing, and stuff) I am not aware of any use for those con­cepts.

So for “gen­eral” op­er­a­tors, this is always true, but there do ex­ist spe­cific op­er­a­tors for which it isn’t.

I be­lieve when math­e­mat­i­ci­ans say that in gen­eral P(x) holds, they mean that for any x in the do­main of in­ter­est P(x) holds. Per­haps you want to you typ­i­cal in­stead of gen­eral here. E.g. there is a no­tion called typ­i­cal ten­sor rank of ten­sors of given shape, which means a ten­sor rank which oc­curs with non-zero prob­a­bil­ity when a ran­dom ten­sor of given shape is sam­pled.

• in­clud­ing 20% who think the Sun re­volves around the Earth

The sun does re­volve around the Earth. The sun re­volv­ing around the earth is equiv­a­lent to the Earth re­volv­ing around the sun.

• Why the gov­ern­ment? Per­haps a gov­ern­ment?

If I loan you an item os­ten­si­bly for a month, and re­gret it, I will do sig­nifi­cantly less dam­age ask­ing you to re­turn it in a week than ask­ing you to re­turn it im­me­di­ately.

So, a good samar­i­tan Alice gave her friend John an item for some time, then she re­al­ized she wants to use the item her­self. It’s already the case that only John is get­ting some­thing out of this agree­ment, so why should Alice take on ad­di­tional costs of wait­ing an­other week?

Edit: un­less John ac­tu­ally gave Alice money for bor­row­ing her item. IMO peo­ple should pay each other money for var­i­ous acts that provide value much more of­ten than the do.

# Reneg­ing proso­cially by Dun­can Sabien

18 Jun 2019 18:52 UTC
59 points
(medium.com)

# [Question] How to de­ter­mine if my sym­pa­thetic or my parasym­pa­thetic ner­vous sys­tem is cur­rently dom­i­nant?

31 May 2019 20:40 UTC
20 points
• It seems “Com­putabil­ity and Logic” doesn’t in­clude Kleene’s re­cur­sion the­o­rem and Rice’s the­o­rem. What sources would you recom­mend for learn­ing those the­o­rems, their proofs, and their corol­laries? Also, which chap­ters of “Com­putabil­ity and Logic” are re­quired to un­der­stand them?

• I think this ar­ti­cle is too vague, be­cause for al­most al­most claims in it I am not sure if I un­der­stand the au­thor cor­rectly. Below I am post­ing my notes. If you want to help me and oth­ers clar­ify un­der­stand­ing of this ar­ti­cle, con­sider an­swer­ing ques­tions in bold, or, if you see a mis­take in my notes, cor­rect­ing it. Also I hope my notes help the au­thor as a piece of feed­back. I’ve only finished 23 of the ar­ti­cle so far, but post­ing notes be­cause I might be­come less in­ter­ested in this later.

Also it’s un­for­tu­nate that un­like in https://​​in­tel­li­gence.org/​​2018/​​11/​​02/​​em­bed­ded-mod­els/​​ ver­sion of this ar­ti­cle we don’t have hy­per­links to ex­pla­na­tions of var­i­ous con­cepts here. Per­haps you could add them un­der the cor­re­spond­ing images? Or have images them­selves be hy­per­links or refer­ence links (like in aca­demic ar­ti­cles) to the bot­tom of the doc­u­ment where all rele­vant links would be stored grouped by image num­ber.

The post says an em­bed­ded agent can’t hold an ex­act model of the en­vi­ron­ment in its head, can’t think through the con­se­quences of ev­ery po­ten­tial course of ac­tion, can’t hold in its head ev­ery pos­si­ble way the en­vi­ron­ment could be. I think this may not be nec­es­sar­ily true and I am not sure what as­sump­tions the au­thor used here.

It seems the whole ar­ti­cle as­sumes countable prob­a­bil­ity spaces (even be­fore the AIXI part). I won­der why and I won­der how re­al­iz­abil­ity is defined for un­countable prob­a­bil­ity space.

--

Re­gard­ing rel­a­tive bounded loss and what this bound is for, my best guess is as fol­lows. Here I use non-con­di­tional prob­a­bil­ity no­ta­tion in­stead of . Sup­pose some event e is ac­tu­ally true. Let be some “ex­pert” event in the prob­a­bil­ity space. Ac­cord­ing to prior, prob­a­bil­ity of e equals , and its log­a­r­ithm of prob­a­bil­ity has a lower bound . Now, ac­cord­ing to the ex­pert h, its prob­a­bil­ity equals just , and its log­a­r­ithm of prob­a­bil­ity equals . I con­clude that rel­a­tive bounded loss is the differ­ence be­tween prior log­a­r­ithm of prob­a­bil­ity and log­a­r­ithm of prob­a­bil­ity of the ex­pert h, which turns out to be at most .

Ini­tially, is your ini­tial trust in ex­pert h, and in each case where it is even a lit­tle bit more cor­rect than you, you in­crease your trust ac­cord­ingly; the way you do this en­sures you as­sign an ex­pert prob­a­bil­ity 1 and hence copy it pre­cisely be­fore you lose more than com­pared to it.

Re­mem­ber, . It fol­lows that prob­a­bil­ity of h in­creases given ev­i­dence e if and only if , i.e. h “is even a lit­tle bit more cor­rect than you”. But I don’t un­der­stand the bit about copy­ing the ex­pert h pre­cisely be­fore los­ing more than , be­cause los­ing more than is log­i­cally im­pos­si­ble (as­sum­ing ), as was shown above.

Com­bin­ing this with the pre­vi­ous idea about view­ing Bayesian learn­ing as a way of al­lo­cat­ing “trust” to “ex­perts” which meets a bounded loss con­di­tion, we can see the Solomonoff prior as a kind of ideal ma­chine learn­ing al­gorithm which can learn to act like any al­gorithm you might come up with, no mat­ter how clever.

It is as­sum­ing all pos­si­ble al­gorithms are com­putable, not that the world is.

I don’t un­der­stand this. Our prob­a­bil­ity space is the carte­sian product of the set of all pos­si­ble UTM pro­grams and the set of all pos­si­ble UTM work­ing tape ini­tial con­figu­ra­tions. Or, equiv­a­lently, the set of out­puts of UTM un­der these con­di­tions. Hence our whole hy­poth­e­sis space only in­cludes com­putable wor­lds. What does “can learn to act like any al­gorithm” mean here? “It’s get­ting bounded loss on its pre­dic­tive ac­cu­racy as com­pared with any com­putable pre­dic­tor.” Huh? Does pre­dic­tor here mean ex­pert h? If yes, what does it mean that h is com­putable and why? All in all, is the au­thor claiming it’s im­pos­si­ble to have a bet­ter com­putable pre­dic­tor than AIXI with Solomonoff prior, even if it has non-com­putable wor­lds in the prob­a­bil­ity space?

prob­a­bil­ities may not be cal­ibrated iden­ti­fi­ca­tion of causal struc­ture may not work

What do these mean? I only know in­for­mally what cal­ibra­tion means re­lated to fore­cast­ing.

So, does AIXI perform well with­out a re­al­iz­abil­ity as­sump­tion?

How is AIXI even defined with­out re­al­iz­abil­ity, i.e. when the ac­tual world isn’t in the prob­a­bil­ity space, or it has zero prior prob­a­bil­ity?

This is fine if the world “holds still” for us; but be­cause the map is in the world, it may im­ple­ment some func­tion.

Is this about the world chang­ing be­cause of the agent just think­ing? Or some­thing else?

It should be noted, though, that there are ad­di­tional bar­ri­ers to get­ting this prop­erty in a game-the­o­retic set­ting; so in their com­mon us­age cases, “grain of truth” is tech­ni­cally de­mand­ing while “re­al­iz­abil­ity” is a tech­ni­cal con­ve­nience.

...

In game the­ory, on the other hand, the as­sump­tion it­self may be in­con­sis­tent. This is be­cause games com­monly yield para­doxes of self-refer­ence.

From the former para­graph I don’t un­der­stand any­thing ex­cept that (the au­thor claims) game the­ory has more prob­lems with grain of truth /​ re­al­iz­abil­ity, than AIXI. After the lat­ter para­graph, my best guess is: for any game, if there is no pure strat­egy equil­ibrium in it, then we say it has no grain of truth, be­cause for ev­ery pos­si­ble out­come ra­tio­nal agents wouldn’t choose it.

If we put weight in both places un­til a proof rules one out, the be­liefs just os­cillate for­ever rather than do­ing any­thing use­ful.

Weights rep­re­sent pos­si­ble wor­lds, there­fore they are on the scales right from the be­gin­ning (the prior), we never put new weights on the scales. My prob­a­bly in­cor­rect guess of what the au­thor is say­ing is some agent which acts like AIXI but in­stead of up­dat­ing on pieces of ev­i­dence as soon as he re­ceives it, he stock­piles it, and at some points he (bound­edly) searches for proofs that these pieces of ev­i­dence are in fa­vor of some hy­poth­e­sis and performs up­date only when he finds them. But still, why os­cilla­tion?

Any com­putable be­liefs about logic must have left out some­thing, since the tree will grow larger than any con­tainer.

I in­ter­pret it as there are in­finitely many the­o­rems, hence an agent with finite amount of space or finite amount of com­pu­ta­tion steps can’t pro­cess all of them.

Another con­se­quence of the fact that the world is big­ger than you is that you need to be able to use high-level world mod­els: mod­els which in­volve things like ta­bles and chairs.

This is re­lated to the clas­si­cal sym­bol ground­ing prob­lem; but since we want a for­mal anal­y­sis which in­creases our trust in some sys­tem, the kind of model which in­ter­ests us is some­what differ­ent. This also re­lates to trans­parency and in­formed over­sight: world-mod­els should be made out of un­der­stand­able parts.

No idea what the sec­ond quoted para­graph means.

All in all, I doubt that high level world mod­els are nec­es­sary. And it’s very not clear what is meant by “high level” or “things” here. Per­haps em­bed­ded agents can (bound­edly) rea­son about the world in other ways, e.g. by mod­el­ing only part of the world.

https://​​in­tel­li­gence.org/​​files/​​On­tolog­i­calCrises.pdf ex­plains the on­tolog­i­cal crisis idea bet­ter. Sup­pose our AIXI-like agent thinks the world is an el­e­men­tary out­come of some pa­ram­e­ter­ized prob­a­bil­ity dis­tri­bu­tion with the pa­ram­e­ter θ. θ is ei­ther 1 or 2. We call the set of el­e­men­tary out­comes with θ=1 the first on­tol­ogy (e.g. pos­si­ble wor­lds run­ning on clas­si­cal me­chan­ics), and the set of el­e­men­tary out­comes with θ=2 the sec­ond on­tol­ogy (e.g. pos­si­ble wor­lds run­ning on su­per­strings the­ory). The pro­gram­mer has only pro­grammed the agent’s util­ity func­tiom for θ=1 part, i.e. a u func­tion from on­tol­ogy 1 to real num­bers. The agent keeps count of which value of θ is more prob­a­ble and chooses ac­tions by con­sid­er­ing only cur­rent on­tol­ogy. If at some point he de­cides that the sec­ond on­tol­ogy is more use­ful, he switches to it. The agent should ex­trap­o­late the util­ity func­tion to θ=2 part. How can he do it?

• maybe re­cent ma­chine learn­ing top­ics are a point of com­par­a­tive advantage

Do you mean re­cent ML top­ics re­lated to AI safety, or just re­cent ML top­ics?

RAISE is already work­ing on the former, it’s an­other course which we in­ter­nally call “main track”. Right now it has the fol­low­ing um­brella top­ics: In­verse Re­in­force­ment Learn­ing; Iter­ated Distil­la­tion and Am­plifi­ca­tion; Cor­rigi­bil­ity. See https://​​www.aisafety.info/​​on­line-course

# AI Safety Pr­ereq­ui­sites Course: Re­vamp and New Lessons

3 Feb 2019 21:04 UTC
33 points
• Is the point of your com­ment that you think peo­ple very rarely read (com­pletely or al­most com­pletely) 3 books in one field?

(if yes, then I agree)

• I find your pre­dic­tions 1 through 3 not clearly defined.

Does OpenAI bot need to defeat a pro team in un­con­strained dota 2 at least once dur­ing 2019? Or does it need to win at least one and more than 50% games against pro teams in 2019?

Sup­pose tesla re­leases a video footage or a re­port of their car reach­ing from one coast to the other, but it had some minor or not so minor prob­lems. How minor should they be to count? Are hu­mans al­lowed to help it recharge or any­thing like that?

How do you define “skil­led” in SC II?

• How did you con­clude that peo­ple who pre­pared GS are ac­tu­ally more likely to help than other peo­ple? Just from eye­bal­ling 1019 and 621 I can’t con­clude that this is enough ev­i­dence, only that this is sug­ges­tive.

• Could you please elab­o­rate what kind of cul­ture fit MIRI re­quire?

• What is the point of spend­ing a sec­tion on dual maps, I won­der? Is the sole pur­pose to show that row rank equals column rank, I won­der? If so, then a lot of my time spent on ex­er­cises on dual maps might be wasted.

• You say

Epistemic Sta­tus: Opinions stated with­out justification

but from the text it seems you be­lieve that act­ing ac­cord­ing to the de­scribed opinions is use­ful and that many of them are true. I don’t like this, I think you should clar­ify epistemic sta­tus.

# Fun­da­men­tals of For­mal­i­sa­tion Level 7: Equiv­alence Re­la­tions and Orderings

10 Aug 2018 15:12 UTC
9 points