Thanks for the good work!

I es­pe­cially like the choice set pa­per.

Ah, yeah, that’s a great point. Although I think act-based agents is a pretty bad name, since those agents may of­ten carry out a whole bunch of acts in a row—in fact, I think that’s what made me over­look the fact that it’s point­ing at the right con­cept. So not sure if I’m com­fortable us­ing it go­ing for­ward, but thanks for point that out.

Thanks for shar­ing!

Most model-based RL al­gorithms I’ve seen as­sume they can eval­u­ate the re­ward func­tions in ar­bi­trary states.

Hmm. AlphaZero can eval­u­ate the true re­ward func­tion in ar­bi­trary states. MuZero can’t—it tries to learn the re­ward func­tion by su­per­vised learn­ing from ob­ser­va­tions of past re­wards (if I un­der­stand cor­rectly). I googled “model-based RL Atari” and the first hit was this which like­wise tries to learn the re­ward func­tion by su­per­vised learn­ing from ob­ser­va­tions of past re­wards (if I un­der­stand cor­rectly). I’m not in­ti­mately fa­mil­iar with the deep RL liter­a­ture, I wouldn’t know what’s typ­i­cal and I’ll take your word for it, but it does seem that both pos­si­bil­ities are out there.

Any­way, I don’t think the neo­cor­tex can eval­u­ate the true re­ward func­tion in ar­bi­trary states, be­cause it’s not a neat math­e­mat­i­cal func­tion, it in­volves messy things like the out­puts of mil­lions of pain re­cep­tors, hor­mones slosh­ing around, the in­put-out­put re­la­tion­ships of en­tire brain sub­sys­tems con­tain­ing tens of mil­lions of neu­rons, etc. So I pre­sume that the neo­cor­tex tries to learn the re­ward func­tion by su­per­vised learn­ing from ob­ser­va­tions of past re­wards—and that’s the whole thing with TD learn­ing and dopamine.

I added a new sub-bul­let at the top to clar­ify that it’s hard to ex­plain by RL un­less you as­sume the plan­ner can query the ground-truth re­ward func­tion in ar­bi­trary hy­po­thet­i­cal states. And then I also added a new para­graph to the “other pos­si­ble ex­pla­na­tions” sec­tion at the bot­tom say­ing what I said in the para­graph just above. Thank you.

I don’t see how you solve this prob­lem in gen­eral in a sam­ple-effi­cient man­ner oth­er­wise.

Well, the rats are try­ing to do the re­ward­ing thing af­ter zero sam­ples, so I don’t think “sam­ple-effi­ciency” is quite the right fram­ing.

In ML to­day, the re­ward func­tion is typ­i­cally a func­tion of states and ac­tions, not “thoughts”. In a brain, the re­ward can de­pend di­rectly on what you’re imag­in­ing do­ing or plan­ning to do, or even just what you’re think­ing about. That’s my pro­posal here.

Well, I guess you could say that this is still a “nor­mal MDP”, but where “hav­ing thoughts” and “hav­ing ideas” etc. are part of the state /​ ac­tion space. But any­way, I think that’s a bit differ­ent than how most ML peo­ple would nor­mally think about things.

After reread­ing the chap­ter in Su­per­in­tel­li­gence, it seems to me that “ge­nie” cap­tures some­thing akin to act-based agents. Do you think that’s the main way to use this con­cept in the cur­rent state of the field, or do you have other ap­pli­ca­tions in mind?

I used your idea of “a” vs. “an” as the ba­sis of a GPT-3 ex­per­i­ment to show that GPT-3 in­deed prob­a­bly does do looka­head. De­tails are at https://​​www.red­dit.com/​​r/​​GPT3/​​com­ments/​​k0mvf3/​​ex­per­i­ment_that_shows_that_gpt3_can_prob­a­bly_plan/​​

Ah, looks like I missed this ques­tion for quite a while!

I agree that it’s not quite one or the other. I think that like wire­head­ing, we can split delu­sion box into “the easy prob­lem” and “the hard prob­lem”. The easy delu­sion box is solved by mak­ing a re­ward/​util­ity which is model-based, and so, knows that the delu­sion box isn’t real. Then, much like ob­ser­va­tion-util­ity func­tions, the agent won’t think en­ter­ing into the delu­sion box is a good idea when it’s plan­ning—and also, won’t get any re­ward even if it en­ters into the delu­sion box ac­ci­den­tally (so long as it knows this has hap­pened).

But the hard prob­lem of delu­sion box would be: we can’t make a perfect model of the real world in or­der to have model-based avoidance of the delu­sion box. So how to we guaran­tee that an agent avoids “gen­er­al­ized delu­sion boxes”?

This is clar­ify­ing, thanks.

WRT the last para­graph, I’m think­ing in terms of con­ver­gent vs di­ver­gent pro­cesses. So , fixed points I guess.

The prob­lem of figur­ing out prefer­ence with­out wire­head­ing seems very similar to the prob­lem of main­tain­ing fac­tual knowl­edge about the world with­out suffer­ing from ap­peals to con­se­quences. In both cases a spe­cial­ized part of agent de­sign (model of prefer­ence or model of a fact in the world) has a pur­pose (ac­cu­rate mod­el­ing of its refer­ent) whose pur­suit might be at odds with con­se­quen­tial­ist de­ci­sion mak­ing of the agent as a whole. The de­sired out­come seems to in­volve main­tain­ing in­tegrity of the spe­cial­ized part, re­sist­ing cor­rup­tion of con­se­quen­tial­ist rea­son­ing.

With this anal­ogy, it might be pos­si­ble to trans­fer les­sons from the more fa­mil­iar prob­lem of learn­ing facts about the world, to the harder prob­lem of pro­tect­ing prefer­ence.

Per­haps the les­son is that ter­minol­ogy that is ac­cept­able in one field (in this case philos­o­phy) might not be suit­able in an­other (in this case ma­chine learn­ing).

Is that from Su­per­in­tel­li­gence? I googled it, and that was the most con­vinc­ing re­sult.

Or­a­cle-ge­nie-sovereign is a re­ally use­ful dis­tinc­tion that I think I (and prob­a­bly many oth­ers) have avoided us­ing mainly be­cause “ge­nie” sounds un­pro­fes­sional/​un­aca­demic. This is a real shame, and a good les­son for fu­ture ter­minol­ogy.

Oh, ac­tu­ally, you’re right (that you were wrong). I think I made the same mis­take in my pre­vi­ous com­ment. Good catch.

Well now I’m less sure that it’s in­cor­rect. I was origi­nally imag­in­ing that like in Solomonoff in­duc­tion, the TMs ba­si­cally di­rectly con­trol­led AIXI’s ac­tions, but that’s not right: there’s an ex­pec­ti­max. And if the TMs re­in­force ac­tions by shap­ing the re­wards, in the AIXI for­mal­ism you learn that im­me­di­ately and throw out those TMs.

Wait, re­ally? I thought it made sense (al­though I’d con­tend that most peo­ple don’t think about AIXI in terms of those TMs re­in­forc­ing hy­pothe­ses, which is the point I’m mak­ing). What’s in­cor­rect about it?

Also, pre­sum­ably many of the com­po­nent Tur­ing ma­chines have reusable pa­ram­e­ters and re­in­force be­havi­our, al­tho this is hid­den by the for­mal­ism.

Ac­tu­ally I think this is to­tal non­sense pro­duced by me for­get­ting the differ­ence be­tween AIXI and Solomonoff in­duc­tion.

Thanks for the clar­ifi­ca­tion! I agree if the plan­ner does not have ac­cess to the re­ward func­tion then it will not be able to solve it. Though, as you say, it could ex­plore more given the un­cer­tainty.

Most model-based RL al­gorithms I’ve seen as­sume they can eval­u­ate the re­ward func­tions in ar­bi­trary states. More­over, it seems to me like this is the key thing that lets rats solve the prob­lem. I don’t see how you solve this prob­lem in gen­eral in a sam­ple-effi­cient man­ner oth­er­wise.

One class of model-based RL ap­proaches is based on [model-pre­dic­tive con­trol](https://​​en.wikipe­dia.org/​​wiki/​​Model_pre­dic­tive_con­trol): sam­ple ran­dom ac­tions, “rol­lout” the tra­jec­to­ries in the model, pick the tra­jec­tory that had the high­est re­turn and then take the first ac­tion from that tra­jec­tory, then re­plan. That said, as­sump­tions vary. [iLQR](https://​​en.wikipe­dia.org/​​wiki/​​Lin­ear%E2%80%93quadratic_reg­u­la­tor) makes the stronger as­sump­tion that re­ward is quadratic and differ­en­tiable.

I think meth­ods based on [Monte Carlo tree search](https://​​en.wikipe­dia.org/​​wiki/​​Monte_Carlo_tree_search) might ex­hibit some­thing like the prob­lem you dis­cuss. Since they sam­ple ac­tions from a policy trained to max­i­mize re­ward, they might end up not ex­plor­ing enough in this novel state if the policy is very con­fi­dent it should not drink the salt wa­ter. That said, they typ­i­cally in­clude ex­plicit meth­ods for ex­plo­ra­tion like [UCB](https://​​en.wikipe­dia.org/​​wiki/​​Thomp­son_sam­pling#Up­per-Con­fi­dence-Bound_(UCB)_al­gorithms) which should miti­gate this.

Oh, OK, I see what you mean. Pos­si­bly re­lated: my com­ment here.

Yes we do: train­ing is our evolu­tion­ary his­tory, de­ploy­ment is an in­di­vi­d­ual life­time. And our genomes are our reusable pa­ram­e­ters.

Un­for­tu­nately I haven’t yet writ­ten any pa­pers/​posts re­ally lay­ing out this anal­ogy, but it’s pretty cen­tral to the way I think about AI, and I’m work­ing on a bunch of re­lated stuff as part of my PhD, so hope­fully I’ll have a more com­plete ex­pla­na­tion soon.

I kind of think the lack of epi­sodes makes it more re­al­is­tic for many prob­lems, but ad­mit­tedly not for simu­lated games. Also, pre­sum­ably many of the com­po­nent Tur­ing ma­chines have reusable pa­ram­e­ters and re­in­force be­havi­our, al­tho this is hid­den by the for­mal­ism. [EDIT: I re­tract the sec­ond sen­tence]