Your re­sults don’t seem to show up­per bounds on the amount of hard­ware over­hang, or very strong lower bounds. What con­crete progress has been made in chess play­ing al­gorithms since deep blue? As far as I can tell, it is a col­lec­tion of rea­son­ably small, chess spe­cific tricks. Bet­ter open­ing libraries, low level perfor­mance tricks, tricks for fine tun­ing eval­u­a­tion func­tions. Ways of rep­re­sent­ing chess­boards in mem­ory that shave 5 pro­ces­sor cy­cles off com­put­ing a knights move ect.

P=PSPACE is still an open con­jec­ture. Chess is brute forcible in PSPACE. If P=PSPACE, and the over­head is a small con­stant, then it might be pos­si­ble to pro­gram a vac­uum tube ma­chine to play perfect chess. Alter­nately, show­ing that stock­fish is or­ders of mag­ni­tude bet­ter than deep blue, doesn’t ac­tu­ally show that there is a hard­ware over­head for chess. It shows that there was one when deep blue was cre­ated.

Also note that there never was a hard­ware over­hang for in­te­ger bi­nary ad­di­tion. The prob­lem is sim­ple enough that the first an­swer that any rea­son­ably smart per­son can come up with is nearly op­ti­mal. (It is fairly straight­for­ward to do ad­di­tion with only a few logic gates per in­put, and as the out­put de­pends bit­wise on all in­puts (chang­ing any sin­gle bit changes the out­put) then you need at least one logic gate per in­put. ) It is plau­si­ble that play­ing chess is a prob­lem sim­ple enough that we have figured out how to do it nearly op­ti­mally, whereas on other prob­lems there is a hard­ware over­hang.

If we as­sume that the ex­pert hu­man brain is about equally effi­cient at AI de­sign, gen­eral pro­gram­ming, airo­plane en­g­ineer­ing and a va­ri­ety of other STEM tasks. (A plau­si­ble as­sump­tion, given that these tasks seem similarly far from the en­vi­ron­ment of evolu­tion­ary adapt­ed­ness, ) Then it should take a similar amount of com­pute to dis­play top hu­man perfor­mance in these, as in chess. On the other hand, do­ing ar­ith­metic used to be a skil­led job, and the com­pute needed for su­per­hu­man chess (with cur­rent al­gorithms) is way higher than that needed for ar­ith­metic.

Right, it’s for ran­domly dis­tributed re­wards. But if I show a prop­erty holds for re­ward func­tions gener­i­cally, then it isn’t nec­es­sar­ily enough to say “we’re go­ing to try to try to provide goals with­out that prop­erty”. Can we provide re­ward func­tions with­out that prop­erty?

Every spe­cific at­tempt so far has been seem­ingly un­suc­cess­ful (un­less you want the AI to choose a policy at ran­dom or shut down im­me­di­ately). The hope might be that fu­ture goals/​ca­pa­bil­ity re­search will help, but I’m not per­son­ally con­vinced that re­searchers will re­ceive good Bayesian ev­i­dence via their sub­hu­man-AI ex­per­i­men­tal re­sults.

I agree it’s rele­vant that we will try to build helpful agents, and might nat­u­rally get bet­ter at that. I don’t know that it makes me feel much bet­ter about fu­ture ob­jec­tives be­ing outer al­igned.

ETA: also, i was refer­ring to the point you made when i said

“the re­sults don’t prove how hard it is tweak the re­ward func­tion dis­tri­bu­tion, to avoid in­stru­men­tal con­ver­gence”

Now the per­haps harder step is try­ing to get trac­tion on them

Yes, very much so. We’re work­ing on a few parts of this now, as part of a differ­ent pro­ject, but I agree that it’s tricky. And there are a num­ber of other things that seem like po­ten­tially very use­ful pro­jects if oth­ers are in­ter­ested in col­lab­o­ra­tions, or just some ideas /​ sug­ges­tions about how they could be ap­proached.

(On the ta­bles, un­for­tu­nately the ta­bles were pasted in as images from an­other pro­gram. We should definitely see if we can get higher-re­s­olu­tion, even if we can’t con­vert to text eas­ily.)

I hadn’t heard of the Delphi method be­fore, so this pa­per brought it to my at­ten­tion.

It’s nice to see con­crete fore­cast­ing ques­tions laid out in a prin­ci­pled way. Now the per­haps harder step is try­ing to get trac­tion on them ;^).

Note: The ta­bles in pages 9 and 10 are a lit­tle blurry to read. They are also not text, so it’s not easy to copy-paste them into an­other for­mat for bet­ter view­ing. I think it’d be good to up­date the images to ei­ther be clearer or trans­late it into a text table.

I whole­heart­edly agree. I think this im­plies:

1. Get­ting very clear on what we want. Can we give a fairly tech­ni­cal speci­fi­ca­tion of the kind of safety that’s nec­es­sary+pos­si­ble?

2. Some de­gree of safety be­yond tool-type non-ma­lig­nancy. A pro­posal which I keep think­ing about is my con­sent-based helpful­ness. The idea is that, in ad­di­tion to be­liev­ing that you want some­thing (with suffi­cient con­fi­dence), the sys­tem should also be­lieve that you un­der­stand the im­pli­ca­tions of that thing (with some kind of suffi­cient de­tail). In the fu­sion ex­am­ple, the sys­tem would en­gage the user in con­ver­sa­tion un­til it was clear that the con­se­quences for so­ciety were un­der­stood and ap­proved of.

Note that the fu­sion power ex­am­ple could be an­swered di­rectly with a value-al­ign­ment type ap­proach, where you have an agent rather than a tool—the agent in­fers your val­ues, and in­fers that you would not re­ally want back­yard fu­sion power if it put the world at risk. That’s the moral that I imag­ine peo­ple more into value learn­ing would give to your story. But I’m reach­ing fur­ther afield for solu­tions, be­cause:

• Value learn­ing sys­tems could Good­hart on the ap­prox­i­mate val­ues learned

• Value learn­ing sys­tems are not cor­rigible if they be­come overly con­fi­dent (which could hap­pen at test time due to un­fore­seen flaws in the sys­tem’s rea­son­ing—hence the de­sire for cor­rigi­bil­ity)

• Value learn­ing sys­tems could ma­nipu­late the hu­man.

But the the­o­rems are ev­i­dence that RL leads to catas­tro­phe at op­ti­mum, at least.

RL with a ran­domly cho­sen re­ward leads to catas­tro­phe at op­ti­mum.

I proved that that op­ti­mal poli­cies are gen­er­ally power-seek­ing in MDPs.

The proof is for ran­domly dis­tributed re­wards.

Ben’s main cri­tique is that the goals evolve in tan­dem with ca­pa­bil­ities, and goals will be de­ter­mined by what hu­mans care about. Th­ese are spe­cific rea­sons to deny the con­clu­sion of anal­y­sis of ran­dom re­wards.

(A ran­dom Python pro­gram will er­ror with near-cer­tainty, yet some­how I still man­age to write Python pro­grams that don’t er­ror.)

I do agree that this isn’t enough rea­son to say “there is no risk”, but it surely is im­por­tant for de­ter­min­ing ab­solute lev­els of risk. (See also this com­ment by Ben.)

Thanks! I’ll try to read that.

Any of the risks of be­ing like a group of hu­mans, only much faster, ap­ply. There are also the mesa al­ign­ment is­sues. I sus­pect that a suffi­ciently pow­er­ful GPT-n might form de­cep­tively al­igned mesa op­ti­misers.

I would also worry that off dis­tri­bu­tion at­trac­tors could be ma­lign and in­tel­li­gent.

Sup­pose you give GPT-n an off train­ing dis­tri­bu­tion prompt. You get it to gen­er­ate text from this prompt. Some­times it might wan­der back into the dis­tri­bu­tion, other times it might stay off dis­tri­bu­tion. How wide is the bor­der be­tween pro­cesses that are safely im­mi­tat­ing hu­mans, and pro­cesses that aren’t perform­ing sig­nifi­cant op­ti­miza­tion?

You could get “viruses”, pat­terns of text that en­courage GPT-n to re­peat them so they don’t drop out of con­text. GPT-n already has an ac­cu­rate world model, a world model that prob­a­bly mod­els the thought pro­cesses of hu­mans in de­tail. You have all the com­po­nents needed to cre­ate pow­er­ful ma­lign in­tel­li­gences, and a pro­cess that smashes them to­gether in­dis­crim­i­nately.

I some­what hope­ful that this is right, but I’m also not so con­fi­dent that I feel like we can ig­nore the risks of GPT-N.

For ex­am­ple, this post makes the ar­gu­ment that, be­cause of GPT’s de­sign and learn­ing mechanism, we need not worry about it com­ing up with sig­nifi­cantly novel things or out­perform­ing hu­mans be­cause it’s op­ti­miz­ing for imi­tat­ing ex­ist­ing hu­man writ­ing, not say­ing true things. On the other hand, it’s man­ag­ing to do pow­er­ful things it wasn’t trained for, like solve math equa­tions we have no rea­son to be­lieve it saw in the train­ing set or write code hasn’t seen be­fore, which makes it pos­si­ble that even if GPT-N isn’t trained to say true things and isn’t re­ally ca­pa­ble of more than hu­mans are, doesn’t mean it might not func­tion like a Han­so­nian em and still be dan­ger­ous by sim­ply do­ing what hu­mans can do, only much faster.

I recom­mend skip­ping to the next post. This post was kind of a stub, the next one ex­plains the same idea bet­ter.

I am usu­ally rea­son­ably good at trans­lat­ing from math to non-ab­stract in­tu­itive ex­am­ples...but I didn’t have much suc­cess here. Do you have “in English, for sim­ple­tons” ex­am­ple to go with this? :-) (You know, some­thing that uses ap­ples and bis­cuits rather than English-but-ab­stract words like “there are many hid­den vari­ables me­di­at­ing the in­ter­ac­tions be­tween ob­serv­ables” :D.)

Other­wise, my cur­rent ab­stract in­ter­pre­ta­tion of this is some­thing like: “There are de­tailed mod­els, and those might vary a lot. And then there are very ab­stract mod­els, which will be more similar to each other...well, ex­cept that they might also be to­tally use­less.” So I was hop­ing that a more spe­cific ex­am­ple would clar­ify things for a bit and tell me whether there is more to this (and also whether I got it all wrong or not :-).)

If you ask GPT-n to pro­duce a de­sign for a fu­sion re­ac­tor, all the prompts that talk about fu­sion are go­ing to say that a work­ing re­ac­tor hasn’t yet been built, or imi­tate cranks or works of fic­tion.

It seems un­likely that a text pre­dic­tor could pick up enough info about fu­sion to be able to de­sign a work­ing re­ac­tor, with­out figur­ing out that hu­mans haven’t made any fu­sion re­ac­tors that pro­duce net power.

If you did some­how get a re­sponse, the level of safety you would get is the level a typ­i­cal hu­man would dis­play. (con­di­tional on the prompt) If some in­for­ma­tion is an ob­vi­ous in­fo­haz­ard, such that no hu­man ca­pa­ble of com­ing up with it would share it, then such data won’t be in GPT-n ’s train­ing dataset, and won’t be pre­dicted. How­ever, the pro­cess of con­di­tion­ing might am­plify tiny prob­a­bil­ities of hu­man failure.

Sup­pose that any easy de­sign of fu­sion re­ac­tor could be turned into a bomb. And ig­nore cranks and fic­tion. Then sup­pose 99% of peo­ple who in­vented a fu­sion re­ac­tor would re­al­ize this, and stay quiet. The other 1% would write an ar­ti­cle that starts with “To make a fu­sion re­ac­tor …” . Then this prompt will cause GPT-n to gen­er­ate the ar­ti­cle that a hu­man that didn’t no­tice the dan­ger would come up with.

This also ap­plies to dan­gers like leak­ing ra­di­a­tion, or just blow­ing up ran­domly if your ma­te­ri­als weren’t pure enough.

Yeah, this makes much more sense.

The claim here is that ei­ther (a) the AI in ques­tion doesn’t achieve the main value prop of AI (i.e. rea­son­ing about sys­tems too com­pli­cated for hu­mans), or (b) the sys­tem it­self has to do the work of mak­ing sure it’s safe.

I see the in­tu­itive ap­peal of this claim, but it seems too strong. I sus­pect if we look at rates of ac­ci­dents over time they’ll have been go­ing down over time, at least for the last few cen­turies. It seems like this can con­tinue go­ing down, to an asymp­tote of zero, in the same way it has been so far—we be­come bet­ter at un­der­stand­ing how ac­ci­dents hap­pen and more care­ful in how we use dan­ger­ous tech­nolo­gies. We already use tools for this (in soft­ware, we use de­bug­gers, pro­filers, type sys­tems, etc) or del­e­gate to other hu­mans (as in a large com­pany). We can con­tinue to do so with AI sys­tems.

I buy that even­tu­ally “most of the work” has to be done by the AI sys­tem, but it seems plau­si­ble that this won’t hap­pen un­til well af­ter ad­vanced AI, and that ad­vanced AI will help us in get­ting there. And so, that from a what-should-we-do per­spec­tive, it’s fine to rely on hu­mans for some as­pects of safety in the short term (though of course it would be prefer­able to del­e­gate en­tirely to a sys­tem we knew was safe and benefi­cial).

(Why bother rely­ing on hu­mans? If you want to build a goal-di­rected AI sys­tem, it sure seems bet­ter if it’s un­der the con­trol of some hu­man, rather than not. It’s not clear what a plau­si­ble op­tion is if you can’t have the AI sys­tem un­der the con­trol of some hu­man.)

In the die-roll anal­ogy, the hope is the rate at which you roll dice ap­prox­i­mately de­cays ex­po­nen­tially, so that you only roll an asymp­tot­i­cally con­stant num­ber of dice.

You definitely don’t un­der­stand what I’m get­ting at here, but I’m not yet sure ex­actly where the in­duc­tive gap is. I’ll em­pha­size a few par­tic­u­lar things; let me know if any of this helps.

There’s this story about an air­plane (I think the B-52 origi­nally?) where the lev­ers for the flaps and land­ing gear were iden­ti­cal and right next to each other. Pilots kept com­ing in to land, and ac­ci­den­tally re­tract­ing the land­ing gear. The point of the story is that this is a de­sign prob­lem with the plane more than a mis­take on the pi­lots’ part; the prob­lem was fixed by putting a lit­tle rub­ber wheel on the land­ing gear lever. If we put two iden­ti­cal lev­ers right next to each other, it’s ba­si­cally in­evitable that mis­takes will be made; that’s bad in­ter­face de­sign.

AI has a similar prob­lem, but far more se­vere, be­cause the sys­tems to which we are in­ter­fac­ing are far more con­cep­tu­ally com­pli­cated. If we have con­fus­ing in­ter­faces on AI, which al­low peo­ple to shoot the world in the foot, then the world will in­evitably be shot in the foot, just like putting two iden­ti­cal lev­ers next to each other guaran­tees that the wrong one will some­times be pul­led.

For tool AI in par­tic­u­lar, the key piece is this:

the big value-propo­si­tion of pow­er­ful AI is its abil­ity to rea­son about sys­tems or prob­lems too com­pli­cated for hu­mans—which are ex­actly the sys­tems/​prob­lems where safety is­sues are likely to be nonob­vi­ous. If we’re go­ing to un­lock the full value of AI at all, we’ll need to use it on prob­lems where hu­mans do not know the rele­vant safety is­sues.

The claim here is that ei­ther (a) the AI in ques­tion doesn’t achieve the main value prop of AI (i.e. rea­son­ing about sys­tems too com­pli­cated for hu­mans), or (b) the sys­tem it­self has to do the work of mak­ing sure it’s safe. If nei­ther of those con­di­tions are met, then mis­takes will ab­solutely be made reg­u­larly. The hu­man op­er­a­tor can­not be trusted to make sure what they’re ask­ing for is safe, be­cause they will definitely make mis­takes.

On the other hand, if the AI it­self is able to eval­u­ate whether its out­puts are safe, then we can po­ten­tially achieve very high lev­els of safety. It could plau­si­bly never go wrong over the life­time of the uni­verse. Just like, if you de­sign a ta­ble­saw with an au­to­matic shut-off, it could plau­si­bly never cut off any­body’s finger. But if you de­sign a ta­ble­saw with­out an au­to­matic shut-off, it is near-cer­tain to cut off a finger from time to time. That level of safety can be achieved, in gen­eral, but it can­not be achieved while rely­ing on the hu­man op­er­a­tor not mak­ing mis­takes.

Com­ing at it from a differ­ent an­gle: if a safety prob­lem is han­dled by a sys­tem’s de­signer, then their die-roll hap­pens once up-front. If that die-roll comes out fa­vor­ably, then the sys­tem is safe (at least with re­spect to the prob­lem un­der con­sid­er­a­tion); it avoids the prob­lem by de­sign. On the other hand, if a safety prob­lem is left to the sys­tem’s users, then a die-roll hap­pens ev­ery time the sys­tem is used, so in­evitably some of those die rolls will come out un­fa­vor­ably. Thus the im­por­tance of de­sign­ing AI for safety up-front, rather than rely­ing on users to use it safely.

Is it more clear what I’m get­ting at now and/​or does this prompt fur­ther ques­tions?

I feel like I don’t un­der­stand what you’re get­ting at here. It seems to me like you’re say­ing “X can­not be guaran­teed to never cause bad things”, which seems to me to be so ob­vi­ous that it’s not worth men­tion­ing.

Noth­ing is in­her­ently safe. There’s always the pos­si­bil­ity that you’ve ne­glected some way by which dan­ger could arise. What I want from ar­gu­ments for risk is an ar­gu­ment that it is high EV to ad­dress the risk (of­ten bro­ken down into im­por­tance, tractabil­ity, ne­glect­ed­ness), or at least that the risk is rea­son­ably high on prob­a­bil­ity * harm (which fo­cuses just on im­por­tance).

Of course Tool AI isn’t in­her­ently safe. A knife isn’t in­her­ently safe, de­spite be­ing a tool; you wouldn’t give a knife to a baby. The ar­gu­ment is usu­ally that Tool AI would, by de­sign, not be ad­ver­sar­i­ally op­ti­miz­ing against hu­mans, be­cause it isn’t op­ti­miz­ing at all, and so a par­tic­u­lar class of risks typ­i­cally con­sid­ered in the AI safety com­mu­nity would be avoided. It is un­clear whether such an ar­gu­ment can work, but it’s cer­tainly not “it is im­pos­si­ble for bad things to hap­pen if you build a Tool AI”, which is clearly wrong.

Per­haps you’re ar­gu­ing that we should fo­cus on the “how do we man­age po­ten­tially dan­ger­ous in­for­ma­tion” prob­lem? Or that we should fo­cus on im­prov­ing hu­man in­tel­li­gence so that we are bet­ter able to use our AI sys­tems?

I re­ally like your ex­am­ples in this post, and it made me think of a tan­gen­tial but ul­ti­mately re­lated is­sue.

I feel like there’s long been some­thing like two camps in the AI safety space: the peo­ple who think it’s very hard to make AI safe and the peo­ple who think it’s very very hard like thread­ing a nee­dle from 10 miles away us­ing hand-made binoc­u­lars and a long stick (yes, there’s a third camp that thinks it will be easy, but they aren’t re­ally in the AI safety con­ver­sa­tion due to se­lec­tion effects). And I sus­pect some of this differ­ence is in how much pur­posed ex­am­ple failure sce­nar­ios feel likely and re­al­is­tic to them. Be­ing my­self in the lat­ter camp, I some­times find I hard to ar­tic­u­late why I think this, and of­ten want bet­ter, more evoca­tive ex­am­ples. Thus I was happy to read your ex­am­ples be­cause I think they achieve a level of evoca­tive­ness that I at least of­ten find hard to cre­ate.

It seems that pri­vacy po­ten­tially could “tame” a not-quite-cor­rigible AI. With a full model, the AGI might re­ceive a re­quest, de­duce that ac­ti­vat­ing a cer­tain set of neu­rons strongly would be the most ro­bust way to make you feel the re­quest was fulfilled, and then de­sign an elec­trode set-up to ac­com­plish that. Whereas the same AI with a weak model wouldn’t be able to think of any­thing like that, and might re­sort to fulfilling the re­quest in a more “nor­mal” way. This doesn’t seem that great, but it does seem to me like this is ac­tu­ally part of what makes hu­mans rel­a­tively cor­rigible.

Pri­vacy as a com­po­nent of AI alignment

[re­al­ized this is ba­si­cally just a be­hav­iorist ge­nie, but post­ing it in case some­one finds it use­ful]

What makes some­thing ma­nipu­la­tive? If I do some­thing with the in­tent of get­ting you to do some­thing, is that ma­nipu­la­tive? A sim­ple re­quest seems fine, but if I have a com­plete model of your mind, and use it phrase things so you do ex­actly what I want, that seems to have crossed an im­por­tant line.

The idea is that us­ing a model of a per­son that is *too* de­tailed is a vi­o­la­tion of hu­man val­ues. In par­tic­u­lar, it vi­o­lates the value of au­ton­omy, since your ac­tions can now be con­trol­led by some­one us­ing this model. And I be­lieve that this is a sig­nifi­cant part of what we are try­ing to pro­tect when we in­voke the col­lo­quial value of pri­vacy.

In or­di­nary situ­a­tions, peo­ple can con­trol how much pri­vacy they have rel­a­tive to an­other en­tity by limit­ing their con­tact with them to cer­tain situ­a­tions. But with an AGI, a per­son may lose a very large amount of pri­vacy from seem­ingly in­nocu­ous in­ter­ac­tions (we’re already see­ing the start of this with “big data” com­pa­nies im­prov­ing their ad­ver­tis­ing effec­tive­ness by us­ing in­for­ma­tion that doesn’t seem that sig­nifi­cant to us). Even worse, an AGI may be able to break the pri­vacy of ev­ery­one (or a very large class of peo­ple) by us­ing in­fer­ences based on just a few peo­ple (lev­er­ag­ing per­haps knowl­edge of the hu­man con­nec­tome, hyp­no­sis, etc...).

If we could re­li­ably point to spe­cific mod­els an AI is us­ing, and have it hon­estly share its model struc­ture with us, we could po­ten­tially limit the strength of its model of hu­man minds. Per­haps even have it use a hard­coded model limited to knowl­edge of the phys­i­cal con­di­tions re­quired to keep it healthy. This would miti­gate is­sues such as de­liber­ate de­cep­tion or mind­crime.

We could also po­ten­tially al­low it to use more de­tailed mod­els in spe­cific cases, for ex­am­ple, we could let it use a de­tailed mind model to figure out what is caus­ing de­pres­sion in a spe­cific case, but it would have to use the limited model in any other con­texts or for any plan­ning as­pects of it. Not sure if that ex­am­ple would work, but I think that there are po­ten­tially safe ways to have it use con­text-limited mind mod­els.

I would say the rea­son to as­sume in­finite com­pute was less about which parts of the prob­lem are hard, and more about which parts can be solved with­out a solu­tion to the rest.

Good solu­tions of­ten have even bet­ter solu­tions nearby. In par­tic­u­lar, we would ex­pect most effi­cient and com­pre­hen­si­ble finite al­gorithms to tend to­wards some nice in­finite be­havi­our in the limit. If we find an in­finite al­gorithm, that’s a good point to start look­ing for finite ap­prox­i­ma­tions. It is also of­ten eas­ier to search for an in­finite al­gorithm than a good ap­prox­i­ma­tion. Back­propi­ga­tion in gra­di­ent de­scent is a trick­ier al­gorithm than brute force search. Log­i­cal in­duc­tion is more com­pli­cated to un­der­stand than brute force proof search.

Well, yes, one way to help some liv­ing en­tity is to (1) in­ter­pret it as an agent, and then (2) act in ser­vice of the ter­mi­nal goals of that agent. But that’s not the only way to be helpful. It may also be pos­si­ble to di­rectly be helpful to a liv­ing en­tity that is not an agent, with­out get­ting any agent con­cepts in­volved at all.

I definitely don’t know how to do this, but the route that avoids agent mod­els en­tirely seems more plau­si­ble me com­pared to work­ing hard to in­ter­pret ev­ery­thing us­ing some agent model that is of­ten a re­ally poor fit, and then helping on the ba­sis of a that poorly-fit­ting agent model.

I’m ex­cited about in­quiring deeply into what the heck “help” means. (All please reach out to me if you’d like to join a study sem­i­nar on this topic)