Very in­ter­est­ing defi­ni­tions! I like the way they’re used here to com­pare differ­ent sce­nar­ios.

Pro­posal: Iter­ated Distil­la­tion and Am­plifi­ca­tion: [...] I cur­rently think of this scheme as de­cently date-com­pet­i­tive but not as cost-com­pet­i­tive or perfor­mance-com­pet­i­tive.

I think IDA’s date-com­pet­i­tive­ness will de­pend on the progress we’ll have in in­ner al­ign­ment (or our will­ing­ness to bet against in­ner al­ign­ment prob­lems oc­cur­ring, and whether we’ll be cor­rect about it). Also, I don’t see why we should ex­pect IDA to not be very perfor­mance-com­pet­i­tive (if I un­der­stand cor­rectly the hope is to get a sys­tem that can do any­thing use­ful that a hu­man with a lot of time can do).

Gen­er­ally, when us­ing these defi­ni­tions for com­par­ing al­ign­ment ap­proaches (rather than sce­nar­ios) I sus­pect we’ll end up talk­ing a lot about “the com­bi­na­tion of date- and perfor­mance-com­pet­i­tive­ness”, be­cause I ex­pect the perfor­mance-com­pet­i­tive­ness of most ap­proaches will de­pend on how much re­search effort is in­vested in them.

Mmm, nice. Thanks! I like your dis­tinc­tion also. I think yours is suffi­ciently differ­ent that we shouldn’t see the two sets of dis­tinc­tions as com­pet­ing.* A sys­tem which has an ob­jec­tive which would be ca­pa­ble on pa­per but isn’t ca­pa­ble in prac­tice due to in­ner-al­ign­ment failures would be perfor­mance-un­com­pet­i­tive but ob­jec­tive-com­pet­i­tive. For this rea­son I think we shouldn’t equate ob­jec­tive and perfor­mance com­pet­i­tive­ness.

If op­er­at­ing an AI sys­tem turns out to be an im­por­tant part of the cost, then cost+date com­pet­i­tive­ness would turn out to be differ­ent from train­ing com­pet­i­tive­ness, be­cause cost com­pet­i­tive­ness in­cludes what­ever the rele­vant costs are. How­ever I ex­pect op­er­at­ing costs will be much less rele­vant to con­trol­ling the fu­ture than costs in­curred dur­ing the cre­ation of the sys­tem (all that train­ing, data-gath­er­ing, in­fras­truc­ture build­ing, etc.) so I think the map­ping be­tween cost+date com­pet­i­tive­ness and train­ing com­pet­i­tive­ness ba­si­cally works.

*In­so­far as they are com­pet­ing, I still pre­fer mine; as you say, it ap­plies to more than just pro­saic AI al­ign­ment pro­pos­als. More­over, it makes it eas­ier for us to talk about com­pe­ti­tions as well, e.g. “In the FOOM sce­nario we need to win a date com­pe­ti­tion; cost-com­pet­i­tive­ness still mat­ters but not as much.” More­over cost, perfor­mance, and date are fairly self-ex­plana­tory terms, whereas as you point out “ob­jec­tive” is more opaque. More­over I think it’s worth dis­t­in­guish­ing be­tween cost and date com­pet­i­tive­ness; in some sce­nar­ios one will be much more im­por­tant than the other, and of course the two kinds of com­pet­i­tive­ness vary in­de­pen­dently in AI safety schemes (in­deed maybe they are mildly anti-cor­re­lated? Some schemes are fairly well-defined and cod­ified already, but would re­quire tons of com­pute, whereas other schemes are more vague and thus would re­quire tons of tweak­ing and cau­tious test­ing to get right, but don’t take that much com­pute. I do like how your ver­sion maps more onto the in­ner vs. outer al­ign­ment dis­tinc­tion.

In the con­text of pro­saic AI al­ign­ment, I’ve re­cently taken to split­ting up com­pet­i­tive­ness into “train­ing com­pet­i­tive­ness” and “ob­jec­tive com­pet­i­tive­ness,”^[1] where train­ing com­pet­i­tive­ness refers to the difficulty of train­ing the sys­tem to suc­ceed at its ob­jec­tive and ob­jec­tive com­pet­i­tive­ness refers to the use­ful­ness of a sys­tem that suc­ceeds at that ob­jec­tive. I think my train­ing com­pet­i­tive­ness broadly maps onto a com­bi­na­tion of your cost and date com­pet­i­tive­ness and my ob­jec­tive com­pet­i­tive­ness broadly maps onto your perfor­mance com­pet­i­tive­ness. I think I mildly like my di­chotomy bet­ter than your tri­chotomy in terms of think­ing about pro­saic AI al­ign­ment schemes, as I think it pro­vides a bet­ter pic­ture of the spe­cific parts of a pro­saic AI al­ign­ment pro­posal that are helping or hin­der­ing its over­all com­pet­i­tive­ness—e.g. if it’s not very ob­jec­tive com­pet­i­tive, that tells you that you need a stronger ob­jec­tive, and if it’s not very train­ing com­pet­i­tive, that tells you that you need a bet­ter train­ing pro­cess (it’s also nice in terms of mir­ror­ing the in­ner/​outer al­ign­ment dis­tinc­tion). That be­ing said, your tri­chotomy is cer­tainly more gen­eral in terms of ap­ply­ing to things that aren’t just pro­saic AI al­ign­ment.

Some thoughts that came to me af­ter I wrote this post:

--I’m not sure I should define date-com­pet­i­tive the way I do. Maybe in­stead of “can be built” it should be “is built.” If we go the lat­ter route, the FOOM sce­nario is an ex­tremely in­tense date com­pe­ti­tion. If we go the former route, the FOOM sce­nario is not nec­es­sar­ily an in­tense date com­pe­ti­tion; it de­pends on what other fac­tors are at play. For ex­am­ple, maybe there are only a few ma­jor AI pro­jects and all of them are pretty so­cially re­spon­si­ble, so a de­sign is more likely to win if it can be built sooner, but it won’t nec­es­sar­ily win; maybe cooler heads will pre­vail and build a safer de­sign in­stead.

--Why is date-com­pet­i­tive­ness worth call­ing a kind of com­pet­i­tive­ness at all? Why not just say: “We want our AI safety scheme/​de­sign to be cost- and perfor­mance-com­pet­i­tive, and also we need to be able to build it fairly quickly com­pared to the other stuff that gets built.” Well, 1. Even that is clunky and awk­ward com­pared to the el­e­gant ”...and also date-com­pet­i­tive.” 2. It re­ally does have the com­par­a­tive fla­vor of com­pe­ti­tion to it; what mat­ters is not how long it takes us to com­plete our safety scheme, but how long it takes rel­a­tive to un­al­igned schemes, and it’s not as sim­ple as just “we need to be first,” rather it’s that sooner is bet­ter but do­ing it later isn’t nec­es­sar­ily game over… 3. It seems to be use­ful for de­scribing date com­pe­ti­tions, which are im­por­tant to dis­t­in­guish from situ­a­tions which are not date com­pe­ti­tions or less so. (Aside: A clas­sic crit­i­cism of the “Let’s build up­loads first, and up­load peo­ple we trust” strat­egy is that neu­ro­mor­phic AI will prob­a­bly come be­fore up­loads. In other words, this strat­egy is not date-com­pet­i­tive.)

--I’m toy­ing with the idea of adding “al­ign­ment-com­pet­i­tive­ness” (mean­ing, as al­igned or more al­igned than com­pet­ing sys­tems) and “al­ign­ment com­pe­ti­tion” to the set of defi­ni­tions. This sounds silly, but it would be con­cep­tu­ally neat, be­cause then we can say: We hope for sce­nar­ios in which con­trol of the fu­ture is a very in­tense al­ign­ment com­pe­ti­tion, and we are work­ing hard to make it that way. ”

It’s a sub­jec­tivist ap­proach similar to Bayesi­anism, start­ing from the per­spec­tive of a given sub­ject. Un­like in ideal­ism, there is no as­ser­tion that ev­ery­thing is men­tal.

To check if I’m un­der­stand­ing you cor­rectly, is this an ideal­ist ap­proach you’re tak­ing here, us­ing agent ter­minol­ogy as the foun­da­tion in­stead of talk­ing in terms of “mind” or “in­ten­tion­al­ity”?

I still don’t get it but prob­a­bly not worth dig­ging fur­ther. My cur­rent con­fu­sion is that even un­der the be­hav­iorist in­ter­pre­ta­tion, it seems like just be­liev­ing con­di­tion 2 im­plies know­ing all the things Carol would ever say (or Alice has a mis­taken be­lief). Prob­a­bly this is a con­fu­sion that would go away with enough for­mal­iza­tion /​ math, but it doesn’t seem worth do­ing that.

I think this is re­lated to the word prob­lem for the rewrit­ing sys­tem defined by your pro­gram­ming lan­guage. When I first read this ques­tion I was think­ing “Some­thing to do with Church-Rosser?”—but you can fol­low the links to see for your­self if that liter­a­ture is what you’re af­ter.

Pro­moted to cu­rated: I’ve been think­ing about this post a lot since it has come out, and it is just a re­ally good pre­sen­ta­tion of all the re­search on AI Align­ment via de­bate. It is quite long, which has made me hes­i­tant about cu­rat­ing it for a while, but I now think cu­rat­ing it is the right choice. I also think while it is rea­son­ably tech­ni­cal, it’s ap­proach­able enough that the ba­sic idea of it should be un­der­stand­able by any­one giv­ing it a se­ri­ous try.

Re­gard­ing re­gret bounds, I don’t think re­gret bounds are re­al­is­tic for an AGI, un­less it queried an op­ti­mal teacher for ev­ery ac­tion (which would make it use­less). In the real world, no ac­tions are re­cov­er­able, and any time picks an ac­tion on its own, we can­not be sure it is act­ing op­ti­mally.

First, you can have a sub­jec­tive re­gret bound which doesn’t re­quire all ac­tions to be re­cov­er­able (it does re­quire some ac­tions to be ap­prox­i­mately re­cov­er­able, which is in­deed the case in the real world).

Se­cond, deal­ing ra­tio­nally with non-re­cov­er­able ac­tions should still trans­late into math­e­mat­i­cal con­di­tions some of which might still look like sort of re­gret bounds, and in any case finite MDPs are a nat­u­ral start­ing point for an­a­lyz­ing them.

Third, check­ing re­gret bounds for pri­ors in which all ac­tions are re­cov­er­able serves as a san­ity test for can­di­date AGI al­gorithms. It is not a suffi­cient desider­a­tum, but I do think it is nec­es­sary.

But I think many of the difficul­ties with gen­eral in­tel­li­gence are not cap­tured in the sim­ple setting

I agree that some of the difficul­ties are not cap­tured. I am cu­ri­ous whether you have more con­crete ex­am­ples in mind than what you wrote in the post?

I don’t quite know what to think of con­tin­u­ous MDPs. I’ll wildly and in­for­mally con­jec­ture that if the state space is com­pact, and if the tran­si­tions are Lip­s­chitz con­tin­u­ous with re­spect to the state, it’s not a whole lot more pow­er­ful than the finite-state MDP for­mal­ism.

This seems wrong to me. Can you elab­o­rate what do you mean by “pow­er­ful” in this con­text? Con­tin­u­ous MDPs definitely de­scribe a large va­ri­ety of en­vi­ron­ments that can­not be cap­tured by a finite state MDP, at least not with­out ap­prox­i­ma­tions. Solv­ing con­tin­u­ous MDPs can also be much more difficult than finite state MDPs. For ex­am­ple, any POMDP can be made into a con­tin­u­ous MDP by treat­ing be­liefs as states, and find­ing the op­ti­mal policy for a POMDP is PSPACE-hard (as op­posed to the case of finite state MDPs which is P-easy).

But the up­shot of those MDP tech­niques is mainly to not search through same plans twice, and if we have an ad­vanced agent that is man­ag­ing to not eval­u­ate many plans even once, I think there’s a good chance that we’ll get for free the don’t-eval­u­ate-plans-twice be­hav­ior.

I guess that you might be think­ing ex­clu­sively of al­gorithms that have some­thing like a uniform prior over tran­si­tion ker­nels. In this case there is ob­vi­ously no way to learn about a state with­out vis­it­ing it. But we can also con­sider al­gorithms with more so­phis­ti­cated pri­ors and get much faster learn­ing rates (if the en­vi­ron­ment is truly sam­pled from this prior ofc). The best ex­am­ple is, I think, the work of Os­band and Van Roy where a re­gret bound is de­rived that scales with a cer­tain di­men­sion pa­ram­e­ter of the hy­poth­e­sis space (that can be much smaller than the num­ber of states and ac­tions), work on which I con­tinued to build.

The prob­lem is not in one of the con­di­tions sep­a­rately but in their con­junc­tion: see my fol­low-up com­ment. You could ar­gue that learn­ing an ex­act model of Carol doesn’t re­ally im­ply con­di­tion 2 since, al­though the model does im­ply ev­ery­thing Carol is ever go­ing to say, Alice is not ca­pa­ble of ex­tract­ing this in­for­ma­tion from the model. But then it be­comes a philo­soph­i­cal ques­tion of what does it mean to “be­lieve” some­thing. I think there is value in the “be­hav­iorist” in­ter­pre­ta­tion that “be­liev­ing X” means “be­hav­ing op­ti­mally given X”. In this sense, Alice can sep­a­rately be­lieve the two facts de­scribed by con­di­tions 1 and 2, but can­not be­lieve their con­junc­tion.

Just wanna say, I in­tend to get around to writ­ing re­but­tals some­day. I definitely have sev­eral coun­ter­ar­gu­ments in mind; the force­ful take­downs you men­tion weren’t very con­vinc­ing to me, though they did make me up­date away from fast take­off.

Think­ing more, I think there are good ar­gu­ments for tak­ing ac­tions that as a by-product in­duce an­thropic un­cer­tainty; these are the stan­dard han­so­nian situ­a­tion where you build lots of ems of your­self to do bits of work then turn them off.

But I still don’t agree with the peo­ple in the situ­a­tion you de­scribe be­cause they’re op­ti­mis­ing over their own epistemic state, I think they’re morally wrong to do that. I’m to­tally fine with a law re­quiring fu­ture gov­ern­ments to re­build you /​ an em of you and give you a nice life (per­haps as a trade for work­ing harder to­day to en­sure that the fu­ture world ex­ists), but that’s con­cep­tu­ally analo­gous to ex­tend­ing your life, and doesn’t re­quire caus­ing you to be­lieve false things. You know you’ll be turned off and then later a copy of you will be turned on, there’s no an­thropic un­cer­tainty, you’re just go­ing to get lots of valuable stuff.

Well, that wasn’t the sce­nario I had in mind. The sce­nario I had in mind was: Peo­ple in the year 2030 pass a law re­quiring fu­ture gov­ern­ments to make an­ces­tor simu­la­tions with happy af­ter­lives, be­cause that way it’s prob­a­ble that they them­selves will be in such a simu­la­tion. (It’s like cry­on­ics, but cheaper!) Then, hun­dreds or billions of years later, the fu­ture gov­ern­ment car­ries out the plan, as re­quired by law.

Not say­ing this is what we should do, just say­ing it’s a de­ci­sion I could sym­pa­thize with, and I imag­ine it’s a de­ci­sion some frac­tion of peo­ple would make, if they thought it was an op­tion.

What’s your model for why those ac­tions weren’t un­done?

Not quite sure what you’re ask­ing here. In the first two cases they even­tu­ally were un­done af­ter peo­ple got fed up with the situ­a­tion, the last is re­cent enough I don’t con­sider it’s not hav­ing already been un­done as ev­i­dence peo­ple like it, only that they don’t have the power to change it. My view is that these changes stayed in place be­cause the dic­ta­tors and their suc­ces­sors con­tinued to be­lieve the good out weighted the harm when ei­ther this was clearly con­trary to the ground truth but served some nar­row pur­pose that was viewed as more im­por­tant or when the ground truth was too hard to dis­cover at the time and we only be­lieve it was net harm­ful through the lens of his­tor­i­cal anal­y­sis.

To pop back up to the origi­nal ques­tion—if you think mak­ing your friend 10x more in­tel­li­gent would be net nega­tive, would you make them 10x dumber? Or per­haps it’s only good to make them 2x smarter, but af­ter that more marginal in­tel­li­gence is bad?

It would be re­ally shock­ing if we were at the op­ti­mal ab­solute level of in­tel­li­gence, so I as­sume that you think we’re at the op­ti­mal rel­a­tive level of in­tel­li­gence, that is, the best situ­a­tion is when your friends are about as in­tel­li­gent as you are. In that case, let’s sup­pose that we in­crease/​de­crease all of your friends and your in­tel­li­gence by a fac­tor of X. For what range of X would you ex­pect this in­ter­ven­tion is net pos­i­tive?

I’m not claiming we’re at some op­ti­mal level of in­tel­li­gence for any par­tic­u­lar pur­pose, only that more in­tel­li­gence leads to greater agency which, in the ab­sence of suffi­cient mechanisms to con­strain ac­tions to benefi­cial ones, re­sults in greater risk of nega­tive out­comes due to things like de­viance and unilat­eral ac­tion. Thus I do in fact think we’d be safer from our­selves, for ex­am­ple screen­ing off ex­is­ten­tial risks hu­man­ity faces due to out­side threats like as­ter­oids, if we were dumber.

By com­par­i­son, chim­panzees may not live what look to us like very happy lives, they are some fac­tor dumber than us, but also they aren’t at risk of mak­ing them­selves ex­tinct be­cause one chimp re­ally wanted a lot of ba­nanas.

I’m not sure how much smarter we could all get with­out putting us at too much risk. I think there’s an an­thropic ar­gu­ment to be made that we are be­low what­ever level of in­tel­li­gence is dan­ger­ous to our­selves with­out greater safe­guards be­cause we wouldn’t ex­ist in such uni­verses due to hav­ing kil­led our­selves, but I feel like I have lit­tle ev­i­dence to make a judge­ment about how much smarter is safe given, for ex­am­ple, be­ing, say, 95th per­centile smart didn’t stop peo­ple from build­ing things like atomic weapons or de­vel­op­ing dan­ger­ous chem­i­cal ap­pli­ca­tions. I would ex­pect mak­ing my friends smarter to risk similarly bad out­comes. Mak­ing them dumber seems safer, es­pe­cially when I’m in the frame of think­ing about AGI.

I just don’t think it’s a good de­ci­sion to make, re­gard­less of the math. If I’m near­ing the end of the uni­verse, I pre­fer to spend all my com­pute in­stead max­imis­ing fun /​ search­ing for a way out. Try­ing to run simu­la­tions to make it so I no longer know if I’m about to die seems like a dumb use of com­pute. I can bear the thought of dy­ing dude, there’s bet­ter uses of that com­pute. You’re not sav­ing your­self, you’re just in­ten­tion­ally mak­ing your­self con­fused be­cause you’re un­com­fortable with the thought of death.

I’m not sure it makes sense ei­ther, but I don’t think it is ac­cu­rately de­scribed as “cause your­self to be­lieve false things.” I think whether or not it makes sense comes down to de­ci­sion the­ory. If you use ev­i­den­tial de­ci­sion the­ory, it makes sense; if you use causal de­ci­sion the­ory, it doesn’t. If you use func­tional de­ci­sion the­ory, or up­date­less de­ci­sion the­ory, I’m not sure, I’d have to think more about it. (My guess is that up­date­less de­ci­sion the­ory would do it in­so­far as you care more about your­self than oth­ers, and func­tional de­ci­sion the­ory wouldn’t do it even then.)

Some ex­am­ples of ac­tions taken by dic­ta­tors that I think were well in­ten­tioned and meant to fur­ther goals that seemed laud­able and not about power grab­bing to the dic­ta­tor but had net nega­tive out­comes for the peo­ple in­volved and the world:

What’s your model for why those ac­tions weren’t un­done?

To pop back up to the origi­nal ques­tion—if you think mak­ing your friend 10x more in­tel­li­gent would be net nega­tive, would you make them 10x dumber? Or per­haps it’s only good to make them 2x smarter, but af­ter that more marginal in­tel­li­gence is bad?

It would be re­ally shock­ing if we were at the op­ti­mal ab­solute level of in­tel­li­gence, so I as­sume that you think we’re at the op­ti­mal rel­a­tive level of in­tel­li­gence, that is, the best situ­a­tion is when your friends are about as in­tel­li­gent as you are. In that case, let’s sup­pose that we in­crease/​de­crease all of your friends and your in­tel­li­gence by a fac­tor of X. For what range of X would you ex­pect this in­ter­ven­tion is net pos­i­tive?

(I’m aware that in­tel­li­gence is not one-di­men­sional, but I feel like this is still a mostly mean­ingful ques­tion.)

Just to be clear about my own po­si­tion, a well in­ten­tioned su­per­in­tel­li­gent AI sys­tem to­tally could make mis­takes. How­ever, it seems pretty un­likely that they’d be of the ex­is­ten­tially-catas­trophic kind. Also, the mis­take could be net nega­tive, but the AI sys­tem over­all should be net pos­i­tive.

Do you think I’m wrong?

No, which is why I want to stop us­ing the ex­am­ple.

(The coun­ter­fac­tual I was think­ing of was more like “imag­ine we handed a lap­top to 19th-cen­tury sci­en­tists, can they mechanis­ti­cally un­der­stand it?” But even that isn’t a good anal­ogy, it over­states the difficulty.)

I think you get “ground truth data” by try­ing stuff and see­ing whether or not the AI sys­tem did what you wanted it to do.

That’s the sort of strat­egy where illu­sion of trans­parency is a big prob­lem, from a trans­la­tion point of view. The difficult cases are ex­actly the cases where the trans­la­tion usu­ally pro­duces the re­sults you ex­pect, but then pro­duce some­thing com­pletely differ­ent in some rare cases.

Another way to put it: if we’re gath­er­ing data by see­ing whether the sys­tem did what we wanted, then the long tail prob­lem works against us pretty badly. Those rare tail-cases are ex­actly the cases we would need to ob­serve in or­der to no­tice prob­lems and im­prove the sys­tem. We’re not go­ing to have very many of them to work with. Abil­ity to gen­er­al­ize from small data sets be­comes a key ca­pa­bil­ity, but then we need to trans­late how-to-gen­er­al­ize in or­der for the AI to gen­er­al­ize in the ways we want (this gets at the can’t-ask-the-AI-to-do-any­thing-novel prob­lem).