ricraz

• ricraz 20 Jan 2020 2:25 UTC

  LW: 2 AF: 1

  AF

  in reply to: abramdemski's comment on: Real­ism about rationality

  I’ll try re­spond prop­erly later this week, but I like the point that em­bed­ded agency is about bound­ed­ness. Nev­er­the­less, I think we prob­a­bly dis­agree about how promis­ing it is “to start with ideal­ized ra­tio­nal­ity and try to drag it down to Earth rather than the other way around”. If the start­ing point is in­co­her­ent, then this ap­proach doesn’t seem like it’ll go far—if AIXI isn’t use­ful to study, then prob­a­bly AIXItl isn’t ei­ther (al­though take this par­tic­u­lar ex­am­ple with a grain of salt, since I know al­most noth­ing about AIXItl).

  I ap­pre­ci­ate that this isn’t an ar­gu­ment that I’ve made in a thor­ough or com­pel­ling way yet—I’m work­ing on a post which does so.

• ricraz 14 Jan 2020 11:01 UTC

  LW: 2 AF: 1

  AF

  in reply to: abramdemski's comment on: Real­ism about rationality

  Yeah, I should have been much more care­ful be­fore throw­ing around words like “real”. See the long com­ment I just posted for more clar­ifi­ca­tion, and in par­tic­u­lar this para­graph:

  I’m not try­ing to ar­gue that con­cepts which we can’t for­mal­ise “aren’t real”, but rather that some con­cepts be­come in­co­her­ent when ex­trap­o­lated a long way, and this tends to oc­cur pri­mar­ily for con­cepts which we can’t for­mal­ise, and that it’s those in­co­her­ent ex­trap­o­la­tions which “aren’t real” (I agree that this was quite un­clear in the origi­nal post).

• ricraz 14 Jan 2020 10:40 UTC

  LW: 9 AF: 3

  AF

  in reply to: abramdemski's comment on: Real­ism about rationality

  I like this re­view and think it was very helpful in un­der­stand­ing your (Abram’s) per­spec­tive, as well as high­light­ing some flaws in the origi­nal post, and ways that I’d been un­clear in com­mu­ni­cat­ing my in­tu­itions. In the rest of my com­ment I’ll try write a syn­the­sis of my in­ten­tions for the origi­nal post with your com­ments; I’d be in­ter­ested in the ex­tent to which you agree or dis­agree.

  We can dis­t­in­guish be­tween two ways to un­der­stand a con­cept X. For lack of bet­ter ter­minol­ogy, I’ll call them “un­der­stand­ing how X func­tions” and “un­der­stand­ing the na­ture of X”. I con­flated these in the origi­nal post in a con­fus­ing way.

  For ex­am­ple, I’d say that study­ing how fit­ness func­tions would in­volve look­ing into the ways in which differ­ent com­po­nents are im­por­tant for the fit­ness of ex­ist­ing or­ganisms (e.g. in­ter­nal or­gans; cir­cu­la­tory sys­tems; etc). Some­times you can gen­er­al­ise that knowl­edge to or­ganisms that don’t yet ex­ist, or even prove things about those com­po­nents (e.g. there’s prob­a­bly use­ful maths con­nect­ing graph the­ory with op­ti­mal nerve wiring), but it’s still very grounded in con­crete ex­am­ples. If we thought that we should study how in­tel­li­gence func­tions in a similar way as we study how fit­ness func­tions, that might look like a com­bi­na­tion of cog­ni­tive sci­ence and ma­chine learn­ing.

  By com­par­i­son, un­der­stand­ing the na­ture of X in­volves perform­ing a con­cep­tual re­duc­tion on X by com­ing up with a the­ory which is ca­pa­ble of de­scribing X in a more pre­cise or com­plete way. The pre-the­o­retic con­cept of fit­ness (if it even ex­isted) might have been some­thing like “the num­ber and qual­ity of an or­ganism’s offspring”. Whereas the evolu­tion­ary no­tion of fit­ness is much more spe­cific, and uses maths to link fit­ness with other con­cepts like allele fre­quency.

  Mo­men­tum isn’t re­ally a good ex­am­ple to illus­trate this dis­tinc­tion, so per­haps we could use an­other con­cept from physics, like elec­tric­ity. We can un­der­stand how elec­tric­ity func­tions in a lawlike way by un­der­stand­ing the re­la­tion­ship be­tween voltage, re­sis­tance and cur­rent in a cir­cuit, and so on, even when we don’t know what elec­tric­ity is. If we thought that we should study how in­tel­li­gence func­tions in a similar way as the dis­cov­er­ers of elec­tric­ity stud­ied how it func­tions, that might in­volve do­ing the­o­ret­i­cal RL re­search. But we also want to un­der­stand the na­ture of elec­tric­ity (which turns out to be the flow of elec­trons). Us­ing that knowl­edge, we can ex­tend our the­ory of how elec­tric­ity func­tions to cases which seem puz­zling when we think in terms of voltage, cur­rent and re­sis­tance in cir­cuits (even if we spend al­most all our time still think­ing in those terms in prac­tice). This illus­trates a more gen­eral point: you can un­der­stand a lot about how some­thing func­tions with­out hav­ing a re­duc­tion­ist ac­count of its na­ture—but not ev­ery­thing. And so in the long term, to un­der­stand re­ally well how some­thing func­tions, you need to un­der­stand its na­ture. (Per­haps un­der­stand­ing how CS al­gorithms work in prac­tice, ver­sus un­der­stand­ing the con­cep­tual re­duc­tion of al­gorithms to Tur­ing Machines, is an­other use­ful ex­am­ple).

  I had pre­vi­ously thought that MIRI was try­ing to un­der­stand how in­tel­li­gence func­tions. What I take from your re­view is that MIRI is first try­ing to un­der­stand the na­ture of in­tel­li­gence. From this per­spec­tive, your ear­lier ob­jec­tion makes much more sense.

  How­ever, I still think that there are differ­ent ways you might go about un­der­stand­ing the na­ture of in­tel­li­gence, and that “some­thing kind of like ra­tio­nal­ity re­al­ism” might be a crux here (as you men­tion). One way that you might try to un­der­stand the na­ture of in­tel­li­gence is by do­ing math­e­mat­i­cal anal­y­sis of what hap­pens in the limit of in­creas­ing in­tel­li­gence. I in­ter­pret work on AIXI, log­i­cal in­duc­tors, and de­ci­sion the­ory as fal­ling into this cat­e­gory. This type of work feels analo­gous to some of Ein­stein’s thought ex­per­i­ments about the limit of in­creas­ing speed. Would it have worked for dis­cov­er­ing evolu­tion? That is, would start­ing with a pre-the­o­retic con­cept of fit­ness and do­ing math­e­mat­i­cal anal­y­sis of its limit­ing cases (e.g. by think­ing about or­ganisms that lived for ar­bi­trar­ily long, or had ar­bi­trar­ily large num­bers of chil­dren) have helped peo­ple come up with evolu­tion? I’m not sure. There’s an ar­gu­ment that Malthus did some­thing like this, by look­ing at long-term pop­u­la­tion dy­nam­ics. But you could also ar­gue that the key in­sights lead­ing up to the dis­cov­ery evolu­tion were pri­mar­ily in­spired by spe­cific ob­ser­va­tions about the or­ganisms around us. And in fact, even know­ing evolu­tion­ary the­ory, I don’t think that the ex­treme cases of fit­ness even make sense. So I would say that I am not a re­al­ist about “perfect fit­ness”, even though the con­cept of fit­ness it­self seems fine.

  So an at­tempted rephras­ing of the point I was origi­nally try­ing to make, given this new ter­minol­ogy, is some­thing like “if we suc­ceed in find­ing a the­ory that tells us the na­ture of in­tel­li­gence, it still won’t make much sense in the limit, which is the place where MIRI seems to be pri­mar­ily study­ing it (with some ex­cep­tions, e.g. your Par­tial Agency se­quence). In­stead, the best way to get that the­ory is to study how in­tel­li­gence func­tions.”

  The rea­son I called it “ra­tio­nal­ity re­al­ism” not “in­tel­li­gence re­al­ism” is that ra­tio­nal­ity has con­no­ta­tions of this limit or ideal ex­ist­ing, whereas in­tel­li­gence doesn’t. You might say that X is very in­tel­li­gent, and Y is more in­tel­li­gent than X, with­out agree­ing that perfect in­tel­li­gence ex­ists. Whereas when we talk about ra­tio­nal­ity, there’s usu­ally an as­sump­tion that “perfect ra­tio­nal­ity” ex­ists. I’m not try­ing to ar­gue that con­cepts which we can’t for­mal­ise “aren’t real”, but rather that some con­cepts be­come in­co­her­ent when ex­trap­o­lated a long way, and this tends to oc­cur pri­mar­ily for con­cepts which we can’t for­mal­ise, and that it’s those in­co­her­ent ex­trap­o­la­tions like “perfect fit­ness” which “aren’t real” (I agree that this was quite un­clear in the origi­nal post).

  My pro­posed re­defi­ni­tion:

  • The “in­tel­li­gence is in­tel­ligible” hy­poth­e­sis is about how lawlike the best de­scrip­tion of how in­tel­li­gence func­tions will turn out to be.

  • The “re­al­ism about ra­tio­nal­ity” hy­poth­e­sis is about how well-defined in­tel­li­gence is in the limit (where I think of the limit of in­tel­li­gence as “perfect ra­tio­nal­ity”, and “well-defined” with re­spect not to our cur­rent un­der­stand­ing, but rather with re­spect to the best un­der­stand­ing of the na­ture of in­tel­li­gence we’ll ever dis­cover).

• ricraz 4 Jan 2020 22:01 UTC

  6 points

  in reply to: ozziegooen's comment on: [Part 1] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – Models of im­pact and challenges

  Cool, thanks for those clar­ifi­ca­tions :) In case it didn’t come through from the pre­vi­ous com­ments, I wanted to make clear that this seems like ex­cit­ing work and I’m look­ing for­ward to hear­ing how fol­low-ups go.

• ricraz 4 Jan 2020 20:55 UTC

  2 points

  in reply to: sil ver's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  Yes, but the fact that the frag­ile wor­lds are much more likely to end in the fu­ture is a rea­son to con­di­tion your efforts on be­ing in a ro­bust world.

  While I do buy Paul’s ar­gu­ment, I think it’d be very helpful if the var­i­ous sum­maries of the in­ter­views with him were ed­ited to make it clear that he’s talk­ing about value-con­di­tioned prob­a­bil­ities rather than un­con­di­tional prob­a­bil­ities—since the claim as origi­nally stated feels mis­lead­ing. (Even if some de­ci­sion the­o­ries only use the former, most peo­ple think in terms of the lat­ter).

• ricraz 4 Jan 2020 20:44 UTC

  LW: 4 AF: 2

  AF

  in reply to: rohinmshah's comment on: [AN #80]: Why AI risk might be solved with­out ad­di­tional in­ter­ven­tion from longtermists

  Some ab­strac­tions are heav­ily de­ter­mined by the ter­ri­tory. The con­cept of trees is pretty heav­ily de­ter­mined by the ter­ri­tory. Whereas the con­cept of be­trayal is de­ter­mined by the way that hu­man minds func­tion, which is de­ter­mined by other peo­ple’s ab­strac­tions. So while it seems rea­son­ably likely to me that an AI “nat­u­rally thinks” in terms of the same low-level ab­strac­tions as hu­mans, it think­ing in terms of hu­man high-level ab­strac­tions seems much less likely, ab­sent some type of safety in­ter­ven­tion. Which is par­tic­u­larly im­por­tant be­cause most of the key hu­man val­ues are very high-level ab­strac­tions.

• ricraz 4 Jan 2020 20:19 UTC

  4 points

  in reply to: ozziegooen's comment on: [Part 1] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – Models of im­pact and challenges

  I have four con­cerns even given that you’re us­ing a proper scor­ing rule, which re­late to the link be­tween that scor­ing rule and ac­tu­ally giv­ing peo­ple money. I’m not par­tic­u­larly well-in­formed on this though, so could be to­tally wrong.

  1. To im­ple­ment some proper scor­ing rules, you need the abil­ity to con­fis­cate money from peo­ple who pre­dict badly. Even when the score always has the same sign, like you have with log-scor­ing (or when you add a con­stant to a quadratic scor­ing sys­tem), if you don’t con­fis­cate money for bad pre­dic­tions, then you’re ba­si­cally just giv­ing money to peo­ple for sign­ing up, which makes hav­ing an open plat­form tricky.

  2. Even if you re­strict signups, you get an analo­gous prob­lem within a fixed pop­u­la­tion who’s already signed up: the in­cen­tives will be skewed when it comes to choos­ing which ques­tions to an­swer. In par­tic­u­lar, if peo­ple ex­pect to get pos­i­tive amounts of money for an­swer­ing ran­domly, they’ll do so even when they have no rele­vant in­for­ma­tion, adding a lot of noise.

  3. If a scor­ing rule is “very capped”, as the log-scor­ing func­tion is, then the ex­pected re­ward from an­swer­ing ran­domly may be very close to the ex­pected re­ward from putting in a lot of effort, and so peo­ple would be in­cen­tivised to an­swer ran­domly and spend their time on other things.

  4. Re­lat­edly, peo­ple’s util­ities aren’t lin­ear in money, so the score func­tion might not re­main a proper one tak­ing that into ac­count. But I don’t think this would be a big effect on the scales this is likely to op­er­ate on.

• ricraz 4 Jan 2020 3:07 UTC

  4 points

  in reply to: RobinHanson's comment on: Char­ac­ter­is­ing utopia

  Apolo­gies for the mischar­ac­ter­i­sa­tion. I’ve changed this to re­fer to Scott Alexan­der’s post which pre­dicts this pres­sure.

• ricraz 3 Jan 2020 19:53 UTC

  4 points

  in reply to: ricraz's comment on: [Part 1] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – Models of im­pact and challenges

  Ac­tu­ally, the key differ­ence be­tween this and pre­dic­tion mar­kets is that this has no down­side risk, it seems? If you can’t lose money for bad pre­dic­tions. So you could ex­ploit it by only mak­ing ex­treme pre­dic­tions, which would make a lot of money some­times, with­out los­ing money in the other cases. Or by mak­ing fake ac­counts to drag the av­er­age down.

• ricraz 3 Jan 2020 19:41 UTC

  3 points

  in reply to: ricraz's comment on: [Part 1] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – Models of im­pact and challenges

  Another point: pre­dic­tion mar­kets al­low you to bet more if you’re more con­fi­dent the mar­ket is off. This doesn’t, ex­cept by bet­ting that the mar­ket is fur­ther off. Which is differ­ent. But idk if that mat­ters very much, you could prob­a­bly recre­ate that dy­namic by let­ting peo­ple weight their own pre­dic­tions.

• ricraz 3 Jan 2020 19:34 UTC

  5 points

  in reply to: jacobjacob's comment on: [Part 2] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – re­sults from a pre­limi­nary exploration

  Okay, so in quite a few cases the fore­cast­ers spent more time on a ques­tion than Eliz­a­beth did? That seems like an im­por­tant point to men­tion.

• ricraz 3 Jan 2020 19:21 UTC

  5 points

  in reply to: jacobjacob's comment on: [Part 1] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – Models of im­pact and challenges

  My in­ter­pre­ta­tion: there’s no such thing as nega­tive value of in­for­ma­tion. If the mean of the crowd­work­ers’ es­ti­mates were re­li­ably in the wrong di­rec­tion (com­pared with Eliz­a­beth’s prior) then that would al­low you to up­date Eliz­a­beth’s prior to make it more ac­cu­rate.

• ricraz 3 Jan 2020 19:19 UTC

  14 points

  on: [Part 1] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – Models of im­pact and challenges

  So the thing I’m won­der­ing here is what makes this “am­plifi­ca­tion” in more than a triv­ial sense. Let me think out loud for a bit. Warn­ing: very ram­bly.

  Let’s say you’re a com­pe­tent re­searcher and you want to find out the an­swers to 100 ques­tions, which you don’t have time to in­ves­ti­gate your­self. The ob­vi­ous strat­egy here is to hire 10 peo­ple, get them to in­ves­ti­gate 10 ques­tions each, and then pay them based on how valuable you think their re­search was. Or, per­haps you don’t even need to as­sign them ques­tions—per­haps they can pick their own ques­tions, and you can fac­tor in how ne­glected each ques­tion was as part of the value-of-re­search calcu­la­tion.

  This is the stan­dard, “freeform” ap­proach; it’s “am­plifi­ca­tion” in the same sense that hav­ing em­ploy­ees is always am­plifi­ca­tion. What does the fore­cast­ing ap­proach change?

  • It gives one spe­cific mechanism for how you (the boss) eval­u­ate the qual­ity of re­search (by com­par­i­son with your own deep dive), and rules out all the oth­ers. This has the ad­van­tage of sim­plic­ity and trans­parency, but has the dis­ad­van­tage that you can’t di­rectly give re­wards for other crite­ria like “how well is this ex­plained”. You also can’t re­ward re­search on top­ics that you don’t do deep dives on.

    • This mainly seems valuable if you don’t trust your own abil­ity to eval­u­ate re­search in an un­bi­ased way. But eval­u­at­ing re­search is usu­ally much eas­ier than do­ing re­search! In par­tic­u­lar, do­ing re­search in­volves eval­u­at­ing a whole bunch of pre­vi­ous liter­a­ture.

    • Fur­ther, if one of your sub­or­di­nates thinks you’re sys­tem­at­i­cally bi­ased, then the fore­cast­ing ap­proach doesn’t give them a mechanism to get re­warded for tel­ling you that. Whereas in the freeform ap­proach to eval­u­at­ing the qual­ity of re­search, you can take that into ac­count in your value calcu­la­tion.

  • It gives one spe­cific mechanism for how you ag­gre­gate all the re­search you re­ceive. But that doesn’t mat­ter very much, since you’re not bound to that—you can do what­ever you like with the re­search af­ter you’ve re­ceived it. And in the freeform ap­proach, you’re also able to ask peo­ple to pro­duce prob­a­bil­ity dis­tri­bu­tions if you think that’ll be use­ful for you to ag­gre­gate their re­search.

  • It might save you time? But I don’t think that’s true in gen­eral. Sure, if you use the strat­egy of read­ing ev­ery­one’s re­search then grad­ing it, that might take a long time. But since the fore­cast­ing ap­proach is highly stochas­tic (peo­ple only get re­wards for ques­tions you ran­domly choose to do a deep dive on) you can be a lit­tle bit stochas­tic in other ways to save time. And pre­sum­ably there are lots of other grad­ing strate­gies you could use if you wanted.

  Okay, let’s take an­other tack. What makes pre­dic­tion mar­kets work?

  1. Any­one with rele­vant in­for­ma­tion can use that in­for­ma­tion to make money, if the mar­ket is wrong.

  2. Peo­ple can see the cur­rent mar­ket value.

  3. They don’t have to re­veal their in­for­ma­tion to make money.

  4. They know that there’s no bias in the eval­u­a­tion—if their in­for­ma­tion is good, it’s graded by re­al­ity, not by some gate­keeper.

  5. They don’t ac­tu­ally have to get the whole ques­tion right—they can just pre­dict a short-term mar­ket move­ment (“this stock is cur­rently un­der­val­ued”) and then make money off that.

  This fore­cast­ing setup also fea­tures 1 and 2. Whether or not it fea­tures 3 de­pends on whether you (the boss) man­age to find that in­for­ma­tion by your­self in the deep dive. And 4 also de­pends on that. I don’t know whether 5 holds, but I also don’t know whether it’s im­por­tant.

  So, for the sort of ques­tions we want to ask, is there sig­nifi­cant pri­vate or hard-to-com­mu­ni­cate in­for­ma­tion?

  • If yes, then peo­ple will worry that you won’t find it dur­ing your deep dive.

  • If no, then you likely don’t have any ad­van­tage over oth­ers who are bet­ting.

  • If it’s in the sweet spot where it’s pri­vate but the in­ves­ti­ga­tor would find it dur­ing their deep dive, then peo­ple with that pri­vate in­for­ma­tion have the right in­cen­tives.

  If ei­ther of the first two op­tions holds, then the fore­cast­ing ap­proach might still have an ad­van­tage over a freeform ap­proach, be­cause peo­ple can see the cur­rent best guess when they make their own pre­dic­tions. Is that visi­bil­ity im­por­tant, for the wis­dom of crowds to work—or does it work even if ev­ery­one sub­mits their prob­a­bil­ity dis­tri­bu­tions in­de­pen­dently? I don’t know—that seems like a cru­cial ques­tion.

  
  

  Any­way, to sum­marise, I think it’s worth com­par­ing this more ex­plic­itly to the most straight­for­ward al­ter­na­tive, which is “ask peo­ple to send you in­for­ma­tion and prob­a­bil­ity dis­tri­bu­tions, then use your in­tu­ition or ex­per­tise or what­ever other crite­ria you like to calcu­late how valuable their sub­mis­sion is, then send them a pro­por­tional amount of money.”

• ricraz 3 Jan 2020 15:48 UTC

  4 points

  on: [Part 2] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – re­sults from a pre­limi­nary exploration

  Per­haps I missed this, but how long were the fore­cast­ers ex­pected to spend per claim?

• ricraz 2 Jan 2020 16:30 UTC

  5 points

  on: hu­man psy­chol­in­guists: a crit­i­cal appraisal

  I broadly agree with the sen­ti­ment of this post, that GPT-2 and BERT tell us new things about lan­guage. I don’t think this claim re­lies on the fact that they’re trans­form­ers though—and am skep­ti­cal when you say that “the trans­former ar­chi­tec­ture was a real rep­re­sen­ta­tional ad­vance”, and that “You need the right ar­chi­tec­ture”. In your post on trans­form­ers, you noted that trans­form­ers are su­per­sets of CNNs, but with fewer in­duc­tive bi­ases. But I don’t think of re­mov­ing in­duc­tive bi­ases as rep­re­sen­ta­tional ad­vances—or else get­ting MLPs to work well would be an even big­ger rep­re­sen­ta­tional ad­vance than trans­form­ers! Rather, what we’re do­ing is con­fess­ing as much ig­no­rance about the cor­rect in­duc­tive bi­ases as we can get away with (with­out run­ning out of com­pute).

  Con­cretely, I’d pre­dict with ~80% con­fi­dence that within 3 years, we’ll be able to achieve com­pa­rable perfor­mance to our cur­rent best lan­guage mod­els with­out us­ing trans­form­ers—say, by only us­ing some­thing built of CNNs and LSTMs, plus bet­ter op­ti­mi­sa­tion and reg­u­lari­sa­tion tech­niques. Would you agree or dis­agree with this pre­dic­tion?

Char­ac­ter­is­ing utopia

• ricraz 22 Dec 2019 18:54 UTC

  12 points

  in reply to: Ben Pace's comment on: We run the Cen­ter for Ap­plied Ra­tion­al­ity, AMA

  Note that Val’s con­fu­sion seems to have been be­cause he mi­s­un­der­stood Oli’s point.

  https://​​www.less­wrong.com/​​posts/​​tMhEv28KJYWsu6Wdo/​​kensh?com­men­tId=SPouGqiWNiJgMB3KW#SPouGqiWNiJgMB3KW

• ricraz 15 Dec 2019 21:49 UTC

  LW: 6 AF: 4

  AF

  in reply to: rohinmshah's comment on: Co­her­ence ar­gu­ments do not im­ply goal-di­rected behavior

  +1, I would have writ­ten my own re­view, but I think I ba­si­cally just agree with ev­ery­thing in this one (and to the ex­tent I wanted to fur­ther elab­o­rate on the post, I’ve already done so here).

• ricraz 2 Dec 2019 0:40 UTC

  2 points

  on: Notic­ing the Taste of Lotus

  This post pro­vides a use­ful con­cep­tual han­dle for zoom­ing on what’s ac­tu­ally hap­pen­ing when I get dis­tracted, or pro­cras­ti­nate. Notic­ing this feel­ing has been a helpful step in pre­vent­ing it.

• ricraz 2 Dec 2019 0:36 UTC

  LW: 2 AF: 1

  AF

  on: Co­her­ence ar­gu­ments do not im­ply goal-di­rected behavior

  This post di­rectly ad­dresses what I think is the biggest con­cep­tual hole in our cur­rent un­der­stand­ing of AGI: what type of goals will it have, and why? I think it’s been im­por­tant in push­ing peo­ple away from un­helpful EU-max­imi­sa­tion fram­ings, and to­wards more nu­anced and use­ful ways of think­ing about goals.