“Taking AI Risk Seriously” (thoughts by Critch)

Con­tent Note: Se­ri­ous dis­cus­sion of end-of-the-world and what to do given limited info. Scrupu­los­ity trig­gers, etc.

Epistemic Sta­tus: The first half of this post is sum­ma­riz­ing my own views. I think I phrase each sen­tence about as strongly as I feel it. (When I in­clude a caveat, please take the caveat se­ri­ously)

Much of this blog­post is sum­ma­riz­ing opinions of An­drew Critch, founder of the Berkeley Ex­is­ten­tial Risk Ini­ti­a­tive, who said them con­fi­dently. Some of them I feel com­fortable defend­ing ex­plic­itly, oth­ers I think are im­por­tant to think se­ri­ously about but don’t nec­es­sar­ily en­dorse my­self.

Critch is pretty busy these days and prob­a­bly won’t have time to clar­ify things, so I’m try­ing to err on the side of pre­sent­ing his opinions cau­tiously.

Table of Con­tents:

  • Core Claims

  • My Rough AGI Timelines

  • Con­ver­sa­tions with Critch

    • Hierarchies

    • Deep Thinking

    • Turn­ing Money into Time

    • Things worth learning

    • Plan­ning, Think­ing, Feed­back, Doing

  • A Fi­nal Note on Marathons

Sum­mary of Claims

In past two years, my AI timelines have gone from “I dunno, any­where from 25-100 years in the fu­ture?” to “ten years seems plau­si­ble, and twenty years seems quite pos­si­ble, and thirty years seems quite likely.”

My ex­act thoughts on this are still a bit im­pre­cise, but I feel con­fi­dent in a few claims:

Claim 1: What­ever your es­ti­mates two years ago for AGI timelines, they should prob­a­bly be shorter and more ex­plicit this year.

Claim 2: Re­lat­edly, if you’ve been wait­ing for con­crete things to hap­pen for you to get wor­ried enough to take AGI x-risk se­ri­ously, that time has come. What­ever your timelines cur­rently are, they should prob­a­bly be in­fluenc­ing your de­ci­sions in ways more spe­cific than pe­ri­od­i­cally say­ing “Well, this sounds con­cern­ing.”

Claim 3: Donat­ing money is helpful (I en­dorse Zvi’s en­dorse­ment of MIRI and I think Lark’s sur­vey of the over­all or­ga­ni­za­tional land­scape is great), but hon­estly, we re­ally need peo­ple who are in­vested in be­com­ing use­ful for mak­ing sure the fu­ture is okay.

What this might mean de­pends on where you are. It took me 5 years to tran­si­tion into be­ing the sort of per­son able to con­sider this se­ri­ously. It was very im­por­tant, for the first cou­ple of those years, for me to be able to think about the ques­tions with­out pres­sure or a sense of obli­ga­tion.

I still don’t see any of this as an obli­ga­tion—just as the ob­vi­ously-right-thing-to-do if you’re a Rae­mon with the par­tic­u­lar sets of be­liefs and life-cir­cum­stances that I cur­rently have.

But I do wish I’d been able to make the tran­si­tion faster. Depend­ing on where you cur­rently are, this might mean:

  1. Get your shit to­gether, in gen­eral. Be­come the sort of per­son who can do things on pur­pose, if you aren’t already.

  2. Develop your abil­ity to think – such that if you spent an ad­di­tional hour think­ing about a prob­lem, you tend to be­come less con­fused about that prob­lem, rather than over­whelmed or run­ning in cir­cles.

  3. Get your fi­nan­cial shit to­gether. (Get enough sta­bil­ity and run­way that you can af­ford to take time off to think, and/​or to spend money on things that im­prove your abil­ity to think and act)

  4. Ar­rang­ing your life such that you are able to learn about the state of AI de­vel­op­ment. There are rea­sons to do this both so that you find things to do to help, and so that you are just in­di­vi­d­u­ally pre­pared for what’s com­ing, what­ever it is.

I think that re­ally get­ting a han­dle on what is hap­pen­ing in the world and what to do about it re­quires more time than “oc­ca­sion­ally, in your off hours.”

Se­ri­ous think­ing re­quires se­ri­ous effort and deep work.

I have some thoughts about “what to do, de­pend­ing on your cur­rent life cir­cum­stances”, but first want to drill down a bit into why I’m con­cerned.

Rough Timelines

My cur­rent AI timelines aren’t very rigor­ous. A some­what em­bar­rass­ing chunk of my “ten to twenty years” num­bers come from “this is what smart peo­ple I re­spect seem to think”, and there’s a de­cent chance those smart peo­ple are un­der­go­ing a run­away in­for­ma­tion cas­cade.

But I still think there’s enough new in­for­ma­tion to make some real up­dates. My views are roughly the ag­gre­gate of the fol­low­ing:

My vague im­pres­sion from read­ing in­for­mal but high-level dis­cus­sions (among peo­ple of differ­ent paradigms) is that the con­ver­sa­tion has shifted from “is AI even a thing to be con­cerned about?” to “what are the spe­cific ways to ap­proach safety in light of the progress we’ve seen with AlphaGo Zero?”

The main thrust of my be­lief is that cur­rent neu­ral nets, while prob­a­bly still a few in­sights away from true AGI a la Sarah Con­stantin...

  • Seem to be able to do a wide va­ri­ety of tasks, us­ing prin­ci­ples similar to how hu­man brains ac­tu­ally do stuff.

  • Seem to be pro­gress­ing faster at play­ing games than most peo­ple expected

  • We’re con­tin­u­ing to de­velop hard­ware (i.e. TPUs) op­ti­mized for AI that may make it eas­ier to ac­com­plish things via brute force even if we don’t have the best al­gorithms yet.

  • Upon hit­ting mile­stones, we seem to quickly go from “less good than a hu­man” at a task to “su­per­hu­man” in a mat­ter of months, and this is be­fore re­cur­sive self im­prove­ment en­ters the pic­ture. (see Su­per­in­tel­li­gence FAQ)

Mean­while, my take­away from Katja Grace sur­vey of ac­tual AI re­searchers is that in­dus­try pro­fes­sion­als say­ing “it’s at least decades away” don’t re­ally have a model at all, since differ­ent fram­ings of the ques­tion yield very differ­ent av­er­age re­sponses.

Sarah Con­stantin offers the strongest ar­gu­ment I’ve seen that AGI isn’t im­mi­nent—that all the progress we’ve seen (even abil­ity to solve ar­bi­trary ar­cade games) still doesn’t seem to in­di­cate AI that un­der­stand con­cepts and can think de­liber­ately about them. A key piece of gen­eral in­tel­li­gence is miss­ing.

She also ar­gues that progress has been fairly lin­ear, even as we have in­cor­po­rated deep learn­ing. “Perfor­mance trends in AI” was writ­ten a year ago and I’m not sure how much AlphaGo Zero would change her opinion.

But this isn’t that re­as­sur­ing to me. At best, this seems to point to­wards a mod­er­ate take­off that be­gins in earnest a cou­ple decades from now. That still sug­gests rad­i­cal changes to the world within our life­times, mov­ing faster than I ex­pect to be able to re­cover from mis­takes.

Mean­while, AlphaGo Zero isn’t over­whelming ev­i­dence to the con­trary, but it seemed at least like a con­crete, sig­nifi­cant bit of ev­i­dence that progress can be faster, sim­pler, and more sur­pris­ing.

From Me­taphors to Evidence

When I first read the se­quences, the ar­gu­ments for fast take­off seemed rea­son­able, but they also sounded like the sort of clever-sound­ing things a smart per­son could say to make any­thing sound plau­si­ble.

By now, I think enough con­crete ev­i­dence has piled up that we’re way be­yond “Is an Earth full of Ein­steins a rea­son­able metaphor?” style de­bates, and squarely into “the ac­tual things hap­pen­ing in the real world seem to firmly point to­wards sooner and faster rather than later.

When you fac­tor in that com­pa­nies don’t share ev­ery­thing they’re work­ing on, and that we should ex­pect Deep­Mind et al to have some pro­jects we don’t even know about yet (and that the last cou­ple of their an­nounce­ments sur­prised many peo­ple), it seems that should fur­ther push prob­a­bil­ity mass to­wards more-progress-than-we-in­tu­itively-ex­pect.

If you aren’t cur­rently per­suaded on AI timelines be­ing short or that you should change your be­hav­ior, that’s fine. This isn’t meant to be a com­pre­hen­sive ar­gu­ment.

But, if you be­lieve that 10-20 year AI timelines are plau­si­ble, and you’re psy­cholog­i­cally, fi­nan­cially, and op­er­a­tionally ready to take that se­ri­ously, I think it’s a good time to kick your “take-se­ri­ously-o-me­ter” up a few notches.

If you’re close but not quite ready to take AI fully se­ri­ously, I think this is a use­ful set of things to start think­ing about now, so that in a year or two when you’re more ca­pa­ble or the timing is bet­ter, the tran­si­tion will be eas­ier.

Con­ver­sa­tions with Critch

I’ve had a few re­cent con­ver­sa­tions with An­drew Critch, who’s been in­volved at MIRI and cur­rently helps run the Cen­ter for Hu­man Com­pat­i­ble AI (CHAI) and the Berkeley Ex­is­ten­tial Risk Ini­ti­a­tive (BERI).

In past con­ver­sa­tions with Critch, I’ve fre­quently run into the pat­tern:

  • Critch: says thing that sounds ridiculous

  • Me: “That’s ridicu­lous”

  • *ar­gues for an hour or two*

  • Me: “Huh, okay, I guess that does make sense.”

This has hap­pened enough that I’ve learned to give him the benefit of the doubt. It’s usu­ally a straight­for­ward case of in­fer­en­tial dis­tance, oc­ca­sion­ally due to differ­ent goals. Usu­ally there are caveats that make the ridicu­lous thing make more sense in con­text.

I men­tion this be­cause when I asked him what ad­vice he’d give to peo­ple look­ing to donate to x-risk or AI safety, he said some­thing to the effect of:

“If you have three years of run­way saved up, quit your job and use the money to fund your­self. Study the AI land­scape full-time. Figure out what to do. Do it.”

This felt a lit­tle ex­treme.

Part of this ex­trem­ity is due to var­i­ous caveats:

  • “Three years of run­way” means com­fortable run­way, not “you can tech­ni­cally live off of ra­men noo­dles” run­way.

  • [Edit for clar­ity] The goal is not to quit your job for three years – the goal is to have as much time as you need (i.e from 6 months to 2 years or so) to learn what you need be­fore scarcity mind­set kicks in. If you’re com­fortable liv­ing with less than a month of run­way, you can get away with less.

  • This re­quires you to already be the sort of per­son who can do self-di­rected study with open ended, am­bigu­ous goals.

  • This re­quires you to, in an im­por­tant sense, know how to think.

  • This makes most sense if you’re not in the mid­dle of plans that seem com­pa­rably im­por­tant.

  • The core un­der­ly­ing idea is more like “it’s more im­por­tant to in­vest in your abil­ity to think, learn and do, than to donate your last spare dol­lar”, rather than the spe­cific con­clu­sion “quit your job to study full-time.”

But… the other part of it is sim­ply...

If you ac­tu­ally think the world might be end­ing or for­ever chang­ing in your life­time – whether in ten years, or fifty…

...maybe you should be tak­ing ac­tions that feel ex­treme?

Even if you’re not in­ter­ested in ori­ent­ing your life around helping with x-risk – if you just want to not be blind­sided by rad­i­cal changes that may be com­ing,

Critch on AI

[This next sec­tion is a sum­mary/​para­phrase of a few con­ver­sa­tions with Critch, writ­ten first-per­son from his per­spec­tive.]

We need more peo­ple who are able to think full-time about AI safety.

I’ve got­ten the sense that you think of me like I’m in some spe­cial in­ner cir­cle of “peo­ple work­ing on x-risk”. But hon­estly, I strug­gle des­per­ately to be use­ful to peo­ple like Stu­art Rus­sell who are ac­tu­ally in the in­ner cir­cle of the world stage, who get to talk to gov­ern­ment and in­dus­try lead­ers reg­u­larly.

Hier­ar­chies are how you get things done for real.

Hu­mans are limited in their time/​at­ten­tion. We need peo­ple fo­cus­ing on spe­cific prob­lems, re­port­ing up to peo­ple who are keep­ing an eye on the big pic­ture. And we need those peo­ple re­port­ing up to peo­ple keep­ing their eye on the big­ger pic­ture.

Right now our abil­ity to grow the hi­er­ar­chy is crip­pled—it can only be a cou­ple lay­ers deep, be­cause there are few peo­ple who a) have their shit to­gether, and b) un­der­stand both the tech­ni­cal the­o­ries/​math/​ma­chine-learn­ing and how in­ter-or­ga­ni­za­tional poli­tics works.

So peo­ple like me can’t just hand com­pli­cated as­sign­ments off and trust they get done com­pe­tently. Some­one might un­der­stand the the­ory but not get the poli­ti­cal nu­ances they need to do some­thing use­ful with the the­ory. Or they get the poli­ti­cal nu­ances, and maybe get the the­ory at-the-time, but aren’t keep­ing up with the evolv­ing tech­ni­cal land­scape.

There are N things I’m work­ing on right now that need to get done in the next 6 months, and I only re­ally have the time to do M of them be­cause there’s no one else with the skills/​cre­den­tials/​net­work who can do it.

So the most im­por­tant thing we need is more peo­ple putting enough time into mak­ing them­selves use­ful.

I think that means fo­cus­ing full-time.

We don’t know ex­actly what will hap­pen, but I ex­pect se­ri­ous changes of some sort over the next 10 years. Even if you aren’t com­mit­ting to sav­ing the world, I think it’s in your in­ter­est just to un­der­stand what is hap­pen­ing, so in a decade or two you aren’t com­pletely lost.

And even ‘un­der­stand­ing the situ­a­tion’ is com­pli­cated enough that I think you need to be able to quit your day-job and fo­cus full-time, in or­der to get ori­ented.

Deep Thinking

How much time have you spent just think­ing about ma­jor prob­lems?

There are in­creas­ing re­turns to deep, un­in­ter­rupted work. A half hour here and there is qual­i­ta­tively differ­ent from spend­ing a four-hour block of time, which is qual­i­ta­tively differ­ent from fo­cus­ing on a prob­lem for an en­tire week.

A tech­nique I use is spend es­ca­lat­ing chunks of time figur­ing out how to spend es­ca­lat­ing chunks of time think­ing. Spend half an hour think­ing “how use­ful would it be to spend four hours think­ing about X?”

When you have a lot of dis­trac­tions—in­clud­ing things like a day job, or wor­ry­ing about run­ning out of money, it can be very difficult to give im­por­tant prob­lems the at­ten­tion they de­serve.

If you’re cur­rently a stu­dent, take ad­van­tage of the fact that you’re life is cur­rently struc­tured to fo­cus on think­ing.

Fund­ing Individuals

I think fund­ing in­di­vi­d­u­als who don’t have that run­way would be a good thing for ma­jor donors to do. The prob­lem is that it’s mod­er­ately ex­pen­sive—even a ma­jor donor can only af­ford to do it a few times. It’s re­ally hard to eval­u­ate which in­di­vi­d­u­als to pri­ori­tize (and if peo­ple know you’re think­ing about it, they’ll show up try­ing to get your money, whether they’re good or not).

The good/​bad news is that, be­cause the whole world may be chang­ing in some fash­ion soon, it’s in an in­di­vi­d­ual’s di­rect in­ter­est to have thought about that a lot in ad­vance.

So while a ma­jor-donor de­cid­ing to give some 2-3 years of run­way to think would be risk­ing a lot on a hard-to-eval­u­ate per­son, an in­di­vi­d­ual per­son who self-funds is more likely to get a lot of value re­gard­less.

If you do know some­one else you highly trust, it may be worth fund­ing them di­rectly.

Small Scale “Money into Time”

A lot of peo­ple have in­ter­nal­ized a “be thrifty” mind­set, which makes it harder to spend money to gain more time. There are a lot of op­tions that might feel a bit ex­trav­a­gant. But right now it looks to me like we may only have 10 years left, and ev­ery op­por­tu­nity to turn money into time is valuable. Ex­am­ples:

  • Buy­ing a larger mon­i­tor, iPad or even large pen-and-pa­per note­book so you have more “exo-brain” to think on. A hu­man can only re­ally keep seven things in their head at once, but hav­ing things writ­ten down ex­ter­nally makes it eas­ier to keep track of more.

  • Pay­ing for cabs that gives you space to think and write dur­ing travel time.

  • Pay­ing for food de­liv­ery rather than mak­ing it or go­ing out.

  • Pay­ing for per­sonal as­sis­tants who can do ran­dom odd-jobs for you. (Get­ting value out of this took a lot of prac­tice – some things turned out to be hard to out­source, and man­ag­ing peo­ple is a nu­anced skill. But if you can put in the time ex­per­i­ment­ing, learn­ing, and find­ing the right as­sis­tant, it’s very worth­while)

  • Pay­ing for a per­sonal trainer to help you get bet­ter at ex­er­cise be­cause it turns out ex­er­cise is pretty im­por­tant over­all.

What to Ac­tu­ally Read and Think About?

What to ac­tu­ally read is a hard ques­tion, since the land­scape is chang­ing fairly quickly, and most of the things worth read­ing aren’t op­ti­mized for easy learn­ing, or figur­ing out if the thing is right-for-you.

But an un­der­ly­ing prin­ci­ple is to think about how minds work, and to study what’s hap­pen­ing in the world of AI de­vel­op­ment. If you don’t un­der­stand what’s go­ing on in the world of AI de­vel­op­ment, figure out what back­ground you need to learn in or­der to un­der­stand it.

[Edit: the goal here is not to be “be­come an AI re­searcher.” The goal is to un­der­stand the land­scape well enough that what­ever you’re do­ing, you’re in­formed enough on it]

A lot of this is nec­es­sar­ily tech­ni­cal, which can be pretty in­timi­dat­ing if you haven’t been think­ing of your­self as a tech­ni­cal per­son. You can by­pass some of this by find­ing tech­ni­cally ori­ented peo­ple who seem to be able to make good pre­dic­tions about the fu­ture, and rely­ing on them to tell you how they ex­pect the world to change. But that will limit how flex­ible a plan you’ll be able to cre­ate for your­self. (And again, this seems rele­vant whether your goal is “help with x-risk” or just “not have your life and ca­reer up­ended as things be­gin chang­ing rad­i­cally).

[Ray note: FWIW, I had ac­quired an image of my­self as a “non-tech­ni­cal per­son”, averse to learn­ing mathy stuff in do­mains I wasn’t already fa­mil­iar with. I re­cently just… got over it, and started learn­ing calcu­lus, and ac­tu­ally en­joyed it and feel kinda lame about spend­ing 10 years self-iden­ti­fy­ing in a way that pre­vented me from grow­ing in that di­rec­tion]

Rough notes on how to go about this:

  • If you can’t viscer­ally feel the differ­ence be­tween .1% and 1%, or a thou­sand and a mil­lion, you will prob­a­bly need more of a statis­tics back­ground to re­ally un­der­stand things like “how much money is flow­ing into AI, and what is be­ing ac­com­plished, and what does it mean?”. A de­cent re­sources for this is Fried­man Statis­tics Fourth Edi­tion.

  • Calcu­lus is pretty im­por­tant back­ground for un­der­stand­ing most tech­ni­cal work.

  • Mu­ti­vari­able Calcu­lus and Lin­ear Alge­bra are im­por­tant for un­der­stand­ing ma­chine learn­ing in par­tic­u­lar.

  • Read the lat­est pub­li­ca­tions by Deep­Mind and OpenAI to have a sense of what progress is be­ing made.

Re­mem­ber as you’re learn­ing all this to think about the fact that minds are made of mat­ter, in­ter­act­ing. Statis­tics is the the­ory of ag­gre­gat­ing in­for­ma­tion. You are a bunch of neu­rons ag­gre­gat­ing in­for­ma­tion. Think about what that means, as you learn the tech­ni­cal back­ground on what the lat­est ma­chine learn­ing is do­ing.

Plan­ning, Think­ing, Feed­back, Doing

A differ­ent tack is, rather than sim­ply catch­ing up on read­ing, to prac­tice for­mu­lat­ing, get­ting feed­back on, and ex­e­cut­ing plans.

A gen­eral strat­egy I find use­ful is to write up plans on google docs, mak­ing your thought pro­cess ex­plicit. Google docs are easy to share, op­ti­mal for peo­ple to provide both in-line com­ments as well as sug­gest­ing ma­jor re­vi­sions.

If you can write up a plan, get feed­back from 2-4 peo­ple who are rep­re­sen­ta­tive of differ­ent thought pro­cesses, who all agree that your plan makes sense, that’s ev­i­dence that you’re got some­thing worth do­ing.

Whereas if you just keep your plan in your head, you may run into a few is­sues:

  1. You only have so much work­ing mem­ory. Writ­ing it down lets you make sure you can see all of your as­sump­tions at once. You can catch ob­vi­ous er­rors. You can build more com­plex mod­els.

  2. You may have ma­jor blindspots. Get­ting feed­back from mul­ti­ple peo­ple with differ­ent out­looks helps en­sure that you’re not run­ning off ma­jorly wrong mod­els.

  3. The pro­cess of find­ing peo­ple to give feed­back is an im­por­tant skill that will be rele­vant to­wards ex­e­cut­ing plans that mat­ter. Get­ting the buy-in from peo­ple to se­ri­ously re­view an idea can be hard. Buy-in to­wards ac­tu­ally ex­e­cut­ing a plan can be harder.

One of our limit­ing ca­pa­bil­ities here is form­ing plans that peo­ple in mul­ti­ple or­ga­ni­za­tions with differ­ent goals are able to col­lab­o­rate on. An early step for this is be­ing aware of how peo­ple from differ­ent or­ga­ni­za­tions think.

An im­por­tant con­sid­er­a­tion is which peo­ple to get feed­back from. The peo­ple you are most fa­mil­iar with at each or­ga­ni­za­tion are prob­a­bly the peo­ple who are most busy. Depend­ing on your cur­rent net­work, some good prac­tice is to start with peo­ple in your so­cial cir­cle who seem gen­er­ally smart, then reach out to peo­ple at differ­ent or­ga­ni­za­tions who aren’t the pri­mary spokesper­son or re­search heads.

Fi­nal Note on Marathons

(Speak­ing now as Rae­mon, again)

I’ve talked a lot lately about burn­ing out, mak­ing sure you have enough slack. In the past, I was the sort of per­son who said “OMG the world is burn­ing” and then be­came in­creas­ingly mis­er­able for 3 years, and I’ve seen other peo­ple do the same.

Ten to twenty year timelines are quite scary. You should be more con­cretely wor­ried than you were be­fore. In the terms of a strat­egy game, we’re tran­si­tion­ing from the mid-game to the late game.

But ten or twenty years is still a marathon, not a sprint. We’re try­ing to max­i­mize the dis­tance cov­ered in the next decade or two, not race as fast as we can for the next 6 months and then col­lapse in a heap. There may come a time when we’re rac­ing to the finish and it’s worth em­ploy­ing strate­gies that are not long-term sus­tain­able, but we are not at that point.

You know bet­ter than I what your own psy­cholog­i­cal, phys­i­cal and fi­nan­cial situ­a­tion is, and what is ap­pro­pri­ate given that situ­a­tion.

There’s room to ar­gue about the ex­act timelines. Smart peo­ple I know seem to agree there’s a rea­son­able chance of AGI in ten years, but dis­agree on whether that’s “likely” or just “pos­si­ble.”

But it is sig­nifi­cant that we are defini­tively in a marathon now, as op­posed to some peo­ple hang­ing out in a park ar­gu­ing over whether a race even ex­ists.

Wher­ever you are cur­rently at, I recom­mend:

...if you haven’t ac­quired the gen­eral abil­ity to do things on pur­pose, or think about things on pur­pose, figure out how to do that. If you haven’t spent 3 hours try­ing to un­der­stand and solve any com­plex prob­lem, try that, on what­ever prob­lem seems most near/​real to you.

...if you haven’t spent 3 hours think­ing about AI in par­tic­u­lar, and things that need do­ing, and skills that you have (or could learn), and plans you might en­act… con­sider carv­ing out those 3 hours.

If you haven’t carved out a full week­end to do deep think­ing about it, maybe try that.

And if you’ve done all that, figure out how to re­ar­range your life to reg­u­larly give your­self large chunks of time to think and make plans. This may take the form of sav­ing a lot of money and quit­ting your job for awhile to ori­ent. It may take the form of build­ing so­cial cap­i­tal at your job so that you can pe­ri­od­i­cally take weeks off to think and learn. It may take the form of get­ting a job where think­ing about the fu­ture is some­how built nat­u­rally into your work­flow.

What­ever your situ­a­tion, take time to pro­cess that the fu­ture is com­ing, in one shape or an­other, and this should prob­a­bly out­put some kind of de­ci­sions that are not busi­ness as usual.


Fur­ther Read­ing:

De­liber­ate Grad School

Bibliog­ra­phy for the Berkeley Cen­ter for Hu­man Com­pat­i­ble AI