# Security Mindset and the Logistic Success Curve

Fol­low-up to: Se­cu­rity Mind­set and Or­di­nary Para­noia

(Two days later, Am­ber re­turns with an­other ques­tion.)

am­ber: Uh, say, Co­ral. How im­por­tant is se­cu­rity mind­set when you’re build­ing a whole new kind of sys­tem—say, one sub­ject to po­ten­tially ad­verse op­ti­miza­tion pres­sures, where you want it to have some sort of ro­bust­ness prop­erty?

coral: How novel is the sys­tem?

am­ber: Very novel.

coral: Novel enough that you’d have to in­vent your own new best prac­tices in­stead of look­ing them up?

am­ber: Right.

coral: That’s se­ri­ous busi­ness. If you’re build­ing a very sim­ple In­ter­net-con­nected sys­tem, maybe a smart or­di­nary para­noid could look up how we usu­ally guard against ad­ver­saries, use as much off-the-shelf soft­ware as pos­si­ble that was checked over by real se­cu­rity pro­fes­sion­als, and not do too hor­ribly. But if you’re do­ing some­thing qual­i­ta­tively new and com­pli­cated that has to be ro­bust against ad­verse op­ti­miza­tion, well… mostly I’d think you were op­er­at­ing in al­most im­pos­si­bly dan­ger­ous ter­ri­tory, and I’d ad­vise you to figure out what to do af­ter your first try failed. But if you wanted to ac­tu­ally suc­ceed, or­di­nary para­noia ab­solutely would not do it.

am­ber: In other words, pro­jects to build novel mis­sion-crit­i­cal sys­tems ought to have ad­vi­sors with the full se­cu­rity mind­set, so that the ad­vi­sor can say what the sys­tem builders re­ally need to do to en­sure se­cu­rity.

am­ber: No?

coral: Let’s say for the sake of con­crete­ness that you want to build a new kind of se­cure op­er­at­ing sys­tem. That is not the sort of thing you can do by at­tach­ing one ad­vi­sor with se­cu­rity mind­set, who has limited poli­ti­cal cap­i­tal to use to try to ar­gue peo­ple into do­ing things. “Build­ing a house when you’re only al­lowed to touch the bricks us­ing tweez­ers” comes to mind as a metaphor. You’re go­ing to need ex­pe­rienced se­cu­rity pro­fes­sion­als work­ing full-time with high au­thor­ity. Three of them, one of whom is a cofounder. Although even then, we might still be op­er­at­ing in the ter­ri­tory of Paul Gra­ham’s De­sign Para­dox.

am­ber: De­sign Para­dox? What’s that?

coral: Paul Gra­ham’s De­sign Para­dox is that peo­ple who have good taste in UIs can tell when other peo­ple are de­sign­ing good UIs, but most CEOs of big com­pa­nies lack the good taste to tell who else has good taste. And that’s why big com­pa­nies can’t just hire other peo­ple as tal­ented as Steve Jobs to build nice things for them, even though Steve Jobs cer­tainly wasn’t the best pos­si­ble de­signer on the planet. Ap­ple ex­isted be­cause of a lucky his­tory where Steve Jobs ended up in charge. There’s no way for Sam­sung to hire some­body else with equal tal­ents, be­cause Sam­sung would just end up with some guy in a suit who was good at pre­tend­ing to be Steve Jobs in front of a CEO who couldn’t tell the differ­ence.

Similarly, peo­ple with se­cu­rity mind­set can no­tice when other peo­ple lack it, but I’d worry that an or­di­nary para­noid would have a hard time tel­ling the differ­ence, which would make it hard for them to hire a truly com­pe­tent ad­vi­sor. And of course lots of the peo­ple in the larger so­cial sys­tem be­hind tech­nol­ogy pro­jects lack even the or­di­nary para­noia that many good pro­gram­mers pos­sess, and they just end up with empty suits talk­ing a lot about “risk” and “safety”. In other words, if we’re talk­ing about some­thing as hard as build­ing a se­cure op­er­at­ing sys­tem, and your pro­ject hasn’t started up already headed up by some­one with the full se­cu­rity mind­set, you are in trou­ble. Where by “in trou­ble” I mean “to­tally, ir­re­triev­ably doomed”.

am­ber: Look, uh, there’s a cer­tain pro­ject I’m in­vested in which has raised a hun­dred mil­lion dol­lars to cre­ate mer­chant drones.

coral: Mer­chant drones?

am­ber: So there are a lot of coun­tries that have poor mar­ket in­fras­truc­ture, and the idea is, we’re go­ing to make drones that fly around buy­ing and sel­l­ing things, and they’ll use ma­chine learn­ing to figure out what prices to pay and so on. We’re not just in it for the money; we think it could be a huge eco­nomic boost to those coun­tries, re­ally help them move for­wards.

coral: Dear God. Okay. There are ex­actly two things your com­pany is about: sys­tem se­cu­rity, and reg­u­la­tory com­pli­ance. Well, and also mar­ket­ing, but that doesn’t count be­cause ev­ery com­pany is about mar­ket­ing. It would be a se­vere er­ror to imag­ine that your com­pany is about any­thing else, such as drone hard­ware or ma­chine learn­ing.

am­ber: Well, the sen­ti­ment in­side the com­pany is that the time to be­gin think­ing about le­gal­ities and se­cu­rity will be af­ter we’ve proven we can build a pro­to­type and have at least a small pi­lot mar­ket in progress. I mean, un­til we know how peo­ple are us­ing the sys­tem and how the soft­ware ends up work­ing, it’s hard to see how we could do any pro­duc­tive think­ing about se­cu­rity or com­pli­ance that wouldn’t just be pure spec­u­la­tion.

coral: Ha! Ha, ha­haha… oh my god you’re not jok­ing.

am­ber: What?

coral: Please tell me that what you ac­tu­ally mean is that you have a se­cu­rity and reg­u­la­tory roadmap which calls for you to do some of your work later, but clearly lays out what work needs to be done, when you are to start do­ing it, and when each mile­stone needs to be com­plete. Surely you don’t liter­ally mean that you in­tend to start think­ing about it later?

am­ber: A lot of times at lunch we talk about how an­noy­ing it is that we’ll have to deal with reg­u­la­tions and how much bet­ter it would be if gov­ern­ments were more liber­tar­ian. That counts as think­ing about it, right?

coral: Oh my god.

am­ber: I don’t see how we could have a se­cu­rity plan when we don’t know ex­actly what we’ll be se­cur­ing. Wouldn’t the plan just turn out to be wrong?

coral: All busi­ness plans for star­tups turn out to be wrong, but you still need them—and not just as works of fic­tion. They rep­re­sent the writ­ten form of your cur­rent be­liefs about your key as­sump­tions. Writ­ing down your busi­ness plan checks whether your cur­rent be­liefs can pos­si­bly be co­her­ent, and sug­gests which crit­i­cal be­liefs to test first, and which re­sults should set off alarms, and when you are fal­ling be­hind key sur­vival thresh­olds. The idea isn’t that you stick to the busi­ness plan; it’s that hav­ing a busi­ness plan (a) checks that it seems pos­si­ble to suc­ceed in any way what­so­ever, and (b) tells you when one of your be­liefs is be­ing falsified so you can ex­plic­itly change the plan and adapt. Hav­ing a writ­ten plan that you in­tend to rapidly re­vise in the face of new in­for­ma­tion is one thing. NOT HAVING A PLAN is an­other.

am­ber: The thing is, I am a lit­tle wor­ried that the head of the pro­ject, Mr. Topaz, isn’t con­cerned enough about the pos­si­bil­ity of some­body fool­ing the drones into giv­ing out money when they shouldn’t. I mean, I’ve tried to raise that con­cern, but he says that of course we’re not go­ing to pro­gram the drones to give out money to just any­one. Can you maybe give him a few tips? For when it comes time to start think­ing about se­cu­rity, I mean.

coral: Oh. Oh, my dear, sweet sum­mer child, I’m sorry. There’s noth­ing I can do for you.

am­ber: Huh? But you haven’t even looked at our beau­tiful busi­ness model!

coral: I thought maybe your com­pany merely had a hope­less case of un­der­es­ti­mated difficul­ties and mis­placed pri­ori­ties. But now it sounds like your leader is not even us­ing or­di­nary para­noia, and re­acts with skep­ti­cism to it. Cal­ling a case like that “hope­less” would be an un­der­state­ment.

am­ber: But a se­cu­rity failure would be very bad for the coun­tries we’re try­ing to help! They need se­cure mer­chant drones!

coral: Then they will need drones built by some pro­ject that is not led by Mr. Topaz.

am­ber: But that seems very hard to ar­range!

coral: …I don’t un­der­stand what you are say­ing that is sup­posed to con­tra­dict any­thing I am say­ing.

am­ber: Look, aren’t you judg­ing Mr. Topaz a lit­tle too quickly? Se­ri­ously.

coral: I haven’t met him, so it’s pos­si­ble you mis­rep­re­sented him to me. But if you’ve ac­cu­rately rep­re­sented his at­ti­tude? Then, yes, I did judge quickly, but it’s a hell of a good guess. Se­cu­rity mind­set is already rare on pri­ors. “I don’t plan to make my drones give away money to ran­dom peo­ple” means he’s imag­in­ing how his sys­tem could work as he in­tends, in­stead of imag­in­ing how it might not work as he in­tends. If some­body doesn’t even ex­hibit or­di­nary para­noia, spon­ta­neously on their own cog­nizance with­out ex­ter­nal prompt­ing, then they can­not do se­cu­rity, pe­riod. Re­act­ing in­dig­nantly to the sug­ges­tion that some­thing might go wrong is even be­yond that level of hope­less­ness, but the base level was hope­less enough already.

am­ber: Look… can you just go to Mr. Topaz and try to tell him what he needs to do to add some se­cu­rity onto his drones? Just try? Be­cause it’s su­per im­por­tant.

coral: I could try, yes. I can’t suc­ceed, but I could try.

am­ber: Oh, but please be care­ful to not be harsh with him. Don’t put the fo­cus on what he’s do­ing wrong—and try to make it clear that these prob­lems aren’t too se­ri­ous. He’s been put off by the me­dia alarmism sur­round­ing apoc­a­lyp­tic sce­nar­ios with armies of evil drones filling the sky, and it took me some trou­ble to con­vince him that I wasn’t just an­other alarmist full of fan­ciful catas­tro­phe sce­nar­ios of drones defy­ing their own pro­gram­ming.

coral:

am­ber: And maybe try to keep your open­ing con­ver­sa­tion away from what might sound like crazy edge cases, like some­body for­get­ting to check the end of a buffer and an ad­ver­sary throw­ing in a huge string of char­ac­ters that over­write the end of the stack with a re­turn ad­dress that jumps to a sec­tion of code some­where else in the sys­tem that does some­thing the ad­ver­sary wants. I mean, you’ve con­vinced me that these far-fetched sce­nar­ios are worth wor­ry­ing about, if only be­cause they might be ca­naries in the coal mine for more re­al­is­tic failure modes. But Mr. Topaz thinks that’s all a bit silly, and I don’t think you should open by try­ing to ex­plain to him on a meta level why it isn’t. He’d prob­a­bly think you were be­ing con­de­scend­ing, tel­ling him how to think. Espe­cially when you’re just an op­er­at­ing-sys­tems guy and you have no ex­pe­rience build­ing drones and see­ing what ac­tu­ally makes them crash. I mean, that’s what I think he’d say to you.

coral:

am­ber: Also, start with the cheaper in­ter­ven­tions when you’re giv­ing ad­vice. I don’t think Mr. Topaz is go­ing to re­act well if you tell him that he needs to start all over in an­other pro­gram­ming lan­guage, or es­tab­lish a re­view board for all code changes, or what­ever. He’s wor­ried about com­peti­tors reach­ing the mar­ket first, so he doesn’t want to do any­thing that will slow him down.

coral:

am­ber: Uh, Co­ral?

coral: … on his novel pro­ject, en­ter­ing new ter­ri­tory, do­ing things not ex­actly like what has been done be­fore, car­ry­ing out novel mis­sion-crit­i­cal sub­tasks for which there are no stan­dard­ized best se­cu­rity prac­tices, nor any known un­der­stand­ing of what makes the sys­tem ro­bust or not-ro­bust.

am­ber: Right!

coral: And Mr. Topaz him­self does not seem much ter­rified of this ter­rify­ing task be­fore him.

am­ber: Well, he’s wor­ried about some­body else mak­ing mer­chant drones first and mi­sus­ing this key eco­nomic in­fras­truc­ture for bad pur­poses. That’s the same ba­sic thing, right? Like, it demon­strates that he can worry about things?

coral: It is ut­terly differ­ent. Mon­keys who can be afraid of other mon­keys get­ting to the ba­nanas first are far, far more com­mon than mon­keys who worry about whether the ba­nanas will ex­hibit weird sys­tem be­hav­iors in the face of ad­verse op­ti­miza­tion.

am­ber: Oh.

coral: I’m afraid it is only slightly more prob­a­ble that Mr. Topaz will over­see the cre­ation of ro­bust soft­ware than that the Moon will spon­ta­neously trans­form into or­gan­i­cally farmed goat cheese.

am­ber: I think you’re be­ing too harsh on him. I’ve met Mr. Topaz, and he seemed pretty bright to me.

coral: Again, as­sum­ing you’re rep­re­sent­ing him ac­cu­rately, Mr. Topaz seems to lack what I called or­di­nary para­noia. If he does have that abil­ity as a cog­ni­tive ca­pac­ity, which many bright pro­gram­mers do, then he ob­vi­ously doesn’t feel pas­sion­ate about ap­ply­ing that para­noia to his drone pro­ject along key di­men­sions. It also sounds like Mr. Topaz doesn’t re­al­ize there’s a skill that he is miss­ing, and would be in­sulted by the sug­ges­tion. I am put in mind of the story of the farmer who was asked by a pass­ing driver for di­rec­tions to get to Point B, to which the farmer replied, “If I was try­ing to get to Point B, I sure wouldn’t start from here.”

am­ber: Mr. Topaz has made some sig­nifi­cant ad­vances in drone tech­nol­ogy, so he can’t be stupid, right?

coral: “Se­cu­rity mind­set” seems to be a dis­tinct cog­ni­tive tal­ent from g fac­tor or even pro­gram­ming abil­ity. In fact, there doesn’t seem to be a level of hu­man ge­nius that even guaran­tees you’ll be skil­led at or­di­nary para­noia. Which does make some se­cu­rity pro­fes­sion­als feel a bit weird, my­self in­cluded—the same way a lot of pro­gram­mers have trou­ble un­der­stand­ing why not ev­ery­one can learn to pro­gram. But it seems to be an ob­ser­va­tional fact that both or­di­nary para­noia and se­cu­rity mind­set are things that can de­cou­ple from g fac­tor and pro­gram­ming abil­ity—and if this were not the case, the In­ter­net would be far more se­cure than it is.

am­ber: Do you think it would help if we talked to the other VCs fund­ing this pro­ject and got them to ask Mr. Topaz to ap­point a Spe­cial Ad­vi­sor on Ro­bust­ness re­port­ing di­rectly to the CTO? That sounds poli­ti­cally difficult to me, but it’s pos­si­ble we could swing it. Once the press started spec­u­lat­ing about drones go­ing rogue and maybe ag­gre­gat­ing into larger Voltron-like robots that could ac­quire laser eyes, Mr. Topaz did tell the VCs that he was very con­cerned about the ethics of drone safety and that he’d had many long con­ver­sa­tions about it over lunch hours.

coral: I’m ven­tur­ing slightly out­side my own ex­per­tise here, which isn’t cor­po­rate poli­tics per se. But on a pro­ject like this one that’s try­ing to en­ter novel ter­ri­tory, I’d guess the per­son with se­cu­rity mind­set needs at least cofounder sta­tus, and must be per­son­ally trusted by any cofounders who don’t have the skill. It can’t be an out­sider who was brought in by VCs, who is op­er­at­ing on limited poli­ti­cal cap­i­tal and needs to win an ar­gu­ment ev­ery time she wants to not have all the ser­vices con­ve­niently turned on by de­fault. I sus­pect you just have the wrong per­son in charge of this startup, and that this prob­lem is not re­pairable.

am­ber: Please don’t just give up! Even if things are as bad as you say, just in­creas­ing our pro­ject’s prob­a­bil­ity of be­ing se­cure from 0% to 10% would be very valuable in ex­pec­ta­tion to all those peo­ple in other coun­tries who need mer­chant drones.

coral: …look, at some point in life we have to try to triage our efforts and give up on what can’t be sal­vaged. There’s of­ten a lo­gis­tic curve for suc­cess prob­a­bil­ities, you know? The dis­tances are mea­sured in mul­ti­plica­tive odds, not ad­di­tive per­centage points. You can’t take a pro­ject like this and as­sume that by putting in some more hard work, you can in­crease the ab­solute chance of suc­cess by 10%. More like, the odds of this pro­ject’s failure ver­sus suc­cess start out as 1,000,000:1, and if we’re very po­lite and nav­i­gate around Mr. Topaz’s sense that he is higher-sta­tus than us and man­age to ex­plain a few tips to him with­out ever sound­ing like we think we know some­thing he doesn’t, we can quin­tu­ple his chances of suc­cess and send the odds to 200,000:1. Which is to say that in the world of per­centage points, the odds go from 0.0% to 0.0%. That’s one way to look at the “law of con­tinued failure”.

If you had the kind of pro­ject where the fun­da­men­tals im­plied, say, a 15% chance of suc­cess, you’d then be on the right part of the lo­gis­tic curve, and in that case it could make a lot of sense to hunt for ways to bump that up to a 30% or 80% chance.

am­ber: Look, I’m wor­ried that it will re­ally be very bad if Mr. Topaz reaches the mar­ket first with in­se­cure drones. Like, I think that mer­chant drones could be very benefi­cial to coun­tries with­out much ex­ist­ing mar­ket back­bone, and if there’s a grand failure—es­pe­cially if some of the would-be cus­tomers have their money or items stolen—then it could poi­son the po­ten­tial mar­ket for years. It will be ter­rible! Really, gen­uinely ter­rible!

coral: Wow. That sure does sound like an un­pleas­ant sce­nario to have wedged your­self into.

am­ber: But what do we do now?

coral: Damned if I know. I do sus­pect you’re screwed so long as you can only win if some­body like Mr. Topaz cre­ates a ro­bust sys­tem. I guess you could try to have some other drone pro­ject come into ex­is­tence, headed up by some­body that, say, Bruce Sch­neier as­sures ev­ery­one is un­usu­ally good at se­cu­rity-mind­set think­ing and hence can hire peo­ple like me and listen to all the harsh things we have to say. Though I have to ad­mit, the part where you think it’s dras­ti­cally im­por­tant that you beat an in­se­cure sys­tem to mar­ket with a se­cure sys­tem—well, that sounds pos­i­tively night­mar­ish. You’re go­ing to need a lot more re­sources than Mr. Topaz has, or some other kind of very ma­jor ad­van­tage. Se­cu­rity takes time.

am­ber: Is it re­ally that hard to add se­cu­rity to the drone sys­tem?

coral: You keep talk­ing about “adding” se­cu­rity. Sys­tem ro­bust­ness isn’t the kind of prop­erty you can bolt onto soft­ware as an af­terthought.

am­ber: I guess I’m hav­ing trou­ble see­ing why it’s so much more ex­pen­sive. Like, if some­body fool­ishly builds an OS that gives ac­cess to just any­one, you could in­stead put a pass­word lock on it, us­ing your clever sys­tem where the OS keeps the hashes of the pass­words in­stead of the pass­words. You just spend a cou­ple of days rewrit­ing all the ser­vices ex­posed to the In­ter­net to ask for pass­words be­fore grant­ing ac­cess. And then the OS has se­cu­rity on it! Right?

coral: NO. Every­thing in­side your sys­tem that is po­ten­tially sub­ject to ad­verse se­lec­tion in its prob­a­bil­ity of weird be­hav­ior is a li­a­bil­ity! Every­thing ex­posed to an at­tacker, and ev­ery­thing those sub­sys­tems in­ter­act with, and ev­ery­thing those parts in­ter­act with! You have to build all of it ro­bustly! If you want to build a se­cure OS you need a whole spe­cial pro­ject that is “build­ing a se­cure op­er­at­ing sys­tem in­stead of an in­se­cure op­er­at­ing sys­tem”. And you also need to re­strict the scope of your am­bi­tions, and not do ev­ery­thing you want to do, and obey other com­mand­ments that will feel like big un­pleas­ant sac­ri­fices to some­body who doesn’t have the full se­cu­rity mind­set. OpenBSD can’t do a tenth of what Ubuntu does. They can’t af­ford to! It would be too large of an at­tack sur­face! They can’t re­view that much code us­ing the spe­cial pro­cess that they use to de­velop se­cure soft­ware! They can’t hold that many as­sump­tions in their minds!

am­ber: Does that effort have to take a sig­nifi­cant amount of ex­tra time? Are you sure it can’t just be done in a cou­ple more weeks if we hurry?

coral: YES. Given that this is a novel pro­ject en­ter­ing new ter­ri­tory, ex­pect it to take at least two years more time, or 50% more de­vel­op­ment time—whichever is less—com­pared to a se­cu­rity-in­cau­tious pro­ject that oth­er­wise has iden­ti­cal tools, in­sights, peo­ple, and re­sources. And that is a very, very op­ti­mistic lower bound.

am­ber: This story seems to be head­ing in a wor­ry­ing di­rec­tion.

coral: Well, I’m sorry, but cre­at­ing ro­bust sys­tems takes longer than cre­at­ing non-ro­bust sys­tems even in cases where it would be re­ally, ex­traor­di­nar­ily bad if cre­at­ing ro­bust sys­tems took longer than cre­at­ing non-ro­bust sys­tems.

am­ber: Couldn’t it be the case that, like, pro­jects which are im­ple­ment­ing good se­cu­rity prac­tices do ev­ery­thing so much cleaner and bet­ter that they can come to mar­ket faster than any in­se­cure com­peti­tors could?

coral: … I hon­estly have trou­ble see­ing why you’re priv­ileg­ing that hy­poth­e­sis for con­sid­er­a­tion. Ro­bust­ness in­volves as­surance pro­cesses that take ad­di­tional time. OpenBSD does not go through lines of code faster than Ubuntu.

But more im­por­tantly, if ev­ery­one has ac­cess to the same tools and in­sights and re­sources, then an un­usu­ally fast method of do­ing some­thing cau­tiously can always be de­gen­er­ated into an even faster method of do­ing the thing in­cau­tiously. There is not now, nor will there ever be, a pro­gram­ming lan­guage in which it is the least bit difficult to write bad pro­grams. There is not now, nor will there ever be, a method­ol­ogy that makes writ­ing in­se­cure soft­ware in­her­ently slower than writ­ing se­cure soft­ware. Any se­cu­rity pro­fes­sional who heard about your bright hopes would just laugh. Ask them too if you don’t be­lieve me.

am­ber: But shouldn’t en­g­ineers who aren’t cau­tious just be un­able to make soft­ware at all, be­cause of or­di­nary bugs?

coral: I am afraid that it is both pos­si­ble, and ex­tremely com­mon in prac­tice, for peo­ple to fix all the bugs that are crash­ing their sys­tems in or­di­nary test­ing to­day, us­ing method­olo­gies that are in­deed ad­e­quate to fix­ing or­di­nary bugs that show up of­ten enough to af­flict a sig­nifi­cant frac­tion of users, and then ship the product. They get ev­ery­thing work­ing to­day, and they don’t feel like they have the slack to de­lay any longer than that be­fore ship­ping be­cause the product is already be­hind sched­ule. They don’t hire ex­cep­tional peo­ple to do ten times as much work in or­der to pre­vent the product from hav­ing holes that only show up un­der ad­verse op­ti­miza­tion pres­sure, that some­body else finds first and that they learn about af­ter it’s too late.

It’s not even the wrong de­ci­sion, for prod­ucts that aren’t con­nected to the In­ter­net, don’t have enough users for one to go rogue, don’t han­dle money, don’t con­tain any valuable data, and don’t do any­thing that could in­jure peo­ple if some­thing goes wrong. If your soft­ware doesn’t de­stroy any­thing im­por­tant when it ex­plodes, it’s prob­a­bly a bet­ter use of limited re­sources to plan on fix­ing bugs as they show up.

… Of course, you need some amount of se­cu­rity mind­set to re­al­ize which soft­ware can in fact de­stroy the com­pany if it silently cor­rupts data and no­body no­tices this un­til a month later. I don’t sup­pose it’s the case that your drones only carry a limited amount of the full cor­po­rate bud­get in cash over the course of a day, and you always have more than enough money to re­im­burse all the cus­tomers if all items in tran­sit over a day were lost, tak­ing into ac­count that the drones might make many more pur­chases or sales than usual? And that the sys­tems are gen­er­at­ing in­ter­nal pa­per re­ceipts that are clearly shown to the cus­tomer and non-elec­tron­i­cally rec­on­ciled once per day, thereby en­abling you to no­tice a prob­lem be­fore it’s too late?

am­ber: Nope!

coral: Then as you say, it would be bet­ter for the world if your com­pany didn’t ex­ist and wasn’t about to charge into this new ter­ri­tory and poi­son it with a spec­tac­u­lar screwup.

am­ber: If I be­lieved that… well, Mr. Topaz cer­tainly isn’t go­ing to stop his pro­ject or let some­body else take over. It seems the log­i­cal im­pli­ca­tion of what you say you be­lieve is that I should try to per­suade the ven­ture cap­i­tal­ists I know to launch a safer drone pro­ject with even more fund­ing.

coral: Uh, I’m sorry to be blunt about this, but I’m not sure you have a high enough level of se­cu­rity mind­set to iden­tify an ex­ec­u­tive who’s suffi­ciently bet­ter than you at it. Try­ing to get enough of a re­source ad­van­tage to beat the in­se­cure product to mar­ket is only half of your prob­lem in launch­ing a com­pet­ing pro­ject. The other half of your prob­lem is sur­pass­ing the prior rar­ity of peo­ple with truly deep se­cu­rity mind­set, and get­ting some­body like that in charge and fully com­mit­ted. Or at least get them in as a highly trusted, fully com­mit­ted cofounder who isn’t on a short bud­get of poli­ti­cal cap­i­tal. I’ll say it again: an ad­vi­sor ap­pointed by VCs isn’t nearly enough for a pro­ject like yours. Even if the ad­vi­sor is a gen­uinely good se­cu­rity pro­fes­sional—

am­ber: This all seems like an un­rea­son­ably difficult re­quire­ment! Can’t you back down on it a lit­tle?

coral: —the per­son in charge will prob­a­bly try to bar­gain down re­al­ity, as rep­re­sented by the un­wel­come voice of the se­cu­rity pro­fes­sional, who won’t have enough so­cial cap­i­tal to bad­ger them into “un­rea­son­able” mea­sures. Which means you fail on full au­to­matic.

am­ber: … Then what am I to do?

coral: I don’t know, ac­tu­ally. But there’s no point in launch­ing an­other drone pro­ject with even more fund­ing, if it just ends up with an­other Mr. Topaz put in charge. Which, by de­fault, is ex­actly what your ven­ture cap­i­tal­ist friends are go­ing to do. Then you’ve just set an even higher com­pet­i­tive bar for any­one ac­tu­ally try­ing to be first to mar­ket with a se­cure solu­tion, may God have mercy on their souls.

Be­sides, if Mr. Topaz thinks he has a com­peti­tor breath­ing down his neck and rushes his product to mar­ket, his chance of cre­at­ing a se­cure sys­tem could drop by a fac­tor of ten and go all the way from 0.0% to 0.0%.

am­ber: Surely my VC friends have faced this kind of prob­lem be­fore and know how to iden­tify and hire ex­ec­u­tives who can do se­cu­rity well?

coral: … If one of your VC friends is Paul Gra­ham, then maybe yes. But in the av­er­age case, NO.

If av­er­age VCs always made sure that pro­jects which needed se­cu­rity had a founder or cofounder with strong se­cu­rity mind­set—if they had the abil­ity to do that even in cases where they de­cided they wanted to—the In­ter­net would again look like a very differ­ent place. By de­fault, your VC friends will be fooled by some­body who looks very sober and talks a lot about how ter­ribly con­cerned he is with cy­ber­se­cu­rity and how the sys­tem is go­ing to be ul­tra-se­cure and re­ject over nine thou­sand com­mon pass­words, in­clud­ing the thirty-six pass­words listed on this slide here, and the VCs will ooh and ah over it, es­pe­cially as one of them re­al­izes that their own pass­word is on the slide. That pro­ject leader is ab­solutely not go­ing to want to hear from me—even less so than Mr. Topaz. To him, I’m a poli­ti­cal threat who might dam­age his line of pat­ter to the VCs.

am­ber: I have trou­ble be­liev­ing all these smart peo­ple are re­ally that stupid.

coral: You’re com­press­ing your in­nate sense of so­cial sta­tus and your es­ti­mated level of how good par­tic­u­lar groups are at this par­tic­u­lar abil­ity into a sin­gle di­men­sion. That is not a good idea.

am­ber: I’m not say­ing that I think ev­ery­one with high sta­tus already knows the deep se­cu­rity skill. I’m just hav­ing trou­ble be­liev­ing that they can’t learn it quickly once told, or could be stuck not be­ing able to iden­tify good ad­vi­sors who have it. That would mean they couldn’t know some­thing you know, some­thing that seems im­por­tant, and that just… feels off to me, some­how. Like, there are all these suc­cess­ful and im­por­tant peo­ple out there, and you’re say­ing you’re bet­ter than them, even with all their in­fluence, their skills, their re­sources—

coral: Look, you don’t have to take my word for it. Think of all the web­sites you’ve been on, with snazzy-look­ing de­sign, maybe with mil­lions of dol­lars in sales pass­ing through them, that want your pass­word to be a mix­ture of up­per­case and low­er­case let­ters and num­bers. In other words, they want you to en­ter “Pass­word1!” in­stead of “cor­rect horse bat­tery sta­ple”. Every one of those web­sites is do­ing a thing that looks hu­morously silly to some­one with a full se­cu­rity mind­set or even just some­body who reg­u­larly reads XKCD. It says that the se­cu­rity sys­tem was set up by some­body who didn’t know what they were do­ing and was blindly imi­tat­ing im­pres­sive-look­ing mis­takes they saw el­se­where.

Do you think that makes a good im­pres­sion on their cus­tomers? That’s right, it does! Be­cause the cus­tomers don’t know any bet­ter. Do you think that lo­gin sys­tem makes a good im­pres­sion on the com­pany’s in­vestors, in­clud­ing pro­fes­sional VCs and prob­a­bly some an­gels with their own startup ex­pe­rience? That’s right, it does! Be­cause the VCs don’t know any bet­ter, and even the an­gel doesn’t know any bet­ter, and they don’t re­al­ize they’re miss­ing a vi­tal skill, and they aren’t con­sult­ing any­one who knows more. An in­no­cent is im­pressed if a web­site re­quires a mix of up­per­case and low­er­case let­ters and num­bers and punc­tu­a­tion. They think the peo­ple run­ning the web­site must re­ally care to im­pose a se­cu­rity mea­sure that un­usual and in­con­ve­nient. The peo­ple run­ning the web­site think that’s what they’re do­ing too.

Peo­ple with deep se­cu­rity mind­set are both rare and rarely ap­pre­ci­ated. You can see just from the lo­gin sys­tem that none of the VCs and none of the C-level ex­ec­u­tives at that startup thought they needed to con­sult a real pro­fes­sional, or man­aged to find a real pro­fes­sional rather than an empty suit if they went con­sult­ing. There was, visi­bly, no­body in the neigh­bor­ing sys­tem with the com­bined knowl­edge and sta­tus to walk over to the CEO and say, “Your lo­gin sys­tem is em­bar­rass­ing and you need to hire a real se­cu­rity pro­fes­sional.” Or if any­body did say that to the CEO, the CEO was offended and shot the mes­sen­ger for not phras­ing it ever-so-po­litely enough, or the CTO saw the out­sider as a poli­ti­cal threat and bad-mouthed them out of the game.

Your wish­ful should-uni­verse hy­poth­e­sis that peo­ple who can touch the full se­cu­rity mind­set are more com­mon than that within the ven­ture cap­i­tal and an­gel in­vest­ing ecosys­tem is just flat wrong. Or­di­nary para­noia di­rected at widely-known ad­ver­sar­ial cases is dense enough within the larger ecosys­tem to ex­ert wide­spread so­cial in­fluence, albeit still com­i­cally ab­sent in many in­di­vi­d­u­als and re­gions. Peo­ple with the full se­cu­rity mind­set are too rare to have the same level of pres­ence. That’s the eas­ily visi­ble truth. You can see the lo­gin sys­tems that want a punc­tu­a­tion mark in your pass­word. You are not hal­lu­ci­nat­ing them.

am­ber: If that’s all true, then I just don’t see how I can win. Maybe I should just con­di­tion on ev­ery­thing you say be­ing false, since, if it’s true, my win­ning seems un­likely—in which case all vic­to­ries on my part would come in wor­lds with other back­ground as­sump­tions.

coral: … is that some­thing you say of­ten?

am­ber: Well, I say it when­ever my vic­tory starts to seem suffi­ciently un­likely.

coral: Good­ness. I could maybe, maybe see some­body say­ing that once over the course of their en­tire life­time, for a sin­gle un­likely con­di­tional, but do­ing it more than once is sheer mad­ness. I’d ex­pect the un­likely con­di­tion­als to build up very fast and drop the prob­a­bil­ity of your men­tal world to effec­tively zero. It’s tempt­ing, but it’s usu­ally a bad idea to slip side­ways into your own pri­vate hal­lu­ci­na­tory uni­verse when you feel you’re un­der emo­tional pres­sure. I tend to be­lieve that no mat­ter what the difficul­ties, we are most likely to come up with good plans when we are men­tally liv­ing in re­al­ity as op­posed to some­where else. If things seem difficult, we must face the difficulty squarely to suc­ceed, to come up with some solu­tion that faces down how bad the situ­a­tion re­ally is, rather than de­cid­ing to con­di­tion on things not be­ing difficult be­cause then it’s too hard.

am­ber: Can you at least try talk­ing to Mr. Topaz and ad­vise him how to make things be se­cure?

coral: Sure. Try­ing things is easy, and I’m a char­ac­ter in a di­alogue, so my op­por­tu­nity costs are low. I’m sure Mr. Topaz is try­ing to build se­cure mer­chant drones, too. It’s suc­ceed­ing at things that is the hard part.

am­ber: Great, I’ll see if I can get Mr. Topaz to talk to you. But do please be po­lite! If you think he’s do­ing some­thing wrong, try to point it out more gen­tly than the way you’ve talked to me. I think I have enough poli­ti­cal cap­i­tal to get you in the door, but that won’t last if you’re rude.

coral: You know, back in main­stream com­puter se­cu­rity, when you pro­pose a new way of se­cur­ing a sys­tem, it’s con­sid­ered tra­di­tional and wise for ev­ery­one to gather around and try to come up with rea­sons why your idea might not work. It’s un­der­stood that no mat­ter how smart you are, most seem­ingly bright ideas turn out to be flawed, and that you shouldn’t be touchy about peo­ple try­ing to shoot them down. Does Mr. Topaz have no ac­quain­tance at all with the prac­tices in com­puter se­cu­rity? A lot of pro­gram­mers do.

am­ber: I think he’d say he re­spects com­puter se­cu­rity as its own field, but he doesn’t be­lieve that build­ing se­cure op­er­at­ing sys­tems is the same prob­lem as build­ing mer­chant drones.

coral: And if I sug­gested that this case might be similar to the prob­lem of build­ing a se­cure op­er­at­ing sys­tem, and that this case cre­ates a similar need for more effort­ful and cau­tious de­vel­op­ment, re­quiring both (a) ad­di­tional de­vel­op­ment time and (b) a spe­cial need for cau­tion sup­plied by peo­ple with un­usual mind­sets above and be­yond or­di­nary para­noia, who have an un­usual skill that iden­ti­fies shaky as­sump­tions in a safety story be­fore an or­di­nary para­noid would judge a fire as be­ing ur­gent enough to need putting out, who can rem­edy the prob­lem us­ing deeper solu­tions than an or­di­nary para­noid would gen­er­ate as par­ries against imag­ined at­tacks?

If I sug­gested, in­deed, that this sce­nario might hold gen­er­ally wher­ever we de­mand ro­bust­ness of a com­plex sys­tem that is be­ing sub­jected to strong ex­ter­nal or in­ter­nal op­ti­miza­tion pres­sures? Pres­sures that strongly pro­mote the prob­a­bil­ities of par­tic­u­lar states of af­fairs via op­ti­miza­tion that searches across a large and com­plex state space? Pres­sures which there­fore in turn sub­ject other sub­parts of the sys­tem to se­lec­tion for weird states and pre­vi­ously un­en­vi­sioned ex­e­cu­tion paths? Espe­cially if some of these pres­sures may be in some sense cre­ative and find states of the sys­tem or en­vi­ron­ment that sur­prise us or vi­o­late our sur­face gen­er­al­iza­tions?

am­ber: I think he’d prob­a­bly think you were try­ing to look smart by us­ing overly ab­stract lan­guage at him. Or he’d re­ply that he didn’t see why this took any more cau­tion than he was already us­ing just by test­ing the drones to make sure they didn’t crash or give out too much money.

coral: I see.

am­ber: So, shall we be off?

coral: Of course! No prob­lem! I’ll just go meet with Mr. Topaz and use ver­bal per­sua­sion to turn him into Bruce Sch­neier.

am­ber: That’s the spirit!

coral: God, how I wish I lived in the ter­ri­tory that cor­re­sponds to your map.

am­ber: Hey, come on. Is it se­ri­ously that hard to be­stow ex­cep­tion­ally rare men­tal skills on peo­ple by talk­ing at them? I agree it’s a bad sign that Mr. Topaz shows no sign of want­ing to ac­quire those skills, and doesn’t think we have enough rel­a­tive sta­tus to con­tinue listen­ing if we say some­thing he doesn’t want to hear. But that just means we have to phrase our ad­vice clev­erly so that he will want to hear it!

coral: I sup­pose you could mod­ify your mes­sage into some­thing Mr. Topaz doesn’t find so un­pleas­ant to hear. Some­thing that sounds re­lated to the topic of drone se­cu­rity, but which doesn’t cost him much, and of course does not ac­tu­ally cause his drones to end up se­cure be­cause that would be all un­pleas­ant and ex­pen­sive. You could slip a lit­tle side­ways in re­al­ity, and con­vince your­self that you’ve got­ten Mr. Topaz to ally with you, be­cause he sounds agree­able now. Your in­stinc­tive de­sire for the high-sta­tus mon­key to be on your poli­ti­cal side will feel like its prob­lem has been solved. You can sub­sti­tute the feel­ing of hav­ing solved that prob­lem for the un­pleas­ant sense of not hav­ing se­cured the ac­tual drones; you can tell your­self that the big­ger mon­key will take care of ev­ery­thing now that he seems to be on your pleas­antly-mod­ified poli­ti­cal side. And so you will be happy. Un­til the mer­chant drones hit the mar­ket, of course, but that un­pleas­ant ex­pe­rience should be brief.

am­ber: Come on, we can do this! You’ve just got to think pos­i­tively!

coral: … Well, if noth­ing else, this should be an in­ter­est­ing ex­pe­rience. I’ve never tried to do any­thing quite this doomed be­fore.

• For ac­tual com­puter se­cu­rity, not AGI, why do you be­lieve that the lax­ness of most soft­ware is a prob­lem rather than a rea­son­able re­sponse?

My un­der­stand­ing is that most peo­ple’s credit card in­for­ma­tion is known to at least some iden­tity thieves. This does not drive most in­di­vi­d­u­als into a panic be­cause we’ve ba­si­cally never heard of a case in our so­cial cir­cle of some­one los­ing more than dol­lars to iden­tity theft—and any­way, banks can re­im­burse the money. It does worry banks and other com­pa­nies that pro­cess iden­ti­fy­ing in­for­ma­tion—and they hire con­sul­tants to help them de­tect and pre­vent iden­tity theft—but they don’t de­vote nearly as many re­sources to this as Bruce Sch­neier would like. (I worked at one of those con­sult­ing com­pa­nies, which is how I know first­hand that fraud pre­ven­tion can be pretty dys­func­tional even at house­hold-name, big, suc­cess­ful com­pa­nies.)

But are they ac­tu­ally mak­ing a strate­gic mis­take? Is the num­ber & power of fraud­sters go­ing to sud­denly in­crease? Or isn’t the situ­a­tion more like “We lose X dol­lars to fraud ev­ery year, we’d like it to be lower but to be hon­est it’s a tiny frac­tion of our yearly costs. We’d go out of busi­ness if our fraud losses were 100x more, but that’s pretty un­likely. So we take these fraud pre­cau­tions—but not the much more elab­o­rate ones that our com­puter se­cu­rity ex­perts would like.”

If se­cu­rity ex­perts are right, then the world fi­nan­cial sys­tem is go­ing to col­lapse any day now. If not, then the sort of se­cu­rity mind­set that is ap­pro­pri­ate to de­sign­ing one piece of soft­ware, where any vuln­er­a­bil­ity should be treated as dan­ger­ous, is not ac­tu­ally ap­pro­pri­ate to al­lo­cat­ing re­sources to se­cu­rity con­cerns in a big or­ga­ni­za­tion or in­dus­try.

• I’m go­ing to naively take Sch­neier’s side here on the grounds that if I no­tice a bunch of small holes in my se­cu­rity I have model un­cer­tainty about those holes un­ex­pect­edly grow­ing larger. I would try to es­tab­lish some bounds on es­ti­mates of this new pa­ram­e­ter by look­ing for high pro­file in­stances. I wouldn’t be that re­as­sured by the fact that most of my peers didn’t seem con­cerned if, af­ter think­ing about it for a few min­utes, I couldn’t see how they were di­rectly in­cen­tivized to be con­cerned. Is this flawed?

• Co­ral isn’t try­ing very hard to be helpful. Why doesn’t she sug­gest that the com­pany offer \$10,000,000 for each se­cu­rity hole that peo­ple can demon­strate? Oh, right, she wants to use this as anal­ogy for AGIs that go foom.

• [Mod note: This com­ment was pre­vi­ously deleted by mods, and on re­flec­tion re­turned.]

From my per­spec­tive, Eliezer comes across as the AI safety equiv­a­lent of a mil­i­tant ve­gan or smug athe­ist in this post. I’m not aware of any so­cial sci­ence on the topic of whether peo­ple like this tend to be use­ful to their cause or not, but my per­sonal im­pres­sion has always been that they aren’t. Even though I agree with his core the­sis, I think posts like this plau­si­bly make it harder for some­one like me to have con­ver­sa­tions with AI peo­ple about safety.

• The post leans hard on the idea that se­cu­rity mind­set is some­thing in­nate you ei­ther have or don’t have, which, as I com­plained pre­vi­ously, is not well-sup­ported. Plenty of sneer­ing to­wards peo­ple who are as­sumed to lack it is un­helpful.

• The post also leans hard on the pass­word re­quire­ments ex­am­ple which I cri­tiqued pre­vi­ously here. This feels like an ex­am­ple of the Copen­hagen In­ter­pre­ta­tion of Ethics. Some com­pa­nies take a ba­sic step to help old peo­ple choose bet­ter pass­words, and they get sneered at… mean­while the many com­pa­nies that do noth­ing to help users choose bet­ter pass­words get off scot-free. Here, Co­ral re­minds me of a mil­i­tant ve­gan who speci­fi­cally tar­gets com­pa­nies that en­gage in hu­mane slaugh­ter prac­tices.

• The anal­ogy it­self is weak. Pay­pal is an ex­am­ple of a com­pany that be­came suc­cess­ful largely due to its abil­ity to com­bat fraud, de­spite hav­ing no idea that fraud is some­thing it would have to deal with in the be­gin­ning. (You can read the book Founders at Work to learn more about this—Pay­pal’s chap­ter is the very first one. Ques­tion for Eliezer: After read­ing the chap­ter, can you tell me whether you think Max Levchin is some­one who has se­cu­rity mind­set? Again, I would ar­gue that the ev­i­dence for Eliezer’s view of se­cu­rity mind­set is very shakey. If I had to bet on ei­ther (a) a de­vel­oper with “or­di­nary para­noia”, us­ing tools they are very fa­mil­iar with, who is given se­cu­rity as a de­sign goal, vs (b) a de­vel­oper with “se­cu­rity mind­set”, us­ing tools they aren’t fa­mil­iar with, who isn’t given se­cu­rity as a de­sign goal, I’d bet on (a). More broadly, Eliezer’s use of “se­cu­rity mind­set” looks kinda like an at­tempt to sneak in the con­no­ta­tion that any­one who doesn’t re­al­ize se­cu­rity was a de­sign goal is in­ca­pable of writ­ing se­cure soft­ware. Note: It’s of­ten not ra­tio­nal for soft­ware com­pa­nies to have se­cu­rity as a de­sign goal.) And Paul Gra­ham writes (granted, this was 2005, so it’s pos­si­ble his opinion has changed in the in­ter­ven­ing 12 years):

...we ad­vise groups to ig­nore is­sues like scal­a­bil­ity, in­ter­na­tion­al­iza­tion, and heavy-duty se­cu­rity at first. [1] I can imag­ine an ad­vo­cate of “best prac­tices” say­ing these ought to be con­sid­ered from the start. And he’d be right, ex­cept that they in­terfere with the pri­mary func­tion of soft­ware in a startup: to be a ve­hi­cle for ex­per­i­ment­ing with its own de­sign. Hav­ing to retrofit in­ter­na­tion­al­iza­tion or scal­a­bil­ity is a pain, cer­tainly. The only big­ger pain is not need­ing to, be­cause your ini­tial ver­sion was too big and rigid to evolve into some­thing users wanted.
• To trans­late Gra­ham’s state­ment back to the FAI prob­lem: In Eliezer’s al­ign­ment talk, he dis­cusses the value of solv­ing a re­laxed con­straint ver­sion of the FAI prob­lem by grant­ing one­self un­limited com­put­ing power. Well, in the same way, the AGI prob­lem can be seen as a re­laxed con­straint ver­sion of the FAI prob­lem. One could ar­gue that it’s a waste of time to try to make a se­cure ver­sion of AGI Ap­proach X if we don’t even know if it’s pos­si­ble to build an AGI us­ing AGI Ap­proach X. (I don’t agree with this view, but I don’t think it’s en­tirely un­rea­son­able.)

• As far as I can tell, Co­ral’s OpenBSD anal­ogy is flat out wrong. Co­ral doesn’t ap­pear to be fa­mil­iar with the con­cept of a “trusted com­put­ing base” (see these lec­ture notes that I linked from the com­ments of his pre­vi­ous post). The most ex­cit­ing pro­ject I know of in the OS se­cu­rity space is Qubes, which, in a cer­tain sense, is do­ing ex­actly what Co­ral says can’t be done. This is a de­cent blog post on the philos­o­phy be­hind Qubes.

• Again, trans­lat­ing this info back to the FAI prob­lem: An­drew Wiles once said:

Per­haps I could best de­scribe my ex­pe­rience of do­ing math­e­mat­ics in terms of en­ter­ing a dark man­sion. One goes into the first room, and it’s dark, com­pletely dark. One stum­bles around bump­ing into the fur­ni­ture, and grad­u­ally, you learn where each piece of fur­ni­ture is, and fi­nally, af­ter six months or so, you find the light switch. You turn it on, and sud­denly, it’s all illu­mi­nated. You can see ex­actly where you were.
• My in­ter­pre­ta­tion of this state­ment is that the key to solv­ing difficult prob­lems is to find a way to frame them that makes them seem easy. As some­one who works for an AI safety non­profit, Eliezer has a fi­nan­cial in­ter­est in mak­ing AI safety seem as difficult as pos­si­ble. Un­for­tu­nately, while that is prob­a­bly a great strat­egy for gath­er­ing dona­tions, it might not be a good strat­egy for ac­tu­ally solv­ing the prob­lem. The Qubes pro­ject is in­ter­est­ing be­cause some­one thought of a way to re­frame the OS se­cu­rity prob­lem to make it much more tractable. (In­stead of need­ing a 100% cor­rect OS, now to a first ap­prox­i­ma­tion you need a 100% cor­rect hy­per­vi­sor.) I’m not nec­es­sar­ily say­ing MIRI is over­funded. But I think MIRI re­searchers, when they are try­ing to ac­tu­ally make progress on FAI, need to rec­og­nize and push back against in­sti­tu­tional in­cen­tives to frame FAI in a way that makes it seem as hard as pos­si­ble to solve. Eliezer’s ad­mo­ni­tion to not “try to solve the whole prob­lem at once” seems like the wrong thing to say, from my per­spec­tive.

Frankly, in­so­far as se­cu­rity mind­set is a thing, this post comes across to me as be­ing writ­ten by some­one who doesn’t have it. I don’t get the sense that the au­thor has even “or­di­nary para­noia” about the pos­si­bil­ity that a post like this could be harm­ful, de­spite the fact Nick Bostrom’s non-con­fronta­tional ad­vo­cacy ap­proach seems to have done a lot more to ex­pand the AI safety Over­ton win­dow, and de­spite the pos­si­bil­ity that a post like this might in­crease already-ex­ist­ing poli­ti­ciza­tion of AI safety. (I’m not even sure Eliezer has a solid story for how this post could end up be­ing helpful!)

Similarly, when I think of the peo­ple I would trust the most to write se­cure soft­ware, Co­ral’s 1,000,000:1 odds es­ti­mate does not seem like the kind of thing they would say—I’d sooner trust some­one who is much more self-skep­ti­cal and spends a lot more time ac­count­ing for model un­cer­tainty. (Model un­cer­tainty and self-skep­ti­cism are both big parts of se­cu­rity mind­set!) Does Co­ral think she can make a mil­lion pre­dic­tions like this and be in­cor­rect on only one of them? How much time did Co­ral spend look­ing in the dark for sto­ries like Max Levchin’s which don’t fit their mod­els?

(An even more straight­for­ward ar­gu­ment for why Eliezer lacks se­cu­rity mind­set: Eliezer says se­cu­rity mind­sest is an in­nate char­ac­ter­is­tic, which means it’s pre­sent from birth. SIAI was founded as an or­ga­ni­za­tion to de­velop seed AI with­out re­gard for safety. The fact that Eliezer didn’t in­stantly re­al­ize the im­por­tance of friendli­ness when pre­sented with the no­tion of seed AI means he lacks se­cu­rity mind­set, and since se­cu­rity mind­set is pre­sent from birth, he doesn’t have it now ei­ther.)

The fact that I con­sider my­self a non-ex­pert on both soft­ware se­cu­rity and AI, yet I’m able to come up with these ob­vi­ous-seem­ing coun­ter­ar­gu­ments, does not bode well. (Note: I keep harp­ing on my non-ex­pert sta­tus be­cause I’m afraid some­one will take my com­ments as literal truth, but I’m acutely aware that I’m just shar­ing facts I re­mem­ber read­ing and ideas I’ve ran­domly had. I want peo­ple to feel com­fortable giv­ing me push­back if they have bet­ter in­for­ma­tion, the way peo­ple don’t seem to do with Eliezer. In fact, this is the main rea­son why I write so many com­ments and so few toplevel posts—peo­ple seem more will­ing to ac­cept toplevel posts as set­tled fact, even if the re­search be­hind them is ac­tu­ally pretty poor. How’s that for se­cu­rity mind­set? :P)

As a side note, I read & agreed with Eliezer’s Inad­e­quate Equil­ibria book. In fact, I’m plan­ning to buy it as a Christ­mas pre­sent for a friend. So if there’s a deeper dis­agree­ment here, that’s not it. A quick shot at iden­ti­fy­ing the deeper dis­agree­ment: From my per­spec­tive, Eliezer fre­quently seems to fall prey to the “What You See Is All There Is” phe­nomenon that Daniel Kah­ne­man de­scribes in Think­ing Fast and Slow. For ex­am­ple, in this di­alogue, the pro­tag­o­nist says that for a se­cure op­er­at­ing sys­tem, “Every­thing ex­posed to an at­tacker, and ev­ery­thing those sub­sys­tems in­ter­act with, and ev­ery­thing those parts in­ter­act with! You have to build all of it ro­bustly!” But this doesn’t ap­pear to ac­tu­ally be true (see Qubes). Just be­cause Eliezer is un­able see a way to do it, doesn’t mean it can’t be done.

• The post leans hard on the idea that se­cu­rity mind­set is some­thing in­nate you ei­ther have or don’t have, which, as I com­plained pre­vi­ously, is not well-sup­ported.

Agreed Eliezer hasn’t given much ev­i­dence for the claim that se­cu­rity mind­set is of­ten un­train­able; it’s good that you’re flag­ging this ex­plic­itly. I think his goal was to pro­mote the “not just any­one can be trained to think like Bruce Sch­neier” hy­poth­e­sis to read­ers’ at­ten­tion, and to say what his own cur­rent model is, not to defend the model in any de­tail.

I found the change in fo­cus use­ful my­self be­cause Inad­e­quate Equil­ibria talks so lit­tle about in­nate com­pe­tence and in­di­vi­d­ual skill gaps, even though it’s clearly one of the cen­tral puz­zle pieces.

This feels like an ex­am­ple of the Copen­hagen In­ter­pre­ta­tion of Ethics. Some com­pa­nies take a ba­sic step to help old peo­ple choose bet­ter pass­words, and they get sneered at… mean­while the many com­pa­nies that do noth­ing to help users choose bet­ter pass­words get off scot-free.

It might not be fair, but I think that’s fine here. The im­por­tant ques­tion is whether the world is ad­e­quate in cer­tain re­spects (e.g., at con­vert­ing re­sources into some level of se­cu­rity and pri­vacy for the av­er­age user), and what that im­plies in do­mains like AI that we care about. I don’t ex­pect com­pa­nies with in­ad­e­quate pass­word sys­tems to suffer any great harm from a blog post crit­i­ciz­ing them with­out spend­ing an equal num­ber of para­graphs crit­i­ciz­ing or­ga­ni­za­tions with even worse se­cu­rity prac­tices. The most ma­te­rial ques­tion is whether pass­word stan­dards are in fact triv­ial to im­prove on in a way that makes users and com­pa­nies much bet­ter off; it’s not clear to me whether we dis­agree about that, since it might be that you’re just fo­cus­ing on a lower ad­e­quacy thresh­old.

The Qubes pro­ject is in­ter­est­ing be­cause some­one thought of a way to re­frame the OS se­cu­rity prob­lem to make it much more tractable. (In­stead of need­ing a 100% cor­rect OS, now to a first ap­prox­i­ma­tion you need a 100% cor­rect hy­per­vi­sor.)

I don’t know much about Qubes, but the idea of mod­u­lariz­ing the prob­lem, dis­t­in­guish­ing trusted and un­trusted sys­tem com­po­nents, min­i­miz­ing re­li­ance on less trusted com­po­nents, and look­ing for work-arounds to make things as easy as pos­si­ble (with­out as­sum­ing they’re easy), sounds like or­di­nary MIRI re­search prac­tice. Eliezer’s idea of cor­rigi­bil­ity is an ex­am­ple of this ap­proach, and Eliezer’s said that if al­ign­ment turns out to be sur­pris­ingly easy, one of the like­liest paths is if there turns out to be a good-enough con­cept of cor­rigi­bil­ity that’s easy to train into sys­tems.

Eliezer’s ad­mo­ni­tion to not “try to solve the whole prob­lem at once” seems like the wrong thing to say, from my per­spec­tive.

Am­bi­tious efforts to take a huge chunk out of the prob­lem, or to find some hacky or el­e­gant way to route around a cen­tral difficulty, seem good to me. I haven’t seen peo­ple make much progress if they “try to solve the whole prob­lem at once” with a few min­utes/​hours of ex­pe­rience think­ing through the prob­lem rather than a few months/​years; usu­ally that looks less like cor­rigi­bil­ity or ALBA and more like “well we’ll shut it off if it starts do­ing scary stuff” or “well we’ll just raise it like a hu­man child”.

I don’t get the sense that the au­thor has even “or­di­nary para­noia” about the pos­si­bil­ity that a post like this could be harmful

It sounds like you’re mak­ing a pre­dic­tion here that Eliezer and oth­ers didn’t put much thought in ad­vance into the po­ten­tial risks or costs of this post. Hav­ing talked with Eliezer and oth­ers about the post be­fore­hand, I can con­firm that this pre­dic­tion is false.

The fact that I con­sider my­self a non-ex­pert on both soft­ware se­cu­rity and AI, yet I’m able to come up with these ob­vi­ous-seem­ing coun­ter­ar­gu­ments, does not bode well.

I think you’re over­es­ti­mat­ing how much over­lap there is be­tween what differ­ent peo­ple tend to think are the most ob­vi­ous coun­ter­ar­gu­ments to this or that AGI al­ign­ment ar­gu­ment. This is ac­tu­ally a hard prob­lem. If Alice thinks counter-ar­gu­ments A and B are ob­vi­ous, Bob thinks counter-ar­gu­ments B and C are ob­vi­ous, and Carol thinks counter-ar­gu­ments C and D are ob­vi­ous, and you only have time to cover two counter-ar­gu­ments be­fore the post gets overly long, then no mat­ter which ar­gu­ments you choose you’ll end up with most read­ers think­ing that you’ve ne­glected one or more “ob­vi­ous” counter-ar­gu­ments.

At the same time, if you try to ad­dress as many counter-ar­gu­ments as you can given length con­straints, you’ll in­evitably end up with most read­ers feel­ing baf­fled at why you’re wast­ing time on triv­ial or straw counter-ar­gu­ments that they don’t care about.

This is also made more difficult if you have to re­ply to all the counter-ar­gu­ments Alice dis­agrees with but thinks some­one else might agree with: Alice might be wrong about who is (or should be) in the tar­get au­di­ence, she might be wrong about the be­liefs of this or that po­ten­tial tar­get au­di­ence, or she might just have an im­prac­ti­cally long list of counter-ar­gu­ments to cover (due to not re­strict­ing her­self to what she thinks is true or even all that prob­a­ble). I think that group dis­cus­sions of­ten end up go­ing in un­pro­duc­tive di­rec­tions when hy­po­thet­i­cal dis­agree­ments take the place of ac­tual dis­agree­ments.

• Thanks for the re­sponse!

The most ma­te­rial ques­tion is whether pass­word stan­dards are in fact triv­ial to im­prove on in a way that makes users and com­pa­nies much bet­ter off; it’s not clear to me whether we dis­agree about that, since it might be that you’re just fo­cus­ing on a lower ad­e­quacy thresh­old.

If there’s a triv­ial way to mea­sure pass­word strength, the method has not oc­curred to me. Sup­pose my pass­word gen­er­a­tion al­gorithm ran­domly sam­ples my pass­word from the set of all alphanu­meric strings that are be­tween 6 and 20 char­ac­ters long. That’s 715971350555965203672729120482208448 pos­si­ble pass­words I’m choos­ing from. Sounds pretty se­cure right? Well, two of those alphanu­meric strings be­tween 6 and 20 char­ac­ters are “aaaaaaaaaa” and “pass­word123″. A server that just sees “aaaaaaaaaa” as my pass­word has no way a pri­ori to know what al­gorithm I used to gen­er­ate it.

I don’t ex­pect it is worth the time of your av­er­age com­pany to write a spe­cial­ized mod­ule that at­tempts to re­verse-en­g­ineer a user’s pass­word in or­der to de­ter­mine the al­gorithm that was likely used to gen­er­ate it. I ex­pect most com­pa­nies who at­tempt to mea­sure pass­word strength this way do so us­ing 3rd party libraries, not al­gorithms that have been de­vel­oped in-house. The difficulty of do­ing this de­pends on whether there’s a good 3rd party library available for the plat­form the com­pany is us­ing, and how quickly an en­g­ineer can ver­ify that the library isn’t do­ing any­thing sus­pi­cious with the pass­words it an­a­lyzes. This ar­ti­cle has more info about the difficulty of mea­sur­ing pass­word strength—it looks like most 3rd party libraries aren’t very good at it.

But any­way, as I said, it typ­i­cally isn’t ra­tio­nal for soft­ware com­pa­nies to in­vest a lot in soft­ware se­cu­rity. If we are try­ing to ap­prox­i­mate a func­tion that takes a com­pany’s fi­nan­cial in­ter­est in se­cu­rity as an in­put, and out­puts the de­gree to which a com­pany’s sys­tems are se­cure, then the pass­word ex­am­ple gives us a data point where the com­pany’s fi­nan­cial in­ter­est is low and the se­cu­rity of their sys­tem is also low. Co­ral ar­gues (cor­rectly IMO) that Mer­chant Drones Inc. has a strong fi­nan­cial in­cen­tive to pre­vent peo­ple from swindling their drones. Ex­trap­o­lat­ing from the pass­word guess­ing ex­am­ple the way she does makes the as­sump­tion that func­tion map­ping fi­nan­cial in­ter­est to se­cu­rity is a con­stant func­tion. I don’t think that’s a rea­son­able as­sump­tion.

I haven’t seen peo­ple make much progress if they “try to solve the whole prob­lem at once” with a few min­utes/​hours of ex­pe­rience think­ing through the prob­lem rather than a few months/​years; usu­ally that looks less like cor­rigi­bil­ity or ALBA and more like “well we’ll shut it off if it starts do­ing scary stuff” or “well we’ll just raise it like a hu­man child”.

The rea­son I’m com­plain­ing about this is be­cause I some­times try to have con­ver­sa­tions with peo­ple in the com­mu­nity about ideas I have re­lated to AI al­ign­ment, typ­i­cally ideas that I can get across the gist of in <5 min­utes but aren’t nearly as naive as “we’ll shut it off if it starts do­ing scary stuff” or “we’ll raise it like a hu­man child”, but I have a hard time get­ting peo­ple to en­gage se­ri­ously. My di­ag­no­sis is that peo­ple in the com­mu­nity have some kind of learned hel­pless­ness around AI safety, be­liev­ing it to be a big se­ri­ous prob­lem that only big se­ri­ous peo­ple are al­lowed to think about it. Try­ing to make progress on AI al­ign­ment with any idea that can be ex­plained in <5 min­utes marks one as un­cool and naive. Even worse, in some cases I think peo­ple get defen­sive about the idea that it might be pos­si­ble to make progress on AI al­ign­ment in a 10-minute con­ver­sa­tion—the com­mu­nity has a lot in­vested in the idea that AI al­ign­ment is a su­per difficult prob­lem, and we’d all look like fools if it was pos­si­ble to make mean­ingful progress in 10 min­utes. I’m re­minded of this quote from Richard Ham­ming:

...if you do some good work you will find your­self on all kinds of com­mit­tees and un­able to do any more work. You may find your­self as I saw Brat­tain when he got a No­bel Prize. The day the prize was an­nounced we all as­sem­bled in Arnold Au­di­to­rium; all three win­ners got up and made speeches. The third one, Brat­tain, prac­ti­cally with tears in his eyes, said, I know about this No­bel-Prize effect and I am not go­ing to let it af­fect me; I am go­ing to re­main good old Walter Brat­tain.″ Well I said to my­self, That is nice.″ But in a few weeks I saw it was af­fect­ing him. Now he could only work on great prob­lems.
When you are fa­mous it is hard to work on small prob­lems. This is what did Shan­non in. After in­for­ma­tion the­ory, what do you do for an en­core? The great sci­en­tists of­ten make this er­ror. They fail to con­tinue to plant the lit­tle acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn’t the way things go. So that is an­other rea­son why you find that when you get early recog­ni­tion it seems to ster­il­ize you. In fact I will give you my fa­vorite quo­ta­tion of many years. The In­sti­tute for Ad­vanced Study in Prince­ton, in my opinion, has ru­ined more good sci­en­tists than any in­sti­tu­tion has cre­ated, judged by what they did be­fore they came and judged by what they did af­ter. Not that they weren’t good af­ter­wards, but they were su­perb be­fore they got there and were only good af­ter­wards.

See also—if Richard Feyn­man was work­ing on AI al­ign­ment, it sounds to me as though he’d see the naive sug­ges­tions of noobs as a source of oc­ca­sional ideas rather than some­thing to be ridiculed.

It sounds like you’re mak­ing a pre­dic­tion here that Eliezer and oth­ers didn’t put much thought in ad­vance into the po­ten­tial risks or costs of this post. Hav­ing talked with Eliezer and oth­ers about the post be­fore­hand, I can con­firm that this pre­dic­tion is false.

That’s good to hear.

The best way to deal with pos­si­ble coun­ter­ar­gu­ments is to re­think your ar­gu­ments so they’re no longer vuln­er­a­ble to them. (Ex­am­ple: Eliezer could have had Co­ral say some­thing like “Why don’t com­pa­nies just use this free and widely praised pass­word strength mea­sure­ment library on Github?” or “Why is there no good open source library to mea­sure pass­word strength?” in­stead of “Why don’t com­pa­nies just mea­sure en­tropy?” Ran­dom note: In­so­far as the num­bers and sym­bols thing is not just se­cu­rity the­ater, I’d guess it mainly makes it harder for friends/​rel­a­tives to cor­rectly guess that you used your dog’s name as your pass­word, in or­der to de­crease the vol­ume of pass­word-re­lated sup­port re­quests.) I’ll con­fess, when some­one writes an ar­ti­cle that seems to me like it’s writ­ten in an in­suffer­ably smug tone, yet I don’t get the sense that they’ve con­sid­ered coun­ter­ar­gu­ments that seem strong and ob­vi­ous to me, that re­ally rubs me the wrong way.

• (Upvoted but dis­agree with the con­clu­sions and about a quar­ter of the as­sump­tions. I should re­ally get around to de­sign func­tion­al­ity that al­lows users to more eas­ily make it clear that they both want to re­ward some­one for writ­ing some­thing, and that they still dis­agree with it)

• Want to make a bet about whether Eliezer will re­ply to my com­ment? I’m bet­ting he won’t, de­spite writ­ing about the im­por­tance of this, be­cause I think Eliezer still has the is­sue Holden iden­ti­fied years ago of be­ing too se­lec­tive in whose feed­back to take. My guess is that he will read over it, figure out some rea­son why he thinks I’m wrong, and then not en­gage with me be­cause he hasn’t in­ter­nal­ized the WYSIATI phe­nomenon, and he un­der­rates the pos­si­bil­ity that I might have some coun­ter­ar­gu­ment to his coun­ter­ar­gu­ment that he hasn’t thought of.

• My model of Eliezer here (which has a high chance of be­ing in­ac­cu­rate) is that he is bot­tle­necked on men­tal en­ergy, and in gen­eral has a lot of peo­ple who try to en­gage with him on the level of depth that your com­ment goes into, all of whom will prob­a­bly have fur­ther coun­ter­ar­gu­ments and dis­cus­sions of their origi­nal posts. (All sig­nifi­cant com­mu­ni­ca­tion is not sim­ple true/​false ar­gu­ments, but de­tailed model shar­ing.) As such he ex­erts high se­lec­tion pres­sure on where to spend his men­tal en­ergy.

• I’ve down­voted this, be­cause I think it is cre­at­ing a very dan­ger­ous pres­sure on peo­ple stat­ing their opinions openly in this com­mu­nity.

As some­one who has been ac­tive in EA and ra­tio­nal­ity com­mu­nity build­ing for a few years now, I have re­peat­edly ex­pe­rienced the pres­sure of peo­ple de­mand­ing ex­pla­na­tions of me and the peo­ple work­ing with me, so much that at peak times around EA Global re­spond­ing to those de­mands took up more than 70% of my time. In ret­ro­spect, re­spond­ing to each ar­gu­ment and com­ment in­di­vi­d­u­ally was a bad use of my time, and ev­ery­one would have been bet­ter served by me tak­ing a step back, and not re­spond­ing to each com­ment in par­tic­u­lar. And then to in­stead keep a tally of what con­fu­sions or mi­s­un­der­stand­ings peo­ple fre­quently seemed to have, and even­tu­ally write up a more ed­u­ca­tional post that had some real effort put into it, that tried to ex­plain my per­spec­tive on a deeper level.

I see Eliezer mostly do­ing ex­actly that, and want him to con­tinue do­ing that. I don’t think it’s a good use of his time to re­spond to ev­ery com­ment, es­pe­cially if he has good rea­son to ex­pect that it will cost him sig­nifi­cant willpower to do so.

Even more so, peo­ple de­mand­ing ex­pla­na­tions or de­mand­ing en­gage­ment has in my ex­pe­rience re­li­ably lead to ex­haust­ing con­ver­sa­tions full of defen­sive­ness and hedg­ing, and so I ex­pect com­ments like this to sig­nifi­cantly re­duce the prob­a­bil­ity that peo­ple whose time is valuable will en­gage with the com­ments on this site.

• As some­one with time that is rel­a­tively val­ue­less com­pared to Elizer’s and Oliver’s, I’d like to sec­ond this com­ment. As much as I’d love to re­spond to ev­ery per­son who has a crit­i­cism of me, it would take up a lot of men­tal en­ergy that I’d rather use for writ­ing. That doesn’t mean that I don’t read crit­i­cisms and take them to heart.

• That’s fair. I find it stim­u­lat­ing to en­gage with peo­ple, so this isn’t re­ally some­thing I em­pathize with.

• For me, writ­ing for one spe­cific per­son, who is al­most guaran­teed to read what I wrote, is also stim­u­lat­ing. When writ­ing an ar­ti­cle, I of­ten feel like talk­ing to an empty room. As a re­sult, I don’t write many ar­ti­cles.

Still, writ­ing ar­ti­cles would prob­a­bly be a bet­ter use of my time, be­cause I of­ten find my­self re­peat­ing the same things in differ­ent 1:1 in­ter­ac­tions. (Or per­haps to col­lect old com­ments on the same topic, and rewrite them as an ar­ti­cle af­ter­wards.) I just haven’t found a way to al­ign my emo­tions with this.

I guess I wanted to say that “ar­ti­cles > com­ments” re­gard­less of one’s emo­tions (as­sum­ing one can write good ar­ti­cles). Un­less the com­ment can be made short, or re­sponds to some­thing new. But “new” is im­pos­si­ble to judge from out­side; we don’t know what kinds of ques­tions Eliezer gets re­peat­edly e.g. out­side LW.

• I can’t work out where you’re go­ing with the Qubes thing. Ob­vi­ously a se­cure hy­per­vi­sor wouldn’t im­ply a se­cure sys­tem, any more than a se­cure ker­nel im­plies a se­cure sys­tem in a non-hy­per­vi­sor based sys­tem.

More deeply, you seem to im­ply that some­one who has made a se­cu­rity er­ror ob­vi­ously lacks the se­cu­rity mind­set. If only the mind­set pro­tected us from all er­rors; sadly it’s not so. But I’ve of­ten been in the situ­a­tion of try­ing to ex­plain some­thing se­cu­rity-re­lated to a smart per­son, and sens­ing the gap that seemed wider than a mere lack of knowl­edge.

• The point I’m try­ing to make is that this statement

Every­thing ex­posed to an at­tacker, and ev­ery­thing those sub­sys­tems in­ter­act with, and ev­ery­thing those parts in­ter­act with! You have to build all of it ro­bustly!

seems false to me, if you have good iso­la­tion—which is what a pro­ject like Qubes tries to ac­com­plish. Ker­nel vs hy­per­vi­sor is dis­cussed in this blog post. It’s pos­si­ble I’m de­scribing Qubes in­cor­rectly; I’m not a sys­tems ex­pert. But I feel pretty con­fi­dent in the broader point about trusted com­put­ing bases.

More deeply, you seem to im­ply that some­one who has made a se­cu­rity er­ror ob­vi­ously lacks the se­cu­rity mind­set. If only the mind­set pro­tected us from all er­rors; sadly it’s not so. But I’ve of­ten been in the situ­a­tion of try­ing to ex­plain some­thing se­cu­rity-re­lated to a smart per­son, and sens­ing the gap that seemed wider than a mere lack of knowl­edge.

This was the im­pli­ca­tion I was get­ting from Eliezer. I at­tempted a re­duc­tio ad ab­sur­dum.

• Every­thing ex­posed to an at­tacker, and ev­ery­thing those sub­sys­tems in­ter­act with, and ev­ery­thing those parts in­ter­act with! You have to build all of it ro­bustly!

seems false to me, if you have good iso­la­tion—which is what a pro­ject like Qubes tries to ac­com­plish.

I agree with you here that Qubes is cool; but the fact that it is (perfor­mantly) pos­si­ble was not ob­vi­ous be­fore it was cooked up. I cer­tainly failed to come up with the idea of Qubes be­fore hear­ing it (even af­ter bluepill), and I am not ashamed of this: Qubes is brilli­ant (and IOMMU is cheat­ing).

Also, in some sense Qubes is do­ing ex­actly what Carol says. Qubes only has a chance of work­ing be­cause the fun­da­men­tal de­sign for hard­ware-as­sisted se­cu­rity-by-iso­la­tion trumps all other con­sid­er­a­tions in their trade-offs. The UI is fun­da­men­tally con­strained (to pre­vent win­dow-re­dress­ing), as is perfor­mance (3d ac­cel­ler­a­tion) and ease-of-use. All these con­straints were known and doc­u­mented be­fore even a sin­gle line of code was writ­ten (afaik). Qubes can only work be­cause it has se­cu­rity as one of its main goals, and has brilli­ant se­cu­rity peo­ple as pro­ject leads with in­finite in­ter­nal poli­ti­cal cap­i­tal.

That said, go­ing on a tan­gent about qubes:

I re­ally want to see painless live-mi­gra­tion of Qubes (mi­grate an ap­pli­ca­tion be­tween differ­ent hosts, with­out in­terupt­ing—say, from a lousy net­book to a fat work­sta­tion and back), this would be a kil­ler fea­ture for non-se­cu­rity-nerds. Un­for­tu­nately xen can­not do x86 <-> arm (qemu?); live-mi­gra­tion
smart­phone<->work­sta­tion would be awe­some (just bring my smart­phone, plug it in as a boot-drive and con­tinue your work on a fat ma­chine—se­cure as long as there is no hard­ware im­plant).

Re Qubes se­cu­rity: You still have the bad prob­lem of timing-sidechan­nels which cross VM bor­ders; you should view Qubes as an awe­some miti­ga­tion, not a solu­tion (not to speak of the not-so-rare xen out­breaks), and you still need to se­cure your soft­ware. That is, Qubes at­tempts to pre­vent privelege es­ca­la­tion, not code exec; if the vuln­er­a­bil­ity is in the ap­pli­ca­tion which han­dles your valuable data, then Qubes can­not help you.

• Also, in some sense Qubes is do­ing ex­actly what Carol says. Qubes only has a chance of work­ing be­cause the fun­da­men­tal de­sign for hard­ware-as­sisted se­cu­rity-by-iso­la­tion trumps all other con­sid­er­a­tions in their trade-offs. The UI is fun­da­men­tally con­strained (to pre­vent win­dow-re­dress­ing), as is perfor­mance (3d ac­cel­ler­a­tion) and ease-of-use. All these con­straints were known and doc­u­mented be­fore even a sin­gle line of code was writ­ten (afaik). Qubes can only work be­cause it has se­cu­rity as one of its main goals, and has brilli­ant se­cu­rity peo­ple as pro­ject leads with in­finite in­ter­nal poli­ti­cal cap­i­tal.

It sounds like you’re say­ing Qubes is a good illus­tra­tion of Co­ral’s claim that re­ally se­cure soft­ware needs se­cu­rity as a de­sign goal from the be­gin­ning and se­cu­rity DNA in the pro­ject lead­er­ship. I agree with that claim.

• Yep. The counter-ex­am­ple would be Ap­ple iOS.

I never ex­pected it to be­come as se­cure as it did. And Ap­ple se­cu­rity are clowns (in­sti­tu­tion­ally, no offense int­eded for the good peo­ple work­ing there), and UI tends to beat se­cu­rity in trade­offs.

• To trans­late Gra­ham’s state­ment back to the FAI prob­lem: In Eliezer’s al­ign­ment talk, he dis­cusses the value of solv­ing a re­laxed con­straint ver­sion of the FAI prob­lem by grant­ing one­self un­limited com­put­ing power. Well, in the same way, the AGI prob­lem can be seen as a re­laxed con­straint ver­sion of the FAI prob­lem. One could ar­gue that it’s a waste of time to try to make a se­cure ver­sion of AGI Ap­proach X if we don’t even know if it’s pos­si­ble to build an AGI us­ing AGI Ap­proach X. (I don’t agree with this view, but I don’t think it’s en­tirely un­rea­son­able.)

Isn’t the point ex­actly that if you can’t solve the whole prob­lem of (AGI + Align­ment) then it would be bet­ter not even to try solv­ing the re­laxed prob­lem (AGI)?

• Maybe not, if you can keep your solu­tion to AGI se­cret and sup­press it if it turns out that there’s no way to solve the al­ign­ment prob­lem in your frame­work.

• Speak­ing of min­i­miz­ing at­tack sur­face, I note that this page at­tempts to load con­tent from 8 differ­ent third-party do­mains, 2 of which are con­trol­led by Google, a ma­jor known tracker.

• Paul Gra­ham’s De­sign Para­dox is that peo­ple who have good taste in UIs can tell when other peo­ple are de­sign­ing good UIs, but most CEOs of big com­pa­nies lack the good taste to tell who else has good taste. And that’s why big com­pa­nies can’t just hire other peo­ple as tal­ented as Steve Jobs to build nice things for them, even though Steve Jobs cer­tainly wasn’t the best pos­si­ble de­signer on the planet. Ap­ple ex­isted be­cause of a lucky his­tory where Steve Jobs ended up in charge. There’s no way for Sam­sung to hire some­body else with equal tal­ents, be­cause Sam­sung would just end up with some guy in a suit who was good at pre­tend­ing to be Steve Jobs in front of a CEO who couldn’t tell the differ­ence.

I think this idea origi­nated in Gra­ham’s startup mis­takes es­say:

...when I think about what kil­led most of the star­tups in the e-com­merce busi­ness back in the 90s, it was bad pro­gram­mers. A lot of those com­pa­nies were started by busi­ness guys who thought the way star­tups worked was that you had some clever idea and then hired pro­gram­mers to im­ple­ment it. That’s ac­tu­ally much harder than it sounds—al­most im­pos­si­bly hard in fact—be­cause busi­ness guys can’t tell which are the good pro­gram­mers. They don’t even get a shot at the best ones, be­cause no one re­ally good wants a job im­ple­ment­ing the vi­sion of a busi­ness guy.
In prac­tice what hap­pens is that the busi­ness guys choose peo­ple they think are good pro­gram­mers (it says here on his re­sume that he’s a Microsoft Cer­tified Devel­oper) but who aren’t. Then they’re mys­tified to find that their startup lum­bers along like a World War II bomber while their com­peti­tors scream past like jet fighters. This kind of startup is in the same po­si­tion as a big com­pany, but with­out the ad­van­tages.
So how do you pick good pro­gram­mers if you’re not a pro­gram­mer? I don’t think there’s an an­swer. I was about to say you’d have to find a good pro­gram­mer to help you hire peo­ple. But if you can’t rec­og­nize good pro­gram­mers, how would you even do that?

http://​​paulgra­ham.com/​​star­tup­mis­takes.html

If it’s true that this is the bot­tle­neck on friendli­ness, one way to ad­dress this might be to try to make the peo­ple who are ac­tu­ally good at se­cu­rity higher sta­tus—by run­ning se­cu­rity com­pe­ti­tions, for ex­am­ple. (I as­sume the com­pe­ti­tions would have to be or­ga­nized by high sta­tus peo­ple for this to work, and you’d have to iden­tify peo­ple who ac­tu­ally had se­cu­rity mind­set in or­der to de­sign & judge them.)

I sus­pect that differ­ent fields have differ­ent deltas be­tween the peo­ple who look im­pres­sive to an out­sider on pa­per vs the peo­ple who are ac­tu­ally com­pe­tent. Pro­gram­ming might have been one of the fields with the high­est deltas, though I think this delta has been ar­bi­traged away some as the meme that Github pro­files are more im­por­tant than de­grees has spread through the pub­lic con­scious­ness. Nowa­days, I think Triple­byte has ac­cu­mu­lated enough sta­tus that a busi­ness guy has a de­cent shot at cor­rectly choos­ing them to choose their first tech­ni­cal em­ployee.

(Didn’t Eliezer claim in a pre­vi­ous es­say that he has the abil­ity to differ­en­ti­ate ex­perts from non-ex­perts in fields he’s not per­son­ally fa­mil­iar with? I don’t re­mem­ber the de­tails of how he says he does this.)

• Real-world anect­data how one big com­pany (med­i­cal equip­ment) got OK at se­cu­rity:

At some time they de­cided that se­cu­rity was more im­por­tant now. Their in-house guy (dev->dev man­age­ment → “con­grats, you are now our chief se­cu­rity guy”) got to hire more con­sul­tants for their pro­jects, went to train­ings and, cru­cially, went to cons (e.g. def­con). He was a pretty nice guy, and af­ter some years he be­came fluent at hacker-cul­ture. In short, he be­came ca­pa­ble of judg­ing con­sul­tant’s work and hiring real se­cu­rity peo­ple. And he made some friends on the way. I think this is the best path to aquire in­sti­tu­tional knowl­edge: Take a smart per­son loyal to the com­pany, im­merse them into the knowl­edgable sub­cul­ture (spend­ing money on failures on the way), use the aquired knowl­edge to hire real pro­fes­sion­als (re­ally hire, or hire as con­sul­tants for pro­jects, or hire to give train­ings).

Differ­ent big com­pany (not soft­ware re­lated), same thing. After some years their se­cu­rity guys be­came fed up with their lack of in­ter­nal poli­ti­cal cap­i­tal, quit and switched ca­reer to “real” se­cu­rity.

• Why do you think a com­pe­ti­tion would mea­sure any­thing mean­ingful? The best way to hire pro­gram­mers doesn’t seem to be about hiring those peo­ple who do best at pro­gram­ming com­pe­ti­tions.

• The goal is to dis­cover a way to mea­sure se­cu­rity mind­set as ac­cu­rately as pos­si­ble, and then make it high sta­tus to do well ac­cord­ing to the mea­sure­ment. There’s no rea­son why your con­test would need to look like our cur­rent idea of a pro­gram­ming con­test. Soft­ware com­pa­nies already have in­cen­tives to mea­sure pro­gram­ming abil­ity as ac­cu­rately as pos­si­ble—see com­pa­nies like Triple­byte which are at­tempt­ing to do this in a data-driven, sci­en­tifi­cally val­i­dated way. But no one has an in­cen­tive to make your score on Triple­byte’s quiz pub­lic and give you sta­tus based on it.

• Another idea is to push for soft­ware li­a­bil­ities, as Bruce Sch­neier de­scribes in this blog post, in or­der to cre­ate a fi­nan­cial in­cen­tive for more de­vel­op­ers to have se­cu­rity mind­set.

• The big­ger prob­lem here is that as noted in the post, (0) it is always faster to do things in a less se­cure man­ner. If you as­sume (1) mul­ti­ple com­peti­tors try­ing to build AI (and if this is not your as­sump­tion, I would like to hear a ba­sis for it), (2) at least some who be­lieve that the first AI cre­ated will be in a po­si­tion of unas­sailable dom­i­nance (this ap­pears to be the be­lief of at least some and in­clude, but not nec­es­sar­ily be limited to, those who be­lieve in a high like­li­hood of a hard take­off), (3) some over­lap be­tween the groups de­scribed in 1 and 2 (again, if you don’t think this is go­ing to be the case, I would like to hear a ba­sis for it) and (4) vary­ing lev­els of con­ern about the po­ten­tial dam­age caused by an un­friendly AI (even if you be­lieve that as we get closer to de­vel­op­ing AI, the av­er­age and min­i­mum level of con­cern will rise, var­i­ance is likely), the first AI to be pro­duced is likely to be highly in­se­cure (i.e. with non-ro­bust friendli­ness).

• This is ter­rify­ing.

• What is ter­rfify­ing is that even if Mr. Topaz had wanted to be se­cure, the state of the AI safety com­mu­nity , sorry the drone econ­omy safety com­mu­nity, means he will prob­a­bly be pushed into cre­at­ing a demo.

Con­sider Mr Beryl’s plight. All the drone econ­omy safety com­mu­ni­ties are work­ing on one of 2 things.

1) solid fuel rocket drones (they can go very quickly but are not very steer­able), played to­day by Ms Onyx

2) Fric­tion­less massless drone safety re­searchers played to­day by Mr Am­ber.

Mr Beryl has the idea that maybe they could use ro­tors in­stead of solid fuel rock­ets, but ro­tors tend to dis­in­te­grate. He thinks he has a good idea for a ro­tor that might not dis­in­te­grate.

Beryl to Onyx: “Hey any­one work­ing on ro­tor based-drones, I have an idea for ro­tors that might not dis­in­te­grate”?

Onyx: “Solid fuel rock­etry might be able to form an econ­omy.”

Beryl: “Sure, okay. It might. Ro­tors drones have very differ­ent flight char­ac­ter­is­tics com­pared to solid fuel rock­ets, so I’m not sure I can re-use your work. But any­one work­ing on ro­tor drones?”

Onyx: Silence

Beryl to Am­ber: “Maybe it would be worth con­sid­er­ing drones to have mass and fric­tion. It changes things a lot when you do?”

Am­ber: “We haven’t solved the prob­lem of drones go­ing faster than light and form­ing black holes with­out con­sid­er­ing mass and fric­tion. Once we do that we’ll con­sider mass and fric­tion.”

Beryl: “Okay let me know when you do. In the mean­time know any­one work­ing on ro­tor drone safety?”

Am­ber: Silence

The only way for Beryl to get peo­ple to talk about and think about a ro­tor drone econ­omy safety seems to be to cre­ate a demo of a ro­tor not fal­ling apart when at­tached to a mo­tor. He might have ideas about how to make a ro­tor drone and a econ­omy and talk about it, but most of his effort needs to be con­vinc­ing peo­ple that this is a re­al­is­tic op­tion. Which seems like a demo is needed. And Beryl has to fol­low the same path as Topaz.

• Per­haps this post is about use­ful ar­ro­gance? Do not be so ar­ro­gant that you think your field is unique and in­de­pen­dent, and re­quires no im­prove­ments from other fields, but be ar­ro­gant enough to point out there is prob­lems to some­one who thinks he is at the top of his field?

• Th­ese posts fi­nally made me get Some­thing Im­por­tant:

Akra­sia is a se­cu­rity prob­lem, and not just against ex­ter­nal ad­ver­saries like ad­ver­tis­ing.

Is there any­thing good writ­ten yet about solv­ing this do­main of prob­lems from a se­cu­rity mind­set per­spec­tive?

• In ex­am­ples like this drone sce­nario (though ob­vi­ously not AGI) my im­me­di­ate gut re­ac­tion is “make a sand­boxed ver­sion, open it up, and offer mon­e­tary bribes for break­ing it”. It’s not too difficult to sell poli­ti­cally, and you throw an ac­tual op­ti­mizer against the sys­tem.

Since this is of the form “why not just...”, I’m sure there’s some good rea­son why it’s not suffi­cient by it­self. But I don’t know what that rea­son is. Could some­one please en­lighten me?

• Mr. Topaz is right (even if for the wrong rea­sons). If he is op­ti­miz­ing for money, be­ing the first to mar­ket may in­deed get him a higher prob­a­bil­ity of be­com­ing rich. Want­ing to be rich is not wrong. Sure, the ser­vice will be in­se­cure, but if he sells be­fore any sig­nifi­cant is­sues pop up he may sell for billions. “Après moi, le déluge.”

Un­for­tu­nately, the ac­tions of Mr. Topaz are also the very op­ti­miza­tion pro­cess that leads to mis­al­igned AGI, already hap­pen­ing at this very mo­ment. And in­deed, try­ing to fix Mr. Topaz is or­di­nary para­noia. Se­cu­rity mind­set would be closer to get­ting rid of The Mar­ket that so per­versely in­cen­tivized him; I don’t like the idea, but if it’s be­tween that and a pa­per­clip max­i­mizer, I know what I would choose.

• Not sure of the mes­sage of this post re: AI al­ign­ment, but I’m in­ter­ested in see­ing the next in­stal­l­ment, hope it brings ev­ery­thing full cir­cle.

• As far as I know there is not a next in­stal­l­ment planned, though I might be wrong.

I am sur­prised that you feel any un­cer­tainty around the mes­sage of this piece re­gard­ing AI al­ign­ment. It seems clear that ev­ery­thing said is di­rectly ap­pli­ca­ble if you re­place ”Drone Econ­omy Com­pany” with “AGI re­search lab”.

• Brain fart. On sec­ond thought, this is not the first time I’m failing to get EY’s mes­sage, so it could be a lack of cog­ni­tive abil­ity in my part.

• I’m happy to see a demon­stra­tion that Eliezer has a good un­der­stand­ing of the top-level is­sues in­volv­ing com­puter se­cu­rity.

One thing I won­der though, is why mak­ing In­ter­net se­cu­rity bet­ter across the board isn’t a more im­por­tant goal in the ra­tio­nal­ity com­mu­nity? Although very difficult (for rea­sons illus­trated here), it seems im­me­di­ately use­ful and also a good pre­req­ui­site for any sort of AI se­cu­rity. If we can’t se­cure the In­ter­net against na­tion-state level at­tacks, what hope is there against an AI that falls into the wrong hands?

In par­tic­u­lar, build­ing “friendly AI” and as­sum­ing it will re­main friendly seems naive at best, since it will copied and then the friendly part will be mod­ified by hos­tile ac­tors.

It seems like some­one with a se­cu­rity mind­set will want to avoid mak­ing any as­sump­tion of friendli­ness and in­stead work on mak­ing crit­i­cal sys­tems that are sim­ple enough to be math­e­mat­i­cally proven se­cure. I won­der why this quote (from the pre­vi­ous post) isn’t treated as a se­ri­ous plan: “If your sys­tem liter­ally has no mis­be­hav­ior modes at all, it doesn’t mat­ter if you have IQ 140 and the en­emy has IQ 160—it’s not an arm-wrestling con­test.”

We are far from be­ing able to build these sys­tems but it still seems like a more plau­si­ble re­search pro­ject than en­sur­ing that no­body in the world makes un­friendly AI.

• In par­tic­u­lar, build­ing “friendly AI” and as­sum­ing it will re­main friendly seems naive at best, since it will copied and then the friendly part will be mod­ified by hos­tile ac­tors.

This is cor­rect. Any rea­son­able AGI de­vel­op­ment strat­egy must have strong clo­sure and se­cu­rity mea­sures in place to min­i­mize the risk of leaks, and de­ploy­ment has to meet the con­di­tions in Dewey (2014).

It seems like some­one with a se­cu­rity mind­set will want to avoid mak­ing any as­sump­tion of friendli­ness and in­stead work on mak­ing crit­i­cal sys­tems that are sim­ple enough to be math­e­mat­i­cally proven se­cure.

This is also cor­rect, if the idea is to en­sure that de­vel­op­ers un­der­stand their sys­tem (and safety-crit­i­cal sub­sys­tems in par­tic­u­lar) well enough for the end product to be “friendly”/​”al­igned.” If you’re in­stead say­ing that al­ign­ment isn’t a good tar­get to shoot for, then I’m not sure I un­der­stand what you’re say­ing. Why not? How do we achieve good long-term out­comes with­out rout­ing through al­ign­ment?

• I think odds are good that, as­sum­ing gen­eral AI hap­pens at all, some­one will build a hos­tile AI and con­nect it to the In­ter­net. I think a proper un­der­stand­ing the se­cu­rity mind­set is that the as­sump­tion “no­body will con­nect a hos­tile AI to the In­ter­net” is some­thing we should stop rely­ing on. (In par­tic­u­lar, main­tain­ing se­crecy and in­ter­na­tonal co­op­er­a­tion seems un­likely. We shouldn’t as­sume they will work.)

We should be look­ing for defenses that aren’t de­pen­dent of the IQ level of the at­tacker, similar to how math­e­mat­i­cal proofs are in­de­pen­dent of IQ. AI al­ign­ment is an im­por­tant re­search prob­lem, but doesn’t seem di­rectly rele­vant for this.

In par­tic­u­lar, I don’t see why you think “rout­ing through al­ign­ment” is im­por­tant for mak­ing sound math­e­mat­i­cal proofs. Nar­row AI should be suffi­cient for mak­ing ad­vances in math­e­mat­ics.

• I think odds are good that, as­sum­ing gen­eral AI hap­pens at all, some­one will build a hos­tile AI and con­nect it to the In­ter­net. I think a proper un­der­stand­ing the se­cu­rity mind­set is that the as­sump­tion “no­body will con­nect a hos­tile AI to the In­ter­net” is some­thing we should stop rely­ing on. (In par­tic­u­lar, main­tain­ing se­crecy and in­ter­na­tonal co­op­er­a­tion seems un­likely. We shouldn’t as­sume they will work.)

Yup, all of this seems like the stan­dard MIRI/​Eliezer view.

In par­tic­u­lar, I don’t see why you think “rout­ing through al­ign­ment” is im­por­tant for mak­ing sound math­e­mat­i­cal proofs. Nar­row AI should be suffi­cient for mak­ing ad­vances in math­e­mat­ics.

I don’t know what the rele­vance of “math­e­mat­i­cal proofs” is. Are you talk­ing about ap­ply­ing for­mal meth­ods of some kind to the prob­lem of en­sur­ing that AGI tech­nol­ogy doesn’t leak, and say­ing that AGI is un­nec­es­sary for this task? I’m guess­ing that part of the story you’re miss­ing is that pro­lifer­a­tion of AGI tech­nol­ogy is at least as much about in­de­pen­dent dis­cov­ery as it is about leaks, splin­ter­ing, or es­pi­onage. You have to ad­dress those is­sues, but the over­all task of achiev­ing non­pro­lifer­a­tion is much larger than that, and it doesn’t do a lot of good to solve part of the prob­lem with­out solv­ing the whole prob­lem. AGI is po­ten­tially a route to solv­ing the whole prob­lem, not to solv­ing the (rel­a­tively easy, though still very im­por­tant) leaks/​es­pi­onage prob­lem.

• I mean things like us­ing math­e­mat­i­cal proofs to en­sure that In­ter­net-ex­posed ser­vices have no bugs that a hos­tile agent might ex­ploit. We don’t need to be able to build an AI to im­prove defences.

• There’s no “friendly part” in an AGI in the same way there’s no “se­cure part” in an OS. The kind of friendli­ness and se­cu­rity we want is deep in the ar­chi­tec­ture.

Most hos­tile ac­tors also don’t want an AGI that kills them. They might still do nasty things with the AGI by giv­ing it bad goals but that’s not the core what the AGI ar­gu­ment is about.

As far as re­mov­ing mis­be­hav­ior modes, that’s done in se­cu­rity cir­cles. Crit­i­cal com­put­ers get air­gapped to pre­vent them from get­ting hacked.

In the quest of get­ting more se­cure Mozilla build a new pro­gram­ming lan­guage that pre­vents a class of er­rors that C had.

• See also Paul Chris­ti­ano’s “Se­cu­rity and AI Align­ment,” which Eliezer agrees with if I re­call cor­rectly.

• Even if there’s no “friendly part,” it seems un­likely that some­one who learns the ba­sic prin­ci­ples be­hind build­ing a friendly AI will be un­able to build an un­friendly AI by ac­ci­dent. I’m happy that we’re mak­ing progress with safe lan­guages, but there is no prac­ti­cal pro­gram­ming lan­guage in which it’s the least bit difficult to write a bad pro­gram.

It would make more sense to as­sume that at some point, a hos­tile AI will get an In­ter­net con­nec­tion, and figure out what needs to be done about that.