Se­cu­rity Mindset

TagLast edit: 16 Feb 2022 0:36 UTC by abramdemski

Security Mindset is a predisposition for thinking about the world in a security-oriented way. A large part of this way of thinking involves always being on the lookout for exploits.

Uncle Milton Industries has been selling ant farms to children since 1956. Some years ago, I remember opening one up with a friend. There were no actual ants included in the box. Instead, there was a card that you filled in with your address, and the company would mail you some ants. My friend expressed surprise that you could get ants sent to you in the mail.

I replied: “What’s really interesting is that these people will send a tube of live ants to anyone you tell them to.”

Security requires a particular mindset. Security professionals — at least the good ones — see the world differently. They can’t walk into a store without noticing how they might shoplift. They can’t use a computer without wondering about the security vulnerabilities. They can’t vote without trying to figure out how to vote twice. They just can’t help it.

SmartWater is a liquid with a unique identifier linked to a particular owner. “The idea is for me to paint this stuff on my valuables as proof of ownership,” I wrote when I first learned about the idea. “I think a better idea would be for me to paint it on your valuables, and then call the police.”

Really, we can’t help it.

-- Bruce Schneier, The security Mindset, Schneier on Security

[I’m unsure of the origin of the term, but Schneier is at least an outspoken advocate. --Abram]

In 2017, Eliezer Yudkowsky wrote a pair of posts on the security mindset:

Amongst other things, these posts forwarded the idea that true security mindset is not just the tendency to spot lots and lots of security flaws. Spotting security flaws is not in itself enough to build secure systems, because you could be spotting flaws with your design forever, patching specific weak points, and moving on to find yet more flaws.

Building secure systems requires coming up with strong positive arguments for the security of a system. These positive arguments have several important features:

  1. They have as few assumptions as possible, because each assumption is an additional chance to be wrong.

  2. Each assumption is individually very certain.

  3. The conclusion of the argument is a meaningful security guarantee.

The mindset required to build tight security arguments like this is different from the mindset required to find security holes.

POC || GTFO cul­ture as par­tial an­ti­dote to al­ign­ment wordcelism

lc15 Mar 2023 10:21 UTC
77 points
7 comments7 min readLW link

Do your­self a FAVAR: se­cu­rity mindset

lukehmiles18 Jun 2022 2:08 UTC
19 points
2 comments2 min readLW link

Six Di­men­sions of Oper­a­tional Ad­e­quacy in AGI Projects

Eliezer Yudkowsky30 May 2022 17:00 UTC
278 points
65 comments13 min readLW link

Cir­cum­vent­ing in­ter­pretabil­ity: How to defeat mind-readers

Lee Sharkey14 Jul 2022 16:59 UTC
100 points
8 comments33 min readLW link

Se­cu­rity Mind­set and Or­di­nary Paranoia

Eliezer Yudkowsky25 Nov 2017 17:53 UTC
108 points
25 comments29 min readLW link

Se­cu­rity Mind­set and the Lo­gis­tic Suc­cess Curve

Eliezer Yudkowsky26 Nov 2017 15:58 UTC
83 points
48 comments20 min readLW link

Se­cu­rity Mind­set and Take­off Speeds

DanielFilan27 Oct 2020 3:20 UTC
54 points
23 comments8 min readLW link

Se­cu­rity Mind­set: Les­sons from 20+ years of Soft­ware Se­cu­rity Failures Rele­vant to AGI Alignment

elspood21 Jun 2022 23:55 UTC
332 points
40 comments7 min readLW link

“Just hiring peo­ple” is some­times still ac­tu­ally possible

lc5 Aug 2022 21:44 UTC
38 points
11 comments5 min readLW link

Con­jec­ture: In­ter­nal In­fo­haz­ard Policy

29 Jul 2022 19:07 UTC
123 points
6 comments19 min readLW link

Builder/​Breaker for Deconfusion

abramdemski29 Sep 2022 17:36 UTC
71 points
9 comments9 min readLW link

It’s time to worry about on­line pri­vacy again

Malmesbury25 Dec 2022 21:05 UTC
64 points
23 comments6 min readLW link

On See­ing Through ‘On See­ing Through: A Unified The­ory’: A Unified Theory

gwern15 Jun 2019 18:57 UTC
26 points
0 comments1 min readLW link

LW Meetup @ DEFCON (Las Ve­gas) − 5-7pm Thu. Aug. 11 at Fo­rum Food Court (Cae­sars)

jchan8 Aug 2022 14:57 UTC
6 points
0 comments1 min readLW link

Why do we post our AI safety plans on the In­ter­net?

Peter S. Park3 Nov 2022 16:02 UTC
3 points
4 comments11 min readLW link

AI can ex­ploit safety plans posted on the Internet

Peter S. Park4 Dec 2022 12:17 UTC
−15 points
4 comments1 min readLW link

Se­cu­rity Mind­set—Fire Alarms and Trig­ger Signatures

elspood9 Feb 2023 21:15 UTC
10 points
0 comments4 min readLW link

AGI With In­ter­net Ac­cess: Why we won’t stuff the ge­nie back in its bot­tle.

Max TK18 Mar 2023 3:43 UTC
5 points
9 comments4 min readLW link

Is AI Safety drop­ping the ball on pri­vacy?

Ishan18 Mar 2023 10:56 UTC
21 points
11 comments7 min readLW link
No comments.