Se­cu­rity Mindset

TagLast edit: 16 Feb 2022 0:36 UTC by abramdemski

Security Mindset is a predisposition for thinking about the world in a security-oriented way. A large part of this way of thinking involves always being on the lookout for exploits.

Uncle Milton Industries has been selling ant farms to children since 1956. Some years ago, I remember opening one up with a friend. There were no actual ants included in the box. Instead, there was a card that you filled in with your address, and the company would mail you some ants. My friend expressed surprise that you could get ants sent to you in the mail.

I replied: “What’s really interesting is that these people will send a tube of live ants to anyone you tell them to.”

Security requires a particular mindset. Security professionals — at least the good ones — see the world differently. They can’t walk into a store without noticing how they might shoplift. They can’t use a computer without wondering about the security vulnerabilities. They can’t vote without trying to figure out how to vote twice. They just can’t help it.

SmartWater is a liquid with a unique identifier linked to a particular owner. “The idea is for me to paint this stuff on my valuables as proof of ownership,” I wrote when I first learned about the idea. “I think a better idea would be for me to paint it on your valuables, and then call the police.”

Really, we can’t help it.

-- Bruce Schneier, The security Mindset, Schneier on Security

[I’m unsure of the origin of the term, but Schneier is at least an outspoken advocate. --Abram]

In 2017, Eliezer Yudkowsky wrote a pair of posts on the security mindset:

Amongst other things, these posts forwarded the idea that true security mindset is not just the tendency to spot lots and lots of security flaws. Spotting security flaws is not in itself enough to build secure systems, because you could be spotting flaws with your design forever, patching specific weak points, and moving on to find yet more flaws.

Building secure systems requires coming up with strong positive arguments for the security of a system. These positive arguments have several important features:

  1. They have as few assumptions as possible, because each assumption is an additional chance to be wrong.

  2. Each assumption is individually very certain.

  3. The conclusion of the argument is a meaningful security guarantee.

The mindset required to build tight security arguments like this is different from the mindset required to find security holes.

Do your­self a FAVAR: se­cu­rity mindset

lcmgcd18 Jun 2022 2:08 UTC
16 points
2 comments2 min readLW link

Six Di­men­sions of Oper­a­tional Ad­e­quacy in AGI Projects

Eliezer Yudkowsky30 May 2022 17:00 UTC
255 points
65 comments13 min readLW link

Se­cu­rity Mind­set and Or­di­nary Paranoia

Eliezer Yudkowsky25 Nov 2017 17:53 UTC
88 points
23 comments29 min readLW link

Se­cu­rity Mind­set and the Lo­gis­tic Suc­cess Curve

Eliezer Yudkowsky26 Nov 2017 15:58 UTC
76 points
48 comments20 min readLW link

Se­cu­rity Mind­set and Take­off Speeds

DanielFilan27 Oct 2020 3:20 UTC
54 points
23 comments8 min readLW link

Se­cu­rity Mind­set: Les­sons from 20+ years of Soft­ware Se­cu­rity Failures Rele­vant to AGI Alignment

elspood21 Jun 2022 23:55 UTC
294 points
34 comments7 min readLW link

On See­ing Through ‘On See­ing Through: A Unified The­ory’: A Unified Theory

gwern15 Jun 2019 18:57 UTC
26 points
0 comments1 min readLW link
No comments.