Upper Bounds on Tolerable Risk

[Diego Zamalloa-Chion]

Most fields require very high probabilities of safety, even when the stakes are much lower than the extinction of humanity.^[1] Here are a few examples with risk thresholds expressed in Log Odds (base 10).

Intro to Log Odds

Instead of writing probabilities like 0.9999 which is hard to read, I’ll use log odds.

$\text{LogOdds}\left(p\right)={\log }_{10}\left(\frac{p}{\neg p}\right)={\log }_{10}\left(\frac{p}{1-p}\right)$

So LogOdds$\left(0.9999\right)\approx 4$

Conversion Trick: rounding LogOdds towards zero^[2] roughly corresponds to the number of nines/​zeros after the dot in the probability. Conveniently, the complementary probability in log space is just negative. LogOdds$(1-p)=-$LogOdds(p)

Civil Engineering

United States

General Structural Design (ASCE/​SEI 7-22)^[3]

Target Reliability Table (excluding earthquake, tsunami, and extraordinary events) Annual probability of various types of failure.

$S=$ sudden, $W=$ widespread progression of damage

Types of Failure	Risk Categories
	I[4]	II[5]	III[6]	IV[7]
$\neg S\wedge \neg W$	-3.90	-4.52	-4.90	-5.30
$S\vee W$	-4.52	-5.30	-5.70	-6.15
$S\wedge W$	-5.30	-6.15	-6.60	-7.00

Bridges (AASHTO LRFD)

Lifetime Reliability Target ($\beta =3.5$)

LogOdds: −3.63 lifetime reliability (-5.51 annual, for a typical 75 year service life)

Dams (Bureau of Reclamation Public Protection Guidelines)

Piecewise log-log fatality-probability guidelines. (-1 slope represents constant Annualized Life Loss)

ALARP = As Low As Reasonably Practicable

“ALARP is now considered to apply anywhere on the risk portrayal chart”

Case Study: Citicorp Center^[8]

¹⁄₁₆ annual chance of collapse was unacceptable

LogOdds: −1.18

Europe

Building Minimum Reliability^[9] (EN 1990)

Reference Periods	Reliability Classes
	RC1[10]	RC2[11]	RC3[12]
1 year	-4.87	-5.89	-7.00
50 years	-3.32	-4.14	-5.07

Netherlands—Fundamentals of Flood Protection (2017)

Log-log death-frequency target for 2050

China

Building Annual Reliability Index (GB 50153‑2008)

Type of failure	Safety Class
	Class III[13]	Class II[14]	Class I[15]
Ductile (gradual)	-2.46	-3.16	-3.97
Brittle (sudden)	-3.16	-3.97	-4.87

Highway Bridges, Culverts, and Tunnels (JTG 2120‑2020)

Type of failure	Safety Class
	Class III[16]	Class II[17]	Class I[18]
Ductile (gradual)	-3.97	-4.87	-5.89
Brittle (sudden)	-4.87	-5.89	-7.00

Automotive

International

Random Hardware Failures (ISO 26262)

ASIL	Examples	Target Failure Rate per Hour
B	Headlights, Brake Lights	-7
C	Cruise Control, Rear Wheel Brakes	-7
D	All Wheel Brakes	-8

Aerospace

United States

Airplane Failure Probability-Severity per flight-hour (AC 25.1309-1)

Effects	Severity (LogOdds Range)
	Minor (-3 to −5)	Major (-5 to −7)	Hazardous (-7 to −9)	Catastrophic (less than −9)
Airplane (reduction in functional capabilities or safety margins)	Slight	Significant	Large	Normally hull loss
Crew	Slight increase in workload	A physical discomfort or significant increase in workload or in conditions impairing the efficiency of the flight-crew	Physical distress or excessive workload such that flight-crew cannot be relied upon to perform their tasks accurately or completely	Fatalities or incapacitation
Passengers	Physical discomfort	Physical distress, possibly including injuries	Serious or fatal injury to a small number of persons	Multiple fatalities

Space (14 CFR 450.101)

Risk per launch to	Overall	Most Endangered Individual
Operations Personnel	-3.7	-5
The Public	-4	-6

Risk per launch to:

• Other Aircraft: −6

• Critical Assets (national interests on the ground): −3

• Critical Payload (national interests on the rocket): −4

Nuclear

OECD (NEA MDEP)

Country/​Org	Page	Event Frequency/​Year
		Core Damage	Large Release
INSAG	11	-5	-6
United States	14[19]	-4	-5
Russia	26	-5	-7
France	17	-5	-7
Japan	23	-4	-5
Finland	18	-5	-6.3
Canada	22	-5	-6

United Kingdom

Initiating Fault Frequency		Basic Safety (mSv)
		Limit	Objective
On-site	> −3	20	0.1
	-3 to −4	200	0.1
	< −4	500	0.1
Off-site	> −3	1	0.01
	-3 to −4	10	0.01
	< −4	100	0.01

Carcinogens

International

Threshold of Toxicological Concern (ICH M7 R1)^[20]

Excess lifetime risk at $1.5\frac{\mu g}{\text{day}}$: −5

United States

EPA: Lifetime Risk from Air Pollutants (NESHAP)

Living <50km from plant: −4

General Public: −6

Electronics

International

Safety Integrity Levels (IEC 61508)

SIL	Low Demand Mode[21] Max Probability of Failure on Demand	High Demand or Continuous Mode[22] Max Probability of Dangerous Failure per Hour
1	-1	-5
2	-2	-6
3	-3	-7
4	-4	-8

Physics

Particle Physics & Gravitational Waves

5 sigma statistical significance (usually single-tailed): 6.54

My Ideological Conclusion (optional)

Things I don’t want:

• clever arguments from AI labs for P(doom) $=1\cdot {10}^{-10}$

• regulation asking for fake low probabilities backed up by clever experiments

What I’d like to convey:

• Most certainty/​risk standards have thresholds set such that failures basically never happen, and those that do probably didn’t follow the standard (though causality is hard to establish)

• Abundant caution is more important than perfectly calibrated risk estimates on catastrophe

• Risk should be “as low as reasonably practicable”

• Many people are too comfortable with probabilities of human extinction far in excess of the guidelines regulating individual human lives.

• There is no looming existential threat against which superintelligence is our only hope, so when you’re gambling with everyone’s lives and you don’t get second chances, your risk tolerance should be considerably lower than for eg a skyscraper collapsing.

• While I’m grateful for LLM research assistance for this report, I’d happily trade the time savings for safety. Please face God and walk backwards away from the cliff edge.

If useful to this report, I’m open to adding:

• more fields with explicit probability thresholds

• studies on safety outcomes downstream of the guidelines

1. ^

   Inspired by Rob Miles on Doom Debates

2. ^

   Round positive values down. Round negative values up.trunc() in programming.

3. ^

   Minimum Design Loads and Associated Criteria for Buildings and Other Structures published by the American Society of Civil Engineers (ASCE) and its Structural Engineering Institute (SEI).

4. ^

   Storage Sheds, Temporary Structures, Low Occupancy, Low-Risk Structures

5. ^

   Homes, Businesses, Most Structures, Standard Occupancy Dwellings

6. ^

   Schools, Gathering Places, Public Utilities, High Occupancy Gathering Halls

7. ^

   Hospitals, Emergency Utilities, Sensitive Occupancy, High Risk Structures

8. ^

   Wikipedia, Veritasium

9. ^

   $\text{LogOdds}\left(\beta \right)={\log }_{10}\left(\frac{\Phi (-\beta )}{1-\Phi (-\beta )}\right)$ where $\Phi$ is the CDF of the standard normal distribution

10. ^

    Agricultural buildings where people do not normally enter. (e.g. storage buildings, greenhouses)

11. ^

    Public buildings where consequences of failure are medium. (e.g. residential, office buildings)

12. ^

    Public buildings where consequences of failure are high. (e.g. grandstands, concert halls)

13. ^

    Small or temporary storage buildings

14. ^

    Ordinary residential buildings and office buildings

15. ^

    Large-scale public buildings

16. ^

    Rural tunnels ≤1000 m, construction access tunnels

17. ^

    2-lane urban tunnels, rural tunnels >1000 m

18. ^

    Expressways, multi-arch tunnels, tunnels ≥3 lanes or ≥3000 m

19. ^

    US Nuclear Regulatory Commission (USNRC) Safety Goal Policy established Quantitative Health Objectives (QHOs) that plant-attributable risk of cancer/​early fatalities should be less than 0.1% of respective background risk. OECD/​NEA evaluated baselines in 1980s US to extrapolate absolute risks, which continue to be referenced and haven’t been rebaselined.

20. ^

    International Council for Harmonization (ICH) of Technical Requirements for Pharmaceuticals for Human Use: Assessment and Control of DNA Reactive (Mutagenic) Impurities in Pharmaceuticals To Limit Potential Carcinogenic Risk—M7(R1)

21. ^

    deployed ≤1 time per year

22. ^

    deployed >1 time per year

[Haiku]

Thank you for compiling these risk thresholds! This is going in my list of handy bookmarks.