I listened to the book This Is How They Tell Me the World Ends by Nicole Perlroth, a book about cybersecurity and the zero-day market. It describes in detail the early days of bug discovery, the social dynamics and moral dilemma of bug hunts.
(It was recommended to me by some EA-adjacent guy very worried about cyber, but the title is mostly bait: the tone of the book is alarmist, but there is very little content about potential catastrophes.)
My main takeaways:
Vulnerabilities used to be dirt-cheap (~$100) but are still relatively cheap (~$1M even for big zero-days);
If you are very good at cyber and extremely smart, you can hide vulnerabilities in 10k-lines programs in a way that less smart specialists will have trouble discovering even after days of examination—code generation/analysis is not really defense favored;
Bug bounties are a relatively recent innovation, and it felt very unnatural to tech giants to reward people trying to break their software;
A big lever companies have on the US government is the threat that overseas competitors will be favored if the US gov meddles too much with their activities;
The main effect of a market being underground is not making transactions harder (people find ways to exchange money for vulnerabilities by building trust), but making it much harder to figure out what the market price is and reducing the effectiveness of the overall market;
Being the target of an autocratic government is an awful experience, and you have to be extremely careful if you put anything they dislike on a computer. And because of the zero-day market, you can’t assume your government will suck at hacking you just because it’s a small country;
It’s not that hard to reduce the exposure of critical infrastructure to cyber-attacks by just making companies air gap their systems more—Japan and Finland have relatively successful programs, and Ukraine is good at defending against that in part because they have been trying hard for a while—but it’s a cost companies and governments are rarely willing to pay in the US;
Electronic voting machines are extremely stupid, and the federal gov can’t dictate how the (red) states should secure their voting equipment;
Hackers want lots of different things—money, fame, working for the good guys, hurting the bad guys, having their effort be acknowledged, spite, … and sometimes look irrational (e.g. they sometimes get frog-boiled).
The US government has a good amount of people who are freaked out about cybersecurity and have good warning shots to support their position. The main difficulty in pushing for more cybersecurity is that voters don’t care about it.
Maybe the takeaway is that it’s hard to build support behind the prevention of risks that 1. are technical/abstract and 2. fall on the private sector and not individuals 3. have a heavy right tail. Given these challenges, organizations that find prevention inconvenient often succeed in lobbying themselves out of costly legislation.
Overall, I don’t recommend this book. It’s very light on details compared to The Hacker and the State despite being longer. It targets an audience which is non-technical and very scope insensitive, is very light on actual numbers, technical details, real-politic considerations, estimates, and forecasts. It is wrapped in an alarmist journalistic tone I really disliked, covers stories that do not matter for the big picture, and is focused on finding who is in the right and who is to blame. I gained almost no evidence either way about how bad it would be if the US and Russia entered a no-holds-barred cyberwar.
If you are very good at cyber and extremely smart, you can hide vulnerabilities in 10k-lines programs in a way that less smart specialists will have trouble discovering even after days of examination—code generation/analysis is not really defense favored;
[...] The NSA invited James Gosler to spend some time at their headquarters in Fort Meade, Maryland in 1987, to teach their analysts [...] about software vulnerabilities. None of the NSA team was able to detect Gosler’s malware, even though it was inserted into an application featuring only 3,000 lines of code. [...]
[Taken from this summary of this passage of the book. The book was light on technical detail, I don’t remember having listened to more details than that.]
I didn’t realize this was so early in the story of the NSA, maybe this anecdote teaches us nothing about the current state of the attack/defense balance.
One example, found by browsing aimlessly through recent high-severity CVE, is CVE-2023-41056. I chose that one by browsing through recent CVEs for one that sounded bad, and was on a project that has a reputation for having clean, well-written, well-tested code, backed by a serious organization. You can see the diff that fixed the CVE here. I don’t think the commit that introduced the vulnerability was intentional… but it totally could have been, and nobody would have caught it despite the Redis project doing pretty much everything right, and there being a ton of eyes on the project.
As a note, CVE stands for “Common Vulnerabilities and Exposures”. The final number in the CVE identifier (i.e.CVE-2023-41056 in this case) is a number that increments sequentially through the year. This should give you some idea of just how frequently vulnerabilities are discovered.
The dirty open secret in the industry is that most vulnerabilities are never discovered, and many of the vulns that are discovered are never publicly disclosed.
Maybe the takeaway is that it’s hard to build support behind the prevention of risks that 1. are technical/abstract and 2. fall on the private sector and not individuals 3. have a heavy right tail. Given these challenges, organizations that find prevention inconvenient often succeed in lobbying themselves out of costly legislation.
Which is also something of a problem for popularising AI alignment. Some aspects of AI (in particular AI art) do have their detractors already, but that won’t necessarily result in policy that helps vs. x-risk.
If you are very good at cyber and extremely smart, you can hide vulnerabilities in 10k-lines programs in a way that less smart specialists will have trouble discovering even after days of examination—code generation/analysis is not really defense favored
I think the first part of the sentence is true, but “not defense favored” isn’t a clear conclusion to me. I think that backdoors work well in closed-source code, but are really hard in open-source widely used code − just look at the amount of effort that went into the recent xz / liblzma backdoor, and the fact that we don’t know of any other backdoor in widely used OSS.
The main effect of a market being underground is not making transactions harder (people find ways to exchange money for vulnerabilities by building trust), but making it much harder to figure out what the market price is and reducing the effectiveness of the overall market
Note this doesn’t apply to all types of underground markets: the ones that regularly get shut down (like darknet drug markets) do have a big issue with trust.
Being the target of an autocratic government is an awful experience, and you have to be extremely careful if you put anything they dislike on a computer. And because of the zero-day market, you can’t assume your government will suck at hacking you just because it’s a small country
This is correct. As a matter of personal policy, I assume that everything I write down somewhere will get leaked at some point (with a few exceptions, like − hopefully − disappearing signal messages).
The reason why xz backdoor was discovered is increased latency, which is textbook side channel. If attacker had more points in security mindset skill tree, it wouldn’t happen.
I just finished listening to The Hacker and the State by Ben Buchanan, a book about cyberattacks, and the surrounding geopolitics. It’s a great book to start learning about the big state-related cyberattacks of the last two decades. Some big attacks /leaks he describes in details:
Wire-tapping/passive listening efforts from the NSA, the “Five Eyes”, and other countries
The multi-layer backdoors the NSA implanted and used to get around encryption, and that other attackers eventually also used (the insecure “secure random number” trick + some stuff on top of that)
The shadow brokers (that’s a *huge* leak that went completely under my radar at the time)
Russia’s attacks on Ukraine’s infrastructure
Attacks on the private sector for political reasons
Stuxnet
The North Korea attack on Sony when they released a documentary criticizing their leader, and misc North Korean cybercrime (e.g. Wannacry, some bank robberies, …)
The leak of Hillary’s emails and Russian interference in US politics
(and more)
Main takeaways (I’m not sure how much I buy these, I just read one book):
Don’t mess with states too much, and don’t think anything is secret—even if you’re the NSA
The US has a “nobody but us” strategy, which states that it’s fine for the US to use vulnerabilities as long as they are the only one powerful enough to find and use them. This looks somewhat nuts and naive in hindsight. There doesn’t seem to be strong incentives to protect the private sector.
There are a ton of different attack vectors and vulnerabilities, more big attacks than I thought, and a lot more is publicly known than I would have expected. The author just goes into great details about ~10 big secret operations, often speaking as if he was an omniscient narrator.
Even the biggest attacks didn’t inflict that much (direct) damage (never >10B in damage?) Unclear if it’s because states are holding back, if it’s because they suck, or if it’s because it’s hard. It seems that even when attacks aim to do what some people fear the most (e.g. attack infrastructure, …) the effect is super underwhelming.
The bottleneck in cyberattacks is remarkably often the will/the execution, much more than actually finding vulnerabilities/entry points to the victim’s network.
The author describes a space where most of the attacks are led by clowns that don’t seem to have clear plans, and he often seems genuinely confused why they didn’t act with more agency to get what they wanted (does not apply to the NSA, but does apply to a bunch of Russia/Iran/Korea-related attacks)
Cyberattacks are not amazing tools to inflict damage or to threaten enemies if you are a state. The damage is limited, and it really sucks that (usually) once you show your capability, it reduces your capability (unlike conventional weapons). And states don’t like to respond to such small threats. The main effect you can have is scaring off private actors from investing in a country / building ties with a country and its companies, and leaking secrets of political importance.
Don’t leak secrets when the US presidential election is happening if they are unrelated to the election, or nobody will care.
(The author seems to be a big skeptic of “big cyberattacks” / cyberwar, and describes cyber as something that always happens in the background and slowly shapes the big decisions. He doesn’t go into the estimated trillion dollar in damages of everyday cybercrime, nor the potential tail risks of cyber.)
I listened to The Failure of Risk Management by Douglas Hubbard, a book that vigorously criticizes qualitative risk management approaches (like the use of risk matrices), and praises a rationalist-friendly quantitative approach. Here are 4 takeaways from that book:
There are very different approaches to risk estimation that are often unaware of each other: you can do risk estimations like an actuary (relying on statistics, reference class arguments, and some causal models), like an engineer (relying mostly on causal models and simulations), like a trader (relying only on statistics, with no causal model), or like a consultant (usually with shitty qualitative approaches).
The state of risk estimation for insurances is actually pretty good: it’s quantitative, and there are strong professional norms around different kinds of malpractice. When actuaries tank a company because they ignored tail outcomes, they are at risk of losing their license.
The state of risk estimation in consulting and management is quite bad: most risk management is done with qualitative methods which have no positive evidence of working better than just relying on intuition alone, and qualitative approaches (like risk matrices) have weird artifacts:
Fuzzy labels (e.g. “likely”, “important”, …) create illusions of clear communication. Just defining the fuzzy categories doesn’t fully alleviate that (when you ask people to say what probabilities each box corresponds to, they often fail to look at the definition of categories).
Inconsistent qualitative methods make cross-team communication much harder.
Coarse categories mean that you introduce weird threshold effects that sometimes encourage ignoring tail effects and make the analysis of past decisions less reliable.
When choosing between categories, people are susceptible to irrelevant alternatives (e.g. if you split the “5/5 importance (loss > $1M)” category into “5/5 ($1-10M), 5⁄6 ($10-100M), 5⁄7 (>$100M)”, people answer a fixed “1/5 (<10k)” category less often).
Following a qualitative method can increase confidence and satisfaction, even in cases where it doesn’t increase accuracy (there is an “analysis placebo effect”).
Qualitative methods don’t prompt their users to either seek empirical evidence to inform their choices.
Qualitative methods don’t prompt their users to measure their risk estimation track record.
Using quantitative risk estimation is tractable and not that weird. There is a decent track record of people trying to estimate very-hard-to-estimate things, and a vocal enough opposition to qualitative methods that they are slowly getting pulled back from risk estimation standards. This makes me much less sympathetic to the absence of quantitative risk estimation at AI labs.
A big part of the book is an introduction to rationalist-type risk estimation (estimating various probabilities and impact, aggregating them with Monte-Carlo, rejecting Knightian uncertainty, doing calibration training and predictions markets, starting from a reference class and updating with Bayes). He also introduces some rationalist ideas in parallel while arguing for his thesis (e.g. isolated demands for rigor). It’s the best legible and “serious” introduction to classic rationalist ideas I know of.
The book also contains advice if you are trying to push for quantitative risk estimates in your team / company, and a very pleasant and accurate dunk on Nassim Taleb (and in particular his claims about models being bad, without a good justification for why reasoning without models is better).
Overall, I think the case against qualitative methods and for quantitative ones is somewhat strong, but it’s far from being a slam dunk because there is no evidence of some methods being worse than others in terms of actual business outputs. The author also fails to acknowledge and provide conclusive evidence against the possibility that people may have good qualitative intuitions about risk even if they fail to translate these intuitions into numbers that make any sense (your intuition sometimes does the right estimation and math even when you suck at doing the estimation and math explicitly).
By Knightian uncertainty, I mean “the lack of any quantifiable knowledge about some possible occurrence” i.e. you can’t put a probability on it (Wikipedia).
The TL;DR is that Knightian uncertainty is not a useful concept to make decisions, while the use subjective probabilities is: if you are calibrated (which you can be trained to become), then you will be better off taking different decisions on p=1% “Knightian uncertain events” and p=10% “Knightian uncertain events”.
For a more in-depth defense of this position in the context of long-term predictions, where it’s harder to know if calibration training obviously works, see the latest scott alexander post.
If you want to get the show-off nerds really on board, then you could make a poast about the expected value of multiplying several distributions (maybe normal distr or pareto distr). Most people get this wrong! I still don’t know how to do it right lol. After I read it I can dunk on my friends and thereby spread the word.
For the product of random variables, there are close form solutions for some common distributions, but I guess Monte-Carlo simulations are all you need in practice (+ with Monte-Carlo can always have the whole distribution, not just the expected value).
Quick convenient monte carlo sim UI seems tractable & neglected & impactful. Like you could reply to a tweet with “hello you are talking about an X=A*B*C thing here. Here’s a histogram of X for your implied distributions of A,B,C” or whatever.
The core idea of the book is that science makes progress by forbidding non-empirical evaluation of hypotheses from publications, focusing on predictions and careful measurements while excluding philosophical interpretations (like Newton’s “I have not as yet been able to deduce from phenomena the reason for these properties of gravity, and I do not feign hypotheses. […] It is enough that gravity really exists and acts according to the laws that we have set forth.”).
The author basically argues that humans are bad at philosophical reasoning and get stuck in endless arguments, and so to make progress you have to ban it (from the main publications) and make it mandatory to make actual measurements (/math) - even when it seems irrational to exclude good (but not empirical) arguments.
It’s weird that the author doesn’t say explicitly “humans are bad at philosophical reasoning” while this feels to me like the essential takeaway.
I’m unsure to what extent this is true, but it’s an interesting claim.
The author doesn’t deny the importance of coming up with good hypotheses, and the role of philosophical reasoning for this part of the process, but he would say that there is clear progress decade by decade only because people did not argue with Einstein by commenting on how crazy the theory was, but instead by they tested the predictions Einstein’s theories made—because that’s the main kind of refutation allowed in scientific venues [Edit: That specific example is wrong and is not in the book, see the comments below.]. Same for evolution, it makes a ton of predictions (though at the time what theory the evidence favored was ambiguous). Before the scientific revolution, lots of people had good ideas, but 1. they had little data to use in their hypotheses’ generation process, and 2. the best ideas had a hard time rising to the top because people argued using arguments instead of collecting data.
(The book also has whole chapters on objectivity, subjectivity, “credibility rankings”, etc. where Bayes and priors aren’t mentioned once. It’s quite sad the extent to which you have to go when you don’t want to scare people with math / when you don’t know math)
Application to AI safety research:
The endless arguments and different schools of thought around the likelihood of scheming and the difficulty of alignment look similar to the historical depictions of people who didn’t know what was going on and should have focused on making experiments.
This makes me more sympathetic to the “just do some experiments” vibe some people, even when it seems like reasoning should be enough if only people understood each other’s arguments.
This makes me more sympathetic towards reviewers/conference organizers rejecting AI safety papers that are mostly about making philosophical points (the rejection may make sense even if the arguments look valid to them).
IIRC Einstein’s theory had a pretty immediate impact on publication on a lot of top physicists even before more empirical evidence came in. Wikipedia on the history of relativity says:
Walter Kaufmann (1905, 1906) was probably the first who referred to Einstein’s work. He compared the theories of Lorentz and Einstein and, although he said Einstein’s method is to be preferred, he argued that both theories are observationally equivalent. Therefore, he spoke of the relativity principle as the “Lorentz–Einsteinian” basic assumption.[76] Shortly afterwards, Max Planck (1906a) was the first who publicly defended the theory and interested his students, Max von Laue and Kurd von Mosengeil, in this formulation. He described Einstein’s theory as a “generalization” of Lorentz’s theory and, to this “Lorentz–Einstein Theory”, he gave the name “relative theory”; while Alfred Bucherer changed Planck’s nomenclature into the now common “theory of relativity” (“Einsteinsche Relativitätstheorie”). On the other hand, Einstein himself and many others continued to refer simply to the new method as the “relativity principle”. And in an important overview article on the relativity principle (1908a), Einstein described SR as a “union of Lorentz’s theory and the relativity principle”, including the fundamental assumption that Lorentz’s local time can be described as real time. (Yet, Poincaré′s contributions were rarely mentioned in the first years after 1905.) All of those expressions, (Lorentz–Einstein theory, relativity principle, relativity theory) were used by different physicists alternately in the next years.[77]
Following Planck, other German physicists quickly became interested in relativity, including Arnold Sommerfeld, Wilhelm Wien, Max Born, Paul Ehrenfest, and Alfred Bucherer.[78] von Laue, who learned about the theory from Planck,[78] published the first definitive monograph on relativity in 1911.[79] By 1911, Sommerfeld altered his plan to speak about relativity at the Solvay Congress because the theory was already considered well established.[78]
Overall I don’t think Einstein’s theories seemed particularly crazy. I think they seemed quite good almost immediately after publication, without the need for additional experiments.
I recently listened to The Righteous Mind. It was surprising to me that many people seem to intrinsically care about many things that look very much like good instrumental norms to me (in particular loyalty, respect for authority, and purity).
The author does not make claims about what the reflective equilibrium will be, nor does he explain how the liberals stopped considering loyalty, respect, and purity as intrinsically good (beyond “some famous thinkers are autistic and didn’t realize the richness of the moral life of other people”), but his work made me doubt that most people will have well-being-focused CEV.
The book was also an interesting jumping point for reflection about group selection. The author doesn’t make the sorts of arguments that would show that group selection happens in practice (and many of his arguments seem to show a lack of understanding of what opponents of group selection think—bees and cells cooperating is not evidence for group selection at all), but after thinking about it more, I now have more sympathy for group-selection having some role in shaping human societies, given that (1) many human groups died, and very few spread (so one lucky or unlucky gene in one member may doom/save the group) (2) some human cultures may have been relatively egalitarian enough when it came to reproductive opportunities that the individual selection pressure was not that big relative to group selection pressure and (3) cultural memes seem like the kind of entity that sometimes survive at the level of the group.
Overall, it was often a frustrating experience reading the author describe a descriptive theory of morality and try to describe what kind of morality makes a society more fit in a tone that often felt close to being normative / fails to understand that many philosophers I respect are not trying to find a descriptive or fitness-maximizing theory of morality (e.g. there is no way that utilitarians think their theory is a good description of the kind of shallow moral intuitions the author studies, since they all know that they are biting bullets most people aren’t biting, such as the bullet of defending homosexuality in the 19th century).
When using length-10 lists (it crushes length-5 no matter the prompt), I get:
32-shot, no fancy prompt: ~25%
0-shot, fancy python prompt: ~60%
0-shot, no fancy prompt: ~60%
So few-shot hurts, but the fancy prompt does not seem to help. Code here.
I’m interested if anyone knows another case where a fancy prompt increases performance more than few-shot prompting, where a fancy prompt is a prompt that does not contain information that a human would use to solve the task. This is because I’m looking for counterexamples to the following conjecture: “fine-tuning on k examples beats fancy prompting, even when fancy prompting beats k-shot prompting” (for a reasonable value of k, e.g. the number of examples it would take a human to understand what is going on).
I listened to the book This Is How They Tell Me the World Ends by Nicole Perlroth, a book about cybersecurity and the zero-day market. It describes in detail the early days of bug discovery, the social dynamics and moral dilemma of bug hunts.
(It was recommended to me by some EA-adjacent guy very worried about cyber, but the title is mostly bait: the tone of the book is alarmist, but there is very little content about potential catastrophes.)
My main takeaways:
Vulnerabilities used to be dirt-cheap (~$100) but are still relatively cheap (~$1M even for big zero-days);
If you are very good at cyber and extremely smart, you can hide vulnerabilities in 10k-lines programs in a way that less smart specialists will have trouble discovering even after days of examination—code generation/analysis is not really defense favored;
Bug bounties are a relatively recent innovation, and it felt very unnatural to tech giants to reward people trying to break their software;
A big lever companies have on the US government is the threat that overseas competitors will be favored if the US gov meddles too much with their activities;
The main effect of a market being underground is not making transactions harder (people find ways to exchange money for vulnerabilities by building trust), but making it much harder to figure out what the market price is and reducing the effectiveness of the overall market;
Being the target of an autocratic government is an awful experience, and you have to be extremely careful if you put anything they dislike on a computer. And because of the zero-day market, you can’t assume your government will suck at hacking you just because it’s a small country;
It’s not that hard to reduce the exposure of critical infrastructure to cyber-attacks by just making companies air gap their systems more—Japan and Finland have relatively successful programs, and Ukraine is good at defending against that in part because they have been trying hard for a while—but it’s a cost companies and governments are rarely willing to pay in the US;
Electronic voting machines are extremely stupid, and the federal gov can’t dictate how the (red) states should secure their voting equipment;
Hackers want lots of different things—money, fame, working for the good guys, hurting the bad guys, having their effort be acknowledged, spite, … and sometimes look irrational (e.g. they sometimes get frog-boiled).
The US government has a good amount of people who are freaked out about cybersecurity and have good warning shots to support their position. The main difficulty in pushing for more cybersecurity is that voters don’t care about it.
Maybe the takeaway is that it’s hard to build support behind the prevention of risks that 1. are technical/abstract and 2. fall on the private sector and not individuals 3. have a heavy right tail. Given these challenges, organizations that find prevention inconvenient often succeed in lobbying themselves out of costly legislation.
Overall, I don’t recommend this book. It’s very light on details compared to The Hacker and the State despite being longer. It targets an audience which is non-technical and very scope insensitive, is very light on actual numbers, technical details, real-politic considerations, estimates, and forecasts. It is wrapped in an alarmist journalistic tone I really disliked, covers stories that do not matter for the big picture, and is focused on finding who is in the right and who is to blame. I gained almost no evidence either way about how bad it would be if the US and Russia entered a no-holds-barred cyberwar.
Do you have concrete examples?
I remembered mostly this story:
[Taken from this summary of this passage of the book. The book was light on technical detail, I don’t remember having listened to more details than that.]
I didn’t realize this was so early in the story of the NSA, maybe this anecdote teaches us nothing about the current state of the attack/defense balance.
The full passage in this tweet thread (search for “3,000”).
One example, found by browsing aimlessly through recent high-severity CVE, is CVE-2023-41056. I chose that one by browsing through recent CVEs for one that sounded bad, and was on a project that has a reputation for having clean, well-written, well-tested code, backed by a serious organization. You can see the diff that fixed the CVE here. I don’t think the commit that introduced the vulnerability was intentional… but it totally could have been, and nobody would have caught it despite the Redis project doing pretty much everything right, and there being a ton of eyes on the project.
As a note, CVE stands for “Common Vulnerabilities and Exposures”. The final number in the CVE identifier (i.e.
CVE-2023-41056in this case) is a number that increments sequentially through the year. This should give you some idea of just how frequently vulnerabilities are discovered.The dirty open secret in the industry is that most vulnerabilities are never discovered, and many of the vulns that are discovered are never publicly disclosed.
Which is also something of a problem for popularising AI alignment. Some aspects of AI (in particular AI art) do have their detractors already, but that won’t necessarily result in policy that helps vs. x-risk.
Same for governments, afaik most still don’t have bug bounty programs for their software66%.Nevermind, a short google shows multiple such programs, although others have been hesitant to adopt them.
I think the first part of the sentence is true, but “not defense favored” isn’t a clear conclusion to me. I think that backdoors work well in closed-source code, but are really hard in open-source widely used code − just look at the amount of effort that went into the recent xz / liblzma backdoor, and the fact that we don’t know of any other backdoor in widely used OSS.
Note this doesn’t apply to all types of underground markets: the ones that regularly get shut down (like darknet drug markets) do have a big issue with trust.
This is correct. As a matter of personal policy, I assume that everything I write down somewhere will get leaked at some point (with a few exceptions, like − hopefully − disappearing signal messages).
The reason why xz backdoor was discovered is increased latency, which is textbook side channel. If attacker had more points in security mindset skill tree, it wouldn’t happen.
I just finished listening to The Hacker and the State by Ben Buchanan, a book about cyberattacks, and the surrounding geopolitics. It’s a great book to start learning about the big state-related cyberattacks of the last two decades. Some big attacks /leaks he describes in details:
Wire-tapping/passive listening efforts from the NSA, the “Five Eyes”, and other countries
The multi-layer backdoors the NSA implanted and used to get around encryption, and that other attackers eventually also used (the insecure “secure random number” trick + some stuff on top of that)
The shadow brokers (that’s a *huge* leak that went completely under my radar at the time)
Russia’s attacks on Ukraine’s infrastructure
Attacks on the private sector for political reasons
Stuxnet
The North Korea attack on Sony when they released a documentary criticizing their leader, and misc North Korean cybercrime (e.g. Wannacry, some bank robberies, …)
The leak of Hillary’s emails and Russian interference in US politics
(and more)
Main takeaways (I’m not sure how much I buy these, I just read one book):
Don’t mess with states too much, and don’t think anything is secret—even if you’re the NSA
The US has a “nobody but us” strategy, which states that it’s fine for the US to use vulnerabilities as long as they are the only one powerful enough to find and use them. This looks somewhat nuts and naive in hindsight. There doesn’t seem to be strong incentives to protect the private sector.
There are a ton of different attack vectors and vulnerabilities, more big attacks than I thought, and a lot more is publicly known than I would have expected. The author just goes into great details about ~10 big secret operations, often speaking as if he was an omniscient narrator.
Even the biggest attacks didn’t inflict that much (direct) damage (never >10B in damage?) Unclear if it’s because states are holding back, if it’s because they suck, or if it’s because it’s hard. It seems that even when attacks aim to do what some people fear the most (e.g. attack infrastructure, …) the effect is super underwhelming.
The bottleneck in cyberattacks is remarkably often the will/the execution, much more than actually finding vulnerabilities/entry points to the victim’s network.
The author describes a space where most of the attacks are led by clowns that don’t seem to have clear plans, and he often seems genuinely confused why they didn’t act with more agency to get what they wanted (does not apply to the NSA, but does apply to a bunch of Russia/Iran/Korea-related attacks)
Cyberattacks are not amazing tools to inflict damage or to threaten enemies if you are a state. The damage is limited, and it really sucks that (usually) once you show your capability, it reduces your capability (unlike conventional weapons). And states don’t like to respond to such small threats. The main effect you can have is scaring off private actors from investing in a country / building ties with a country and its companies, and leaking secrets of political importance.
Don’t leak secrets when the US presidential election is happening if they are unrelated to the election, or nobody will care.
(The author seems to be a big skeptic of “big cyberattacks” / cyberwar, and describes cyber as something that always happens in the background and slowly shapes the big decisions. He doesn’t go into the estimated trillion dollar in damages of everyday cybercrime, nor the potential tail risks of cyber.)
Thanks! I read and enjoyed the book based on this recommendation
I listened to The Failure of Risk Management by Douglas Hubbard, a book that vigorously criticizes qualitative risk management approaches (like the use of risk matrices), and praises a rationalist-friendly quantitative approach. Here are 4 takeaways from that book:
There are very different approaches to risk estimation that are often unaware of each other: you can do risk estimations like an actuary (relying on statistics, reference class arguments, and some causal models), like an engineer (relying mostly on causal models and simulations), like a trader (relying only on statistics, with no causal model), or like a consultant (usually with shitty qualitative approaches).
The state of risk estimation for insurances is actually pretty good: it’s quantitative, and there are strong professional norms around different kinds of malpractice. When actuaries tank a company because they ignored tail outcomes, they are at risk of losing their license.
The state of risk estimation in consulting and management is quite bad: most risk management is done with qualitative methods which have no positive evidence of working better than just relying on intuition alone, and qualitative approaches (like risk matrices) have weird artifacts:
Fuzzy labels (e.g. “likely”, “important”, …) create illusions of clear communication. Just defining the fuzzy categories doesn’t fully alleviate that (when you ask people to say what probabilities each box corresponds to, they often fail to look at the definition of categories).
Inconsistent qualitative methods make cross-team communication much harder.
Coarse categories mean that you introduce weird threshold effects that sometimes encourage ignoring tail effects and make the analysis of past decisions less reliable.
When choosing between categories, people are susceptible to irrelevant alternatives (e.g. if you split the “5/5 importance (loss > $1M)” category into “5/5 ($1-10M), 5⁄6 ($10-100M), 5⁄7 (>$100M)”, people answer a fixed “1/5 (<10k)” category less often).
Following a qualitative method can increase confidence and satisfaction, even in cases where it doesn’t increase accuracy (there is an “analysis placebo effect”).
Qualitative methods don’t prompt their users to either seek empirical evidence to inform their choices.
Qualitative methods don’t prompt their users to measure their risk estimation track record.
Using quantitative risk estimation is tractable and not that weird. There is a decent track record of people trying to estimate very-hard-to-estimate things, and a vocal enough opposition to qualitative methods that they are slowly getting pulled back from risk estimation standards. This makes me much less sympathetic to the absence of quantitative risk estimation at AI labs.
A big part of the book is an introduction to rationalist-type risk estimation (estimating various probabilities and impact, aggregating them with Monte-Carlo, rejecting Knightian uncertainty, doing calibration training and predictions markets, starting from a reference class and updating with Bayes). He also introduces some rationalist ideas in parallel while arguing for his thesis (e.g. isolated demands for rigor). It’s the best legible and “serious” introduction to classic rationalist ideas I know of.
The book also contains advice if you are trying to push for quantitative risk estimates in your team / company, and a very pleasant and accurate dunk on Nassim Taleb (and in particular his claims about models being bad, without a good justification for why reasoning without models is better).
Overall, I think the case against qualitative methods and for quantitative ones is somewhat strong, but it’s far from being a slam dunk because there is no evidence of some methods being worse than others in terms of actual business outputs. The author also fails to acknowledge and provide conclusive evidence against the possibility that people may have good qualitative intuitions about risk even if they fail to translate these intuitions into numbers that make any sense (your intuition sometimes does the right estimation and math even when you suck at doing the estimation and math explicitly).
Is there a short summary on the rejecting Knightian uncertainty bit?
By Knightian uncertainty, I mean “the lack of any quantifiable knowledge about some possible occurrence” i.e. you can’t put a probability on it (Wikipedia).
The TL;DR is that Knightian uncertainty is not a useful concept to make decisions, while the use subjective probabilities is: if you are calibrated (which you can be trained to become), then you will be better off taking different decisions on p=1% “Knightian uncertain events” and p=10% “Knightian uncertain events”.
For a more in-depth defense of this position in the context of long-term predictions, where it’s harder to know if calibration training obviously works, see the latest scott alexander post.
If you want to get the show-off nerds really on board, then you could make a poast about the expected value of multiplying several distributions (maybe normal distr or pareto distr). Most people get this wrong! I still don’t know how to do it right lol. After I read it I can dunk on my friends and thereby spread the word.
For the product of random variables, there are close form solutions for some common distributions, but I guess Monte-Carlo simulations are all you need in practice (+ with Monte-Carlo can always have the whole distribution, not just the expected value).
Quick convenient monte carlo sim UI seems tractable & neglected & impactful. Like you could reply to a tweet with “hello you are talking about an
X=A*B*Cthing here. Here’s a histogram of X for your implied distributions of A,B,C” or whatever.Both causal.app and getguesstimate.com have pretty good monte carlo uis
Oh sweet
Tiny review of The Knowledge Machine (a book I listened to recently)
The core idea of the book is that science makes progress by forbidding non-empirical evaluation of hypotheses from publications, focusing on predictions and careful measurements while excluding philosophical interpretations (like Newton’s “I have not as yet been able to deduce from phenomena the reason for these properties of gravity, and I do not feign hypotheses. […] It is enough that gravity really exists and acts according to the laws that we have set forth.”).
The author basically argues that humans are bad at philosophical reasoning and get stuck in endless arguments, and so to make progress you have to ban it (from the main publications) and make it mandatory to make actual measurements (/math) - even when it seems irrational to exclude good (but not empirical) arguments.
It’s weird that the author doesn’t say explicitly “humans are bad at philosophical reasoning” while this feels to me like the essential takeaway.
I’m unsure to what extent this is true, but it’s an interesting claim.
The author doesn’t deny the importance of coming up with good hypotheses, and the role of philosophical reasoning for this part of the process, but he would say that there is clear progress decade by decade only because people did not argue with Einstein by commenting on how crazy the theory was, but instead by they tested the predictions Einstein’s theories made—because that’s the main kind of refutation allowed in scientific venues [Edit: That specific example is wrong and is not in the book, see the comments below.]. Same for evolution, it makes a ton of predictions (though at the time what theory the evidence favored was ambiguous). Before the scientific revolution, lots of people had good ideas, but 1. they had little data to use in their hypotheses’ generation process, and 2. the best ideas had a hard time rising to the top because people argued using arguments instead of collecting data.
(The book also has whole chapters on objectivity, subjectivity, “credibility rankings”, etc. where Bayes and priors aren’t mentioned once. It’s quite sad the extent to which you have to go when you don’t want to scare people with math / when you don’t know math)
Application to AI safety research:
The endless arguments and different schools of thought around the likelihood of scheming and the difficulty of alignment look similar to the historical depictions of people who didn’t know what was going on and should have focused on making experiments.
This makes me more sympathetic to the “just do some experiments” vibe some people, even when it seems like reasoning should be enough if only people understood each other’s arguments.
This makes me more sympathetic towards reviewers/conference organizers rejecting AI safety papers that are mostly about making philosophical points (the rejection may make sense even if the arguments look valid to them).
Did Einstein’s theory seem crazy to people at the time?
IIRC Einstein’s theory had a pretty immediate impact on publication on a lot of top physicists even before more empirical evidence came in. Wikipedia on the history of relativity says:
Overall I don’t think Einstein’s theories seemed particularly crazy. I think they seemed quite good almost immediately after publication, without the need for additional experiments.
Thanks for the fact check. I was trying to convey the vibe the book gave me, but I think this specific example was not in the book, my bad!
Thanks! And makes sense, you did convey the vibe. And good to know it isn’t in the book.
I recently listened to The Righteous Mind. It was surprising to me that many people seem to intrinsically care about many things that look very much like good instrumental norms to me (in particular loyalty, respect for authority, and purity).
The author does not make claims about what the reflective equilibrium will be, nor does he explain how the liberals stopped considering loyalty, respect, and purity as intrinsically good (beyond “some famous thinkers are autistic and didn’t realize the richness of the moral life of other people”), but his work made me doubt that most people will have well-being-focused CEV.
The book was also an interesting jumping point for reflection about group selection. The author doesn’t make the sorts of arguments that would show that group selection happens in practice (and many of his arguments seem to show a lack of understanding of what opponents of group selection think—bees and cells cooperating is not evidence for group selection at all), but after thinking about it more, I now have more sympathy for group-selection having some role in shaping human societies, given that (1) many human groups died, and very few spread (so one lucky or unlucky gene in one member may doom/save the group) (2) some human cultures may have been relatively egalitarian enough when it came to reproductive opportunities that the individual selection pressure was not that big relative to group selection pressure and (3) cultural memes seem like the kind of entity that sometimes survive at the level of the group.
Overall, it was often a frustrating experience reading the author describe a descriptive theory of morality and try to describe what kind of morality makes a society more fit in a tone that often felt close to being normative / fails to understand that many philosophers I respect are not trying to find a descriptive or fitness-maximizing theory of morality (e.g. there is no way that utilitarians think their theory is a good description of the kind of shallow moral intuitions the author studies, since they all know that they are biting bullets most people aren’t biting, such as the bullet of defending homosexuality in the 19th century).
List sorting does not play well with few-shot mostly doesn’t replicate with davinci-002.
When using length-10 lists (it crushes length-5 no matter the prompt), I get:
32-shot, no fancy prompt: ~25%
0-shot, fancy python prompt: ~60%
0-shot, no fancy prompt: ~60%
So few-shot hurts, but the fancy prompt does not seem to help. Code here.
I’m interested if anyone knows another case where a fancy prompt increases performance more than few-shot prompting, where a fancy prompt is a prompt that does not contain information that a human would use to solve the task. This is because I’m looking for counterexamples to the following conjecture: “fine-tuning on k examples beats fancy prompting, even when fancy prompting beats k-shot prompting” (for a reasonable value of k, e.g. the number of examples it would take a human to understand what is going on).