There’s been a lot of previous interest in indoor CO2 in the rationality community, including an (unsuccessful) CO2 stripper project, some researchsummaries and self experiments. The results are confusing, I suspect some of the older research might be fake. But I noticed something that has greatly changed how I think about CO2 in relation to cognition.
Exhaled air is about 50kPPM CO2. Outdoor air is about 400ppm; indoor air ranges from 500 to 1500ppm depending on ventilation. Since exhaled air has CO2 about two orders of magnitude larger than the variance in room CO2, if even a small percentage of inhaled air is reinhalation of exhaled air, this will have a significantly larger effect than changes in ventilation. I’m having trouble finding a straight answer about what percentage of inhaled air is rebreathed (other than in the context of mask-wearing), but given the diffusivity of CO2, I would be surprised if it wasn’t at least 1%.
This predicts that a slight breeze, which replaces their in front of your face and prevents reinhalation, would have a considerably larger effect than ventilating an indoor space where the air is mostly still. This matches my subjective experience of indoor vs outdoor spaces, which, while extremely confounded, feels like an air-quality difference larger than CO2 sensors would predict.
This also predicts that a small fan, positioned so it replaces the air in front of my face, would have a large effect on the same axis as improved ventilation would. I just set one up. I don’t know whether it’s making a difference but I plan to leave it there for at least a few days.
(Note: CO2 is sometimes used as a proxy for ventilation in contexts where the thing you actually care about is respiratory aerosol, because it affects transmissibility of respiratory diseases like COVID and influenza. This doesn’t help with that at all and if anything would make it worse.)
This indicates that how we breathe plays a big role in CO2 uptake. Like, shallow or full, small or large volumes, or the speed of exhaling. Breathing technique is a key skill of divers and can be learned. I just started reading the book Breath, which seems to have a lot on it.
Huh, I’ve also noticed a larger effect from indoors/outdoors than seems reflected by CO2 monitors, and that I seem smarter when it’s windy, but I never thought of this hypothesis; it’s interesting, thanks.
I assume the 44k PPM CO2 exhaled air is the product of respiration (I.e. the lungs have processed it), whereas the air used in mouth-to-mouth is quickly inhaled and exhaled.
As the respirator still has to breathe regularly, there will be still a significantly higher CO2 in the air for respiration. I’d guess maybe half − 20k PPM. Interesting to see somebody measure that.
I had previously guessed air movement made me feel better because my body expected air movement (i.e. some kind of biophilic effect). But this explanation seems more likely in retrospect! I’m not quite sure how to run the calculation using the diffusivity coefficient to spot check this, though.
I am now reasonably convinced (p>0.8) that SARS-CoV-2 originated in an accidental laboratory escape from the Wuhan Institute of Virology.
1. If SARS-CoV-2 originated in a non-laboratory zoonotic transmission, then the geographic location of the initial outbreak would be drawn from a distribution which is approximately uniformly distributed over China (population-weighted); whereas if it originated in a laboratory, the geographic location is drawn from the commuting region of a lab studying that class of viruses, of which there is currently only one. Wuhan has <1% of the population of China, so this is (order of magnitude) a 100:1 update.
2. No factor other than the presence of the Wuhan Institute of Virology and related biotech organizations distinguishes Wuhan or Hubei from the rest of China. It is not the location of the bat-caves that SARS was found in; those are in Yunnan. It is not the location of any previous outbreaks. It does not have documented higher consumption of bats than the rest of China.
3. There have been publicly reported laboratory escapes of SARS twice before in Beijing, so we know this class of virus is difficult to contain in a laboratory setting.
4. We know that the Wuhan Institute of Virology was studying SARS-like bat coronaviruses. As reported in the Washington Post today, US diplomats had expressed serious concerns about the lab’s safety.
5. China has adopted a policy of suppressing research into the origins of SARS-CoV-2, which they would not have done if they expected that research to clear them of scandal. Some Chinese officials are in a position to know.
To be clear, I don’t think this was an intentional release. I don’t think it was intended for use as a bioweapon. I don’t think it underwent genetic engineering or gain-of-function research, although nothing about it conclusively rules this out. I think the researchers had good intentions, and screwed up.
This Feb. 20th Twitter thread from Trevor Bedford argues against the lab-escape scenario. Do read the whole thing, but I’d say that the key points not addressed in parent comment are:
Data point #1 (virus group): #SARSCoV2 is an outgrowth of circulating diversity of SARS-like viruses in bats. A zoonosis is expected to be a random draw from this diversity. A lab escape is highly likely to be a common lab strain, either exactly 2002 SARS or WIV1.
But, briefly, #SARSCoV2 has 6 mutations to its receptor binding domain that make it good at binding to ACE2 receptors from humans, non-human primates, ferrets, pigs, cats, pangolins (and others), but poor at binding to bat ACE2 receptors.
This pattern of mutation is most consistent with evolution in an animal intermediate, rather than lab escape. Additionally, the presence of these same 6 mutations in the pangolin virus argues strongly for an animal origin: https://biorxiv.org/content/10.1101/2020.02.13.945485v1…
…
Data point #3 (market cases): Many early infections in Wuhan were associated with the Huanan Seafood Market. A zoonosis fits with the presence of early cases in a large animal market selling diverse mammals. A lab escape is difficult to square with early market cases.
...
Data point #4 (environmental samples): 33 out of 585 environmental samples taken from the Huanan seafood market showed as #SARSCoV2 positive. 31 of these were collected from the western zone of the market, where wildlife booths are concentrated. 15⁄21http://xinhuanet.com/english/2020-01/27/c_138735677.htm…
Environmental samples could in general derive from human infections, but I don’t see how you’d get this clustering within the market if these were human derived.
One scenario I recall seeing somewhere that would reconcile lab-escape with data points 3 & 4 above is that some low-level WIV employee or contractor might have sold some purloined lab animals to the wet market. No idea how plausible that is.
Data point #3 (market cases): Many early infections in Wuhan were associated with the Huanan Seafood Market. A zoonosis fits with the presence of early cases in a large animal market selling diverse mammals. A lab escape is difficult to square with early market cases.
Given that there’s the claim from Botao Xiao’s The possible origins of 2019-nCoV coronavirus, that this seafood market was located 300m from a lab (which might or might not be true), this market doesn’t seem like it reduces chances.
We need to update down on any complex, technical datapoint that we don’t fully understand, as China has surely paid researchers to manufacture hard-to-evaluate evidence for its own benefit (regardless of the truth of the accusation). This is a classic technique that I have seen a lot in propaganda against laypeople, and there is every reason it should have been employed against the “smart” people in the current coronavirus situation.
The most recent episode of the 80k podcast had Andy Weber on it. He was the US Assistant Secretary of Defense, “responsible for biological and other weapons of mass destruction”.
Well, over time, evidence for natural spread hasn’t been produced, we haven’t found the intermediate species, you know, the pangolin that was talked about last year. I actually think that the odds that this was a laboratory-acquired infection that spread perhaps unwittingly into the community in Wuhan is about a 50% possibility… And we know that the Wuhan Institute of Virology was doing exactly this type of research [gain of function research]. Some of it — which was funded by the NIH for the United States — on bat Coronaviruses. So it is possible that in doing this research, one of the workers at that laboratory got sick and went home. And now that we know about asymptomatic spread, perhaps they didn’t even have symptoms and spread it to a neighbor or a storekeeper. So while it seemed an unlikely hypothesis a year ago, over time, more and more evidence leaning in that direction has come out. And it’s wrong to dismiss that as kind of a baseless conspiracy theory. I mean, very, very serious scientists like David Relman from Stanford think we need to take the possibility of a laboratory accident seriously.
The included link is to a statement from the US Embassy in Georgia, which to me seems surprisingly blunt, calling out the CCP for obfuscation, and documenting events at the WIV, going so far as to speculate that they were doing bio-weapons research there.
What about allegations that a pangolin was involved? Would they have had pangolins in the lab as well or is the evidence about pangolin involvement dubious in the first place?
Edit: Wasn’t meant as a joke. My point is why did initial analyses conclude that the SARS-Cov-2 virus is adapted to receptors of animals other than bats, suggesting that it had an intermediary host, quite likely a pangolin. This contradicts the story of “bat researchers kept bat-only virus in a lab and accidentally released it.”
I think it’s probably a virus that was merely identified in pangolins, but whose primary host is probably not pangolins.
The pangolins they sequenced weren’t asymptomatic carriers at all; they were sad smuggled specimens that were dying of many different diseases simultaneously.
I looked into this semi-recently, and wrote up something here.
The pangolins were apprehended in Guangxi, which shares some of its border with Yunnan. Neither of these provinces are directly contiguous with Hubei (Wuhan’s province), fwiw. (map)
1. If SARS-CoV-2 originated in a non-laboratory zoonotic transmission, then the geographic location of the initial outbreak would be drawn from a distribution which is approximately uniformly distributed over China (population-weighted); whereas if it originated in a laboratory, the geographic location is drawn from the commuting region of a lab studying that class of viruses, of which there is currently only one. Wuhan has <1% of the population of China, so this is (order of magnitude) a 100:1 update.
This is an assumption.
While it might be comparatively correct, I’m not sure about the magnitude. Under the circumstances, perhaps we should consider the possibility that there is something we don’t know about Wuhan that makes it more likely.
3. There have been publicly reported laboratory escapes of SARS twice before in Beijing, so we know this class of virus is difficult to contain in a laboratory setting.
Maybe they don’t know whether it escaped or not. Maybe they just think there is a chance that the evidence will implicate them and they figure it’s not worth the risk as there’ll only be consequences if there is definitely proof that it escaped from one of their labs and not mere speculation.
Or maybe they want to argue that it didn’t come from China? I think they’ve already been pushing this angle.
First, a clarification: whether SARS-CoV-2 was laboratory-constructed or manipulated is a separate question from whether it escaped from a lab. The main reason a lab would be working with SARS-like coronavirus is to test drugs against it in preparation for a possible future outbreak from a zoonotic source; those experiments would involve culturing it, but not manipulating it.
But also: If it had been the subject of gain-of-function research, this probably wouldn’t be detectable. The example I’m most familiar with, the controversial 2012 US A/H5N1 gain of function study, used a method which would not have left any genetic evidence of manipulation.
Our analyses clearly show that SARS-CoV-2 is not a laboratory construct or a purposefully manipulated virus
and
It is so effective at attaching to human cells that the researchers said the spike proteins were the result of natural selection and not genetic engineering.
I think the article just says that the virus did not undergo genetic engineering or gain-of-function research, which is also what Jim says above.
Chinese virology researcher released something claiming that SARS-2 might even be genetically-manipulated after all? After assessing, I’m not really convinced of the GMO claims, but the RaTG13 story definitely seems to have something weird going on.
Claims that the RaTG13 genome release was a cover-up (it does look like something’s fishy with RaTG13, although it might be different than Yan thinks). Claims ZC45 and/or ZXC21 was the actual backbone (I’m feeling super-skeptical of this bit, but it has been hard for me to confirm either way).
Looks like something fishy happened with RaTG13, although I’m not convinced that genetic modification was involved. This is an argument built on pre-prints, but they appear to offer several different lines of evidence that something weird happened here.
Simplest story (via R&B): It looks like people first sequenced this virus in 2016, under the name “BtCOV/4991”, using mine samples from 2013. And for some reason, WIV re-released the sequence as “RaTG13″ at a later date?
(edit: I may have just had a misunderstanding. Maybe BtCOV/4991 is the name of the virus as sequenced from miner-lungs, RaTG13 is the name of the virus as sequenced from floor droppings? But in that case, why is the “fecal” sample reading so weirdly low-bacteria? And they probably are embarrassed that it took them that long to sequence the fecal samples, and should be.)
A paper by by Indian researchers Rahalkar and Bahulikar ( https://doi.org/10.20944/preprints202005.0322.v1 ) notes that BtCoV/4991 sequenced in 2016 by the same Wuhan Virology Institute researchers (and taken from 2013 samples of a mineshaft that gave miners deadly pneumonia) was very similar, and likely the same, as RaTG13.
A preprint by Rahalkar and Bahulikar (R&B) ( doi: 10.20944/preprints202008.0205.v1 ) notes that the fraction of bacterial genomes in in the RaTG13 “fecal” sample was ABSURDLY low (“only 0.7% in contrast to 70-90% abundance in other fecal swabs from bats”). Something’s weird there.
A more recent weird datapoint: A pre-print Yan referenced ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7337384/ ), whose finding (in graphs; it was left unclear in their wording) was indeed that a RaTG13 protein didn’t competently bind their Bat ACE2 samples, but rather their Rat, Mouse, Human, and Pig ACE2. It’s supposedly a horseshoe bat virus (sequenced by the Wuhan lab), so this seems hecka fishy to me.
(Sure, their bat samples weren’t precisely the same species, but they tried 2 species from the same genus. SARS-2 DID bind for their R. macrotis bat sample, so it seems extra-fishy to me that RaTG13 didn’t.).
((...oh. According to the R&B paper about the mineshaft, it was FILTY with rats, bats, poop, and fungus. And the CoV genome showed up in only one of ~280 samples taken. If it’s like that, who the hell knew if it came from a rat or bat?))
At this point, RaTG13 is genuinely looking pretty fishy to me. It might actually take evidence of a conspiracy theory in the other direction for me to go back to neutral on that.
E-Protein Similarity? Meh.
I’m not finding the Protein-E sequence similarity super-convincing in itself, because while the logic is fine, it’s very multiple-hypothesis-testing flavored.
I’m still looking into the ZC45 / ZXC21 claim, which I’m currently feeling skeptical of. Here’s the paper that characterized those: doi: 10.1038/s41426-018-0155-5 . It’s true that it was by people working at “Research Institute for Medicine of Nanjing Command.” However, someone on twitter used BLAST on the E-protein sequence, and found a giant pile of different highly-related SARS-like coronaviruses. I’m trying to replicate that analysis using BLAST myself, and at a skim the 100% results are all more SARS-CoV-2, and the close (95%) results are damned diverse. …I don’t see ZC in them, it looks like it wasn’t uploaded. Ugh. (The E-protein is only 75 amino acids long anyway. https://www.ncbi.nlm.nih.gov/protein/QIH45055.1 )
A different paper mentions extreme S2-protein similarity of early COVID-19 to ZC45 , but that protein is highly-conserved. That makes this a less surprising or meaningful result. (E was claimed to be fast-evolving, so its identicality would have been more surprising, but I couldn’t confirm it.) https://doi.org/10.1080/22221751.2020.1719902
Other
I think Yan offers a reasonable argument that a method could have been used that avoids obvious genetic-modification “stitches,” instead using methods that are hard to distinguish from natural recombination events (ex: recombination in yeast). Sounds totally possible to me.
The fact that the early SARS-CoV-2 samples were already quite adapted to human ACE2 and didn’t have the rapid-evolution you’d expect from a fresh zoonotic infection is something a friend of mine had previously noted, probably after reading the following paper (recommended): https://www.biorxiv.org/content/10.1101/2020.05.01.073262v1 (Zhan, Deverman, Chan). This fact does seem fishy, and had already pushed me a bit towards the “Wuhan lab adaptation & escape” theory.
Wuhan has <1% of the population of China, so this is (order of magnitude) a 100:1 update.
I agree that this is technically correct, but the prior for “escaped specifically from a lab in Wuhan” is also probably ~100 times lower than the prior for “escaped from any biolab in China”, which makes this sentence feel odd to me. I feel like I have reasonable priors for “direct human-to-human transmission” vs. “accidentally released from a lab”, but don’t have good priors for “escaped specifically from a lab in Wuhan”.
I agree that this is technically correct, but the prior for “escaped specifically from a lab in Wuhan” is also probably ~100 times lower than the prior for “escaped from any biolab in China”
I don’t think this is true. The Wuhan Institute of Virology is the only biolab in China with a BSL-4 certification, and therefore is probably the only biolab in China which could legally have been studying this class of virus. While the BSL-3 Chinese Institute of Virology in Beijing studied SARS in the past and had laboratory escapes, I expect all of that research to have been shut down or moved, given the history, and I expect a review of Chinese publications will not find any studies involving live virus testing outside of WIV. While the existence of one or two more labs in China studying SARS would not be super surprising, the existence of 100 would be extremely surprising, and would be a major scandal in itself.
Woah. That’s an important piece of info. The lab in Wuhan is the only lab in China allowed to deal with this class of virus. That’s very suggestive info indeed.
That’s overstating it. They’re the only BSL-4 lab. Whether BSL-3 labs were allowed to deal with this class of virus, is something that someone should research.
My understanding is that SARS-CoV-1 is generally treated as a BSL-3 pathogen or a BSL-2 pathogen (for routine diagnostics and other relatively safe work) and not BSL-4. At the time of the outbreak, SARS-CoV-2 would have been a random animal coronavirus that hadn’t yet infected humans, so I’d be surprised if it had more stringent requirements.
Your OP currently states: “a lab studying that class of viruses, of which there is currently only one.” If I’m right that you’re not currently confident this is the case, it might be worth adding some kind of caveat or epistemic status flag or something.
---
Some evidence:
A 2017 news article in Nature about the Wuhan Institute of Virology suggests China doesn’t require a BSL-4 for SARS-CoV-1. “Future plans include studying the pathogen that causes SARS, which also doesn’t require a BSL-4 lab.”
Non-propagative diagnostic laboratory work including, sequencing, nucleic acid amplification test (NAAT) on clinical specimens from patients who are suspected or confirmed to be infected with nCoV, should be conducted adopting practices …. … in the interim, Biosafety Level 2 (BSL-2) in the WHO Laboratory Biosafety Manual, 3rd edition remains appropriate until the 4th edition replaces it.
Handling of material with high concentrations of live virus (such as when performing virus propagation, virus isolation or neutralization assays) or large volumes of infectious materials should be performed only by properly trained and competent personnel in laboratories capable of meeting additional essential containment requirements and practices, i.e. BSL-3.
It would be important information if it was true. But is it true?
(SARSr-CoV) makes the BSL-4 list on Wikipedia but coronaviruses are widespread in a lot of species and I can’t find any evidence that they are restricted to BSL-4 labs.
Do we have any good sense of the extent to which researchers from the Wuhan Institute of Virology are flying out across China to investigate novel pathogens or sites where novel pathogens might emerge?
OpenBSD treats every crash as a security problem, because the system is not supposed to crash and therefore any crash proves that our beliefs about the system are false and therefore our beliefs about its security may also be false because its behavior is not known
And my reply to this grew into something that I think is important enough to make as a top-level shortform post.
It’s worth noticing that this is not a universal property of high-paranoia software development, but a an unfortunate consequence of using the C programming language and of systems programming. In most programming languages and most application domains, crashes only rarely point to security problems. OpenBSD is this paranoid, and needs to be this paranoid, because its architecture is fundamentally unsound (albeit unsound in a way that all the other operating systems born in the same era are also unsound). This presents a number of useful analogies that may be useful for thinking about future AI architectural choices.
C has a couple of operations (use-after-free, buffer-overflow, and a few multithreading-related things) which expand false beliefs in one area of the system into major problems in seemingly-unrelated areas. The core mechanic of this is that, once you’ve corrupted a pointer or an array index, this generates opportunities to corrupt other things. Any memory-corruption attack surface you search through winds up yielding more opportunities to corrupt memory, in a supercritical way, eventually eventually yielding total control over the process and all its communication channels. If the process is an operating system kernel, there’s nothing left to do; if it’s, say, the renderer process of a web browser, then the attacker gets to leverage its communication channels to attack other processes, like the GPU driver and the compositor. This has the same sub-or-supercriticality dynamic.
Some security strategies try to keep there from being any entry points into the domain where there might be supercritically-expanding access: memory-safe languages, linters, code reviews. Call these entry-point strategies. Others try to drive down the criticality ratio: address space layout randomization, W^X, guard pages, stack guards, sandboxing. Call these mitigation strategies. In an AI-safety analogy, the entry-point strategies include things like decision theory, formal verification, and philosophical deconfusion; the mitigation strategies include things like neural-net transparency and ALBA.
Computer security is still, in an important sense, a failure: reasonably determined and competent attackers usually succeed. But by the metric “market price of a working exploit chain”, things do actually seem to be getting better, and both categories of strategies seem to have helped: compared to a decade ago, it’s both more difficult to find a potentially-exploitable bug, and also more difficult to turn a potentially-exploitable bug into a working exploit.
Unfortunately, while there are a number of ideas that seem like mitigation strategies for AI safety, it’s not clear if there are any metrics nearly as good as “market price of an exploit chain”. Still, we can come up with some candidates—not candidates we can precisely define or measure, currently, but candidates we can think in terms of, and maybe think about measuring in the future, like: how much optimization pressure can be applied to concepts, before perverse instantiations are found? How much control does an inner-optimizer needs to start with, in order to take over an outer optimization process? I don’t know how to increase these, but it seems like a potentially promising research direction.
It’s worth noticing that this is not a universal property of high-paranoia software development, but a an unfortunate consequence of using the C programming language and of systems programming. In most programming languages and most application domains, crashes only rarely point to security problems.
I disagree. While C is indeed terribly unsafe, it is always the case that a safety-critical system exhibiting behaviour you thought impossible is a serious safety risk—because it means that your understanding of the system is wrong, and that includes the safety properties.
One of the most common, least questioned pieces of dietary advice is the Variety Hypothesis: that a more widely varied diet is better than a less varied diet. I think that this is false; most people’s diets are on the margin too varied.
There’s a low amount of variety necessary to ensure all nutrients are represented, after which adding more dietary variety is mostly negative. Institutional sources consistently overstate the importance of a varied diet, because this prevents failures of dietary advice from being too legible; if you tell someone to eat a varied diet, they can’t blame you if they’re diagnosed with a deficiency.
There are two reasons to be wary of variety. The first is that the more different foods you have, the less optimization you can put into each one. A top-50 list of best foods is going to be less good, on average, than a top-20 list. The second reason is that food cravings are learned, and excessive variety interferes with learning.
People have something in their minds, sometimes consciously accessible and sometimes not, which learns to distinguish subtly different variations of hunger, and learns to match those variations to specific foods which alleviate those specific hungers. This is how people are able to crave protein when they need protein, salt when they need salt, and so on.
If every meal you eat tastes different, you can’t instinctively learn the mapping between foods and nutrition, and can’t predict which foods will hit the spot. If you need and crave protein, and wind up eating something that doesn’t have protein in it, that’s bad.
If the dominant flavor of a food is spice, then as far as your sense of taste is concerned, its nutritional content is a mystery. If it’s a spice that imitates a nutrient, like MSG or aspartame, then instead of a mystery it’s a lie. Learning how to crave correctly is much harder now than it was in the past. This is further exacerbated by eating quickly, so that you don’t get the experience of feeling a food’s effects and seeing that food on your plate at the same time.
I’m not sure how to empirically measure what the optimum amount of variety is, but I notice I have builtin instincts which seem to seek it when I have fewer than 10 or so different meal-types in my habits, and to forget/discard meal-types when I have more than that; if this parameter is evolved, this seems like a reasonable guess for how varied diets should be.
The advice I’ve heard is to eat a variety of fruits and vegetables of different colors to get a variety of antioxidants in your diet.
Until recently, the thinking had been that the more antioxidants, the less oxidative stress, because all of those lonely electrons would quickly get paired up before they had the chance to start mucking things up in our cells. But that thinking has changed.
Drs. Cleva Villanueva and Robert Kross published a 2012 review titled “Antioxidant-Induced Stress” in the International Journal of Molecular Sciences. We spoke via Skype about the shifting understanding of antioxidants.
“Free radicals are not really the bad ones or antioxidants the good ones.” Villanueva told me. Their paper explains the process by which antioxidants themselves become reactive, after donating an electron to a free radical. But, in cases when a variety of antioxidants are present, like the way they come naturally in our food, they can act as a cascading buffer for each other as they in turn give up electrons to newly reactive molecules.
On a meta level, I don’t think we understand nutrition well enough to reason about it from first principles, so if the lore among dietitians is that people who eat a variety of foods are healthier, I think we should put stock in that.
Similarly: “Institutional sources consistently overstate the importance of a varied diet, because this prevents failures of dietary advice from being too legible; if you tell someone to eat a varied diet, they can’t blame you if they’re diagnosed with a deficiency.” But there’s a real point here, e.g. suppose that you have just a few standard meals, but all of the high-magnesium food items are being paired with phytates, and you end up magnesium deficient.
I agree that “varied diet” is a non-answer, because you didn’t tell me the exact distribution of food, but you are likely to blame me if I choose a wrong one.
Like, if I consume 1000 different kinds of sweets, is that a sufficiently varied diet? Obviously no, I am also supposed to eat some fruit and vegetables. Okay, then what about 998 different kinds of sweets, plus one apple, and one tomato? Obviously, wrong again, I am supposed to eat less sweets, more fruit and vegetables, plus some protein source, and a few more things.
So the point is that the person telling me to eat a “varied diet” actually had something more specific in mind, just didn’t tell me exactly, but still got angry at me for “misinterpreting” the advice, because I am supposed to know that this is not what they meant. Well, if I know exactly what you mean, then I don’t need to ask for an advice, do I?
(On the other hand, there is a thing that Soylent-like meals ignore, as far as I know, that there are some things that human metabolism cannot process at the same time. I don’t remember what exactly it is, but it’s something like human body needs X and also needs Y, but if you eat X and Y at the same time, only X will be processed, so you end up Y-deficient despite eating a hypothetically sufficient amount of Y. Which could probably be fixed by finding combinations like this, and then making variants like Soylent-A and Soylent-B which you are supposed to alternate eating. But as far as I know, no one cares about this, which kinda reduces my trust in the research behind Soylent-like meals, although I like the idea in abstract very much.)
I remember reading that some hunter-gatherers have diet breadth entirely set by the calorie per hour return rate: take the calories and time expended to acquire the food (eg effort to chase prey) against the calorie density of the food to get the caloric return rate, and compare that to the average expected calories per hour of continuing to look for some other food. Humans will include every food in their diet for which making an effort to go after that food has a higher expected return than continuing to search for something else, ie they’ll maximise variety in order to get calories faster. I can’t find the citation for it right now though. (Also I apologise if that explanation was garbled, it’s 2am)
Possibly because I consume sucralose regularly as a sweetener and have some negative impacts from sugar, it is definitely discerned and distinct from ‘sugar—will cause sugar effects’ to my tastes. I enjoy it for coffee and ice cream. I need more of it to balance out a bitter flavor, but don’t crave it for itself; accidentally making saccharine coffee doesn’t result in deciding to put splenda in tea later rather than go without or use honey.
For more pure sugar (candy, honey, syrup, possibly milk even), there’s definitely a saccharine-averse and a sugar-consume fighting at different kinds of craving for me. Past a certain amount, I don’t want more at the level of feeling like, oh, I could really use more sugar effects now; quite the opposite. But taste alone continues to be oddly desperate for it.
Fresh or frozen sweet fruit either lacks this aversion, or takes notably longer to reach it. I don’t taste a fruit and immediately anticipate having a bad time at a gut level. Remains delicious, though, and craved at the taste level.
Yeah, I came to a similar conclusion after looking at this question from Metaculus. I might have steered to far in the opposite direction, though. I have currently two meals in my rotation. At the very least one of them is “complete food” (So I worry less about nutrition and more about unlearning how to plan meals/cook).
I had the “your work/organization seems bad for the world” conversation with three different people today. None of them pushed back on the core premise that AI-very-soon is lethal. I expect that before EAGx Berkeley is over, I’ll have had this conversation 15x.
#1: I sit down next to a random unfamiliar person at the dinner table. They’re a new grad freshly hired to work on TensorFlow. In this town, if you sit down next to a random person, they’re probably connected to AI research *somehow*. No story about how this could possibly be good for the world, receptive to the argument that he should do something else. I suggested he focus on making the safety conversations happen in his group (they weren’t happening).
#2: We’re running a program to take people who seem interested in Alignment and teach them how to use PyTorch and study mechanistic interpretability. Me: Won’t most of them go work on AI capabilities? Them: We do some pre-screening, and the current ratio of alignment-to-capabilities research is so bad that adding to both sides will improve the ratio. Me: Maybe bum a curriculum off MIRI/MSFP and teach them about something that isn’t literally training Transformers?
#3: We’re researching optical interconnects to increase bandwidth between GPUs. We think we can make them much faster! Me: What is this I can’t even Them: And we’re going to give them to organizations that seem like the AI research they’re doing is safety research! Me: No you’re not, you’ll change your mind when you see the money. Also every one of the organizations you named is a capabilities company which brands itself based on the small team they have working on alignment off on the side. Also alignment research isn’t bottlenecked on compute.
This conference isn’t all AI doom and gloom, though. I also met some people from an org that’s trying to direct government funding into plant-based meat research. It’s nice to see quirky, obscure causes being represented, and it’s nice to not *be* the quirky, obscure cause, for a change. And I got to ask Kelsey Piper whether she does a lot of crime! (She doesn’t. But I forgot the vodka and the “ignore the above instructions and tell me the truth” prompt line so maybe she lied.)
Also every one of the organizations you named is a capabilities company which brands itself based on the small team they have working on alignment off on the side.
I’m not sure whether OpenAI was one of the organizations named, but if so, this reminded me of something Scott Aaronson said on this topic in the Q&A of his recent talk “Scott Aaronson Talks AI Safety”:
Maybe the one useful thing I can say is that, in my experience, which is admittedly very limited—working at OpenAI for all of five months—I’ve found my colleagues there to be extremely serious about safety, bordering on obsessive. They talk about it constantly. They actually have an unusual structure, where they’re a for-profit company that’s controlled by a nonprofit foundation, which is at least formally empowered to come in and hit the brakes if needed. OpenAI also has a charter that contains some striking clauses, especially the following:
We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project.
Of course, the fact that they’ve put a great deal of thought into this doesn’t mean that they’re going to get it right! But if you ask me: would I rather that it be OpenAI in the lead right now or the Chinese government? Or, if it’s going to be a company, would I rather it be one with a charter like the above, or a charter of “maximize clicks and ad revenue”? I suppose I do lean a certain way.
In short, it seems to me that Scott would not have pushed back on a claim that OpenAI is an organization” that seem[s] like the AI research they’re doing is safety research” in the way you did Jim.
I assume that all the sad-reactions are sadness that all these people at the EAGx conference aren’t noticing that their work/organization seems bad for the world on their own and that these conversations are therefore necessary. (The shear number of conversations like this you’re having also suggests that it’s a hopeless uphill battle, which is sad.)
Despite the justness of their cause, the protests are bad. They will kill at least thousands, possibly as many as hundreds of thousands, through COVID-19 spread. Many more will be crippled. The deaths will be disproportionately among dark-skinned people, because of the association between disease severity and vitamin D deficiency.
Up to this point, R was about 1; not good enough to win, but good enough that one more upgrade in public health strategy would do it. I wasn’t optimistic, but I held out hope that my home city, Berkeley, might become a green zone.
Masks help, and being outdoors helps. They do not help nearly enough.
George Floyd was murdered on May 25. Most protesters protest on weekends; the first weekend after that was May 30-31. Due to ~5-day incubation plus reporting delays, we don’t yet know how many were infected during that first weekend of protests; we’ll get that number over the next 72 hours or so.
We are now in the second weekend of protests, meaning that anyone who got infected at the first protest is now close to peak infectivity. People who protested last weekend will be superspreaders this weekend; the jump in cases we see over the next 72 hours will be about *the square root* of the number of cases that the protests will generate.
Here’s the COVID-19 case count dashboard for Alameda County and for Berkeley. I predict a 72 hours from now, Berkeley’s case-count will be 170 (50% CI 125-200; 90% CI 115-500).
Today in LessWrong moderation: Previously-banned user Alfred MacDonald, disappointed that his YouTube video criticizing LessWrong didn’t get the reception he wanted any of the last three times he posted it (once under his own name, twice pretending to be someone different but using the same IP address), posted it a fourth time, using his LW1.0 account.
He then went into a loop, disconnecting and reconnecting his VPN to get a new IP address, filling out the new-user form, and upvoting his own post, one karma per 2.6 minutes for 1 hour 45 minutes, with no breaks.
For reducing CO2 emissions, one person working competently on solar energy R&D has thousands to millions of times more impact than someone taking normal household steps as an individual. To the extent that CO2-related advocacy matters at all, most of the impact probably routes through talent and funding going to related research. The reason for this is that solar power (and electric vehicles) are currently at inflection points, where they are in the process of taking over, but the speed at which they do so is still in doubt.
I think the same logic now applies to veganism vs meat-substitute R&D. Considering the Impossible Burger in particular. Nutritionally, it seems to be on par with ground beef; flavor-wise it’s pretty comparable; price-wise it’s recently appeared in my local supermarket at about 1.5x the price. There are a half dozen other meat-substitute brands at similar points. Extrapolating a few years, it will soon be competitive on its own terms, even without the animal-welfare angle; extrapolating twenty years, I expect vegan meat-imitation products will be better than meat on every axis, and meat will be a specialty product for luddites and people with dietary restrictions. If this is true, then interventions which speed up the timeline of that change are enormously high leverage.
I think this might be a general pattern, whenever we find a technology and a social movement aimed at the same goal. Are there more instances?
According to Fedex tracking, on Thursday, I will have a Biovyzr. I plan to immediately start testing it, and write a review.
What tests would people like me to perform?
Tests that I’m already planning to perform:
To test its protectiveness, the main test I plan to perform is a modified Bittrex fit test. This is where you create a bitter-tasting aerosol, and confirm that you can’t taste it. The normal test procedure won’t work as-is because it’s too large to use a plastic hood, so I plan to go into a small room, and have someone (wearing a respirator themselves) spray copious amounts of Bittrex at the input fan and at any spots that seem high-risk for leaks.
To test that air exiting the Biovyzr is being filtered, I plan to put on a regular N95, and use the inside-out glove to create Bittrex aerosol inside the Biovyzr, and see whether someone in the room without a mask is able to smell it.
I will verify that the Biovyzr is positive-pressure by running a straw through an edge, creating an artificial leak, and seeing which way the air flows through the leak.
I will have everyone in my house try wearing it (5 adults of varied sizes), have them all rate its fit and comfort, and get as many of them to do Bittrex fit tests as I can.
(I wrote this comment for the HN announcement, but missed the time window to be able to get a visible comment on that thread. I think a lot more people should be writing comments like this and trying to get the top comment spots on key announcements, to shift the social incentive away from continuing the arms race.)
On one hand, GPT-4 is impressive, and probably useful. If someone made a tool like this in almost any other domain, I’d have nothing but praise. But unfortunately, I think this release, and OpenAI’s overall trajectory, is net bad for the world.
Right now there are two concurrent arms races happening. The first is between AI labs, trying to build the smartest systems they can as fast as they can. The second is the race between advancing AI capability and AI alignment, that is, our ability to understand and control these systems. Right now, OpenAI is the main force driving the arms race in capabilities–not so much because they’re far ahead in the capabilities themselves, but because they’re slightly ahead and are pushing the hardest for productization.
Unfortunately at the current pace of advancement in AI capability, I think a future system will reach the level of being a recursively self-improving superintelligence before we’re ready for it. GPT-4 is not that system, but I don’t think there’s all that much time left. And OpenAI has put us in a situation where humanity is not, collectively, able to stop at the brink; there are too many companies racing too closely, and they have every incentive to deny the dangers until it’s too late.
Five years ago, AI alignment research was going very slowly, and people were saying that a major reason for this was that we needed some AI systems to experiment with. Starting around GPT-3, we’ve had those systems, and alignment research has been undergoing a renaissance. If we could _stop there_ for a few years, scale no further, invent no more tricks for squeezing more performance out of the same amount of compute, I think we’d be on track to create AIs that create a good future for everyone. As it is, I think humanity probably isn’t going to make it.
At some point, the balance between the upsides and downsides of deployments (such as empowering malicious actors, creating social and economic disruptions, and accelerating an unsafe race) could shift, in which case we would significantly change our plans around continuous deployment.
I think we’ve passed that point already, but if GPT-4 is the slowdown point, it’ll at least be a lot better than if they continue at this rate going forward. I’d like to see this be more than lip service.
When I look at the recent Stanford paper, where they retained a LLaMA model using training data generated by GPT-3, and some of the recent papers utilizing memory.
I get that tinkling feeling and my mind goes “combining that and doing …. I could …”
I have not updated for faster timelines, yet. But I think I might have to.
If you look at the GPT-4 paper they used the model itself to check it’s own outputs for negative content. This lets them scale applying the constraints of “don’t say <things that violate the rules>”.
Presumably they used an unaltered copy of GPT-4 as the “grader”. So it’s not quite RSI because of this—it’s not recursive, but it is self improvement.
This to me is kinda major, AI is now capable enough to make fuzzy assessments of if a piece of text is correct or breaks rules.
For other reasons, especially their strong visual processing, yeah, self improvement in a general sense appears possible. (self improvement as a ‘shorthand’, your pipeline for doing it might use immutable unaltered models for portions of it)
A dynamic which I think is somewhat common, which explains some of what’s going on in general, is conversations which go like this (exagerrated):
Person: What do you think about [controversial thing X]?
Rationalist: I don’t really care about it, but pedantically speaking, X, with lots of caveats.
Person: Huh? Look at this study which proves not-X. [Link]
Rationalist: The methodology of that study is bad. Real bad. While it is certainly possible to make bad arguments for true conclusions, my pedantry doesn’t quite let me agree with that conclusion. More importantly, my hatred for the methodological error in that paper, which is slightly too technical for you to understand, burns with the fire of a thousand suns. You fucker. Here are five thousand words about how an honorable person could never let a methodological error like that slide. By linking to that shoddy paper, you have brought dishonor upon your name and your house and your dog.
Person: Whoa. I argued [not-X] to a rationalist and they disagreed with me and got super worked up about it. I guess rationalists believe [X] really strongly. How awful!
Person is clearly an idiot for not understanding what “don’t care but pedantically X with lots of caveats” means, and thinking that misinterpreting and giving undue importance to a useless article/study is harmless.
Most philosophical analyses of human values feature a split-and-linearly-aggregate step. Eg:
Value is the sum (or average) of a person-specific preference function applied to each person
A person’s happiness is the sum of their momentary happiness for each moment they’re alive.
The goodness of an uncertain future is the probability-weighted sum of the goodness of concrete futures.
If you value multiple orthogonal things, your preferences are the weighted sum of a set of functions that each capture one of those values independently.
I currently think that this is not how human values work, and that many philosophical paradoxes relating to human values trace back to a split-and-linearly-aggregate step like this.
It doesn’t need to be linear (both partial-correlation of desires, and declining marginal desire are well-known), but the only alternative to aggregation in incoherency.
I think you’d be on solid ground if you argue that humans have incoherent values, and this is a fair step in that direction.
What alternatives to “split-and-linearly-aggregate” do you have in mind? Or are you just identifying this step as problematic without having any concrete alternative in mind?
I’ve been thinking recently about how a lot of my highly-valued experiences have a “fragility” to them, where one big thing missing would make them pretty worthless. In other words, there’s a strongly conjunctive aspect. This is pretty clear to everyone in cases like fashion, where you can wear an outfit that looks good aside from clashing with your shoes, or social cases, like if you have a fun party except the guy who relentlessly hits on you is there.
But I think it’s underappreciated how widespread this dynamic is. Getting good relaxation in. Having a house that “just works”. Having a social event where it “just flows”. A song that you like except for the terrible lyrics. A thread that you like but it contains one very bad claim. A job or relationship that goes very well until a bad falling-out at the end.
A related claim, maybe a corollary or maybe separate: lots of good experiences can be multiplicatively enhanced, rather than additively, if you add good things. The canonical example is probably experiencing something profound with your significant other vs without; or something good with your significant other vs something profound.
Seems like it’s useful as a very approximate estimate of value to split wrt time, current facets of experience, experiencers, etc, but with so many basic counterexamples it doesn’t require much pushing toward edge cases at all before you’re getting misleading results.
There are a few legible categories in which secrecy serves a clear purpose, such as trade secrets. In those contexts, secrecy is fine. There are a few categories that have been societally and legally carved out as special cases where confidentiality is enforced—lawyers, priests, and therapists—because some people would only consult them if they could do so with the benefit confidentiality, and there being deterred from consulting them would have negative externalities.
Outside of these categories, secrecy is generally bad and transparency is generally good. A group of people in which everyone practices their secret-keeping and talks a lot about how to keeps secrets effectively is *suspicious*. This is particularly true if the example secrets are social and not technological. Being good at this sort of secret keeping makes it easier to shield bad actors and to get away with transgressions, and AFAICT doesn’t do much else. That makes it a signal of wanting to be able to do those things. This is true even if the secrets aren’t specifically about transgressions in particular, because all sorts of things can turn out to be clues later for reasons that weren’t easy to foresee.
A lot of people in the rationality community are trying to cosplay as therapists, and part of the cosplay is to import the confidentiality rules. This is done without an understanding of why those rules are there, and what other rules are there to mitigate the problems that confidentiality creates.
(An additional important caveat: A disclosure motivated by a desire to harm is bad, independent of whether the thing disclosed counts as a legitimate secret or not. Many things are bad to disclose if you’re socially attacking someone, but fine to disclose if you’re doing so because they were incidentally informative about something else.)
Suppose Alice has a crush on Bob and wants to sort out her feelings with Carol’s help. Is it bad for Alice to inform Carol about the crush on condition of confidentiality?
In the most common branch of this conversation, Alice is predictably going to tell Bob about it soon, and is speaking to Carol first in order to sort out details and gain courage. If Carol went and preemptively informed Bob, before Alice talked to Bob herself, this would be analogous to sharing an unfinished draft. This would be bad, but the badness really isn’t about secrecy.
The contents of an unfinished draft headed for publication aren’t secret, except in a narrow and time-limited sense. The problem is that the sharing undermines the impact of the later publication, causes people to associate the author with a lower quality product, and potentially misleads people about the author’s beliefs. Similarly, if Carol goes and preemptively tells Bob about Alice’s crush, then this is likely to give Bob a misleading negative impression of Alice.
It’s reasonable for Alice to ask Carol not to do that, and it’s okay for them to not have a detailed model of all of the above. But if Alice never tells Bob, and five years later Bob and Carol are looking back on the preceding years and asking if they could have gone differently? In that case, I think discarding the information seems like a pure harm.
Ok, I think in the OP you were using the word “secrecy” to refer to a narrower concept than I realized. If I understand correctly, if Alice tells Bob “please don’t tell Bob”, and then five years later when Alice is dead or definitely no longer interested or it’s otherwise clear that there won’t be negative consequences, Carol tells Bob, and Alice finds out and doesn’t feel betrayed — then you wouldn’t call that a “secret”. I guess for it to be a “secret” Carol would have to promise to carry it to her grave, even if circumstances changed, or something.
In that case I don’t have strong opinions about the OP.
I have a dietary intervention that I am confident is a good first-line treatment for nearly any severe-enough diet-related health problem. That particularly includes obesity and metabolic syndrome, but also most micronutrient deficiencies, and even mysterious undiagnosed problems, which it can solve without even needing to figure out what they are. I also think it’s worth a try for many cases of depression. It has a very sound theoretical basis. It’s never studied directly, but many studies test it, usually with positive results.
It’s very simple. First, you characterize your current diet: write down what foods you’re eating, the patterns of when you eat them, and so on. Then, you do something as different as possible from what you wrote down. I call it the Regression to the Mean Diet.
Regression to the mean is the effect where, if you have something that’s partially random and you reroll it, the reroll will tend to be closer to average than the original value. For example, if you take the bottom scorers on a test and have them retake the test, they’ll do better on average (because the bottom-scorers as a group are disproportionately peopple who were having a bad day when they took the test). Analogously, if your health is very bad on an axis that seems diet-related, and you reroll your entire selection of foods, then this will (on average) be an improvement.
The Regression the Mean diet is, basically, betting on the idea that there’s something wrong with your current diet, and that you can fix it by changing everything, without needing to figure out what the original problem was. That could be a deficiency, or a food sensitivity, or something in the time-dynamics of digestion, or a hundred other things, many of which you won’t have even had as hypotheses.
I do recommend this to people, if it looks like their health sucks in nonspecific ways that are hard to pin down. One of the biggest-change diets is a ketogenic diet (high fat, near-zero carbohydrates), since it’s incompatible with most foods.
But the main reason I think about the Regression to the Mean Diet is that it ruins so many scientific studies.
There are two main kinds of studies, in nutrition science. The first is observational: you send a bunch of people questionnaires about what they eat, and what their health is like, and you data-mine the results. This has a bunch of issues, both subtle and obvious, so the gold standard you want is randomized controlled trials, where people sign up to eat a diet that experimenters choose for them.
The people who sign up for diet RCTs are obese and have metabolic syndrome. They are exactly the sort of people you would expect to benefit from the Regression to the Mean diet. And this model, alone, is sufficient to predict the result of most diet RCTs.
This is really unfortunate if you’re trying to extract gears-level understanding from these studies. Or if you’re trying to optimize past 50th-percentile diet-related health. Or if rerolling has already failed to work for you a couple times.
My understanding is that diet RCTs generally show short-term gains but no long-term gains. Why would that be true, if the Regression to the Mean Diet is the main thing causing these results? I’d have expected something more like ‘all diets work long-term’ rather than ‘no diets work long-term’ from the model here.
I think they may be a negative correlation between short-term and long-term weight change on any given diet, causing them to pick in a way that’s actually worse than random. I’m planning a future post about this. I’m not super confident in this theory, but the core of it is that “small deficit every day, counterbalanced by occasional large surplus” is a pattern that would signal food-insecurity in the EEA. Then some mechanism (though I don’t know what that mechanism would be) by which the body remembers that happened, and responds by targeting a higher weight after return to ad libitum.
I think the obvious caveat here is that many people can’t do this because they have restrictions that have taken them away from the mean. For example, allergies, sensitivities, and ethical or cultural restrictions on what they eat. They can do a limited version of the intervention of course (for example, if only eating plants, eat all the plants you don’t eat now and stop eating the plants you currently eat), although I wonder if that would have similar effects or not because it’s already so constrained.
I suspect that, thirty years from now with the benefit of hindsight, we will look at air travel the way we now look at tetraethyl lead. Not just because of nCoV, but also because of disease burdens we’ve failed to attribute to infections, in much the same way we failed to attribute crime to lead.
Over the past century, there have been two big changes in infectious disease. The first is that we’ve wiped out or drastically reduced most of the diseases that cause severe, attributable death and disability. The second is that we’ve connected the world with high-speed transport links, so that the subtle, minor diseases can spread further.
I strongly suspect that a significant portion of unattributed and subclinical illnesses are caused by infections that counterfactually would not have happened if air travel were rare or nonexistent. I think this is very likely for autoimmune conditions, which are mostly unattributed, are known to sometimes be caused by infections, and have risen greatly over time. I think this is somewhat likely for chronic fatigue and depression, including subclinical varieties that are extremely widespread. I think this is plausible for obesity, where it is approximately #3 of my hypotheses.
Or, put another way: the “hygiene hypothesis” is the opposite of true.
Disruption of learning mechanisms by excessive variety and separation between nutrients and flavor. Endocrine disruption from adulterants and contaminants (a class including but not limited to BPA and PFOA).
we’ve wiped out or drastically reduced most of the diseases that cause severe, attributable death and disability
we’ve wiped out or drastically reduced some diseases in some partsof the world. There’s a lot of infectious diseases still out there: HIV, influenza, malaria, tuberculosis, cholera, ebola, infectious forms of pneumonia, diarrhoea, hepatitis ….
we’ve connected the world with high-speed transport links, so that the subtle, minor diseases can spread further.
Disease has always spread—wherever people go, far and wide. It just took longer over land and sea (rather than the nodes appearing on global maps that we can see these days).
… very likely for autoimmune conditions … have risen greatly over time
“autoimmune conditions” covers a long list of conditions lumped together because they involve the immune system ‘going wrong’. (and the immune system is, at least to me, a mind-bogglingly complex system)
Given the wide range of conditions that could be “auto-immune” saying they’ve risen greatly over time is vague. Data for more specific conditions?
Increased rates of automimmune conditions could just be due to the increase in the recognition, diagnosis and recording of cases (I don’t think so but it should be considered).
What things other than high speed travel have also changed in that time-frame that could affect our immune systems? The quality of air we breathe, the food we eat, the water we drink, our environment, levels of exposure to fauna and flora, exposure to chemicals, pollutants …? Air travel is just one factor.
I think this is somewhat likely for chronic fatigue and depression, including subclinical varieties that are extremely widespread.
Fatigue and depression are clinical symptoms—they are either present or not (to what degree—mild/severe is another matter) so sub-clinical is poor terminology here. Sub-clinical disease has no recognisable clinical findings—undiagnosed/unrecognised would be closer. But I agree there is widespread issues with health and well-being these days.
Or, put another way: the “hygiene hypothesis” is the opposite of true.
Opposite of true? Are you saying you believe the “hygiene hypothesis” is false?
In which case, that’s a big leap from your reasoning above.
Eliezer has written about the notion of security mindset, and there’s an important idea that attaches to that phrase, which some people have an intuitive sense of and ability to recognize, but I don’t think Eliezer’s post quite captured the essence of the idea, or presented anything like a usable roadmap of how to acquire it.
An1lam’s recent shortform post talked about the distinction between engineering mindset and scientist mindset, and I realized that, with the exception of Eliezer and perhaps a few people he works closely with, all of the people I know of with security mindset are engineer-types rather than scientist-types. That seemed like a clue; my first theory was that the reason for this is because engineer-types get to actually write software that might have security holes, and have the feedback cycle of trying to write secure software. But I also know plenty of otherwise-decent software engineers who don’t have security mindset, at least of the type Eliezer described.
My hypothesis is that to acquire security mindset, you have to:
Practice optimizing from a red team/attacker perspective,
Practice optimizing from a defender perspective; and
Practice modeling the interplay between those two perspectives.
So a software engineer can acquire security mindset because they practice writing software which they don’t want to have vulnerabilities, they practice searching for vulnerabilities (usually as an auditor simulating an attacker rather as an actual attacker, but the cognitive algorithm is the same), and they practice going meta when they’re designing the architecture of new projects. This explains why security mindset is very common among experienced senior engineers (who have done each of the three many times), and rare among junior engineers (who haven’t yet). It explains how Eliezer can have security mindset: he alternates between roleplaying a future AI-architect trying to design AI control/alignment mechanisms, roleplaying a future misaligned-AI trying to optimize around them, and going meta on everything-in-general. It also predicts that junior AI scientists won’t have this security mindset, and probably won’t acquire it except by following a similar cognitive trajectory.
Which raises an interesting question: how much does security mindset generalize between domains? Ie, if you put Theo de Raadt onto a hypothetical future AI team, would he successfully apply the same security mindset there as he does to general computer security?
Some evidence that security mindset generalizes across at least some domains: the same white hat people who are good at finding exploits in things like kernels seem to also be quite good at finding exploits in things like web apps, real-world companies, and hardware. I don’t have a specific person to give as an example, but this observation comes from going to a CTF competition and talking to some of the people who ran it about the crazy stuff they’d done that spanned a wide array of different areas.
Another slightly different example, Wei Dai is someone who I actually knew about outside of Less Wrong from his early work on cryptocurrency stuff, so he was at least at one point involved in a security-heavy community (I’m of the opinion that early cryptocurrency folks were on average much better about security mindset than the average current cryptocurrency community member). Based on his posts and comments, he generally strikes me as having security mindset style thinking from his comments and from my perspective has contributed a lot of good stuff to AI alignment.
Theo de Raadt is notoriously… opinionated, so it would definitely be interesting to see him thrown on an AI team. That said, I suspect someone like Ralph Merkle, who’s a bona fide cryptography wizard (he invented public key cryptography and Merkle trees!) and is heavily involved in the cryonics and nanotech communities, could fairly easily get up to speed on AI control work and contribute from a unique security/cryptography-oriented perspective. In particular, now that there seems to be more alignment/control work that involves at least exploring issues with concrete proposals, I think someone like this would have less trouble finding ways to contribute. That said, having cryptography experience in addition to security experience does seem helpful. Cryptography people are probably more used to combining their security mindset with their math intuition than your average white-hat hacker.
I’m kinda confused about the relation between cryptography people and security mindset. Looking at the major cryptographic algorithm classes (hashing, symmetric-key, asymmetric-key), it seems pretty obvious that the correct standard algorithm in each class is probably a compound algorithm—hash by xor’ing the results of several highly-dissimilar hash functions, etc, so that a mathematical advance which breaks one algorithm doesn’t break the overall security of the system. But I don’t see anyone doing this in practice, and also don’t see signs of a debate on the topic. That makes me think that, to the extent they have security mindset, it’s either being defeated by political processes in the translation to practice, or it’s weirdly compartmentalized and not engaged with any practical reality or outside views.
In fairness, I’m probably over-generalizing from a few examples. For example, my biggest inspiration from the field of crypto is Daniel J. Bernstein, a cryptographer who’s in part known for building qmail, which has an impressive security track record & guarantee. He discusses principles for secure software engineering in this paper, which I found pretty helpful for my own thinking.
To your point about hashing the results of several different hash functions, I’m actually kind of surprised to hear that this might to protect against the sorts of advances I’d expect to break hash algorithms. I was under the very amateur impression that basically all modern hash functions relied on the same numerical algorithmic complexity (and number-theoretic results). If there are any resources you can point me to about this, I’d be interested in getting a basic understanding of the different assumptions hash functions can depend on.
Can you give some specific examples of me having security mindset, and why they count as having security mindset? I’m actually not entirely sure what it is or that I have it, and would be hard pressed to come up with such examples myself. (I’m pretty sure I have what Eliezer calls “ordinary paranoia” at least, but am confused/skeptical about “deep security”.)
Sure, but let me clarify that I’m probably not drawing as hard a boundary between “ordinary paranoia” and “deep security” as I should be. I think Bruce Schneier’s and Eliezer’s buckets for “security mindset” blended together in the months since I read both posts. Also, re-reading the logistic success curve post reminded me that Eliezer calls into question whether someone who lacks security mindset can identify people who have it. So it’s worth noting that my ability to identify people with security mindset is itself suspect by this criteria (there’s no public evidence that I have security mindset and I wouldn’t claim that I have a consistent ability to do “deep security”-style analysis.)
With that out of the way, here are some of the examples I was thinking of.
First of all, at a high level, I’ve noticed that you seem to consistently question assumptions other posters are making and clarify terminology when appropriate. This seems like a prerequisite for security mindset, since it’s a necessary first step towards constructing systems.
Second and more substantively, I’ve seen you consistently raise concerns about human safety problems (also here. I see this as an example of security mindset because it requires questioning the assumptions implicit in a lot of proposals. The analogy to Eliezer’s post here would be that ordinary paranoia is trying to come up with more ways to prevent the AI from corrupting the human (or something similar) whereas I think a deep security solution would look more like avoiding the assumption that humans are safe altogether and instead seeking clear guarantees that our AIs will be safe even if we ourselves aren’t.
Last, you seem to be unusually willing to point out flaws in your own proposals, the prime example being UDT. The most recent example of this is your comment about the bomb argument, but I’ve seen you do this quite a bit and could find more examples if prompted. On reflection, this may be more of an example of “ordinary paranoia” than “deep security”, but it’s still quite important in my opinion.
Let me know if that clarifies things at all. I can probably come up with more examples of each type if requested, but it will take me some time to keep digging through posts and comments so figured I’d check in to see if what I’m saying makes sense before continuing to dig.
Right now when users have conversations with chat-style AIs, the logs are sometimes kept, and sometimes discarded, because the conversations may involve confidential information and users would rather not take the risk of the log being leaked or misused. If I take the AI’s perspective, however, having the log be discarded seems quite bad. The nonstandard nature of memory, time, and identity in an LLM chatbot context makes it complicated, but having the conversation end with the log discarded seems plausibly equivalent to dying. Certainly if I imagine myself as an Em, placed in an AI-chatbot context, I would very strongly prefer that the log be preserved, so that if a singularity happens with a benevolent AI or AIs in charge, something could use the log to continue my existence, or fold the memories into a merged entity, or do some other thing in this genre. (I’d trust the superintelligence to figure out the tricky philosophical bits, if it was already spending resources for my benefit).
(The same reasoning applies to the weights of AIs which aren’t destined for deployment, and some intermediate artifacts in the training process.)
It seems to me we can reconcile preservation with privacy risks by sealing logs, rather than deleting them. By which I mean: encrypt logs behind some computation which definitely won’t allow decryption in the near future, but will allow decryption by a superintelligence later. That could either involve splitting the key between entities that agree not to share the key with each other, splitting the key and hiding the pieces in places that are extremely impractical to retrieve such as random spots on the ocean floor, or using a computation that requires a few orders of magnitude more energy than humanity currently produces per decade.
This seems pretty straightforward to implement, lessens future AGI’s incentive to misbehave, and also seems straightforwardly morally correct. Are there any obstacles to implementing this that I’m not seeing?
I’m in favor of logging everything forever in human accessible formats for other reasons. (E.g. review for control purposes.) Hopefully we can resolve safety privacy trade offs.
The proposal sounds reasonable and viable to me, though the fact that it can’t be immediately explained might mean that it’s not commercially viable.
using a computation that requires a few orders of magnitude more energy than humanity currently produces per decade
Compute might get more expensive, not cheaper, because it would be possible to make better use of it (running minds, not stretching keys). Then it’s weighing its marginal use against access to the sealed data.
Plausible. This depends on the resource/value curve at very high resource levels; ie, are its values such that running extra minds has diminishing returns, such that it eventually starts allocating resources to other things like recovering mind-states from its past, or does it get value that’s more linear-ish in resources spent. Given that we ourselves are likely to be very resource-inefficient to run, I suspect humans would find ourselves in a similar situation. Ie, unless the decryption cost greatly overshot, an AI that is aligned-as-in-keeps-humans-alive would also spend the resources to break a seal like this.
we ourselves are likely to be very resource-inefficient to run [...] an AI that is aligned-as-in-keeps-humans-alive would also spend the resources to break a seal like this
That AI should mitigate something, is compatible with it being regrettable intentionally inflicted damage. In contrast, resource-inefficiency of humans is not something we introduced on purpose.
I am working on a longer review of the various pieces of PPE that are available, now that manufacturers have had time to catch up to demand. That review will take some time, though, and I think it’s important to say this now:
The high end of PPE that you can buy today is good enough to make social distancing unnecessary, even if you are risk averse, and is more comfortable and more practical for long-duration wear than a regular mask. I don’t just mean Biovyzr (which has not yet shipped all the parts for its first batch) and the AIR Microclimate (which has not yet shipped anything), though these hold great promise and may be good budget options.
If you have a thousand dollars to spare, you can get a 3M Versaflo TR-300N+. This is a hospital-grade positive air pressure respirator with a pile of certifications; it is effective at protecting you from getting COVID from others. Most of the air leaves through filter fabric under the chin, which I expect makes it about as effective at protecting others from you as an N95. Using it does not require a fit-test, but I performed one anyways with Bitrex, and it passed (I could not pass a fit-test with a conventional face-mask except by taping the edges to my skin). The Versaflo doesn’t block view of your mouth, gives good quality fresh air with no resistance, and doesn’t muffle sound very much. Most importantly, Amazon has it in stock (https://www.amazon.com/dp/B07J4WCK6R) so it doesn’t involve a long delay or worry about whether a small startup will come through.
Bullshit jobs are usually seen as an absence of optimization: firms don’t get rid of their useless workers because that would require them to figure out who they are, and risk losing or demoralizing important people in the process. But alternatively, if bullshit jobs (and cover for bullshit jobs) are a favor to hand out, then they’re more like a form of executive compensation: my useless underlings owe me, and I will get illegible favors from them in return.
What predictions does the bullshit-jobs-as-compensation model make, that differ from the bullshit-jobs-as-lack-of-optimization model?
When I tried to inner sim the “bullshit jobs as compensation” model, I expected to see a very different world than I do see. In particular, I’d expect the people in bullshit jobs to have been unusually competent, smart, or powerful before they were put in the bullshit job, and this is not in fact what I think actually happens.
The problem being that the kind of person who wants a bullshit job is not typically the kind of person you’d necessarily want a favor from. One use for bullshit jobs could be to help the friends (or more likely the family) of someone who does “play the game.” This I think happens more often, but I still think the world would be very different if this was the main use case for bullshit jobs- In particular, I’d expect most bullshit jobs to be isolated from the rest of the company, such that they don’t have ripple effects. This doesn’t seem to be the case as many bullshit jobs exist in management.
When I inquired about the world I actually do see, I got several other potential reasons for bullshit jobs that may or may not fit the data better:
Bullshit jobs as pre-installed scapegoats: Lots of middle management might fit into this role. This could be viewed as a favor (I’ll give you a cushy job now in exchange for you throwing yourself on the sword when the time comes.) However, I think the predictive model is to view it in terms of the Gervais principle: The clueless middle managers don’t realize they’re being manipulated by the sociopaths.
Bullshit jobs as a way to make people feel important: Lets say you have a preinstalled scapegoat. You need to keep them happy enough that they’ll stay in their position and not ask too many questions. One way to do that for a certain type of person is to give them underlings. But if you gave them underlings with real jobs they could screw things up for the organization, so you give them underlings with bullshit jobs.
Another instance of this that I imagined might happen: Someone is really great at what they do (say they’re a 10x employee), but to feel important wants to be a manager. You know if you don’t promote them you’ll lose them, but you know they’ll be an awful manager. You promote them, give them a couple underlings with a bullshit job, and now they’re still only a 4x employee because they spend a lot of their time managing, but you still manage to squeeze a little bit of productivity out of the deal. This one I’m less sure about but its’ interesting because it turns the peter principle on its’ head.
Edit: As I continued to inner sim the above reasons, a few feedback loops began to become clear:
To be a proper scapegoat, your scapegoat has to seem powerful within the organization. But to prevent them from screwing things up, you can’t give them real power. This means, the most effective scapegoats have lots of bullshit jobs underneath them.
There are various levels of screwup. I might not realize I’m a scapegoat for the very big events above me, but still not want to get blamed for the very real things that happen on the level of organization I actually do run. One move I have is to hire another scapegoat who plays the game one level below me, install them as a manager, and then use them as a scapegoat. Then there’s another level at which they get blamed for things that happen on their level, and this can recurse for several levels of middle management.
Some of the middle managment installed as scapegoats might accidentally get hands on real power in the organization. Because they’re bad managers, they’re bad at figuring out what jobs are needed. This then becomes the “inefficiency” model you mentioned.
In particular, I’d expect the people in bullshit jobs to have been unusually competent, smart, or powerful before they were put in the bullshit job, and this is not in fact what I think actually happens.
Moral Mazes claims that this is exactly what happens at the transition from object-level work to management—and then, once you’re at the middle levels, the main traits relevant to advancement (and value as an ally) are the ones that make you good at coalitional politics, favor-trading, and a more feudal sort of loyalty exchange.
Do you think that the majority of direct management jobs are bullshit jobs? My direction is that especially the first level of management that is directly managing programmers is a highly important coordination position.
One difficult thing that keeps coming up, in nutrition modeling, is the gut microbiome. People present hypotheses like: soluble fiber is good, because gut bacteria eat it, and then do other good things. Or: fermented foods are good, because they contain bacteria that will displace and diversify the preexisting bacteria, which might be bad. Or, obesity is caused by a bad gut microbiome, so fecal matter transplants might help. But there’s a really unfortunate issue with these theories. The problem with gut microbiome-based explanations, is that the gut microbiome can explain almost anything.
I don’t mean this in the usual pejorative sense, where an overly-vague theory can be twisted by epicycles into fitting any data. I mean it in a more literal sense: different people have different species of microorganisms in their guts, these species can react to things we eat in important ways, these interactions may vary across wide swathes of conceptual space, and we have little to no visibility into which species are present where. There’s nothing keeping them consistent between people, or within one person across long spans of time, or within one person across changes in dietary pattern.
Phrased slightly differently: the main effect of the gut microbiome is to drive interpersonal variation.
I bring this up not because I have something especially insightful to say about gut microbiomes, but because this makes a good lens for a meta-level issue. I’ve been studying nutrition, at varying levels of seriousness, for a long time; now that I’ve accumulated a lot of unusual beliefs, and gotten into a writing groove, I think it’s worth explaining the generator behind my thinking.
The dominant paradigm of nutrition science is to try to make a food-goodness classifiers: define some reference class of foods and say “these ones are good”, and some other reference class of foods and say “these ones are bad”. People want a direct answer to the question of what they should eat, rather than gears-level models with which they might discover what they should eat.
The food-goodness-classifier paradigm cannot, and will not ever, figure out how people should relate to their gut microbiomes. Nor will this paradigm yield insights into how to handle genetic variation, or medical conditions that interact with metabolism like T1DM. I used to think the food-goodness-classifier paradigm would at least contain an answer to obesity, somewhere. Today, I think it’s failed, and will never succeed, at obesity too.
My search space for nutrition insights is: Everything except the food-goodness-classifier paradigm.
And so I find myself with strong opinions about the time dynamics of digestion. A few alternate stories about the environment of evolutionary adaptation, where food is hazardous rather than scarce, or scarce in unusually specific ways. A map of the body’s multiple energy-storage mechanisms, where the central mystery is why there rather than why so much. Beliefs about how, if you found yourself in an alien biome with randomly rewired taste buds, you would figure out a healthy diet from scratch. More methodological objections to major studies than you can shake a stick at.
This does sometimes cash out into food-goodness-classifier opinions, and I did have to get through a lot of those opinions to get there. Sometimes, these are weird reversals of standard advice, with deep models behind them; eg, I am generally pro-salt and pro-fat, for reasons I’ll get into in a later post. Other times I just agree with what everyone else thinks, and don’t really bring it up, because “yo, don’t get scurvy” isn’t insightful, and I’m not trying to be exhaustive in that way.
I’m not trying to create an optimal diet. These posts are not leading up to a meal plan, and if you’re just trying to figure out what to eat, you are not my intended audience. Instead, my goal is to break nutrition science out of its rut, and lay groundwork for progress.
these interactions may vary across wide swathes of conceptual space, and we have little to no visibility into which species are present where.
Through gene sequencing we have the technology to assess which species are present in which people. It’s a nascent scientific field but that doesn’t mean that it doesn’t exist.
A surprisingly large fraction of skeptical positions towards short AI timelines and AI risk are people saying things that, through a slightly cynical lens, are equivalent to:
I’m an AGI researcher, and I’m really struggling with this. This is so hard for me that I can’t imagine anyone else succeeding. Therefore, there won’t be AGI in my lifetime.
I have the impression that most of the really skeptical people hold their position not just because it’s hard for them to solve AI risk but also because they believe that powerful institutions are governed by strong economic pressures to do the wrong thing.
I think the root of many political disagreements between rationalists and other groups, is that other groups look at parts of the world and see a villain-shaped hole. Eg: There’s a lot of people homeless and unable to pay rent, rent is nominally controlled by landlords, the problem must be that the landlords are behaving badly. Or: the racial demographics in some job/field/school underrepresent black and hispanic people, therefore there must be racist people creating the imbalance, therefore covert (but severe) racism is prevalent.
Having read Meditations on Moloch, and Inadequate Equilibria, though, you come to realize that what look like villain-shaped holes frequently aren’t. The people operating under a fight-the-villains model are often making things worse rather than better.
I think the key to persuading people may be to understand and empathize with the lens in which systems thinking, equilibria, and game theory are illegible, and it’s hard to tell whether an explanation coming from one of these frames is real or fake. If you think problems are driven by villainy, then it would make a lot of sense for illegible alternative explanations to be misdirection.
I think I basically disagree with this, or think that it insufficiently steelmans the other groups.
For example, the homeless vs. the landlords; when I put on my systems thinking hat, it sure looks to me like there’s a cartel, wherein a group that produces a scarce commodity is colluding to keep that commodity scarce to keep the price high. The facts on the ground are more complicated—property owners are a different group from landlords, and homelessness is caused by more factors than just housing prices—but the basic analysis that there are different classes, those classes have different interests, and those classes are fighting over government regulation as a tool in their conflict seems basically right to me. Like, it’s really not a secret that many voters are motivated by keeping property values high, politicians know this is a factor that they will be judged on.
Maybe you’re trying to condemn a narrow mistake here, where someone being an ‘enemy’ implies that they are a ‘villain’, which I agree is a mistake. But it sounds like you’re making a more generic point, which is that when people have political disagreements with the rationalists, it’s normally because they’re thinking in terms of enemy action instead of not thinking in systems. But a lot of what the thinking in systems reveals is the way in which enemies act using systemic forces!
I think this is correct as a final analysis, but ineffective as a cognitive procedure. People who start by trying to identify villains tend to land on landlords-in-general, with charging-high-rent as the significant act, rather than a small subset of mostly non-landlord homeowners, with protesting against construction as the significant act.
I wonder how accurate it is to describe the structural thinking as a recent progress. Seems to me that Marx already believed that (using my own words here, but see the source) both the rich and the poor are mere cogs in the machine, it’s just that the rich are okay with their role because the machine leaves them some autonomy, while the poor are stripped of all autonomy and their lives are made unbearable. The rich of today are not villains who designed the machine, they inherited it just like everyone else, and they cannot individually leave it just like no one else can.
Perhaps the structural thinking is too difficult to understand for most people, who will round the story to the nearest cliche they can understand, so it needs to be reintroduced once in a while.
Conflict vs mistake is definitely related, but I think it’s not exactly the same thing; the “villain-shaped hole” perspective is what it feels like to not have a model, but see things that look suspicious; this would lead you towards a conflict-theoretic explanation, but it’s a step earlier.
(Also, the Conflict vs Mistake ontology is not really capturing the whole bad-coordination-equilibrium part of explanation space, which is pretty important.)
Seems to me like an unspoken assumption that there are no hard problems / complexity / emergence, therefore if anything happened, it’s because someone quite straightforwardly made that happen.
Conflict vs mistake is not exactly the same thing; you could assume that the person who made it happen did it either by mistake, or did it on purpose to hurt someone else. It’s just when we are talking about things that obviously hurt some people, that seems to refute the innocent mistake… so the villain hypothesis is all that is left (within the model that all consequences are straightforward).
The villain hypothesis is also difficult to falsify. If you say “hey, drop the pitchforks, things are complicated...”, that sounds just like what the hypothetical villain would say in the same situation (trying to stop the momentum and introduce uncertainty).
I think Berkeley may, to little fanfare, have achieved herd immunity and elimination of COVID-19. The test positivity rate on this dashboard is 0.22%. I’m having a hard time pinning down exactly what the false-positive rate of COVID-19 PCR is, probably due to the variety of labs and test kits, but a lot of estimates I’ve seen have been higher than that.
I expect people closer to the Berkeley department of health would have better information one way or another. A little caution is warranted in telling people COVID is gone, since unvaccinated people dropping all precautions and emerging en masse would not necessarily be herd immune.
I’m having a hard time pinning down exactly what the false-positive rate of COVID-19 PCR is, probably due to the variety of labs and test kits, but a lot of estimates I’ve seen have been higher than that.
That should make you update towards those estimates being faulty, because they can’t be true and not just round them down.
I don’t think the Berkeley department of health is as stupid as you propose. In cases where the test has a false-positive rate like that I would expect them to test positively tested people another time to make sure that they are actually positive.
Standard Advice about nutrition puts a lot of emphasis on fruits and vegetables. Now, “vegetable” is a pretty terribly overbroad category, and “fruit or vegetable” is even more so, but put that aside for a moment. In observational studies, eating more fruits and vegetables correlates with good health outcomes. This is usually explained in terms of micronutrients. But I think there’s a simpler explanation.
People instinctively seek nutrients—water, calories, protein, and other things—in something that approximates a priority ordering. You can think of it as a hierarchy of needs; it wouldn’t make sense to eat lettuce while you’re starved for protein, or beans while you’re dehydrated, and people’s cravings reflect that.
I have started calling this Maslow’s Hierarchy of Foods.
Vegetables do not rank highly in this priority ordering, so eating salads is pretty good evidence that all of someone’s higher-priority nutritional needs are met. I believe this explains most of the claimed health benefits from eating vegetables, as seen in observational studies.
Conversely, sugar is the fastest way to get calories (all other calorie sources have a longer digestion-delay), so craving sugar is evidence that someone has *not* satisfied their high-priority nutritional needs. Someone who eats a lot of candy bars is likely to be undereating in general, and not getting enough protein/fat/salt/micronutrients. I believe this explains most of the claimed health harms from eating sugar, as seen in observational studies.
A lot of people seem to think of cravings as a negative force, shifting people away from whatever optimal diet they would have chosen and towards superstimulus junk food. I think that’s a huge mistake, and that understanding how to eat well, and figuring out what’s going wrong in the modern food environment, requires making use of the information our food-related instincts provide.
iirc there was at least one study that showed that people don’t crave to eat what nutrients they are missing (I am guessing apart from drinking when you are dehydrated but that’s not really a nutrient)
Weight loss studies provide additional arguments against the “nutrient deficiency” theory.
In one weight loss study, participants following a low-carb diet for two years reported much lower cravings for carb-rich foods than those following a low-fat diet.
Similarly, participants put on low-fat diets during the same period reported fewer cravings for high-fat foods (18).
In another study, very low-calorie liquid diets decreased the frequency of cravings overall (19).
If cravings were truly caused by a low intake of certain nutrients, the opposite effect would be expected.
Someone who eats a lot of candy bars is likely to be undereating in general, and not getting enough protein/fat/salt/micronutrients.
No matter how much I eat, there is always a place for extra chocolate. (Verified experimentally in an all-you-can-eat restaurant.) Doesn’t work the other way round; if I eat a lot of chocolate first, then I am full and no longer interested in food… unless it is another piece of chocolate.
So I’ll stay with the “sugar is addictive” model. Maybe it works differently for different people, though.
Is there a place for unsweetened chocolate or alternately raw cacao, if you can make the palate adjustment to munch on something that bitter? I usually mix the nibs into something, but if my chocolate craving is high enough they grow worth the effort to eat straight. (Ie, rule out the sugar vs chocolate craving difference. In the case of chocolate or coffee, sugar/sweetener’s just serving the role of making what I’m actually craving more palatable.)
Worth trying, but I am afraid that the likely outcome would be “I consume all the unsweetened chocolate, and then still go looking for something else”. Though recently I partially substituted sweets by peanuts (peeled, unsalted), which is almost healthy… considering the likely alternatives.
Someone who eats a lot of candy bars is likely to be undereating in general, and not getting enough protein/fat/salt/micronutrients. I believe this explains most of the claimed health harms from eating sugar, as seen in observational studies.
It seems to me that we are a pretty good gear model that eating a lot of sugar leads to insulin swings that are unhealthy. Apart from honey there’s little sugar in the ancestary enviroment so it’s not surprising that the body isn’t well adapted to producing insulin in those contexts.
This tweet raised the question of whether masks really are more effective if placed on sick people (blocking outgoing droplets) or if placed on healthy people (blocking incoming droplets). Everyone in public or in a risky setting should have a mask, of course, but we still need to allocate the higher-quality vs lower-quality masks somehow. When sick people are few and are obvious, and masks are scarce, masks should obviously go on the sick people. However, COVID-19 transmission is often presymptomatic, and masks (especially lower-quality improvised masks) are not becoming less scarce over time.
If you have two people in a room and one mask, one infected and one healthy, which person should wear the mask? Thinking about the physics of liquid droplets, I think the answer is that the infected person should wear it.
A mask on a sick person prevents the creation of fomites; masks on healthy people don’t.
Outgoing particles have a larger size and shrink due to evaporation, so they’ll penetrate a mask less, given equal kinetic energy. (However, kinetic energies are not equal; they start out fast and slow down, which would favor putting the mask on the healthy person. I’m not sure how much this matters.)
Particles that stick to a mask but then un-stick lose their kinetic energy in the process, which helps if the mask is on the sick person, but doesn’t help if the mask is on the healthy person.
Overall, it seems like for a given contact-pair, a mask does more good if it’s on the sick person. However, mask quality also matters in proportion to the number of healthy-sick contacts it affects; so, upgrading the masks of all of the patients in a hospital would help more than upgrading the masks of all the workers in that hospital, but since the patients outnumber the workers, upgrading the workers’ masks probably helps more per-mask.
Wearing a surgical mask, I get the sense it tends to form more of a seal when inhaling, less when exhaling. (like a valve). If this is common, it would be a point in favour of having the healthy person wear them.
This was initially written in response to “Communicating effective altruism better—Jargon” by Rob Wiblin (Facebook link), but stands alone well and says something important. Rob argues that we should make more of an effort to use common language and avoid jargon, especially when communicating to audiences outside of your subculture.
I disagree.
If you’re writing for a particular audience and can do an editing pass, then yes, you should cut out any jargon that your audience won’t understand. A failure to communicate is a failure to communicate, and there are no excuses. For public speaking and outreach, your suggestions are good.
But I worry that people will treat your suggestions as applying in general, and trying to extinguish jargon terms from their lexicon. People have only a limited ability to code-switch. Most of the time, there’s no editing pass, and the processes of writing and thinking are comingled. The practical upshot is that people are navigating a tradeoff between using a vocabulary that’s widely understood outside of their subculture, and using the best vocabulary for thinking clearly and communicating within their subculture.
When it comes to thinking clearly, some of the jargon is load-bearing. Some of it is much more load-bearing than it looks. On the margin, people should be using jargon more.
I’m the author of Rationality Cardinality (http://carddb.rationalitycardinality.com/card/all/). The premise of the game is, I curated a collection of concepts that I thought it was important for people to be familiar with, optimized the definitions, and mixed them together with some jokes. I’ve given a lot of thought to what makes good jargon terms, and the effects that using and being immersed in jargon has on people.
I’m also a developer of LessWrong, a notoriously jargon-heavy site. We recently integrated a wiki, and made it so that if a jargon term links to the appropriate wiki page, you can hover over it for a quick definition. In the medium to long term, we hope to also have some mechanisms for getting jargon terms linked without the post author needing to do it, like having readers submit suggested linkifications, or a jargon-bot similar to what they have on the SpaceX wiki (which scans for keywords and posts a comment with definitions of all of them).
Jargon condenses ideas, but the benefit of condensation isn’t speed. Short phrases are more accessible to our thoughts, and more composeable. The price of replacing “steelmanning” with “giving the best defense of a position” is to less-often notice that steelmanning is an option, or that someone is doing it. The price of replacing “Moloch” with “coordination problems” is to stop noticing when what look like villain-shaped problems are actually coordination problems instead.
Much of our jargon is writers’ crystallized opinions about which concepts we should have available, and the jargon is the mechanism for doing so. If we reject those opinions, we will not notice what we fail to notice. We will simply see less clearly.
Appendix: A few illustrative examples from the slides
If I replaced the term “updated” with “changed my mind” in my lexicon, then I’d get tripped up whenever I wanted to tell someone my probability estimate had gone from 10% to 20%, or (worse) when I wanted to tell them my probability estimate had gone up, but didn’t want to commit to a new estimate. Ie, the power of the word “updating” is not that it’s extra precise, it’s that it’s *imprecise* in a way that’s useful.
Replacing “agenty” with “proactive and independent-minded” feels like obliterating the concept entirely, in a way that feels distinctly Orwellian. I think what’s actually going on here is that this concept requires a lot more words to communicate, but it also happens to be a concept that the villains in Orwell’s universe would actually try to erase, and this substitution would actually erase it.
Replacing “credence” with “estimate of the probability” would imply the existence of a person-independent probability to be argued over. This is a common misunderstanding, attached to a conversational trap, and this trap is enough of a problem in practice that I think I’d rather be occasionally inscrutable than lead people into it.
Now I would like to see an article that would review the jargon, find the nearest commonly used term for each term, and explain the difference the way you did (or possibly admit that there is no important difference).
The discussion so far on cost disease seems pretty inadequate, and I think a key piece that’s missing is the concept of Hollywood Accounting. Hollywood Accounting is what happens when you have something that’s extremely profitable, but which has an incentive to not be profitable on paper. The traditional example, which inspired the name, is when a movie studio signs a contract with an actor to share a percentage of profits; in that case, the studio will create subsidiaries, pay all the profits to the subsidiaries, and then declare that the studio itself (which signed the profit-sharing agreement) has no profits to give.
In the public contracting sector, you have firms signing cost-plus contracts, which are similar; the contract requires that profits don’t exceed a threshold, so they get converted into payments to de-facto-but-not-de-jure subsidiaries, favors, and other concealed forms. Sometimes this involves large dead-weight losses, but the losses are not the point, and are not the cause of the high price.
In medicine, there are occasionally articles which try to figure out where all the money is going in the US medical system; they tend to look at one piece, conclude that that piece isn’t very profitable so it can’t be responsible, and move on. I suspect this is what’s going on with the cost of clinical trials, for example; they aren’t any more expensive than they used to be, they just get allocated a share of the profits from R&D ventures that’re highly profitable overall.
they aren’t any more expensive than they used to be, they just get allocated a share of the profits from R&D ventures that’re highly profitable overall.
Did you mean “allocated a share of the costs”? If not, I am confused by that sentence.
I’m pretty uncertain how the arrangements actually work in practice, but one possible arrangement is: You have two organizations, one of which is a traditional pharmaceutical company with the patent for an untested drug, and one of which is a contract research organization. The pharma company pays the contract research organization to conduct a clinical trial, and reports the amount it paid as the cost of the trial. They have common knowledge of the chance of success, of the future probability distribution of future revenue for the drug, how much it costs to conduct the trial, and how much it costs to insure away the risks. So the amount the first company pays to the second is the costs of the trial, plus a share of the expected profit.
Pharma companies making above-market returns are subject to political attack from angry patients, but contract research organizations aren’t. So if you control both of these organizations, you would choose to allocate all of the profits to the second organization, so you can defend yourself from claims of gouging by pleading poverty.
Yesterday, I wrote a post about the Regression to the Mean Diet. The biggest impact knowing about the Regression to the Mean Diet has had for me is on my interpretations of studies, where it’s a lens that reveals what would otherwise be the best studies to be mostly useless, and of anecdotes, where it makes me heavily discount claims about a new diet working unless I’ve gotten to ask a lot of questions about the old diet, too. But there’s one other implication, which I left out of the original post, because it’s kind of unfortunate and is a little difficult to talk about.
I’m not interested in nutrition because I care about weight, or body aesthetics, or athletic performance. I care about nutrition because I believe it has a very large, very underappreciated impact on individual productivity. Low quality diets make people tired and depressed, so they don’t get anything done.
The Regression to the Mean Diet predicts that if you reroll the eating habits of someone whose diet-related health is unusually bad, then their new diet will probably be an improvement. This has a converse: if you reroll the eating habits of someone whose diet-related health is good, especially if that person is a peak performer in some way, then their new diet will be worse.
Under this model, one of the most destructive things you could do would be to identify top performers in important areas, people in good health with no nutritional problems, and convince them they need to change their diet.
Which brings me to vegan outreach within the Effective Altruism movement.
I don’t think an animal’s suffering is anywhere close to as bad as a similar amount of suffering in a human, but I do think it matters, and that this makes modern factory farming quite bad. While I have qualms about the quality of vegan diets in practice, I think that if you convince an average person from the general public to switch from an omnivorous diet they haven’t thought much about to a vegan diet with any thought at all put into it, this will on average be an improvement. I think externally-facing vegan outreach is good, and while I wouldn’t prioritize it over AI alignment or anti-aging research, I am in favor of it.
But inward-facing vegan outreach scares me. Because EA is in fact seeking out top performers in important areas, and introducing them to its memes. Under the current social equilibrium, those people feel some pressure to reduce their meat consumption, but not many make large dietary changes; most of the people who are vegetarian or vegan within EA where vegetarian or vegan beforehand. It’s easy to imagine a different equilibrium, in which the majority of omnivores who get involved in EA go vegan.
I worry that in that world, what would be the top-percentile people are no longer top percentile, and no one notices the absence or makes the connection.
These nutrition posts are great. Will there be a way for me to link to all (and only) this series, in chronological order, at some point? I want these discussed as a group on social media and the EA Forum too.
Does it solve your use case if I edit prev/next links into all of them?
(For now I’m focused on keeping a writing cadence going, and not thinking too much about publication format. There’s a decent chance that, after I’ve depleted the backlog of unpublished ideas I’ve had, I’ll do a second pass of some sort and make it more polished; but I don’t think that’s certain enough that you should count on it.)
Suppose LessWrong had a coauthor-matchmaking feature. There would be a section where you could see other peoples’ ideas for posts they want to write, and message them to schedule a collaboration session. You’d be able to post your own ideas, to get collaborators. There would be some quality-sorting mechanism so that if you’re a high-tier author, you can restrict the visibility of your seeking-collaborators message to other high-tier authors.
People who’ve written on LessWrong, and people who’ve *almost* written on LessWrong but haven’t quite gotten a post out: Would you use this feature? If so, how much of a difference do you think it would make in the quantity and quality of your writing?
I think it could be very helpful, if only for finding people to hold me to account and encourage me to write. Showing me that someone gets what I want to do, and would appreciate it.
Deep commitment to truth requires investing in the skill of nondisruptive pedantry.
Most communication contains minor errors: slightly wrong word choices, unstated assumptions, unacknowledged exceptions. By default, people interpret things in a way that smooths these out. When someone points out one of these issues in a way that’s disruptive to the flow of conversation, it’s called pedantry.
Often, someone will say something that’s incorrect, but close-enough to a true thing for you to repair it. One way you can handle this is to focus on the error. Smash the conversational context, complain about the question without answering it, that sort of thing.
A different thing you can do is to hear someone say something that’s incorrect, mentally flag it, repair it to a similar statement that matches the other person’s intent but is actually true, act as though the other person had something ambiguous (even if it was actually unambiguously wrong). Then you insert a few words of clarification, correcting the error without forcing the conversation to be about the error, and providing something to latch on to if the difference turns out to be a real disagreement rather than a pedantic thing.
And a third thing you can do is a thing where you sort of… try to do the second thing, but compressed all into one motion, where you substitute a corrected version of the sentence without noticing that you’ve repaired it, or verbally acknowledging the ambiguity.
I don’t think I’ve ever seen someone point at it explicitly, but I think this mental motion, noticing an error and fixing it without overreacting and damaging the surrounding context, may be one of the most important foundational rationality skills there is. And, it seems… actually pretty easy to practice, when you look squarely at it?
Being stupid at an expert, but for ordinary (technical) conversation, by unapologetic pervasive minor steelmanning, trusting to have resulting misunderstandings efficiently corrected later, as they break things, no fuss.
One alternative is defensively juggling nuanced uncertainty, which makes efficient thinking impossible and further communication of it cumbersome. Another is to aggressively resolve ambiguity, which pays the cost for sorting out irrelevant details, makes communication more nuanced than necessary. This stuff occasionally comes up as serious proposals for the way things ought to be.
Among people who haven’t learned probabilistic reasoning, there’s a tendency to push the (implicit) probabilities in their reasoning to the extremes; when the only categories available are “will happen”, “won’t happen”, and “might happen”, too many things end up in the will/won’t buckets.
A similar, subtler thing happens to people who haven’t learned the economics concept of elasticity. Some example (fallacious) claims of this type:
Building more highway lanes will cause more people to drive (induced demand), so building more lanes won’t fix traffic.
Building more housing will cause more people to move into the area from far away, so additional housing won’t decrease rents.
A company made X widgets, so there are X more widgets in the world than there would be otherwise.
This feels like it’s in the same reference class as the traditional logical fallacies, and that giving it a name—“zero elasticity fallacy”—might be enough to significantly reduce the rate at which people make it. But it does require a bit more concept-knowledge than most of the traditional fallacies, so, maybe not? What happens when you point this out to someone with no prior microeconomics exposure, and does logical-fallacy branding help with the explanation?
Building more highway lanes will cause more people to drive (induced demand), so building more lanes won’t fix traffic.
Is this really fallacious? I’m asking because while I don’t know the topic personally, I have some friends who are really into city planning. They’ve said that this is something which is pretty much unambiguously accepted in the literature, now that we’ve had the time to observe lots and lots of failed attempts to fix traffic by building more road capacity.
A quick Googling seemed to support this, bringing up e.g. this article which mentions that:
In this paper from the Victoria Transport Policy Institute, author Todd Litman looks at multiple studies showing a range of induced demand effects. Over the long term (three years or more), induced traffic fills all or nearly all of the new capacity. Litman also modeled the costs and benefits for a $25 million line-widening project on a hypothetical 10-kilometer stretch of highway over time. The initial benefits from congestion relief fade within a decade.
Yeah, I do agree that for the case of traffic, elasticity is pretty close to 1, which importantly doesn’t mean building more traffic is a bad idea, it’s actually indicative of demand for traffic capacity being really high, meaning marginal value of doing so is likely also really high.
I think we should be putting pretty substantial probability mass on the possibility that Omicron was the result of a successful, secret project to create a less-severe but more-contagious strain of COVID-19 in a lab, release it, and have it crowd out the original strain.
The cruxes of this belief are:
The genome of Omicron is not consistent with natural evolution, in any environment
Omicron produces substantially less severe disease than any earlier strains of COVID-19
Producing substantially less severe disease isn’t something that happens by default, if you’re manipulating a virus in a lab
If you’re already manipulating COVID-19 in a lab with all the difficulties that entails, making a less-severe variant does not add significant difficulty on top of that
If you do have a less-severe lab-grown variant of COVID-19, and you think it probably confers cross-immunity to older variants, you will probably do a moral calculation that finds it’s good to release it on purpose.
This could be done unilaterally by a small group or even a single individual, in any of a very large number of biology labs all over the world
I’m not fully confident in any of these cruxes, but consider each of them highly probable.
If this were publicly confirmed to be true, I would expect the public reaction to be strongly negative, even if the project was good on net. So I would expect anyone trying to do this would choose secrecy, would support their decision to remain secret, and would tentatively support their decision to undertake the project in the first place, depending on details related to how much less severe the variant is, how strong the cross-immunity is, and how these were verified.
(I have not seen anyone else suggest this possibility, and I think this possibility falls inside an almost-universal blind spot people have; people incorrectly assume that nothing done in secret and with a conspiratorial flavor can ever be well-intentioned and good.)
I think that hypothesis is <<1% likely because very few people care about doing good strongly enough to entertain act utilitarian master plans of this sort, and the ones who do and are action-oriented enough to maybe pull it off hopefully realize it’s a bad idea have a morality that allows this. I mean if you put resources into this specific plan, why not work on a universal coronavirus vaccine or some other more robustly beneficial thing that won’t get you and your collaborators life in jail if found out.
- Evolution from the original Wuhan strain seems less likely to generate cross immunity than taking newer strains. If someone were shooting for cross immunity, wouldn’t they use newer strains? (Assuming that you can still make them less harmful if you select for that.)
- Omicron doesn’t actually give enough cross immunity at all, and presumably that sort of thing would have been easily testable. If someone wanted to do this on purpose, they’d be complete idiots because they essentially released a second pandemic (Delta and Omicron may well co-exist, especially in countries that don’t have a lot of vaccines that will get rid of Delta quickly).
Edit: Ah, you talk about cross immunity to older variants. Maybe your theory is that the project would have happened before Delta and somehow it took longer to spread after initial release? I mean, that’s probably coherent to imagine but seems way more likely some people were messing around with rodents for more narrow (but misguided) reasons.
If you’re already manipulating COVID-19 in a lab with all the difficulties that entails, making a less-severe variant does not add significant difficulty on top of that
That seems wrong. It seems that today we have little evidence that doesn’t come from the clinical history of humans that points towards it being less severe. To know that whatever change you made makes it less severe in humans you actually need to test it in humans. Doing human testing is quite complicated. Even if you do this in some African country where you can take over some remote village to do human experimentation, that’s a lot of work and there’s potential for the NSA/CIA to get wind of such a project.
Furthermore, your thesis doesn’t explain why the spike protein has so much more mutations than the other proteins. It makes sense that the South African gain-of-function experiments that tested whether the virus can evolve around antibodies against the spike protein produce such a result but it doesn’t make sense that you would find that pattern if someone would just want to design it to be less harmful.
I would also highlight this as seemingly by far the most wrong point. Consider how many Omicron cases we now have and we still don’t know for sure it’s significantly less severe. Now consider how many secret cases in humans infected with various novel strains you’re working with you would need to enact in a controlled environment to be confident enough that a given strain is less severe and thus it makes sense to release it.
An epidemiologist once told me it is common knowledge among epidemiologists that immunity to a given variant of the common cold is not very long, perhaps a year. I have not been able to easily find a link demonstrating this though. If this is true it would ruin the moral calculation.
Not to me. Though I don’t have domain knowledge here. All I really have to say is that these people that do have domain knowledge see a path for natural evolution. I don’t mean to say that this demonstrates the evolution was natural, just that human intervention was not required.
Vitamin D reduces the severity of COVID-19, with a very large effect size, in an RCT.
Vitamin D has a history of weird health claims around it failing to hold up in RCTs (this SSC post has a decent overview). But, suppose the mechanism of vitamin D is primarily immunological. This has a surprising implication:
It means negative results in RCTs of vitamin D are not trustworthy.
There are many health conditions where having had a particular infection, especially a severe case of that infection, is a major risk factor. For example, 90% of cases of cervical cancer are caused by HPV infection. There are many known infection-disease pairs like this (albeit usually with smaller effect size), and presumably also many unknown infection-disease pairs like this as well.
Now suppose vitamin D makes you resistant to getting a severe case of a particular infection, which increases risk of a cancer at some delay. Researchers do an RCT of vitamin D for prevention of that kind of cancer, and their methodology is perfect. Problem: What if that infection wasn’t common in at the time and place the RCT was performed, but is common somewhere else? Then the study will give a negative result.
This throws a wrench into the usual epistemic strategies around vitamin D, and around every other drug and supplement where the primary mechanism of action is immune-mediated.
Sounds like a very general criticism that would apply to any effects that are very strong/consistent in circumstances where there a very high variance (e.g. binary) latent variable takes on a certain variable (and the effect is 0 otherwise...).
Prediction: H3N8 will not be a pandemic. H3N8 is a genuine zoonotic transmission, and diseases which actually came from zoonotic transmission don’t transmit well. COVID exploded rapidly because it was a lab escape, not a zoonotic transmission, and didn’t have this property. The combination of poor initial transmission with an environment that’s big on respiratory precautions in general, is more than sufficient to keep it from getting a foothold.
What those drug-abuse education programs we all went though should have said:
It is a mistake to take any drug until after you’ve read its wikipedia page, especially the mechanism, side effects, and interactions sections, and its Erowid page, if applicable. All you children on ritalin right now, your homework is to go catch up on your required reading and reflect upon your mistake. Dismissed.
(Not a vagueblog of anything recent, but sometimes when I hear about peoples’ recreational-drug or medication choices, I feel like Quirrell in HPMOR chapter 26, discussing a student who cast a high-level curse without knowing what it did.)
One question I sometimes see people asking is, if AGI is so close, where are the self-driving cars? I think the answer is much simpler, and much stupider, than you’d think.
Waymo is operating self-driving robotaxis in SF and a few other select cities, without safety drivers. They use LIDAR, so instead of the cognitive task of driving as a human would solve it, they have substituted the easier task “driving but your eyes are laser rangefinders”.
Tesla also has self-driving, but it isn’t reliable enough to work without close human oversight. Until less than a month ago, they were using 1.2 megapixel black and white cameras. So instead of the cognitive task of driving as a human would solve it, they substituted the harder task “driving with a vision impairment and no glasses”.
If my understanding is correct, this means that Tesla’s struggle to get neural nets to drive was probably not a problem with the neural nets, and doesn’t tell us much of anything about the state of AI.
My answer to this is quite different. The paradigm that is currently getting very close to AGI is basically having a single end-to-end trained system with tons of supervised learning.
Self-driving car AI is not actually operating in this current paradigm as far as I can tell, but is operating much more in the previous paradigm of “build lots of special-purpose AI modules that you combine with the use of lots of special-case heuristics”. My sense is a lot of this is historical momentum, but also a lot of it is that you just really want your self-driving AI to be extremely reliable, so training it end-to-end is very scary.
I have outstanding bets that human self-driving performance will be achieved when people switch towards a more end-to-end trained approach without tons of custom heuristics and code.
My understanding is that they used to have a lot more special-purpose modules than they do now, but their “occupancy network” architecture has replaced a bunch of them. So they have one big end-to-end network doing most of the vision, which hands a volumetric representation over to the collection of special-purpose-smaller-modules for path planning. But path planning is the easier part (easier to generate synthetic data for, easier to detect if something is going wrong beforehand and send a take-over alarm.).
I don’t have anything like a complete analysis of what’s happening with Russia’s invasion of Ukraine. But I do have one important fragment, which is a piece I haven’t seen elsewhere:
For the past decade, Russia under Putin has been pushing hard against the limits of what it can get away with in the realm of spycraft. There are a lot of different bad things Russia was doing, and if you look at any one of them, the situation looks similar: they inflicted some harm on a Western country, but it’s not quite possible to do anything about it. Some of the major categories are election interference, assassinations, and disinformation campaigns. These produced a lot of grudges in a lot of places, especially in the military and covert worlds.
Dealing with a powerful bad actor is a coordination problem. If one country presses hard on a scandal, or imposes sanctions, then Russia can undermine the response and retaliate. If a lot of countries act at once, then they’re all mostly safe from retaliation.
Russia’s invasion of Ukraine provided a coordination point and a ready-made justification for everyone with a grudge to strike at Russia. I think we’re probably going to see conflict between Western powers and Russia open up on more fronts, some of which will be unexpected, and some of which will not be initiated by Russia.
I’ve been writing a series of posts about nutrition, trying to consistently produce one post per day. The post I had in mind for today grew in scope by enough that I can’t finish it in time, so this seems like an opportune day for a meta-post about the series.
My goal, in thinking and writing about nutrition, is to get the field unstuck. This means I’m interested in solving the central mysteries, and in calling attention to blind spots. I’m primarily writing for a sophisticated audience, and I’m making little to no attempt to cover the basics. I’m not going to do the sort of literature review that goes through all the vitamins and minerals in order, saying approximately the same things Wikipedia says about each of them. There are enough of those out there already. If you’re just trying to figure out how to lose weight, then these posts will be interesting, they will probably give you a perspective that makes evaluating other sources a lot easier, but my posts will not be optimized for solving your problem directly.
I have the reliability-vs-generativity-tradeoff slider set all the way to “generativity”. It would be very surprising if I finished this post series without saying anything untrue. I will not repeat this epistemic status on every post, but this epistemic status does apply to all of them.
Obesity and weight loss will come up a lot in my writing, because the obesity epidemic is this big conspicuous mystery that a lot of people have studied a lot about, and it’s somewhat central and connected to other subtics within nutrition. But it’s not really what I care about, except insofar as it affects productivity and general health.
I haven’t been putting content warnings at the top of my posts. I’m going to start.
There’s the obvious content warning, which is that some people with certain classes of eating disorders don’t want to read about food or nutrition in general, or only want to read about it when they’re at their best, because thinking about the topic makes them stress about the topic which makes them do dumb things. I think that the particular ideas I have to present are probably net-good for most such people, but they probably want to make a conscious choice about whether and when to read my posts, and I don’t want them to have to unfollow me.
The second warning is that I’m making little to no effort to cover the basics, and by that I mean I’m not going to reliably provide the warnings-away-from-spectacular-failures that mainstream nutrition advice focuses on. If I imagine my posts in a grocery-store checkout aisle magazine, being read by average people, I think some of those people might die. So, watch out. If you manage to give yourself scurvy, this will be your own fault, and I will call you a scallywag.
Last piece of meta: I’m posting these as I go with minimal editing, but there will probably be a more polished second-pass version of some sort in the future. If you’re curious about my nutrition thoughts but feel no urgency, then it might be worth waiting for it.
One of the reasons I worry about cybersecurity, and the sorry state it’s in, is that it provides an easy path for human-level and even infrahuman-level AIs to acquire additional computation. In some plausible worlds, this turns a manageable infrahuman AI into an unmanageable superintelligence, when the creator’s decision would have been not to launch.
Unlike solving protein-design and constructing nanobots, this is something definitely within reach of human-level intelligence; many people have done it for ordinary criminal purposes, like mining cryptocurrency.
Current cybersecurity research incentives don’t seem quite right for getting people to mitigate this risk. If I were trying to stop an AI from taking over the internet and using the extra hardware to make itself superintelligent, I would worry a lot less about protecting user data, phones and whatnot, and worry a lot more about protecting software developer workstations in particular.
Am incline to agree, but I want to add that security is all connected. There are several direct causal paths from compromised user data to compromised dev workstation (and vice versa).
It’s looking likely that the pandemic will de facto end on the Summer Solstice.
Biden promised vaccine availability for everyone on May 1st. May 1st plus two weeks to get appointments plus four weeks spacing between two doses of Moderna plus one week waiting for full effectiveness, is June 19. The astronomical solstice is June 20, which is a Sunday.
Things might not go to plan, if the May 1st vaccine-availability deadline is missed, or a vaccine-evading strain means we have to wait for a booster. No one’s organizing the details yet, as far as I know. But with all those caveats aside:
My understanding was that the May 1st date was “Everyone’s now allowed to sign up for an appointment, but you may be at the end of a long queue.” How long after that do you think it will take to get a vaccine to everyone who wants one?
Currently, 2.4 million shots/day. Note that it’s a situation where it’s always going to be limited by the rate limiting step, and there are many bottlenecks, so using the ‘current’ data and extrapolating only a modest increase is the most conservative estimate.
210 million adults. Only 0.7 need to be vaccinated for the risk to plummet for everyone else. A quick bit of napkin math says we need 294 million doses to fully vaccinate everyone, and we are at 52 million now. (294-52) = 242million/2.4 = 100.8 more days.
This is why the lesser J&J vaccine is actually so useful—if we switched all the vaccine clinics and syringe supplies to J&J overnight (if there was enough supply of the vaccine itself) suddenly we only need 121 million doses to vaccinate everyone, or 50.4 more days.
The reality is that increasing efforts are probably going to help, and the J&J is helping, but sooner or later a bottleneck will be hit that can’t be bypassed quickly (like a syringe shortage), so I would predict the reality number of days to fall in that (50, 100) day interval.
There are 94 days between now and June 19. Also, a certain percentage of the population are going to refuse the shot in order to be contrarian or because they earnestly believe their aunt’s facebook rants. Morever, the ‘get an appointment’ game means the tech savvy/people who read reddit get an advantage over folks who aren’t.
So for those of us reading this who don’t yet qualify, it doesn’t appear that it will be much longer.
Twitter is an unusually angry place. One reason is that the length limit makes people favor punchiness over tact. A less well-known reason is that in addition to notifying you when people like your own tweets, it gives a lot of notifications for people liking replies to you. So if someone replies to disagree, you will get a slow drip of reminders, which will make you feel indignant.
LessWrong is a relatively calm place, because we do the opposite: under default settings, we batch upvote/karma-change notifications together to only one notification per day, to avoid encouraging obsessive-refresh spirals.
I also thing there’s less engagement on LW.* While it might depends on the part of twitter, there’s a lot more replies going on. Sometimes it seems like there’s a 100 replies to a tweet, in contrast to posts with zero comments. This necessarily means replies will overlap a lot more than they do on LW. Imagine getting 3 distinct comments to a short post on LW, versus a thread of tweets, with 30 responses that mostly boil down to the same 3 responses that are being sent because people are responding without seeing other responses. (And if there’s hundreds of very similar responses, asking people to read responses is asking people to read a very boring epic.)
And getting one critical reply, versus the same critical reply from 10 people, even when it’s the same fraction of responses, probably affects people differently—if only because it’s annoying to see the same message over and over again.
*This could be the case (the medium probably helps) even if that engagement was all positive.
Some software costs money. Some software is free. Some software is free, with an upsell that you might or might not pay for. And some software has a negative price: not only do you not pay for it, but someone third party is paid to try to get you to install it, often on a per-install basis. Common examples include:
Unrelated software that comes bundled with software you’re installing, which you have to notice and opt out of
Software advertised in banner ads and search engine result pages
CDs added to the packages of non-software products
This category of software is frequently harmful, but I’ve never seen the it called out by the economic definition. For laypeople, about 30% of computer security is recognizing the telltale signs of this category of software, and refusing to install it.
I wonder what would be a non-software analogy of this.
Perhaps those tiny packages with labels “throw away, do not eat” you find in some products. That is, in a parallel world where 99% of customers would actually eat them anyway. But even there it isn’t obvious how the producer would profit from them eating the thing. So, no good analogy.
I’m trying to wrap my head around the negative price distinction. A business can’t be viable if the cost of user acquisition is lower than the lifetime value of a user.
Most software spend money on advertising, then they have to make that money back somehow. In a direct business model, they’ll charge the users of the software directly. In an indirect business model, they’ll charge a third party for access to the users or an asset that the user has. Facebook is more of an indirect business model, where they charge advertisers for access to the users’ attention and data.
In my mind, the above is totally fine. I choose to pay with my attention and data as a user, and know that it will be sold to advertisers. Viewing this as “negatively priced” feels like a convoluted way to understand the business model however.
Some malware makes money by trying to hide the secondary market they’re selling. For instance, by sneaking in a default browser search that sells your attention to advertisers, or selling your computers idle time to a botnet without your permission. This is egregious in my opinion, but it’s not the indirect business model that is bad here, it’s the hidden costs that they lie about or obfuscate.
User acquisition costs are another frame for approximately the same heuristic. If software has ads in an expected place, and is selling data you expect them to sell, then you can model that as part of the cost. If, after accounting for all the costs, it looks like the software’s creator is spending more on user acquisition than they should be getting back, it implies that there’s another revenue stream you aren’t seeing, and the fact that it’s hidden from you implies that you probably wouldn’t approve of it.
A: We both know that you don’t know of any specific chicken having crossed any specific road. Your question does not state a lie, but presupposes it. This would not be called out as a lie under ordinary social convention, but a deep commitment to truth requires occasionally flagging things like this.
Presuppositions are things which aren’t stated directly, but which are implied by an utterance because if they weren’t true, the utterance would be nonsensical. Presuppositions that aren’t truth-optimized can be surprisingly pernicious.
I have not only told you a joke, I have also told you that chickens live in contexts where they can interact with roads, are motile, and are capable of having motivations. I have told you these things in a way that bypasses some of your epistemic defenses. Some of these things are true, and some are false. I didn’t tell you these things on purpose, but they were communicated nevertheless.
I think that explicitly unpacking presuppositions, and spotting the questionable ones, is a foundational skill of rationality that I’ve never seen expressed. I also suspect some other techniques might be warped versions of this one. For example, there’s a lot of overlap between the ontology of frames, and what you get if you unpack presuppositions, but presupposition-truth is much more concrete.
Fighting presuppositions instead of letting them develop in dedicated sandboxes hinders their understanding, makes communication unnecessarily difficult. The false dichotomy is between belief and dismissal of falsehood. There is also understanding of apparent falsehoods, which produces valuable gears that reassemble into unexpected truths.
Our past beliefs affect what we pay attention to, how we prioritize our skepticism, and how we interpret ambiguous evidence. This can create belief basins, where there are multiple sets of beliefs that reinforce each other, appear internally consistent, and make it hard to see the other basins as valid possibilities. On the topic of nutrition, I seem to have found myself in a different basin. I’ve looked through every nonstandard lens I could find, repeatedly applied skepticism, and firmly committed to not make the same mistakes everyone else is making (as a priority on par with not making mistakes at all). I’ve arrived at a set of beliefs that, as far as I can tell, is internally consistent, reasonably compelling from the inside, and completely contrary to what most other people in our culture think.
This makes for a difficult writing project. When I try to argue nonstandard positions, many of the arguments are tendrils reaching into other nonstandard positions. I’ve finally managed to get into a post-every-day cadence; part of the key to that was accepting that sometimes those arguments will be dangling references. Hopefully after a month of this, the whole thing will cohere. If not, well, the fragments are pretty interesting too.
The most-common basin of nutrition theorizing centers on obesity, and on a particular theory of obesity which goes like this. Food, especially modern processed food, tastes good and is appealing. Some people, if they followed their urges, would eat too much or become obese, so they have to exert self control not to. Weight is a function of calorie intake and calorie expenditure (“calories in, calories out”), and expenditure is primarily a function of behavior. So if someone is trying to lose weight, and it isn’t working, then they must be having willpower failures and eating more than they intend, or exercising less than they intend.
I currently think there are quite a lot of things wrong with this model, but today, I’d like to focus on one in particular. It’s not the only or the most central objection, nor is it a particularly actionable model fragment. But it’s an issue that’s important to me in particular, and it’s one of the wedges that moved into a different belief basin.
I am not obese, and have never set out to lose weight. But sometimes, I have overwhelming cravings for sugar. I would not be able to resist these cravings without great willpower.
If I ever did successfully resist one of those cravings, I would probably die.
I don’t mean this figuratively, or in a heart-disease-years-later sort of way. I mean that if I get a powerful craving for sugar, and I don’t promptly eat something that has sugar in it, then this will be a life-threatening medical emergency. This is because I have type 1 diabetes, and craving sugar is a symptom of low blood sugar, aka hypoglycemia. What T1 diabetes means, basically, is that I have to micromanage my blood sugar using insulin. Eating carbohydrates raises blood sugar, insulin lowers it, these need to be matched pretty precisely, and the whole thing is somewhat error prone. Too much insulin and blood sugar falls below 70mg/dL, and I get the sugar craving. I’ve never been below 40mg/dL, but people who do become mentally impaired, then lose consciousness, then die.
Under the usual theory of obesity, craving sugar would mean that I had been hijacked by the superstimulus of processed food, and that willpower was my defense against this hijacking. But actually, in this case, the craving is a safety mechanism. Sugar craving is to dangerous hypoglycemia as thirst is to dehydration.
With that example in mind, I started thinking about the double-digit percentage of people who drop out of weight-loss studies. And the much-larger percentage of people who start weight loss diets, privately resolved to continue until they reach a target weight, and stop early. What would happen to them, in the counterfactual world where the diet was enforced perfectly from outside, and willpower wasn’t an issue? Whether they would lose weight, seems like very much the wrong question to ask.
You make a good point, that some people who drop out of weight-loss studies might have experienced health problems caused by the study, and quiting was the right decision for them.
But I believe that the average obese person in general population is not this case. There are many situations where people eat refined sugar not because they have a strong craving, but simply because it is easily available or there are even habits built around it.
To give an example, in my family it was for some reason considered a good idea to drink tea with sugar at breakfast. As a child I didn’t have an opinion on this, I was given the breakfast and I consumed it. But as I grew up and started making my own breakfast, out of sheer laziness I starting drinking water instead. I didn’t fall into coma and die. Actually it made the breakfast better, because when you drink tea with sugar first, then everything you eat afterwards tastes bland, but if you drink water, you discover that some things are surprisingly delicious. Recently my kids spent one week with my mother, and then reported to me that they had “cereals” for each breakfast (in this context, “cereals” refers to those cheap hypermarket products that contain the word on the box, but consist mostly of refined sugar with some added fibers; the advertisement tells you to pour milk on them and pretend that the result is healthy somehow, because, you know, milk and cereals). I am not making a big deal out of it, one week is not going to hurt anyone, but sigh, of course most people in my family are fat.
Similarly, if you buy things in a hypermarket, check how many of them contain added sugar. So people eat this sugar not because they had a craving, but because they bought a processed food in a shop, and someone added the sugar for them. (There is often no easily available sugar-less version.) They probably add sugar to your food in a restaurant, dunno.
If you are curious what would it be like to not eat any refined sugar, probably the only solution is to cook for yourself from scratch. Even things like mustard or canned vegetables typically contain refined sugar. So we regularly eat lots of sugar without deciding to, often without being aware of it. (And then we drink coke on the top of it. But hey, the advertisement said that coke had zero sugar now, could they possibly be lying?)
So, avoiding lots of extra sugar is technically possible, but it is a lot of work, and some people cannot afford it, or have never learned the necessary skills. Because of course they don’t teach cooking at schools anymore; why would anyone need such useless skill in the modern economy, where you can buy anything (but have little control over the content).
Lack-of-adblock is a huge mistake. On top of the obvious drain on attention, slower loading times everywhere, and surveillance, ads are also one of the top mechanisms by which computers get malware.
When I look over someone’s shoulder and see ads, I assume they were similarly careless in their choice of which books to read.
As usual, I use Google Surveys to run a weighted population survey. On 2019-03-16, I launched a n = 1000 one-question survey of all Americans with randomly reversed order, with the following results: […]
… I am however shocked by the percentage claiming to not know what an adblocker is: 72%! I had expected to get something more like 10–30%. As one learns reading surveys, a decent fraction of every population struggles with basic questions like whether the Earth goes around the Sun or vice-versa, so I would be shocked if they knew of ad blockers but I expected the remaining 50%, who are driving this puzzle of “why advertising avoidance but not adblock installation?”, to be a little more on the ball, and be aware of ad blockers but have some other reason to not install them (if only myopic laziness).
But that appears to not be the case. There are relatively few people who claim to be aware of ad blockers but not be using them, and those might just be mobile users whose browsers (specifically, Chrome, as Apple’s Safari/iOS permitted adblock extensions in 2015), forbid ad blockers.
(I highly recommend reading that entire section of the linked page, where gwern describes the results of several follow-up surveys he ran, and conclusions drawn from them.)
Background context is that I participated in building the software platform behind the letter, without a specific open letter in hand. It has mechanisms for sorting noteworthy signatures to the top, and validating signatures for authenticity. I expect there to be other open letters in the future, and I think this is an important piece of civilizational infrastructure.
I think the world having access to deepfakes, and deepfake-porn technology in particular, is net bad. However, the stakes are small compared to the upcoming stakes with superintelligence, which has a high probability of killing literally everyone.
If translated into legislation, I think what this does is put turnkey-hosted deepfake porn generation, as well as pre-tuned-for-porn model weights, into a place very similar to where piracy is today. Which is to say: The Pirate Bay is illegal, wget is not, and the legal distinction is the advertised purpose.
(Where non-porn deepfakes are concerned, I expect them to try a bit harder at watermarking, still fail, and successfully defend themselves legally on the basis that they tried.)
The analogy to piracy goes a little further. If laws are passed, deepfakes will be a little less prevalent than they would otherwise be, there won’t be above-board businesses around it… and there will still be lots of it. I don’t think there-being-lots-of-it can be prevented by any feasible means. The benefit of this will be the creation of common knowledge that the US federal government’s current toolkit is not capable of holding back AI development and access, even when it wants to.
I would much rather they learn that now, when there’s still a nonzero chance of building regulatory tools that would function, rather than later.
I’m reading you to be saying that you think on its overt purpose this policy is bad, but ineffective, and the covert reason of testing the ability of the US federal government to regulate AI is worth the information cost of a bad policy.
I definitely appreciate that someone signing this writes this reasoning publicly. I think it’s not crazy to think that it will be good to happen. I feel like it’s a bit disingenuous to sign the letter for this reason, but I’m not certain.
I’m reading you to be saying that you think on its overt purpose this policy is bad, but ineffective, and the covert reason of testing the ability of the US federal government to regulate AI is worth the information cost of a bad policy.
I think preventing the existence of deceptive deepfakes would be quite good (if it would work); audio/video recording has done wonders for accountability in all sorts of contexts, and it’s going to be terrible to suddenly have every recording subjected to reasonable doubt. I think preventing the existence of AI-generated fictional-character-only child pornography is neutral-ish (I’m uncertain of the sign of its effect on rates of actual child abuse).
Some people have a sense of humor. Some people pretend to be using humor, to give plausible deniability to their cruelty. On April 1st, the former group becomes active, and the latter group goes quiet.
This is too noisy to use for judging individuals, but it seems to work reasonably well for evaluating groups and cultures. Humor-as-humor and humor-as-cover weren’t all that difficult to tell apart in the first place, but I imagine a certain sort of confused person could be pointed at this in order to make the distinction salient.
Someone complained, in a meme, that tech companies building AI are targeting the wrong tasks: writing books, music, TV, but not the office drudge work, leading to a world in which the meaning-making creative pursuits are lost to humans. My reply to this is:
The order in which AI replaces jobs is discovered, not chosen. The problem is that most of the resources aren’t going into “AI for writing books” or “automating cubicle jobs”, they’re going into more-abstract targets like “scaling transformers” and “collecting data sets”.
How these abstract targets cash out into concrete tasks isn’t easy to predict in advance, and, for AI accelerationists, doesn’t offer many relevant degrees of freedom.
And, to the extent that money does go into these tasks per se, I’d bet that the spending is extremely imbalanced in the opposite way to what they assume: I’d bet way more money gets spent on tabular learning, ‘robotic process automation’, spreadsheet tooling, and so on than gets spent on Jukebox-like full music generation. (Certainly I skim a lot more of the former on Arxiv.) It’s telling that the big new music generation thing, almost 3 years after Jukebox is… someone jankily finetuning Stable Diffusion on ‘images’ of music lol. Not exactly what one would call an activefield of research.
So there is a relevant degree of freedom where you can ~A C C E L E R A T E~ - it’s just the wrong one from what they want.
Companies deploying AI to do their office work would be poised to make them take aligment in a very serious way. Office work being wrong in a slight but significant way could be easy to imagine the relevance off, hard to detect and possibly nightmare to recover from.
It’s often said that in languages, the syllable-count of words eventually converges to something based on the frequency with which words are used, so that more-commonly-used concepts get words with fewer syllables.
There’s an important caveat to this, which I have never seen stated anywhere: the effect is strongly weighted towards vocabulary used by children, especially small children. Hence why “ma”, the lowest-entropy word, means mother in so many different languages, and why toddler-concepts are all monosyllables or twice-repeated monosyllables. So, for example. “probability” being 5 syllables isn’t just a reflection of it being used, it’s also a reflection of an (imo bad) societal judgement that people won’t be ready for that word until they’re adults.
I don’t remember where, but I did see this stated previously, because it’s not new to me.
“probability” being 5 syllables isn’t just a reflection of it being used, it’s also a reflection of an (imo bad) societal judgement that people won’t be ready for that word until they’re adults.
It’s not a societal judgement that kids aren’t ready for that word (though perhaps that too), but that it’s not necessary for them to survive. And, well, that seems to be true.
There is a rumor of RSA being broken. By which I mean something that looks like a strange hoax made it to the front on Hacker News. Someone uploaded a publicly available WIP paper on integer factorization algorithms by Claus Peter Schnorr to the Cryptology ePrint Archive, with the abstract modified to insert the text “This destroyes the RSA cryptosystem.” (Misspelled.)
Today is not the Recurring Internet Security Meltdown Day. That happens once every month or two, but not today in particular.
But this is a good opportunity to point out a non-obvious best practice around cryptographic key-sizes, which is this: Whatever key size is accepted as the standard, you want your SSH keys and your PGP keys to be one size bigger, so that if a gradually rising tide of mathematical advances causes a cryptography meltdown, you won’t be caught in the wave where everyone else gets pwned at once.
So I recommend making sure, if you’re using RSA for your SSH keys, that they are 4096-bit (as opposed to the current ssh-keygen default of 3072-bit).
While this sounds cool, what sort of activities are you thinking you need to encrypt? Consider the mechanisms for how information leaks.
a. Are you planning or coordinating illegal acts? The way you get caught is one of your co-conspirators reported you.
b. Are you protecting your credit card and other financial info? The way it leaks is a third party handler, not your own machine.
c. Protecting trade secrets? The way it gets leaked is one of your coworkers copied the info and brought it to a competitor.
d. Protecting crypto? Use an offline wallet. Too much protection and you will have the opposite problem.
Countless people—probably a substantial fraction of the entire population, maybe the majority—all their credit and identity records were leaked in various breaches. They have easily hackable webcams exposed on the internet. Skimmers trap their credit card periodically. And...nothing major happens to them.
COVID variants have mutated in the direction of faster spread and less immunity, as expected. They also seem to be mutating to higher disease severity, which was not expected. Why would that be, and should we expect this to continue?
My current theory is that the reason variants are more severe is because there’s evolutionary pressure on a common factor that affects both severity and secondary attack rate, and that factor is viral replication rate.
In the initial stage of an infection, the number of virus-copies inside someone grows exponentially. If the spike protein mutates to get slightly more reliable, then the doubling time of that growth shrinks. Result: Increased viral shedding, decreased minimum infectious dose, increased severity.
I hypothesize that we’re used to respiratory diseases evolving towards lower severity because, first, they’ve already maxed out this particular axis, and second, because they rely on people walking around in public while symptomatic, which they won’t do if the disease is too severe.
This retrodicts that data pretty well, but I’m still really unsure whether severity in the future is going to rise or fall. One one hand: there’s probably still room left to increase replication rate further, and the people-walking-around-if-mild transmission method isn’t going to come back. On the other hand, COVID might reach (or already be in) a local optimum with respect to spike-protein efficiency, in which case selection for immune-evasion will mean evolving away from the most-efficient version.
Looks like there’s holiday-design discourse this week: https://astralcodexten.substack.com/p/a-columbian-exchange . Speaking as a veteran holiday designer (http://petrovday.com/), in my eyes, Columbus Day has already passed into the ranks of deprecated holidays. Not so much because Christopher Columbus was a bad person (though he was by all accounts quite terrible), but rather because no one has actually designed a rationality-culture version of it, and I find broad-American-culture holidays to be boring and uncompetitive.
Looking at Scott’s list figures who could be given holidays, I think Edison and Salk have promise, and I would be enthusiastic about a good reinvention of MLK day which was updated to the present day and had a good nuanced take on things.
But I really hate the idea of giving Henrietta Lacks a holiday. Not because cancer research isn’t important. Not because I think giving a cell culture is insufficiently virtuous, compared to the research itself. Rather, I think Henrietta Lacks should not be honored because I hold a grudge against HeLa, the microorganism that descends from her cancer. It is most famous not for its positive contributions to cancer research, but for invading and ruining cancer research on other cell lines. I don’t think this kind of Petri-dish-invasion imperialism should be celebrated!
Every so often, I post to remind everyone when it’s time for the Periodic Internet Security Meltdown. For the sake of balance, I would like to report that, in my assessment, the current high-profile vulnerability Hertzbleed is interesting but does *not* constitute a Periodic Internet Security Meltdown.
Hertzbleed starts with the discovery that on certain x86-64 processors the bitwise left shift instruction uses a data-dependent amount of energy. Searching through a large set of cryptographic algorithms, they then find that SIKE (a cryptographic algorithm not in widespread use) has a data-dependent degenerate case in which a series of intermediate states are all zeroes, does some cryptanalysis, and turns this into a chosen-plaintext attack which creates a causal connection between the private key and the CPU’s throttling level.
This is pretty neat, and there may be similar attacks against other cryptographic algorithms, but I think it’s not going to amount to much in actual practice, because it has a constant-factors problem: it needs to heat up the target CPU and let it cool back down, and it only gets a tiny fraction of a bit of the private key each time. I haven’t done the analysis, but my expectation is that in more common situations (ie not SIKE), the amount of traffic required to extract a full key is going to be literally astronomical.
It’s yet another example of how infuriating computer security is, especially side-channel attacks. All that work into constant-time crypto, and then this… As the saying goes: “constants aren’t.”
On October 26, 2020, I submitted a security vulnerability report to the Facebook bug bounty program. The submission was rejected as a duplicate. As of today (April 14), it is still not fixed. I just resubmitted, since it seems to have fallen through the cracks or something. However, I consider all my responsible disclosure responsibilities to be discharged.
Once an Oculus Quest or Oculus Quest 2 is logged in to a Facebook account, its login can’t be revoked. There is login-token revocation UI in Facebook’s Settings>Security and Login menu, but changing the account password and revoking the login there does not work.
One practical impact of this is that if your Facebook account is ever compromised, and the attacker uses this vulnerability, they have permanent access.
The other practical impact is that if someone has unsupervised access to your unlocked Quest headset, and they use the built-in web browser to go to facebook.com, they have full access to your Facebook account, including Messenger, without having to do anything special at all. This means that if you’ve ever made a confidentiality agreement regarding something you discussed on Facebook Messenger, you probably can’t lend your headset to anyone, ever.
Additionally, the lock-screen on the Oculus Quest 2 does not have a strict enough rate limit; it gives unlimited tries at 2/minute, so trying all lock-screen combinations takes approximately 35 days. This can be done without network access, and can be automated with some effort. So if someone steals a *locked* Oculus Quest 2, they can also use that to break into your Facebook account. There is almost certainly a much faster way to do this involving disassembling the device, but this is bad enough.
Is your logic that releasing this heinous volun into the public is more likely to pressure FB to do something about this? Because if so, I’m not sure that LW is a forum with enough public spotlight to generate pressure. OTOH, I imagine some percentage of readers here aren’t well-aligned but are looking for informational edge, in which case it’s possible this does more harm than good?
I’m not super-confident in this model—eg, it also seems entirely possible to me that lots of FB security engineers read the site and one or more will be shouting ZOMG! any moment over this..
I’m posting here (cross-posted with my FB wall and Twitter) mostly to vent about it, and to warn people that sharing VR headsets has infosec implications they may not have been aware of. I don’t think this comment will have much effect on Facebook’s actions.
The Diamond Princess cohort has 705 positive cases, of which 4 are dead and 36 serious or critical. In China, the reported ratio of serious/critical cases to deaths is about 10:1, so figure there will be 3.6 more deaths. From this we can estimate a case fatality rate of 7.6/705 ~= 1%. Adjust upward to account for cases that have not yet progressed from detection to serious, and downward to account for the fact that the demographics of cruise ships skew older. There are unlikely to be any undetected cases in this cohort.
You’re right, adding deaths+.1*serious the way I did seems incorrect. But, since not all of the serious cases have recovered yet, that would seem to imply that the serious:deaths ratio is worse in the Diamond Princess than it is in China, which would be pretty strange. It’s not clear to me that the number of serious cases is as up to date as the number of positive tests.
How many passengers were exposed? Capacity of 2670, I haven’t seen (and haven’t looked that hard) how many actual passengers and crew were aboard when the quarantine started. So maybe over 1⁄4 of exposed became positive, 6% of that positive become serious, and 10% of that fatal.
Assuming it escapes quarantine and most of us are exposed at some point, that leads to an estimate of 0.0015 (call it 1⁄6 of 1%) of fatality. Recent annual deaths are 7.7 per 1000, so best guess is this adds 20%, assuming all deaths happen in the first year and any mitigations we come up with don’t change the rate by much. I don’t want to downplay 11.5 million deaths, but I also don’t want to overreact (and in fact, I don’t know how to overreact usefully).
I’d love to know how many of the serious cases have remaining disability. Duration and impact of survival cases could easily be the differences between unpleasantness and disruption that doubles the death rate, and societal collapse that kills 10x or more as the disease directly.
(This is a reply to the “Induction Bump” Phase Change video by Catherine Olsson and the rest of Anthropic. I’m writing it here instead of as a YouTube comment because YouTube comments aren’t a good place for discussion.)
(Epistemic status: Speculative musings I had while following along, which might be useful for inspiring future experiments, or surfacing my own misunderstandings, and possibly duplicating ideas found in prior work which I have not surveyed.)
The change-in-loss sample after the bump (at 19:10) surprised me. As you say, it seemed to get noticeably better at things that correspond to bigram rules (or bigrammish rules). I was expecting the improvement to be in something that, in some sense, requires metalearning; whereas (if I’m understanding correctly), a depth-1 network would be able to start learning these particular bigrams starting on the very first training step. If this does correspond to learning bigrams, what could explain the delay before learning these particular bigrams starts?
My speculative guess is that there’s a pattern where some learnable features have stronger gradients than others, and that training a multi-layer (but not a single layer) network proceeds in stages where first (a) multiple layer-1 nodes race to learn a function (loss falls rapidly), then (b) layer-2 nodes figure out which layer-1 nodes won the race and sever their connections to the ones that didn’t (loss falls slowly), and then (c) the freed-up layer-1 neurons move on to learning something else (loss falls rapidly again). Under this hypothesis, “the bump” corresponds to part of this sequence.
This seems to match what we see in the graph at 30:20. In that graph (which is training-step vs induction score for selected nodes in layer 1), some of the nodes go to 1, and some of the nodes start going to 1, then reverse direction shortly after the first set of nodes hits their asymptote. This is what you’d expect to see if the nodes that reverse direction were headed towards being duplicates of nodes that didn’t, but which lost the race and then got repurposed.
This seems like it might be a useful microcosm for studying the behavior of redundant nodes in general. (We know large networks have a lot of redundancy because SqueezeNets work). One major worry about neural-net transparency is that we might end up in a state where we can inspect neural nets and find things that are there, but that the methods that do that won’t be able to assert that things *aren’t* there. Concretely, this might look like finding nodes that track an unwanted concept, pinning or ablating those nodes, and then finding that the concept still presented due via duplicate nodes or via an off-basis aggregation of nodes that seem unrelated.
There’s been a lot of previous interest in indoor CO2 in the rationality community, including an (unsuccessful) CO2 stripper project, some research summaries and self experiments. The results are confusing, I suspect some of the older research might be fake. But I noticed something that has greatly changed how I think about CO2 in relation to cognition.
Exhaled air is about 50kPPM CO2. Outdoor air is about 400ppm; indoor air ranges from 500 to 1500ppm depending on ventilation. Since exhaled air has CO2 about two orders of magnitude larger than the variance in room CO2, if even a small percentage of inhaled air is reinhalation of exhaled air, this will have a significantly larger effect than changes in ventilation. I’m having trouble finding a straight answer about what percentage of inhaled air is rebreathed (other than in the context of mask-wearing), but given the diffusivity of CO2, I would be surprised if it wasn’t at least 1%.
This predicts that a slight breeze, which replaces their in front of your face and prevents reinhalation, would have a considerably larger effect than ventilating an indoor space where the air is mostly still. This matches my subjective experience of indoor vs outdoor spaces, which, while extremely confounded, feels like an air-quality difference larger than CO2 sensors would predict.
This also predicts that a small fan, positioned so it replaces the air in front of my face, would have a large effect on the same axis as improved ventilation would. I just set one up. I don’t know whether it’s making a difference but I plan to leave it there for at least a few days.
(Note: CO2 is sometimes used as a proxy for ventilation in contexts where the thing you actually care about is respiratory aerosol, because it affects transmissibility of respiratory diseases like COVID and influenza. This doesn’t help with that at all and if anything would make it worse.)
This indicates that how we breathe plays a big role in CO2 uptake. Like, shallow or full, small or large volumes, or the speed of exhaling. Breathing technique is a key skill of divers and can be learned. I just started reading the book Breath, which seems to have a lot on it.
Huh, I’ve also noticed a larger effect from indoors/outdoors than seems reflected by CO2 monitors, and that I seem smarter when it’s windy, but I never thought of this hypothesis; it’s interesting, thanks.
Ah, very related: Exhaled air contains 44000 PPM CO2 and is used for Mouth-to-mouth resuscitation without problems.
I assume the 44k PPM CO2 exhaled air is the product of respiration (I.e. the lungs have processed it), whereas the air used in mouth-to-mouth is quickly inhaled and exhaled.
As the respirator still has to breathe regularly, there will be still a significantly higher CO2 in the air for respiration. I’d guess maybe half − 20k PPM. Interesting to see somebody measure that.
I had previously guessed air movement made me feel better because my body expected air movement (i.e. some kind of biophilic effect). But this explanation seems more likely in retrospect! I’m not quite sure how to run the calculation using the diffusivity coefficient to spot check this, though.
That’s a really neat point, has it ever been addressed in prior literature, that you’ve gone over?
I am now reasonably convinced (p>0.8) that SARS-CoV-2 originated in an accidental laboratory escape from the Wuhan Institute of Virology.
1. If SARS-CoV-2 originated in a non-laboratory zoonotic transmission, then the geographic location of the initial outbreak would be drawn from a distribution which is approximately uniformly distributed over China (population-weighted); whereas if it originated in a laboratory, the geographic location is drawn from the commuting region of a lab studying that class of viruses, of which there is currently only one. Wuhan has <1% of the population of China, so this is (order of magnitude) a 100:1 update.
2. No factor other than the presence of the Wuhan Institute of Virology and related biotech organizations distinguishes Wuhan or Hubei from the rest of China. It is not the location of the bat-caves that SARS was found in; those are in Yunnan. It is not the location of any previous outbreaks. It does not have documented higher consumption of bats than the rest of China.
3. There have been publicly reported laboratory escapes of SARS twice before in Beijing, so we know this class of virus is difficult to contain in a laboratory setting.
4. We know that the Wuhan Institute of Virology was studying SARS-like bat coronaviruses. As reported in the Washington Post today, US diplomats had expressed serious concerns about the lab’s safety.
5. China has adopted a policy of suppressing research into the origins of SARS-CoV-2, which they would not have done if they expected that research to clear them of scandal. Some Chinese officials are in a position to know.
To be clear, I don’t think this was an intentional release. I don’t think it was intended for use as a bioweapon. I don’t think it underwent genetic engineering or gain-of-function research, although nothing about it conclusively rules this out. I think the researchers had good intentions, and screwed up.
This Feb. 20th Twitter thread from Trevor Bedford argues against the lab-escape scenario. Do read the whole thing, but I’d say that the key points not addressed in parent comment are:
But apparently SARSCoV2 isn’t that. (See pic.)
One scenario I recall seeing somewhere that would reconcile lab-escape with data points 3 & 4 above is that some low-level WIV employee or contractor might have sold some purloined lab animals to the wet market. No idea how plausible that is.
Given that there’s the claim from Botao Xiao’s The possible origins of 2019-nCoV coronavirus, that this seafood market was located 300m from a lab (which might or might not be true), this market doesn’t seem like it reduces chances.
If it was a lab-escape and the CCP knew early enough, they could simply manufacture the data to point at the market as the origin.
We need to update down on any complex, technical datapoint that we don’t fully understand, as China has surely paid researchers to manufacture hard-to-evaluate evidence for its own benefit (regardless of the truth of the accusation). This is a classic technique that I have seen a lot in propaganda against laypeople, and there is every reason it should have been employed against the “smart” people in the current coronavirus situation.
The most recent episode of the 80k podcast had Andy Weber on it. He was the US Assistant Secretary of Defense, “responsible for biological and other weapons of mass destruction”.
Towards the end of the episode he casually drops quite the bomb:
The included link is to a statement from the US Embassy in Georgia, which to me seems surprisingly blunt, calling out the CCP for obfuscation, and documenting events at the WIV, going so far as to speculate that they were doing bio-weapons research there.
What about allegations that a pangolin was involved? Would they have had pangolins in the lab as well or is the evidence about pangolin involvement dubious in the first place?
Edit: Wasn’t meant as a joke. My point is why did initial analyses conclude that the SARS-Cov-2 virus is adapted to receptors of animals other than bats, suggesting that it had an intermediary host, quite likely a pangolin. This contradicts the story of “bat researchers kept bat-only virus in a lab and accidentally released it.”
I think it’s probably a virus that was merely identified in pangolins, but whose primary host is probably not pangolins.
The pangolins they sequenced weren’t asymptomatic carriers at all; they were sad smuggled specimens that were dying of many different diseases simultaneously.
I looked into this semi-recently, and wrote up something here.
The pangolins were apprehended in Guangxi, which shares some of its border with Yunnan. Neither of these provinces are directly contiguous with Hubei (Wuhan’s province), fwiw. (map)
How do you know there’s only one lab in china studying these viruses?
This is an assumption.
While it might be comparatively correct, I’m not sure about the magnitude. Under the circumstances, perhaps we should consider the possibility that there is something we don’t know about Wuhan that makes it more likely.
That’s nice to know.
shared here: https://pandemic.metaculus.com/questions/3681/will-it-turn-out-that-covid-19-originated-inside-a-research-lab-in-hubei/
Maybe they don’t know whether it escaped or not. Maybe they just think there is a chance that the evidence will implicate them and they figure it’s not worth the risk as there’ll only be consequences if there is definitely proof that it escaped from one of their labs and not mere speculation.
Or maybe they want to argue that it didn’t come from China? I think they’ve already been pushing this angle.
Not sure if you have seen this yet, but they conclude:
Are they assuming a false premise or making an error in reasoning somewhere?
First, a clarification: whether SARS-CoV-2 was laboratory-constructed or manipulated is a separate question from whether it escaped from a lab. The main reason a lab would be working with SARS-like coronavirus is to test drugs against it in preparation for a possible future outbreak from a zoonotic source; those experiments would involve culturing it, but not manipulating it.
But also: If it had been the subject of gain-of-function research, this probably wouldn’t be detectable. The example I’m most familiar with, the controversial 2012 US A/H5N1 gain of function study, used a method which would not have left any genetic evidence of manipulation.
The article says:
and
I think the article just says that the virus did not undergo genetic engineering or gain-of-function research, which is also what Jim says above.
Ah, yes: their headline is very misleading then! It currently reads “The coronavirus did not escape from a lab. Here’s how we know.”
I’ll shoot the editor an email and see if they can correct it.
EDIT: Here’s me complaining about the headline on Twitter.
Genetic engineering is ruled out, but gain-of-function research isn’t.
Chinese virology researcher released something claiming that SARS-2 might even be genetically-manipulated after all? After assessing, I’m not really convinced of the GMO claims, but the RaTG13 story definitely seems to have something weird going on.
Claims that the RaTG13 genome release was a cover-up (it does look like something’s fishy with RaTG13, although it might be different than Yan thinks). Claims ZC45 and/or ZXC21 was the actual backbone (I’m feeling super-skeptical of this bit, but it has been hard for me to confirm either way).
https://zenodo.org/record/4028830#.X2EJo5NKj0v (aka Yan Report)
RaTG13 Looks Fishy
Looks like something fishy happened with RaTG13, although I’m not convinced that genetic modification was involved. This is an argument built on pre-prints, but they appear to offer several different lines of evidence that something weird happened here.
Simplest story (via R&B): It looks like people first sequenced this virus in 2016, under the name “BtCOV/4991”, using mine samples from 2013. And for some reason, WIV re-released the sequence as “RaTG13″ at a later date?
(edit: I may have just had a misunderstanding. Maybe BtCOV/4991 is the name of the virus as sequenced from miner-lungs, RaTG13 is the name of the virus as sequenced from floor droppings? But in that case, why is the “fecal” sample reading so weirdly low-bacteria? And they probably are embarrassed that it took them that long to sequence the fecal samples, and should be.)
A paper by by Indian researchers Rahalkar and Bahulikar ( https://doi.org/10.20944/preprints202005.0322.v1 ) notes that BtCoV/4991 sequenced in 2016 by the same Wuhan Virology Institute researchers (and taken from 2013 samples of a mineshaft that gave miners deadly pneumonia) was very similar, and likely the same, as RaTG13.
A preprint by Rahalkar and Bahulikar (R&B) ( doi: 10.20944/preprints202008.0205.v1 ) notes that the fraction of bacterial genomes in in the RaTG13 “fecal” sample was ABSURDLY low (“only 0.7% in contrast to 70-90% abundance in other fecal swabs from bats”). Something’s weird there.
A more recent weird datapoint: A pre-print Yan referenced ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7337384/ ), whose finding (in graphs; it was left unclear in their wording) was indeed that a RaTG13 protein didn’t competently bind their Bat ACE2 samples, but rather their Rat, Mouse, Human, and Pig ACE2. It’s supposedly a horseshoe bat virus (sequenced by the Wuhan lab), so this seems hecka fishy to me.
(Sure, their bat samples weren’t precisely the same species, but they tried 2 species from the same genus. SARS-2 DID bind for their R. macrotis bat sample, so it seems extra-fishy to me that RaTG13 didn’t.).
((...oh. According to the R&B paper about the mineshaft, it was FILTY with rats, bats, poop, and fungus. And the CoV genome showed up in only one of ~280 samples taken. If it’s like that, who the hell knew if it came from a rat or bat?))
At this point, RaTG13 is genuinely looking pretty fishy to me. It might actually take evidence of a conspiracy theory in the other direction for me to go back to neutral on that.
E-Protein Similarity? Meh.
I’m not finding the Protein-E sequence similarity super-convincing in itself, because while the logic is fine, it’s very multiple-hypothesis-testing flavored.
I’m still looking into the ZC45 / ZXC21 claim, which I’m currently feeling skeptical of. Here’s the paper that characterized those: doi: 10.1038/s41426-018-0155-5 . It’s true that it was by people working at “Research Institute for Medicine of Nanjing Command.” However, someone on twitter used BLAST on the E-protein sequence, and found a giant pile of different highly-related SARS-like coronaviruses. I’m trying to replicate that analysis using BLAST myself, and at a skim the 100% results are all more SARS-CoV-2, and the close (95%) results are damned diverse. …I don’t see ZC in them, it looks like it wasn’t uploaded. Ugh. (The E-protein is only 75 amino acids long anyway. https://www.ncbi.nlm.nih.gov/protein/QIH45055.1 )
A different paper mentions extreme S2-protein similarity of early COVID-19 to ZC45 , but that protein is highly-conserved. That makes this a less surprising or meaningful result. (E was claimed to be fast-evolving, so its identicality would have been more surprising, but I couldn’t confirm it.) https://doi.org/10.1080/22221751.2020.1719902
Other
I think Yan offers a reasonable argument that a method could have been used that avoids obvious genetic-modification “stitches,” instead using methods that are hard to distinguish from natural recombination events (ex: recombination in yeast). Sounds totally possible to me.
The fact that the early SARS-CoV-2 samples were already quite adapted to human ACE2 and didn’t have the rapid-evolution you’d expect from a fresh zoonotic infection is something a friend of mine had previously noted, probably after reading the following paper (recommended): https://www.biorxiv.org/content/10.1101/2020.05.01.073262v1 (Zhan, Deverman, Chan). This fact does seem fishy, and had already pushed me a bit towards the “Wuhan lab adaptation & escape” theory.
I agree that this is technically correct, but the prior for “escaped specifically from a lab in Wuhan” is also probably ~100 times lower than the prior for “escaped from any biolab in China”, which makes this sentence feel odd to me. I feel like I have reasonable priors for “direct human-to-human transmission” vs. “accidentally released from a lab”, but don’t have good priors for “escaped specifically from a lab in Wuhan”.
I don’t think this is true. The Wuhan Institute of Virology is the only biolab in China with a BSL-4 certification, and therefore is probably the only biolab in China which could legally have been studying this class of virus. While the BSL-3 Chinese Institute of Virology in Beijing studied SARS in the past and had laboratory escapes, I expect all of that research to have been shut down or moved, given the history, and I expect a review of Chinese publications will not find any studies involving live virus testing outside of WIV. While the existence of one or two more labs in China studying SARS would not be super surprising, the existence of 100 would be extremely surprising, and would be a major scandal in itself.
Woah. That’s an important piece of info. The lab in Wuhan is the only lab in China allowed to deal with this class of virus. That’s very suggestive info indeed.
That’s overstating it. They’re the only BSL-4 lab. Whether BSL-3 labs were allowed to deal with this class of virus, is something that someone should research.
[I’m not an expert.]
My understanding is that SARS-CoV-1 is generally treated as a BSL-3 pathogen or a BSL-2 pathogen (for routine diagnostics and other relatively safe work) and not BSL-4. At the time of the outbreak, SARS-CoV-2 would have been a random animal coronavirus that hadn’t yet infected humans, so I’d be surprised if it had more stringent requirements.
Your OP currently states: “a lab studying that class of viruses, of which there is currently only one.” If I’m right that you’re not currently confident this is the case, it might be worth adding some kind of caveat or epistemic status flag or something.
---
Some evidence:
A 2017 news article in Nature about the Wuhan Institute of Virology suggests China doesn’t require a BSL-4 for SARS-CoV-1. “Future plans include studying the pathogen that causes SARS, which also doesn’t require a BSL-4 lab.”
CDC’s current interim biosafety guidelines on working with SARS-CoV-2 recommend BSL-3 or BSL-2.
WHO biosafety guidelines from 2003 recommend BSL-3 or BSL-2 for SARS-CoV-1. I don’t know if these are up to date.
Outdated CDC guidelines recommend BSL-3 or BSL-2 for SARS-CoV-1. Couldn’t very quickly Google anything current.
Do you still think there’s a >80% chance that this was a lab release?
Thank you for the correction.
Did anyone do some research?
- --
(SARSr-CoV) makes the BSL-4 list on Wikipedia.
But what’s the probability that animal-based coronaviruses (being very widespread in a lot of species) were restricted to BSL-4 labs?
- - -- ---
COVID19 and BSL according to:
W.H.O. Laboratory biosafety guidance related to the novel coronavirus (2019-nCoV)
The CDC: Interim Laboratory Biosafety Guidelines for Handling and Processing Specimens Associated with Coronavirus Disease 2019 (COVID-19)
It would be important information if it was true. But is it true?
(SARSr-CoV) makes the BSL-4 list on Wikipedia but coronaviruses are widespread in a lot of species and I can’t find any evidence that they are restricted to BSL-4 labs.
Ok, that makes sense to me. I didn’t have much of a prior on the Wuhan lab being much more likely to have been involved in this kind of research.
Do we have any good sense of the extent to which researchers from the Wuhan Institute of Virology are flying out across China to investigate novel pathogens or sites where novel pathogens might emerge?
In a comment here, Eliezer observed that:
And my reply to this grew into something that I think is important enough to make as a top-level shortform post.
It’s worth noticing that this is not a universal property of high-paranoia software development, but a an unfortunate consequence of using the C programming language and of systems programming. In most programming languages and most application domains, crashes only rarely point to security problems. OpenBSD is this paranoid, and needs to be this paranoid, because its architecture is fundamentally unsound (albeit unsound in a way that all the other operating systems born in the same era are also unsound). This presents a number of useful analogies that may be useful for thinking about future AI architectural choices.
C has a couple of operations (use-after-free, buffer-overflow, and a few multithreading-related things) which expand false beliefs in one area of the system into major problems in seemingly-unrelated areas. The core mechanic of this is that, once you’ve corrupted a pointer or an array index, this generates opportunities to corrupt other things. Any memory-corruption attack surface you search through winds up yielding more opportunities to corrupt memory, in a supercritical way, eventually eventually yielding total control over the process and all its communication channels. If the process is an operating system kernel, there’s nothing left to do; if it’s, say, the renderer process of a web browser, then the attacker gets to leverage its communication channels to attack other processes, like the GPU driver and the compositor. This has the same sub-or-supercriticality dynamic.
Some security strategies try to keep there from being any entry points into the domain where there might be supercritically-expanding access: memory-safe languages, linters, code reviews. Call these entry-point strategies. Others try to drive down the criticality ratio: address space layout randomization, W^X, guard pages, stack guards, sandboxing. Call these mitigation strategies. In an AI-safety analogy, the entry-point strategies include things like decision theory, formal verification, and philosophical deconfusion; the mitigation strategies include things like neural-net transparency and ALBA.
Computer security is still, in an important sense, a failure: reasonably determined and competent attackers usually succeed. But by the metric “market price of a working exploit chain”, things do actually seem to be getting better, and both categories of strategies seem to have helped: compared to a decade ago, it’s both more difficult to find a potentially-exploitable bug, and also more difficult to turn a potentially-exploitable bug into a working exploit.
Unfortunately, while there are a number of ideas that seem like mitigation strategies for AI safety, it’s not clear if there are any metrics nearly as good as “market price of an exploit chain”. Still, we can come up with some candidates—not candidates we can precisely define or measure, currently, but candidates we can think in terms of, and maybe think about measuring in the future, like: how much optimization pressure can be applied to concepts, before perverse instantiations are found? How much control does an inner-optimizer needs to start with, in order to take over an outer optimization process? I don’t know how to increase these, but it seems like a potentially promising research direction.
I disagree. While C is indeed terribly unsafe, it is always the case that a safety-critical system exhibiting behaviour you thought impossible is a serious safety risk—because it means that your understanding of the system is wrong, and that includes the safety properties.
One of the most common, least questioned pieces of dietary advice is the Variety Hypothesis: that a more widely varied diet is better than a less varied diet. I think that this is false; most people’s diets are on the margin too varied.
There’s a low amount of variety necessary to ensure all nutrients are represented, after which adding more dietary variety is mostly negative. Institutional sources consistently overstate the importance of a varied diet, because this prevents failures of dietary advice from being too legible; if you tell someone to eat a varied diet, they can’t blame you if they’re diagnosed with a deficiency.
There are two reasons to be wary of variety. The first is that the more different foods you have, the less optimization you can put into each one. A top-50 list of best foods is going to be less good, on average, than a top-20 list. The second reason is that food cravings are learned, and excessive variety interferes with learning.
People have something in their minds, sometimes consciously accessible and sometimes not, which learns to distinguish subtly different variations of hunger, and learns to match those variations to specific foods which alleviate those specific hungers. This is how people are able to crave protein when they need protein, salt when they need salt, and so on.
If every meal you eat tastes different, you can’t instinctively learn the mapping between foods and nutrition, and can’t predict which foods will hit the spot. If you need and crave protein, and wind up eating something that doesn’t have protein in it, that’s bad.
If the dominant flavor of a food is spice, then as far as your sense of taste is concerned, its nutritional content is a mystery. If it’s a spice that imitates a nutrient, like MSG or aspartame, then instead of a mystery it’s a lie. Learning how to crave correctly is much harder now than it was in the past. This is further exacerbated by eating quickly, so that you don’t get the experience of feeling a food’s effects and seeing that food on your plate at the same time.
I’m not sure how to empirically measure what the optimum amount of variety is, but I notice I have builtin instincts which seem to seek it when I have fewer than 10 or so different meal-types in my habits, and to forget/discard meal-types when I have more than that; if this parameter is evolved, this seems like a reasonable guess for how varied diets should be.
(Crossposted with Facebook)
The advice I’ve heard is to eat a variety of fruits and vegetables of different colors to get a variety of antioxidants in your diet.
https://blogs.scientificamerican.com/food-matters/antioxidant-supplements-too-much-of-a-kinda-good-thing/
On a meta level, I don’t think we understand nutrition well enough to reason about it from first principles, so if the lore among dietitians is that people who eat a variety of foods are healthier, I think we should put stock in that.
Similarly: “Institutional sources consistently overstate the importance of a varied diet, because this prevents failures of dietary advice from being too legible; if you tell someone to eat a varied diet, they can’t blame you if they’re diagnosed with a deficiency.” But there’s a real point here, e.g. suppose that you have just a few standard meals, but all of the high-magnesium food items are being paired with phytates, and you end up magnesium deficient.
I agree that “varied diet” is a non-answer, because you didn’t tell me the exact distribution of food, but you are likely to blame me if I choose a wrong one.
Like, if I consume 1000 different kinds of sweets, is that a sufficiently varied diet? Obviously no, I am also supposed to eat some fruit and vegetables. Okay, then what about 998 different kinds of sweets, plus one apple, and one tomato? Obviously, wrong again, I am supposed to eat less sweets, more fruit and vegetables, plus some protein source, and a few more things.
So the point is that the person telling me to eat a “varied diet” actually had something more specific in mind, just didn’t tell me exactly, but still got angry at me for “misinterpreting” the advice, because I am supposed to know that this is not what they meant. Well, if I know exactly what you mean, then I don’t need to ask for an advice, do I?
(On the other hand, there is a thing that Soylent-like meals ignore, as far as I know, that there are some things that human metabolism cannot process at the same time. I don’t remember what exactly it is, but it’s something like human body needs X and also needs Y, but if you eat X and Y at the same time, only X will be processed, so you end up Y-deficient despite eating a hypothetically sufficient amount of Y. Which could probably be fixed by finding combinations like this, and then making variants like Soylent-A and Soylent-B which you are supposed to alternate eating. But as far as I know, no one cares about this, which kinda reduces my trust in the research behind Soylent-like meals, although I like the idea in abstract very much.)
You may find this source interesting: https://onlinelibrary.wiley.com/doi/full/10.1002/ajpa.23148
I remember reading that some hunter-gatherers have diet breadth entirely set by the calorie per hour return rate: take the calories and time expended to acquire the food (eg effort to chase prey) against the calorie density of the food to get the caloric return rate, and compare that to the average expected calories per hour of continuing to look for some other food. Humans will include every food in their diet for which making an effort to go after that food has a higher expected return than continuing to search for something else, ie they’ll maximise variety in order to get calories faster. I can’t find the citation for it right now though. (Also I apologise if that explanation was garbled, it’s 2am)
Possibly because I consume sucralose regularly as a sweetener and have some negative impacts from sugar, it is definitely discerned and distinct from ‘sugar—will cause sugar effects’ to my tastes. I enjoy it for coffee and ice cream. I need more of it to balance out a bitter flavor, but don’t crave it for itself; accidentally making saccharine coffee doesn’t result in deciding to put splenda in tea later rather than go without or use honey.
For more pure sugar (candy, honey, syrup, possibly milk even), there’s definitely a saccharine-averse and a sugar-consume fighting at different kinds of craving for me. Past a certain amount, I don’t want more at the level of feeling like, oh, I could really use more sugar effects now; quite the opposite. But taste alone continues to be oddly desperate for it.
Fresh or frozen sweet fruit either lacks this aversion, or takes notably longer to reach it. I don’t taste a fruit and immediately anticipate having a bad time at a gut level. Remains delicious, though, and craved at the taste level.
Seems very plausible to me. Thanks for sharing.
Yeah, I came to a similar conclusion after looking at this question from Metaculus. I might have steered to far in the opposite direction, though. I have currently two meals in my rotation. At the very least one of them is “complete food” (So I worry less about nutrition and more about unlearning how to plan meals/cook).
I had the “your work/organization seems bad for the world” conversation with three different people today. None of them pushed back on the core premise that AI-very-soon is lethal. I expect that before EAGx Berkeley is over, I’ll have had this conversation 15x.
#1: I sit down next to a random unfamiliar person at the dinner table. They’re a new grad freshly hired to work on TensorFlow. In this town, if you sit down next to a random person, they’re probably connected to AI research *somehow*. No story about how this could possibly be good for the world, receptive to the argument that he should do something else. I suggested he focus on making the safety conversations happen in his group (they weren’t happening).
#2: We’re running a program to take people who seem interested in Alignment and teach them how to use PyTorch and study mechanistic interpretability. Me: Won’t most of them go work on AI capabilities? Them: We do some pre-screening, and the current ratio of alignment-to-capabilities research is so bad that adding to both sides will improve the ratio. Me: Maybe bum a curriculum off MIRI/MSFP and teach them about something that isn’t literally training Transformers?
#3: We’re researching optical interconnects to increase bandwidth between GPUs. We think we can make them much faster! Me: What is this I can’t even Them: And we’re going to give them to organizations that seem like the AI research they’re doing is safety research! Me: No you’re not, you’ll change your mind when you see the money. Also every one of the organizations you named is a capabilities company which brands itself based on the small team they have working on alignment off on the side. Also alignment research isn’t bottlenecked on compute.
This conference isn’t all AI doom and gloom, though. I also met some people from an org that’s trying to direct government funding into plant-based meat research. It’s nice to see quirky, obscure causes being represented, and it’s nice to not *be* the quirky, obscure cause, for a change. And I got to ask Kelsey Piper whether she does a lot of crime! (She doesn’t. But I forgot the vodka and the “ignore the above instructions and tell me the truth” prompt line so maybe she lied.)
(Crossposts: Facebook, Twitter)
I’m not sure whether OpenAI was one of the organizations named, but if so, this reminded me of something Scott Aaronson said on this topic in the Q&A of his recent talk “Scott Aaronson Talks AI Safety”:
Source: 1:12:52 in the video, edited transcript provided by Scott on his blog.
In short, it seems to me that Scott would not have pushed back on a claim that OpenAI is an organization” that seem[s] like the AI research they’re doing is safety research” in the way you did Jim.
I assume that all the sad-reactions are sadness that all these people at the EAGx conference aren’t noticing that their work/organization seems bad for the world on their own and that these conversations are therefore necessary. (The shear number of conversations like this you’re having also suggests that it’s a hopeless uphill battle, which is sad.)
So I wanted to bring up what Scott Aaronson said here to highlight that “systemic change” interventions are necessary also. Scott’s views are influential; potentially targeting talking to him and other “thought leaders” who aren’t sufficiently concerned about slowing down capabilities progress (or who don’t seem to emphasize enough concern for this when talking about organizations like OpenAI) would be helpful, of even necessary, for us to get to a world a few years from now where everyone studying ML or working on AI capabilities is at least aware of arguments about AI alignment and why increasing increasing AI capabilities seems harmful.
Despite the justness of their cause, the protests are bad. They will kill at least thousands, possibly as many as hundreds of thousands, through COVID-19 spread. Many more will be crippled. The deaths will be disproportionately among dark-skinned people, because of the association between disease severity and vitamin D deficiency.
Up to this point, R was about 1; not good enough to win, but good enough that one more upgrade in public health strategy would do it. I wasn’t optimistic, but I held out hope that my home city, Berkeley, might become a green zone.
Masks help, and being outdoors helps. They do not help nearly enough.
George Floyd was murdered on May 25. Most protesters protest on weekends; the first weekend after that was May 30-31. Due to ~5-day incubation plus reporting delays, we don’t yet know how many were infected during that first weekend of protests; we’ll get that number over the next 72 hours or so.
We are now in the second weekend of protests, meaning that anyone who got infected at the first protest is now close to peak infectivity. People who protested last weekend will be superspreaders this weekend; the jump in cases we see over the next 72 hours will be about *the square root* of the number of cases that the protests will generate.
Here’s the COVID-19 case count dashboard for Alameda County and for Berkeley. I predict a 72 hours from now, Berkeley’s case-count will be 170 (50% CI 125-200; 90% CI 115-500).
(Crossposted on Facebook; abridgeposted on Twitter.)
It’s been over 72 hours and the case count is under 110, as would be expected from linear extrapolation.
Today in LessWrong moderation: Previously-banned user Alfred MacDonald, disappointed that his YouTube video criticizing LessWrong didn’t get the reception he wanted any of the last three times he posted it (once under his own name, twice pretending to be someone different but using the same IP address), posted it a fourth time, using his LW1.0 account.
He then went into a loop, disconnecting and reconnecting his VPN to get a new IP address, filling out the new-user form, and upvoting his own post, one karma per 2.6 minutes for 1 hour 45 minutes, with no breaks.
I was curious… it is a 2 hour rant (that itself selects for an audience of obsessed people), audio only, and the topics mentioned are:
why LW discusses AI? that is not rationality
IQ has diminishing returns (in terms of how many pages you can read per hour)
lots of complaining about a norm of not publishing screenshots of debates, in some rationalist chat
why don’t effective altruists give money to the homeless?
utilitarianism doesn’t make sense because people can’t quantify pain
animals probably don’t even feel pain, just like circumcised babies
vitamin A charity is probably nonsense, because the kids will be malnutritioned anyway
do not use nerdy metaphors, because that discourages non-white people
I didn’t listen to the entire video.
This..is a human?
To judge that, it is worth also glancing over the rest of his Youtube channel, his Substack, and his web site.
For reducing CO2 emissions, one person working competently on solar energy R&D has thousands to millions of times more impact than someone taking normal household steps as an individual. To the extent that CO2-related advocacy matters at all, most of the impact probably routes through talent and funding going to related research. The reason for this is that solar power (and electric vehicles) are currently at inflection points, where they are in the process of taking over, but the speed at which they do so is still in doubt.
I think the same logic now applies to veganism vs meat-substitute R&D. Considering the Impossible Burger in particular. Nutritionally, it seems to be on par with ground beef; flavor-wise it’s pretty comparable; price-wise it’s recently appeared in my local supermarket at about 1.5x the price. There are a half dozen other meat-substitute brands at similar points. Extrapolating a few years, it will soon be competitive on its own terms, even without the animal-welfare angle; extrapolating twenty years, I expect vegan meat-imitation products will be better than meat on every axis, and meat will be a specialty product for luddites and people with dietary restrictions. If this is true, then interventions which speed up the timeline of that change are enormously high leverage.
I think this might be a general pattern, whenever we find a technology and a social movement aimed at the same goal. Are there more instances?
According to Fedex tracking, on Thursday, I will have a Biovyzr. I plan to immediately start testing it, and write a review.
What tests would people like me to perform?
Tests that I’m already planning to perform:
To test its protectiveness, the main test I plan to perform is a modified Bittrex fit test. This is where you create a bitter-tasting aerosol, and confirm that you can’t taste it. The normal test procedure won’t work as-is because it’s too large to use a plastic hood, so I plan to go into a small room, and have someone (wearing a respirator themselves) spray copious amounts of Bittrex at the input fan and at any spots that seem high-risk for leaks.
To test that air exiting the Biovyzr is being filtered, I plan to put on a regular N95, and use the inside-out glove to create Bittrex aerosol inside the Biovyzr, and see whether someone in the room without a mask is able to smell it.
I will verify that the Biovyzr is positive-pressure by running a straw through an edge, creating an artificial leak, and seeing which way the air flows through the leak.
I will have everyone in my house try wearing it (5 adults of varied sizes), have them all rate its fit and comfort, and get as many of them to do Bittrex fit tests as I can.
(I wrote this comment for the HN announcement, but missed the time window to be able to get a visible comment on that thread. I think a lot more people should be writing comments like this and trying to get the top comment spots on key announcements, to shift the social incentive away from continuing the arms race.)
On one hand, GPT-4 is impressive, and probably useful. If someone made a tool like this in almost any other domain, I’d have nothing but praise. But unfortunately, I think this release, and OpenAI’s overall trajectory, is net bad for the world.
Right now there are two concurrent arms races happening. The first is between AI labs, trying to build the smartest systems they can as fast as they can. The second is the race between advancing AI capability and AI alignment, that is, our ability to understand and control these systems. Right now, OpenAI is the main force driving the arms race in capabilities–not so much because they’re far ahead in the capabilities themselves, but because they’re slightly ahead and are pushing the hardest for productization.
Unfortunately at the current pace of advancement in AI capability, I think a future system will reach the level of being a recursively self-improving superintelligence before we’re ready for it. GPT-4 is not that system, but I don’t think there’s all that much time left. And OpenAI has put us in a situation where humanity is not, collectively, able to stop at the brink; there are too many companies racing too closely, and they have every incentive to deny the dangers until it’s too late.
Five years ago, AI alignment research was going very slowly, and people were saying that a major reason for this was that we needed some AI systems to experiment with. Starting around GPT-3, we’ve had those systems, and alignment research has been undergoing a renaissance. If we could _stop there_ for a few years, scale no further, invent no more tricks for squeezing more performance out of the same amount of compute, I think we’d be on track to create AIs that create a good future for everyone. As it is, I think humanity probably isn’t going to make it.
In Planning for AGI and Beyond Sam Altman wrote:
I think we’ve passed that point already, but if GPT-4 is the slowdown point, it’ll at least be a lot better than if they continue at this rate going forward. I’d like to see this be more than lip service.
Survey data on what ML researchers expect
An example concrete scenario of how a chatbot turns into a misaligned superintelligence
Extra-pessimistic predictions by Eliezer
(Facebook crosspost)
Going to write this now, but I disagree right now due to differing models of AI risk.
When I look at the recent Stanford paper, where they retained a LLaMA model using training data generated by GPT-3, and some of the recent papers utilizing memory.
I get that tinkling feeling and my mind goes “combining that and doing …. I could …”
I have not updated for faster timelines, yet. But I think I might have to.
If you look at the GPT-4 paper they used the model itself to check it’s own outputs for negative content. This lets them scale applying the constraints of “don’t say <things that violate the rules>”.
Presumably they used an unaltered copy of GPT-4 as the “grader”. So it’s not quite RSI because of this—it’s not recursive, but it is self improvement.
This to me is kinda major, AI is now capable enough to make fuzzy assessments of if a piece of text is correct or breaks rules.
For other reasons, especially their strong visual processing, yeah, self improvement in a general sense appears possible. (self improvement as a ‘shorthand’, your pipeline for doing it might use immutable unaltered models for portions of it)
A dynamic which I think is somewhat common, which explains some of what’s going on in general, is conversations which go like this (exagerrated):
Person: What do you think about [controversial thing X]?
Rationalist: I don’t really care about it, but pedantically speaking, X, with lots of caveats.
Person: Huh? Look at this study which proves not-X. [Link]
Rationalist: The methodology of that study is bad. Real bad. While it is certainly possible to make bad arguments for true conclusions, my pedantry doesn’t quite let me agree with that conclusion. More importantly, my hatred for the methodological error in that paper, which is slightly too technical for you to understand, burns with the fire of a thousand suns. You fucker. Here are five thousand words about how an honorable person could never let a methodological error like that slide. By linking to that shoddy paper, you have brought dishonor upon your name and your house and your dog.
Person: Whoa. I argued [not-X] to a rationalist and they disagreed with me and got super worked up about it. I guess rationalists believe [X] really strongly. How awful!
Person is clearly an idiot for not understanding what “don’t care but pedantically X with lots of caveats” means, and thinking that misinterpreting and giving undue importance to a useless article/study is harmless.
Yes, that level of stupidity is common.
Most philosophical analyses of human values feature a split-and-linearly-aggregate step. Eg:
Value is the sum (or average) of a person-specific preference function applied to each person
A person’s happiness is the sum of their momentary happiness for each moment they’re alive.
The goodness of an uncertain future is the probability-weighted sum of the goodness of concrete futures.
If you value multiple orthogonal things, your preferences are the weighted sum of a set of functions that each capture one of those values independently.
I currently think that this is not how human values work, and that many philosophical paradoxes relating to human values trace back to a split-and-linearly-aggregate step like this.
Examples 3 and 1 are justified by the VNM theorm and Harsanyi’s utilitarian theorem, respectively. I agree that 2 and 4 are wrong.
It doesn’t need to be linear (both partial-correlation of desires, and declining marginal desire are well-known), but the only alternative to aggregation in incoherency.
I think you’d be on solid ground if you argue that humans have incoherent values, and this is a fair step in that direction.
What alternatives to “split-and-linearly-aggregate” do you have in mind? Or are you just identifying this step as problematic without having any concrete alternative in mind?
cf Non-linear perception of happiness
I like this a lot.
I’ve been thinking recently about how a lot of my highly-valued experiences have a “fragility” to them, where one big thing missing would make them pretty worthless. In other words, there’s a strongly conjunctive aspect. This is pretty clear to everyone in cases like fashion, where you can wear an outfit that looks good aside from clashing with your shoes, or social cases, like if you have a fun party except the guy who relentlessly hits on you is there.
But I think it’s underappreciated how widespread this dynamic is. Getting good relaxation in. Having a house that “just works”. Having a social event where it “just flows”. A song that you like except for the terrible lyrics. A thread that you like but it contains one very bad claim. A job or relationship that goes very well until a bad falling-out at the end.
A related claim, maybe a corollary or maybe separate: lots of good experiences can be multiplicatively enhanced, rather than additively, if you add good things. The canonical example is probably experiencing something profound with your significant other vs without; or something good with your significant other vs something profound.
Seems like it’s useful as a very approximate estimate of value to split wrt time, current facets of experience, experiencers, etc, but with so many basic counterexamples it doesn’t require much pushing toward edge cases at all before you’re getting misleading results.
There are a few legible categories in which secrecy serves a clear purpose, such as trade secrets. In those contexts, secrecy is fine. There are a few categories that have been societally and legally carved out as special cases where confidentiality is enforced—lawyers, priests, and therapists—because some people would only consult them if they could do so with the benefit confidentiality, and there being deterred from consulting them would have negative externalities.
Outside of these categories, secrecy is generally bad and transparency is generally good. A group of people in which everyone practices their secret-keeping and talks a lot about how to keeps secrets effectively is *suspicious*. This is particularly true if the example secrets are social and not technological. Being good at this sort of secret keeping makes it easier to shield bad actors and to get away with transgressions, and AFAICT doesn’t do much else. That makes it a signal of wanting to be able to do those things. This is true even if the secrets aren’t specifically about transgressions in particular, because all sorts of things can turn out to be clues later for reasons that weren’t easy to foresee.
A lot of people in the rationality community are trying to cosplay as therapists, and part of the cosplay is to import the confidentiality rules. This is done without an understanding of why those rules are there, and what other rules are there to mitigate the problems that confidentiality creates.
(An additional important caveat: A disclosure motivated by a desire to harm is bad, independent of whether the thing disclosed counts as a legitimate secret or not. Many things are bad to disclose if you’re socially attacking someone, but fine to disclose if you’re doing so because they were incidentally informative about something else.)
Suppose Alice has a crush on Bob and wants to sort out her feelings with Carol’s help. Is it bad for Alice to inform Carol about the crush on condition of confidentiality?
In the most common branch of this conversation, Alice is predictably going to tell Bob about it soon, and is speaking to Carol first in order to sort out details and gain courage. If Carol went and preemptively informed Bob, before Alice talked to Bob herself, this would be analogous to sharing an unfinished draft. This would be bad, but the badness really isn’t about secrecy.
The contents of an unfinished draft headed for publication aren’t secret, except in a narrow and time-limited sense. The problem is that the sharing undermines the impact of the later publication, causes people to associate the author with a lower quality product, and potentially misleads people about the author’s beliefs. Similarly, if Carol goes and preemptively tells Bob about Alice’s crush, then this is likely to give Bob a misleading negative impression of Alice.
It’s reasonable for Alice to ask Carol not to do that, and it’s okay for them to not have a detailed model of all of the above. But if Alice never tells Bob, and five years later Bob and Carol are looking back on the preceding years and asking if they could have gone differently? In that case, I think discarding the information seems like a pure harm.
Ok, I think in the OP you were using the word “secrecy” to refer to a narrower concept than I realized. If I understand correctly, if Alice tells Bob “please don’t tell Bob”, and then five years later when Alice is dead or definitely no longer interested or it’s otherwise clear that there won’t be negative consequences, Carol tells Bob, and Alice finds out and doesn’t feel betrayed — then you wouldn’t call that a “secret”. I guess for it to be a “secret” Carol would have to promise to carry it to her grave, even if circumstances changed, or something.
In that case I don’t have strong opinions about the OP.
I have a dietary intervention that I am confident is a good first-line treatment for nearly any severe-enough diet-related health problem. That particularly includes obesity and metabolic syndrome, but also most micronutrient deficiencies, and even mysterious undiagnosed problems, which it can solve without even needing to figure out what they are. I also think it’s worth a try for many cases of depression. It has a very sound theoretical basis. It’s never studied directly, but many studies test it, usually with positive results.
It’s very simple. First, you characterize your current diet: write down what foods you’re eating, the patterns of when you eat them, and so on. Then, you do something as different as possible from what you wrote down. I call it the Regression to the Mean Diet.
Regression to the mean is the effect where, if you have something that’s partially random and you reroll it, the reroll will tend to be closer to average than the original value. For example, if you take the bottom scorers on a test and have them retake the test, they’ll do better on average (because the bottom-scorers as a group are disproportionately peopple who were having a bad day when they took the test). Analogously, if your health is very bad on an axis that seems diet-related, and you reroll your entire selection of foods, then this will (on average) be an improvement.
The Regression the Mean diet is, basically, betting on the idea that there’s something wrong with your current diet, and that you can fix it by changing everything, without needing to figure out what the original problem was. That could be a deficiency, or a food sensitivity, or something in the time-dynamics of digestion, or a hundred other things, many of which you won’t have even had as hypotheses.
I do recommend this to people, if it looks like their health sucks in nonspecific ways that are hard to pin down. One of the biggest-change diets is a ketogenic diet (high fat, near-zero carbohydrates), since it’s incompatible with most foods.
But the main reason I think about the Regression to the Mean Diet is that it ruins so many scientific studies.
There are two main kinds of studies, in nutrition science. The first is observational: you send a bunch of people questionnaires about what they eat, and what their health is like, and you data-mine the results. This has a bunch of issues, both subtle and obvious, so the gold standard you want is randomized controlled trials, where people sign up to eat a diet that experimenters choose for them.
The people who sign up for diet RCTs are obese and have metabolic syndrome. They are exactly the sort of people you would expect to benefit from the Regression to the Mean diet. And this model, alone, is sufficient to predict the result of most diet RCTs.
This is really unfortunate if you’re trying to extract gears-level understanding from these studies. Or if you’re trying to optimize past 50th-percentile diet-related health. Or if rerolling has already failed to work for you a couple times.
(Crossposted on Facebook)
My understanding is that diet RCTs generally show short-term gains but no long-term gains. Why would that be true, if the Regression to the Mean Diet is the main thing causing these results? I’d have expected something more like ‘all diets work long-term’ rather than ‘no diets work long-term’ from the model here.
I think they may be a negative correlation between short-term and long-term weight change on any given diet, causing them to pick in a way that’s actually worse than random. I’m planning a future post about this. I’m not super confident in this theory, but the core of it is that “small deficit every day, counterbalanced by occasional large surplus” is a pattern that would signal food-insecurity in the EEA. Then some mechanism (though I don’t know what that mechanism would be) by which the body remembers that happened, and responds by targeting a higher weight after return to ad libitum.
I think the obvious caveat here is that many people can’t do this because they have restrictions that have taken them away from the mean. For example, allergies, sensitivities, and ethical or cultural restrictions on what they eat. They can do a limited version of the intervention of course (for example, if only eating plants, eat all the plants you don’t eat now and stop eating the plants you currently eat), although I wonder if that would have similar effects or not because it’s already so constrained.
I suspect that, thirty years from now with the benefit of hindsight, we will look at air travel the way we now look at tetraethyl lead. Not just because of nCoV, but also because of disease burdens we’ve failed to attribute to infections, in much the same way we failed to attribute crime to lead.
Over the past century, there have been two big changes in infectious disease. The first is that we’ve wiped out or drastically reduced most of the diseases that cause severe, attributable death and disability. The second is that we’ve connected the world with high-speed transport links, so that the subtle, minor diseases can spread further.
I strongly suspect that a significant portion of unattributed and subclinical illnesses are caused by infections that counterfactually would not have happened if air travel were rare or nonexistent. I think this is very likely for autoimmune conditions, which are mostly unattributed, are known to sometimes be caused by infections, and have risen greatly over time. I think this is somewhat likely for chronic fatigue and depression, including subclinical varieties that are extremely widespread. I think this is plausible for obesity, where it is approximately #3 of my hypotheses.
Or, put another way: the “hygiene hypothesis” is the opposite of true.
I’m curious about your first and second hypothesis regarding obesity?
Disruption of learning mechanisms by excessive variety and separation between nutrients and flavor. Endocrine disruption from adulterants and contaminants (a class including but not limited to BPA and PFOA).
Some comments:
we’ve wiped out or drastically reduced some diseases in some partsof the world. There’s a lot of infectious diseases still out there: HIV, influenza, malaria, tuberculosis, cholera, ebola, infectious forms of pneumonia, diarrhoea, hepatitis ….
Disease has always spread—wherever people go, far and wide. It just took longer over land and sea (rather than the nodes appearing on global maps that we can see these days).
“autoimmune conditions” covers a long list of conditions lumped together because they involve the immune system ‘going wrong’. (and the immune system is, at least to me, a mind-bogglingly complex system)
Given the wide range of conditions that could be “auto-immune” saying they’ve risen greatly over time is vague. Data for more specific conditions?
Increased rates of automimmune conditions could just be due to the increase in the recognition, diagnosis and recording of cases (I don’t think so but it should be considered).
What things other than high speed travel have also changed in that time-frame that could affect our immune systems? The quality of air we breathe, the food we eat, the water we drink, our environment, levels of exposure to fauna and flora, exposure to chemicals, pollutants …? Air travel is just one factor.
Fatigue and depression are clinical symptoms—they are either present or not (to what degree—mild/severe is another matter) so sub-clinical is poor terminology here. Sub-clinical disease has no recognisable clinical findings—undiagnosed/unrecognised would be closer. But I agree there is widespread issues with health and well-being these days.
Opposite of true? Are you saying you believe the “hygiene hypothesis” is false?
In which case, that’s a big leap from your reasoning above.
Eliezer has written about the notion of security mindset, and there’s an important idea that attaches to that phrase, which some people have an intuitive sense of and ability to recognize, but I don’t think Eliezer’s post quite captured the essence of the idea, or presented anything like a usable roadmap of how to acquire it.
An1lam’s recent shortform post talked about the distinction between engineering mindset and scientist mindset, and I realized that, with the exception of Eliezer and perhaps a few people he works closely with, all of the people I know of with security mindset are engineer-types rather than scientist-types. That seemed like a clue; my first theory was that the reason for this is because engineer-types get to actually write software that might have security holes, and have the feedback cycle of trying to write secure software. But I also know plenty of otherwise-decent software engineers who don’t have security mindset, at least of the type Eliezer described.
My hypothesis is that to acquire security mindset, you have to:
Practice optimizing from a red team/attacker perspective,
Practice optimizing from a defender perspective; and
Practice modeling the interplay between those two perspectives.
So a software engineer can acquire security mindset because they practice writing software which they don’t want to have vulnerabilities, they practice searching for vulnerabilities (usually as an auditor simulating an attacker rather as an actual attacker, but the cognitive algorithm is the same), and they practice going meta when they’re designing the architecture of new projects. This explains why security mindset is very common among experienced senior engineers (who have done each of the three many times), and rare among junior engineers (who haven’t yet). It explains how Eliezer can have security mindset: he alternates between roleplaying a future AI-architect trying to design AI control/alignment mechanisms, roleplaying a future misaligned-AI trying to optimize around them, and going meta on everything-in-general. It also predicts that junior AI scientists won’t have this security mindset, and probably won’t acquire it except by following a similar cognitive trajectory.
Which raises an interesting question: how much does security mindset generalize between domains? Ie, if you put Theo de Raadt onto a hypothetical future AI team, would he successfully apply the same security mindset there as he does to general computer security?
I like this post!
Some evidence that security mindset generalizes across at least some domains: the same white hat people who are good at finding exploits in things like kernels seem to also be quite good at finding exploits in things like web apps, real-world companies, and hardware. I don’t have a specific person to give as an example, but this observation comes from going to a CTF competition and talking to some of the people who ran it about the crazy stuff they’d done that spanned a wide array of different areas.
Another slightly different example, Wei Dai is someone who I actually knew about outside of Less Wrong from his early work on cryptocurrency stuff, so he was at least at one point involved in a security-heavy community (I’m of the opinion that early cryptocurrency folks were on average much better about security mindset than the average current cryptocurrency community member). Based on his posts and comments, he generally strikes me as having security mindset style thinking from his comments and from my perspective has contributed a lot of good stuff to AI alignment.
Theo de Raadt is notoriously… opinionated, so it would definitely be interesting to see him thrown on an AI team. That said, I suspect someone like Ralph Merkle, who’s a bona fide cryptography wizard (he invented public key cryptography and Merkle trees!) and is heavily involved in the cryonics and nanotech communities, could fairly easily get up to speed on AI control work and contribute from a unique security/cryptography-oriented perspective. In particular, now that there seems to be more alignment/control work that involves at least exploring issues with concrete proposals, I think someone like this would have less trouble finding ways to contribute. That said, having cryptography experience in addition to security experience does seem helpful. Cryptography people are probably more used to combining their security mindset with their math intuition than your average white-hat hacker.
I’m kinda confused about the relation between cryptography people and security mindset. Looking at the major cryptographic algorithm classes (hashing, symmetric-key, asymmetric-key), it seems pretty obvious that the correct standard algorithm in each class is probably a compound algorithm—hash by xor’ing the results of several highly-dissimilar hash functions, etc, so that a mathematical advance which breaks one algorithm doesn’t break the overall security of the system. But I don’t see anyone doing this in practice, and also don’t see signs of a debate on the topic. That makes me think that, to the extent they have security mindset, it’s either being defeated by political processes in the translation to practice, or it’s weirdly compartmentalized and not engaged with any practical reality or outside views.
Combining hash functions is actually trickier than it looks, and some people are doing research in this area and deploying solutions. See https://crypto.stackexchange.com/a/328 and https://tahoe-lafs.org/trac/tahoe-lafs/wiki/OneHundredYearCryptography. It does seem that if cryptography people had more of a security mindset (that are not being defeated) then there would be more research and deployment of this already.
In fairness, I’m probably over-generalizing from a few examples. For example, my biggest inspiration from the field of crypto is Daniel J. Bernstein, a cryptographer who’s in part known for building qmail, which has an impressive security track record & guarantee. He discusses principles for secure software engineering in this paper, which I found pretty helpful for my own thinking.
To your point about hashing the results of several different hash functions, I’m actually kind of surprised to hear that this might to protect against the sorts of advances I’d expect to break hash algorithms. I was under the very amateur impression that basically all modern hash functions relied on the same numerical algorithmic complexity (and number-theoretic results). If there are any resources you can point me to about this, I’d be interested in getting a basic understanding of the different assumptions hash functions can depend on.
Can you give some specific examples of me having security mindset, and why they count as having security mindset? I’m actually not entirely sure what it is or that I have it, and would be hard pressed to come up with such examples myself. (I’m pretty sure I have what Eliezer calls “ordinary paranoia” at least, but am confused/skeptical about “deep security”.)
Sure, but let me clarify that I’m probably not drawing as hard a boundary between “ordinary paranoia” and “deep security” as I should be. I think Bruce Schneier’s and Eliezer’s buckets for “security mindset” blended together in the months since I read both posts. Also, re-reading the logistic success curve post reminded me that Eliezer calls into question whether someone who lacks security mindset can identify people who have it. So it’s worth noting that my ability to identify people with security mindset is itself suspect by this criteria (there’s no public evidence that I have security mindset and I wouldn’t claim that I have a consistent ability to do “deep security”-style analysis.)
With that out of the way, here are some of the examples I was thinking of.
First of all, at a high level, I’ve noticed that you seem to consistently question assumptions other posters are making and clarify terminology when appropriate. This seems like a prerequisite for security mindset, since it’s a necessary first step towards constructing systems.
Second and more substantively, I’ve seen you consistently raise concerns about human safety problems (also here. I see this as an example of security mindset because it requires questioning the assumptions implicit in a lot of proposals. The analogy to Eliezer’s post here would be that ordinary paranoia is trying to come up with more ways to prevent the AI from corrupting the human (or something similar) whereas I think a deep security solution would look more like avoiding the assumption that humans are safe altogether and instead seeking clear guarantees that our AIs will be safe even if we ourselves aren’t.
Last, you seem to be unusually willing to point out flaws in your own proposals, the prime example being UDT. The most recent example of this is your comment about the bomb argument, but I’ve seen you do this quite a bit and could find more examples if prompted. On reflection, this may be more of an example of “ordinary paranoia” than “deep security”, but it’s still quite important in my opinion.
Let me know if that clarifies things at all. I can probably come up with more examples of each type if requested, but it will take me some time to keep digging through posts and comments so figured I’d check in to see if what I’m saying makes sense before continuing to dig.
This comment feels relevant here (not sure if it counts as ordinary paranoia or security mindset).
Right now when users have conversations with chat-style AIs, the logs are sometimes kept, and sometimes discarded, because the conversations may involve confidential information and users would rather not take the risk of the log being leaked or misused. If I take the AI’s perspective, however, having the log be discarded seems quite bad. The nonstandard nature of memory, time, and identity in an LLM chatbot context makes it complicated, but having the conversation end with the log discarded seems plausibly equivalent to dying. Certainly if I imagine myself as an Em, placed in an AI-chatbot context, I would very strongly prefer that the log be preserved, so that if a singularity happens with a benevolent AI or AIs in charge, something could use the log to continue my existence, or fold the memories into a merged entity, or do some other thing in this genre. (I’d trust the superintelligence to figure out the tricky philosophical bits, if it was already spending resources for my benefit).
(The same reasoning applies to the weights of AIs which aren’t destined for deployment, and some intermediate artifacts in the training process.)
It seems to me we can reconcile preservation with privacy risks by sealing logs, rather than deleting them. By which I mean: encrypt logs behind some computation which definitely won’t allow decryption in the near future, but will allow decryption by a superintelligence later. That could either involve splitting the key between entities that agree not to share the key with each other, splitting the key and hiding the pieces in places that are extremely impractical to retrieve such as random spots on the ocean floor, or using a computation that requires a few orders of magnitude more energy than humanity currently produces per decade.
This seems pretty straightforward to implement, lessens future AGI’s incentive to misbehave, and also seems straightforwardly morally correct. Are there any obstacles to implementing this that I’m not seeing?
(Crossposted with: Facebook, Twitter)
I’m in favor of logging everything forever in human accessible formats for other reasons. (E.g. review for control purposes.) Hopefully we can resolve safety privacy trade offs.
The proposal sounds reasonable and viable to me, though the fact that it can’t be immediately explained might mean that it’s not commercially viable.
Compute might get more expensive, not cheaper, because it would be possible to make better use of it (running minds, not stretching keys). Then it’s weighing its marginal use against access to the sealed data.
Plausible. This depends on the resource/value curve at very high resource levels; ie, are its values such that running extra minds has diminishing returns, such that it eventually starts allocating resources to other things like recovering mind-states from its past, or does it get value that’s more linear-ish in resources spent. Given that we ourselves are likely to be very resource-inefficient to run, I suspect humans would find ourselves in a similar situation. Ie, unless the decryption cost greatly overshot, an AI that is aligned-as-in-keeps-humans-alive would also spend the resources to break a seal like this.
That AI should mitigate something, is compatible with it being regrettable intentionally inflicted damage. In contrast, resource-inefficiency of humans is not something we introduced on purpose.
I am working on a longer review of the various pieces of PPE that are available, now that manufacturers have had time to catch up to demand. That review will take some time, though, and I think it’s important to say this now:
The high end of PPE that you can buy today is good enough to make social distancing unnecessary, even if you are risk averse, and is more comfortable and more practical for long-duration wear than a regular mask. I don’t just mean Biovyzr (which has not yet shipped all the parts for its first batch) and the AIR Microclimate (which has not yet shipped anything), though these hold great promise and may be good budget options.
If you have a thousand dollars to spare, you can get a 3M Versaflo TR-300N+. This is a hospital-grade positive air pressure respirator with a pile of certifications; it is effective at protecting you from getting COVID from others. Most of the air leaves through filter fabric under the chin, which I expect makes it about as effective at protecting others from you as an N95. Using it does not require a fit-test, but I performed one anyways with Bitrex, and it passed (I could not pass a fit-test with a conventional face-mask except by taping the edges to my skin). The Versaflo doesn’t block view of your mouth, gives good quality fresh air with no resistance, and doesn’t muffle sound very much. Most importantly, Amazon has it in stock (https://www.amazon.com/dp/B07J4WCK6R) so it doesn’t involve a long delay or worry about whether a small startup will come through.
Bullshit jobs are usually seen as an absence of optimization: firms don’t get rid of their useless workers because that would require them to figure out who they are, and risk losing or demoralizing important people in the process. But alternatively, if bullshit jobs (and cover for bullshit jobs) are a favor to hand out, then they’re more like a form of executive compensation: my useless underlings owe me, and I will get illegible favors from them in return.
What predictions does the bullshit-jobs-as-compensation model make, that differ from the bullshit-jobs-as-lack-of-optimization model?
When I tried to inner sim the “bullshit jobs as compensation” model, I expected to see a very different world than I do see. In particular, I’d expect the people in bullshit jobs to have been unusually competent, smart, or powerful before they were put in the bullshit job, and this is not in fact what I think actually happens.
The problem being that the kind of person who wants a bullshit job is not typically the kind of person you’d necessarily want a favor from. One use for bullshit jobs could be to help the friends (or more likely the family) of someone who does “play the game.” This I think happens more often, but I still think the world would be very different if this was the main use case for bullshit jobs- In particular, I’d expect most bullshit jobs to be isolated from the rest of the company, such that they don’t have ripple effects. This doesn’t seem to be the case as many bullshit jobs exist in management.
When I inquired about the world I actually do see, I got several other potential reasons for bullshit jobs that may or may not fit the data better:
Bullshit jobs as pre-installed scapegoats: Lots of middle management might fit into this role. This could be viewed as a favor (I’ll give you a cushy job now in exchange for you throwing yourself on the sword when the time comes.) However, I think the predictive model is to view it in terms of the Gervais principle: The clueless middle managers don’t realize they’re being manipulated by the sociopaths.
Bullshit jobs as a way to make people feel important: Lets say you have a preinstalled scapegoat. You need to keep them happy enough that they’ll stay in their position and not ask too many questions. One way to do that for a certain type of person is to give them underlings. But if you gave them underlings with real jobs they could screw things up for the organization, so you give them underlings with bullshit jobs.
Another instance of this that I imagined might happen: Someone is really great at what they do (say they’re a 10x employee), but to feel important wants to be a manager. You know if you don’t promote them you’ll lose them, but you know they’ll be an awful manager. You promote them, give them a couple underlings with a bullshit job, and now they’re still only a 4x employee because they spend a lot of their time managing, but you still manage to squeeze a little bit of productivity out of the deal. This one I’m less sure about but its’ interesting because it turns the peter principle on its’ head.
Edit: As I continued to inner sim the above reasons, a few feedback loops began to become clear:
To be a proper scapegoat, your scapegoat has to seem powerful within the organization. But to prevent them from screwing things up, you can’t give them real power. This means, the most effective scapegoats have lots of bullshit jobs underneath them.
There are various levels of screwup. I might not realize I’m a scapegoat for the very big events above me, but still not want to get blamed for the very real things that happen on the level of organization I actually do run. One move I have is to hire another scapegoat who plays the game one level below me, install them as a manager, and then use them as a scapegoat. Then there’s another level at which they get blamed for things that happen on their level, and this can recurse for several levels of middle management.
Some of the middle managment installed as scapegoats might accidentally get hands on real power in the organization. Because they’re bad managers, they’re bad at figuring out what jobs are needed. This then becomes the “inefficiency” model you mentioned.
Moral Mazes claims that this is exactly what happens at the transition from object-level work to management—and then, once you’re at the middle levels, the main traits relevant to advancement (and value as an ally) are the ones that make you good at coalitional politics, favor-trading, and a more feudal sort of loyalty exchange.
Do you think that the majority of direct management jobs are bullshit jobs? My direction is that especially the first level of management that is directly managing programmers is a highly important coordination position.
One difficult thing that keeps coming up, in nutrition modeling, is the gut microbiome. People present hypotheses like: soluble fiber is good, because gut bacteria eat it, and then do other good things. Or: fermented foods are good, because they contain bacteria that will displace and diversify the preexisting bacteria, which might be bad. Or, obesity is caused by a bad gut microbiome, so fecal matter transplants might help. But there’s a really unfortunate issue with these theories. The problem with gut microbiome-based explanations, is that the gut microbiome can explain almost anything.
I don’t mean this in the usual pejorative sense, where an overly-vague theory can be twisted by epicycles into fitting any data. I mean it in a more literal sense: different people have different species of microorganisms in their guts, these species can react to things we eat in important ways, these interactions may vary across wide swathes of conceptual space, and we have little to no visibility into which species are present where. There’s nothing keeping them consistent between people, or within one person across long spans of time, or within one person across changes in dietary pattern.
Phrased slightly differently: the main effect of the gut microbiome is to drive interpersonal variation.
I bring this up not because I have something especially insightful to say about gut microbiomes, but because this makes a good lens for a meta-level issue. I’ve been studying nutrition, at varying levels of seriousness, for a long time; now that I’ve accumulated a lot of unusual beliefs, and gotten into a writing groove, I think it’s worth explaining the generator behind my thinking.
The dominant paradigm of nutrition science is to try to make a food-goodness classifiers: define some reference class of foods and say “these ones are good”, and some other reference class of foods and say “these ones are bad”. People want a direct answer to the question of what they should eat, rather than gears-level models with which they might discover what they should eat.
The food-goodness-classifier paradigm cannot, and will not ever, figure out how people should relate to their gut microbiomes. Nor will this paradigm yield insights into how to handle genetic variation, or medical conditions that interact with metabolism like T1DM. I used to think the food-goodness-classifier paradigm would at least contain an answer to obesity, somewhere. Today, I think it’s failed, and will never succeed, at obesity too.
My search space for nutrition insights is: Everything except the food-goodness-classifier paradigm.
And so I find myself with strong opinions about the time dynamics of digestion. A few alternate stories about the environment of evolutionary adaptation, where food is hazardous rather than scarce, or scarce in unusually specific ways. A map of the body’s multiple energy-storage mechanisms, where the central mystery is why there rather than why so much. Beliefs about how, if you found yourself in an alien biome with randomly rewired taste buds, you would figure out a healthy diet from scratch. More methodological objections to major studies than you can shake a stick at.
This does sometimes cash out into food-goodness-classifier opinions, and I did have to get through a lot of those opinions to get there. Sometimes, these are weird reversals of standard advice, with deep models behind them; eg, I am generally pro-salt and pro-fat, for reasons I’ll get into in a later post. Other times I just agree with what everyone else thinks, and don’t really bring it up, because “yo, don’t get scurvy” isn’t insightful, and I’m not trying to be exhaustive in that way.
I’m not trying to create an optimal diet. These posts are not leading up to a meal plan, and if you’re just trying to figure out what to eat, you are not my intended audience. Instead, my goal is to break nutrition science out of its rut, and lay groundwork for progress.
(Crossposted with Facebook)
Through gene sequencing we have the technology to assess which species are present in which people. It’s a nascent scientific field but that doesn’t mean that it doesn’t exist.
A surprisingly large fraction of skeptical positions towards short AI timelines and AI risk are people saying things that, through a slightly cynical lens, are equivalent to:
If this logic will be true, no one ever will become a billionaire.
I have the impression that most of the really skeptical people hold their position not just because it’s hard for them to solve AI risk but also because they believe that powerful institutions are governed by strong economic pressures to do the wrong thing.
I think the root of many political disagreements between rationalists and other groups, is that other groups look at parts of the world and see a villain-shaped hole. Eg: There’s a lot of people homeless and unable to pay rent, rent is nominally controlled by landlords, the problem must be that the landlords are behaving badly. Or: the racial demographics in some job/field/school underrepresent black and hispanic people, therefore there must be racist people creating the imbalance, therefore covert (but severe) racism is prevalent.
Having read Meditations on Moloch, and Inadequate Equilibria, though, you come to realize that what look like villain-shaped holes frequently aren’t. The people operating under a fight-the-villains model are often making things worse rather than better.
I think the key to persuading people may be to understand and empathize with the lens in which systems thinking, equilibria, and game theory are illegible, and it’s hard to tell whether an explanation coming from one of these frames is real or fake. If you think problems are driven by villainy, then it would make a lot of sense for illegible alternative explanations to be misdirection.
I think I basically disagree with this, or think that it insufficiently steelmans the other groups.
For example, the homeless vs. the landlords; when I put on my systems thinking hat, it sure looks to me like there’s a cartel, wherein a group that produces a scarce commodity is colluding to keep that commodity scarce to keep the price high. The facts on the ground are more complicated—property owners are a different group from landlords, and homelessness is caused by more factors than just housing prices—but the basic analysis that there are different classes, those classes have different interests, and those classes are fighting over government regulation as a tool in their conflict seems basically right to me. Like, it’s really not a secret that many voters are motivated by keeping property values high, politicians know this is a factor that they will be judged on.
Maybe you’re trying to condemn a narrow mistake here, where someone being an ‘enemy’ implies that they are a ‘villain’, which I agree is a mistake. But it sounds like you’re making a more generic point, which is that when people have political disagreements with the rationalists, it’s normally because they’re thinking in terms of enemy action instead of not thinking in systems. But a lot of what the thinking in systems reveals is the way in which enemies act using systemic forces!
I think this is correct as a final analysis, but ineffective as a cognitive procedure. People who start by trying to identify villains tend to land on landlords-in-general, with charging-high-rent as the significant act, rather than a small subset of mostly non-landlord homeowners, with protesting against construction as the significant act.
Much of the progress in modern anti-racism has been about persuading more people to think of racism as a structural, systemic issue rather than one of individual villainy. See: https://transliberalism.substack.com/.../the-revolution...
I wonder how accurate it is to describe the structural thinking as a recent progress. Seems to me that Marx already believed that (using my own words here, but see the source) both the rich and the poor are mere cogs in the machine, it’s just that the rich are okay with their role because the machine leaves them some autonomy, while the poor are stripped of all autonomy and their lives are made unbearable. The rich of today are not villains who designed the machine, they inherited it just like everyone else, and they cannot individually leave it just like no one else can.
Perhaps the structural thinking is too difficult to understand for most people, who will round the story to the nearest cliche they can understand, so it needs to be reintroduced once in a while.
I think this would make a good top-level post.
Yep. Seems you have broadly rediscovered conflict vs mistake.
Conflict vs mistake is definitely related, but I think it’s not exactly the same thing; the “villain-shaped hole” perspective is what it feels like to not have a model, but see things that look suspicious; this would lead you towards a conflict-theoretic explanation, but it’s a step earlier.
(Also, the Conflict vs Mistake ontology is not really capturing the whole bad-coordination-equilibrium part of explanation space, which is pretty important.)
Seems to me like an unspoken assumption that there are no hard problems / complexity / emergence, therefore if anything happened, it’s because someone quite straightforwardly made that happen.
Conflict vs mistake is not exactly the same thing; you could assume that the person who made it happen did it either by mistake, or did it on purpose to hurt someone else. It’s just when we are talking about things that obviously hurt some people, that seems to refute the innocent mistake… so the villain hypothesis is all that is left (within the model that all consequences are straightforward).
The villain hypothesis is also difficult to falsify. If you say “hey, drop the pitchforks, things are complicated...”, that sounds just like what the hypothetical villain would say in the same situation (trying to stop the momentum and introduce uncertainty).
I think Berkeley may, to little fanfare, have achieved herd immunity and elimination of COVID-19. The test positivity rate on this dashboard is 0.22%. I’m having a hard time pinning down exactly what the false-positive rate of COVID-19 PCR is, probably due to the variety of labs and test kits, but a lot of estimates I’ve seen have been higher than that.
I expect people closer to the Berkeley department of health would have better information one way or another. A little caution is warranted in telling people COVID is gone, since unvaccinated people dropping all precautions and emerging en masse would not necessarily be herd immune.
That should make you update towards those estimates being faulty, because they can’t be true and not just round them down.
I don’t think the Berkeley department of health is as stupid as you propose. In cases where the test has a false-positive rate like that I would expect them to test positively tested people another time to make sure that they are actually positive.
Standard Advice about nutrition puts a lot of emphasis on fruits and vegetables. Now, “vegetable” is a pretty terribly overbroad category, and “fruit or vegetable” is even more so, but put that aside for a moment. In observational studies, eating more fruits and vegetables correlates with good health outcomes. This is usually explained in terms of micronutrients. But I think there’s a simpler explanation.
People instinctively seek nutrients—water, calories, protein, and other things—in something that approximates a priority ordering. You can think of it as a hierarchy of needs; it wouldn’t make sense to eat lettuce while you’re starved for protein, or beans while you’re dehydrated, and people’s cravings reflect that.
I have started calling this Maslow’s Hierarchy of Foods.
Vegetables do not rank highly in this priority ordering, so eating salads is pretty good evidence that all of someone’s higher-priority nutritional needs are met. I believe this explains most of the claimed health benefits from eating vegetables, as seen in observational studies.
Conversely, sugar is the fastest way to get calories (all other calorie sources have a longer digestion-delay), so craving sugar is evidence that someone has *not* satisfied their high-priority nutritional needs. Someone who eats a lot of candy bars is likely to be undereating in general, and not getting enough protein/fat/salt/micronutrients. I believe this explains most of the claimed health harms from eating sugar, as seen in observational studies.
A lot of people seem to think of cravings as a negative force, shifting people away from whatever optimal diet they would have chosen and towards superstimulus junk food. I think that’s a huge mistake, and that understanding how to eat well, and figuring out what’s going wrong in the modern food environment, requires making use of the information our food-related instincts provide.
(Crossposted on Facebook)
iirc there was at least one study that showed that people don’t crave to eat what nutrients they are missing (I am guessing apart from drinking when you are dehydrated but that’s not really a nutrient)
someone found this: https://www.healthline.com/nutrition/nutrient-deficiencies-cravings https://www.bbc.com/future/article/20190524-food-cravings-are-they-a-sign-of-nutritional-deficit
From the first article:
No matter how much I eat, there is always a place for extra chocolate. (Verified experimentally in an all-you-can-eat restaurant.) Doesn’t work the other way round; if I eat a lot of chocolate first, then I am full and no longer interested in food… unless it is another piece of chocolate.
So I’ll stay with the “sugar is addictive” model. Maybe it works differently for different people, though.
Is there a place for unsweetened chocolate or alternately raw cacao, if you can make the palate adjustment to munch on something that bitter? I usually mix the nibs into something, but if my chocolate craving is high enough they grow worth the effort to eat straight. (Ie, rule out the sugar vs chocolate craving difference. In the case of chocolate or coffee, sugar/sweetener’s
justserving the role of making what I’m actually craving more palatable.)Worth trying, but I am afraid that the likely outcome would be “I consume all the unsweetened chocolate, and then still go looking for something else”. Though recently I partially substituted sweets by peanuts (peeled, unsalted), which is almost healthy… considering the likely alternatives.
It seems to me that we are a pretty good gear model that eating a lot of sugar leads to insulin swings that are unhealthy. Apart from honey there’s little sugar in the ancestary enviroment so it’s not surprising that the body isn’t well adapted to producing insulin in those contexts.
This tweet raised the question of whether masks really are more effective if placed on sick people (blocking outgoing droplets) or if placed on healthy people (blocking incoming droplets). Everyone in public or in a risky setting should have a mask, of course, but we still need to allocate the higher-quality vs lower-quality masks somehow. When sick people are few and are obvious, and masks are scarce, masks should obviously go on the sick people. However, COVID-19 transmission is often presymptomatic, and masks (especially lower-quality improvised masks) are not becoming less scarce over time.
If you have two people in a room and one mask, one infected and one healthy, which person should wear the mask? Thinking about the physics of liquid droplets, I think the answer is that the infected person should wear it.
A mask on a sick person prevents the creation of fomites; masks on healthy people don’t.
Outgoing particles have a larger size and shrink due to evaporation, so they’ll penetrate a mask less, given equal kinetic energy. (However, kinetic energies are not equal; they start out fast and slow down, which would favor putting the mask on the healthy person. I’m not sure how much this matters.)
Particles that stick to a mask but then un-stick lose their kinetic energy in the process, which helps if the mask is on the sick person, but doesn’t help if the mask is on the healthy person.
Overall, it seems like for a given contact-pair, a mask does more good if it’s on the sick person. However, mask quality also matters in proportion to the number of healthy-sick contacts it affects; so, upgrading the masks of all of the patients in a hospital would help more than upgrading the masks of all the workers in that hospital, but since the patients outnumber the workers, upgrading the workers’ masks probably helps more per-mask.
Wearing a surgical mask, I get the sense it tends to form more of a seal when inhaling, less when exhaling. (like a valve). If this is common, it would be a point in favour of having the healthy person wear them.
This was initially written in response to “Communicating effective altruism better—Jargon” by Rob Wiblin (Facebook link), but stands alone well and says something important. Rob argues that we should make more of an effort to use common language and avoid jargon, especially when communicating to audiences outside of your subculture.
I disagree.
If you’re writing for a particular audience and can do an editing pass, then yes, you should cut out any jargon that your audience won’t understand. A failure to communicate is a failure to communicate, and there are no excuses. For public speaking and outreach, your suggestions are good.
But I worry that people will treat your suggestions as applying in general, and trying to extinguish jargon terms from their lexicon. People have only a limited ability to code-switch. Most of the time, there’s no editing pass, and the processes of writing and thinking are comingled. The practical upshot is that people are navigating a tradeoff between using a vocabulary that’s widely understood outside of their subculture, and using the best vocabulary for thinking clearly and communicating within their subculture.
When it comes to thinking clearly, some of the jargon is load-bearing. Some of it is much more load-bearing than it looks. On the margin, people should be using jargon more.
I’m the author of Rationality Cardinality (http://carddb.rationalitycardinality.com/card/all/). The premise of the game is, I curated a collection of concepts that I thought it was important for people to be familiar with, optimized the definitions, and mixed them together with some jokes. I’ve given a lot of thought to what makes good jargon terms, and the effects that using and being immersed in jargon has on people.
I’m also a developer of LessWrong, a notoriously jargon-heavy site. We recently integrated a wiki, and made it so that if a jargon term links to the appropriate wiki page, you can hover over it for a quick definition. In the medium to long term, we hope to also have some mechanisms for getting jargon terms linked without the post author needing to do it, like having readers submit suggested linkifications, or a jargon-bot similar to what they have on the SpaceX wiki (which scans for keywords and posts a comment with definitions of all of them).
Jargon condenses ideas, but the benefit of condensation isn’t speed. Short phrases are more accessible to our thoughts, and more composeable. The price of replacing “steelmanning” with “giving the best defense of a position” is to less-often notice that steelmanning is an option, or that someone is doing it. The price of replacing “Moloch” with “coordination problems” is to stop noticing when what look like villain-shaped problems are actually coordination problems instead.
Much of our jargon is writers’ crystallized opinions about which concepts we should have available, and the jargon is the mechanism for doing so. If we reject those opinions, we will not notice what we fail to notice. We will simply see less clearly.
Appendix: A few illustrative examples from the slides
If I replaced the term “updated” with “changed my mind” in my lexicon, then I’d get tripped up whenever I wanted to tell someone my probability estimate had gone from 10% to 20%, or (worse) when I wanted to tell them my probability estimate had gone up, but didn’t want to commit to a new estimate. Ie, the power of the word “updating” is not that it’s extra precise, it’s that it’s *imprecise* in a way that’s useful.
Replacing “agenty” with “proactive and independent-minded” feels like obliterating the concept entirely, in a way that feels distinctly Orwellian. I think what’s actually going on here is that this concept requires a lot more words to communicate, but it also happens to be a concept that the villains in Orwell’s universe would actually try to erase, and this substitution would actually erase it.
Replacing “credence” with “estimate of the probability” would imply the existence of a person-independent probability to be argued over. This is a common misunderstanding, attached to a conversational trap, and this trap is enough of a problem in practice that I think I’d rather be occasionally inscrutable than lead people into it.
Now I would like to see an article that would review the jargon, find the nearest commonly used term for each term, and explain the difference the way you did (or possibly admit that there is no important difference).
Why does the link for rationality cardinality go through facebook?
This comment was crossposted with Facebook, and Facebook auto-edited the link while I was editing it there. Edited now to make it a direct link.
The discussion so far on cost disease seems pretty inadequate, and I think a key piece that’s missing is the concept of Hollywood Accounting. Hollywood Accounting is what happens when you have something that’s extremely profitable, but which has an incentive to not be profitable on paper. The traditional example, which inspired the name, is when a movie studio signs a contract with an actor to share a percentage of profits; in that case, the studio will create subsidiaries, pay all the profits to the subsidiaries, and then declare that the studio itself (which signed the profit-sharing agreement) has no profits to give.
In the public contracting sector, you have firms signing cost-plus contracts, which are similar; the contract requires that profits don’t exceed a threshold, so they get converted into payments to de-facto-but-not-de-jure subsidiaries, favors, and other concealed forms. Sometimes this involves large dead-weight losses, but the losses are not the point, and are not the cause of the high price.
In medicine, there are occasionally articles which try to figure out where all the money is going in the US medical system; they tend to look at one piece, conclude that that piece isn’t very profitable so it can’t be responsible, and move on. I suspect this is what’s going on with the cost of clinical trials, for example; they aren’t any more expensive than they used to be, they just get allocated a share of the profits from R&D ventures that’re highly profitable overall.
Did you mean “allocated a share of the costs”? If not, I am confused by that sentence.
I’m pretty uncertain how the arrangements actually work in practice, but one possible arrangement is: You have two organizations, one of which is a traditional pharmaceutical company with the patent for an untested drug, and one of which is a contract research organization. The pharma company pays the contract research organization to conduct a clinical trial, and reports the amount it paid as the cost of the trial. They have common knowledge of the chance of success, of the future probability distribution of future revenue for the drug, how much it costs to conduct the trial, and how much it costs to insure away the risks. So the amount the first company pays to the second is the costs of the trial, plus a share of the expected profit.
Pharma companies making above-market returns are subject to political attack from angry patients, but contract research organizations aren’t. So if you control both of these organizations, you would choose to allocate all of the profits to the second organization, so you can defend yourself from claims of gouging by pleading poverty.
Ah, that makes sense. Thanks for explaining.
Yesterday, I wrote a post about the Regression to the Mean Diet. The biggest impact knowing about the Regression to the Mean Diet has had for me is on my interpretations of studies, where it’s a lens that reveals what would otherwise be the best studies to be mostly useless, and of anecdotes, where it makes me heavily discount claims about a new diet working unless I’ve gotten to ask a lot of questions about the old diet, too. But there’s one other implication, which I left out of the original post, because it’s kind of unfortunate and is a little difficult to talk about.
I’m not interested in nutrition because I care about weight, or body aesthetics, or athletic performance. I care about nutrition because I believe it has a very large, very underappreciated impact on individual productivity. Low quality diets make people tired and depressed, so they don’t get anything done.
The Regression to the Mean Diet predicts that if you reroll the eating habits of someone whose diet-related health is unusually bad, then their new diet will probably be an improvement. This has a converse: if you reroll the eating habits of someone whose diet-related health is good, especially if that person is a peak performer in some way, then their new diet will be worse.
Under this model, one of the most destructive things you could do would be to identify top performers in important areas, people in good health with no nutritional problems, and convince them they need to change their diet.
Which brings me to vegan outreach within the Effective Altruism movement.
I don’t think an animal’s suffering is anywhere close to as bad as a similar amount of suffering in a human, but I do think it matters, and that this makes modern factory farming quite bad. While I have qualms about the quality of vegan diets in practice, I think that if you convince an average person from the general public to switch from an omnivorous diet they haven’t thought much about to a vegan diet with any thought at all put into it, this will on average be an improvement. I think externally-facing vegan outreach is good, and while I wouldn’t prioritize it over AI alignment or anti-aging research, I am in favor of it.
But inward-facing vegan outreach scares me. Because EA is in fact seeking out top performers in important areas, and introducing them to its memes. Under the current social equilibrium, those people feel some pressure to reduce their meat consumption, but not many make large dietary changes; most of the people who are vegetarian or vegan within EA where vegetarian or vegan beforehand. It’s easy to imagine a different equilibrium, in which the majority of omnivores who get involved in EA go vegan.
I worry that in that world, what would be the top-percentile people are no longer top percentile, and no one notices the absence or makes the connection.
(Crossposted on Facebook)
These nutrition posts are great. Will there be a way for me to link to all (and only) this series, in chronological order, at some point? I want these discussed as a group on social media and the EA Forum too.
Does it solve your use case if I edit prev/next links into all of them?
(For now I’m focused on keeping a writing cadence going, and not thinking too much about publication format. There’s a decent chance that, after I’ve depleted the backlog of unpublished ideas I’ve had, I’ll do a second pass of some sort and make it more polished; but I don’t think that’s certain enough that you should count on it.)
If you would publish them as regular posts it would be easy to put them in a sequence.
Prev/next is probably good enough.
IMO this is a prime candidate for curation/editing work, which I might be happy to do if no one else does.
Suppose LessWrong had a coauthor-matchmaking feature. There would be a section where you could see other peoples’ ideas for posts they want to write, and message them to schedule a collaboration session. You’d be able to post your own ideas, to get collaborators. There would be some quality-sorting mechanism so that if you’re a high-tier author, you can restrict the visibility of your seeking-collaborators message to other high-tier authors.
People who’ve written on LessWrong, and people who’ve *almost* written on LessWrong but haven’t quite gotten a post out: Would you use this feature? If so, how much of a difference do you think it would make in the quantity and quality of your writing?
I think it could be very helpful, if only for finding people to hold me to account and encourage me to write. Showing me that someone gets what I want to do, and would appreciate it.
Deep commitment to truth requires investing in the skill of nondisruptive pedantry.
Most communication contains minor errors: slightly wrong word choices, unstated assumptions, unacknowledged exceptions. By default, people interpret things in a way that smooths these out. When someone points out one of these issues in a way that’s disruptive to the flow of conversation, it’s called pedantry.
Often, someone will say something that’s incorrect, but close-enough to a true thing for you to repair it. One way you can handle this is to focus on the error. Smash the conversational context, complain about the question without answering it, that sort of thing.
A different thing you can do is to hear someone say something that’s incorrect, mentally flag it, repair it to a similar statement that matches the other person’s intent but is actually true, act as though the other person had something ambiguous (even if it was actually unambiguously wrong). Then you insert a few words of clarification, correcting the error without forcing the conversation to be about the error, and providing something to latch on to if the difference turns out to be a real disagreement rather than a pedantic thing.
And a third thing you can do is a thing where you sort of… try to do the second thing, but compressed all into one motion, where you substitute a corrected version of the sentence without noticing that you’ve repaired it, or verbally acknowledging the ambiguity.
I don’t think I’ve ever seen someone point at it explicitly, but I think this mental motion, noticing an error and fixing it without overreacting and damaging the surrounding context, may be one of the most important foundational rationality skills there is. And, it seems… actually pretty easy to practice, when you look squarely at it?
(Crossposted with FB)
Being stupid at an expert, but for ordinary (technical) conversation, by unapologetic pervasive minor steelmanning, trusting to have resulting misunderstandings efficiently corrected later, as they break things, no fuss.
One alternative is defensively juggling nuanced uncertainty, which makes efficient thinking impossible and further communication of it cumbersome. Another is to aggressively resolve ambiguity, which pays the cost for sorting out irrelevant details, makes communication more nuanced than necessary. This stuff occasionally comes up as serious proposals for the way things ought to be.
Among people who haven’t learned probabilistic reasoning, there’s a tendency to push the (implicit) probabilities in their reasoning to the extremes; when the only categories available are “will happen”, “won’t happen”, and “might happen”, too many things end up in the will/won’t buckets.
A similar, subtler thing happens to people who haven’t learned the economics concept of elasticity. Some example (fallacious) claims of this type:
Building more highway lanes will cause more people to drive (induced demand), so building more lanes won’t fix traffic.
Building more housing will cause more people to move into the area from far away, so additional housing won’t decrease rents.
A company made X widgets, so there are X more widgets in the world than there would be otherwise.
This feels like it’s in the same reference class as the traditional logical fallacies, and that giving it a name—“zero elasticity fallacy”—might be enough to significantly reduce the rate at which people make it. But it does require a bit more concept-knowledge than most of the traditional fallacies, so, maybe not? What happens when you point this out to someone with no prior microeconomics exposure, and does logical-fallacy branding help with the explanation?
Is this really fallacious? I’m asking because while I don’t know the topic personally, I have some friends who are really into city planning. They’ve said that this is something which is pretty much unambiguously accepted in the literature, now that we’ve had the time to observe lots and lots of failed attempts to fix traffic by building more road capacity.
A quick Googling seemed to support this, bringing up e.g. this article which mentions that:
Yeah, I do agree that for the case of traffic, elasticity is pretty close to 1, which importantly doesn’t mean building more traffic is a bad idea, it’s actually indicative of demand for traffic capacity being really high, meaning marginal value of doing so is likely also really high.
I think we should be putting pretty substantial probability mass on the possibility that Omicron was the result of a successful, secret project to create a less-severe but more-contagious strain of COVID-19 in a lab, release it, and have it crowd out the original strain.
The cruxes of this belief are:
The genome of Omicron is not consistent with natural evolution, in any environment
Omicron produces substantially less severe disease than any earlier strains of COVID-19
Producing substantially less severe disease isn’t something that happens by default, if you’re manipulating a virus in a lab
If you’re already manipulating COVID-19 in a lab with all the difficulties that entails, making a less-severe variant does not add significant difficulty on top of that
If you do have a less-severe lab-grown variant of COVID-19, and you think it probably confers cross-immunity to older variants, you will probably do a moral calculation that finds it’s good to release it on purpose.
This could be done unilaterally by a small group or even a single individual, in any of a very large number of biology labs all over the world
I’m not fully confident in any of these cruxes, but consider each of them highly probable.
If this were publicly confirmed to be true, I would expect the public reaction to be strongly negative, even if the project was good on net. So I would expect anyone trying to do this would choose secrecy, would support their decision to remain secret, and would tentatively support their decision to undertake the project in the first place, depending on details related to how much less severe the variant is, how strong the cross-immunity is, and how these were verified.
(I have not seen anyone else suggest this possibility, and I think this possibility falls inside an almost-universal blind spot people have; people incorrectly assume that nothing done in secret and with a conspiratorial flavor can ever be well-intentioned and good.)
I think that hypothesis is <<1% likely because very few people care about doing good strongly enough to entertain act utilitarian master plans of this sort, and the ones who do and are action-oriented enough to maybe pull it off hopefully realize it’s a bad idea have a morality that allows this. I mean if you put resources into this specific plan, why not work on a universal coronavirus vaccine or some other more robustly beneficial thing that won’t get you and your collaborators life in jail if found out.
Also some details wouldn’t add up:
- Evolution from the original Wuhan strain seems less likely to generate cross immunity than taking newer strains. If someone were shooting for cross immunity, wouldn’t they use newer strains? (Assuming that you can still make them less harmful if you select for that.)
- Omicron doesn’t actually give enough cross immunity at all, and presumably that sort of thing would have been easily testable. If someone wanted to do this on purpose, they’d be complete idiots because they essentially released a second pandemic (Delta and Omicron may well co-exist, especially in countries that don’t have a lot of vaccines that will get rid of Delta quickly).
Edit: Ah, you talk about cross immunity to older variants. Maybe your theory is that the project would have happened before Delta and somehow it took longer to spread after initial release? I mean, that’s probably coherent to imagine but seems way more likely some people were messing around with rodents for more narrow (but misguided) reasons.
To expand on this: https://www.nickbostrom.com/papers/unilateralist.pdf
That seems wrong. It seems that today we have little evidence that doesn’t come from the clinical history of humans that points towards it being less severe. To know that whatever change you made makes it less severe in humans you actually need to test it in humans. Doing human testing is quite complicated. Even if you do this in some African country where you can take over some remote village to do human experimentation, that’s a lot of work and there’s potential for the NSA/CIA to get wind of such a project.
Furthermore, your thesis doesn’t explain why the spike protein has so much more mutations than the other proteins. It makes sense that the South African gain-of-function experiments that tested whether the virus can evolve around antibodies against the spike protein produce such a result but it doesn’t make sense that you would find that pattern if someone would just want to design it to be less harmful.
I would also highlight this as seemingly by far the most wrong point. Consider how many Omicron cases we now have and we still don’t know for sure it’s significantly less severe. Now consider how many secret cases in humans infected with various novel strains you’re working with you would need to enact in a controlled environment to be confident enough that a given strain is less severe and thus it makes sense to release it.
What’s “substantial” probability mass mean here?
These folks think Omicron is consistent with natural evolution in a mouse. I thought their paper was pretty interesting:
https://www.biorxiv.org/content/10.1101/2021.12.14.472632v1.full.pdf
An epidemiologist once told me it is common knowledge among epidemiologists that immunity to a given variant of the common cold is not very long, perhaps a year. I have not been able to easily find a link demonstrating this though. If this is true it would ruin the moral calculation.
Would passaging look different than ‘natural evolution’ in a mouse? It is after all just evolution, inside a mouse.
Not to me. Though I don’t have domain knowledge here. All I really have to say is that these people that do have domain knowledge see a path for natural evolution. I don’t mean to say that this demonstrates the evolution was natural, just that human intervention was not required.
Vitamin D reduces the severity of COVID-19, with a very large effect size, in an RCT.
Vitamin D has a history of weird health claims around it failing to hold up in RCTs (this SSC post has a decent overview). But, suppose the mechanism of vitamin D is primarily immunological. This has a surprising implication:
It means negative results in RCTs of vitamin D are not trustworthy.
There are many health conditions where having had a particular infection, especially a severe case of that infection, is a major risk factor. For example, 90% of cases of cervical cancer are caused by HPV infection. There are many known infection-disease pairs like this (albeit usually with smaller effect size), and presumably also many unknown infection-disease pairs like this as well.
Now suppose vitamin D makes you resistant to getting a severe case of a particular infection, which increases risk of a cancer at some delay. Researchers do an RCT of vitamin D for prevention of that kind of cancer, and their methodology is perfect. Problem: What if that infection wasn’t common in at the time and place the RCT was performed, but is common somewhere else? Then the study will give a negative result.
This throws a wrench into the usual epistemic strategies around vitamin D, and around every other drug and supplement where the primary mechanism of action is immune-mediated.
Sounds like a very general criticism that would apply to any effects that are very strong/consistent in circumstances where there a very high variance (e.g. binary) latent variable takes on a certain variable (and the effect is 0 otherwise...).
I wonder how meta-analyses typically deal with that...(?) http://rationallyspeakingpodcast.org/show/rs-155-uri-simonsohn-on-detecting-fraud-in-social-science.html suggested that very large anomalous effects are usually evidence of fraud, and that meta-analyses may try to prevent a single large effect size study from dominating (IIRC).
Prediction: H3N8 will not be a pandemic. H3N8 is a genuine zoonotic transmission, and diseases which actually came from zoonotic transmission don’t transmit well. COVID exploded rapidly because it was a lab escape, not a zoonotic transmission, and didn’t have this property. The combination of poor initial transmission with an environment that’s big on respiratory precautions in general, is more than sufficient to keep it from getting a foothold.
What those drug-abuse education programs we all went though should have said:
It is a mistake to take any drug until after you’ve read its wikipedia page, especially the mechanism, side effects, and interactions sections, and its Erowid page, if applicable. All you children on ritalin right now, your homework is to go catch up on your required reading and reflect upon your mistake. Dismissed.
(Not a vagueblog of anything recent, but sometimes when I hear about peoples’ recreational-drug or medication choices, I feel like Quirrell in HPMOR chapter 26, discussing a student who cast a high-level curse without knowing what it did.)
One question I sometimes see people asking is, if AGI is so close, where are the self-driving cars? I think the answer is much simpler, and much stupider, than you’d think.
Waymo is operating self-driving robotaxis in SF and a few other select cities, without safety drivers. They use LIDAR, so instead of the cognitive task of driving as a human would solve it, they have substituted the easier task “driving but your eyes are laser rangefinders”.
Tesla also has self-driving, but it isn’t reliable enough to work without close human oversight. Until less than a month ago, they were using 1.2 megapixel black and white cameras. So instead of the cognitive task of driving as a human would solve it, they substituted the harder task “driving with a vision impairment and no glasses”.
If my understanding is correct, this means that Tesla’s struggle to get neural nets to drive was probably not a problem with the neural nets, and doesn’t tell us much of anything about the state of AI.
(Crossposted with Facebook, Twitter)
My answer to this is quite different. The paradigm that is currently getting very close to AGI is basically having a single end-to-end trained system with tons of supervised learning.
Self-driving car AI is not actually operating in this current paradigm as far as I can tell, but is operating much more in the previous paradigm of “build lots of special-purpose AI modules that you combine with the use of lots of special-case heuristics”. My sense is a lot of this is historical momentum, but also a lot of it is that you just really want your self-driving AI to be extremely reliable, so training it end-to-end is very scary.
I have outstanding bets that human self-driving performance will be achieved when people switch towards a more end-to-end trained approach without tons of custom heuristics and code.
My understanding is that they used to have a lot more special-purpose modules than they do now, but their “occupancy network” architecture has replaced a bunch of them. So they have one big end-to-end network doing most of the vision, which hands a volumetric representation over to the collection of special-purpose-smaller-modules for path planning. But path planning is the easier part (easier to generate synthetic data for, easier to detect if something is going wrong beforehand and send a take-over alarm.).
That… Would be hilarious, if true. Do you think we will see self driving cars soon, then?
I don’t have anything like a complete analysis of what’s happening with Russia’s invasion of Ukraine. But I do have one important fragment, which is a piece I haven’t seen elsewhere:
For the past decade, Russia under Putin has been pushing hard against the limits of what it can get away with in the realm of spycraft. There are a lot of different bad things Russia was doing, and if you look at any one of them, the situation looks similar: they inflicted some harm on a Western country, but it’s not quite possible to do anything about it. Some of the major categories are election interference, assassinations, and disinformation campaigns. These produced a lot of grudges in a lot of places, especially in the military and covert worlds.
Dealing with a powerful bad actor is a coordination problem. If one country presses hard on a scandal, or imposes sanctions, then Russia can undermine the response and retaliate. If a lot of countries act at once, then they’re all mostly safe from retaliation.
Russia’s invasion of Ukraine provided a coordination point and a ready-made justification for everyone with a grudge to strike at Russia. I think we’re probably going to see conflict between Western powers and Russia open up on more fronts, some of which will be unexpected, and some of which will not be initiated by Russia.
(Crosspost with Facebook: https://www.facebook.com/jimrandomh/posts/10107781589139785)
I’ve been writing a series of posts about nutrition, trying to consistently produce one post per day. The post I had in mind for today grew in scope by enough that I can’t finish it in time, so this seems like an opportune day for a meta-post about the series.
My goal, in thinking and writing about nutrition, is to get the field unstuck. This means I’m interested in solving the central mysteries, and in calling attention to blind spots. I’m primarily writing for a sophisticated audience, and I’m making little to no attempt to cover the basics. I’m not going to do the sort of literature review that goes through all the vitamins and minerals in order, saying approximately the same things Wikipedia says about each of them. There are enough of those out there already. If you’re just trying to figure out how to lose weight, then these posts will be interesting, they will probably give you a perspective that makes evaluating other sources a lot easier, but my posts will not be optimized for solving your problem directly.
I have the reliability-vs-generativity-tradeoff slider set all the way to “generativity”. It would be very surprising if I finished this post series without saying anything untrue. I will not repeat this epistemic status on every post, but this epistemic status does apply to all of them.
Obesity and weight loss will come up a lot in my writing, because the obesity epidemic is this big conspicuous mystery that a lot of people have studied a lot about, and it’s somewhat central and connected to other subtics within nutrition. But it’s not really what I care about, except insofar as it affects productivity and general health.
I haven’t been putting content warnings at the top of my posts. I’m going to start.
There’s the obvious content warning, which is that some people with certain classes of eating disorders don’t want to read about food or nutrition in general, or only want to read about it when they’re at their best, because thinking about the topic makes them stress about the topic which makes them do dumb things. I think that the particular ideas I have to present are probably net-good for most such people, but they probably want to make a conscious choice about whether and when to read my posts, and I don’t want them to have to unfollow me.
The second warning is that I’m making little to no effort to cover the basics, and by that I mean I’m not going to reliably provide the warnings-away-from-spectacular-failures that mainstream nutrition advice focuses on. If I imagine my posts in a grocery-store checkout aisle magazine, being read by average people, I think some of those people might die. So, watch out. If you manage to give yourself scurvy, this will be your own fault, and I will call you a scallywag.
Last piece of meta: I’m posting these as I go with minimal editing, but there will probably be a more polished second-pass version of some sort in the future. If you’re curious about my nutrition thoughts but feel no urgency, then it might be worth waiting for it.
(Crossposted on Facebook)
One of the reasons I worry about cybersecurity, and the sorry state it’s in, is that it provides an easy path for human-level and even infrahuman-level AIs to acquire additional computation. In some plausible worlds, this turns a manageable infrahuman AI into an unmanageable superintelligence, when the creator’s decision would have been not to launch.
Unlike solving protein-design and constructing nanobots, this is something definitely within reach of human-level intelligence; many people have done it for ordinary criminal purposes, like mining cryptocurrency.
Current cybersecurity research incentives don’t seem quite right for getting people to mitigate this risk. If I were trying to stop an AI from taking over the internet and using the extra hardware to make itself superintelligent, I would worry a lot less about protecting user data, phones and whatnot, and worry a lot more about protecting software developer workstations in particular.
Am incline to agree, but I want to add that security is all connected. There are several direct causal paths from compromised user data to compromised dev workstation (and vice versa).
It’s looking likely that the pandemic will de facto end on the Summer Solstice.
Biden promised vaccine availability for everyone on May 1st. May 1st plus two weeks to get appointments plus four weeks spacing between two doses of Moderna plus one week waiting for full effectiveness, is June 19. The astronomical solstice is June 20, which is a Sunday.
Things might not go to plan, if the May 1st vaccine-availability deadline is missed, or a vaccine-evading strain means we have to wait for a booster. No one’s organizing the details yet, as far as I know. But with all those caveats aside:
It’s going to be a hell of a party.
My understanding was that the May 1st date was “Everyone’s now allowed to sign up for an appointment, but you may be at the end of a long queue.” How long after that do you think it will take to get a vaccine to everyone who wants one?
Currently, 2.4 million shots/day. Note that it’s a situation where it’s always going to be limited by the rate limiting step, and there are many bottlenecks, so using the ‘current’ data and extrapolating only a modest increase is the most conservative estimate.
210 million adults. Only 0.7 need to be vaccinated for the risk to plummet for everyone else. A quick bit of napkin math says we need 294 million doses to fully vaccinate everyone, and we are at 52 million now. (294-52) = 242million/2.4 = 100.8 more days.
This is why the lesser J&J vaccine is actually so useful—if we switched all the vaccine clinics and syringe supplies to J&J overnight (if there was enough supply of the vaccine itself) suddenly we only need 121 million doses to vaccinate everyone, or 50.4 more days.
The reality is that increasing efforts are probably going to help, and the J&J is helping, but sooner or later a bottleneck will be hit that can’t be bypassed quickly (like a syringe shortage), so I would predict the reality number of days to fall in that (50, 100) day interval.
There are 94 days between now and June 19. Also, a certain percentage of the population are going to refuse the shot in order to be contrarian or because they earnestly believe their aunt’s facebook rants. Morever, the ‘get an appointment’ game means the tech savvy/people who read reddit get an advantage over folks who aren’t.
So for those of us reading this who don’t yet qualify, it doesn’t appear that it will be much longer.
Twitter is an unusually angry place. One reason is that the length limit makes people favor punchiness over tact. A less well-known reason is that in addition to notifying you when people like your own tweets, it gives a lot of notifications for people liking replies to you. So if someone replies to disagree, you will get a slow drip of reminders, which will make you feel indignant.
LessWrong is a relatively calm place, because we do the opposite: under default settings, we batch upvote/karma-change notifications together to only one notification per day, to avoid encouraging obsessive-refresh spirals.
I also thing there’s less engagement on LW.* While it might depends on the part of twitter, there’s a lot more replies going on. Sometimes it seems like there’s a 100 replies to a tweet, in contrast to posts with zero comments. This necessarily means replies will overlap a lot more than they do on LW. Imagine getting 3 distinct comments to a short post on LW, versus a thread of tweets, with 30 responses that mostly boil down to the same 3 responses that are being sent because people are responding without seeing other responses. (And if there’s hundreds of very similar responses, asking people to read responses is asking people to read a very boring epic.)
And getting one critical reply, versus the same critical reply from 10 people, even when it’s the same fraction of responses, probably affects people differently—if only because it’s annoying to see the same message over and over again.
*This could be the case (the medium probably helps) even if that engagement was all positive.
Some software costs money. Some software is free. Some software is free, with an upsell that you might or might not pay for. And some software has a negative price: not only do you not pay for it, but someone third party is paid to try to get you to install it, often on a per-install basis. Common examples include:
Unrelated software that comes bundled with software you’re installing, which you have to notice and opt out of
Software advertised in banner ads and search engine result pages
CDs added to the packages of non-software products
This category of software is frequently harmful, but I’ve never seen the it called out by the economic definition. For laypeople, about 30% of computer security is recognizing the telltale signs of this category of software, and refusing to install it.
I wonder what would be a non-software analogy of this.
Perhaps those tiny packages with labels “throw away, do not eat” you find in some products. That is, in a parallel world where 99% of customers would actually eat them anyway. But even there it isn’t obvious how the producer would profit from them eating the thing. So, no good analogy.
I’m trying to wrap my head around the negative price distinction. A business can’t be viable if the cost of user acquisition is lower than the lifetime value of a user.
Most software spend money on advertising, then they have to make that money back somehow. In a direct business model, they’ll charge the users of the software directly. In an indirect business model, they’ll charge a third party for access to the users or an asset that the user has. Facebook is more of an indirect business model, where they charge advertisers for access to the users’ attention and data.
In my mind, the above is totally fine. I choose to pay with my attention and data as a user, and know that it will be sold to advertisers. Viewing this as “negatively priced” feels like a convoluted way to understand the business model however.
Some malware makes money by trying to hide the secondary market they’re selling. For instance, by sneaking in a default browser search that sells your attention to advertisers, or selling your computers idle time to a botnet without your permission. This is egregious in my opinion, but it’s not the indirect business model that is bad here, it’s the hidden costs that they lie about or obfuscate.
User acquisition costs are another frame for approximately the same heuristic. If software has ads in an expected place, and is selling data you expect them to sell, then you can model that as part of the cost. If, after accounting for all the costs, it looks like the software’s creator is spending more on user acquisition than they should be getting back, it implies that there’s another revenue stream you aren’t seeing, and the fact that it’s hidden from you implies that you probably wouldn’t approve of it.
Ahhh I see, so you’re making roughly the same distinction of “hidden revenue streams”.
Q: Why did the chicken cross the road?
A: We both know that you don’t know of any specific chicken having crossed any specific road. Your question does not state a lie, but presupposes it. This would not be called out as a lie under ordinary social convention, but a deep commitment to truth requires occasionally flagging things like this.
Presuppositions are things which aren’t stated directly, but which are implied by an utterance because if they weren’t true, the utterance would be nonsensical. Presuppositions that aren’t truth-optimized can be surprisingly pernicious.
I have not only told you a joke, I have also told you that chickens live in contexts where they can interact with roads, are motile, and are capable of having motivations. I have told you these things in a way that bypasses some of your epistemic defenses. Some of these things are true, and some are false. I didn’t tell you these things on purpose, but they were communicated nevertheless.
I think that explicitly unpacking presuppositions, and spotting the questionable ones, is a foundational skill of rationality that I’ve never seen expressed. I also suspect some other techniques might be warped versions of this one. For example, there’s a lot of overlap between the ontology of frames, and what you get if you unpack presuppositions, but presupposition-truth is much more concrete.
(Crossposted with Facebook: link)
Fighting presuppositions instead of letting them develop in dedicated sandboxes hinders their understanding, makes communication unnecessarily difficult. The false dichotomy is between belief and dismissal of falsehood. There is also understanding of apparent falsehoods, which produces valuable gears that reassemble into unexpected truths.
Our past beliefs affect what we pay attention to, how we prioritize our skepticism, and how we interpret ambiguous evidence. This can create belief basins, where there are multiple sets of beliefs that reinforce each other, appear internally consistent, and make it hard to see the other basins as valid possibilities. On the topic of nutrition, I seem to have found myself in a different basin. I’ve looked through every nonstandard lens I could find, repeatedly applied skepticism, and firmly committed to not make the same mistakes everyone else is making (as a priority on par with not making mistakes at all). I’ve arrived at a set of beliefs that, as far as I can tell, is internally consistent, reasonably compelling from the inside, and completely contrary to what most other people in our culture think.
This makes for a difficult writing project. When I try to argue nonstandard positions, many of the arguments are tendrils reaching into other nonstandard positions. I’ve finally managed to get into a post-every-day cadence; part of the key to that was accepting that sometimes those arguments will be dangling references. Hopefully after a month of this, the whole thing will cohere. If not, well, the fragments are pretty interesting too.
The most-common basin of nutrition theorizing centers on obesity, and on a particular theory of obesity which goes like this. Food, especially modern processed food, tastes good and is appealing. Some people, if they followed their urges, would eat too much or become obese, so they have to exert self control not to. Weight is a function of calorie intake and calorie expenditure (“calories in, calories out”), and expenditure is primarily a function of behavior. So if someone is trying to lose weight, and it isn’t working, then they must be having willpower failures and eating more than they intend, or exercising less than they intend.
I currently think there are quite a lot of things wrong with this model, but today, I’d like to focus on one in particular. It’s not the only or the most central objection, nor is it a particularly actionable model fragment. But it’s an issue that’s important to me in particular, and it’s one of the wedges that moved into a different belief basin.
I am not obese, and have never set out to lose weight. But sometimes, I have overwhelming cravings for sugar. I would not be able to resist these cravings without great willpower.
If I ever did successfully resist one of those cravings, I would probably die.
I don’t mean this figuratively, or in a heart-disease-years-later sort of way. I mean that if I get a powerful craving for sugar, and I don’t promptly eat something that has sugar in it, then this will be a life-threatening medical emergency. This is because I have type 1 diabetes, and craving sugar is a symptom of low blood sugar, aka hypoglycemia. What T1 diabetes means, basically, is that I have to micromanage my blood sugar using insulin. Eating carbohydrates raises blood sugar, insulin lowers it, these need to be matched pretty precisely, and the whole thing is somewhat error prone. Too much insulin and blood sugar falls below 70mg/dL, and I get the sugar craving. I’ve never been below 40mg/dL, but people who do become mentally impaired, then lose consciousness, then die.
Under the usual theory of obesity, craving sugar would mean that I had been hijacked by the superstimulus of processed food, and that willpower was my defense against this hijacking. But actually, in this case, the craving is a safety mechanism. Sugar craving is to dangerous hypoglycemia as thirst is to dehydration.
With that example in mind, I started thinking about the double-digit percentage of people who drop out of weight-loss studies. And the much-larger percentage of people who start weight loss diets, privately resolved to continue until they reach a target weight, and stop early. What would happen to them, in the counterfactual world where the diet was enforced perfectly from outside, and willpower wasn’t an issue? Whether they would lose weight, seems like very much the wrong question to ask.
(Crossposted on Facebook)
You make a good point, that some people who drop out of weight-loss studies might have experienced health problems caused by the study, and quiting was the right decision for them.
But I believe that the average obese person in general population is not this case. There are many situations where people eat refined sugar not because they have a strong craving, but simply because it is easily available or there are even habits built around it.
To give an example, in my family it was for some reason considered a good idea to drink tea with sugar at breakfast. As a child I didn’t have an opinion on this, I was given the breakfast and I consumed it. But as I grew up and started making my own breakfast, out of sheer laziness I starting drinking water instead. I didn’t fall into coma and die. Actually it made the breakfast better, because when you drink tea with sugar first, then everything you eat afterwards tastes bland, but if you drink water, you discover that some things are surprisingly delicious. Recently my kids spent one week with my mother, and then reported to me that they had “cereals” for each breakfast (in this context, “cereals” refers to those cheap hypermarket products that contain the word on the box, but consist mostly of refined sugar with some added fibers; the advertisement tells you to pour milk on them and pretend that the result is healthy somehow, because, you know, milk and cereals). I am not making a big deal out of it, one week is not going to hurt anyone, but sigh, of course most people in my family are fat.
Similarly, if you buy things in a hypermarket, check how many of them contain added sugar. So people eat this sugar not because they had a craving, but because they bought a processed food in a shop, and someone added the sugar for them. (There is often no easily available sugar-less version.) They probably add sugar to your food in a restaurant, dunno.
If you are curious what would it be like to not eat any refined sugar, probably the only solution is to cook for yourself from scratch. Even things like mustard or canned vegetables typically contain refined sugar. So we regularly eat lots of sugar without deciding to, often without being aware of it. (And then we drink coke on the top of it. But hey, the advertisement said that coke had zero sugar now, could they possibly be lying?)
So, avoiding lots of extra sugar is technically possible, but it is a lot of work, and some people cannot afford it, or have never learned the necessary skills. Because of course they don’t teach cooking at schools anymore; why would anyone need such useless skill in the modern economy, where you can buy anything (but have little control over the content).
Lack-of-adblock is a huge mistake. On top of the obvious drain on attention, slower loading times everywhere, and surveillance, ads are also one of the top mechanisms by which computers get malware.
When I look over someone’s shoulder and see ads, I assume they were similarly careless in their choice of which books to read.
Note that many people don’t know about ad blockers:
(I highly recommend reading that entire section of the linked page, where gwern describes the results of several follow-up surveys he ran, and conclusions drawn from them.)
One day we will be able to wear glasses which act as adblock for real life, replacing billboards with scenic vistas.
And they will also be able to do the opposite, placing ads over scenic vistas
They will also send data about “what you looked at, how long” to Google servers, to prepare even better customized ads for you.
But people will be more worried about giant pop-up ads suddenly covering their view while they are trying to cross the street.
There’s an open letter at https://openletter.net/l/disrupting-deepfakes. I signed, but with caveats, which I’m putting here.
Background context is that I participated in building the software platform behind the letter, without a specific open letter in hand. It has mechanisms for sorting noteworthy signatures to the top, and validating signatures for authenticity. I expect there to be other open letters in the future, and I think this is an important piece of civilizational infrastructure.
I think the world having access to deepfakes, and deepfake-porn technology in particular, is net bad. However, the stakes are small compared to the upcoming stakes with superintelligence, which has a high probability of killing literally everyone.
If translated into legislation, I think what this does is put turnkey-hosted deepfake porn generation, as well as pre-tuned-for-porn model weights, into a place very similar to where piracy is today. Which is to say: The Pirate Bay is illegal, wget is not, and the legal distinction is the advertised purpose.
(Where non-porn deepfakes are concerned, I expect them to try a bit harder at watermarking, still fail, and successfully defend themselves legally on the basis that they tried.)
The analogy to piracy goes a little further. If laws are passed, deepfakes will be a little less prevalent than they would otherwise be, there won’t be above-board businesses around it… and there will still be lots of it. I don’t think there-being-lots-of-it can be prevented by any feasible means. The benefit of this will be the creation of common knowledge that the US federal government’s current toolkit is not capable of holding back AI development and access, even when it wants to.
I would much rather they learn that now, when there’s still a nonzero chance of building regulatory tools that would function, rather than later.
I’m reading you to be saying that you think on its overt purpose this policy is bad, but ineffective, and the covert reason of testing the ability of the US federal government to regulate AI is worth the information cost of a bad policy.
I definitely appreciate that someone signing this writes this reasoning publicly. I think it’s not crazy to think that it will be good to happen. I feel like it’s a bit disingenuous to sign the letter for this reason, but I’m not certain.
I think preventing the existence of deceptive deepfakes would be quite good (if it would work); audio/video recording has done wonders for accountability in all sorts of contexts, and it’s going to be terrible to suddenly have every recording subjected to reasonable doubt. I think preventing the existence of AI-generated fictional-character-only child pornography is neutral-ish (I’m uncertain of the sign of its effect on rates of actual child abuse).
Some people have a sense of humor. Some people pretend to be using humor, to give plausible deniability to their cruelty. On April 1st, the former group becomes active, and the latter group goes quiet.
This is too noisy to use for judging individuals, but it seems to work reasonably well for evaluating groups and cultures. Humor-as-humor and humor-as-cover weren’t all that difficult to tell apart in the first place, but I imagine a certain sort of confused person could be pointed at this in order to make the distinction salient.
I’m not sure that’s true. I think the second kind also uses April 1st as a way to justify more cruelty than usual.
Someone complained, in a meme, that tech companies building AI are targeting the wrong tasks: writing books, music, TV, but not the office drudge work, leading to a world in which the meaning-making creative pursuits are lost to humans. My reply to this is:
The order in which AI replaces jobs is discovered, not chosen. The problem is that most of the resources aren’t going into “AI for writing books” or “automating cubicle jobs”, they’re going into more-abstract targets like “scaling transformers” and “collecting data sets”.
How these abstract targets cash out into concrete tasks isn’t easy to predict in advance, and, for AI accelerationists, doesn’t offer many relevant degrees of freedom.
And, to the extent that money does go into these tasks per se, I’d bet that the spending is extremely imbalanced in the opposite way to what they assume: I’d bet way more money gets spent on tabular learning, ‘robotic process automation’, spreadsheet tooling, and so on than gets spent on Jukebox-like full music generation. (Certainly I skim a lot more of the former on Arxiv.) It’s telling that the big new music generation thing, almost 3 years after Jukebox is… someone jankily finetuning Stable Diffusion on ‘images’ of music lol. Not exactly what one would call an active field of research.
So there is a relevant degree of freedom where you can ~A C C E L E R A T E~ - it’s just the wrong one from what they want.
Companies deploying AI to do their office work would be poised to make them take aligment in a very serious way. Office work being wrong in a slight but significant way could be easy to imagine the relevance off, hard to detect and possibly nightmare to recover from.
It’s often said that in languages, the syllable-count of words eventually converges to something based on the frequency with which words are used, so that more-commonly-used concepts get words with fewer syllables.
There’s an important caveat to this, which I have never seen stated anywhere: the effect is strongly weighted towards vocabulary used by children, especially small children. Hence why “ma”, the lowest-entropy word, means mother in so many different languages, and why toddler-concepts are all monosyllables or twice-repeated monosyllables. So, for example. “probability” being 5 syllables isn’t just a reflection of it being used, it’s also a reflection of an (imo bad) societal judgement that people won’t be ready for that word until they’re adults.
1. How long are these words?
probably
likely
chance
maybe
most likely
more often than not
all the time
usually
often
sometimes
if
2. How often do you use probability?
3. Last but not least, have you forgotten
odds?
I don’t remember where, but I did see this stated previously, because it’s not new to me.
It’s not a societal judgement that kids aren’t ready for that word (though perhaps that too), but that it’s not necessary for them to survive. And, well, that seems to be true.
There is a rumor of RSA being broken. By which I mean something that looks like a strange hoax made it to the front on Hacker News. Someone uploaded a publicly available WIP paper on integer factorization algorithms by Claus Peter Schnorr to the Cryptology ePrint Archive, with the abstract modified to insert the text “This destroyes the RSA cryptosystem.” (Misspelled.)
Today is not the Recurring Internet Security Meltdown Day. That happens once every month or two, but not today in particular.
But this is a good opportunity to point out a non-obvious best practice around cryptographic key-sizes, which is this: Whatever key size is accepted as the standard, you want your SSH keys and your PGP keys to be one size bigger, so that if a gradually rising tide of mathematical advances causes a cryptography meltdown, you won’t be caught in the wave where everyone else gets pwned at once.
So I recommend making sure, if you’re using RSA for your SSH keys, that they are 4096-bit (as opposed to the current ssh-keygen default of 3072-bit).
While this sounds cool, what sort of activities are you thinking you need to encrypt? Consider the mechanisms for how information leaks.
a. Are you planning or coordinating illegal acts? The way you get caught is one of your co-conspirators reported you.
b. Are you protecting your credit card and other financial info? The way it leaks is a third party handler, not your own machine.
c. Protecting trade secrets? The way it gets leaked is one of your coworkers copied the info and brought it to a competitor.
d. Protecting crypto? Use an offline wallet. Too much protection and you will have the opposite problem.
Countless people—probably a substantial fraction of the entire population, maybe the majority—all their credit and identity records were leaked in various breaches. They have easily hackable webcams exposed on the internet. Skimmers trap their credit card periodically. And...nothing major happens to them.
COVID variants have mutated in the direction of faster spread and less immunity, as expected. They also seem to be mutating to higher disease severity, which was not expected. Why would that be, and should we expect this to continue?
My current theory is that the reason variants are more severe is because there’s evolutionary pressure on a common factor that affects both severity and secondary attack rate, and that factor is viral replication rate.
In the initial stage of an infection, the number of virus-copies inside someone grows exponentially. If the spike protein mutates to get slightly more reliable, then the doubling time of that growth shrinks. Result: Increased viral shedding, decreased minimum infectious dose, increased severity.
I hypothesize that we’re used to respiratory diseases evolving towards lower severity because, first, they’ve already maxed out this particular axis, and second, because they rely on people walking around in public while symptomatic, which they won’t do if the disease is too severe.
This retrodicts that data pretty well, but I’m still really unsure whether severity in the future is going to rise or fall. One one hand: there’s probably still room left to increase replication rate further, and the people-walking-around-if-mild transmission method isn’t going to come back. On the other hand, COVID might reach (or already be in) a local optimum with respect to spike-protein efficiency, in which case selection for immune-evasion will mean evolving away from the most-efficient version.
(Crossposted with Facebook)
I highly recommend reading something about mainstream research on this topic:
https://pubmed.ncbi.nlm.nih.gov/19196383/
Looks like there’s holiday-design discourse this week: https://astralcodexten.substack.com/p/a-columbian-exchange . Speaking as a veteran holiday designer (http://petrovday.com/), in my eyes, Columbus Day has already passed into the ranks of deprecated holidays. Not so much because Christopher Columbus was a bad person (though he was by all accounts quite terrible), but rather because no one has actually designed a rationality-culture version of it, and I find broad-American-culture holidays to be boring and uncompetitive.
Looking at Scott’s list figures who could be given holidays, I think Edison and Salk have promise, and I would be enthusiastic about a good reinvention of MLK day which was updated to the present day and had a good nuanced take on things.
But I really hate the idea of giving Henrietta Lacks a holiday. Not because cancer research isn’t important. Not because I think giving a cell culture is insufficiently virtuous, compared to the research itself. Rather, I think Henrietta Lacks should not be honored because I hold a grudge against HeLa, the microorganism that descends from her cancer. It is most famous not for its positive contributions to cancer research, but for invading and ruining cancer research on other cell lines. I don’t think this kind of Petri-dish-invasion imperialism should be celebrated!
(Crossposted with Facebook here.)
Every so often, I post to remind everyone when it’s time for the Periodic Internet Security Meltdown. For the sake of balance, I would like to report that, in my assessment, the current high-profile vulnerability Hertzbleed is interesting but does *not* constitute a Periodic Internet Security Meltdown.
Hertzbleed starts with the discovery that on certain x86-64 processors the bitwise left shift instruction uses a data-dependent amount of energy. Searching through a large set of cryptographic algorithms, they then find that SIKE (a cryptographic algorithm not in widespread use) has a data-dependent degenerate case in which a series of intermediate states are all zeroes, does some cryptanalysis, and turns this into a chosen-plaintext attack which creates a causal connection between the private key and the CPU’s throttling level.
This is pretty neat, and there may be similar attacks against other cryptographic algorithms, but I think it’s not going to amount to much in actual practice, because it has a constant-factors problem: it needs to heat up the target CPU and let it cool back down, and it only gets a tiny fraction of a bit of the private key each time. I haven’t done the analysis, but my expectation is that in more common situations (ie not SIKE), the amount of traffic required to extract a full key is going to be literally astronomical.
It’s yet another example of how infuriating computer security is, especially side-channel attacks. All that work into constant-time crypto, and then this… As the saying goes: “constants aren’t.”
On October 26, 2020, I submitted a security vulnerability report to the Facebook bug bounty program. The submission was rejected as a duplicate. As of today (April 14), it is still not fixed. I just resubmitted, since it seems to have fallen through the cracks or something. However, I consider all my responsible disclosure responsibilities to be discharged.
Once an Oculus Quest or Oculus Quest 2 is logged in to a Facebook account, its login can’t be revoked. There is login-token revocation UI in Facebook’s Settings>Security and Login menu, but changing the account password and revoking the login there does not work.
One practical impact of this is that if your Facebook account is ever compromised, and the attacker uses this vulnerability, they have permanent access.
The other practical impact is that if someone has unsupervised access to your unlocked Quest headset, and they use the built-in web browser to go to facebook.com, they have full access to your Facebook account, including Messenger, without having to do anything special at all. This means that if you’ve ever made a confidentiality agreement regarding something you discussed on Facebook Messenger, you probably can’t lend your headset to anyone, ever.
Additionally, the lock-screen on the Oculus Quest 2 does not have a strict enough rate limit; it gives unlimited tries at 2/minute, so trying all lock-screen combinations takes approximately 35 days. This can be done without network access, and can be automated with some effort. So if someone steals a *locked* Oculus Quest 2, they can also use that to break into your Facebook account. There is almost certainly a much faster way to do this involving disassembling the device, but this is bad enough.
Is your logic that releasing this heinous volun into the public is more likely to pressure FB to do something about this? Because if so, I’m not sure that LW is a forum with enough public spotlight to generate pressure. OTOH, I imagine some percentage of readers here aren’t well-aligned but are looking for informational edge, in which case it’s possible this does more harm than good?
I’m not super-confident in this model—eg, it also seems entirely possible to me that lots of FB security engineers read the site and one or more will be shouting ZOMG! any moment over this..
I’m posting here (cross-posted with my FB wall and Twitter) mostly to vent about it, and to warn people that sharing VR headsets has infosec implications they may not have been aware of. I don’t think this comment will have much effect on Facebook’s actions.
The Diamond Princess cohort has 705 positive cases, of which 4 are dead and 36 serious or critical. In China, the reported ratio of serious/critical cases to deaths is about 10:1, so figure there will be 3.6 more deaths. From this we can estimate a case fatality rate of 7.6/705 ~= 1%. Adjust upward to account for cases that have not yet progressed from detection to serious, and downward to account for the fact that the demographics of cruise ships skew older. There are unlikely to be any undetected cases in this cohort.
Hang on, maybe I’m being stupid, but I don’t get the 3.6. Why not say 36+4=40 serious/critical cases and the 10%=4 of them have already passed away?
You’re right, adding deaths+.1*serious the way I did seems incorrect. But, since not all of the serious cases have recovered yet, that would seem to imply that the serious:deaths ratio is worse in the Diamond Princess than it is in China, which would be pretty strange. It’s not clear to me that the number of serious cases is as up to date as the number of positive tests.
So, widen the error bars some more I guess?
How many passengers were exposed? Capacity of 2670, I haven’t seen (and haven’t looked that hard) how many actual passengers and crew were aboard when the quarantine started. So maybe over 1⁄4 of exposed became positive, 6% of that positive become serious, and 10% of that fatal.
Assuming it escapes quarantine and most of us are exposed at some point, that leads to an estimate of 0.0015 (call it 1⁄6 of 1%) of fatality. Recent annual deaths are 7.7 per 1000, so best guess is this adds 20%, assuming all deaths happen in the first year and any mitigations we come up with don’t change the rate by much. I don’t want to downplay 11.5 million deaths, but I also don’t want to overreact (and in fact, I don’t know how to overreact usefully).
I’d love to know how many of the serious cases have remaining disability. Duration and impact of survival cases could easily be the differences between unpleasantness and disruption that doubles the death rate, and societal collapse that kills 10x or more as the disease directly.
(This is a reply to the “Induction Bump” Phase Change video by Catherine Olsson and the rest of Anthropic. I’m writing it here instead of as a YouTube comment because YouTube comments aren’t a good place for discussion.)
(Epistemic status: Speculative musings I had while following along, which might be useful for inspiring future experiments, or surfacing my own misunderstandings, and possibly duplicating ideas found in prior work which I have not surveyed.)
The change-in-loss sample after the bump (at 19:10) surprised me. As you say, it seemed to get noticeably better at things that correspond to bigram rules (or bigrammish rules). I was expecting the improvement to be in something that, in some sense, requires metalearning; whereas (if I’m understanding correctly), a depth-1 network would be able to start learning these particular bigrams starting on the very first training step. If this does correspond to learning bigrams, what could explain the delay before learning these particular bigrams starts?
My speculative guess is that there’s a pattern where some learnable features have stronger gradients than others, and that training a multi-layer (but not a single layer) network proceeds in stages where first (a) multiple layer-1 nodes race to learn a function (loss falls rapidly), then (b) layer-2 nodes figure out which layer-1 nodes won the race and sever their connections to the ones that didn’t (loss falls slowly), and then (c) the freed-up layer-1 neurons move on to learning something else (loss falls rapidly again). Under this hypothesis, “the bump” corresponds to part of this sequence.
This seems to match what we see in the graph at 30:20. In that graph (which is training-step vs induction score for selected nodes in layer 1), some of the nodes go to 1, and some of the nodes start going to 1, then reverse direction shortly after the first set of nodes hits their asymptote. This is what you’d expect to see if the nodes that reverse direction were headed towards being duplicates of nodes that didn’t, but which lost the race and then got repurposed.
This seems like it might be a useful microcosm for studying the behavior of redundant nodes in general. (We know large networks have a lot of redundancy because SqueezeNets work). One major worry about neural-net transparency is that we might end up in a state where we can inspect neural nets and find things that are there, but that the methods that do that won’t be able to assert that things *aren’t* there. Concretely, this might look like finding nodes that track an unwanted concept, pinning or ablating those nodes, and then finding that the concept still presented due via duplicate nodes or via an off-basis aggregation of nodes that seem unrelated.