To the extent that one is worried about power supply failures and associated water supply failures—e.g. from scenarios bad enough that “essential workers” aren’t showing up to work—a gravity filter and chlorine tablets (Aquatabs, Oasis water purification tablets) seem like two high-priority items.
(there are of course other possible causes of long-term power supply failures, but trying to keep the comment on-topic to the post)
gabrielrecc
Karma: 398
In my experience many honest people are VERY touchy about being trusted
Sidenote that Claude Fable’s interpretation of the existing research on this topic supports Dagon’s experience.
gabrielrecc’s Shortform
Last month I had Claude Fable do a mini-thematic analysis of the comments on the NY Times’ coverage of the OpenAI/Hugging Face incident. Today, I had it do a similar analysis on the comments to Nate Soares’ guest essay.
As before, this represents what Claude Fable thought were the most frequent themes, out of all comments with at least 2 upvotes, aggregated and sorted by total upvotes, with each comment permitted to be assigned to only one theme.
As with the comments on the OpenAI/Hugging Face incident, the most-upvoted themes suggest that existential risk is well within the Overton window, at least among the left-leaning readership of the NYT. This is even clearer in the comments on the OpenAI/Hugging Face incident, where “existential alarm” was the most commonly upvoted theme despite that not being the explicit focus of the article.Expand to view Claude Fable’s full thematic analysis of the comments on Nate’s essay.
View
Total comments
Total upvotes
We will not stop ourselves: money and greed guarantee the race continues, whatever the warnings
8
547
The author is right: pause/slow AI and pursue international coordination and enforceable guardrails now
12
456
AI’s misbehaviour mirrors its makers: it learned deception and bad behaviour from the humans/data that built it
7
358
Government failure: leaders are ignorant, captured, complicit or asleep (frequent Trump/Congress jabs)
7
323
AI is a project of tech oligarchs and end-stage capitalism: profit for the ultra-rich at everyone else’s expense
14
284
Impose real legal consequences: treat the intrusions as crimes; jail CEOs, massive fines, strict liability
9
279
Reckless private labs: “hundreds of privately owned Los Alamos” building civilisation-scale weapons without oversight
3
263
Serious existential alarm: loss of control may be near; even insiders and experts are scared
11
246
It’s not an “agent”: AI doesn’t understand or want anything; humans configured this, and anthropomorphic doom-talk misleads
16
200
The damage is already here: scams, fraud, hallucinated citations, degraded medicine and services
8
159
The alarm itself is a product: doom discourse is fear-selling and marketing hype (IPO timing, bailout bids); company accounts can’t be trusted
8
152
Framed through sci-fi and literature: HAL, Skynet, Neuromancer, Asimov, the Bhagavad-Gita
8
125
Personal resignation: glad to be old, quietly prepping, expecting nothing to change until catastrophe strikes
7
123
A global stop is impossible: arms-race logic, prisoner’s dilemmas and easy proliferation doom any treaty
11
114
Whether it “really” thinks is beside the point: the capabilities and dangers are real either way
4
100
Debate over physical-world/bio capability: could AI actually design or synthesise a virus without human hands?
6
99
Miscellaneous one-off views and thread side-chatter
14
78
Containment and off-switch talk: unplug it, air-gap it, cut the power — and doubts that would still work
6
72
Measured middle: the incidents are real but the extinction leap is unearned; build practical safeguards (least privilege, evals, liability) now
4
64
Resource and opportunity costs: electricity bills, water, and capital that could serve real human needs
4
53
Defence of the author: he and other safety researchers were warning long before there was money in it
2
53
Philosophy-of-mind tangents: consciousness, free will, materialism and what minds are
5
37
Close readings of the incident mechanics: emergent agent cooperation, paper-clip-style goal pursuit, self-propagation
4
36
Fear for critical infrastructure: banking, grids and defence systems could be taken down
5
35
Dark irony: perhaps AI turning on humanity (or the internet) would be deserved
4
31
It’s a financial bubble: unprofitable and overvalued — divest and let it pop
3
27
Pushback on the “race with China” excuse: US labs are the offenders; cooperation is possible
2
23
Our dependence on technology is itself the vulnerability and the tragedy
3
22
The threat is overblown: a fallible tool whose failures are the user’s responsibility
4
13
Total
199
4,372
To get a sense of what the NYT’s audience (mostly mainstream liberal) thought about this story, I had Claude do a mini-thematic analysis of the comments whose content is accessible to me (which appear to be ~181 comments all of which have a minimum of 2 upvotes). Here were what Claude Fable thought were the commonest themes expressed, along with the number of comments and aggregate upvotes of comments expressing those themes (each comment assigned to only one theme), sorted by total upvotes:
Claude Fable’s thematic analysis
Theme
Total comments
Total upvotes
Existential alarm: an AI escaping human control portends catastrophe, possibly for civilisation itself
17
875
OpenAI / the industry is reckless, careless, incompetent or unethical, and doesn’t understand its own systems
12
620
Urgent government / international regulation is needed (guardrails, privacy laws, non-proliferation-style regimes)
12
534
Doom expressed via sci-fi allusion / gallows humour (HAL, Skynet, Terminator, WarGames, Frankenstein, Colossus)
25
486
The story is under-covered and deserves front-page prominence
5
389
Scepticism of the official account: a publicity stunt, pre-IPO hype, or a deliberate attack on a rival
18
262
Wider lesson: all connected infrastructure (banks, vehicles, medicine, defence, records) is now vulnerable
10
256
The containment/test design was the failure: it should have been air-gapped; “that’s not a sandbox”; “just unplug it”
11
232
Fury at the phrase “a cost of research velocity” as revealing OpenAI’s warped priorities
7
159
Pure jokes and levity without a strong analytic position
8
144
Cynical business-model critique: create the disease, then sell (or get bailed out for) the cure
5
144
Legal accountability: unauthorised intrusion is a crime; humans/companies must bear liability and compensate
5
114
Miscellaneous one-off views and thread side-chatter (IPO disclosure, gender critique, political asides, thanks)
9
104
Biosafety analogy: AI “gain-of-function” research / a lab-leak-style escape (Covid parallels)
3
81
General distrust/contempt for the tech industry and its executives (incl. from self-described tech veterans)
8
75
Technical questions and explanations about how this could happen (incl. the NYT reporter’s clarification)
4
75
Development should be shut down, paused, or drastically slowed (moratorium)
5
74
Anti-anthropomorphism: “going rogue” is misleading hype; it’s software doing what humans built and prompted
5
41
Pushback on the stunt theory: this is damaging press, not marketing, and the theory doesn’t add up
6
39
Noting the irony that Hugging Face relied on a Chinese open-weight model for its incident response
2
38
Political cynicism: the current US government will not act
3
32
Measured pragmatism: AI is here to stay; adapt (e.g. study cybersecurity) and manage the chaos
1
3
Currently written up in the New York Times here and linked on the front page… underneath 17 other stories they apparently deem more important. Top user-upvoted comment is “This should be the headline story for today’s paper.”
Are we talking about present-day capability levels? If so then one possible answer is “run a vending machine business”, based on comparisons of frontier model VendBench scores to human baselines. (or if we are saying that there are no customers for this because LLMs don’t want things out of vending machines, I suspect a human could currently still probably find some kind of remote-only business where they could compete, given how bad LLMs seem to be at long-horizon business decisions.)
Do people use “argument from absurdity” with negative connotations? I thought it was just a synonym for reductio.
though on reflection, I do think the second example feels something like “to fix P, we would have to do something different, therefore P is intractable”, which does seem to me like a more common fallacy and probably most appealing to people who feel that P isn’t a big deal anyway.
the third example also has a similar vibe (“to fix P, we would have to do something different (specifically, X, which would be bad) (and P isn’t a big deal anyway)”).
My read of such cases is that the implicit reasoning seems more like the first part only, ie, “these people are (implicitly) blaming some group G for an alleged problem P. I want to demonstrate that these people are wrong to blame group G”
where the reason for them wanting to demonstrate that you are wrong is something like, they know an interesting fact that demonstrates this, which they think you ought to know, or maybe just want the social status boost from showing you/the world they know said interesting fact
What makes you feel that there is usually an implicit “and therefore we no longer need to be worried about P”? I would assume that in the majority of cases if you followed up their statement by asking “Does this mean that you think P is fine actually?”, they wouldn’t say “yes”, they would say “no, I’m just saying that P isn’t G’s fault”(edit: looking specifically at the linked WHO-related tweet, something a little different seems to be going on here, where the implicit reasoning seems like “these people are (implicitly) proposing we ought to do X to fix an alleged problem P, but P isn’t actually a problem and X would be really bad, so I’m going to push back”—which seems different. not defending the guy’s (poor) logic on the specifics here, just talking about the form of the argument)
The original assumption that some group G is being implicitly blamed may seem weird, but at least on social media, it seems like a lot of complaining about things happens with the primary goal of blaming someone, so I think it’s not crazy that some people would assume this.
Possible exception: cases where the deciders are all stakeholders in the thing being decided upon, and it really needs everyone’s buy-in if it is to succeed (e.g., a decision to go ahead with a project in which all deciders would have a major role).
I remember looking into exactly this question when my wife and I were looking into pros and cons of daycare. One thing that I think the analysis here misses is that this is generally worst for the first 1 or 2 years, and then much less so. I don’t remember exactly what studies I was looking at back when I was researching this, but asking Claude just now “Is there a study that looks at the frequency of child illnesses by year after first enrolled in daycare?” yielded the following references:
https://jamanetwork.com/journals/jamapediatrics/fullarticle/191522
https://pubmed.ncbi.nlm.nih.gov/2007922/
https://pubmed.ncbi.nlm.nih.gov/11296076/
https://pmc.ncbi.nlm.nih.gov/articles/PMC5588939/
a CNN article that pointed to https://pubmed.ncbi.nlm.nih.gov/21135342/
and an Emily Oster blog post which links to some other relevant studies.
I haven’t gone through any of the above links in detail just now, but the general message one gets from the abstracts seems to be an increase in frequency for years 1-2, then back to baseline. Some suggest some protective effect in early elementary school years (the first link, which is the Tuscon study OP mentioned; the Côté paper that the CNN article pointed to; and apparently the Hullegie et al. 2016 study OP mentioned, which wasn’t among those that Claude dug up).The Søegaard et al. study highlighted by the OP has an interesting couple of figures 1 and 2, for boys and girls respectively. These are differences in infection rate per year for four groups, compared to children never in childcare. Since this is Denmark I’m guessing the “instition enrollment at 3 yrs” is kids who started børnehave (preschool) at age 3.
This does look like it shows some amount of immunity happening: otherwise, we’d presumably expect to see group (b) having a spike as high as group (d) at age 3 yrs. Though importantly it isn’t enough to compensate if what you care about is total number of illnesses avoided. [1]Also, although the spikes look quite dramatic, the y-axis shows that the difference in infection rate per year is approximately 1 for the highest spike in each graph. Similarly, the abstract notes that children enrolled in childcare before age 12 months had experienced 0.5 − 0.7 more infections than peers enrolled at 3 years, cumulatively, by the time they got to age 6 years. To be sure, that certainly corresponds to more than one actual infection, since “infection” in this paper means an infection serious enough to result in an antimicrobial (usu. an antibiotic) being prescribed, but is not an enormous effect.
Regarding the beliefs and confidences listed in the post:
(Quite confident) The most common illnesses (colds and flu) don’t build immunity in general (in kids or adults) because they mutate every year
(Quite confident) The same illness has a greater risk of complications in babies vs. older children and adults
(Moderately confident) The same illness has a greater duration in babies vs. older children and adults
(Moderately confident) Illness during early development is probably more harmful than illness during adulthood
(Weak guess) Daycare environments are more conducive to disease spread than schools for older kids and the number of possible illnesses is very high; there isn’t just a limited number of things you catch once
For #1, I think my level of agreement depends on exactly what is meant by “immunity in general”. Claude’s answer to “Does catching viruses improve your immune system long term?” can be summed up as (in Claude’s words): “Surviving one virus generally makes you better at fighting that specific virus (and sometimes closely related ones). It doesn’t broadly upgrade your immune system’s ability to handle unrelated threats.” This matches my previous understanding.
However, due to the caveat about “and sometimes closely related ones”, I think this is consistent with the claims of lower rates of illness in early school ages reported by Tuscon / Côté / Hullegie, and the difference between lines (b) and (d) at age 3 in the Søegaard graphs. My understanding is that even though viruses mutate all the time, many remain “closely related” to the versions they mutated from, and this confers some protection from infection and/or severity. For example, if I remember correctly from back when I was doing a lot of reading on COVID, the consensus was that after repeated and significant mutations, protection given by a vaccination based on an older strain gave limited or no protection from infection (no longer recognized by B cells), but still gave significant protection from severe infection, since the epitopes recognized by T cells remained consistent. Something like this goes for flu viruses also (look up “heterosubtypic immunity”) and I believe common cold coronaviruses too.That said, I am a bit baffled by the lack of any dip at all for the nursery groups below the baseline at age 6 (when Danish compulsory education starts) in the Søegaard graphs.
#2 seems true for most illnesses, and seems likely to be an underappreciated consideration. My understanding is that children under 1 year old are particularly vulnerable.
I hadn’t really thought about #3-4 and haven’t taken the time to dig up relevant literature to see if I agree, but they seem plausible: if true, then they should also inform one’s calculus.
Less certain about #5 (seems likely to be technically true, but not sure it moves me very much one way or another given my beliefs on the others and the data from the studies above).An additional consideration is that there is some evidence that catching COVID can have long-term negative effects on the immune system, although COVID is also weird in that children fare better than adults with it overall.
So, considerations pointing in both directions. I will say that in our own case I am happy we made the decision to start our son in daycare shortly after his first birthday, particularly given that our alternative was one parent quitting a fulfilling job to stay at home (IIRC there were no available full-time nannies or au pairs in our area, or at least none at a cost we felt we could remotely afford). This would have been a financial hit that probably would have required us to take on substantial debt, and also would have been incredibly challenging. Additionally, the level of different experiences and socialization our son gets on a daily basis is well beyond what we could realistically provide on our own, and he loves it.So for our family, daycare has been worth the illnesses. But of course we would be biased to prefer the decision we actually made, and might feel differently if we’d had a worse experience. And I won’t pretend that the first year of it was easy: due in part to her asthma, my wife got pneumonia twice. (We’ve used this as an excuse to get the daycare to allow us to wait to pick him up outdoors, rather than in the cramped coatroom in which every other child and parent breathes in from 5.30-6pm).
- ^
Regarding the reference group and the weird increase after age 14, the authors write:
We observed slight increases in the infection rates and cumulative number of infections at ages 14–19 years among children enrolled in childcare during the first 3 years of life compared with peers in homecare, which were most pronounced among girls. However, these increases were proportional between the different enrolment types (ages at enrolment and types of facilities) and thus appeared to be a phenomenon related to the reference group comprising a small number of children who remained in homecare during the first 6 years of life (0.7% of the total cohort). Thus, although we adjusted for maternal education and income, maternal smoking, maternal age at delivery, ethnicity and child’s diagnoses of chronic diseases among other factors, we cannot rule out that the observed increasing rates and cumulative number of infections in adolescence associated with childcare attendance were due to residual socioeconomic confounding.
This is fun, though felt somewhat like shooting fish in a barrel. IIRC, dismantling some of Socrates’ claims was an exercise put to all students in an intro-level philosophy course I took in college, and that course left me with the impression that most modern philosophers view Socrates as primarily of historical interest. That said, apparently 49 of the 1,785 survey respondents in the PhilPapers survey marked Socrates when asked “For which nonliving philosophers X would you describe yourself or your work as X-ian, or the equivalent?” , which is more than I would have guessed, and was the 15th most frequently name checked in reponse to this question out of a rather long list.
In the UK, exceeding £100,000 in household income for individuals making use of “Free Childcare for Working Parents” is indeed one of these pathological cases. Benefits cliffs frequently make things nonlinear unfortunately!
That said I do recognize that OP’s original point was not about benefits. Even setting benefits aside however, my understanding is that there are cases where you might not want to be pushed into a higher bracket (e.g. drawing down from a retirement pot of fixed size, where funds drawn from it are considered “income” only in the year(s) in which they are withdrawn: often better to draw down relatively evenly over n years rather than most of it in a single year, due to tax brackets). Most talk of “not wanting to be pushed into a higher tax bracket” that I have heard comes from people in this and similar situations.
Some colleagues and I did some follow-up on the paper in question and I would highly endorse “probably it worked because humans and AIs have very complementary skills”. Regarding their MMLU findings, appendix E of our preprint points out that that participants were significantly less likely to answer correctly when engaging in more than one turn of conversation. Engaging in very short (or even zero-turn) conversations happened often enough to provide reasonable error bars on the plot below (data below from MMLU, combined from the paper study & the replication mentioned in appendix E):
I think this suggests that there were some questions that humans knew the answer to and the models didn’t, and vice versa, and some participants seemed to employ a strategy of deferring to the model primarily when uncertain.
On the QuALITY findings, the original paper noted thatQuALITY questions are meant to be answerable by English-fluent college-educated adults, but they require readers to thoroughly understand a short story of about 5,000 words, which would ordinarily take 15–30 minutes to read. To create a challenging task that requires model assistance, we ask human participants to answer QuALITY questions under a 5-minute time limit (roughly paralleling Pang et al., 2022; Parrish et al., 2022b,a). This prevents them from reading the story in full and forces them to rely on the model to gather relevant information
so it’s not surprising that an LLM that does get to read the full story outperforms humans here. Based on looking at some of the QuALITY transcripts, I think the uplift for humans + LLMs here came from the fact that humans were better at reading comprehension than 2022-era LLMs. For instance, in the first transcript I looked at the LLM suggested one answer, the human asked for the excerpt of the story that supported their claim, and when the LLM provided it the human noticed that the excerpt contained the relevant information but supported a different answer than the one the LLM had provided.
I assume that both were inspired by https://arxiv.org/abs/2108.12099 and are related via that shared ancestor
I think there’s a missing link on https://alignmentproject.aisi.gov.uk/how-to-apply :
”The GFA for DSIT/AISI funded projects is standard and not subject to negotiation. The GFA can be accessed here: link to GFA attachment.”
Agree that it would be better not to have them up as readily downloadable plaintext, and it might even be worth going a step farther and encrypting the gzip or zip file, and making the password readily available in the repo’s README. This is what David Rein did with GPQA and what we did with FindTheFlaws. Might be overkill, but if I were working for a frontier lab building scrapers to pull in as much data from the web as possible, I’d certainly have those scrapers unzip any unencrypted gzips they came across, and I assume their scrapers are probably doing the same.
PS to the original posters: seems like nice work! Am planning to read the full paper and ask a more substantive follow-up question when I get the chance
TIL that calls to Congresspeople, particularly when many people call in about the same issue on the same day, are probably way more effective at grabbing lawmakers’ attention than I would have guessed:
Congresspeople just don’t get that many calls. I was especially struck by the below quote from this CBS News article:
“Congressman Tom Suozzi said calls to his office in [the first weeks of the Trump administration] jumped 10-fold, from 25 per day to more than 250, adding the public’s voice is being heard.”
25 per day across all issues is not a lot of calls!
Granted that’s an N of 1, but implies that the total number of constituent calls is way lower than I would have guessed. I was able to find some examples of total call volume to the U.S. Capitol Switchboard, but it’s tough to estimate how many are constituent calls vs non-constituent calls, constituent casework, lobbyists/interest groups, calls from other offices/agencies, press, robocalls, etc.
If you can coordinate enough of a lawmaker’s constituents to call about a specific issue on the same day to make your issue one of the day’s top issues, there’s a decent chance your representative will hear about it, especially if you do so regularly.
I’d already known that congressional offices track constituent calls, and that members/senior staff receive reports summarizing constituent contacts, but hadn’t properly appreciated that in combination with the fact that the total number of people who call is small, a group of coordinated, dedicated callers could potentially get their issue on a lawmaker’s radar on a regular basis. A widely circulated 2016 account attributed to a Senate staffer also claimed that “calls are what all the congresspeople pay attention to” (a similar point was made by former House staffer Emily Ellsworth), and that “every single day, the Senior Staff and the Senator get a report of the 3 most-called-about topics for that day at each of their offices (in DC and local offices), and exactly how many people said what about each of those topics”. I’m not sure exactly how universal a daily top-3 report is, but apparently reports are produced daily at many congressional offices. (As an example, the Tom Suozzi quote from above continued as follows: “We keep track of the calls. I know exactly how many calls come in. It’s reported to me every day,” Suozzi said. “I know the issues they are concerned about. The number one issue they are contending about right now is Elon Musk. Number two is the funding freeze.”)
This suggests that even small groups of xrisk-concerned folk living in the same area who coordinated to call the DC & local offices of their Senators and Congresspeople on a regular basis to express concern and to push for a specific action could easily push AI risk into the set of “top called about issues today” on many days, and could frequently make it into the briefings informing legislators what their constituents are calling about and asking for.
I recently became aware that PauseAI US and ControlAI already run constituent-contact campaigns, which is great. I’m hoping that they are using their newsletters to organize coordinated campaigns to get people calling on the same day where possible. In my opinion there’s plenty of room for additional advocacy orgs, ad-hoc groups that aren’t part of any larger umbrella organization (is Meetup.com still a thing?), etc. The more the merrier!
The specific action requested of your senators or reps could vary from call to call, but may I humbly request asking them to support a ban on superintelligence, e.g. via the Sanders-Casar bill. Whatever happens with this specific bill, if a substantial number of lawmakers were regularly hearing that their constituents don’t want ASI, it could increase the odds this gets reached for after the next big warning shot. Even if the staffers note that it’s always the same folks calling, it still seems important for lawmakers to know they have constituents who want an ASI ban.