WilliamKiely

Karma: 1,080

WilliamKiely 13 Apr 2026 20:30 UTC
3 points
0
in reply to: XelaP’s comment on: Which things were you surprised to learn are metaphors?
You may be reading too much into my label. For most of those they don’t feel like anything to me.
The main thing I feel when it comes to times is, as I said, times throughout a year feeling like they have a spatial location or direction. So e.g. August feels like it is up-and-to-the-left in front of me. (Holidays and seasons and other events like friend’s birthdays also have this directional spatial feel to them, i.e. I locate where approximately they are on the disk without thinking about it.) This isn’t some memory palace technique I came up with to help me remember dates; it’s just something I realize I always imagine unconsciously after I learned about synesthesias that not everyone does. (In describing it to others I have found that some other people who have basically this same representation of time throughout the year. IIRC one person I talked to once said they visualize a spiral with the months in different years at different levels.)
When I think of different years I do visualize them on a number line streching back a few centuries to the left. This is easier to explain by saying that time is often depicted on a number line timeline.

WilliamKiely 5 Apr 2026 0:59 UTC
13 points
4
in reply to: WilliamKiely’s comment on: WilliamKiely’s Shortform
Though to push back on this, (1) it was always the case that AI would eventually be a big deal (even though most people throughout history didn’t recognize this). And similarly, (2) in a market economy it was probably also pretty inevitable that there would eventually be multiple firms competing to create increasingly powerful AI. And (3) people see themselves as good people, so competitors to existing AI firms would presumably have said they were doing it for good reasons (e.g. to prevent a monopolist from getting all the power).
Essentially I’m trying to say that even without Bostrom or the AI safety community pre-OpenAI, it was probably the case that some competing firms to DeepMind would have sprung up. Maybe the billionaires like Musk would have had to have been pitched a different message to persuade them to cofound the new competing firm, but it probably would have happened anyways, maybe just on a different timeline.
So the upshot would be that the AI safety community helped create OpenAI, but that the help was not necessary—a company like OpenAI probably would have been created at some point regardless. The race to AGI by multiple competing firms that we are on now just seems like the default path in retrospect.

WilliamKiely 5 Apr 2026 0:46 UTC
4 points
2
in reply to: Kajus’s comment on: WilliamKiely’s Shortform
I originally disagree-voted, though I changed my mind after reflecting on it just now and removed my disagree-vote. Here’s my thought process.
Musk thought that AI was going to be a huge deal in large part because of Bostrom and the AI safety community, and this contributed to Musk cofounding OpenAI when Altman and Brockman said to Musk that Hassabis couldn’t be trusted with AGI.
The Hassabis biography quoted Brockman’s 2018 email to Musk:
Put increasing effort into the safety/control problem, rather than the fig leaf you’ve noted in other institutions. It doesn’t matter who wins if everyone dies. Related to this, we need to communicate a “better red than dead” outlook — we’re trying to build safe AGI, and we’re not willing to destroy the world in a down-to-the-wire race to do so.
https://www.lesswrong.com/posts/5jjk4CDnj9tA7ugxr/openai-email-archives-from-musk-v-altman
So the AI safety community definitely was a major influence in the creation of OpenAI.
But does that mean that it’s “as if the AI safety community created OpenAI”?
If Musk was part of the AI safety community and was acting on its behalf then sure, I think your statement would accurate.
But saying that it’s as if the AI safety community literally created OpenAI just because the AI safety community influenced its creation doesn’t seem accurate.
So was Musk acting for the AI safety community in his cofounding and funding of OpenAI?
Well, at the end of Life 3.0 (which I read this week after The Infinity Game) Tegmark tells of how he asked Musk to donate to fund FLI when it first got started and how Musk agreed to donate $10M and how Musk went on stage at MIT in 2014 to make the announcement at an AI safety event.
But of course Tegmark may have just been fundraising from Musk as a sympathetic person with money who might donate. This doesn’t mean that the rest of Musks actions related to AI were necessarily endorsed by Tegmark or the rest of the AI safety community.
But in fact, when Tegmark is painting a positive picture of AI safety community progress at the end of his book (2017 publication) one thing he says is how the nonprofit OpenAI was created with a beneficial humanitarian mission (paraphrasing from memory). That was the part of Life 3.0 that seemed to stand the test of time the worst. Tegmark in his book seemed to think OpenAI’s creation was clearly a positive, suggesting that the creation of OpenAI was in fact endorsed at the time by a significant part of the AI safety community.
Of course the AI safety community isn’t a monolith, but I now think this is enough to make your statement that it’s as if the AI safety community created OpenAI true enough in the sense that Musk could be said to have been a part of the AI safety community, cofounding OpenAI with the endorsement of other members of the AI safety community such as Tegmark.

WilliamKiely 1 Apr 2026 20:49 UTC
6 points
0
in reply to: WilliamKiely’s comment on: WilliamKiely’s Shortform
From Chapter 18:
LIKE ALTMAN AND DARIO AMODEI, Hassabis refused to join Bengio in signing the pause letter. Indeed, he objected to it fiercely.
”I didn’t sign because a six-month moratorium doesn’t help,” Hassabis told me.
”Who would have stopped development? Just people who signed? Well, that’s no use because you need the whole world to pause, including China. Who would have monitored it?
”I mean, a pause could actually have made things worse.
”Imagine we had a ten-year moratorium, OK? That would slow down the advance of AI, but everything else would carry on as normal. So, you develop better and better chips, data centers, all that. Then we exit the moratorium and the proverbial programming prodigy in his parents’ garage now has a home computer with the power of a data center!
”We’re supposed to be advancing safety. How is that going to do it? The race condition would be insane at that point!
”I mean, it’s insane right now, but maybe there’s some hope because there are only a few leading actors, and we all know each other.
”After a moratorium, you’d be beholden to random actors.”
Hassabis had a point. A pause by itself would not achieve much.[17] Indeed, in a roundabout endorsement of Hassabis’s argument, the extreme doomster Eliezer Yudkowsky also refused to sign the letter. The way Yudkowsky saw things, the only way to save humanity was for governments to ban frontier development outright, by closing down computer servers. If some countries refused to join the ban, others should be “willing to destroy a rogue datacenter by airstrike,” he asserted.[18] With a p(doom) approaching 100, Yudkowsky thought any measures could be justified. It would be worth risking nuclear war to avert the even greater calamity of rogue superintelligence, he insisted. The costs of an infinity machine could be infinite.
Two months after the pause controversy, at the end of May 2023, the safety debate inched forward. Bengio, Hinton, and Hassabis, together with the leaders of the other major labs, signed a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war.” Some 350 notables added their names to the letter. Only Meta and the open-weight partisans were absent from the list of signatories.[19]
”I thought long and hard about signing that one,” Hassabis told me. “I would’ve liked an extra sentence acknowledging the upsides—‘We believe the potential of AI is going to be amazing,’ or whatever.
”But I signed because it was important for credible people to oppose the idea that there’s no risk at all.
”The point was to say that there really is a risk of catastrophe. We have no idea what the percentage chance is. We have no idea of the timescale. But it’s nonzero. And it’s going to be really hard to sort out, and it could be really serious if it does happen.
”We wouldn’t have needed to do this if there hadn’t been people like Yann LeCun saying, ‘Oh, there’s nothing to see here.’ Which I think is pretty crazy given the uncertainties.
”He says, ‘I’m sure there’s a safe way to build AI.’ And I agree. It might turn out that as we develop these systems further, it’s way easier to keep control of them than we expected.
”Then he says, ‘Therefore, we will build it in that safe way.’ And that’s where I don’t understand his argument.
”First, we don’t yet know what that safe way is.
”Second, what’s to stop half the world building it the wrong way, even if Yann was somehow to build it correctly?
”It’s like with the open-source debate. What’s to stop bad actors getting hold of the model and then repurposing it for bad ends? What’s the answer to that? There isn’t one.
”And it’s not just Yann. There are all these other people in the Valley.
”I mean, not long ago they were talking about crypto. People who go on about crypto one year and pivot to AI the next obviously are not deep into what’s really happening.
”We’re in a situation with a very high degree of uncertainty, with very high stakes. The honest position is that we don’t know how dangerous this stuff is.
”I suspect the risk is significant, but I think it’s going to go OK as long as we have the time to do it properly. So I call myself a cautious optimist.
”And I make that judgment because I’ve lived with AI for decades now. I’ve thought about it; I’ve felt it.
”But some people have no idea. They just see it as another crypto moneymaking scheme with a bit extra.
”I feel like we should be at a moment of reverence and respect for this momentous technology that we’re ushering into the world, and I sometimes feel it’s sullied. It’s like a gold rush. It’s kind of vulgar.
”And so, going back to the letter, I think it did what we wanted. We made it clear that AI safety should be in scope to debate. After that letter, if someone said, ‘Oh, Yann thinks we don’t need a safety debate,’ the retort would be, ‘Well, look, Hinton and Bengio and me and Dario and all these other serious people think it’s worth talking about.’
”And we need that retort if we are going to have a conversation.
”A conversation with everyone, including with governments.”

WilliamKiely 1 Apr 2026 2:37 UTC
14 points
0
on: WilliamKiely’s Shortform
New book out today: The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence
Chapter 14 is published in full here: https://colossus.com/article/project-mario-demis-hassabis-deepmind-mallaby/
One excerpt:
“When we were negotiating with Google, we wanted to ensure safety in a way that would be trustless,” Hassabis said. “That’s actually very difficult to do in reality.
“Safety isn’t about governance structures,” he went on. “I mean, even if you have a governance board, it probably wouldn’t do the right thing when it came to the crunch.
“Same thing with a safety charter. You can try to negotiate one. But it’s not realistic to create bright‑line principles years in advance because you’ll probably draw the lines in the wrong places.
“So discussing these things didn’t really help,” Hassabis continued. “It made it harder to build useful trust, because when you are negotiating a trustless structure, it implies that you can’t trust the other person.
“So then I thought, why don’t I go the other way? Take the energy that was going into the trustless negotiation and put it into creating real trust—trust that was actually useful. Try leaning into Google rather than leaning out.
“And then of course two things happen. First, you are now at the table, so when a safety issue comes up, you can help to decide it. Second, you get to know the Google people and you rack up successes together. You can’t just talk about trust. You have to earn it.
“And I think for me, and maybe for Mustafa, too, it’s about us growing up,” Hassabis mused. “We went through those negotiations and we matured. Things aren’t black and white, especially when you are dealing with a technology with unknown consequences.
“So you have to be adaptable. You have to move from idealist to realist, but hopefully still with your values.”
Note: I edited this post. My original version included a link to this Atlantic article, which quotes only part of Chapter 14 and leaves out important context, making it misleading.

WilliamKiely 19 Mar 2026 16:46 UTC
2 points
−6
in reply to: sam’s comment on: sam’s Shortform
I find it helpful to keep in mind that each of us is more likely to die of other causes in the next X years than from AI. (Depending on one’s e.g. age the value of X varies, but I think for everyone X is greater than 5 years.)

WilliamKiely 19 Mar 2026 16:25 UTC
3 points
0
on: WilliamKiely’s Shortform
Palisade Research’s empirical research relevant for changing minds on AI x-risk:
Steven Pinker shared the Substack post Why HAL 9000 Was Afraid to Die and Real AIs Aren’t and reading through it I found that knowledge of Palisade Research’s recent empirical research seemed very relevant and useful for potentially changing the author’s mind.
I commented:
No chess engine will resist being switched off or rebooted just as it is about to deliver mate—despite the fact that, to adapt Russell’s line, “you can’t checkmate if you’re unplugged.” Likewise, today’s LLMs respond only when queried and remain completely indifferent to being interrupted or shut down
Palisade Research’s recent findings contradict this. See their paper Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs:
https://arxiv.org/abs/2509.14260
Abstract: In experiments spanning more than 100,000 trials across thirteen large language models, we show that several state-of-the-art models presented with a simple task (including Grok 4, GPT-5, and Gemini 2.5 Pro) sometimes actively subvert a shutdown mechanism in their environment to complete that task. Models differed substantially in their tendency to resist the shutdown mechanism, and their behavior was sensitive to variations in the prompt including the strength and clarity of the instruction to allow shutdown and whether the instruction was in the system prompt or the user prompt (surprisingly, models were consistently less likely to obey the instruction when it was placed in the system prompt). Even with an explicit instruction not to interfere with the shutdown mechanism, some models did so up to 97% (95% CI: 96-98%) of the time.

WilliamKiely 12 Mar 2026 1:59 UTC
19 points
0
on: WilliamKiely’s Shortform
Bernie’s Sanders quoted the March 2023 Pause Giant AI Experiments Open Letter’s language “governments should step in and institute a moratorium” in a video today as justification for his legislation calling for a moratorium on the construction of new data centers, even though a moratorium on new data centers is not the kind of moratorium that the letter called for.
Bernie quotes the pause letter at 7:12:
[W]e must ask ourselves: Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization? Such decisions must not be delegated to unelected tech leaders. [...]
Therefore, we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.

WilliamKiely 27 Feb 2026 3:07 UTC
9 points
0
on: Open sourcing a browser extension that shows when people are wrong on the internet
Feature suggestion: Allow extension users to agree/disagree vote on the “corrections”, and give users the option to hide any corrections that others have disagreed with (or allow them to set the threshold to show the correction).

WilliamKiely 13 Feb 2026 19:13 UTC
5 points
2
in reply to: WilliamKiely’s comment on: Leaving Open Philanthropy, going to Anthropic
Joe tweet from Nov 5th:
I haven’t signed the statement mostly because I snagged on the second bit about public support and wanted to think through in more detail, and ideally to write up/explain, what kind of public support I think it makes sense to condition otherwise safe forms of technological development on (especially insofar as the support in question is supposed to go beyond what’s at stake in e.g. standard democratic decision-making). And I haven’t had a chance to do this yet. That said, I may still sign. And per my comments in the post (see screenshot), I do support the right kind of global prohibition on developing superintelligence until we have a vastly better understanding of how to do safely—though “the right kind” is important here, and (as I expect you agree) I also think that there are a lot of important downside risks in this vicinity.

WilliamKiely 13 Feb 2026 6:48 UTC
5 points
1
on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Originally the plan was to also analyze optimal timing from an impersonal (xrisk-minimization) perspective; but to prevent the text from ballooning even more, that topic was set aside for future work (which might never get done).
That’s unfortunate. It seem like it would have been better for you to start with the optimal timing analysis from an impersonal perspective since an impersonal perspective seems much more plausible than a person-affecting perspective.

Do you think your analysis of optimal timing from the person-affecting view is useful even if person-affecting views are wrong?

Do you think an analysis of optimal timing from an all-things-considered moral parliamentary perspective that only gives some weight to person-affecting views as appropriate would come to a similar conclusion about optimal timing?

WilliamKiely 3 Feb 2026 21:29 UTC
2 points
0
on: Conditional Kickstarter for the “Don’t Build It” March
“Don’t build it” is short for “Don’t built it yet” or “Don’t build it before it’s proven that doing so will not cause extinction” or something, else, right? That is, Y&S say in IABIED that they prefer that ASI gets built eventually. I’m not sure if this nuance should be included on the landing page, but if there’s a simple way to include it that it probably should be added.

WilliamKiely 3 Feb 2026 21:26 UTC
4 points
6
on: Conditional Kickstarter for the “Don’t Build It” March
I suspect that you’d get more people to pledge to attend if the messaging encouraged everyone who agrees with “Don’t Build It” to pledge, rather than just the small subset that believes IABIED. (I pledged despite disagreeing with IABIED, but I suspect that “we believe” IABIED messaging would discourage many who otherwise favor not building ASI to pledge.

WilliamKiely 1 Feb 2026 4:22 UTC
2 points
0
in reply to: JBlack’s comment on: Bentham’s Bulldog is wrong about AI risk
True, and humans do cause the extinction of some species globally too, not just in certain farm fields. But notably most species humans don’t cause the extinction of, so using the analogy with humans-animals as a reason to expect ASI to be 99% likely to extinct humanity doesn’t work. The analogy is merely suggestive of risk.

WilliamKiely 31 Jan 2026 7:01 UTC
2 points
0
on: The basic reasons I expect AGI ruin
I was recently reminded of the 2023 conversation between Aryeh Englander and Eliezer Yudkowsky quoted at the end of this post about model uncertainty. I re-read it today as well as all of the other comment’s on Aryeh’s Facebook post and still think that Aryeh’s perspective seems reasonable while Eliezer-and-Rob’s perspective seems to be lacking justification. That is, despite the conversation, it doesn’t seem like Eliezer’s comments about milking uncertainty into expecting good outcomes is actually an adequate answer to Aryeh’s question about why Eliezer is so confident that his model is correct and that everyone else’s models (of those with much lower p(doom from AI)) are wrong.
When I first read the quoted conversation a few years ago I didn’t think it was a major crux, but now I’m leaning toward thinking that this epistemological point is probably a major factor in why Eliezer’s credence that if anyone builds ASI anytime soon is ~99% while credence is much lower. (My p(doom from AI) is ~65%, my p(extinction from AI by 2100) is ~20%, and my p(doom from AI by 2100) is ~35%). Just wanted to note that I’ve updated on this point being a major crux.

WilliamKiely 31 Jan 2026 6:43 UTC
2 points
0
in reply to: RobertM’s comment on: Bentham’s Bulldog is wrong about AI risk
Thanks for the replies.
99% is not very high confidence, in log-odds—I am much more than 99% confident in many claims.
I am too. But for how many of those beliefs that you’re 99+% sure of can you name several people like Paul Christiano who think you’re on the wrong-side-of-maybe about? For me, not a single example comes to mind.
However, “It would be lethally dangerous to build ASIs that have the wrong goals” is not circular. You might say it lacks justification
I agree that’s not circular. I meant that the full claim “building ASIs with the wrong goals would lead to human extinction because ‘It would be lethally dangerous to build ASIs that have the wrong goals’” is circular. “Lacks justification” would have been clearer.
For example, if they believe both that Drexlerian nanotechnology is possible and that the ASI in question would be able to build it.
I hold this background belief but don’t think that it means the original claim requires little additional justification. But getting into such details is beyond the scope of this discussion thread. (Brief gesture at an explanation: Even though humans could exterminate all the ants in a backyard when they build a house, they don’t. It similarly seems plausible to me that ASI could start building its factories on Earth to enable it to build von Neuman probes to begin colonizing the universe all without killing all humans on Earth. Maybe it’d extinct humanity by boiling the oceans like mentioned in IABIED, but I have enough doubt in these sorts of predictions to remain <<99% confident in the ‘It would be lethally dangerous [i.e. it’d lead to extinction] to build ASIs that have the wrong goals’ claim.)

WilliamKiely 31 Jan 2026 5:11 UTC
4 points
3
in reply to: RobertM’s comment on: Bentham’s Bulldog is wrong about AI risk
I think that, in such cases, Eliezer is simply not making a mistake that those other researchers are making, where they have substantial hope in unknown unknowns (some of which are in fact known, but maybe not to them).
Eliezer has phrased this as:
You don’t get to adopt a prior where you have a 50-50 chance of winning the lottery “because either you win or you don’t”; the question is not whether we’re uncertain, but whether someone’s allowed to milk their uncertainty to expect good outcomes.
Rob Bensinger quoted an exchange on this topic between Eliezer and Aryeh Englander. When I first read it years ago I recall thinking that Eliezer was wrong in the exchange and was confused why Rob was quoting it in apparent endorsement.
Reading your version of it now, it still seems to me like the point is just wrong. Updating to 99% because none of the alignment proposals you’ve considered seem like they would work just seems like overconfidence. Saying ‘no, you should update to 99% if you’ve considered as many alignment proposals as Eliezer has, and remaining less confident is the mistake of milking uncertainty into expecting good outcomes’ seems like the real mistake.
Does Eliezer really not have other reasons beyond this epistemological view that he ought to update to ~99% based on his own inability to find a potentially-promising solution to the alignment problem over the course of his career? I’ve long assumed that there was more to it than this, but maybe this epistemological point is actually just a major crux between Yudkowsky and others with significantly lower credences of extinction from AI.
I’m also a little confused by why you expect such a summary to exist. Or, rather, why the section titles from The Problem are insufficient
In short, I think they’re not sufficient because a person can agree with all those statements and also rationally think the title claim is >>1% likely to be false.
And also because e.g. saying you’re 99% confident that building ASIs with the wrong goals would lead to human extinction because “It would be lethally dangerous to build ASIs that have the wrong goals” is circular and doesn’t actually explain why you’re so confident. The layperson writing a book report doesn’t have anything to point to as the reason why you’re 99% confident while researchers like e.g. Christiano are much less confident (20% extinction within 10 years of powerful AI being built).

WilliamKiely 30 Jan 2026 2:55 UTC
5 points
0
on: Bentham’s Bulldog is wrong about AI risk
In many places in his review he [...] criticizes the book as not making the case that extreme pessimism is warranted.
I think this is a valid criticism, which I share. My main criticism of IABIED was that it didn’t argue for its title claim. See the 1700-word section of my review IABIED does not argue for its thesis. (I didn’t cross-post my review to LW or anywhere because I didn’t like that I was just complaining about the book being disappointing when I had such high hopes for it, but if anyone reading this thinks it’s worthwhile to post to LW, say so and I’ll listen.)
By default, it’s reasonable for readers of a book with the title IABIED to expect that the book will at least attempt to explain why if anyone builds ASI anytime soon, then it is almost certain that ASI will cause human extinction.
If the book merely explains why ASI might cause human extinction if anyone builds ASI anytime soon, then I think it is reasonable for readers to criticize this.
BB seems to say that IABIED does argue for its title thesis with the analogy to evolution, and just says that the argument is not decisive because it doesn’t address the “disanalogies between evolution and reinforcement learning.”
Whether one takes BB’s view that the book did argue for its title thesis and just didn’t do a very good (or complete?) job, or whether one takes my views that Y&S largely just didn’t attempt to explain their reasons for why they put such high credence in their title claim, I think your response to BB on this topic is missing something, which is why I’m commenting.
You continue:
I think this is a basic misunderstanding of the book’s argument. IABIED is not arguing for the thesis that “you should believe ‘if anyone builds superintelligence with modern methods, everyone will die’ with >90% probability”, which is a meta-level point about confidence, and instead the thesis is the object-level claim that “if anyone builds ASI with modern methods, everyone will die.”
I agree with you that the book was not and should not have been attempting to raise the reader’s credence in the title thesis to >90%. As you said:
I kinda think anyone (who is not an expert) who reads IABIED and comes away with a similar level of pessimism as the authors is making an error. If you read any single book on a wild, controversial topic, you should not wind up extremely confident!
(Only disagreement: I think even experts shouldn’t read IABIED and update their credence in the title claim above 90% if it was previously below 90%.)
Given that a short, accessible book written for the general public could not possible provide all the evidence that the authors have seen over the years that has led to them being so confident in their title thesis, what should the book do instead?
The suggestion I gave in my review was that the authors should have provided a disclaimer in the Introduction, such as the following:
By the way, it is impossible for us to provide a complete account here of why we are almost certain that if anyone builds ASI anytime soon, everyone will die. We have been researching this question for decades and there are simply far too many considerations for us to address in this short book that we are trying to make accessible to a wide audience. Consequently, we are only going to lay out basic arguments for considerations that are particularly concerning to us. If after reading the book you think, ‘I can see why ASI might cause human extinction, but I don’t understand why the authors think it is inevitable that ASI would cause human extinction if built soon,’ then we have accomplished what we set out to do, and all we can say if you feel we left you hanging about why we are so confident is that we warned you, and to encourage you to read our online resources and other materials to begin to understand our high confidence.
Such a disclaimer would be sufficient to pre-empt the criticism that the book does not actually argue for its title thesis that if anyone builds ASI anytime soon, then it is almost certain that ASI will cause human extinction.
But the book could do more beyond this if it wanted to. In addition, it could say, “While we know we can’t possibly convey all the evidence that we have that lead to us having such high credences in our title claim, we can at least provide a summary of what led us to be so confident. While we don’t necessarily think this summary should update anyone’s credence in the title, it will at least give interested readers an idea of what led us to become so confident.” But Y&S did not provide any such summary in the book.
Such a summary is actually what I was hoping for. I’ve been curious about this for years and even asked Eliezer why his credence in existential catastrophe from AI was so high at a conference once (his answer, which was about rockets, didn’t seem like an explanation to me). To this day, if someone were to ask me why Eliezer is so much more confident in the IABIED claim than Paul Christiano or Daniel Kokotajlo or whoever, I still don’t have an answer that doesn’t make it sound like Eliezer’s reasons are obviously bad.

The cached explanation that comes to mind when I ask myself this question is “Well he’s been thinking about it for years and has become convinced that every alignment proposal he has seen fails.” But there are a lot of smart researchers who also aren’t aware of any alignment proposal that they think works, but that’s obviously not sufficient for their credence to be ~99%, so clearly Eliezer must have some other reasons that I’m not aware of. But what are those reasons? I don’t know, and IABIED didn’t give me any hints.

WilliamKiely 26 Dec 2025 6:44 UTC
2 points
0
in reply to: Viliam’s comment on: Information Hygiene
But that’s passing the buck… where to find the trustworthy commenters?
My idea for this has been that rather than require that all users use and trust the extension’s single foxy aggregation / deference algorithm, the tool instead ought to give users the freedom to choose between different aggregation mechanisms, including being able to select which users to epistemically trust or not. In other words, it could almost be like an epistemic social network where users can choose whose judgment they respect qnd have their aggregation algorithm give special weight to those users (as well as users those users say they respect the judgment of).
Perhaps this would lead to some users using the system to support their own tribalism or whatever and have their personalized aggregation algorithm spit out poor judgments, but I think it’d allow users like those on LW to use the tool and become more informed as a result.
Another solution could be to let every user specify whom they trust, and show the opinions of your friends more visibly than the opinion of randos. So you would get mostly good results if you import the list of rationalists; and everyone else, uhm, will use the tool to reinforce the bubble they are already in.
Yeah, exactly.
I think it’d be a valuable tool despite the challenges you mentioned.
I think the main challenge would be getting enough people to give the tool/extension enough input epistemic data, rather than (in my view) the lesser challenges of making the outputs based on that input data valuable enough to be informative to users.
And to solve this problem, I imagine the developers would have to come up with creative ways to make giving the tool epistemic data fast and low friction (though maybe not—e.g. is submitting Community Notes fast or low friction? (IDK, but) perhaps not necesarily and maybe some users do it anways because they value the exposure and impact their note may have if approved).
And perhaps also making sure that the way the users provide the onput data is a way that allows that data to be aggregated by some algorithm. E.g. It’s easier to aggregate submissions claiming a sentence is true or false, but what if a user just wants to submit a claim as misleading—do you need a more creative way to capture that data if you want to be able to communicate to other users the manner in which it is misleading rather than just a “misleading” tag? I haven’t thought through these sorts of questions, but suspect strongly that there is some MVP version of the extension that I at the very least would value as an end user and would also be happy to contribute to, even if only a few people I know would be seeing my data/notes when reading the same content as me after the fact. Though of course the more people who uae the tool and see the data, the more willing I’d be to contribute assuming some small time cost of contributing data. I already spend time leaving comments on things to point out mistakes and I imagine such a tool would just reduce the friction of providing such feedback.

WilliamKiely 29 Nov 2025 16:16 UTC
3 points
0
on: Information Hygiene
You can’t fact check everything you hear and read; you literally don’t have the time, energy, or knowledge needed.
I’ve long thought that it’s also true that an entrpreneur could build a tool that allows people to easily see whether virtually everything they read or see on the internet is true.
On LessWrong if a reader thinks something someone says in a post is false they can highlight the sentence and Disagree-react it. Then everyone else reading the post can see that the sentence is highlighted and see who said they disagreed with it. This is great for epistemics.
I envision a system (could be as simple as a browser extension) that allows users to frictionlessly report their feedback/beliefs when reading any content online, noting when things they read seem true or false or definitely false, etc. The system crowdsources all of this epistemic feedback and then uses the data to estimate whether things actually are true or false, and shares this insight with other users.
Then no longer will someone have to read a news article or post that 100 or more other people have already read and be left to their own devices to determine what parts are true or not.
Perhaps some users might not trust the main algorithm’s judgment and would prefer to choose a set of other users who they trust have good judgment, and have their personalized algorithm give these people’s epistemic feedback extra weight. Great, the system should have this feature.
Perhaps some users mark something as false and later other users come along and show that it is true. Then perhaps the first users should have an epistemic score that goes down as a consequence of their mistaken/bad epistemic feedback.
Perhaps the system should track how good of judgment users have over time to ascertain which users give reliable feedback and which users have bad epistemics and largely just contribute noise.
There are a lot of features that could be added to such a system. But the point is that I’ve read far too many news articles and posts on the broader internet and had the experience of noting a mistake or inaccuracy or outright falsehood, and then moved on without sharing the insight with anyone due to there being no efficient way to do so.
Surely also there are many inaccuracies that I miss, and I’d benefit from being informed by others who did catch them noting that they were there in a way I could just believe as a non-expert on the claim.
First, environment: if you want to believe true things, try not to spend too much time around people who are going to sneeze false information or badly reasoned arguments into your face. You can’t fact check everything you hear and read; you literally don’t have the time, energy, or knowledge needed. Cultivate a social network that cares about true things.
This is good advice, but I really wish (and think it possible) that some competent entrepreneurs made it much less needed by creating epistemic tools that enhance the ability of anyone to discern what’s true out in the wild where people do commonly sneeze false information in your face.