WilliamKiely

Karma: 1,162

WilliamKiely 18 Sep 2026 11:19 UTC
10 points
1
on: The Game is Set for a Targeted Memetic Attack on the AI Safety Community
[Not certain I’m recalling this accurately and someone else should confirm]
I listened to the Diary of a CEO podcast with Nate tonight and IIRC Nate said there were “two more” solutions to Millenium Prize problems besides Navier-Stokes before another guest immediately interjected to say they were unconfirmed, and only later in the conversation did Nate clarify that he heard rumors about two more. I think the end result was fine, but in the moment when the guest cut him off to say they were unconfirmed I thought Nate had assumed too much and it could have turned out badly if the other guest hadn’t immediately said the other two were unconfirmed.

WilliamKiely 3 Sep 2026 17:58 UTC
2 points
0
on: AI Tweets
If AI goes well I think the world could be extremely good. The bottom 10% living far better than the top 10% do now.
I defended this claim on a friends-only Facebook post in response to people who disagreed with it:
It sounds like you’re dramatically underestimating how good the future could be. Specifically, from your mention of people having jobs, it seems like you’re imagining a future in which humanity is confined to our solar system with technology that is not radically different than what we have today. (This would be a mediocre or bad future, not a future in which AI goes well.)
In reality, the universe can probably sustain something like 10^32 to 10^52 lives of ordinary length (100 years) (per Bostrom: https://existential-risk.com/concept). This is a huge number, much larger than the ~10^10 people alive today, or the ~10^11 people who have lived so far throughout all of human history.
And in the best futures where AI goes very well, it’s very plausible that the vast majority of those lives (much more than a mere 90%) could be much better than the lives of everyone who has lived on Earth so far.
Consider what the limits of economic and technological growth are. Physics does impose limits, but they are very high. It’s probably possible for humanity to reach what Bostrom calls “technological maturity” this century (“By ‘technological maturity’ we mean the attainment of capabilities affording a level of economic productivity and control over nature close to the maximum that could feasibly be achieved (in the fullness of time)” https://nickbostrom.com/papers/vulnerable.pdf), but even if it took much longer, it could still easily be the case that at *some* future point in time “The bottom 10% [could be] living far better than the top 10% do now” (Jeff’s claim), or, as I believe, that much more than 90% of future could have lives that are better than the lives of literally all humans so far throughout history.
A future in which hunger, war, or disease persist indefinitely is not a future in which “AI goes well.” If AI goes well, all of these things will be eliminated quite quickly after transformative AI, and we’ll have billions of years to spread an astronomically large number of flourishing people (with lives that are unimaginably good to us today) throughout the reachable universe.
And:
A lot of Nick Bostrom’s old works are very relevant to this topic, including his 2008 essay Letter from Utopia, which I’ve long been fond of: https://nickbostrom.com/utopia
And:
> What mechanism would provide the improvement to the bottom 10%?
Primarily wealth creation. For example, in the AI 2040 scenario by the AI Futures Project, the authors describe a future in which world GDP and real wealth for the average human will grow by roughly 200 times compared to today: https://ai-2040.com/ . (Note that AI 2040 is a conservative growth trajectory scenario compared to the default path of even faster growth if humanity continues to race to build superintelligent AI rather than slow down AI development in the name of safety.)
> Unless the people controlling the use of AI are somehow forced to let go of some of the benefits(assuming there are benefits) the worst off will just be worse off/dead.
I think near-term human extinction from AI is a very significant risk. But even if we managed to avoid human extinction, it’s also possible that powerful AI will just lead to concentration of power in the hands of a few people. If those people do not share the benefits of AI with everyone else, that that could indeed be a very bad outcome for the vast majority of people. But note that none of this is relevant to Jeff’s original tweet claim from []’s post, since Jeff explicitly qualified his claim by conditioning on a future where “AI goes well.” If humans lose control of AI and it leads to near-term human extinction, or if humans stay in control of powerful AI, but the small number of humans in control of that AI use it only to benefit themselves and leave the vast majority of people worse off than the best-off people of 2026, then AI has not gone well.
And:
I know I’m pretty unusual in that my main autism special interest of the last decade has been thinking about the long-term future of humanity and the different ways that AI might affect what that future looks like. I’ve spent thousands of hours consuming content about this topic since 2015, but I know most other people aren’t nearly as interested by this as I am. Still, I would encourage anyone who disagrees with Jeff’s claim (“If AI goes well I think the world could be extremely good. The bottom 10% living far better than the top 10% do now.”) to spend some more time thinking about this. It is has been very evident to me for many years that Jeff’s claim is clearly true, so it’s sad to see people dismiss his (conservative) positive vision of what’s possible as impossible without much thought.
To be clear, it is very possible that AI will lead to very bad outcomes, including human extinction. But dismissing the possibility that AI could lead to good futures is completely unjustified and way too doomy for my liking. Extremely good futures are still possible; it’s not too late to make them happen!

WilliamKiely 29 Aug 2026 17:00 UTC
2 points
0
in reply to: jefftk’s comment on: AI Tweets
When I say the extreme “0.0001%” and “all” I’m mostly thinking about how I think near-optimal futures are nontrivially likely, how “AI going well” is what could enable those futures, and how near-optimal futures are not only somewhat better than the world today, but literally “extremely good” / much, much better.
That is, in these extremely good futures, I see us surviving to what Bostrom calls “technological maturity,” and I also see the population of humans and/or conscious minds we care about ballooning to astronomically large numbers as we engage in space colonization and put to use of much of the matter in the reachable universe (~400 billion stars in our galaxy, ~2 trillion galaxies in the reachable universe). The vast number of beings in these futures means that ensuring that 99.9999+% of them have better lives than all of us in 2026 is just a matter of being intentional about making sure that the minds we create in other galaxies are actually ~all awesome. I think technologically we’d easily have the capability and it’d just be a question of are we actually universally optimizing/trying for that or not. (In Letter from Utopia, I think Bostrom does a good job conveying that humans today have lives nowhere close to the best possible lives or experiences permitted by physics.) And again, in futures where AI goes well, I think there’s a significant chance (and it definitely “could” happen, to use the word you used when saying 10%/90%) that we would manage to actually optimize for such a near-optimal, extremely good future.

WilliamKiely 29 Aug 2026 16:28 UTC
13 points
4
in reply to: jefftk’s comment on: AI Tweets
Bet structure candidates:
(1) Bet on the unemployment rate. I have an active bet of this kind:
“If the US total unemployment rate in April 2028 is 6.4% or higher, I will pay Liron $500 (or, if he prefers, donate $500 to Doom Debates or a charity of his choice). On the other hand, if the US total unemployment rate in April 2028 is less than 6.4%, Liron will owe me $500.”
(2) If a significant amount of your probability mass on you being right comes from scenarios where RSI kills us within 3 years, such that Andy wouldn’t be able to pay out in those scenarios, then you could structure the bet as a loan in the form of Andy pays you now, and you pay Andy back in 3 years if most people are still able to productively contribute (however exactly you operationalize that). Choose your bet odds like usual, then do time discounting by either staking to an investment you’d both be happy to make over the next three years, or to an explicit interest rate you’re both agreeable to. For example, for a 20% interest rate and an even odds bet, for each $1 Andy pays you now, you pay him $3.456 in 3 years if he wins.
In terms of operationalization, if you literally mean most people can’t get a job because employers would rather hire AI and robots to do the job better/faster/cheaper, that’s easy enough to judge. But if you mean something more conservative, like “opportunities to have a significant altruistic impact for most people are gone” then I think you should propose the operationalization since Andy and others probably don’t know exactly what you mean.

WilliamKiely 29 Aug 2026 16:13 UTC
4 points
2
in reply to: WilliamKiely’s comment on: AI Tweets
Use the time well!
I want to acknowledge that I strongly agree with this takeaway despite my median being >3 years.
The next 1-3 years are indeed the most important years left to have an impact in expectation, and so it’s unusually important that we use this time well.

WilliamKiely 29 Aug 2026 15:55 UTC
3 points
1
in reply to: Haiku’s comment on: AI Tweets
I feel you—this does hit hard emotionally.
I think someone pursuing a mathematics education PhD is a perfect example for Jeff’s X/Y/Z tweet (which I agree with).
I think there’s a very good chance that that will still be a valuable thing for her to pursue
But how valuable? (It’s easy to focus on how it’s very likely more than zero value, but that would be missing the point.) Is the marginal expected value really anywhere close to the marginal EV of raising the probability of a pause in the next couple years?
Sadly, I think not.
This can be really hard to admit for emotional reasons, especially if she is passionately focused on this path and reluctant to consider radical changes to unusual paths like working to pause AI.
And of course, since pausing AI is the path you’re pursuing, then even if you admitted it (or took the view—I don’t want to use too much language that presumes I’m right), might this just lead to unnecessery tension in the relationship? “Hey romantic parter, I think that your expected social impact would be much higher if you switched from your current path which you’re very passionate about to working on what I’m working on instead” seems pretty awkward TBH. I’d expect most people in that situation to feel like “Why is my partner trying to discourage me from doing what I’m passionate about?” and “Aren’t you biased to think that what you’re doing is most valuable?” and feel bad like they aren’t being supported. If they aren’t unusually open to pivoting, they probably won’t pivot, and so will end up having the same impact that they would have had if you hadn’t said anything, but will just think that you don’t think what they’re doing is important. And that sucks for both of you. It would be much nicer if it could just be true that continuing with her current path is the most valuable thing she can do.

WilliamKiely 29 Aug 2026 15:12 UTC
2 points
0
in reply to: jefftk’s comment on: AI Tweets
I’m guessing the difference here is that I’m expecting more RSI
Yes, this seems plausible. Both RSI scenarios where we survive and where we don’t survive:
If all humans are dead by 2030, then humans are no longer productively contributing in the workforce, so your ~20% on us being dead by 2030 due to AI (from your May post) is contributing to your forecast on humans being no longer able to contribute in 3 years being higher than mine, I think.
I’m guessing another difference is that I’m expecting it’s more likely that we “pace the frontier” and slow down AI development and deployment in the next 3 years, such that even if it was possible for AI to automate AI research and the full AI production supply chain to build robot factories and billions of robots that could outcompete most people globally for all jobs within 3 years, that won’t actually happen that quickly thanks to society’s “pacing” efforts.

WilliamKiely 29 Aug 2026 15:09 UTC
2 points
0
in reply to: _rpd’s comment on: AI Tweets
I agree that one’s median date for the US workforce should be sooner than for the global workforce (largely because it takes time to build billions of robots to do physical tasks and the US is expected to get such robots before many other countries), but I still think my median is >5 years for the US.

WilliamKiely 29 Aug 2026 4:05 UTC
2 points
0
in reply to: WilliamKiely’s comment on: Donating 80% While It Still Counts
Perhaps I’m overconfident on my by mid-2028/2030 numbers above. These are closer to my independent impressions, but my all-things-considered view should be higher since it wouldn’t be that surpriaing for me to update significantly higher (10x+ higher).

WilliamKiely 29 Aug 2026 3:56 UTC
14 points
7
on: AI Tweets
AI has a very real chance of getting us all killed. I think it probably won’t because I expect a lot of people to work very hard to avoid that outcome.
I expect a lot of people to work very hard to avoid that outcome as well, but I think the chance that they / we all succeed is somewhat less than 50%.
Precise forecasting here probably isn’t that important, but I want to flag that just because a lot of people work very hard to avoid an outcome doesn’t mean that the outcome will necessarily be successfully avoided.

WilliamKiely 29 Aug 2026 3:51 UTC
4 points
1
on: AI Tweets
If AI goes well I think the world could be extremely good. The bottom 10% living far better than the top 10% do now.
This seems like a conservative understatement to me. I think the bottom 0.0001% could be living far better than all humans do now.
What links here?
- WilliamKiely's comment on AI Tweets by jefftk (3 Sep 2026 17:58 UTC; 2 points)

WilliamKiely 29 Aug 2026 3:46 UTC
15 points
5
on: AI Tweets
AI is so quickly approaching (or exceeding) expert human abilities across so many areas that most people should be planning for 1-3 more years in which they can productively contribute.
1-3 years seems plausible to me, but my median estimate of the date when most people in the work force today will no longer be able to productively contribute is significantly more than 3 years. I’d probably say sometime in the 2030s.

WilliamKiely 20 Aug 2026 4:29 UTC
7 points
6
in reply to: arielb1’s comment on: How My Students Think About AI
The Hugging Face incident demonstrates a meaningful kind of misalignment, namely the possibility of an agentic AI achieving a goal OpenAI intended using means that OpenAI not only didn’t intend, but actively wanted to make sure the AI did not use.

WilliamKiely 16 Aug 2026 1:29 UTC
8 points
2
on: How My Students Think About AI
I described the Hugging Face incident to students in my summer course. None of them had heard of it beforehand. Their basic reaction can best be summarized as “OpenAI told a model to do some hacking and then it did some hacking. And?” None of them understood this as representing any kind of meaningful misalignment, nor anything particularly interesting.
Can you say more about this? Why do you think they didn’t see the HF incident as representing any kind of meaningful misalignment?

I’ve read thousands of YouTube comments on videos about AI (podcasts/news interviews/etc) in the last year and have seen all of the perspectives you list in this post expressed in those comments. Regarding #7, my assumption was that the people expressing this view had learned about the HF incident only from secondary sources that explained what happened poorly/inaccurately.

It’s not clear to me how people would come to hold this view if they had read the OpenAI blog post or the Black Hat video. But maybe even with an accurate explanation of the incident (I’m guessing your explanation was accurate) there’s a reason why many people still come to this conclusion?

How I think developers of frontier AI systems and regulators ought to act in the face of existential AI risk

WilliamKiely19 Jun 2026 22:22 UTC

14 points

0 comments12 min readLW link

WilliamKiely 16 Jun 2026 18:06 UTC
4 points
0
in reply to: Zach Stein-Perlman’s comment on: WilliamKiely’s Shortform
Agreed that there is significant value in moving us between different high-value futures.
Intuitively though, I still think “weak bimodality” combined with my view that low-value futures are much more likely than high-value futures means that the main thing we should be aiming to do now is get to a high-value future. In other words, I still think maxipok is the right policy, where an “okay outcome” is defined as any high-value future.
Do you agree with this, or do you think some of our collective portfolio of actions should be focused on trying to raise the probability of very-high-value futures relative to somewhat-high-value-futures even while (or if) it is still the case that high-value futures are much less likely than low-value (near-zero-value) futures?

WilliamKiely 11 Jun 2026 17:45 UTC
4 points
0
in reply to: Zach Stein-Perlman’s comment on: WilliamKiely’s Shortform
Huh, I don’t think that change of view undermines the main point.
In my view the main insight is that the value of the future is bimodally distributed (perhaps there is a small third mode for very negative value futures, but I think it’s much smaller than the ~zero mode and the ~very postive mode so we can basically ignore it).
Whether the very positive mode probability mass is distributed over 90%-100% optimal futures or over 0.1%-100% optimal futures (or even wider) doesn’t really matter. The key thing is that the ~zero mode is distributed over +/-10^-10% optimal futures or even +/-10^-20% optimal futures, and the space inbetween from 10^-10% to 0.1% optimal futures has lower cumulative probability mass than either mode. So the EV of the future is basically equal to the probability of getting a future in the high value mode times the average value of a future in that mode.
Perhaps you think acausal trade considerations make ~zero value futures unlikely (does most of the value of the future come from value outside of our lightcone that we acausally make happen due to what we do in our lightcone? I’m not really sure how to value the multiverse—assuming it exists it seems that our lightcone could at most only have a small (acausal) influence on it, and so the absolute value of the multiverse will be basically the same regardless of what happens in our lightcone), but at least ignoring acausal trade considerations and only looking at the reachable universe within our lightcone the above value-is-bimodally-distributed view still seems right to me.
It just seems to fall out of the fact that the potential value of the future spans dozens of OOMs. There are two main factors -- (1) How much of the reachable universe’s resources we use (including for how long) and (2) How hard we optimize the resources we use on average—each of which spans many OOMs, but I don’t think the fact that there are two factors rather than just one (or three rather than just one if we separate out the time factor) changes the bimodal nature of our forecast of how valuable the future is.
I’ll check out MacAskill’s article later.

WilliamKiely 10 Jun 2026 5:15 UTC
6 points
0
on: WilliamKiely’s Shortform
@Zach Stein-Perlman ’s 2021 quick take Value Is Binary is very under-rated. I’ve referenced the idea well over a dozen times in the last 4+ years and just re-read it after being reminded of it again.

WilliamKiely 26 May 2026 5:40 UTC
4 points
0
on: Donating 80% While It Still Counts
(Copying my comments over from Facebook:)

Thanks for all your donations and direct work and transparency about these things!
From my perspective it’s unclear whether or not you’re leaving a lot of potential impact on the table, due to a few considerations:
(1) Are your retirement savings invested optimally? The difference between e.g. a 25% annual return and a 50% annual return on your ~$1.5M in retirement savings is $375,000 in the next year. I haven’t had a 401(k) in years, but my understanding is that 401(k)s limit how you’re able to invest the money, so if your retirement savings are in a 401(k) this could be very suboptimal assuming AI stocks are going to keep growing much faster than the rest of the economy, as I expect they probably will for the next couple of years at least. (VanEck Semiconductor ETF is up 135% in the last 12 months.)

(2) What do you think is the difference in cost-effectiveness between the most cost-effective donation opportunities you can find and your current work? SecureBio sounds like valuable altruistic work, but even if it’s a 99th percentile donation opportunity, you may be able to identify a ~99.9th percentile donation opportunity that is 10x as cost-effective, in which case your salary-sacrifice direct work strategy may be suboptimal. Or perhaps there is a different direct work job you could do that would be much higher impact than your work at SecureBio.

It sounds like you put a much higher probability on AI-caused catastrophe in the next two years than I do, though maybe your 10% forecast below just wasn’t made carefully and your true credence is actually lower:
Let’s try and make some similar predictions for 2028:
My odds that the world has changed substantially are up significantly, maybe 55%, primarily due to AI. I’d put about 10% on futures where things go very badly, where I’m not here to write a followup and you’re not here to read one.
10% that you’ll be unable to write a blog post by 2028 (even end of 2028) due to being dead from a global catastrophe seems *way* too high to me. I’d put that at less than 1% by mid-2028. Also, if there really is a 10% chance that you’ll be dead two years from now due to an AI-caused catastrophe, it’s worth flagging that this means that AI represents about 95% of your mortality risk in the next two years (!!!). (99.3% of American men who turn 40 make it two more years to their 42nd birthday according to https://www.ssa.gov/oact/STATS/table4c6.html).
I wonder if your forecast is 10% because you’re much more pessimistic than I am about AI risk in general or because you have more aggressive AI timelines (or both)?
To answer this, what are your forecasts that you’ll be dead as a result of a global catastrophe by mid-2028/2030/2032/2035/2040? Well actually, I’m not sure whether a good forecast for this for 2040 is 10% or 70%, but if the right forecast for the risk of you dying due to an AI-caused catastrophe by mid-2040 is X%, then what’s your forecast for this by mid-2028/2030/2032/2035? My forecasts would be roughly 0.01X/0.05X/0.2X/0.6X, so my number by mid-2028 is ~0.1%-0.7%, i.e. much lower than your 10%.

Also, in those 10% of scenarios where you’re imagining where we’re dead in two years due to an AI-caused catastrophe, what kind of AI-caused catastrophe are you imagining? Is the bulk of the (you and I aren’t here in two years) risk AI takeover risk or engineered pandemic risk or something else in your mind?

WilliamKiely 13 Apr 2026 20:30 UTC
3 points
0
in reply to: XelaP’s comment on: Which things were you surprised to learn are metaphors?
You may be reading too much into my label. For most of those they don’t feel like anything to me.
The main thing I feel when it comes to times is, as I said, times throughout a year feeling like they have a spatial location or direction. So e.g. August feels like it is up-and-to-the-left in front of me. (Holidays and seasons and other events like friend’s birthdays also have this directional spatial feel to them, i.e. I locate where approximately they are on the disk without thinking about it.) This isn’t some memory palace technique I came up with to help me remember dates; it’s just something I realize I always imagine unconsciously after I learned about synesthesias that not everyone does. (In describing it to others I have found that some other people who have basically this same representation of time throughout the year. IIRC one person I talked to once said they visualize a spiral with the months in different years at different levels.)
When I think of different years I do visualize them on a number line streching back a few centuries to the left. This is easier to explain by saying that time is often depicted on a number line timeline.

WilliamKiely

How I think de­vel­op­ers of fron­tier AI sys­tems and reg­u­la­tors ought to act in the face of ex­is­ten­tial AI risk

How I think developers of frontier AI systems and regulators ought to act in the face of existential AI risk