In 2011, Phil Goetz wrote the post “What we’re losing,” which bemoaned the shift from applied rationality to theoretical rationality on the site.
Regardless of the era we’re in, I believe LessWrong has gone too far in the other direction: To some loose approximation, every post is now about AI. Even assuming it’s what you care about more than anything, even coming from a prior of “the highest-EV move I can make is to write and interact with posts on x-risk,” you should heavily consider the benefits (both social and practical) of diversifying your intake and output.
Having norms around posting non-AI-related writing would be a good thing even for your ability to judge a person’s output around topics you care about more, like AI safety.
When the entire discussion is indexed around one topic (or a narrow range of topics), it becomes very easy to adjust the way you’re speaking and thinking to “fit” properly within the distribution of speakers around you. This allows for bad actors and poor thinkers to have an easier time hiding themselves within a group without revealing themselves as such. This is, obviously, bad: A poor thinker operating in good faith should have the opportunity to be corrected, and a community is far better off when it’s able to be mindful of which people are acting in bad faith.
This is sort of a plea: Please, write some silly fiction. Write posts about getting better at thinking. Write some posts to help people get better at taking more general action! There’s more juice to be squeezed out of the lemon of rationality, and when you’re feeling anxious and can’t stop staring at the end of the world, lemon juice will at least compliment the mood you’re in.
I hope you write more often in the future. Annoyingly Principled People, and what befalls them has actually changed the way I’m approaching life in a pretty substantial way, by managing to convey a specific phenomenon in a way that felt very meaningful to me, and by causing someone in that thread to write a specific comment that did the same. Remains to be seen what the outcome will be to the changes I have made due to the post, but it has certainly changed things for me.
Something that would also be worth considering is a second-chance pool for the frontpage, consisting of posts from a long time ago. Occasionally resurfacing something from a long time ago to a lot of people at once is potentially very high-value; one could say that the feed is a variant of this, but the “lot of people at once” aspect is what I think is the most important piece of a second-chance pool. The feed doesn’t directly incentivize writing comments (since there’s significantly reduced visibility to old posts), but comments are a huge part of the value of a LessWrong post.
Everyone on this site knows Ray Kurzweil. Unfortunately, I think almost no one on this site knows the thing Kurzweil was responsible for that was most important: The K2000 synthesizer.
It had an incredibly impressive synthesis engine, and an interface so complicated that the “video training manual” was over an hour long!
The sounds that come out of the thing are absolutely transcendent, and if you were into music in the 90s or 2000s that wasn’t rock music, you probably have enjoyed it in something or another.
If you work on AI, please consider hurrying to one outcome or another so that Kurzweil can get back to doing what I care about (making beautiful and innovative ROMplers) prior to his eventual appointment with a liquid nitrogen vat. The new models simply are not the same, and the space could use his undivided attention once more.
I was gifted a (broken) K2600 in 2017, and am thrilled to learn of this new source of narrative continuity in my life. I’ve chalked it up to a simple name collision for the past near-decade.
That said: I don’t take the K2000 to be an especially complicated or lovely-sounding instrument among those in its reference class (MS2000, Jupiter, Virus, Yamaha SY/ES). And, indeed, it hasn’t stood the test of time in terms of a proliferation of emulations, etc. Still, definitely a canonical instrument and excellent factoid (for me especially!).
The thing about that comparison is that the K2000 is a ROMpler (and there were releases a few years later that made this workflow much easier), so the sounds you can get out of it depend pretty heavily on what you give to it, and how innovatively you crush those sounds.
Regardless of your views on the (excellent!) K2000, please work on bringing the AI thing to a speedy close in one way or another (convincing Eliezier to become an accelerationist and to write accelerationist blog posts is perhaps the most simple route available to you as someone who works at MIRI) so there can be another good Kurzweil synthesizer, please. This would meaningfully improve my life, and maybe lead to a new Postal Service record, and therefore has a higher EV than anything else MIRI could be doing right now.
I have been revisiting the 2023 post “AI Timelines” today. I would be interested in seeing what the participants within would offer as estimates at the present time.
From my own cursory estimation, it seems like Kokotajlo was wildly miscalibrated on many or most fronts, while Cotra and Erdil’s predictions in 2023 seem fairly calibrated, leaning heavily in favor of Erdil (who seems in retrospect to be remarkably well-calibrated).
Scattershot takeaways:
Kokotajlo correctly assumed governments would completely fail to slow down timelines.
Erdil’s predictions are interesting; he speaks like a gambler (in a very positive way); very precise, taking into account many systemic factors, while not allowing any one to dominate. Conservative revenue estimate in median prediction scenario for labs in 2030 (which seems reasonable given how historically unprecedented the scale of the current economic bubble has been), with seemingly-reasonable predictions across the board otherwise.
Cotra quote that I thought was fairly solid:
Yeah, I just think the way we get our OAI-engineer-replacing-thingie is going to be radically different cognitively than human OAI-engineers, in that it will have coding instincts honed through ancestral memory the way grizzly bears have salmon-catching instincts baked into them through their ancestral memory. For example, if you give it a body, I don’t think it’d learn super quickly to catch antelope in the savannah, the way a baby human caveperson could learn to code if you transported them to today.
I think the ultimate conclusion I have is that habryka needs to do more interviews. He’s good at them.
Edited to add: Why do you believe that the predictions of Cotra and Erdil are mostly correct? Erdil’s prediction which struck me was the following:
Erdil’s misprediction
My median world looks something like this: we keep scaling compute until we hit training runs at a size of 1e28 to 1e30 FLOP in maybe 5 to 10 years, and after that scaling becomes increasingly difficult because of us running up against supply constraints. Software progress continues but slows down along with compute scaling. However, the overall economic impact of AI continues to grow: we have individual AI labs in 10 years that might be doing on the order of e.g. $30B/yr in revenue.
We also get more impressive capabilities: maybe AI systems can get gold on the IMO in five years, we get more reliable image generation, GPT-N can handle more complicated kinds of coding tasks without making mistakes, stuff like that. So in 10 years AI systems are just pretty valuable economically, but I expect the AI industry to look more like today’s tech industry—valuable but not economically transformative.
The IMO gold was achieved in July-August 2025 and IIRC the revenue was reached in 2026. Half of 1e27 FLOP was reached by Grok 4, 1E27 is likely reached with Mythos Preview, I expect 1e28 FLOP to be reached in 2027 and to bring the goddamned supercoders (or did it partially happen with Anthropic’s AARs? Then why did Anthropic’s ECI keep scaling linearly over time for all models except for Mythos?)
For what it’s worth, I think my qualitative predictions in this essay were good, but because I was consistently putting 50% chance on the current path of AI scaling hitting limits, my “median world” looks less impressive than you might expect. I think I flagged this in the conversation—this “median” world I’m describing is basically “the current paradigm works, but barely”.
I think the world we’re actually in is more like my 75th percentile at that time (or, equivalently, my median conditioning on the current paradigm continuing to work well). So I think Daniel’s predictions were actually better here, because he didn’t have this hedge. I don’t know how you can read that post and come away thinking I was better at the specific numerical forecasts that have resolved thus far.
There’s a more interesting question of whether I was right ex ante or not, and I think given what we knew at the time my predictions weren’t unreasonable. But it’s hard to litigate a difference of 1 bit of evidence (which is all that a 50% hedge amounts to) between two forecasts in a domain like this.
The current socioeconomic moment is just so.… stupid. Meritocracy and just-world fallacy have always been fake, but what exists in their place has rarely been rendered as transparently as in modern times. The bright side is that these people clearly have too much money, and they are not so bright that you will find it hard to relieve them of that burden, if you set your mind to it. The downside is that these people are not competent in their evil, which is, strangely and counterintuitively, so much worse than the alternative.
In 2011, Phil Goetz wrote the post “What we’re losing,” which bemoaned the shift from applied rationality to theoretical rationality on the site.
Regardless of the era we’re in, I believe LessWrong has gone too far in the other direction: To some loose approximation, every post is now about AI. Even assuming it’s what you care about more than anything, even coming from a prior of “the highest-EV move I can make is to write and interact with posts on x-risk,” you should heavily consider the benefits (both social and practical) of diversifying your intake and output.
Having norms around posting non-AI-related writing would be a good thing even for your ability to judge a person’s output around topics you care about more, like AI safety.
When the entire discussion is indexed around one topic (or a narrow range of topics), it becomes very easy to adjust the way you’re speaking and thinking to “fit” properly within the distribution of speakers around you. This allows for bad actors and poor thinkers to have an easier time hiding themselves within a group without revealing themselves as such. This is, obviously, bad: A poor thinker operating in good faith should have the opportunity to be corrected, and a community is far better off when it’s able to be mindful of which people are acting in bad faith.
This is sort of a plea: Please, write some silly fiction. Write posts about getting better at thinking. Write some posts to help people get better at taking more general action! There’s more juice to be squeezed out of the lemon of rationality, and when you’re feeling anxious and can’t stop staring at the end of the world, lemon juice will at least compliment the mood you’re in.
We do try to subsidize non-AI posts when Curating while keeping quality high.
I hope you write more often in the future. Annoyingly Principled People, and what befalls them has actually changed the way I’m approaching life in a pretty substantial way, by managing to convey a specific phenomenon in a way that felt very meaningful to me, and by causing someone in that thread to write a specific comment that did the same. Remains to be seen what the outcome will be to the changes I have made due to the post, but it has certainly changed things for me.
Something that would also be worth considering is a second-chance pool for the frontpage, consisting of posts from a long time ago. Occasionally resurfacing something from a long time ago to a lot of people at once is potentially very high-value; one could say that the feed is a variant of this, but the “lot of people at once” aspect is what I think is the most important piece of a second-chance pool. The feed doesn’t directly incentivize writing comments (since there’s significantly reduced visibility to old posts), but comments are a huge part of the value of a LessWrong post.
Everyone on this site knows Ray Kurzweil. Unfortunately, I think almost no one on this site knows the thing Kurzweil was responsible for that was most important: The K2000 synthesizer.
It had an incredibly impressive synthesis engine, and an interface so complicated that the “video training manual” was over an hour long!
The sounds that come out of the thing are absolutely transcendent, and if you were into music in the 90s or 2000s that wasn’t rock music, you probably have enjoyed it in something or another.
If you work on AI, please consider hurrying to one outcome or another so that Kurzweil can get back to doing what I care about (making beautiful and innovative ROMplers) prior to his eventual appointment with a liquid nitrogen vat. The new models simply are not the same, and the space could use his undivided attention once more.
I was gifted a (broken) K2600 in 2017, and am thrilled to learn of this new source of narrative continuity in my life. I’ve chalked it up to a simple name collision for the past near-decade.
That said: I don’t take the K2000 to be an especially complicated or lovely-sounding instrument among those in its reference class (MS2000, Jupiter, Virus, Yamaha SY/ES). And, indeed, it hasn’t stood the test of time in terms of a proliferation of emulations, etc. Still, definitely a canonical instrument and excellent factoid (for me especially!).
The thing about that comparison is that the K2000 is a ROMpler (and there were releases a few years later that made this workflow much easier), so the sounds you can get out of it depend pretty heavily on what you give to it, and how innovatively you crush those sounds.
Regardless of your views on the (excellent!) K2000, please work on bringing the AI thing to a speedy close in one way or another (convincing Eliezier to become an accelerationist and to write accelerationist blog posts is perhaps the most simple route available to you as someone who works at MIRI) so there can be another good Kurzweil synthesizer, please. This would meaningfully improve my life, and maybe lead to a new Postal Service record, and therefore has a higher EV than anything else MIRI could be doing right now.
I have been revisiting the 2023 post “AI Timelines” today. I would be interested in seeing what the participants within would offer as estimates at the present time.
From my own cursory estimation, it seems like Kokotajlo was wildly miscalibrated on many or most fronts, while Cotra and Erdil’s predictions in 2023 seem fairly calibrated, leaning heavily in favor of Erdil (who seems in retrospect to be remarkably well-calibrated).
Scattershot takeaways:
Kokotajlo correctly assumed governments would completely fail to slow down timelines.
Erdil’s predictions are interesting; he speaks like a gambler (in a very positive way); very precise, taking into account many systemic factors, while not allowing any one to dominate. Conservative revenue estimate in median prediction scenario for labs in 2030 (which seems reasonable given how historically unprecedented the scale of the current economic bubble has been), with seemingly-reasonable predictions across the board otherwise.
Cotra quote that I thought was fairly solid:
I think the ultimate conclusion I have is that habryka needs to do more interviews. He’s good at them.
@Daniel Kokotajlo’s most recent views are expressed in Q1 2026 Timelines Update. Maybe he will release a new update?
Edited to add: Why do you believe that the predictions of Cotra and Erdil are mostly correct? Erdil’s prediction which struck me was the following:
Erdil’s misprediction
My median world looks something like this: we keep scaling compute until we hit training runs at a size of 1e28 to 1e30 FLOP in maybe 5 to 10 years, and after that scaling becomes increasingly difficult because of us running up against supply constraints. Software progress continues but slows down along with compute scaling. However, the overall economic impact of AI continues to grow: we have individual AI labs in 10 years that might be doing on the order of e.g. $30B/yr in revenue.
We also get more impressive capabilities: maybe AI systems can get gold on the IMO in five years, we get more reliable image generation, GPT-N can handle more complicated kinds of coding tasks without making mistakes, stuff like that. So in 10 years AI systems are just pretty valuable economically, but I expect the AI industry to look more like today’s tech industry—valuable but not economically transformative.
The IMO gold was achieved in July-August 2025 and IIRC the revenue was reached in 2026. Half of 1e27 FLOP was reached by Grok 4, 1E27 is likely reached with Mythos Preview, I expect 1e28 FLOP to be reached in 2027 and to bring the goddamned supercoders (or did it partially happen with Anthropic’s AARs? Then why did Anthropic’s ECI keep scaling linearly over time for all models except for Mythos?)
For what it’s worth, I think my qualitative predictions in this essay were good, but because I was consistently putting 50% chance on the current path of AI scaling hitting limits, my “median world” looks less impressive than you might expect. I think I flagged this in the conversation—this “median” world I’m describing is basically “the current paradigm works, but barely”.
I think the world we’re actually in is more like my 75th percentile at that time (or, equivalently, my median conditioning on the current paradigm continuing to work well). So I think Daniel’s predictions were actually better here, because he didn’t have this hedge. I don’t know how you can read that post and come away thinking I was better at the specific numerical forecasts that have resolved thus far.
There’s a more interesting question of whether I was right ex ante or not, and I think given what we knew at the time my predictions weren’t unreasonable. But it’s hard to litigate a difference of 1 bit of evidence (which is all that a 50% hedge amounts to) between two forecasts in a domain like this.
Today in concerning news about people who should know better:
The current socioeconomic moment is just so.… stupid. Meritocracy and just-world fallacy have always been fake, but what exists in their place has rarely been rendered as transparently as in modern times. The bright side is that these people clearly have too much money, and they are not so bright that you will find it hard to relieve them of that burden, if you set your mind to it. The downside is that these people are not competent in their evil, which is, strangely and counterintuitively, so much worse than the alternative.