I personally believe that my exposure to the rationalist community has made me significantly more intelligent and capable along a number of axes. It has also caused me to move from working in chemistry to working in AI alignment. Overall, this is probably a 10-100x multiplier on my impact on the future, it’s just that most people don’t have any impact at all.
This is a really very impressive feat for a web forum and Harry Potter fanfiction to achieve, (N=1 but I doubt I’m atypical) but like, 1-2 OOMs isn’t much in the grand scheme of things. If everyone was as strongly benefited as I was, then yeah, the world would change in aggregate, but for a few hundred people (upper bound guess) I think you shouldn’t expect anything massive.
I also suspect that my specific personality factors are important here. I have very high g which meant there was a lot of horsepower in my brain, which engagement with rationalist literature was able to focus. A lot of the benefits for me have been un-hobbling, though not all. I lack the ability or desire to work 60 hour weeks (or even 40 hour weeks, to be honest, I think about 30 hours reliable productivity + 5 hours of sporadic insight might be a better description) and lack a certain insane agency which is somewhat common among rats (the kind that Mikhail Samin has, for example) though I’m still more agentic than most people I know. I think both of these things are slowly improving, perhaps given five or ten years I’d be able to be a real founder, but then again perhaps not!
Maybe a good model is there are three or four important axes which define achievement. For any person, they have a raw skill factor (heavy tailed) and a hobbledness factor (between zero and one) and you multiply all the parameters together to get someone’s overall score. Rationality is good at unhobbling but not so good at bumping up the heavy-tailed factors by an OOM.
When you say your exposure to the rationalist community has made you significantly more intelligent, not just more capable (I assume you were alluding to these separately in the way that you’d allude to a base model’s capability level separately from its agent harness), how do you mean?
Mostly it has been along the “unhobbling” axes. I have (I think) better instincts for probability and estimation, which only requires a little feedback, it’s mostly about getting in touch with one’s inner sim.
I can more easily spot flaws in arguments, particularly of the flavor “this evidence is far too strong, you have done something very wrong” e.g. “small Mistral models can find the same vulnerabilities in code that Mythos did”, which was obviously not true, even if Mythos was overhyped.
Most of this feels like my brain is “de-noised” in a way, e.g. better at searching through an argument for flaws. It’s like my thoughts are less grasping, less likely to stick to the first thing I notice or am presented with, so I’m better at thinking.
E.g. in the Mythos example, a worse version of me might have grabbed onto the claim that Mythos is overhyped, and started arguing about Anthropic’s overall integrity, etc. instead of noticing the authors having made a claim that on lots of their benchmarks, small open models were better than GPT-5 and Claude 4.5, and really no models were better than any other, which is obvious nonsense and discredits their entire blogpost.
Interesting, thanks, that jives with my own experience as well. I’m mainly concerned about the thing Buck pointed out, that my “brain de-noising” has progressed more for evaluating external arguments than the ones I come up with.
I personally believe that my exposure to the rationalist community has made me significantly more intelligent and capable along a number of axes. It has also caused me to move from working in chemistry to working in AI alignment. Overall, this is probably a 10-100x multiplier on my impact on the future, it’s just that most people don’t have any impact at all.
This is a really very impressive feat for a web forum and Harry Potter fanfiction to achieve, (N=1 but I doubt I’m atypical) but like, 1-2 OOMs isn’t much in the grand scheme of things. If everyone was as strongly benefited as I was, then yeah, the world would change in aggregate, but for a few hundred people (upper bound guess) I think you shouldn’t expect anything massive.
I also suspect that my specific personality factors are important here. I have very high g which meant there was a lot of horsepower in my brain, which engagement with rationalist literature was able to focus. A lot of the benefits for me have been un-hobbling, though not all. I lack the ability or desire to work 60 hour weeks (or even 40 hour weeks, to be honest, I think about 30 hours reliable productivity + 5 hours of sporadic insight might be a better description) and lack a certain insane agency which is somewhat common among rats (the kind that Mikhail Samin has, for example) though I’m still more agentic than most people I know. I think both of these things are slowly improving, perhaps given five or ten years I’d be able to be a real founder, but then again perhaps not!
Maybe a good model is there are three or four important axes which define achievement. For any person, they have a raw skill factor (heavy tailed) and a hobbledness factor (between zero and one) and you multiply all the parameters together to get someone’s overall score. Rationality is good at unhobbling but not so good at bumping up the heavy-tailed factors by an OOM.
When you say your exposure to the rationalist community has made you significantly more intelligent, not just more capable (I assume you were alluding to these separately in the way that you’d allude to a base model’s capability level separately from its agent harness), how do you mean?
Mostly it has been along the “unhobbling” axes. I have (I think) better instincts for probability and estimation, which only requires a little feedback, it’s mostly about getting in touch with one’s inner sim.
I can more easily spot flaws in arguments, particularly of the flavor “this evidence is far too strong, you have done something very wrong” e.g. “small Mistral models can find the same vulnerabilities in code that Mythos did”, which was obviously not true, even if Mythos was overhyped.
Most of this feels like my brain is “de-noised” in a way, e.g. better at searching through an argument for flaws. It’s like my thoughts are less grasping, less likely to stick to the first thing I notice or am presented with, so I’m better at thinking.
E.g. in the Mythos example, a worse version of me might have grabbed onto the claim that Mythos is overhyped, and started arguing about Anthropic’s overall integrity, etc. instead of noticing the authors having made a claim that on lots of their benchmarks, small open models were better than GPT-5 and Claude 4.5, and really no models were better than any other, which is obvious nonsense and discredits their entire blogpost.
Interesting, thanks, that jives with my own experience as well. I’m mainly concerned about the thing Buck pointed out, that my “brain de-noising” has progressed more for evaluating external arguments than the ones I come up with.