When you say your exposure to the rationalist community has made you significantly more intelligent, not just more capable (I assume you were alluding to these separately in the way that you’d allude to a base model’s capability level separately from its agent harness), how do you mean?
Mostly it has been along the “unhobbling” axes. I have (I think) better instincts for probability and estimation, which only requires a little feedback, it’s mostly about getting in touch with one’s inner sim.
I can more easily spot flaws in arguments, particularly of the flavor “this evidence is far too strong, you have done something very wrong” e.g. “small Mistral models can find the same vulnerabilities in code that Mythos did”, which was obviously not true, even if Mythos was overhyped.
Most of this feels like my brain is “de-noised” in a way, e.g. better at searching through an argument for flaws. It’s like my thoughts are less grasping, less likely to stick to the first thing I notice or am presented with, so I’m better at thinking.
E.g. in the Mythos example, a worse version of me might have grabbed onto the claim that Mythos is overhyped, and started arguing about Anthropic’s overall integrity, etc. instead of noticing the authors having made a claim that on lots of their benchmarks, small open models were better than GPT-5 and Claude 4.5, and really no models were better than any other, which is obvious nonsense and discredits their entire blogpost.
Interesting, thanks, that jives with my own experience as well. I’m mainly concerned about the thing Buck pointed out, that my “brain de-noising” has progressed more for evaluating external arguments than the ones I come up with.
When you say your exposure to the rationalist community has made you significantly more intelligent, not just more capable (I assume you were alluding to these separately in the way that you’d allude to a base model’s capability level separately from its agent harness), how do you mean?
Mostly it has been along the “unhobbling” axes. I have (I think) better instincts for probability and estimation, which only requires a little feedback, it’s mostly about getting in touch with one’s inner sim.
I can more easily spot flaws in arguments, particularly of the flavor “this evidence is far too strong, you have done something very wrong” e.g. “small Mistral models can find the same vulnerabilities in code that Mythos did”, which was obviously not true, even if Mythos was overhyped.
Most of this feels like my brain is “de-noised” in a way, e.g. better at searching through an argument for flaws. It’s like my thoughts are less grasping, less likely to stick to the first thing I notice or am presented with, so I’m better at thinking.
E.g. in the Mythos example, a worse version of me might have grabbed onto the claim that Mythos is overhyped, and started arguing about Anthropic’s overall integrity, etc. instead of noticing the authors having made a claim that on lots of their benchmarks, small open models were better than GPT-5 and Claude 4.5, and really no models were better than any other, which is obvious nonsense and discredits their entire blogpost.
Interesting, thanks, that jives with my own experience as well. I’m mainly concerned about the thing Buck pointed out, that my “brain de-noising” has progressed more for evaluating external arguments than the ones I come up with.