My reasons for disagreement with Habryka and Ben Pace on their impact largely derive from me being way more optimistic on AI risk and AI Alignment than I used to be, which implies Habryka and Ben Pace had way more positive impact than they thought.
You’re using a time difference as evidence for the sign of one causal arrow?
I don’t understand what you’re saying, but what I was saying is that I used to be much more pessimistic around how hard AI Alignment was, and a large part of the problem of AI Alignment is that it’s not very amenable to iterative solutions. Now, however, I believe that I was very wrong on how hard alignment ultimately turned out to be, and in retrospect, that means that the funding of AI safety research is much more positive, since I now give way higher chances to the possibility that empirical, iterative alignment is enough to solve AI Alignment.
Sure, but why do you think that means they had a positive impact? Even if alignment turns out to be easy instead of hard, that doesn’t seem like it’s evidence that Lightcone had a positive impact.
[I agree a simple “alignment hard → Lightcone bad” model gets contradicted by it, but that’s not how I read their model.]
My reading is that Noosphere89 thinks that Lightcone has helped in bringing in/upskiling a number of empirical/prosaic alignment researchers. In worlds where alignment is relatively easy, this is net positive as the alignment benefits are higher than the capabilities costs, while in worlds where alignment is very hard, we might expect the alignment benefits to be marginal while the capabilities costs continue to be very real.
I did argue that closing the Lightcone offices was the right thing, but my point is that part of the reasoning relies on a core assumption that AI Alignment isn’t very iterable and will generally cost capabilities that I find probably false.
I am open to changing my mind, but I see a lot of reasoning on AI Alignment that is kinda weird to me by Habryka and Ben Pace.
You’re using a time difference as evidence for the sign of one causal arrow?
I don’t understand what you’re saying, but what I was saying is that I used to be much more pessimistic around how hard AI Alignment was, and a large part of the problem of AI Alignment is that it’s not very amenable to iterative solutions. Now, however, I believe that I was very wrong on how hard alignment ultimately turned out to be, and in retrospect, that means that the funding of AI safety research is much more positive, since I now give way higher chances to the possibility that empirical, iterative alignment is enough to solve AI Alignment.
Sure, but why do you think that means they had a positive impact? Even if alignment turns out to be easy instead of hard, that doesn’t seem like it’s evidence that Lightcone had a positive impact.
[I agree a simple “alignment hard → Lightcone bad” model gets contradicted by it, but that’s not how I read their model.]
My reading is that Noosphere89 thinks that Lightcone has helped in bringing in/upskiling a number of empirical/prosaic alignment researchers. In worlds where alignment is relatively easy, this is net positive as the alignment benefits are higher than the capabilities costs, while in worlds where alignment is very hard, we might expect the alignment benefits to be marginal while the capabilities costs continue to be very real.
I did argue that closing the Lightcone offices was the right thing, but my point is that part of the reasoning relies on a core assumption that AI Alignment isn’t very iterable and will generally cost capabilities that I find probably false.
I am open to changing my mind, but I see a lot of reasoning on AI Alignment that is kinda weird to me by Habryka and Ben Pace.