I think the thrust of this argument is definitely correct. I think EAs and the AI safety crowd should have taken a different tack back in the early 2020s, and I think they should change tack now. I also think this article slightly overreaches in ways which harm it’s broader point.
There’s pretty much an unassailable case along the lines of “all the money that went to METR, Apollo, etc. would have been better spent on at Palisade, or similar”, but this post mixes in some unsupported claims like AISI being “repurposed” followed by this claim that particularly irked me:
… and was housemates with the CEO of Anthropic.
What is being implied here? This was nine years ago! If you want to make a claim that Paul Christiano is overly friendly with Anthropic in a way that affects his professional work, you have to actually make that claim with substantive evidence, not innuendo.
(Ok, yes, it’s Bayesian evidence, but it’s not very strong evidence, and if this is the strongest evidence you’re presenting, I’ll assume you don’t have better)
Again, I note that this doesn’t even strengthen the case very much, this whole section could be cut wholecloth from the article without damaging the important parts of its case very much.
I’d be happy with the standards of this post if I came across it in a newspaper, or on Substack, but it doesn’t reach the bar I expect from LessWrong.
>(Okay, yes, it’s Bayesian evidence, but it’s not very strong evidence)
I think it’s stronger than you’re making it out to be, although i’m not entirely confident i can state the case well. It’s something like… insofar as the claim is that evaluators are independent, they really aren’t. it’s all the same big social cluster, with similar beliefs and worldviews
evaluators really ought to minimize conflict of interest. if you’re already in the scene, then there’s sorta a sense in which it feels like the scene is all there is? or at least, i feel this sometimes. but it really would be better if there were some evaluation firms that were not so… part of the in-group of the social vibe inside the ai labs. if some random research team formed a competent evaluation organization that was entirely disconnected from that in-group, it would probably be a very good thing.
and if you’ve already got this mindset, then learning that christiano and amodei were housemates in the past is very strong evidence that aisi does not meet this higher bar for being truly independent of the ai labs
I think the thrust of this argument is definitely correct. I think EAs and the AI safety crowd should have taken a different tack back in the early 2020s, and I think they should change tack now. I also think this article slightly overreaches in ways which harm it’s broader point.
There’s pretty much an unassailable case along the lines of “all the money that went to METR, Apollo, etc. would have been better spent on at Palisade, or similar”, but this post mixes in some unsupported claims like AISI being “repurposed” followed by this claim that particularly irked me:
What is being implied here? This was nine years ago! If you want to make a claim that Paul Christiano is overly friendly with Anthropic in a way that affects his professional work, you have to actually make that claim with substantive evidence, not innuendo.
(Ok, yes, it’s Bayesian evidence, but it’s not very strong evidence, and if this is the strongest evidence you’re presenting, I’ll assume you don’t have better)
Again, I note that this doesn’t even strengthen the case very much, this whole section could be cut wholecloth from the article without damaging the important parts of its case very much.
I’d be happy with the standards of this post if I came across it in a newspaper, or on Substack, but it doesn’t reach the bar I expect from LessWrong.
>(Okay, yes, it’s Bayesian evidence, but it’s not very strong evidence)
I think it’s stronger than you’re making it out to be, although i’m not entirely confident i can state the case well. It’s something like… insofar as the claim is that evaluators are independent, they really aren’t. it’s all the same big social cluster, with similar beliefs and worldviews
evaluators really ought to minimize conflict of interest. if you’re already in the scene, then there’s sorta a sense in which it feels like the scene is all there is? or at least, i feel this sometimes. but it really would be better if there were some evaluation firms that were not so… part of the in-group of the social vibe inside the ai labs. if some random research team formed a competent evaluation organization that was entirely disconnected from that in-group, it would probably be a very good thing.
and if you’ve already got this mindset, then learning that christiano and amodei were housemates in the past is very strong evidence that aisi does not meet this higher bar for being truly independent of the ai labs