Ok. I think I might bow out for now unless there’s something especially salient that I should look at, but by way of a bit of summary: I think we agree that Yudkowsky was somewhat overconfident about solving FAI, and that there’s a super high bar that should be met before making an AGI, and no one knows how to meet that bar; my guess would be that we disagree about
the degree to which he was overconfident,
how high a bar would have to be met before making an AGI, in desperate straits.
(I did read that one; it’s interesting but basically in line with how I think he’s overconfident; it’s possible one or both of us is incorrectly reading in / not reading in to what he wrote there, about his absolute level of confidence in solving the philosophical problems involved.)
Hmm, did you also read my immediate reply to him, where I made the point “if you’re the only philosopher in the team, how will others catch your mistakes?” How to understand his (then) plan except that he would have been willing to push the “launch” button even if there were zero other similarly capable philosophers available to scrutinize his philosophical ideas?
(Also just recording that I appreciate the OP and these threads, and people finding historical info. I think the topic of how “we” have been going wrong on strategy is important. I’m participating because I’m interested, though my contributions may not be very helpful because
I was a relative latecomer, in that much of the strategic direction (insofar as that existed) had already been fixed and followed;
I didn’t especially think about strategy that much initially, so I didn’t have many mental hooks for tracking what was happening in the social milieu in terms of strategic thinking and actions.)
(Oh I hadn’t read the full thread, now I have; still no big update? Like, I continue to see him being seemingly overconfident in his ability to get those solutions, but I’m not seeing “oh he would have mistakenly come to think he had a solution when he didn’t”, if that’s what you’re trying to say.)
Ok. I think I might bow out for now unless there’s something especially salient that I should look at, but by way of a bit of summary: I think we agree that Yudkowsky was somewhat overconfident about solving FAI, and that there’s a super high bar that should be met before making an AGI, and no one knows how to meet that bar; my guess would be that we disagree about
the degree to which he was overconfident,
how high a bar would have to be met before making an AGI, in desperate straits.
Definitely read the second link if you haven’t already (it’s very short and salient), but otherwise, sure.
(I did read that one; it’s interesting but basically in line with how I think he’s overconfident; it’s possible one or both of us is incorrectly reading in / not reading in to what he wrote there, about his absolute level of confidence in solving the philosophical problems involved.)
Hmm, did you also read my immediate reply to him, where I made the point “if you’re the only philosopher in the team, how will others catch your mistakes?” How to understand his (then) plan except that he would have been willing to push the “launch” button even if there were zero other similarly capable philosophers available to scrutinize his philosophical ideas?
(Also just recording that I appreciate the OP and these threads, and people finding historical info. I think the topic of how “we” have been going wrong on strategy is important. I’m participating because I’m interested, though my contributions may not be very helpful because
I was a relative latecomer, in that much of the strategic direction (insofar as that existed) had already been fixed and followed;
I didn’t especially think about strategy that much initially, so I didn’t have many mental hooks for tracking what was happening in the social milieu in terms of strategic thinking and actions.)
(Oh I hadn’t read the full thread, now I have; still no big update? Like, I continue to see him being seemingly overconfident in his ability to get those solutions, but I’m not seeing “oh he would have mistakenly come to think he had a solution when he didn’t”, if that’s what you’re trying to say.)