As much as it maybe ruins the fun for me to just point out the message: the major point of the story was that you weren’t supposed to condition on us knowing that nuclear weapons are real, and instead ask whether the Gradualist or Catastrophist’s arguments actually make sense given what they knew.
That’s the situation I think we’re in with Fast AI Takeoff. We’re trying to interpret what the existence of general intelligences like humans (the Sun) implies for future progress on ML algorithms (normal explosives), without either a clear underlying theory for what the Sun’s power really is, or any direct evidence that there’ll be a jump.
Richard Ngo: You don’t have a specific argument about utility functions and their relationship to AGIs in a precise, technical way. Instead, it’s more like utility functions are like a pointer towards the type of later theory that will give us a much more precise understanding of how to think about intelligence and agency and AGIs pursuing goals and so on. And to Eliezer, it seems like we’ve got a bunch of different handles on what the shape of this larger scale theory might look like, but he can’t really explain it in precise terms. It’s maybe in the same way that for any other scientific theory, before you latch onto it, you can only gesture towards a bunch of different intuitions that you have and be like, “Hey guys, there are these links between them that I can’t make precise or rigorous or formal at this point.”
As much as it maybe ruins the fun for me to just point out the message: the major point of the story was that you weren’t supposed to condition on us knowing that nuclear weapons are real, and instead ask whether the Gradualist or Catastrophist’s arguments actually make sense given what they knew.
That’s the situation I think we’re in with Fast AI Takeoff. We’re trying to interpret what the existence of general intelligences like humans (the Sun) implies for future progress on ML algorithms (normal explosives), without either a clear underlying theory for what the Sun’s power really is, or any direct evidence that there’ll be a jump.
That remark about the ‘micro-foundational explanation for why the sun looks qualitatively new but really isn’t’ refers to Richard Ngo’s explanation of why humans are so much better than chimps: https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/gf9hhmSvpZfyfS34B#13_1__Alignment_difficulty_debate__Richard_Ngo_s_case