I have problems getting the first point. If bugs are hard to find then shouldn’t this precisely entail that dangerous AI is hard to differentiate from benign AI?! Any literature you can suggest on the subject?
Regarding the second point. I don’t find Eliezer’s idea entirely convincing. But I don’t think the fire thesis hinges on his view. Rather, it is built on the much weaker and simpler view that if we don’t know the utility function of some AGI system then this system is dangerous—I find it very hard to see any convincing reasons for thinking this is false. Eliezer thinks doom is default. I just assume that ignorance makes it rational to air on the side of caution.
I have problems getting the first point. If bugs are hard to find then shouldn’t this precisely entail that dangerous AI is hard to differentiate from benign AI?! Any literature you can suggest on the subject?
Regarding the second point. I don’t find Eliezer’s idea entirely convincing. But I don’t think the fire thesis hinges on his view. Rather, it is built on the much weaker and simpler view that if we don’t know the utility function of some AGI system then this system is dangerous—I find it very hard to see any convincing reasons for thinking this is false. Eliezer thinks doom is default. I just assume that ignorance makes it rational to air on the side of caution.