However, for the large part, the world model would differ, in totality, very little between a paperclip maximizer and a friendly AI. While the Friendly AI certainly has to keep track of more things which are irrelevant to the paperclip maximizer, both AIs would have to have world models which have to be able to model human behavior in order for the AIs to be effective, which one would expect would account for the bulk of the complexity of the world model in the first place.
This is, as I understand it, the crux of the argument. Perhaps it takes an AI of complexity 10 to model the world well enough to interact with it and pursue simple values, but an AI of complexity 11 to model the world well enough to understand and preserve human values. If fooming is possible, that means any AIs of complexity 10 will take over the world and not preserve human values, and the only way to get a friendly AI is for no one to make an AI of complexity 10 and the first AI to be complexity 11 (and human-friendly).
This is, as I understand it, the crux of the argument. Perhaps it takes an AI of complexity 10 to model the world well enough to interact with it and pursue simple values, but an AI of complexity 11 to model the world well enough to understand and preserve human values. If fooming is possible, that means any AIs of complexity 10 will take over the world and not preserve human values, and the only way to get a friendly AI is for no one to make an AI of complexity 10 and the first AI to be complexity 11 (and human-friendly).