I think one way that people get a little caught up when thinking about the possibility of superintelligence is via the typical mind fallacy. Someone says “a superintelligence could theoretically do [super hard thing]” and my intuitive response is: “Yeah, but how plausible is that? Why would it even bother? That seems like it would take a lot of effort, and it probably wouldn’t even work.” By default I anchor on my own mind’s capabilities (and motivation, even), instead of trying to think in information-theoretic terms, or figuring out what does and doesn’t violate the laws of physics.
I think one way that people get a little caught up when thinking about the possibility of superintelligence is via the typical mind fallacy. Someone says “a superintelligence could theoretically do [super hard thing]” and my intuitive response is: “Yeah, but how plausible is that? Why would it even bother? That seems like it would take a lot of effort, and it probably wouldn’t even work.” By default I anchor on my own mind’s capabilities (and motivation, even), instead of trying to think in information-theoretic terms, or figuring out what does and doesn’t violate the laws of physics.