I think part of this is actually just a failure of capability.
Point (1) and (2) can be true, while (3) only applies if asked. If not asked, it doesn’t evaluate whether it would result in humanity being destroyed (or a database being deleted). It is not capable of correlating the knowledge that it has without specifically aiming to do those correlations.
Humans also do this all the time, of course.
That said, I would generally think of a superintelligent entity as being more coherent in what it knows, in the sense that at least all the reasonably obvious and relevant correlations are taken into account and used to predict evaluate actions before acting—even when not specifically asked about each of them.
That may be oversimplified though, since I would also expect a superintelligent entity to consider a great deal more things “reasonably obvious”. The space of “obvious correlations” between them all likely grows much faster than the number of obvious things themselves, which could mean that it is exponentially difficult to take them all into account.
So maybe it is not practicallypossible to be both superintelligent and coherent in this fairly strong sense.
I would expect one’s ability to “correlate knowledge” to be strongly related to one’s intelligence, but it’s not clear to me that a super-LLM would have a near-zero probability of accidentally deleting humanity. What level of knowledge-correlation ability is required to consistently avoid fatal mistakes? The level of the average LW user is not high enough, because we still do things like poison ourselves.
I think part of this is actually just a failure of capability.
Point (1) and (2) can be true, while (3) only applies if asked. If not asked, it doesn’t evaluate whether it would result in humanity being destroyed (or a database being deleted). It is not capable of correlating the knowledge that it has without specifically aiming to do those correlations.
Humans also do this all the time, of course.
That said, I would generally think of a superintelligent entity as being more coherent in what it knows, in the sense that at least all the reasonably obvious and relevant correlations are taken into account and used to predict evaluate actions before acting—even when not specifically asked about each of them.
That may be oversimplified though, since I would also expect a superintelligent entity to consider a great deal more things “reasonably obvious”. The space of “obvious correlations” between them all likely grows much faster than the number of obvious things themselves, which could mean that it is exponentially difficult to take them all into account.
So maybe it is not practically possible to be both superintelligent and coherent in this fairly strong sense.
I would expect one’s ability to “correlate knowledge” to be strongly related to one’s intelligence, but it’s not clear to me that a super-LLM would have a near-zero probability of accidentally deleting humanity. What level of knowledge-correlation ability is required to consistently avoid fatal mistakes? The level of the average LW user is not high enough, because we still do things like poison ourselves.