The AI is asked mathematical questions relating to the big unsolved problems of physics: quantum mechanics vs. general relativity, the structure of space-time at the Planck scale, dark matter, dark energy, and so on. Its answers enable the easy construction of superweapons that convert matter directly into energy; or make anything radioactive decay instantly to its end state; or collapse the false vacuum; etc. Shortly thereafter, everyone dies.
The AI is asked mathematical questions relating to the big unsolved problems of physics: quantum mechanics vs. general relativity, the structure of space-time at the Planck scale, dark matter, dark energy, and so on. Its answers enable the easy construction of superweapons that convert matter directly into energy; or make anything radioactive decay instantly to its end state; or collapse the false vacuum; etc. Shortly thereafter, everyone dies.
ETA: see also.
The problem here is that there can be questions whose answers are dangerous to know.