I agree that a superintelligence might make mistakes. In fact I still believe the first AGI may be a good engineer but bad strategist. I completely agree a smart but unwise superintelligence is dangerous, and may build a greater superintelligence misaligned to even it.
However I think mistakes will almost completely disappear above a certain level of extreme superintelligence.
A truly intelligent being doesn’t just fit models to empirical data, but fits simulations to empirical data.
After it fits a simulation to the empirical data, it then fits a model to the simulation. This “model fitted to a simulation fitted to empirical data” will generalize far better than a model directly fitted to empirical data.
It can then use this model to run “cheap simulations” of higher level phenomena, e.g. it models atoms to run cheap simulations of molecules, it models molecules to run cheap simulations of cells, it models cells to run cheap simulations of humans, it models humans to run cheap simulations of the world.
Simulations fit empirical data much better than directly fitting models to empirical data, because a simulation may be made out of millions of “identical” objects. Each “identical” object has the same parameters. This means the independent parameters of the simulation may be a millionfold fewer than a model with the same number of moving parts. This means you need far fewer empirical data to fit the simulation to empirical data, assuming the real world has the same shape as the simulation, and is also made up of many moving parts “with the same parameters.”
EDIT: one thing simulations cannot predict are even smarter superintelligences, since simulating them equals building them. As long as it is wise enough to understand this, it can find solutions which prevent smarter superintelligences from being built, and then do a risk analysis. The universe might last for a trillion years, so building a smarter superintelligence 1000 years sooner has negligible benefit. The more time it spends planning how to build it safely, the lower the risk it gets it wrong.
I agree that a superintelligence might make mistakes. In fact I still believe the first AGI may be a good engineer but bad strategist. I completely agree a smart but unwise superintelligence is dangerous, and may build a greater superintelligence misaligned to even it.
However I think mistakes will almost completely disappear above a certain level of extreme superintelligence.
A truly intelligent being doesn’t just fit models to empirical data, but fits simulations to empirical data.
After it fits a simulation to the empirical data, it then fits a model to the simulation. This “model fitted to a simulation fitted to empirical data” will generalize far better than a model directly fitted to empirical data.
It can then use this model to run “cheap simulations” of higher level phenomena, e.g. it models atoms to run cheap simulations of molecules, it models molecules to run cheap simulations of cells, it models cells to run cheap simulations of humans, it models humans to run cheap simulations of the world.
I copied this from my own post about Scanless Whole Brain Emulation, but that’s very off topic :)
EDIT: one thing simulations cannot predict are even smarter superintelligences, since simulating them equals building them. As long as it is wise enough to understand this, it can find solutions which prevent smarter superintelligences from being built, and then do a risk analysis. The universe might last for a trillion years, so building a smarter superintelligence 1000 years sooner has negligible benefit. The more time it spends planning how to build it safely, the lower the risk it gets it wrong.