This doesn’t matter for predicting the outcome of a hypothetical war between 16th century Britain and 21st century USA.
If AI systems can make 500 years of progress before we notice it’s uncontrolled, it’s already assuming it’s a insanely strong superintelligence.
We could probably understand how a von Neumann probe or an anti-aging cure worked too, if someone taught us.
Probably, if it’s of a type we can imagine and is comprehensible in those terms—but that’s assuming the conclusion! As Gwern noted, we can’t understand chess endgames. Similarly, in the case of a strong ASI, the ASI- created probe or cure could look more like a random set of actions that aren’t explainable in our terms which cause the outcome than it does like an engineered / purpose driven system that is explainable at all.
As Gwern noted, we can’t understand chess endgames.
On this example specifically, a) it’s possible AI is too stupid to have a theory of mind of humans such that it can write good chess textbooks on these endgames. Maybe there is an elegant way of looking at it that isn’t brute force b) chess endgames are amenable to brute force in a way that “invent a microscope” is not. Scientific discovery is searching through an exponential space so you need a good heuristic or model for every major step you take, you can’t brute force it.
I agree tech beyond human comprehension is possible. I’m just giving an intuition as to why a lot of radically powerful tech likely still lies within human comprehension. 500 [1] years of progress is likely to still be within comprehension, so is 50 years or 5 years.
The most complex tech that exists in the universe is arguably human brains themselves and we could probably understand a good fraction of their working too, if someone explained it.
Important point here being the AI has to want to explain it in simple terms to us.
If you get a 16th century human to visit a nuclear facility for a day that’s not enough information for them to figure out what it does or how it works. You need to provide them textbooks that break down each of the important concepts.
[1] society in 2000 is explainable to society in 1500 but society in 2500 may or may not be explainable to society in 2000 because acceleration
I think you are fooling yourself about how similar people in 1600 are to people today. The average person at the time was illiterate, superstitious, and could maybe do single digit addition and subtraction. You’re going to explain nuclear physics?
If AI systems can make 500 years of progress before we notice it’s uncontrolled, it’s already assuming it’s a insanely strong superintelligence.
Probably, if it’s of a type we can imagine and is comprehensible in those terms—but that’s assuming the conclusion! As Gwern noted, we can’t understand chess endgames. Similarly, in the case of a strong ASI, the ASI- created probe or cure could look more like a random set of actions that aren’t explainable in our terms which cause the outcome than it does like an engineered / purpose driven system that is explainable at all.
On this example specifically, a) it’s possible AI is too stupid to have a theory of mind of humans such that it can write good chess textbooks on these endgames. Maybe there is an elegant way of looking at it that isn’t brute force b) chess endgames are amenable to brute force in a way that “invent a microscope” is not. Scientific discovery is searching through an exponential space so you need a good heuristic or model for every major step you take, you can’t brute force it.
I agree tech beyond human comprehension is possible. I’m just giving an intuition as to why a lot of radically powerful tech likely still lies within human comprehension. 500 [1] years of progress is likely to still be within comprehension, so is 50 years or 5 years.
The most complex tech that exists in the universe is arguably human brains themselves and we could probably understand a good fraction of their working too, if someone explained it.
Important point here being the AI has to want to explain it in simple terms to us.
If you get a 16th century human to visit a nuclear facility for a day that’s not enough information for them to figure out what it does or how it works. You need to provide them textbooks that break down each of the important concepts.
[1] society in 2000 is explainable to society in 1500 but society in 2500 may or may not be explainable to society in 2000 because acceleration
I think you are fooling yourself about how similar people in 1600 are to people today. The average person at the time was illiterate, superstitious, and could maybe do single digit addition and subtraction. You’re going to explain nuclear physics?
There is a similar hypothesis that is testable. Find someone who is illiterate and superstitious today and fund their education upto university level.
Edit: Bonus points if they are selected from an isolated tribal community existing today