However, if it is the case that the difference between humans and monkeys is mostly due to a one-shot discrete difference (ie language), then this cannot necessarily be repeated to get a similar gain in intelligence a second time.
Perhaps language is a zero-one, i.e. language renders a mind “cognitively complete” in the sense that the mind can represent anything about the external world, and make any inferences using those representations. But intelligence is not thereby zero-one because intelligence depends on continuous variables like computional speed, memory, etc.
More concretely, I am sceptic that “we end up with AI geniuses, but not AI gods”, because running a genius at 10,000x speed, parallelised over 10,000x cores, with instantaneous access to the internet does (I think) make an AI god. A difference is quantity is a difference in kind.
Thar said, there might exist plausible threat models which require an AI which doesn’t spatiotemporally decompose into less smart AIs. Could you sketch one out?
So, the claim is (of course) not that intelligence is zero-one. We know that this is not the case, from the fact that some people are smarter than other people.
Perhaps language is a zero-one, i.e. language renders a mind “cognitively complete” in the sense that the mind can represent anything about the external world, and make any inferences using those representations. But intelligence is not thereby zero-one because intelligence depends on continuous variables like computional speed, memory, etc.
More concretely, I am sceptic that “we end up with AI geniuses, but not AI gods”, because running a genius at 10,000x speed, parallelised over 10,000x cores, with instantaneous access to the internet does (I think) make an AI god. A difference is quantity is a difference in kind.
Thar said, there might exist plausible threat models which require an AI which doesn’t spatiotemporally decompose into less smart AIs. Could you sketch one out?
So, the claim is (of course) not that intelligence is zero-one. We know that this is not the case, from the fact that some people are smarter than other people.
As for the other two points, see this comment and this comment.