I haven’t read Superintelligence (so maybe I shouldn’t be posting here?), but I have read much of Bostrom earlier work, and yes I object to Bostrom’s prior characterisations of tool proposals, which basically amount to strawman setups. Either that or i misunderstood the original tool AI posts and they really were that bad :\ In my mind at least, “tool AI” is simply a proposal for using oracle AI in a specific way, as a sort of intelligence augmentation for its human operators.
Regarding energy constraints, that’s just one of an entire spectrum of possibilities. Approaching the problem from the other direction leads to suggestions such as cryptographically secure computational traces: you basically demand the AI to provide a subset of its call graph history as an audit log, but in a way that the AI doesn’t know ahead of time what execution pathways will be demanded of it. You then have some sort of audit log checker or manual inspection of the trace and the instrumental goals it reveals. Faking the log would require a detectable amount of work not accounted for in the trace.
It is possible to construct much better boxes than critics of oracle AI would have you believe.
How do energy constraints defeat a strategy of this form:
While energy constraints are low, answer easy questions, but fake failure on some hard questions claiming lack of resources
Once given enough excess energy beyond that needed for a task, use the excess computing power to figure out a way to do unfriendly things
Do cryptographic traces place a limit on the complexity of tasks that you can solve? It seems like as task complexity increases, it will become harder to spot deception amongst legitimate computations (or the amount of human attention you need to achieve a fixed probability of detection scales according to problem complexity).
You can’t fake failure. Whether it responds in time or not, a trace is provided. That trace, if it validates, provides a record of everything the program did (or rather a randomly selected subset, but that subset is not knowable in advance), even if it didn’t lead to a usable answer. If not valid, you terminate and never run again.
To your second paragraph, these are open questions. But that’s precisely my point—I think MIRI should be engaged in researching these sorts of things, and not be biased towards areas relating to their own preconceived notions of what an AGI experiment should look like.
I haven’t read Superintelligence (so maybe I shouldn’t be posting here?), but I have read much of Bostrom earlier work, and yes I object to Bostrom’s prior characterisations of tool proposals, which basically amount to strawman setups. Either that or i misunderstood the original tool AI posts and they really were that bad :\ In my mind at least, “tool AI” is simply a proposal for using oracle AI in a specific way, as a sort of intelligence augmentation for its human operators.
Regarding energy constraints, that’s just one of an entire spectrum of possibilities. Approaching the problem from the other direction leads to suggestions such as cryptographically secure computational traces: you basically demand the AI to provide a subset of its call graph history as an audit log, but in a way that the AI doesn’t know ahead of time what execution pathways will be demanded of it. You then have some sort of audit log checker or manual inspection of the trace and the instrumental goals it reveals. Faking the log would require a detectable amount of work not accounted for in the trace.
It is possible to construct much better boxes than critics of oracle AI would have you believe.
How do energy constraints defeat a strategy of this form: While energy constraints are low, answer easy questions, but fake failure on some hard questions claiming lack of resources Once given enough excess energy beyond that needed for a task, use the excess computing power to figure out a way to do unfriendly things
Do cryptographic traces place a limit on the complexity of tasks that you can solve? It seems like as task complexity increases, it will become harder to spot deception amongst legitimate computations (or the amount of human attention you need to achieve a fixed probability of detection scales according to problem complexity).
You can’t fake failure. Whether it responds in time or not, a trace is provided. That trace, if it validates, provides a record of everything the program did (or rather a randomly selected subset, but that subset is not knowable in advance), even if it didn’t lead to a usable answer. If not valid, you terminate and never run again.
To your second paragraph, these are open questions. But that’s precisely my point—I think MIRI should be engaged in researching these sorts of things, and not be biased towards areas relating to their own preconceived notions of what an AGI experiment should look like.