I suspect that Holden is imagining tool AGI as both transparent and much less generally powerful than what you are discussing.
Much of your objection inherently relies on the tool making vast chains of inferences in its plans that are completely opaque to human operators.
Many current AI systems have an internal knowledge representation that is human-readable and their inference and planning systems are thus transparently debuggable. Even neuroscience heavy designs could be made transparent: for example human brains have an inner monologue which could be recorded and made available for external debugging/monitoring for brain emulation type designs.
If the AGI is vastly more complex and faster-thinking, certainly monitoring may become more difficult in proportion, but it’s hardly clear that monitoring necessarily becomes impossible and necessarily out of the reach of algorithmic optimization.
And even then one could employ lesser monitoring AGI’s to bridge the gap so to speak, so that humans can transparently montior AGI1, which monitors AGI2, and so on.
Sure transparency applies to agent AGI as well and undoubtedly has been discusses here before, but it’s much more useful for holden’s more constrained ‘tool-AGI’ notion. Moreover, Holden’s view seems to imply transparency whereas your attack implies opaqueness.
I suspect that Holden is imagining tool AGI as both transparent and much less generally powerful than what you are discussing.
Much of your objection inherently relies on the tool making vast chains of inferences in its plans that are completely opaque to human operators.
Many current AI systems have an internal knowledge representation that is human-readable and their inference and planning systems are thus transparently debuggable. Even neuroscience heavy designs could be made transparent: for example human brains have an inner monologue which could be recorded and made available for external debugging/monitoring for brain emulation type designs.
If the AGI is vastly more complex and faster-thinking, certainly monitoring may become more difficult in proportion, but it’s hardly clear that monitoring necessarily becomes impossible and necessarily out of the reach of algorithmic optimization.
And even then one could employ lesser monitoring AGI’s to bridge the gap so to speak, so that humans can transparently montior AGI1, which monitors AGI2, and so on.
Sure transparency applies to agent AGI as well and undoubtedly has been discusses here before, but it’s much more useful for holden’s more constrained ‘tool-AGI’ notion. Moreover, Holden’s view seems to imply transparency whereas your attack implies opaqueness.