I think we’re misunderstanding each other. You seemed to think that this “He talks about that and thinks Oracle AIs are distinct from tools.” was a mistake.
I understand Holden to be trying to invent a new category of AI, called “tool-AI”, which is not just an AGI with a utility function for answering questions nor a UFAI in a box (he may be wrong about which definition/interpretation is more popular, but that’s mostly irrelevant to his claim because he’s just trying to distinguish his idea from these other ideas). He claims that this category has not been discussed much.
He says “Yes, I agree AI’s with utility functions for answering questions will do terrible things just like UFAI in a box, but my idea is qualitatively different either of these, and it hasn’t been discussed”.
I think we’re misunderstanding each other. You seemed to think that this “He talks about that and thinks Oracle AIs are distinct from tools.” was a mistake.
I understand Holden to be trying to invent a new category of AI, called “tool-AI”, which is not just an AGI with a utility function for answering questions nor a UFAI in a box (he may be wrong about which definition/interpretation is more popular, but that’s mostly irrelevant to his claim because he’s just trying to distinguish his idea from these other ideas). He claims that this category has not been discussed much.
He says “Yes, I agree AI’s with utility functions for answering questions will do terrible things just like UFAI in a box, but my idea is qualitatively different either of these, and it hasn’t been discussed”.
Are we still talking past each other?
Probably not.