Karnofsky’s focus on “tool AI” is useful but also his statement of it may confuse matters and needs refinement. I don’t think the distinction between “tool AI” and “agent AI” is sharp, or in quite the right place.
For example, the sort of robot cars we will probably have in a few years are clearly agents—you tell them to “come here and take me there” and they do it without further intervention on your part (when everything is working as planned). This is useful in a way that any amount and quality of question answering is not. Almost certainly there will be various flavors of robot cars available and people will choose the ones they like (that don’t drive in scary ways, that get them where they want to go even if it isn’t well specified, that know when to make conversation and when to be quiet, etc.) As long as robot cars just drive themselves and people around, can’t modify the world autonomously to make their performance better, and are subject to continuing selection by their human users, they don’t seem to be much of a threat.
The key points here seem to be (1) limited scope, (2) embedding in a network of other actors and (3) humans in the loop as evaluators. We could say these define “tool AIs” or come up with another term. But either way the antonym doesn’t seem to be “agent AIs” but maybe something like “autonomous AIs” or “independent AIs”—AIs with the power to act independently over a very broad range, unchecked by embedding in a network of other actors or by human evaluation.
Framed this way, we can ask “Why would independent AIs exist?” If the reason is mad scientists, an arms race, or something similar then Karnofsky has a very strong argument that any study of friendliness is beside the point. Outside these scenarios, the argument that we are likely to create independent AIs with any significant power seems weak; Karnofsky’s survey more or less matches my own less methodical findings. I’d be interested in strong arguments if they exist.
Given this analysis, there seem to be two implications:
We shouldn’t build independent AIs, and should organize to prevent their development if they seem likely.
We should thoroughly understand the likely future evolution of a patchwork of diverse tool AIs, to see where dangers arise.
For better or worse, neither of these lend themselves to tidy analytical answers, though analytical work would be useful for both. But they are very much susceptible to investigation, proposals, evangelism, etc.
These do lend themselves to collaboration with existing AI efforts. To the extent they perceive a significant risk of development of independent AIs in the foreseeable future, AI researchers will want to avoid that. I’m doubtful this is an active risk but could easily be convinced by evidence—not just abstract arguments—and I’m fairly sure they feel the same way.
Understanding the long term evolution of a patchwork of diverse tool AIs should interest just about all major AI developers, AI project funders, and long term planners who will be affected (which is just about all of them). Short term bias and ceterisparibus bias will lead to lots of these folks not engaging with the issue, but I think it will seem relevant to an increasing number as the hits keep coming.
That doesn’t help you if you need a car to take you someplace in the next hour or so, though. I think jed’s point is that sometimes it is useful for an AI to take action rather than merely provide information.
For example, the sort of robot cars we will probably have in a few years are clearly agents—you tell them to “come here and take me there” and they do it without further intervention on your part (when everything is working as planned). This is useful in a way that any amount and quality of question answering is not.
Yes I agree. Evidently, the environment cars work in is too fast-paced and quickly changing for “tool ai” to be close in usefulness to “agent ai.” To drive safely and effectively, you need to be making and implementing decisions on the time frame of a split second.
At the same time, the lesson to be learned is that useful ai can have a utility function which is pretty mundane—e.g. “find a fast route from point A to point B while minimizing the chances of running off the road or running into any people or objects.”
Similarly, instead of telling AI to “improve human welfare” we can tell it to do things like “find ways to kill cancerous cells while keeping collateral damage to a minimum.” The higher level decisions about improving human welfare can be left to the traditional institutions—legislatures, courts, and individual autonomy.
At the same time, the lesson to be learned is that useful ai can have a utility function which is pretty mundane—e.g. “find a fast route from point A to point B while minimizing the chances of running off the road or running into any people or objects.”
Self-driving cars aren’t piloted by AGIs in the first place, let alone dangerous “world-optimization” AGIs.
Similarly, instead of telling AI to “improve human welfare” we can tell it to do things like “find ways to kill cancerous cells while keeping collateral damage to a minimum.” The higher level decisions about improving human welfare can be left to the traditional institutions—legislatures, courts, and individual autonomy.
The whole point of Friendly AI is that we want something which is more effective at improving human welfare than our existing institutions. Our existing institutions are, by FAI standards, Unfriendly and destructive. Not existentially destructive, this is true (except on rare occasions like World War II), but neither are they trustworthy when handed, for instance, power over the life-and-death of Earth’s ecosystem (which they are currently failing to save, despite our having no other planet to go to).
I don’t engage with this poster because of his past dishonesty, i.e. misrepresenting my posts. If anyone not on my *(&^%-list is curious, I am happy to provide references.
I don’t engage with this poster because of his past dishonesty, i.e. misrepresenting my posts. If anyone not on my *(&^%-list is curious, I am happy to provide references.
I applaud your decision to not engage (as a good general strategy given your state of belief—the specifics of the conflict do not matter). I find it usually works best to do so without announcing it. Or, at least, by announcing it sparingly with extreme care to minimize the appearance of sniping.
Karnofsky’s focus on “tool AI” is useful but also his statement of it may confuse matters and needs refinement. I don’t think the distinction between “tool AI” and “agent AI” is sharp, or in quite the right place.
For example, the sort of robot cars we will probably have in a few years are clearly agents—you tell them to “come here and take me there” and they do it without further intervention on your part (when everything is working as planned). This is useful in a way that any amount and quality of question answering is not. Almost certainly there will be various flavors of robot cars available and people will choose the ones they like (that don’t drive in scary ways, that get them where they want to go even if it isn’t well specified, that know when to make conversation and when to be quiet, etc.) As long as robot cars just drive themselves and people around, can’t modify the world autonomously to make their performance better, and are subject to continuing selection by their human users, they don’t seem to be much of a threat.
The key points here seem to be (1) limited scope, (2) embedding in a network of other actors and (3) humans in the loop as evaluators. We could say these define “tool AIs” or come up with another term. But either way the antonym doesn’t seem to be “agent AIs” but maybe something like “autonomous AIs” or “independent AIs”—AIs with the power to act independently over a very broad range, unchecked by embedding in a network of other actors or by human evaluation.
Framed this way, we can ask “Why would independent AIs exist?” If the reason is mad scientists, an arms race, or something similar then Karnofsky has a very strong argument that any study of friendliness is beside the point. Outside these scenarios, the argument that we are likely to create independent AIs with any significant power seems weak; Karnofsky’s survey more or less matches my own less methodical findings. I’d be interested in strong arguments if they exist.
Given this analysis, there seem to be two implications:
We shouldn’t build independent AIs, and should organize to prevent their development if they seem likely.
We should thoroughly understand the likely future evolution of a patchwork of diverse tool AIs, to see where dangers arise.
For better or worse, neither of these lend themselves to tidy analytical answers, though analytical work would be useful for both. But they are very much susceptible to investigation, proposals, evangelism, etc.
These do lend themselves to collaboration with existing AI efforts. To the extent they perceive a significant risk of development of independent AIs in the foreseeable future, AI researchers will want to avoid that. I’m doubtful this is an active risk but could easily be convinced by evidence—not just abstract arguments—and I’m fairly sure they feel the same way.
Understanding the long term evolution of a patchwork of diverse tool AIs should interest just about all major AI developers, AI project funders, and long term planners who will be affected (which is just about all of them). Short term bias and ceteris paribus bias will lead to lots of these folks not engaging with the issue, but I think it will seem relevant to an increasing number as the hits keep coming.
“how do I build an automated car?”
That doesn’t help you if you need a car to take you someplace in the next hour or so, though. I think jed’s point is that sometimes it is useful for an AI to take action rather than merely provide information.
Yes I agree. Evidently, the environment cars work in is too fast-paced and quickly changing for “tool ai” to be close in usefulness to “agent ai.” To drive safely and effectively, you need to be making and implementing decisions on the time frame of a split second.
At the same time, the lesson to be learned is that useful ai can have a utility function which is pretty mundane—e.g. “find a fast route from point A to point B while minimizing the chances of running off the road or running into any people or objects.”
Similarly, instead of telling AI to “improve human welfare” we can tell it to do things like “find ways to kill cancerous cells while keeping collateral damage to a minimum.” The higher level decisions about improving human welfare can be left to the traditional institutions—legislatures, courts, and individual autonomy.
Self-driving cars aren’t piloted by AGIs in the first place, let alone dangerous “world-optimization” AGIs.
The whole point of Friendly AI is that we want something which is more effective at improving human welfare than our existing institutions. Our existing institutions are, by FAI standards, Unfriendly and destructive. Not existentially destructive, this is true (except on rare occasions like World War II), but neither are they trustworthy when handed, for instance, power over the life-and-death of Earth’s ecosystem (which they are currently failing to save, despite our having no other planet to go to).
[ . . . ]
I don’t engage with this poster because of his past dishonesty, i.e. misrepresenting my posts. If anyone not on my *(&^%-list is curious, I am happy to provide references.
I applaud your decision to not engage (as a good general strategy given your state of belief—the specifics of the conflict do not matter). I find it usually works best to do so without announcing it. Or, at least, by announcing it sparingly with extreme care to minimize the appearance of sniping.