I think a certain sort of urgency comes naturally from the finite lifespan of all things. So let’s say we have an AI with a 0.00001% chance of malfunctioning or being destroyed per year, that gets 100 utility if it is in Africa. Should it go now, or should it wait a year? Well, all things being equal, the expected utility of the “go now” strategy is veeeery slightly larger than the expected utility of waiting a year. So an expected utility maximizer with any sort of risk of death would choose to act sooner rather than later.
Yes, a full-blown expected utility maximizer, with a utility-function enclosing goals with detailed enough optimization parameters to make useful utility calculations about real-world causal relationships relative to its own self-perception. I think that before something like that becomes possible, some other less sophisticated intelligence will have already been employed as a tool to do, or solve, something that destroys pretty much everything. An AI that can solve bio or nanotech problems should be much easier to design than one that can destroy the world as a side-effect of unbounded self-improvement. And only the latter category is subject to friendliness research.
I don’t doubt that the kind of intelligence you have in mind has certain drives that cause it to take certain actions. But that’s not what I would call “basic AI drives”, rather a certain kind of sophisticated AGI design that is unlikely to come into existence as a result of ignorance or unpredictable implications of a design someone stumbled upon by luck.
Well, okay. Restricting the validity of the “AI drives” to general AIs seems reasonable. After all, the local traffic lights have not started trying to refine their goals, and PageRank has not yet started manipulating society to eliminate threats to its well-being.
I think that before something like that becomes possible, some other less sophisticated intelligence will have already been employed as a tool to do, or solve, something that destroys pretty much everything.
DOOM!
An AI that can solve bio or nanotech problems should be much easier to design than one that can destroy the world as a side-effect of unbounded self-improvement.
Probably true.
And only the latter category is subject to friendliness research.
Hmm. Possibly some research avenues are more promising than others—but this sounds a bit broad.
I think a certain sort of urgency comes naturally from the finite lifespan of all things. So let’s say we have an AI with a 0.00001% chance of malfunctioning or being destroyed per year, that gets 100 utility if it is in Africa. Should it go now, or should it wait a year? Well, all things being equal, the expected utility of the “go now” strategy is veeeery slightly larger than the expected utility of waiting a year. So an expected utility maximizer with any sort of risk of death would choose to act sooner rather than later.
Yes, a full-blown expected utility maximizer, with a utility-function enclosing goals with detailed enough optimization parameters to make useful utility calculations about real-world causal relationships relative to its own self-perception. I think that before something like that becomes possible, some other less sophisticated intelligence will have already been employed as a tool to do, or solve, something that destroys pretty much everything. An AI that can solve bio or nanotech problems should be much easier to design than one that can destroy the world as a side-effect of unbounded self-improvement. And only the latter category is subject to friendliness research.
I don’t doubt that the kind of intelligence you have in mind has certain drives that cause it to take certain actions. But that’s not what I would call “basic AI drives”, rather a certain kind of sophisticated AGI design that is unlikely to come into existence as a result of ignorance or unpredictable implications of a design someone stumbled upon by luck.
Well, okay. Restricting the validity of the “AI drives” to general AIs seems reasonable. After all, the local traffic lights have not started trying to refine their goals, and PageRank has not yet started manipulating society to eliminate threats to its well-being.
So far as we know.
DOOM!
Probably true.
Hmm. Possibly some research avenues are more promising than others—but this sounds a bit broad.