My argument doesn’t require that anybody be able to formally define “self” or “maximize paperclips”; it doesn’t require the goal G to be picked among those that are easily defined in English.
Show 3 examples of goal G. Somewhere I’ve read awesome technique for avoiding the abstraction mistakes—asking to show 3 examples.
What’s the point? Are you going to nitpick that my goals aren’t formal enough, even though I’m not making any claim at all about what kind of goals those could be?
Are you claiming that it’s impossible for an agent to have goals? That the set of goals that it’s even conceivable for an AI to have (without immediately wireheading or something) is much narrower than what most people here assume?
I’m not even sure what this disagreement is about right now, or even if there is a disagreement.
Ya, I think the set of goals is very narrow. The AI here starts of Descartes level genius and proceeds to self preserve, understand the map-territory distinction for non-wireheading, foreseeing the possibility that instrumental goals which look good may destroy the terminal goal, and such.
The AI I imagine starts off stupid and has some really narrowly (edit: or should i say, short-foresighted) self improving non self destructive goal likely having to do with maximization of complexity in some way. Think evolution, don’t think fully grown Descartes waking up after amnesia. It ain’t easy to reinvent the ‘self’. It’s also not easy to look at agent (yourself) and say—wow, this agent works to maximize G—without entering infinite recursion. We humans, if we escaped out of our universe into some super-universe, we might wreck some havoc but we’d sacrifice a bit of utility to preserve anything resembling life. Why? Well, we started stupid, and that’s how we got our goals.
Show 3 examples of goal G. Somewhere I’ve read awesome technique for avoiding the abstraction mistakes—asking to show 3 examples.
What’s the point? Are you going to nitpick that my goals aren’t formal enough, even though I’m not making any claim at all about what kind of goals those could be?
Are you claiming that it’s impossible for an agent to have goals? That the set of goals that it’s even conceivable for an AI to have (without immediately wireheading or something) is much narrower than what most people here assume?
I’m not even sure what this disagreement is about right now, or even if there is a disagreement.
Ya, I think the set of goals is very narrow. The AI here starts of Descartes level genius and proceeds to self preserve, understand the map-territory distinction for non-wireheading, foreseeing the possibility that instrumental goals which look good may destroy the terminal goal, and such.
The AI I imagine starts off stupid and has some really narrowly (edit: or should i say, short-foresighted) self improving non self destructive goal likely having to do with maximization of complexity in some way. Think evolution, don’t think fully grown Descartes waking up after amnesia. It ain’t easy to reinvent the ‘self’. It’s also not easy to look at agent (yourself) and say—wow, this agent works to maximize G—without entering infinite recursion. We humans, if we escaped out of our universe into some super-universe, we might wreck some havoc but we’d sacrifice a bit of utility to preserve anything resembling life. Why? Well, we started stupid, and that’s how we got our goals.