At best, I think they may’ve encoded a simple objective definition of a convergent AI drive like ‘keep your options open and acquire any kind of influence’ but nothing in it seems to map onto utility functions or anything like that.
I think that’s an accurate informal summary of their basic mechanism. Personally, I’m not impressed by utility functions (or much else in AGI, for that matter), so I don’t rate the fact that they aren’t using them as a point against.
I do, because it seems like in any nontrivial situation, simply grasping for entropies ignores the point of having power or options, which is to aim at some state of affairs which is more valuable than others. Simply buying options is worthless and could well be actively harmful if you keep exposing yourself to risks you could’ve shut down. They mention that it works well as a strategy in Go playing… but I can’t help but think that it must be in situations where it’s not feasible to do any board evaluation at all and where one is maximally ignorant about the value of anything at that point.
I do, because it seems like in any nontrivial situation, simply grasping for entropies ignores the point of having power or options, which is to aim at some state of affairs which is more valuable than others.
As I understand it, it’s more a denial of that claim. The point is to maximise entropy, and values are a means to that end.
Obviously, this is counter-intuitive, since orthodoxy has this relationship the other way around: claiming that organisms maximise correlates of their own power—and the entropy they produce is a byproduct. MaxEnt suggests that this perspective may have things backwards.
“Value” is just another word for “utility”, isn’t it? It’s the whole idea of utility maximisation as a fundamental principle that I think is misguided. No, I don’t have a better idea; I just think that that one is a road that, where AGI is concerned, leads nowhere.
But AGI is not something I work on. There is no reason for anyone who does to pay any attention to my opinions on the subject.
I think that’s an accurate informal summary of their basic mechanism. Personally, I’m not impressed by utility functions (or much else in AGI, for that matter), so I don’t rate the fact that they aren’t using them as a point against.
I do, because it seems like in any nontrivial situation, simply grasping for entropies ignores the point of having power or options, which is to aim at some state of affairs which is more valuable than others. Simply buying options is worthless and could well be actively harmful if you keep exposing yourself to risks you could’ve shut down. They mention that it works well as a strategy in Go playing… but I can’t help but think that it must be in situations where it’s not feasible to do any board evaluation at all and where one is maximally ignorant about the value of anything at that point.
As I understand it, it’s more a denial of that claim. The point is to maximise entropy, and values are a means to that end.
Obviously, this is counter-intuitive, since orthodoxy has this relationship the other way around: claiming that organisms maximise correlates of their own power—and the entropy they produce is a byproduct. MaxEnt suggests that this perspective may have things backwards.
what’s your preferred system for encoding values?
“Value” is just another word for “utility”, isn’t it? It’s the whole idea of utility maximisation as a fundamental principle that I think is misguided. No, I don’t have a better idea; I just think that that one is a road that, where AGI is concerned, leads nowhere.
But AGI is not something I work on. There is no reason for anyone who does to pay any attention to my opinions on the subject.