Some people (me included) value a certain level of non-manipulation. I’m trying to cash out that instinct. And it’s also needed for some ideas like corrigibility. Manipulation also combines poorly with value learning, see eg our paper here https://arxiv.org/abs/2004.13654
I do agree that saving the world is a clearly positive case of that ^_^
I have an article on “Anthropic decision theory”. with the video version here.
Basically, it’s not that the presumptuous philosopher is more likely to be right in a given universe, its that there are far more presumptuous philosophers in the large universe. So if we count “how many presumptuous philosophers are correct”, we get a different answer to “in how many universes is the presumptuous philosopher correct”. These things only come apart in anthropic situations.
Suart, by ” is complex” are you referring to...
I mean that that defining can be done in many different ways, and hence has a lot of contingent structure. In contrast, in , the $\rho is a complex distribution on , conditional on ; hence itself is trivial and just encodes “apply to and in the obvious way.
This is a link to “An Increasingly Manipulative Newsfeed” about potential social media manipulation incentives (eg FaceBook).
I’m putting the link here because I keep losing the original post (since it wasn’t published by me, but I co-wrote it).
A boundedly-rational agent is assumed to be mostly rational, failing to be fully rational because of a failure to figure things out in enough detail.
Humans are occasionally rational, often biased, often inconsistent, sometimes consciously act against their best interests, often follow heuristics without thinking, sometimes do think things through. This doesn’t seem to correspond to what is normally understood as “boundedly-rational”.
that paper was about fitting observations of humans to a mathematical model of “boundedly-rational agent pursuing a utility function”
It was “any sort of agent pursuing a reward function”.
We don’t need a special module to get an everyday definition of doorknobs, and likewise I don’t think we don’t need a special module to get an everyday definition of human motivation.
I disagree. Doornobs exist in the world (even if the category is loosely defined, and has lots of edge cases), whereas goals/motivations are interpretations that we put upon agents. The main result of the Occam’s razor paper is that there the goals of an agent are not something that you can know without putting your own interpretation on it—even if you know every physical fact about the universe. And two very different interpretations can be equally valid, with no way of distinguishing between them.
(I like the anthropomorphising/dehumanising symmetry, but I’m focusing on the aspects of dehumanising that cause you to make errors of interpretation. For example, out-groups are perceived as being coherent, acting in concert without disagreements, and often being explicitly evil. This is an error, not just a reduction in social emotions)
For instance throughout history people have been able to model and interact with traders from neighbouring or distant civilizations, even though they might think very differently.
Humans think very very similarly to each other, compared with random minds from the space of possible minds. For example, we recognise anger, aggression, fear, and so on, and share a lot of cultural universals https://en.wikipedia.org/wiki/Cultural_universal
There haven’t been as many big accomplishments.
I think we should look at the demand side, not the supply side. We are producing lots of technological innovations, but there aren’t so many major problems left for them to solve. The flush toilet was revolutionary; a super-flush ecological toilet with integrated sensors that can transform into a table… is much more advanced from the supply side, but barely more from the demand side: it doesn’t fulfil many more needs than the standard flush toilet.
Cool, good summary.
Humans have a theory of mind, that makes certain types of modularizations easier. That doesn’t mean that the same modularization is simple for an agent that doesn’t share that theory of mind.
Then again, it might be. This is worth digging into empirically. See my post on the optimistic and pessimistic scenarios; in the optimistic scenario, preferences, human theory of mind, and all the other elements, are easy to deduce (there’s an informal equivalence result; if one of those is easy to deduce, all the others are).
So we need to figure out if we’re in the optimistic or the pessimistic scenario.
My understanding of the OP was that there is a robot [...]
That understanding is correct.
Then my question was: what if none of the variables, functions, etc. corresponds to “preferences”? What if “preferences” is a way that we try to interpret the robot, but not a natural subsystem or abstraction or function or anything else that would be useful for the robot’s programmer?
I agree that preferences is a way we try to interpret the robot (and how we humans try to interpret each other). The programmer themselves could label the variables; but its also possible that another labelling would be clearer or more useful for our purposes. It might be a “natural” abstraction, once we’ve put some effort into defining what preferences “naturally” are.
but “white box” is any source code that produces the same input-output behavior
What that section is saying is that there are multiple white boxes that produce the same black box behaviour (hence we cannot read the white box simply from the black box).
modularization is super helpful for simplifying things.
The best modularization for simplification will not likely correspond to the best modularization for distinguishing preferences from other parts of the agent’s algorithm (that’s the “Occam’s razor” result).
“within the limits of their intelligence” can mean anything, excuse any error, bias, and failure. Thus, they are not rational, and (form one perspective) very very far from it.