I think writing something like this is a bit like a rite of passage. So, welcome to LW :P
When we talk about someone’s values, we’re using something like Dan Dennett’s intentional stance. You might also enjoy this LW post about not applying the intentional stance.
Long story short, there is no “truly true” answer to what people want, and no “true boundary” between person and environment, but there are answers and boundaries that are good enough for what people usually mean.
I’m still reading Dan Dennett’s intentional stance now so I won’t address that right now, but in terms of /not/ applying the intentional stance, I think we can be considered different from the “blue minimizer” since the blue minimizer assumes it has no access to its source code—we do actually have access to our source code so can see what laws govern us. Since we “want” to do things, we should be able to figure out why we “want” anything or really, why we “do” anything. To be clear, are you saying that instead of the equations being X=”good points” and Y=”good points” and the law is “maximize good points” the law might just be DO X and Y? If so I still don’t think things like “survival” and “friendship” are terminal values or laws of the form “SURVIVE” and “MAKE FRIENDS”. When these two are in conflict we still are able to choose a course of action therefore there must be some lower level law that determines the thing we “want” to do (or more accurately, just do if you don’t want to assign intention to people).
I also want to address the point that you said there are answers and boundaries good enough for what people usually mean—I think what we should really be going for is “answers and boundaries good enough to get what we really /want/.” I think a common model of humans in this community is somewhat effective optimizers upon a set of terminal values, if that’s really true, in order to optimize our terminal value(s) we should be trying to know them, and as I said I think the current idea that we can have multiple changeable terminal values contradicts the definition of a terminal value.
I think writing something like this is a bit like a rite of passage. So, welcome to LW :P
When we talk about someone’s values, we’re using something like Dan Dennett’s intentional stance. You might also enjoy this LW post about not applying the intentional stance.
Long story short, there is no “truly true” answer to what people want, and no “true boundary” between person and environment, but there are answers and boundaries that are good enough for what people usually mean.
Thanks so much for replying!
I’m still reading Dan Dennett’s intentional stance now so I won’t address that right now, but in terms of /not/ applying the intentional stance, I think we can be considered different from the “blue minimizer” since the blue minimizer assumes it has no access to its source code—we do actually have access to our source code so can see what laws govern us. Since we “want” to do things, we should be able to figure out why we “want” anything or really, why we “do” anything. To be clear, are you saying that instead of the equations being X=”good points” and Y=”good points” and the law is “maximize good points” the law might just be DO X and Y? If so I still don’t think things like “survival” and “friendship” are terminal values or laws of the form “SURVIVE” and “MAKE FRIENDS”. When these two are in conflict we still are able to choose a course of action therefore there must be some lower level law that determines the thing we “want” to do (or more accurately, just do if you don’t want to assign intention to people).
I also want to address the point that you said there are answers and boundaries good enough for what people usually mean—I think what we should really be going for is “answers and boundaries good enough to get what we really /want/.” I think a common model of humans in this community is somewhat effective optimizers upon a set of terminal values, if that’s really true, in order to optimize our terminal value(s) we should be trying to know them, and as I said I think the current idea that we can have multiple changeable terminal values contradicts the definition of a terminal value.