We have only access to our current map to tell us about the territory, yes. But we have strong intuitions about how we would act if we could explicitly choose that our future map permanently diverge from our current map (which we currently see as the territory). If we (again by our current map) believe that this divergence would conform less to the territory (as opposed to a new map created by learning information), many of us would oppose that change even against pretty high stakes.
I mean, if Omega told me that I had to choose between
(A) my sister on Mars being well but cut off from all contact with me, or
(B) my sister being killed but a nonsentient chatbot impersonating her to me in happy weekly chats,
and that in either case my memory of this choice would be wiped when I made it, I would choose (A) without hesitation.
I understand that calling our current map “the territory” looks like a categorical error, but rejecting conchis’ point entirely is the wrong response. There’s a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.
Of course, because the immediate pain of the thought of choosing B would outweigh the longer-term lesser pain of the thought of losing contact with your sister.
This has nothing to do with whether the events actually occur, and everything to do with your mapping of the experience of the conditions, as you imagine them for purposes of making a decision.
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences
It so happens that there is some correspondence between this (real) process and the way we would prefer to think we establish and evaluate our preferences. Specifically, both models will generate similar results, most of the time. It’s just that the reasons we end up with for the responses are quite different.
There’s a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.
But calling that latter concept “territory” is still a category error, because what you are using to evaluate it is still your perception of how you would experience the change.
We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have “rational” preferences it is because they will ultimately lead to some desired emotion or sensation.
However, our brains are constructed in such a way so as to allow us to plausibly overlook and deny this fact, so that we can be honestly “sincere” in our altruism… specifically by claiming that our responses are “really” about things outside ourselves.
For example, your choice of “A” allows you to self-signal altruism, even if your sister would actually prefer death to being imprisoned on Mars for the rest of her life! Your choice isn’t about making her life better, it’s about you feeling better for the brief moment that you’re aware you did something.
(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences.
I affirm this, but it does not follow that:
This has nothing to do with whether the events actually occur...
Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.
Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.
And a thermostat’s map is also “entangled” with the territory, but as loqi pointed out, what it really prefers is only that its input sensor match its temperature setting!
I am not saying there are no isomorphisms between the shape of our preferences and the shape of reality, I am saying that assuming this isomorphism means the preferences are therefore “about” the territory is mind projection.
If you look at a thermostat, you can project that it was made by an optimizing process that “wanted” it to do certain things by responding to the territory, and that thus, the thermostat’s map is “about” the territory. And in the same way, you can look at a human and project that it was made by an optimizing process (evolution) that “wanted” it to do certain thing by responding to the territory.
However, the “aboutness” of the thermostat does not reside in the thermostat; it resides in the maker of the thermostat, if it can be said to exist at all! (In fact, this “aboutness” cannot exist, because it is not a material entity; it’s a mental entity—the idea of aboutness.)
So despite the existence of inputs and outputs, both the human and the thermostat do their “preference” calculations inside the closed box of their respective models of the world.
It just so happens that humans’ model of the world also includes a Mind Projection device, that causes humans to see intention and purpose everywhere they look. And when they look through this lens at themselves, they imagine that their preferences are about the territory… which then keeps them from noticing various kinds of erroneous reasoning and subgoal stomps.
For that matter, it keeps them from noticing things like the idea that if you practice being a pessimist, nothing good can last for you, because you’ve trained yourself to find bad things about anything. (And vice versa for optimists.)
Ostensibly, optimism and pessimism are “about” the outside world, but in fact, they’re simply mechanical, homeostatic processes very much like a thermostat.
I am not a solipsist nor do I believe people “create your own reality”, with respect to the actual territory. What I’m saying is that people are deluded about the degree of isomorphism between their preferences and reality, because they confuse the map with the territory. And even with maximal isomorphism between preference and reality, they are still living in the closed box of their model.
It is reasonable to assume that existence actually exists, but all we can actually reason about is our experience of it, “inside the box”.
That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.
Then, as I said, he cares about something closer to the reality.
The major point I’ve been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations. And not asking the question is a perfect example of the sort of decision error this can produce!
However, asking the question doesn’t magically make the preference about the territory either; in order to prefer the future include his sister’s best interests, he must first have an experience of the sister and a reason to wish well of her. But it’s still better than not asking, which is basically wireheading.
The irony I find in this discussion is that people seem to think I’m in favor of wireheading because I point out that we’re all doing it, all the time. When in fact, the usefulness of being aware that it’s all wireheading, is that it makes you better at noticing when you’re doing it less-usefully.
The fact that he hadn’t asked his sister, or about his sister’s actual well-being instantly jumped off the screen at me, because it was (to me) obvious wireheading.
So, you could say that I’m biased by my belief to notice wireheading more, but that’s an advantage for a rationalist, not a disadvantage.
The major point I’ve been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations.
Is human knowledge also not just in the map, but exclusively of the map? If not, what’s the difference?
Is human knowledge also not just in the map, but exclusively of the map? If not, what’s the difference?
Any knowledge about the actual territory can in principle be reduced to mechanical form without the presence of a human being in the system.
To put it another way, a preference is not a procedure, process, or product. The very use of the word “preference” is a mind projection—mechanical systems do not have “preferences”—they just have behavior.
The only reason we even think we have preferences in the first place (let alone that they’re about the territory!) is because we have inbuilt mind projection. The very idea of having preferences is hardwired into the model we use for thinking about other animals and people.
So, knowledge exists in the structure of map and is about the territory, while preference can’t be implemented in natural artifacts. Preference is a magical property of subjective experience, and it is over maps, or about subjective experience, but not, for example, about the brain. Saying that preference exists in the structure of map or that it is about the territory is a confusion, that you call “mind projection” Does that summarize your position? What are the specific errors in this account?
Preference is a magical property of subjective experience
No, “preference” is an illusory magical property projected by brains onto reality, which contains only behaviors.
Our brains infer “preferences” as a way of modeling expected behaviors of other agents: humans, animals, and anything else we perceive as having agency (e.g. gods, spirits, monsters). When a thing has a behavior, our brains conclude that the thing “prefers” to have either the behavior or the outcome of the behavior, in a particular circumstance. In other words, “preference” is a label attached to a clump of behavior-tendency observations and predictions in the brain—not a statement about the nature of the thing being observed.
Thus, presuming that these “preferences” actually exist in the territory is supernaturalism, i.e., acting as though basic mental entities exist.
My original point had more to do with the types of delusion that occur when we reason on the basis of preferences actually existing, rather than the idea simply being a projection of our own minds. However, the above will do for a start, as I believe my other conclusions can be easily reached from this point.
Thus, presuming that these “preferences” actually exist in the territory is supernaturalism, i.e., acting as though basic mental entities exist.
Do you think someone is advocating the position that goodness of properties of the territory is an inherent property of territory (that sounds like a kind of moral realism)? This looks like the lack of distinction between 1-place and 2-place words. You could analogize preference (and knowledge) as a relation between the mind and the (possible states of the) territory, that is neither a property of the mind alone, nor of the territory alone, but a property of them being involved in a certain interaction.
Do you think someone is advocating the position that goodness of properties of the territory is an inherent property of territory?
No, I assume that everybody who’s been seriously participating has at least got that part straight.
This looks like the lack of distinction between 1-place and 2-place words. You could analogize preference (and knowledge) as a relation between the mind and the (possible states of the) territory, that is neither a property of the mind alone, nor of the territory alone, but a property of them being involved in a certain interaction.
Now you’re getting close to what I’m saying, but on the wrong logical level. What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
If I assert that my preferences are “about” the real world, I am making a category error because my preferences are relationships between portions of my map, some of which I have labeled as representing the territory.
The fact that there is a limited isomorphism between that portion of my map, and the actual territory, does not make my preferences “about” the territory, unless you represent that idea in another map.
That is, I can represent the idea that “your” preferences are about the territory in my map… in that I can posit a relationship between the part of my map referring to “you”, and the part of my map referring to “the territory”. But that “aboutness” relationship is only contained in my map; it doesn’t exist in reality either.
That’s why it’s always a mind projection fallacy to assert that preferences are “about” territory: one cannot assert it of one’s own preferences, because that implies the territory is inside the map. And if one asserts it of another person’s preferences, then that one is projecting their own map onto the territory.
I initially only picked on the specific case of self-applied projection, because understanding that case can be very practically useful for mind hacking. In particular, it helps to dissolve certain irrational fears that changing one’s preferences will necessarily result in undesirable futures. (That is, these fears are worrying that the gnomes and fairies will be destroyed by the truth, when in fact they were never there to start with.)
Now you’re getting close to what I’m saying, but on the wrong logical level. What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
How’s that? You can write Newton’s law of universal gravitation describing the orbit of the Earth around the Sun on a piece of paper located on the surface of a table standing in a house on the surface of the Earth. Where does this analogy break from your point of view?
″...but, you can’t fold up the territory and put it in your glove compartment”
How’s that? You can write Newton’s law of universal gravitation describing the orbit of the Earth around the Sun on a piece of paper located on the surface of a table standing in a house on the surface of the Earth. Where does this analogy break from your point of view?
The “aboutness” relationship between the written version of Newton’s law and the actual instances of it is something that lives in the map in your head.
IOW, the aboutness is not on the piece of paper. Nor does it exist in some supernatural link between the piece of paper and the objects acting in accordance with the expressed law.
The “aboutness” relationship between the written version of Newton’s law and the actual instances of it is something that lives in the map in your head.
Your head describes how your head rotates around the Sun.
What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
Your head describes how your head rotates around the Sun.
No, your head is rotating around the Sun, and it contains a description relating the ideas of “head” and “Sun”. You are confusing head 1 (the real head) with head 2 (the “head” pictured inside head 1), as well as Sun 1 (the real Sun) and Sun 2 (the “Sun” pictured inside head 1).
No, I’m not confusing them. They are different things. Yet the model simulates the real thing, which means the following (instead of magical aboutness): By examining the model it’s possible to discover new properties of its real counterpart, that were not apparent when the model was being constructed, and that can’t be observed directly (or it’s just harder to do), yet can be computed from the model.
By examining the model it’s possible to discover new properties of its real counterpart, that were not apparent when the model was being constructed, and that can’t be observed directly (or it’s just harder to do), yet can be computed from the model.
Indeed. Although more precisely, examining the model merely suggests or predicts these “new” (rather, previously undiscovered, unnoticed, or unobservable) properties.
That is what I mean by isomorphism between model and territory. The common usage of “about”, however, projects an intention onto this isomorphism—a link that can only exist in the mind of the observer, not the similarity of shapes between one physical process and another.
Since agent’s possible actions are one of the things in the territory captured by the model, it’s possible to use the model to select an action leading to a preferable outcome, and to perform thus selected action, determining the territory to conform with the plan. The correspondence between the preferred state of the world in the mind and the real world is ensured by this mechanism for turning plans into actuality. Pathologies aside, or course.
I don’t disagree with anything you’ve just said, but it does nothing to support the idea of an isomorphism inherently meaning that one thing is “about” another.
If I come across a near-spherical rock that resembles the moon, does this make the rock “about” the moon? If I find another rock that is shaped the same, does that mean it is about the moon? The first rock? Something else entirely?
The :”aboutness” of a thing can’t be in the thing, and that applies equally to thermostats and humans.
The (external) aboutness of a thermostat’s actions don’t reside in the thermostat’s map, and humans are deluded when they project that the (external) aboutness of their own actions actually resides within the same map they’re using to decide those actions. It is merely a sometimes-useful (but often harmful) fiction.
Taboo “aboutness” already. However unfathomably confused the philosophic and folk usage of this word is doesn’t interest me much. What I mean by this word I described in thesecomments, and this usage seems reasonably close to the usual one, which justifies highjacking the word for the semi-technical meaning rather than inventing a new one. This is also the way meaning/aboutness is developed in formal theories of semantics.
What I mean by this word I described in these comments, and this usage seems reasonably close to the usual one, which justifies highjacking the word for the semi-technical meaning rather than inventing a new one.
So, you are saying that you have no argument with my position, because you have not been using either “about” or “preference” with their common usage?
If that is the case, why couldn’t you simply say that, instead of continued argument and posturing about your narrower definition of the words? ISTM you could have pointed that out days ago and saved us all a lot of time.
This is also not the first time where I have been reducing the common usage of a word (e.g. “should”) and then had you argue that I was wrong, based on your own personal meaning of the word.
Since I have no way of knowing in advance all of the words you have chosen to redefine in your specialized vocabulary, would it be too much to ask if you point out which words you are treating as specialized when you argue that my objection to (or reduction of) the common meaning of the word is incorrect, because it does not apply to your already-reduced personal version of the word?
Then, I could simply nod, and perhaps ask for your reduction in the case where I do not have a good one already, and we would not need to have an extended argument where we are using utterly incompatible definitions for such words as “about”, “preference”, and “should.”
Actually, the point is that most of the other usages of these words are meaningless confusion, and the argument is that this particular semi-technical sense is what the word actually means, when you get the nonsense out of it. It’s not how it’s used, but it’s the only meaningful thing that fits the idea.
Since you don’t just describe the usage of the word, but argue for the confusion behind it, we have a disagreement. Presenting a clear definition is the easy part. Showing that ten volumes of the Encyclopedia of Astrology is utter nonsense is harder, and arguing with each point made in its chapters is a wrong approach. It should be debunked on meta-level, with an argument that doesn’t require the object-level details, but that requires the understanding of the general shape of the confusion.
Actually, the point is that most of the other usages of these words are meaningless confusion
Yes, but ones which most people do not understand to be confusion, and the only reason I started this discussion in the first place was because I was trying to clear up one point in that confusion.
Since you don’t just describe the usage of the word, but argue for the confusion behind it, we have a disagreement
I am arguing against the confusion, not for the confusion. So, as far as I can tell, there should be no disagreement.
In practice, however, you have been making arguments that sound like you are still confusing map and territory in your own thinking, despite seeming to agree with my reasoning on the surface. You are consistently treating “about” as a 2-way relationship, when to be minimally cohesive, it requires 3 entities: the 2 entities that have an isomorphism, and the third entity whose map ascribes some significance to this isomorphism.
You’ve consistently omitted the presence of the third entity, making it sound as though you do not believe it to be required, and thereby committing the mind projection fallacy.
Also, please stop using “mind projection fallacy”, you are misapplying the term.
How is that, precisely? My understanding is that it is mind projection when you mistakenly believe a property of an object to be intrinsic, rather than merely attributed.
I am pointing out that “aboutness” (whose definition I never agreed on, because you handwaved away the subject by saying it is I who should define it), is not an intrinsic property of isomorphic relationships.
Rather, it is a property being attributed to that relationship, a label that is being expressed in some map.
(Prediction: your next reply will still not address this point, nor clarify your definition of “about”, but simply handwave again why it is that I am doing something else wrong. Anything but actually admitting that you have been using a mind-projecting definition of “about” since the beginning of this conversation, right up until the point where you ducked the question by asking me to taboo it, rather than defend the imprecise definition you’ve been using, or clear up any of the other handwaving you’ve been using to beg the question. I base this prediction on the rapid increase in non-responsive replies that, instead of defending the weak points of your position, represent attempts to find new ways to attack me and/or my position. A rationalist should be able to attack their own weak points, let alone being able to defend them, without resorting to distraction, subject-changing, and playing to the gallery.)
There are natural categories, like “tigers”, that don’t require much of a mind to define. It’s not mind projection fallacy to say that something is a tiger.
P.S. I’m correcting self-censoring threshold, so expect silence where before I’d say something for the fifth time.
There are natural categories, like “tigers”, that don’t require much of a mind to define. It’s not mind projection fallacy to say that something is a tiger.
Is that actually an argument? ’cause it sounds like a random sentence injected into the conversation, perhaps as an invitation for me to waste time tearing “natural categories” to shreds, while leaving you still able to deny that your statement actually relates in any substantial way to your point… thereby once again relieving you of any need to actually defend your position.
That is, are you actually claiming aboutness to be a natural category? Or just trying to get me to treat your argument as if you were doing so?
P.S. I’m correcting self-censoring threshold, so expect silence
I already did and do expect it; see my “prediction” in the parent to your comment. I predicted that you would remain silent on any substantive issues, and avoid admitting anywhere where you were mistaken or incorrect. (I notice, for example, that you went back and deleted the comment where you said I was using “mind projection fallacy” incorrectly, rather than admit your attack was in error.)
And, as predicted, you avoided directly addressing the actual point of contention, instead choosing to enter a new piece of handwaving to imply that I am doing something else wrong.
That is, you appear to now be implying that I am using an overbroad definition of the MPF, without actually saying that I am doing it, or that your statement is in any way connected to your own position. This is a nice double bind, since either way I interpret the statement, you can retreat… and throw in more irrelevancies.
I don’t know if “troll” is a natural category, but you’re sure getting close to where I’d mind-project your behavior as matching that of one. ;-)
For the record, I thought it obvious that my argument above implied that I claim aboutness to be a natural category (although I’m not perfectly sure it’s a sound argument). I deleted my comment because I deemed it low-quality, before knowing you responded to it.
I claim aboutness to be a natural category (although I’m not perfectly sure it’s a sound argument)
It’s not.
First, the only way it can be one is if “natural category” has the reductionist meaning of “a category based on distinctions that humans are biased towards using as discriminators”, rather than “a category that ‘naturally’ exists in the territory”. (Categories are abstractions, not physical entities, after all.)
And second, even if you do use the reductionist meaning of “natural category”, then this does not in any way undermine the conclusion that “aboutness” is mind projection when you omit the entity mapping that aboutness from the description.
In other words, this argument appears to result in only one of two possibilities: either “aboutness” is not a natural category per the reductionist definition, and thus inherently a mind projection when the attribution source is omitted, or “aboutness” is a natural category per the reductionist definition… in which case the attribution source has to be a human brain (i.e., in another map).
Finally, if we entirely reject the reductionist definition of “natural category”, then “natural category” is itself an instance of the mind projection fallacy, since the description omits any definition of for whom the category is “natural”.
In short, QED: the argument is not sound. (I just didn’t want to bother typing all this if you were going to retreat to a claim this was never your argument.)
To the (unknowable*) extent that the portion of my map labelled “territory” is an accurate reflection of the relevant portion of the territory, do I get to say that my preferences are “about” the territory (implicitly including disclaimers like “as mediated by the map”)?
* due at the very least to Matrix/simulation scenarios
To the (unknowable*) extent that the portion of my map labelled “territory” is an accurate reflection of the relevant portion of the territory, do I get to say that my preferences are “about” the territory (implicitly including disclaimers like “as mediated by the map”)?
You can say it all you want, it just won’t make it true. ;-) Your preference is “about” your experience, just as the thermostat’s heating and cooling preferences are “about” the temperature of its sensor, relative to its setting.
For there to be an “about”, there has to be another observer, projecting a relationship of intention onto the two things. It’s a self-applied mind projection—a “strange loop” in your model—to assert that you can make such statements about your own preferences, like a drawing of Escher wherein Escher is pictured, making the drawing. The whole thing only makes sense within the surface of the paper.
(Heck, it’s probably a similar strange loop to make statements about one’s self in general, but this probably doesn’t lead to the same kind of confusion and behavioral problems that result from making assertions about one’s preferences.… No, wait, actually, yes it does! Self-applied nominalizations, like “I’m bad at math” are an excellent example. Huh. I keep learning interesting new things in this discussion.)
I feel your frustration, but throwing the word “magical” in there is just picking a fight, IMO. Anyway, I too would like to see P.J. Eby summarize his position in this format.
I have a certain technical notion of magic in mind. This particular comment wasn’t about frustration (some of the others were), I’m trying out something different of which I might write a post later.
(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)
Be charitable in your interpretation, and remember the Least Convenient Possible World principle. I was presuming that the setup was such that being alive on Mars wouldn’t be a ‘fate worse than death’ for her; if it were, I’d choose differently. If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences.
I would say rather that my reaction is my evaluation of an imagined future world. The essence of many decision algorithms is to model possible futures and compare them to some criteria. In this case, I have complicated unconscious affective criteria for imagined futures (which dovetail well with my affective criteria for states of affairs I directly experience), and my affective reaction generally determines my actions.
We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have “rational” preferences it is because they will ultimately lead to some desired emotion or sensation.
To the extent this is true (as in the sense of my previous sentence), it is a tautology. I understand what you’re arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals. I am not asserting this; I believe that our affective algorithms do often operate under more selfish and basic criteria, and that they fixate on the most salient possibilities instead of weighing probabilities properly, among other things.
However, these affective algorithms do appear to respond more strongly to certain facets of “how I expect the world to be” than to facets of “how I expect to think the world is” when the two conflict (with an added penalty for the expectation of being deceived), and I don’t find that problematic on any level.
If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.
As I said, it’s still going to be about your experience during the moments until your memory is erased.
I understand what you’re arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals.
I took that as a given, actually. ;-) What I’m really arguing against is the naive self-applied mind projection fallacy that causes people to see themselves as decision-making agents—i.e., beings with “souls”, if you will. Asserting that your preferences are “about” the territory is the same sort of error as saying that the thermostat “wants” it to be a certain temperature. The “wanting” is not in the thermostat, it’s in the thermostat’s maker.
Of course, it makes for convenient language to say it wants, but we should not confuse this with thinking the thermostat can really “want” anything but for its input and setting to match. And the same goes for humans.
(This is not a mere fine point of tautological philosophy; human preferences in general suffer from high degrees of subgoal stomp, chaotic loops, and other undesirable consequences arising as a direct result of this erroneous projection. Understanding the actual nature of preferences makes it easier to dissolve these confusions.)
What features of that comment made it communicate something new to you? What was it that got communicated?
The comment restated a claim that a certain relationship is desirable as a claim that given that it’s desirable, there is a process that establishes it to be true. It’s interesting how this restatement could pierce inferential distance: is preference less trustworthy than a fact, and so demonstrating the conversion of preference into a fact strengthens the case?
Given the length of the thread I branched from, it looks like you and P.J. Eby ended up talking past each other to some extent, and I think that you both failed to distinguish explicitly between the current map (which is what you calculate the territory to be) and a hypothetical future map.
P.J. Eby was (correctly) insisting that your utility function is only in contact with your current map, not the territory directly. You were (correctly) insisting that your utility function cares about (what it calculates to be) the future territory, and not just the future map.
Utility function is no more “in contact” with your current map than the actual truth of 2+2=4 is in contact with display of a calculator that displays the statement. Utility function may care about past territory (and even counterfactual territory) as well as future territory, with map being its part. Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn’t be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.
Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we’re talking about practical purposes, this is a relevant point.
Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I’m generally more repulsed by the ones in which my map says it’s fine where it’s not than by the opposite.
any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
This something of a nitpick, but this isn’t strictly true. If others are trying to calculate your utility function (in order to help you), this will depend on their maps rather than yours (though probably including their map of your map). The difference becomes important if their maps are more accurate than yours in some respect (or if they can affect how accurate your map is).
For example, if you know that I value not being deceived (and not merely the subjective experience of not being deceived), and you care about my welfare, then I think that you should not deceive me, even if you know that I might perceive my welfare to be higher if you did.
If you install an alarm system that uses a video camera to recognize movement and calls the police if it’s armed, you are delegating some of the map-making and decision-making to the alarm system. You are neither aware of the exact nature of possible intruders, nor making a decision regarding calling the police before any intrusion actually occurs. The system decides what to do by itself, according to the aspect of your values it implements. You map is not involved.
Yes, but your decision to install it (as well as your decision to arm it) comes from your map. You would not install it if you thought you had virtually no chance of being burglarized, or if you thought that it would have a false alarm every five minutes when the train went past.
We can make choices that cause other (human, mechanical, etc) agents to act in particular ways, as one of the manners in which we affect possible futures. But these sorts of choices are evaluated by us in the same way as others.
I fear we’ve resorted to arguing about the semantics of “map” versus “territory”, as I don’t see a scenario where we’d predict or decide differently from each other on account of this disagreement. As such, I’m willing to drop it for now unless you see such a scenario.
(My disagreement with Mr. Eby, on the other hand, appears to be more substantive.)
The [alarm] system decides what to do by itself, according to the aspect of your values it implements.
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
My argument is simply that humans are no different than this hypothetical alarm system; the things we call preferences are no different than variables in the alarm system’s controller—an implementation of values that are not our own.
If there are any “preferences about reality” in the system, they belong to the maker of the alarm system, as it is merely an implementation of the maker’s values.
By analogy, if our preferences are the implementation of any values, they are the “values” of natural selection, not our own.
If now you say that natural selection doesn’t have any preferences or values, then we are left with no preferences anywhere—merely isomorphism between control systems and their environments. Saying this isomorphism is “about” something is saying that a mental entity (the “about” relationship) exists in the real world, i.e., supernaturalism.
In short, what I’m saying is that anybody who argues human preferences are “about” reality is anthropomorphizing the alarm system.
However, if you say that the alarm system does have preferences by some reductionistic definition of “preference”, and you assert that human preference is exactly the same, then we are still left to determine the manner in which these preferences are “about” reality.
If nobody made the alarm system, but it just happened to be formed by a spontaneous jumbling of parts, can it still be said to have preferences? Are its “preferences” still “about” reality in that case?
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
So an alarm system has preferences? That is not most people’s understanding of the word “preference”, which requires a degree of agency that most rationalists wouldn’t attribute to an alarm system.
Nonetheless, let us say an alarm system has preferences. You didn’t answer any of my follow-on questions for that case.
As for explaining away the rainbow, you seem to have me confused with an anti-reductionist. See Explaining vs. Explaining Away, in particular:
If you don’t distinguish between the multi-level map and the mono-level territory, then when someone tries to explain to you that the rainbow is not a fundamental thing in physics, acceptance of this will feel like erasing rainbows from your multi-level map, which feels like erasing rainbows from the world.
At this point, I am attempting to show that the very concept of a “preference” existing in the first place is something projected onto the world by an inbuilt bias in human perception. Reality does not have preferences, it has behaviors.
This is not erasing the rainbow from the world, it’s attempting to erase the projection of a mind-modeling variable (“preference”) from the world, in much the same way as Eliezer broke down the idea of “possible” actions in one of his series.
So, if you are claiming that preference actually exists, please give your definition of a preference, such that alarm systems and humans both have them.
Note that probability is also in the mind, but yet your see all the facts through it, and you can’t ever revoke it, each mind is locked in its subjectively objective character. What do you think of that?
I think that those things have already been very well explained by Eliezer—so much so that I assumed that you (and the others participating in this discussion) would have already internalized them to the same degree as I have, such that asserting “preferences” to be “about” things would be a blatantly obvious instance of the mind projection fallacy.
That’s why, early on, I tended to just speak as though it was bloody obvious, and why I haven’t been painstakingly breaking it all out piece by piece, and why I’ve been baffled by the argument, confusion, and downvoting from people for whom this sort of basic reductionism ought to be a bloody simple matter.
Oh, and finally, I think that you still haven’t given your definition of “preference”, such that humans and alarm systems both have it, so that we can then discuss how it can then be “about” something… and whether that “aboutness” exists in the thing having the preference, or merely in your mental model of the thing.
I think that those things have already been very well explained by Eliezer
That in reply to a comment full of links to Eliezer’s articles. You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument. I wasn’t even talking about preference.
I know. That’s the problem. See this comment and this one, where I asked for your definition of preference, which you still haven’t given.
You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument.
That’s because you also “didn’t answer my comment, but wrote some text that doesn’t help me in our argument.” I was attempting to redirect you to answering the question which you’ve now ducked twice in a row.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation. Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation. I’m currently trying to understand your position in terms of my position, not to explain to you my position.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation.
We reached a point in the discussion where it appears the only way we could disagree is if we had a different definition of “preference”. Since I believe I’ve made my definition quite clear, I wanted to know what yours is.
It might not help you, but it would certainly help me to understand your position, if you are not using the common definition of preference.
Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation.
I asked you first, and you responded with (AFAICT) a non-answer. You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
So, I actually answered your question (i.e. “what do you think?”), even though you still haven’t answered mine.
You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
I chose the articles in the comment above because they were in surface-match with what you are talking about, as a potential point on establishing understanding. I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
And I answered that I agree with them, and that I considered it foundational material to what I’m talking about.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
Indeed, which is why I’d now like to have the answer to my question, please. What definition of “preferences” are you using, such that an alarm system, thermostat, and human all have them? (Since this is not the common, non-metaphorical usage of “preference”.)
Preference is order on the lotteries of possible worlds (ideally established by expected utility), usually with agent a part of the world. Computations about this structure are normally performed by a mind inside the mind. The agent tries to find actions that determine the world to be as high as possible on the preference order, given the knowledge about it. Now, does it really help?
Yes, as it makes clear that what you’re talking about is a useful reduction of “preference”, unrelated to the common, “felt” meaning of “preference”. That alleviates the need to further discuss that portion of the reduction.
The next step of reduction would be to unpack your phrase “determine the world”… because that’s where you’re begging the question that the agent is determining the world, rather than determining the thing it models as “the world”.
So far, I have seen no-one explain how an agent can go beyond its own model of the world, except as perceived by another agent modeling the relationship between that agent and the world. It is simply repeatedly asserted (as you have effectively just done) as an obvious fact.
But if it is an obvious fact, it should be reducible, as “preference” is reducible, should it not?
I’d been following this topic and getting frustrated with my inability to put my opinion on the whole preferences-about-the-territory thing into words, and I thought that orthonomal’s comment accomplished it very nicely. I don’t think I understand your other question.
We have only access to our current map to tell us about the territory, yes. But we have strong intuitions about how we would act if we could explicitly choose that our future map permanently diverge from our current map (which we currently see as the territory). If we (again by our current map) believe that this divergence would conform less to the territory (as opposed to a new map created by learning information), many of us would oppose that change even against pretty high stakes.
I mean, if Omega told me that I had to choose between
(A) my sister on Mars being well but cut off from all contact with me, or
(B) my sister being killed but a nonsentient chatbot impersonating her to me in happy weekly chats,
and that in either case my memory of this choice would be wiped when I made it, I would choose (A) without hesitation.
I understand that calling our current map “the territory” looks like a categorical error, but rejecting conchis’ point entirely is the wrong response. There’s a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.
Of course, because the immediate pain of the thought of choosing B would outweigh the longer-term lesser pain of the thought of losing contact with your sister.
This has nothing to do with whether the events actually occur, and everything to do with your mapping of the experience of the conditions, as you imagine them for purposes of making a decision.
That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality—your present-tense experience in response to a constructed fiction made of previous experiences
It so happens that there is some correspondence between this (real) process and the way we would prefer to think we establish and evaluate our preferences. Specifically, both models will generate similar results, most of the time. It’s just that the reasons we end up with for the responses are quite different.
But calling that latter concept “territory” is still a category error, because what you are using to evaluate it is still your perception of how you would experience the change.
We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have “rational” preferences it is because they will ultimately lead to some desired emotion or sensation.
However, our brains are constructed in such a way so as to allow us to plausibly overlook and deny this fact, so that we can be honestly “sincere” in our altruism… specifically by claiming that our responses are “really” about things outside ourselves.
For example, your choice of “A” allows you to self-signal altruism, even if your sister would actually prefer death to being imprisoned on Mars for the rest of her life! Your choice isn’t about making her life better, it’s about you feeling better for the brief moment that you’re aware you did something.
(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you’d have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)
I affirm this, but it does not follow that:
Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.
And a thermostat’s map is also “entangled” with the territory, but as loqi pointed out, what it really prefers is only that its input sensor match its temperature setting!
I am not saying there are no isomorphisms between the shape of our preferences and the shape of reality, I am saying that assuming this isomorphism means the preferences are therefore “about” the territory is mind projection.
If you look at a thermostat, you can project that it was made by an optimizing process that “wanted” it to do certain things by responding to the territory, and that thus, the thermostat’s map is “about” the territory. And in the same way, you can look at a human and project that it was made by an optimizing process (evolution) that “wanted” it to do certain thing by responding to the territory.
However, the “aboutness” of the thermostat does not reside in the thermostat; it resides in the maker of the thermostat, if it can be said to exist at all! (In fact, this “aboutness” cannot exist, because it is not a material entity; it’s a mental entity—the idea of aboutness.)
So despite the existence of inputs and outputs, both the human and the thermostat do their “preference” calculations inside the closed box of their respective models of the world.
It just so happens that humans’ model of the world also includes a Mind Projection device, that causes humans to see intention and purpose everywhere they look. And when they look through this lens at themselves, they imagine that their preferences are about the territory… which then keeps them from noticing various kinds of erroneous reasoning and subgoal stomps.
For that matter, it keeps them from noticing things like the idea that if you practice being a pessimist, nothing good can last for you, because you’ve trained yourself to find bad things about anything. (And vice versa for optimists.)
Ostensibly, optimism and pessimism are “about” the outside world, but in fact, they’re simply mechanical, homeostatic processes very much like a thermostat.
I am not a solipsist nor do I believe people “create your own reality”, with respect to the actual territory. What I’m saying is that people are deluded about the degree of isomorphism between their preferences and reality, because they confuse the map with the territory. And even with maximal isomorphism between preference and reality, they are still living in the closed box of their model.
It is reasonable to assume that existence actually exists, but all we can actually reason about is our experience of it, “inside the box”.
And what if he did ask?
Then, as I said, he cares about something closer to the reality.
The major point I’ve been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations. And not asking the question is a perfect example of the sort of decision error this can produce!
However, asking the question doesn’t magically make the preference about the territory either; in order to prefer the future include his sister’s best interests, he must first have an experience of the sister and a reason to wish well of her. But it’s still better than not asking, which is basically wireheading.
The irony I find in this discussion is that people seem to think I’m in favor of wireheading because I point out that we’re all doing it, all the time. When in fact, the usefulness of being aware that it’s all wireheading, is that it makes you better at noticing when you’re doing it less-usefully.
The fact that he hadn’t asked his sister, or about his sister’s actual well-being instantly jumped off the screen at me, because it was (to me) obvious wireheading.
So, you could say that I’m biased by my belief to notice wireheading more, but that’s an advantage for a rationalist, not a disadvantage.
Is human knowledge also not just in the map, but exclusively of the map? If not, what’s the difference?
Any knowledge about the actual territory can in principle be reduced to mechanical form without the presence of a human being in the system.
To put it another way, a preference is not a procedure, process, or product. The very use of the word “preference” is a mind projection—mechanical systems do not have “preferences”—they just have behavior.
The only reason we even think we have preferences in the first place (let alone that they’re about the territory!) is because we have inbuilt mind projection. The very idea of having preferences is hardwired into the model we use for thinking about other animals and people.
You never answered my question.
You said, “if not, what’s the difference”, and I gave you the difference. i..e, we can have “knowledge” of the territory.
So, knowledge exists in the structure of map and is about the territory, while preference can’t be implemented in natural artifacts. Preference is a magical property of subjective experience, and it is over maps, or about subjective experience, but not, for example, about the brain. Saying that preference exists in the structure of map or that it is about the territory is a confusion, that you call “mind projection” Does that summarize your position? What are the specific errors in this account?
No, “preference” is an illusory magical property projected by brains onto reality, which contains only behaviors.
Our brains infer “preferences” as a way of modeling expected behaviors of other agents: humans, animals, and anything else we perceive as having agency (e.g. gods, spirits, monsters). When a thing has a behavior, our brains conclude that the thing “prefers” to have either the behavior or the outcome of the behavior, in a particular circumstance. In other words, “preference” is a label attached to a clump of behavior-tendency observations and predictions in the brain—not a statement about the nature of the thing being observed.
Thus, presuming that these “preferences” actually exist in the territory is supernaturalism, i.e., acting as though basic mental entities exist.
My original point had more to do with the types of delusion that occur when we reason on the basis of preferences actually existing, rather than the idea simply being a projection of our own minds. However, the above will do for a start, as I believe my other conclusions can be easily reached from this point.
Do you think someone is advocating the position that goodness of properties of the territory is an inherent property of territory (that sounds like a kind of moral realism)? This looks like the lack of distinction between 1-place and 2-place words. You could analogize preference (and knowledge) as a relation between the mind and the (possible states of the) territory, that is neither a property of the mind alone, nor of the territory alone, but a property of them being involved in a certain interaction.
No, I assume that everybody who’s been seriously participating has at least got that part straight.
Now you’re getting close to what I’m saying, but on the wrong logical level. What I’m saying is that the logical error is that you can’t express a 2-place relationship between a map, and the territory covered by that map, within that same map, as that amounts to claiming the territory is embedded within that map.
If I assert that my preferences are “about” the real world, I am making a category error because my preferences are relationships between portions of my map, some of which I have labeled as representing the territory.
The fact that there is a limited isomorphism between that portion of my map, and the actual territory, does not make my preferences “about” the territory, unless you represent that idea in another map.
That is, I can represent the idea that “your” preferences are about the territory in my map… in that I can posit a relationship between the part of my map referring to “you”, and the part of my map referring to “the territory”. But that “aboutness” relationship is only contained in my map; it doesn’t exist in reality either.
That’s why it’s always a mind projection fallacy to assert that preferences are “about” territory: one cannot assert it of one’s own preferences, because that implies the territory is inside the map. And if one asserts it of another person’s preferences, then that one is projecting their own map onto the territory.
I initially only picked on the specific case of self-applied projection, because understanding that case can be very practically useful for mind hacking. In particular, it helps to dissolve certain irrational fears that changing one’s preferences will necessarily result in undesirable futures. (That is, these fears are worrying that the gnomes and fairies will be destroyed by the truth, when in fact they were never there to start with.)
How’s that? You can write Newton’s law of universal gravitation describing the orbit of the Earth around the Sun on a piece of paper located on the surface of a table standing in a house on the surface of the Earth. Where does this analogy break from your point of view?
″...but, you can’t fold up the territory and put it in your glove compartment”
The “aboutness” relationship between the written version of Newton’s law and the actual instances of it is something that lives in the map in your head.
IOW, the aboutness is not on the piece of paper. Nor does it exist in some supernatural link between the piece of paper and the objects acting in accordance with the expressed law.
Located on the planet Earth.
And this helps your position how?
Your head describes how your head rotates around the Sun.
No, your head is rotating around the Sun, and it contains a description relating the ideas of “head” and “Sun”. You are confusing head 1 (the real head) with head 2 (the “head” pictured inside head 1), as well as Sun 1 (the real Sun) and Sun 2 (the “Sun” pictured inside head 1).
No, I’m not confusing them. They are different things. Yet the model simulates the real thing, which means the following (instead of magical aboutness): By examining the model it’s possible to discover new properties of its real counterpart, that were not apparent when the model was being constructed, and that can’t be observed directly (or it’s just harder to do), yet can be computed from the model.
Indeed. Although more precisely, examining the model merely suggests or predicts these “new” (rather, previously undiscovered, unnoticed, or unobservable) properties.
That is what I mean by isomorphism between model and territory. The common usage of “about”, however, projects an intention onto this isomorphism—a link that can only exist in the mind of the observer, not the similarity of shapes between one physical process and another.
Since agent’s possible actions are one of the things in the territory captured by the model, it’s possible to use the model to select an action leading to a preferable outcome, and to perform thus selected action, determining the territory to conform with the plan. The correspondence between the preferred state of the world in the mind and the real world is ensured by this mechanism for turning plans into actuality. Pathologies aside, or course.
I don’t disagree with anything you’ve just said, but it does nothing to support the idea of an isomorphism inherently meaning that one thing is “about” another.
If I come across a near-spherical rock that resembles the moon, does this make the rock “about” the moon? If I find another rock that is shaped the same, does that mean it is about the moon? The first rock? Something else entirely?
The :”aboutness” of a thing can’t be in the thing, and that applies equally to thermostats and humans.
The (external) aboutness of a thermostat’s actions don’t reside in the thermostat’s map, and humans are deluded when they project that the (external) aboutness of their own actions actually resides within the same map they’re using to decide those actions. It is merely a sometimes-useful (but often harmful) fiction.
Taboo “aboutness” already. However unfathomably confused the philosophic and folk usage of this word is doesn’t interest me much. What I mean by this word I described in these comments, and this usage seems reasonably close to the usual one, which justifies highjacking the word for the semi-technical meaning rather than inventing a new one. This is also the way meaning/aboutness is developed in formal theories of semantics.
So, you are saying that you have no argument with my position, because you have not been using either “about” or “preference” with their common usage?
If that is the case, why couldn’t you simply say that, instead of continued argument and posturing about your narrower definition of the words? ISTM you could have pointed that out days ago and saved us all a lot of time.
This is also not the first time where I have been reducing the common usage of a word (e.g. “should”) and then had you argue that I was wrong, based on your own personal meaning of the word.
Since I have no way of knowing in advance all of the words you have chosen to redefine in your specialized vocabulary, would it be too much to ask if you point out which words you are treating as specialized when you argue that my objection to (or reduction of) the common meaning of the word is incorrect, because it does not apply to your already-reduced personal version of the word?
Then, I could simply nod, and perhaps ask for your reduction in the case where I do not have a good one already, and we would not need to have an extended argument where we are using utterly incompatible definitions for such words as “about”, “preference”, and “should.”
Actually, the point is that most of the other usages of these words are meaningless confusion, and the argument is that this particular semi-technical sense is what the word actually means, when you get the nonsense out of it. It’s not how it’s used, but it’s the only meaningful thing that fits the idea.
Since you don’t just describe the usage of the word, but argue for the confusion behind it, we have a disagreement. Presenting a clear definition is the easy part. Showing that ten volumes of the Encyclopedia of Astrology is utter nonsense is harder, and arguing with each point made in its chapters is a wrong approach. It should be debunked on meta-level, with an argument that doesn’t require the object-level details, but that requires the understanding of the general shape of the confusion.
Yes, but ones which most people do not understand to be confusion, and the only reason I started this discussion in the first place was because I was trying to clear up one point in that confusion.
I am arguing against the confusion, not for the confusion. So, as far as I can tell, there should be no disagreement.
In practice, however, you have been making arguments that sound like you are still confusing map and territory in your own thinking, despite seeming to agree with my reasoning on the surface. You are consistently treating “about” as a 2-way relationship, when to be minimally cohesive, it requires 3 entities: the 2 entities that have an isomorphism, and the third entity whose map ascribes some significance to this isomorphism.
You’ve consistently omitted the presence of the third entity, making it sound as though you do not believe it to be required, and thereby committing the mind projection fallacy.
So you are saying that my definition with which you’ve just agreed is unreasonable. Pick something tangible.
(Also, please stop using “mind projection fallacy”, you are misapplying the term.)
How is that, precisely? My understanding is that it is mind projection when you mistakenly believe a property of an object to be intrinsic, rather than merely attributed.
I am pointing out that “aboutness” (whose definition I never agreed on, because you handwaved away the subject by saying it is I who should define it), is not an intrinsic property of isomorphic relationships.
Rather, it is a property being attributed to that relationship, a label that is being expressed in some map.
That sounds like a textbook case of the mind projection fallacy, i.e. “the error of projecting your own mind’s properties into the external world.”
(Prediction: your next reply will still not address this point, nor clarify your definition of “about”, but simply handwave again why it is that I am doing something else wrong. Anything but actually admitting that you have been using a mind-projecting definition of “about” since the beginning of this conversation, right up until the point where you ducked the question by asking me to taboo it, rather than defend the imprecise definition you’ve been using, or clear up any of the other handwaving you’ve been using to beg the question. I base this prediction on the rapid increase in non-responsive replies that, instead of defending the weak points of your position, represent attempts to find new ways to attack me and/or my position. A rationalist should be able to attack their own weak points, let alone being able to defend them, without resorting to distraction, subject-changing, and playing to the gallery.)
There are natural categories, like “tigers”, that don’t require much of a mind to define. It’s not mind projection fallacy to say that something is a tiger.
P.S. I’m correcting self-censoring threshold, so expect silence where before I’d say something for the fifth time.
Is that actually an argument? ’cause it sounds like a random sentence injected into the conversation, perhaps as an invitation for me to waste time tearing “natural categories” to shreds, while leaving you still able to deny that your statement actually relates in any substantial way to your point… thereby once again relieving you of any need to actually defend your position.
That is, are you actually claiming aboutness to be a natural category? Or just trying to get me to treat your argument as if you were doing so?
I already did and do expect it; see my “prediction” in the parent to your comment. I predicted that you would remain silent on any substantive issues, and avoid admitting anywhere where you were mistaken or incorrect. (I notice, for example, that you went back and deleted the comment where you said I was using “mind projection fallacy” incorrectly, rather than admit your attack was in error.)
And, as predicted, you avoided directly addressing the actual point of contention, instead choosing to enter a new piece of handwaving to imply that I am doing something else wrong.
That is, you appear to now be implying that I am using an overbroad definition of the MPF, without actually saying that I am doing it, or that your statement is in any way connected to your own position. This is a nice double bind, since either way I interpret the statement, you can retreat… and throw in more irrelevancies.
I don’t know if “troll” is a natural category, but you’re sure getting close to where I’d mind-project your behavior as matching that of one. ;-)
For the record, I thought it obvious that my argument above implied that I claim aboutness to be a natural category (although I’m not perfectly sure it’s a sound argument). I deleted my comment because I deemed it low-quality, before knowing you responded to it.
It’s not.
First, the only way it can be one is if “natural category” has the reductionist meaning of “a category based on distinctions that humans are biased towards using as discriminators”, rather than “a category that ‘naturally’ exists in the territory”. (Categories are abstractions, not physical entities, after all.)
And second, even if you do use the reductionist meaning of “natural category”, then this does not in any way undermine the conclusion that “aboutness” is mind projection when you omit the entity mapping that aboutness from the description.
In other words, this argument appears to result in only one of two possibilities: either “aboutness” is not a natural category per the reductionist definition, and thus inherently a mind projection when the attribution source is omitted, or “aboutness” is a natural category per the reductionist definition… in which case the attribution source has to be a human brain (i.e., in another map).
Finally, if we entirely reject the reductionist definition of “natural category”, then “natural category” is itself an instance of the mind projection fallacy, since the description omits any definition of for whom the category is “natural”.
In short, QED: the argument is not sound. (I just didn’t want to bother typing all this if you were going to retreat to a claim this was never your argument.)
Indeed. If this didn’t work then there wouldn’t be any practical point in modeling physics!
To the (unknowable*) extent that the portion of my map labelled “territory” is an accurate reflection of the relevant portion of the territory, do I get to say that my preferences are “about” the territory (implicitly including disclaimers like “as mediated by the map”)?
* due at the very least to Matrix/simulation scenarios
You can say it all you want, it just won’t make it true. ;-) Your preference is “about” your experience, just as the thermostat’s heating and cooling preferences are “about” the temperature of its sensor, relative to its setting.
For there to be an “about”, there has to be another observer, projecting a relationship of intention onto the two things. It’s a self-applied mind projection—a “strange loop” in your model—to assert that you can make such statements about your own preferences, like a drawing of Escher wherein Escher is pictured, making the drawing. The whole thing only makes sense within the surface of the paper.
(Heck, it’s probably a similar strange loop to make statements about one’s self in general, but this probably doesn’t lead to the same kind of confusion and behavioral problems that result from making assertions about one’s preferences.… No, wait, actually, yes it does! Self-applied nominalizations, like “I’m bad at math” are an excellent example. Huh. I keep learning interesting new things in this discussion.)
That’s one way of writing. Another is to edit what you intend to post before you click ‘comment’.
I feel your frustration, but throwing the word “magical” in there is just picking a fight, IMO. Anyway, I too would like to see P.J. Eby summarize his position in this format.
I have a certain technical notion of magic in mind. This particular comment wasn’t about frustration (some of the others were), I’m trying out something different of which I might write a post later.
Be charitable in your interpretation, and remember the Least Convenient Possible World principle. I was presuming that the setup was such that being alive on Mars wouldn’t be a ‘fate worse than death’ for her; if it were, I’d choose differently. If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.
I would say rather that my reaction is my evaluation of an imagined future world. The essence of many decision algorithms is to model possible futures and compare them to some criteria. In this case, I have complicated unconscious affective criteria for imagined futures (which dovetail well with my affective criteria for states of affairs I directly experience), and my affective reaction generally determines my actions.
To the extent this is true (as in the sense of my previous sentence), it is a tautology. I understand what you’re arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals. I am not asserting this; I believe that our affective algorithms do often operate under more selfish and basic criteria, and that they fixate on the most salient possibilities instead of weighing probabilities properly, among other things.
However, these affective algorithms do appear to respond more strongly to certain facets of “how I expect the world to be” than to facets of “how I expect to think the world is” when the two conflict (with an added penalty for the expectation of being deceived), and I don’t find that problematic on any level.
As I said, it’s still going to be about your experience during the moments until your memory is erased.
I took that as a given, actually. ;-) What I’m really arguing against is the naive self-applied mind projection fallacy that causes people to see themselves as decision-making agents—i.e., beings with “souls”, if you will. Asserting that your preferences are “about” the territory is the same sort of error as saying that the thermostat “wants” it to be a certain temperature. The “wanting” is not in the thermostat, it’s in the thermostat’s maker.
Of course, it makes for convenient language to say it wants, but we should not confuse this with thinking the thermostat can really “want” anything but for its input and setting to match. And the same goes for humans.
(This is not a mere fine point of tautological philosophy; human preferences in general suffer from high degrees of subgoal stomp, chaotic loops, and other undesirable consequences arising as a direct result of this erroneous projection. Understanding the actual nature of preferences makes it easier to dissolve these confusions.)
I wish I could upvote this two or three times. Thank you.
What features of that comment made it communicate something new to you? What was it that got communicated?
The comment restated a claim that a certain relationship is desirable as a claim that given that it’s desirable, there is a process that establishes it to be true. It’s interesting how this restatement could pierce inferential distance: is preference less trustworthy than a fact, and so demonstrating the conversion of preference into a fact strengthens the case?
Given the length of the thread I branched from, it looks like you and P.J. Eby ended up talking past each other to some extent, and I think that you both failed to distinguish explicitly between the current map (which is what you calculate the territory to be) and a hypothetical future map.
P.J. Eby was (correctly) insisting that your utility function is only in contact with your current map, not the territory directly. You were (correctly) insisting that your utility function cares about (what it calculates to be) the future territory, and not just the future map.
Is that a fair statement of the key points?
Utility function is no more “in contact” with your current map than the actual truth of 2+2=4 is in contact with display of a calculator that displays the statement. Utility function may care about past territory (and even counterfactual territory) as well as future territory, with map being its part. Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn’t be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.
Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we’re talking about practical purposes, this is a relevant point.
Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I’m generally more repulsed by the ones in which my map says it’s fine where it’s not than by the opposite.
This something of a nitpick, but this isn’t strictly true. If others are trying to calculate your utility function (in order to help you), this will depend on their maps rather than yours (though probably including their map of your map). The difference becomes important if their maps are more accurate than yours in some respect (or if they can affect how accurate your map is).
For example, if you know that I value not being deceived (and not merely the subjective experience of not being deceived), and you care about my welfare, then I think that you should not deceive me, even if you know that I might perceive my welfare to be higher if you did.
Oh, good point. I should have restricted it to “any calculation you personally do”, in which case I believe it holds.
At which point it becomes trivial: any calculation that is done on your map is done using your map, just Markovity of computation...
A related point is that you can create tools that make decisions themselves, in situations only of possibility of which you are aware.
Right. It’s trivial, but relevant when discussing in what sense our decision algorithms refer to territory versus map.
I can’t parse this. What do you mean?
If you install an alarm system that uses a video camera to recognize movement and calls the police if it’s armed, you are delegating some of the map-making and decision-making to the alarm system. You are neither aware of the exact nature of possible intruders, nor making a decision regarding calling the police before any intrusion actually occurs. The system decides what to do by itself, according to the aspect of your values it implements. You map is not involved.
Yes, but your decision to install it (as well as your decision to arm it) comes from your map. You would not install it if you thought you had virtually no chance of being burglarized, or if you thought that it would have a false alarm every five minutes when the train went past.
We can make choices that cause other (human, mechanical, etc) agents to act in particular ways, as one of the manners in which we affect possible futures. But these sorts of choices are evaluated by us in the same way as others.
I fear we’ve resorted to arguing about the semantics of “map” versus “territory”, as I don’t see a scenario where we’d predict or decide differently from each other on account of this disagreement. As such, I’m willing to drop it for now unless you see such a scenario.
(My disagreement with Mr. Eby, on the other hand, appears to be more substantive.)
It appears to lead nowhere: your comments are clear, while his are all smoke and mirrors, in many many words.
And does this alarm system have “preferences” that are “about” reality? Or does it merely generate outputs in response to inputs, according to the “values it implements”?
My argument is simply that humans are no different than this hypothetical alarm system; the things we call preferences are no different than variables in the alarm system’s controller—an implementation of values that are not our own.
If there are any “preferences about reality” in the system, they belong to the maker of the alarm system, as it is merely an implementation of the maker’s values.
By analogy, if our preferences are the implementation of any values, they are the “values” of natural selection, not our own.
If now you say that natural selection doesn’t have any preferences or values, then we are left with no preferences anywhere—merely isomorphism between control systems and their environments. Saying this isomorphism is “about” something is saying that a mental entity (the “about” relationship) exists in the real world, i.e., supernaturalism.
In short, what I’m saying is that anybody who argues human preferences are “about” reality is anthropomorphizing the alarm system.
However, if you say that the alarm system does have preferences by some reductionistic definition of “preference”, and you assert that human preference is exactly the same, then we are still left to determine the manner in which these preferences are “about” reality.
If nobody made the alarm system, but it just happened to be formed by a spontaneous jumbling of parts, can it still be said to have preferences? Are its “preferences” still “about” reality in that case?
Both. You are now trying to explain away the rainbow, by insisting that it consists of atoms, which can’t in themselves possess the properties of a rainbow.
So an alarm system has preferences? That is not most people’s understanding of the word “preference”, which requires a degree of agency that most rationalists wouldn’t attribute to an alarm system.
Nonetheless, let us say an alarm system has preferences. You didn’t answer any of my follow-on questions for that case.
As for explaining away the rainbow, you seem to have me confused with an anti-reductionist. See Explaining vs. Explaining Away, in particular:
At this point, I am attempting to show that the very concept of a “preference” existing in the first place is something projected onto the world by an inbuilt bias in human perception. Reality does not have preferences, it has behaviors.
This is not erasing the rainbow from the world, it’s attempting to erase the projection of a mind-modeling variable (“preference”) from the world, in much the same way as Eliezer broke down the idea of “possible” actions in one of his series.
So, if you are claiming that preference actually exists, please give your definition of a preference, such that alarm systems and humans both have them.
A good reply, if only you approached the discussion this constructively more often.
Note that probability is also in the mind, but yet your see all the facts through it, and you can’t ever revoke it, each mind is locked in its subjectively objective character. What do you think of that?
I think that those things have already been very well explained by Eliezer—so much so that I assumed that you (and the others participating in this discussion) would have already internalized them to the same degree as I have, such that asserting “preferences” to be “about” things would be a blatantly obvious instance of the mind projection fallacy.
That’s why, early on, I tended to just speak as though it was bloody obvious, and why I haven’t been painstakingly breaking it all out piece by piece, and why I’ve been baffled by the argument, confusion, and downvoting from people for whom this sort of basic reductionism ought to be a bloody simple matter.
Oh, and finally, I think that you still haven’t given your definition of “preference”, such that humans and alarm systems both have it, so that we can then discuss how it can then be “about” something… and whether that “aboutness” exists in the thing having the preference, or merely in your mental model of the thing.
That in reply to a comment full of links to Eliezer’s articles. You also didn’t answer my comment, but wrote some text that doesn’t help me in our argument. I wasn’t even talking about preference.
I know. That’s the problem. See this comment and this one, where I asked for your definition of preference, which you still haven’t given.
That’s because you also “didn’t answer my comment, but wrote some text that doesn’t help me in our argument.” I was attempting to redirect you to answering the question which you’ve now ducked twice in a row.
Writing text that doesn’t help is pointless and mildly destructive. I don’t see how me answering your questions would help this situation. Maybe you have the same sentiment towards answering my questions, but that’s separate from reciprocation. I’m currently trying to understand your position in terms of my position, not to explain to you my position.
We reached a point in the discussion where it appears the only way we could disagree is if we had a different definition of “preference”. Since I believe I’ve made my definition quite clear, I wanted to know what yours is.
It might not help you, but it would certainly help me to understand your position, if you are not using the common definition of preference.
I asked you first, and you responded with (AFAICT) a non-answer. You appear to have been projecting entirely different arguments and thesis on to me, and posting links to articles whose conclusions I appear to be more in line with than you are—again, as far as I can tell.
So, I actually answered your question (i.e. “what do you think?”), even though you still haven’t answered mine.
That’s why philosophy is such a bog, and why it’s necessary to arrive at however insignificant but technical conclusions in order to move forward reliably.
I chose the articles in the comment above because they were in surface-match with what you are talking about, as a potential point on establishing understanding. I asked basically how you can characterize your agreement/disagreement with them, and how it carries over to the preference debate.
And I answered that I agree with them, and that I considered it foundational material to what I’m talking about.
Indeed, which is why I’d now like to have the answer to my question, please. What definition of “preferences” are you using, such that an alarm system, thermostat, and human all have them? (Since this is not the common, non-metaphorical usage of “preference”.)
Preference is order on the lotteries of possible worlds (ideally established by expected utility), usually with agent a part of the world. Computations about this structure are normally performed by a mind inside the mind. The agent tries to find actions that determine the world to be as high as possible on the preference order, given the knowledge about it. Now, does it really help?
Yes, as it makes clear that what you’re talking about is a useful reduction of “preference”, unrelated to the common, “felt” meaning of “preference”. That alleviates the need to further discuss that portion of the reduction.
The next step of reduction would be to unpack your phrase “determine the world”… because that’s where you’re begging the question that the agent is determining the world, rather than determining the thing it models as “the world”.
So far, I have seen no-one explain how an agent can go beyond its own model of the world, except as perceived by another agent modeling the relationship between that agent and the world. It is simply repeatedly asserted (as you have effectively just done) as an obvious fact.
But if it is an obvious fact, it should be reducible, as “preference” is reducible, should it not?
Hmm… Okay, this should’ve been easier if the possibility of this agreement was apparent to you. This thread is thereby merged here.
I’d been following this topic and getting frustrated with my inability to put my opinion on the whole preferences-about-the-territory thing into words, and I thought that orthonomal’s comment accomplished it very nicely. I don’t think I understand your other question.