There was the example of discovering how to cue your students into signalling they understand the content. I think this is about engaging with a reality-masking puzzle that might show up as “how can I avoid my students probing at my flaws while teaching” or “how can I have my students recommend me as a good tutor” or etc.
It’s a puzzle in the sense that it’s an aspect of reality you’re grappling with. It’s reality-masking in that the pressure was away from building true/accurate maps.
To say this more slowly:
Let’s take “tinkering” to mean “a process of fiddling with a [thing that can provide outputs] while having some sort of feedback-loop whereby the [outputs provided by the thing] impacts what fiddling is tried later, in such a way that it doesn’t seem crazy to say there is some ‘learning’ going on.”
Examples of tinkering:
A child playing with legos. (The “[thing that provides outputs]” here is the [legos + physics], which creates an output [an experience of how the legos look, whether they fall down, etc.] in reply to the child’s “what if I do this?” attempts. That output then affects the child’s future play-choices some, in such a way that it doesn’t seem crazy to say there is some “learning” happening.)
An person doodling absent-mindedly while talking on the phone, even if the doodle has little to no conscious attention;
A person walking. (Since the walking process (I think) contains at least a bit of [exploration / play / “what happens if I do this?”—not necessarily conscious], and contains some feedback from “this is what happens when you send those signals to your muscles” to future walking patterns)
A person explicitly reasoning about how to solve a math problem
A family member A mostly-unconsciously taking actions near another family member B [while A consciously or unconscoiusly notices something about how the B responds, and while A has some conscious or unconscious link between [how B responds] and [what actions A takes in future].
By a “puzzle”, I mean a context that gets a person to tinker. Puzzles can be person-specific. “How do I get along with Amy?” may be a puzzle for Bob and may not be a puzzle for Carol (because Bob responds to it by tinkering, and Carol responds by, say, ignoring it). A kong toy with peanut butter inside is a puzzle for some dogs (i.e., it gets thesedogs to tinker), but wouldn’t be for most people. Etc.
And… now for the hard part. By a “reality-masking puzzle”, I mean a puzzle such that the kind of tinkering it elicits in a given person will tend to make that person’s “I” somehow stupider, or in less contact with the world.
The usual way this happens is that, instead of the tinkering-with-feedback process gradually solving an external problem (e.g., “how do I get the peaut butter out of the kong toy?”), the tinkering-with-feedback process is gradually learning to mask things from part of their own mind (e.g. “how do I not-notice that I feel X”).
However, it differs from that distinction in that “rationalization” usually refers to processes happening within a single person’s mind. And in many examples of “reality-masking puzzles,” the [process that figures out how to mask a bit of reality from a person’s “I”] is spread across multiple heads, with several different tinkering processes feeding off each other and the combined result somehow being partially about blinding someone.
I am actually not all that satisfied by the “reality-revealing puzzles” vs “reality-masking puzzles” ontology. It was more useful to me than what I’d had before, and I wanted to talk about it, so I posted it. But… I understand what it means for the evidence to run forwards vs backwards, as in Eliezer’s Sequences post about rationalization. I want a similarly clear-and-understood generalization of the “reasoning vs rationalizing” distinction that applies also to processes to spread across multiple heads. I don’t have that yet. I would much appreciate help toward this. (Incremental progress helps too.)
The core distinction between tinkering that is “reality-revealing” and tinkering that is “reality-masking,” is which process is learning to predict/understand/manipulate which other process.
When a process that is part of your core “I” is learning to predict/manipulate an outside process (as with the child who is whittling, and is learning to predict/manipulate the wood and pocket knife), what is happening is reality-revealing.
When a process that is not part of your core “I” is learning to predict/manipulate/screen-off parts of your core “I”s access to data, what is happening is often reality-masking.
(Multiple such processes can be occurring simultaneously, as multiple processes learn to predict/manipulate various other processes all at once.)
The “learning” in a given reality-masking process can be all in a single person’s head (where a person learns to deceive themselves just by thinking self-deceptive thoughts), but it often occurs via learning to impact outside systems that then learn to impact the person themselves (like in the example of me as a beginning math tutor learning to manipulate my tutees into manipulating me into thinking I’d explained things clearly)).
The “reality-revealing” vs “reality-masking” distinction is in attempt to generalize the “reasoning” vs “rationalizing” distinction to processes that don’t all happen in a single head.
There are some edge cases I am confused about, many of which are quite relevant to the “epistemic immune system vs Sequences/rationality” stuff discussed above:
Let us suppose a person has two faculties that are both pretty core parts of their “I”—for example, deepset “yuck/this freaks me out” reactions (“A”), and explicit reasoning (“B”). Now let us suppose that the deepset “yuck/this freaks me out” reactor (A) is being used to selectively turn off the person’s contact with explicit reasoning in cases where it predicts that B “reasoning” will be mistaken / ungrounded / not conducive to the goals of the organism. (Example: a person’s explicit models start saying really weird things about anthropics, and then they have a less-explicit sense that they just shouldn’t take arguments seriously in this case.)
What does it mean to try to “help” a person in such as case, where two core faculties are already at loggerheads, or where one core faculty is already masking things from another?
If a person tinkers in such a case toward disabling A’s ability to disable B’s access to the world… the exact same process, in its exact same aspect, seems “reality-revealing” (relative to faculty B) and “reality-masking” (relative to faculty A).
I want a similarly clear-and-understood generalization of the “reasoning vs rationalizing” distinction that applies also to processes to spread across multiple heads. I don’t have that yet. I would much appreciate help toward this.
I’m not entirely happy with any of the terminology suggested in that post; something like “seeing your preferences realized” vs. “seeing the world clearly” would in my mind be better than either “self vs. no-self” or “design specifications vs. engineering constraints”.
In particular, Vaniver’s post makes the interesting contribution of pointing out that while “reasoning vs. rationalization” suggests that the two would be opposed, seeing the world clearly vs. seeing your preferences realized can be opposed, mutually supporting, or orthogonal. You can come to see your preferences more realized by deluding yourself, but you can also deepen both, seeing your preferences realized more because you are seeing the world more clearly.
In that ontology, instead of something being either reality-masking or reality-revealing, it can
A. Cause you to see your preferences more realized and the world more clearly
B. Cause you to see your preferences more realized but the world less clearly
C. Cause you to see your preferences less realized but the world more clearly
D. Cause you to see your preferences less realized and the world less clearly
But the problem is that a system facing a choice between several options has no general way to tell whether some option it could take is actually an instance of A, B, C or D or if there is a local maximum that means that choosing one possiblity increases one variable a little, but another option would have increased it even more in the long term.
E.g. learning about the Singularity makes you see the world more clearly, but it also makes you see that fewer of your preferences might get realized than you had thought. But then the need to stay alive and navigate the Singularly successfully, pushes you into D, where you are so focused on trying to invest all your energy into that mission that you fail to see how this prevents you from actually realizing any of your preferences… but since you see yourself as being very focused on the task and ignoring “unimportant” things, you think that you are doing A while you are actually doing D.
In the spirit of incremental progress, there is an interpersonal reality-masking pattern I observe.
Perhaps I’m meeting someone I don’t know too well, and we’re sort of feeling each other out. It becomes clear that they’re sort of hoping for me to be shaped a certain way. To take the concrete example at hand, perhaps they’re hoping that I reliably avoid reality-masking puzzles. Unless I’m quite diligent, then I will shape my self-presentation to match that desire.
This has two larger consequences. The first is if that person is trying to tell if they want to have more regular contact with me, we’re starting to build a relationship with a rotten plank that will spawn many more reality-masking puzzles.
The second is that I might buy my own bullshit, and identify with avoiding reality-masking puzzles. And I might try to proselytize for this behavior. But I don’t really understand it. So when talking to people, I’ll be playing with the puzzle of how to mask my lack of understanding / actually holding the virtue. And if I’m fairly confident about the goodness of this virtue, then I’ll also be pushing those around me to play with the puzzle of how they can feel they have this virtue without knowing what it really is
To me terminology like “puzzle” seems to suggest it is a search for an answer but the process seems also be characterised by avoidance of information generation.
You could have a challenge of lifting a weigth and one could struggle by pulling or pressing hard with their muscles. “tinkering” seems to refer to cognitive adaptation so weightlifting doesn’t fit into the definition. But to me it seems it is more about success rather than smarting up. If one phrases it as “I feel uncomfortable when X happens, let’s do something different” and “Now I feel comfortable” it is a challenge and a struggle but not a question or a puzzle. If one were to ask “What I could do to make myself comfortable?” that could be answered with knowledge or knowledge generation. But it doesn’t seem clear to me whether the struggle actually has question structure.
At most extreme it would not be totally crazy to describe a weightlifter as answering the question “How do I lift these weights?” and the answer being “give muscle motor commands in the order x, y ,z”. I guess somebody could help with weigthlifting with turning it into a puzzle “hey I see your technique is wrong. Try lifting like this.”. But more usually it is a challenge of bothering the effort and maybe living throught the uncomfortability of the lift. And while even those could be turned into emotional intelligence questions (“emotional technique”) they are not standardly tackled as questions.
Someone that is interested in “instrumental epistemology” should be interested in instrumental anything and succeeding at a task often involves succeding in dimensions other than epistemology too. All models are wrong but some are useful so in some situations it might be easy to find models that are very useful but very simple. Like being a religious zealot might give a lot of confidence which could be very useful so a consequentialist mind might recognise the success and lean into that direction. Is such an inductive inference reasonable? Maybe doing quantum mechanics as a bind fate black box leads to “shut up and calculate” be a more succesfull strategy than trying to form a broken understading/intuition and suffer many mistakes. Thus competence might mean abstraction supression.
A couple people asked for a clearer description of what a “reality-masking puzzle” is. I’ll try.
JamesPayor’s comment speaks well for me here:
To say this more slowly:
Let’s take “tinkering” to mean “a process of fiddling with a [thing that can provide outputs] while having some sort of feedback-loop whereby the [outputs provided by the thing] impacts what fiddling is tried later, in such a way that it doesn’t seem crazy to say there is some ‘learning’ going on.”
Examples of tinkering:
A child playing with legos. (The “[thing that provides outputs]” here is the [legos + physics], which creates an output [an experience of how the legos look, whether they fall down, etc.] in reply to the child’s “what if I do this?” attempts. That output then affects the child’s future play-choices some, in such a way that it doesn’t seem crazy to say there is some “learning” happening.)
An person doodling absent-mindedly while talking on the phone, even if the doodle has little to no conscious attention;
A person walking. (Since the walking process (I think) contains at least a bit of [exploration / play / “what happens if I do this?”—not necessarily conscious], and contains some feedback from “this is what happens when you send those signals to your muscles” to future walking patterns)
A person explicitly reasoning about how to solve a math problem
A family member A mostly-unconsciously taking actions near another family member B [while A consciously or unconscoiusly notices something about how the B responds, and while A has some conscious or unconscious link between [how B responds] and [what actions A takes in future].
By a “puzzle”, I mean a context that gets a person to tinker. Puzzles can be person-specific. “How do I get along with Amy?” may be a puzzle for Bob and may not be a puzzle for Carol (because Bob responds to it by tinkering, and Carol responds by, say, ignoring it). A kong toy with peanut butter inside is a puzzle for some dogs (i.e., it gets these dogs to tinker), but wouldn’t be for most people. Etc.
And… now for the hard part. By a “reality-masking puzzle”, I mean a puzzle such that the kind of tinkering it elicits in a given person will tend to make that person’s “I” somehow stupider, or in less contact with the world.
The usual way this happens is that, instead of the tinkering-with-feedback process gradually solving an external problem (e.g., “how do I get the peaut butter out of the kong toy?”), the tinkering-with-feedback process is gradually learning to mask things from part of their own mind (e.g. “how do I not-notice that I feel X”).
This distinction is quite related to the distinction between reasoning and rationalization.
However, it differs from that distinction in that “rationalization” usually refers to processes happening within a single person’s mind. And in many examples of “reality-masking puzzles,” the [process that figures out how to mask a bit of reality from a person’s “I”] is spread across multiple heads, with several different tinkering processes feeding off each other and the combined result somehow being partially about blinding someone.
I am actually not all that satisfied by the “reality-revealing puzzles” vs “reality-masking puzzles” ontology. It was more useful to me than what I’d had before, and I wanted to talk about it, so I posted it. But… I understand what it means for the evidence to run forwards vs backwards, as in Eliezer’s Sequences post about rationalization. I want a similarly clear-and-understood generalization of the “reasoning vs rationalizing” distinction that applies also to processes to spread across multiple heads. I don’t have that yet. I would much appreciate help toward this. (Incremental progress helps too.)
To try yet again:
The core distinction between tinkering that is “reality-revealing” and tinkering that is “reality-masking,” is which process is learning to predict/understand/manipulate which other process.
When a process that is part of your core “I” is learning to predict/manipulate an outside process (as with the child who is whittling, and is learning to predict/manipulate the wood and pocket knife), what is happening is reality-revealing.
When a process that is not part of your core “I” is learning to predict/manipulate/screen-off parts of your core “I”s access to data, what is happening is often reality-masking.
(Multiple such processes can be occurring simultaneously, as multiple processes learn to predict/manipulate various other processes all at once.)
The “learning” in a given reality-masking process can be all in a single person’s head (where a person learns to deceive themselves just by thinking self-deceptive thoughts), but it often occurs via learning to impact outside systems that then learn to impact the person themselves (like in the example of me as a beginning math tutor learning to manipulate my tutees into manipulating me into thinking I’d explained things clearly)).
The “reality-revealing” vs “reality-masking” distinction is in attempt to generalize the “reasoning” vs “rationalizing” distinction to processes that don’t all happen in a single head.
There are some edge cases I am confused about, many of which are quite relevant to the “epistemic immune system vs Sequences/rationality” stuff discussed above:
Let us suppose a person has two faculties that are both pretty core parts of their “I”—for example, deepset “yuck/this freaks me out” reactions (“A”), and explicit reasoning (“B”). Now let us suppose that the deepset “yuck/this freaks me out” reactor (A) is being used to selectively turn off the person’s contact with explicit reasoning in cases where it predicts that B “reasoning” will be mistaken / ungrounded / not conducive to the goals of the organism. (Example: a person’s explicit models start saying really weird things about anthropics, and then they have a less-explicit sense that they just shouldn’t take arguments seriously in this case.)
What does it mean to try to “help” a person in such as case, where two core faculties are already at loggerheads, or where one core faculty is already masking things from another?
If a person tinkers in such a case toward disabling A’s ability to disable B’s access to the world… the exact same process, in its exact same aspect, seems “reality-revealing” (relative to faculty B) and “reality-masking” (relative to faculty A).
You are talking about it as though it is a property of the puzzle, when it seems likely to be an interaction between the person and puzzle
(These last two comments were very helpful for me, thanks.)
I feel like Vaniver’s interpretation of self vs. no-self is pointing at a similar thing; would you agree?
I’m not entirely happy with any of the terminology suggested in that post; something like “seeing your preferences realized” vs. “seeing the world clearly” would in my mind be better than either “self vs. no-self” or “design specifications vs. engineering constraints”.
In particular, Vaniver’s post makes the interesting contribution of pointing out that while “reasoning vs. rationalization” suggests that the two would be opposed, seeing the world clearly vs. seeing your preferences realized can be opposed, mutually supporting, or orthogonal. You can come to see your preferences more realized by deluding yourself, but you can also deepen both, seeing your preferences realized more because you are seeing the world more clearly.
In that ontology, instead of something being either reality-masking or reality-revealing, it can
A. Cause you to see your preferences more realized and the world more clearly
B. Cause you to see your preferences more realized but the world less clearly
C. Cause you to see your preferences less realized but the world more clearly
D. Cause you to see your preferences less realized and the world less clearly
But the problem is that a system facing a choice between several options has no general way to tell whether some option it could take is actually an instance of A, B, C or D or if there is a local maximum that means that choosing one possiblity increases one variable a little, but another option would have increased it even more in the long term.
E.g. learning about the Singularity makes you see the world more clearly, but it also makes you see that fewer of your preferences might get realized than you had thought. But then the need to stay alive and navigate the Singularly successfully, pushes you into D, where you are so focused on trying to invest all your energy into that mission that you fail to see how this prevents you from actually realizing any of your preferences… but since you see yourself as being very focused on the task and ignoring “unimportant” things, you think that you are doing A while you are actually doing D.
In the spirit of incremental progress, there is an interpersonal reality-masking pattern I observe.
Perhaps I’m meeting someone I don’t know too well, and we’re sort of feeling each other out. It becomes clear that they’re sort of hoping for me to be shaped a certain way. To take the concrete example at hand, perhaps they’re hoping that I reliably avoid reality-masking puzzles. Unless I’m quite diligent, then I will shape my self-presentation to match that desire.
This has two larger consequences. The first is if that person is trying to tell if they want to have more regular contact with me, we’re starting to build a relationship with a rotten plank that will spawn many more reality-masking puzzles.
The second is that I might buy my own bullshit, and identify with avoiding reality-masking puzzles. And I might try to proselytize for this behavior. But I don’t really understand it. So when talking to people, I’ll be playing with the puzzle of how to mask my lack of understanding / actually holding the virtue. And if I’m fairly confident about the goodness of this virtue, then I’ll also be pushing those around me to play with the puzzle of how they can feel they have this virtue without knowing what it really is
To me terminology like “puzzle” seems to suggest it is a search for an answer but the process seems also be characterised by avoidance of information generation.
You could have a challenge of lifting a weigth and one could struggle by pulling or pressing hard with their muscles. “tinkering” seems to refer to cognitive adaptation so weightlifting doesn’t fit into the definition. But to me it seems it is more about success rather than smarting up. If one phrases it as “I feel uncomfortable when X happens, let’s do something different” and “Now I feel comfortable” it is a challenge and a struggle but not a question or a puzzle. If one were to ask “What I could do to make myself comfortable?” that could be answered with knowledge or knowledge generation. But it doesn’t seem clear to me whether the struggle actually has question structure.
At most extreme it would not be totally crazy to describe a weightlifter as answering the question “How do I lift these weights?” and the answer being “give muscle motor commands in the order x, y ,z”. I guess somebody could help with weigthlifting with turning it into a puzzle “hey I see your technique is wrong. Try lifting like this.”. But more usually it is a challenge of bothering the effort and maybe living throught the uncomfortability of the lift. And while even those could be turned into emotional intelligence questions (“emotional technique”) they are not standardly tackled as questions.
Someone that is interested in “instrumental epistemology” should be interested in instrumental anything and succeeding at a task often involves succeding in dimensions other than epistemology too. All models are wrong but some are useful so in some situations it might be easy to find models that are very useful but very simple. Like being a religious zealot might give a lot of confidence which could be very useful so a consequentialist mind might recognise the success and lean into that direction. Is such an inductive inference reasonable? Maybe doing quantum mechanics as a bind fate black box leads to “shut up and calculate” be a more succesfull strategy than trying to form a broken understading/intuition and suffer many mistakes. Thus competence might mean abstraction supression.