Yeah, I think that’s quite close to this concept—thanks for the link.
TurnTrout
Karma: 3,580
A common delusion among grandiose schizophrenics in institutions is that they are themselves psychiatrists. Consider a particularly underfunded mental hospital, in which the majority of people who “know” themselves to be psychiatrists are wrong. No examination of the evidence will convince them otherwise. No matter how overwhelming, some reason to disbelieve will be found.
Given this, should any amount of evidence suffice to convince you that you are such a psychiatrist?
I am not aware of any resolution to this paradox.
I don’t think there’s a resolution to this kind of thing. “What algorithm produces good epistemics, even though your memories and beliefs may be arbitrarily corrupted?”. In the general case, you can’t do better than approximating normatively correct reasoning as best you can, given the information you have.
Of course, as actual people, we quickly run into the following issue:
The simplest ideal of thinking deals extensively with uncertainty of external facts, but trusts its own reasoning implicitly. Directly imitating this, when your own reasoning is not 100% trustworthy, is a bad plan. Hopefully this sequence will provide some alternatives.
While reading Focusing today, I thought about the book and wondered how many exercises it would have. I felt a twinge of aversion. In keeping with my goal of increasing internal transparency, I said to myself: “I explicitly and consciously notice that I felt averse to some aspect of this book”.
I then Focused on the aversion. Turns out, I felt a little bit disgusted, because a part of me reasoned thusly:
If the book does have exercises, it’ll take more time. That means I’m spending reading time on things that aren’t math textbooks. That means I’m slowing down.
(Transcription of a deeper Focusing on this reasoning)
I’m afraid of being slow. Part of it is surely the psychological remnants of the RSI I developed in the summer of 2018. That is, slowing down is now emotionally associated with disability and frustration. There was a period of meteoric progress as I started reading textbooks and doing great research, and then there was pain. That pain struck even when I was just trying to take care of myself, sleep, open doors. That pain then left me on the floor of my apartment, staring at the ceiling, desperately willing my hands to just get better. They didn’t (for a long while), so I just lay there and cried. That was slow, and it hurt. No reviews, no posts, no typing, no coding. No writing, slow reading. That was slow, and it hurt.
Part of it used to be a sense of “I need to catch up and learn these other subjects which [Eliezer / Paul / Luke / Nate] already know”. Through internal double crux, I’ve nearly eradicated this line of thinking, which is neither helpful nor relevant nor conducive to excitedly learning the beautiful settled science of humanity. Although my most recent post touched on impostor syndrome, that isn’t really a thing for me. I feel reasonably secure in who I am, now (although part of me worries that others wrongly view me as an impostor?).
However, I mostly just want to feel fast, efficient, and swift again. I sometimes feel like I’m in a race with Alex, and I feel like I’m losing.
Awesome. I should also note this generator is post hoc; I was (trying to) do this for a few years before I was even thinking about the planning fallacy.
I think the important generator is: being robust seems like a solution to this “generalized planning fallacy”[1], where you don’t correctly anticipate which corners should not be cut. So, even though you could theoretically excise some wasted motions by cutting pointless corners, you can’t tell which corners are pointless. Therefore, a better policy is just not cutting corners by default.
I think you should probably do most of these things, but not sure which order to do them in,
TBC, the main point isn’t that people should do these specific things per se, the main thing is the overall mindset.
and meanwhile, I think so long as you’re afraid of being unmasked part of the problem seems like it’s about the fear itself?
This is what I was getting at with
There are a lot of things to say about the impostor syndrome on a psychological basis (the fears are often unrealistic / unmerited, etc). But I’d like to take another angle.
I think the fear itself is the key problem with impostor syndrome, and I wasn’t trying to say “just be so good you feel secure” should be the main line of attack on that insecurity.
I don’t particularly like this name, but it’s just a temporary handle. In the planning fallacy, you optimistically stress-test your planned schedule. In this case, you optimistically stress-test your life in a broader sense; being robust attempts to counter that. ↩︎
On Being Robust
Your definition says “no”, which seems fine. But it might leave you with an awkward distinction between “instrumentally rational decisions” and “decisions rooted in instrumental rationality.”
My definition was the first, which is “instrumental rationality = acting so you win”. So, wouldn’t it say that following your gut was instrumentally rational? At least, if it’s a great idea in expectation given what you knew—I wouldn’t say lottery winners were instrumentally rational.
I define it the first way, and don’t see the case for the second way. Analogously, for a while, Bayesian reasoning was our best guess of what the epistemic Way might look like. But then we find out about logical induction, and that seems to tell us a little more about what to do when you’re embedded. So, we now see it would have been a mistake to define “epistemic rationality” as “adhering to the dictates of probability theory as best as possible”.
I think that Eliezer’s other usage of “instrumental rationality” points to fields of study for theoretical underpinning of effective action.
I don’t think is quite the same thing as instrumental rationality (although it’s tightly entwined). If your goals are simple and well-understood, and you’re interfacing in a social domain with clear rules, the most instrumentally rational thing might be to not overthink it and follow common wisdom.
But it’s particularly important if you want to coordinate with other agents, over the long term. Especially on ambitious, complicated projects in novel domains.
On my initial read, I read this as saying “this is the right thing for some people, even when it isn’t instrumentally rational” (?!). But
I think it’s important to have at least one concept that is “anyone with goals should ultimately be trying to solve them the best way possible”, and at least one concept that is “you might consider specifically studying cognitive patterns and policies and a cluster of related things, as a strategy to pursue particular goals.”
makes me think this isn’t what you meant. Maybe clarify the OP?
Windows machines circa ~2013. Let’s say 128GB hard drives which magically never fail, for 10 PCs.
Suppose you could choose how much time to spend at your local library, during which:
you do not age. Time stands still outside; no one enters or exits the library (which is otherwise devoid of people).
you don’t need to sleep/eat/get sunlight/etc
you can use any computers, but not access the internet or otherwise bring in materials with you
you can’t leave before the requested time is up
Suppose you don’t go crazy from solitary confinement, etc. Remember that value drift is a potential thing.
How long would you ask for?
There can’t be too many things that reduce the expected value of the future by 10%; if there were, there would be no expected value left. So, the prior that any particular thing has such an impact should be quite low.
I don’t follow this argument; I also checked the transcript, and I still don’t see why I should buy it. Paul said:
A priori you might’ve been like, well, if you’re going to build some AI, you’re probably going to build the AI so it’s trying to do what you want it to do. Probably that’s that. Plus, most things can’t destroy the expected value of the future by 10%. You just can’t have that many things, otherwise there’s not going to be any value left in the end. In particular, if you had 100 such things, then you’d be down to like 1/1000th of your values. 1⁄10 hundred thousandth? I don’t know, I’m not good at arithmetic.
Anyway, that’s a priori, just aren’t that many things are that bad and it seems like people would try and make AI that’s trying to do what they want.
In my words, the argument is “we agree that the future has nontrivial EV, therefore big negative impacts are a priori unlikely”.
But why do we agree about this? Why are we assuming the future can’t be that bleak in expectation? I think there are good outside-view arguments to this effect, but that isn’t the reasoning here.
Earlier today, I became curious why extrinsic motivation tends to preclude or decrease intrinsic motivation. This phenomenon is known as overjustification. There’s likely agreed-upon theories for this, but here’s some stream-of-consciousness as I reason and read through summarized experimental results. (ETA: Looks like there isn’t consensus on why this happens)
My first hypothesis was that recognizing external rewards somehow precludes activation of curiosity-circuits in our brain. I’m imagining a kid engrossed in a puzzle. Then, they’re told that they’ll be given $10 upon completion. I’m predicting that the kid won’t become significantly less engaged, which surprises me?
third graders who were rewarded with a book showed more reading behaviour in the future, implying that some rewards do not undermine intrinsic motivation.
Might this be because the reward for reading is more reading, which doesn’t undermine the intrinsic interest in reading? You aren’t looking forward to escaping the task, after all.
While the provision of extrinsic rewards might reduce the desirability of an activity, the use of extrinsic constraints, such as the threat of punishment, against performing an activity has actually been found to increase one’s intrinsic interest in that activity. In one study, when children were given mild threats against playing with an attractive toy, it was found that the threat actually served to increase the child’s interest in the toy, which was previously undesirable to the child in the absence of threat.
A few experimental summaries:
1 Researchers at Southern Methodist University conducted an experiment on 188 female university students in which they measured the subjects’ continued interest in a cognitive task (a word game) after their initial performance under different incentives.
The subjects were divided into two groups. Members of the first group were told that they would be rewarded for competence. Above-average players would be paid more and below-average players would be paid less. Members of the second group were told that they would be rewarded only for completion. Their pay was scaled by the number of repetitions or the number of hours playing. Afterwards, half of the subjects in each group were told that they over-performed, and the other half were told that they under-performed, regardless of how well each subject actually did.
Members of the first group generally showed greater interest in the game and continued playing for a longer time than the members of the second group. “Over-performers” continued playing longer than “under-performers” in the first group, but “under-performers” continued playing longer than “over-performers” in the second group. This study showed that, when rewards do not reflect competence, higher rewards lead to less intrinsic motivation. But when rewards do reflect competence, higher rewards lead to greater intrinsic motivation.
2 Richard Titmuss suggested that paying for blood donations might reduce the supply of blood donors. To test this, a field experiment with three treatments was conducted. In the first treatment, the donors did not receive compensation. In the second treatment, the donors received a small payment. In the third treatment, donors were given a choice between the payment and an equivalent-valued contribution to charity. None of the three treatments affected the number of male donors, but the second treatment almost halved the number of female donors. However, allowing the contribution to charity fully eliminated this effect.
From a glance at the Wikipedia page, it seems like there’s not really expert consensus on why this happens. However, according to self-perception theory,
a person infers causes about his or her own behavior based on external constraints. The presence of a strong constraint (such as a reward) would lead a person to conclude that he or she is performing the behavior solely for the reward, which shifts the person’s motivation from intrinsic to extrinsic.
This lines up with my understanding of self-consistency effects.
5′s confidence seems a bit high, as does 10.
But several of your predictions seem way too confident, given how specific they are. 6,7,14 in particular. 40% for
Hot, but limited war between China and USA resulting in an internal Civil War in mainland China driven by Muslim revolt in north-western China, Taiwan & Hong Kong and economic interests in southern China
seems wrong due to its burdensome details.
What would 15) mean, exactly?
Too vague a prediction indeed, but also collaborative filtering seems to have become a cornerstone of modern online advertising / content recommendation services.
I don’t think about my self-study as “know a bunch of math things”. Rather, it’s a) continually improve at mathematical reasoning, and b) accumulate a bunch of problem-solving strategies and ways-of-looking-at-the-world. I can
consider the analytic properties of a real polynomial (how quickly does the output change with the input? all polynomials are analytic. polynomial approximation theorem)
put on my group theory hat (what kinds of operations does the space of polynomials admit?)
consider it from a linear algebraic standpoint ( is secretly )
put on a number-theoretic hat (consider the distributional behavior of integer polynomials in a modular context)
think about how polynomials might be used in ML (hypothesis classes, complexity-of-fit, what’s the search space behavior like for SGD if the model space is just the coefficients for degree-50 polynomials?)
think about complexity theory (the nice closure properties of polynomial composition)
or even consider the computability properties of polynomials (this is a bit of a stretch, but… for every decision problem containing a finite amount of
YESanswers, there exists a polynomial with roots on the base-ten encodings of thoseYESinputs).
Then, when I actually need to do stuff with polynomials for my research, I can see how the difference of returns from two policies can be represented by a polynomial in the agent’s discount rate, which is a nice result (lemma 32). Insights build on insights.
So, I’m not trying to memorize everything. Leafing through Linear Algebra Done Right, I don’t remember much about what self-adjointness means, or Jordan normal form, or whatever. However, I don’t think that really matters. If I need to use the extraneous stuff, I know it exists, and could just pick it back up.
I am, however, able to regenerate fundamental things I actually use / run into in later studies. I have a mental habit of making myself regenerate random claims in every proof I consider. If we’re relying on the commutativity of addition on the reals, I reflexively supply a proof of that property. I came up with: use the Cauchy sequence limit notion of reals, then rely on the commutativity of rationals under addition for each member of the sequence.
It’s not like I’ve perfectly retained everything I need. I can tell I’m a little rusty on some things. However, I never do conscious reviewing (beyond building on past insights through further study, using the insights in my professional research, and rederiving things periodically). I have empirical feedback that this works pretty well. My PhD qualifier exam involved a matrix analysis question; without having even taken the class, I was able to get the right answer by reasoning using the skills and knowledge I got from self-study.
ETA: FWIW, when I talk with math undergrads at my university about areas we’ve both studied / solve problems with them, my impression is that my comprehension is often better.
uses it to make a confusing claim that “there’s nothing wrong” as though we have no more agency over the development of AI than we do over the chaotic motion of a dice.
It’s foolish to liken the development of AI to a roll of the dice. Given the stakes, we must try to study, prepare for, and guide the development of AI as best we can.
I think you’re misinterpreting the original comment. Scott was talking about there being “nothing wrong” with this conception of epistemic uncertainty before the 1 arrives, where each new roll doesn’t tell you anything about when the 1 will come. He isn’t advocating pacifism about AI risk, though. Ironically enough, in his capacity as lead of the Agent Foundations team at MIRI, Scott is arguably one of the least AI-risk-passive people on the planet.
Just a cheeky way to I decided to refer to the planning fallacy, but for allocating money instead of time.
I’ve developed this skill some. To me, it feels like part of my brain is “slipping sideways”, tugging me harder than appropriate towards a particular line of thinking or conclusion. I think I’m reasonably good at noticing rationalization, but part of my brain still tries to rationalize even after I notice it. I want to get better at responding appropriately.