Mostly I want to improve my cardiovascular health (and, this is optimized a lot around “what I’m empirically actually willing to do a lot of that makes me actually sweat. Previously I’ve tried more typical weightlifting or other activities, but this one I actually stick with.”
Does your “eh, doesn’t matter to much one way another” takeaway stay the same if the weight is increased?
FWIW, I think “efficient” beat saber playing tends to focus on elbow moves, but because I’m optimizing a bit for workout I think I move my shoulders more than usual.
One of my current beliefs, based on skimming older posts periodically (esp. since recommendations), is that a lot of the old comments just weren’t that good. Not sure about posts.
I agree that there’s a better, crisper version of this that has those more distinct.
I’m not sure if the end product, for most people, should keep them distinct because by default humans seem to use blurry clusters of concepts to simplify things into something manageable.
But, I think if you’re aiming to be a robust agent, or build a robustly agentic organization, there’s is something valuable about keeping these crisply separate so you can reason about them well. (you’ve previously mentioned that this is analogous to the friendly AI problem and I agree). I think it’s a good project for many people in the rationalsphere to have undertaken to deepen our understanding, even if it turns out not to be practical for the average person.
I ended up having thoughts here that grew beyond the context (how to think about this feels related to how to think about depleted willpower). Wrote a shortform post.
My current best guess is that you’d get the best results (measured roughly “useful ideas generated and non-useful ideas pruned”) from a collection of norms where
a) people need to take responsibility for their own feelings of fear, and there is help/guidelines on how to go about that if you’re not very good at it yet, and
b) people also take responsibility for learning to the social and writing skills to avoid particularly obvious failure modes.
i.e. “I’m feeling defensive” shouldn’t be a get out of jail free card. (in particular, any request that someone change their communication behavior should come with a corresponding costly signal that you are working on improving your ability to listen while triggered)
And while I think I’ve believed this for a couple weeks, I don’t think I was doing the work to actually embody it, and I think that’s been a mistake I’ve been making.
I am very confused about how to think (and feel!) about willpower, and about feelings of safety.
My impression from overviews of the literature is something like “The depletion model of willpower is real if you believe it’s real. But also it’s at least somewhat real even if you don’t?”
Like, doing cognitive work costs resources. That seems like it should just be true. But your stance towards your cognitive work affects what sort of work you are doing.
Similarly, I have a sense that physiological responses to potentially threatening situations are real. People who feel defensive have a harder time thinking in truthseeking mode rather than “keep myself safe” mode. But, it also seems plausibly-true that if you naively reinforce feelings of defensiveness they get stronger. i.e. if you make saying “I’m feeling defensive” a get out of jail free card, people will use it, intentionally or no. (there’s a weird video about parents tricking babies into thinking they’ve been hit on the head when they haven’t, and the babies start crying as if they’re hurt. I have no idea if this is just cute selection effect but sort of illustrates the point)
There’s a practical question re: “what sort of norms or advice do you want to encourage about how people deal with defensiveness, and with willpower?”. But then there’s also just a “what the hell epistemic state do you actually want to have?” as well as “how do you communicate about this?”
If you’re a manager at a company, maybe you want to have a model of willpower depletion so you can make good tradeoffs, but you don’t want your employees to dwell upon it much? But this obviously collapses as soon as there’s multiple management layers. And in any case it’s not a sustainable equilibrium. (You can’t stop people from learning about willpower. Although you can make a choice not to emphasize it)
Meanwhile, if I’m just trying to manage myself as a person with goals, what should my epistemic state on willpower be?
I’d originally called this “Becoming a Robust Person or Organization”, but then was noticing that this suggested the truncation “Robust Organization” as shorthand for “what an organization looks like when it’s a robust agent.” And that didn’t feel like it had the right connotations.
I’m still not sure of a great phrase. “Agentic Organization” has differently slightly-off connotations. “Robust Agentic Organization” does actually get it across maybe but I don’t believe anyone will ever say it. :P
I’ve been trying to use the phrase ‘feeling of safety’ when it comes up but it has the unfortunate property that ‘aspiring rationalist’ had, where there isn’t a stable equilibrium where people reliably say the whole phrase.
I think there are separate worthwhile skills of “focus on learning empathy/modeling and let clear language flow from that”, and also “writing skills exist that are separate from epistemics” (such as brevity, which I think actually factors in here a bit)
Something that may not have been clear from my past discussion is that when I say “this could have been written in a way that was less triggering”, or something, I’m not (usually) meaning that to be a harsh criticism. Just, the sort of thing that you should say ‘ah, that makes sense. I will work on that’ for the future.
Thanks! I’m not sure this is a place where steelmanning is quite the appropriate tool. My past self was optimized for being my past self, not being right. He was mostly just not trying to solve this question.
But, in this case, I think the best tool is more properly called “modeling people” and maybe “empathy”.
Things my past self cared about and/or believed included:
All the probability stuff feels too hard to think about, and it doesn’t seem like it’s really going to help me that much even if I put a lot of work into it. So for me personally, I’m just going to try to “remember base rates” and a few other simple heuristics and call it a day. I was glad other people took it more seriously though
Truth seems like one of many important things. What matters is getting things accomplished. (I’ve never been optimizing against truth, I have just prioritized other things. There’s been times where I, say, only put 20 minutes into checking an essay for being right, rather than 2 hours, when I had reason to suspect I might have had motivated reasoning.)
I thought (and still think, although less strongly and for more nuanced reasons) that the in person rationality community is unhealthy because it only selects for a few narrow types of person, who are min-maxed in a particular skillset. And I think the in person community is important (both for epistemic and instrumental reasons). It is important to be a community that doesn’t actively drive away people who bring other skills to the table.
I still roughly believe all that. The main update is that there should a) be dedicated spaces that focus on truthseeking as their [probably] sacred value, b) that LessWrong should be such a space. (But, as noted in Tensions in Truthseeking, there are still different tradeoffs you can make in your truthseeking frame, and I think it’s good to have spaces that have made different min-max tradeoffs to explore those tradeoffs. For example, there might be math-heavy spaces, there might be “blunt communication” spaces that optimize for directness, there might be feelings-heavy spaces that optimize for understanding and owning your internal state)
(I have made a bit of conceptual progress on probability stuff. I probably will never do real Bayesian Wizardry but I think grok it better now – I can follow some conversations I didn’t used to be able to follow and in some cases I can participate in and uphold norms that help others on their way to learning it better than I)
There is an interesting thing in all this space I recently re-read while perusing the old critiques of Gleb. A paraphrase of the linked comment is:
I think a problem with effective altruists is they often end up with a conception that marketing is icky, and that without marketing they are ineffective. I think Gleb might have just said “I’d rather be effective and icky than ineffective and pure.” And this is maybe an unhelpful frame that other people are implicitly using. There are ways you can market effectively without actually being icky.
And, while I’m not sure, I think I might have held a frame somewhat like that (I don’t have clear memories of biting either particular bullet). But my current position is “effective altruists should hold to a high epistemic standard, even when marketing. But, learn to market well within those constraints.”
I’m not advocating lying here, I’m advocating learning the communication skills necessary to a) actually get people to understand your point (which they’ll have a harder time with if they’re defensive), and b) not wasting dozens of hours unnecessarily (which could be better spent on figuring other things out).
[and to be clear, I also advocate gaining the courage to speak the truth even if your voice trembles, and be willing to fight for it when it’s important. Just, those aren’t the only skills a rationalist or a rationalist space needs. Listening, communicating clearly, avoiding triggering people’s “use language as politics mode”, and modeling minds and frames different from your own are key skills too]
Okay, but geez man, my past self had different beliefs. What do you want here? What is your incredulity here aiming to accomplish? If you can’t simulate the mind of a person who showed up on LessWrong with one set of beliefs and gradually updated their beliefs in a set of directions that are common on the site, I think you should prioritize learning to simulate other minds a bit
If the takeaway were what mattered about the post, why include all the other stuff?
I think habryka was trying to get across a more cohesive worldview rather than just a few points. I also don’t know that my interpretation is the same as his. But here are some points that I took from this. (Hmm. These may not have been even slightly obvious in the current post, but were part of the background conversation that prompted it, and would probably have eventually been brought up in a future post. And I think the OP at least hints at them)
First, I think there are people in the LessWrong readership who still have some naive conception of “be accountable to the public”, which is in fact a recipe for
It’s as old as “you can only serve one master,”
This is pretty different from how I’d describe this.
In some sense, you only get to serve one master. But value is complex and fragile. So there many be many facets of integrity or morality that you find important to pay attention to, and you might be missing some of them. Your master may contain multitudes.
Any given facet of morality (or more generally, things I care about it) is complicated. I might want, to hold myself to a higher standard that I currently meet, is to have several people whom I hold myself accountable to, each of which pays deep attention to a different facet.
If I’m running a company, I might want to be accountable to
people who deeply understand the industry I’m working in
people who deeply understand human needs, coercion, etc, who can tell me if I’m mistreating my workers,
people who understand how my industry interacts with other industries, the general populace, or the environment, who can call me out if I’m letting negative externalities run wild.
 maybe just having a kinda regular person who just sanity checks if I seem crazy
I might want multiple people for each facet, who look at that facet through a different lens.
By having these facets represented in concrete individuals, I also improve my ability to resolve confusions about how to trade off multiple sacred values. Each individual might deeply understand their domain and see it as most important. But if they disagree, they can doublecrux with each other, or with me, and I can try to integrate their views into something coherent and actionable.
There’s also the important operationalization of “what does accountable mean?” There’s different powers you could give these people, possibly including:
Emergency Doublecrux button – they can demand N hours of your time per year, at least forcing you to have a conversation to justify yourself
Vote of No Confidence – i.e. if your project still seems good but you seem corrupt, they can fire you and replace you
Shut down your project, if it seems net negative
There might be some people you trust with some powers but not others (i.e. you might think someone has good perspectives that justify the emergency double crux button but not the “fire you” button)
There’s a somewhat different conception you could have of all this that’s more coalition focused than personal development focused.
Something that I’m maybe able to put into words now:
The classical example of “sacred values run amok” in my mind is when you ask people how much money a hospital should spend on a heart transplant for a dying child. People try to dodge the question, avoiding trading off a sacred value for a mundane value. Despite the fact that money can buy hospital equipment that saves other lives.
It’s plausible that hospital should hold “keeping people healthy and alive” as an overall sacred value, which they never trade off against. This might forbid some paths where resources are spent on things that weren’t necessary to keep people healthy and alive. But it doesn’t tell you what are the best strategies to go about it are. You’re allowed to sacrifice a boy’s life to buy hospital equipment. You’re even allowed to sacrifice a boy’s life to make sure your employees are well rested and not overly stressed. Running a hospital is a marathon, not a sprint.
Over the past couple years, I have updated to “yes, LessWrong should be the place focused on truthseeking.” I think I came to believe that right around the time I wrote Tensions in Truthseeking, in the process of writing the paragraph about instrumental sacredness. But that tells us what the question is, not what the answer is.
Some of the language about “holding truth sacred” (things you’ve said, and others) has came across to me with a tone of single-minded focus that feels like not being willing to put an upper bound on a heart transplant, rather than earnestly asking the question “how do we get the most valuable truthseeking the most effective way?”
There’s also the bit where operationalization matters. “Minimize falsehood” is a different function than “maximize true, good ideas over time” which is a different function than “maximize true, good ideas that are communicated well enough to impact the world.”
The concerns here make sense.
Something I still can’t tell about your concern, though: one of the things that seemed like the “primary takeaway” here, at least to me, is the concept of thinking carefully about who you want to be accountable to (and to be wary of holding yourself accountable to too many people that won’t be able to understand more complex moral positions you might hold)
So far, having thought through that concept through the various lenses of integrity you list here, it doesn’t seem like something that likely to make things worse. Do you think it is?
(the other claim, to think about what sort of incentives you want for yourself, does seem like the sort of thing some people might interpret as instruction to create coercive environments for themselves. This was fairly different from what I think habryka was aiming at, but I’d agree that might not be clear from this post)
Eliezer notes that this story needs some kind of checking against the fact that ancient tribes were actually egalitarian. Does anyone have a sense of, given that, how to think about this question?
Counterfactual revolutions are basically good, revolutions are basically bad
(The political sort of revolution, not the scientific sort)
(And yes I’d be interested in talking more in person if you’re up for it)
This is an awesome comment on many dimensions, thanks. I both agree with your summary of my position, and I think your cruxes are pretty similar to my cruxes.
There are a few additional considerations of mine which I’ll list, followed by attempting to tease out some deeper cruxes of mine about “what facts would have to be true for me to want to backpropagate the level of fear it seems like you feel into my aesthetic judgment.” [This is a particular metaframe I’m currently exploring]
[Edit: turned out to be more than a few straightforward assumptions, and I haven’t gotten to the aesthetic or ontology cruxes yet]
Additional considerations from my own beliefs:
I define clarity in terms of what gets understood, rather than what gets said. So, using words with non-standard connotations, without doing a lot of up-front work to redefine your terms, seems to me to be reducing clarity, and/or mixing clarity, rather than improving it.
I think it’s especially worthwhile to develop non-court language, for public discourse, if your intent is not to be punative – repurposing court language for non-punative action is particularly confusing. The first definition for “fraud” that comes up on google is “wrongful or criminal deception intended to result in financial or personal gain”. The connotation I associate it with is “the kind of lying you pay fines or go to jail for or get identified as a criminal for”.
By default, language-processing is a mixture of truthseeking and politicking. The more political a conversation feels, the harder it will be for people to remain in truthseeking mode. I see the primary goal of a rationalist/truthseeking space to be to ensure people remain in truthseeking mode. I don’t think this is completely necessary but I do think it makes the space much more effective (in terms of time spent getting points across).
I think it’s very important for language re: how-to-do-politics-while-truthseeking be created separately from any live politics – otherwise, one of the first things that’ll happen is the language get coopted and distorted by the political process. People are right/just to fear you developing political language if you appear to be actively
Fact that is (quite plausibly) my true rejection – Highly tense conversations that I get defensive at are among the most stressful things I experience, which cripple my ability to sleep well while doing them. This is high enough cost that if I had to do it all the time, I would probably just tune them out.
This is a selfish perspective, and I should perhaps be quite suspicious of the rest of my arguments in light of it. But it’s not obviously wrong to me in the first place – having stressful weeks of sleep wrecked is really bad. When I imagine a world where people are criticizing me all the time [in particular when they’re misunderstanding my frame, see below about deep model differences], it’s not at all obvious that the net benefit I or the community gets from people getting to express their criticism more easily outways the cost in productivity (which would, among other things, be spent on other truthseeking pursuits). When I imagine this multiplied across all orgs it’s not very surprising or unreasonable seeming for people to have learned to tune out criticism.
Single Most Important Belief that I endorse – I think trying to develop a language for truthseeking-politics (or politics-adjaecent stuff) could potentially permanently destroy the ability for a given space do politics sanely. It’s possible to do it right, but also very easy to fuck up, and instead of properly transmitting truthseeking-into-politics, politics backpropogates into truthseeking, causes people to view truthseeking norms as a political weapon. I think this is basically what happened with the American Right Wing and their view of science (and I think things like the March for Science are harmful because they exacerbate Science as Politics).
In the same way that it’s bad to tell a lie, to accomplish some locally good thing (because the damage you do to the ecosystem is far worse than whatever locally good thing you accomplished), I think it is bad to try to invent truthseeking-politics-on-the-fly without explaining well what you are doing while also making claims that people are (rightly) worried will cost them millions of dollars. Whatever local truth you’re outputting is much less valuable than the risks you are playing with re: the public commons of “ability to ever discuss politics sanely.”
I really wish we had developed good tools to discuss politics sanely before we got access to billions of dollars. That was an understandable mistake (I didn’t think about it until just this second), but it probably cost us deeply. Given that we didn’t, I think creating good norms requires much more costly signaling of good faith (on everyone’s part) than it might have needed. [this paragraph is all weak confidence since I just thought of it but feels pretty true to me]
People have deep models, in which certain things seem obvious them that are not obvious to others. I think I drastically disagree with you about what your prior should be that “Bob has a non-motivated deep model (or, not any more motivated than average) that you don’t understand”, rather than “Bob’s opinion or his model is different/frightening because he is motivated, deceptive and/or non-truth-tracking.”
My impression is that everyone with a deep, weird model that I’ve encountered was overly biased in favor of their deep model (including you and Ben), but this seems sufficiently explained by “when you focus all your attention on one particular facet of reality, that facet looms much larger in your thinking, and other facets loom less large”, with some amount of “their personality or circumstance biased them towards their model” (but, not to a degree that seems particularly weird or alarming).
Seeing “true reality” involves learning lots of deep models into narrow domains and then letting them settle.
[For context/frame, remember that it took Eliezer 2 years of blogging every day to get everyone up to speed on how to think in his frame. That’s roughly the order-of-magnitude of effort that seems like you should expect to expend to explain a counterintuitive worldview to people]
In particular, a lot of the things that seem alarming to you (like, Givewell’s use of numbers that seem wrong) is pretty well (but not completely) explained by “it’s actually very counterintuitive to have the opinions you do about what reasonable numbers are.” I have updated more towards your view on the matter, but a) it took me a couple years, b) it still doesn’t seem very obvious to me. Drowning-Children-are-Rare is a plausible hypothesis but doesn’t seem so overdetermined that anyone thinks otherwise must be deeply motivated or deceptive.
I’m not saying this applies across the board. I can think of several people in EA or rationalist space who seem motivated in important ways. My sense of deep models specifically comes from the combination of “the deep model is presented to me when I inquire about it, and makes sense”, and “they have given enough costly signals of trustworthiness that I’m willing to give them the benefit of the doubt.”
I have updated over the past couple years on how bad “PR management” and diplomacy are for your ability to think, and I appreciate the cost a bit more, but it still seems less than the penalties you get for truthseeking when people feel unsafe.
I have (low confidence) models that seem fairly different from Ben (and I assume your) model of what exactly early LessWrong was like, and what happened to it. This is complicated and I think beyond scope for this comment.
Unknown Unknowns, and model-uncertainty. I’m not actually that worried about scissor-attacks, and I’m not sure how confident I am about many of the previous models. But they are all worrisome enough that I think caution is warranted.
Many of the above bullet-points are cruxy and suggest natural crux-reframes. I’m going to go into some detail for a few:
I could imagine learning that my priors on “deep model divergence” vs “nope, they’re just really deceptive” are wrong. I don’t actually have all that many data points to have longterm confidence here. It’s just that so far, most of the smoking guns that have been presented to me didn’t seem very definitive.
The concrete observations that would shift this are “at least one of the people that I have trusted turns out to have a smoking gun that makes me think their deep model was highly motivated” [I will try to think privately about what concrete examples of this might be, to avoid a thing where I confabulate justifications in realtime.]
It might be a lot easier than I think to create a public truthseeking space that remains sane in the face of money and politics. Relatedly, I might be overly worried about the risk of destroying longterm ability to talk-about-politics-sanely.
If I saw an existing community that operated on a public forum and onboarded new people all the time, which had the norms you are advocating, and interviewing various people involved seemed to suggest it was working sanely, I’d update. I’m not sure if there are easier bits of evidence to find.
The costs that come from diplomacy might be higher than the costs of defensiveness.
Habryka has described experiences where diplomacy/PR-concerns seemed bad-for-his-soul in various ways. [not 100% sure this is quite the right characterization but seems about right]. I think so far I haven’t really been “playing on hard mode” in this domain, and I think there’s a decent chance that I will be over the next few years. I could imagine updating about how badly diplomacy cripples thought after having that experience, and for it to turn out to be greater than defensiveness.
I might be the only person that suffers from sleep loss or other stress-side-effects as badly as I do.
These were the easier ones. I’m trying to think through the “ontology doublecrux” thing and think about what sorts of things would change my ontology. That may be another while.