I’ve been a LessWrong organizer since 2011, with roughly equal focus on the cultural, practical and intellectual aspects of the community. My first project was creating the Secular Solstice and helping groups across the world run their own version of it. More recently I’ve been interested in improving my own epistemic standards and helping others to do so as well.
Raemon(Raymond Arnold)
Karma: 28,474
Sort of meta, but… I’m thinking about the use of acronyms in order to get more density into your title.
If you only get about 5 words, but one of those words is PONDS, have you successfully gotten 9 words? Maybe, and it’s worth checking. If you can spend one of your words on unfolding complexity, that’s great.
But my experience is “not really” – I tend to totally forget acronyms, unless they are hammered into me a lot over time, which requires other infrastructure. And I could go out of my way to set up a spaced-repetition thingy for this particular post, to get it to stick into my head. But, that wouldn’t be scalable. I can only space-repeat so many things.
This is not a strong claim that this acronym won’t work or that this post isn’t useful. (I think this post works sort of as a space-repeat for the collective LW commentariat about “things to keep in mind for commenting”, even if people don’t end up remembering this particular acronym). But, I dunno, something to keep in mind.
Sort of relatedly, or on the flipside of the coin:
In these threads, I’ve seen a lot of concern with using language “consequentially”, rather than rooted in pure epistemics and map-territory correspondence.
And those arguments have always seemed weird to me. Because… what could you possibly be grounding this all out in, other than consequences? It seems useful to have a concept of “appeals to consequence” being logically invalid. But in terms of what norms to have on as public forum, the key issue on a public forum is that appeals to shortsighted consequences are bad, for the same reason shortsighted consequentialism is often bad.
If you don’t call the president a Vargath (despite them obviously supporting Varg), because they’d be offended, it seems fairly straightforward to argue that this has bad consequences. You just have to model it out more steps.
I would agree with the claim “if you’re constantly checking ‘hey, in this particular instance, maybe it’s net positive to lie?’ you end lying all the time, and end up in a world where people can’t trust each other”, so it’s worth treating appeals to consequences as forbidden as part of a Rule Consequentialism framework. But, why not just say that?
I do, of course, think that the LessWrong community should be and to an extent is such a place.
Something about this has been bugging me and I maybe finally have a grasp on it.
It’s perhaps somewhat entangled with this older Benquo comment elsewhere in this thread. I’m not sure if you endorse this phrasing but your prior paragraph seems similar:
Discourse about how to speak the truth efficiently, on a site literally called “Less Wrong,” shouldn’t have to explicitly disclaim that it’s meant as advice within that context every time, even if it’s often helpful to examine what that means and when and how it is useful to prioritize over other desiderata.
Since a couple-years-ago, I’ve updated “yes, LessWrong should be a fundamentally truthseeking place, optimizing for that at the expense of other things.” (this was indeed an update for me, since I came here for the Impact and vague-appreciation-of-truthseeking, and only later updated that yes, Epistemics are one of the most important cause areas)
But, one of the most important things I want to get out of LessWrong is a clear map of how the rest of the world works, and how to interface with it.
So when I read the conclusion here...
Similarly, the primary thing when you take a word in your lips is your intention to reflect the territory, whatever the means. Whenever you categorize, label, name, define, or draw boundaries, you must cut through to the correct answer in the same movement. If you think only of categorizing, labeling, naming, defining, or drawing boundaries, you will not be able actually to reflect the territory.
...I feel like my epistemics are kind of under attack.
I feel like this statement is motte-and-bailey-ing between “These are the rules of thinking rationally, for forming accurate beliefs, and for communicating about that in particular contexts” and “these are the rules of communicating, in whatever circumstances you might find yourself.”
And it’s actually a pretty big epistemic deal for a site called LessWrong to not draw that distinction. “How coordination works, even with non-rationalists”, is as really big deal, not a special edge case, and I want to be maintaining an accurate map of it the entire time that I’m building out my theory of rationality.
Note: If you haven’t been voting because it felt overwhelming, and want to at least vote on posts that you remember well from 2019 – we’ve recently added a feature on the voting page where you can see which posts you previously voted on. You’ll be able to quickly scan for posts that presumably you have at least some vague memories of.
Hrm. Sent a PM to chat more about it.
Mod note: this is frontpaged, which is an exception to our usual rules. See here for explanation.
I ended up chatting with Habryka (who had originally inspired this post, and now wasn’t sure he agreed with it).
One key additional point here is “gee, doing anything at all is really goddamn hard, and you might not want people to feel any additional disincentive from doing anything, including building monopolies.”
There’s the frustrating “consider reversing all advice you hear, because you maybe filter yourself to hear advice that reinforces your own biases” thingy. I think I endorse the specific phrasings used in this post (which I think were properly caveated). But, I wouldn’t want the takeaway to be people being too overly worried about ensuring they have competitors when they themselves haven’t gotten off the ground.
There’s an additional confusing issue where...
it’s really bad if a mediocre organization is a monopoly, or if organizations stretch beyond their capacity.
it… actually can be good if a highly competent organization does a lot of vertical integration.
And the problem is that figuring out “am I competent or not?” is one off the hardest things you can try to figure out.
Self Review.
I still endorse the broad thrusts of this post. But I think it should change at least somewhat. I’m not sure how extensively, but here are some considerations
Clearer distinctions between Prisoner’s Dilemma and Stag Hunts
I should be more clear about what the game theoretical distinctions I’m actually making between Prisoners Dilemma and Stag Hunt. I think Rob Bensinger rightly criticized the current wording, which equivocates between “stag hunting is meaningfully different” and “‘hunting rabbit’ has nicer aesthetic properties than ‘defect’”.
I think Turntrout spelled out in the comments why it’s meaningful to think in terms of stag hunts. I’m not sure it’s the post’s job to lay it out in the exhaustive detail that his comment does, but it should at least gesture at the idea.
Future Work: Explore a lot of coordination failures and figure out what the actual most common rules / payoff structures are.
Stag Hunting is relevant sometimes, but not always. I think it’s probably more relevant than Prisoner’s Dilemma, which is a step up, but I think it’s worth actually checking which game theory archetypes are most relevant most of the time.
Reworked Example
Some people comment that my proposed stag hunt… wasn’t a stag hunt. I think that’s actually kind of the point (i.e. most things that look like stag hunts are more complicated than you think, and people may not agree on the utility payoff). Coming up with good examples is hard, but I think at the very least the post should make it more clear that no, my original intended Stag Hunt did not have the appropriate payoff matrix after all.
What’s the correct title?
While I endorse most of the models and gears in this post, I… have mixed feelings about the title. I’m not actually sure what the key takeaway of the post is meant to be. Abram’s comment gets at some of the issues here. Benquo also notes that we do have plenty of stag hunts where the schelling choice is Stag (i.e. don’t murder)
I think there were two primary intended ‘negative’ points (i.e. “things you SHOULDN’T do”)
Don’t hunt stag unsustainably. I’ve seen a bunch of people get burned out and frustrated from hitting the Stag button and then not having enough support. Either make sure you have buy-in, or only hunt stag when you can afford to lose.
Be careful punishing people when they choose Rabbit. Often, you’re just wrong about either your proposal being a staghunt, or about people having common knowledge that your proposal is good. (punishing in a way people consider unjust often shuts down coordination potential). I think this is a fairly complex point that this post doesn’t quite address.
There are multiple positive-advice-takeaways you might have from this post, but they depend a lot on what situation you’re actually in.
I’ve alluded to this in other comments, but I think worth spelling out more comprehensively here.
I think this post makes a few main points:
Categories are not arbitrary. You might need different categories for different purposes, but categories are for helping you think about the things you care about, and a category that doesn’t correspond to the territory will be less helpful for thinking and communciating.
Some categories might sort of look like they correspond to something in reality, but they are gerrymandered in a way optimized for deception.
You might sometimes wish something were a member of a category that it isn’t, and it is better to admit that so that you can actually communicate about the current state of reality.
I realize the three points cleave together pretty closely in the author’s model, and make sense to think about in conjunction. But I think trying to introduce them all at once makes for more confusing reading.
I think the followup post Unnatural Categories Are Optimized For Deception does a pretty good job of spelling out the details of points #2 and #3. I think the current post does a good job at #1, a decent job at #2, but a fairly confused job at #3.
In particular, these 2.5 paragraphs feel like a meandering vagueblog. I know what point they’re trying to make, but by trying to avoid the object level political disagreement, the post leaves me very confused about why I might be making these particular mistakes, or what to do about it.
If what you want isn’t currently true in reality, maybe there’s some action you could take to make it become true. To search for that action, you’re going to need accurate beliefs about what reality is currently like. To enlist the help of others in your planning, you’re going to need precise terminology to communicate accurate beliefs about what reality is currently like. Even when—especially when—the current reality is inconvenient.
Even when it hurts.
(Oh, and if you’re actually trying to optimize other people’s models of the world, rather than the world itself—you could just lie, rather than playing clever category-gerrymandering mind games. It would be a lot simpler!)
(By contrast, the Unnatural Categories gives concrete real world examples that explain why you’d make this particular class of mistake. Somewhat oddly, it does use some politically charged examples, but I think it does a good job of laying out in as gearsy, not-too-politicized fashion. I think it actually could probably have gotten away with directly involving the original motivating example for the series)
If the point of Unnatural Categories is to replace this post, and this post is more like a first draft, then… seems potentially fine for Unnatural Categories to become the new canonical version of it and not worry overmuch about this one.
But, I think there’s a lot of distinct concepts here. Breaking them into multiple posts that deal with theseems worthwhile to me.
If this post is appearing in the 2019 Review as a standalone piece, I think it’d be clearer if it just cut the paragraphs I listed (and then reorganized itself slightly), rather than doing a rough, vague job of explaining them. When I got to like “even if it hurts” my reaction was “what? why would it hurt? what are you talking about? I think I know what the underlying political argument is and I’m still kinda confused about what’s going on here.”
I’ve also mentioned elsethread that I think the Bob the Vice President of Sorting example would be helpful to have earlier in the piece, to give a clearer example of when this whole problem might come up. But I realize people may vary in what pedagogy works best for them.
...
A sidepoint I notice while thinking about this: when I go back to the older sequence posts that this essay is referencing...
...well, they totally are vague and don’t spell out what real world examples you might run into that would motivate the philosophical confusion. But they are also much shorter, usually focus on one idea at a time instead of three, and are intermixed with other posts that do lay out more of the motivating-examples.
That seems true, but I think it still makes sense to concentrate the discussion on particular posts. (Zack specifically disavowed this post and the meta-honesty response, so I think it makes most sense to concentrate on Where To Draw The Boundaries and Heads I Win, Tails Never Heard Of Her)
I think it’s reasonable to bring up “this post seems rooted in a wrong frame” on both of those, linking to other examples. But my own voting algorithm for those posts will personally be asking “does this single post have a high overall mix of ‘true’ and ‘important’?”
I think most posts in the review, even the top posts, have something wrong with them, and in some cases I disagree with the author about which things are wrong-enough-to-warrant-fixing. I do feel that the overall review process isn’t quite solid enough for me to really endorse the Best Of book as a statement of “The LessWrong Community fully endorses this post”, and I think that’s a major problem to be fixed for next year. But meanwhile I think it makes more sense to accept that some posts will have flaws.
Hmm. I don’t I want to commit to a huge discussion of it. I’m happy to continue doing async LW comments about it. I’m busier than usual this month. There might turn out to be a day I had a spare hour or two to chat in more detail but don’t think I want to spend cognition planning around that.
I think I’ve mostly said my main piece and am fairly happy with “LW members can read what Matt and Ray have said so far and vote accordingly.” If you raise specific points on specific posts I (and others) might change their vote for those posts.
Yeah I thought about trying to specify ‘finance stuff’ but then wasn’t actually sure I knew the definition of finance and whether that was quite the right word
Okay, that is compatible with the rest of my Paul model. Does still seem to fit into the ‘what’s least impossible’ frame.
Matt Levine on “Fraud is no fun without friends.”
I’d still argue that at the level of individual posts rather than the sequence as a whole. (In this particular context. If there was something like a “Algorithms of Deception Sequence Book” getting considered as a whole I’d have a pretty different attitude)
The thing the “timelines are too short” was trying to get at was “it has to be competitive with mainstream AI in order to work” (pretty sure Paul has explicitly said this), with, what I thought was basically a followup assumption of “and timelines are too short to have time to get a competitive thing based off the kind of deconfusion work that MIRI does.”
Update: in the past year, since writing this comment, I periodically thought ‘man, I should probably post some somewhat controversial opinions, to get into the habit of actually having the backbone to do that sometimes.’
And… well, I haven’t gotten around to it. And I think there at least is a little bit of “Ray is just being cowardly here”, but, also… it just doesn’t feel super worth it. It’s effortful to write up stuff in the first place, and I have a ton of stuff I want to write that isn’t controversial which I think is really important, and meanwhile I don’t think on the object level that any given controversial opinion was really that valuable to express except for building up my social resilience backbone.
I think “build up social resilience” is actually pretty important, enough that I lean towards “yes actually feel bad about this even though any given controversial opinion doesn’t feel very important to express”. But, whelp, here we are.
On one hand, I think I probably agree with the overall thrust of your criticism. But I don’t think I endorse it in the context of the review.
Some of the posts in the review are sort of a stand-in for a whole sequence (Moral Mazes, Multiagent Models), but I don’t think Zack’s posts are (or at least I have not interpreted them as such). So I think it makes more sense to look at the given particular posts up for review and for each of them ask “okay, is the writing on this particular post embodying a wrong or confusing frame?”
I think the answer is yes in some cases, no in others.
In general: I probably still have some deep disagreements with Zack. (I’m not entirely sure, see below). But I also think I’ve learned a bunch from watching him follow this train of thought, and been impressed with how thoroughly he investigates it. And I don’t think makes sense to write a sequence off because the author has a frame you disagree with. I’m evaluating our intellectual progress at the group level, and I think it’s a pretty key tool in the toolkit for individuals to take their assumptions and run with them and see how far they can get.
I think it’s more useful to flag specific places where you think he’s making a mistake on individual posts, than to make a vague metacriticism.
This particular meetup is over (it’s possible we will host a similar meetup next weekend, although I’d currently lean towards not doing it again).
But, I was interested in replying to your comment about that, and may do so now.
(I say all of this largely agreeing with the thrust of what the post and your (Abram’s) comments are pointing at, but feeling like something about the exact reasoning is off. And it feeling consistently off has been part of why I’ve taken awhile to come around to the reasoning)