I mostly agree with this post. I wrote some notes, trying to understand and extend this idea. Apologies for the length which ended up being longer than your post while having less important content, I was a bit rushed and therefore spent less time making my post succinct than the topic deserved.
Linch
Karma: 3,940
How to Write Fast, Weird, and Well
Middlemen Are Eating the World (And That’s Good, Actually)
What model?
[Linkpost] How to Win Board Games
I think this discussion is very hard to have because people have substantial variation in how good their base intuitions, including social intuitions, are and also substantial variation in how good their rational minds (S2′s) are.
Most good forecasters start off with pretty good base intuitions for probabilities, and forecasting/calibration practice helps refine their intuitions to be even better. Some good forecasters start with terrible probabilistic intuitions and the formal exercises help improve their intuitions. (I suppose it’s also theoretically possible for some good forecasters to start with terrible probabilistic intuitions, continue to have terrible probabilistic intuitions, but forecast well because their formal practice and analysis allows them to override their intuitions. I’ve just never heard about this in practice).
Many mental health conditions come from creating a wrong intuition/prior about the world that rational-adjacent therapy methods like CBT teach you to override. Depression being the most famous/common. When I’m depressed, it’s natural to fixate on specific high-noise signals and automatically assume bad interpretations of the evidence (eg someone didn’t respond to a text because I’ve upset them, or they don’t like me for other reasons). Presumably there’s the inverse problem as well (eg manics have a very positive “rose-colored goggles” prior for everything), but I’ve never heard of this as a serious problem in practice.
In the realm of social intuitions, I think there’s large variation in how good different social intuitions are. Autism in particular is a disorder which has a core manifestation of being bad at parsing social intuitions. I imagine most young autistic people would benefit from learning higher-order rationality and principles about social questions counteracting/overriding their own poor social intuitions (“vibes”), and to a lesser extent training/calibrating their poor social intuitions to become better. On the flip side, neurotypical people that are prone to ideological capture should potentially learn to trust their hearts more, and be less inclined to trust higher order ideologies and “rationality” (which often manifests as rationalizations), in favor of their base intuitions.
But even all of these comments are overly high-order glosses. Autism does not make you immune to rationalization, nor does ideological capture suddenly make you incapable of rationality. Reverse all advice you hear, etc.
My own background is in academic social science and national security, for whatever that’s worth
Why should we assume the AI wants to survive? If it does, then what exactly wants to survive?
...
Why should we assume that the AI has boundless, coherent drives?
Are you familiar with the “realist” school of international relations, and in particular their theoretical underpinnings?
If so, I think it’d be helpful to consider Yudkowsky and Soares’s arguments in that light. In particular, how closely does the international order for emerging superintelligences look like the anarchic international order for realist states? What are the weaknesses of the realist school of analysis, and do they apply to AIs?
I’ll just hop on the bandwagon and say that I’ll be posting my thoughts over at inchpin.substack.com!
I’m doing Inkhaven! For people interested in reading my daily content starting November 1st, consider subscribing to inchpin.substack.com!
Do you want to come up with some other “obvious exceptions” to your “Nobody says X” claim?
Tbh, I find this comment kinda bizarre.
Popular belief analogizes internet arguments to pig-wrestling: “Never wrestle with a pig because you both get dirty and the pig likes it” But does the pig, in fact, like it? I set out to investigate.
Nobody writes a story whose moral is that you should be selfish and ignore the greater good,
This seems obviously false. Ayn Rand comes to mind as the most iconic example, but eg Camus’ The Stranger also had this as a major theme, as does various self-help books. It is also the implicit moral of JJ Thompson’s violinist thought-experiment. My impression from reading summaries is that it’s also a common theme for early 20th century Japanese novels (though I don’t like them so I never read one myself).
[Linkpost] A Field Guide to Writing Styles
I agree if you model people as along some Pareto frontier of perfectly selfish to perfectly (direct) utilitarian, then in no point on that frontier does offsetting ever make sense. However, I think most people have, and endorse, having other moral goals.
For example, a lot of the intuition for offsetting may come from believing you want to be the type of person who internalizes the (large, predictably negative) externalities of your actions, so offsetting comes from your consumption rather than altruism budget.
Though again, I agree that perfect utilitarians, or people aspiring to be perfect utilitarians, should not offset. And this generalizes also to people whose idealized behavior is best described as a linear combination of perfectly utilitarian and perfectly selfish.
I think this post underestimates the value of practicing and thinking in classic style, even if you chose to ultimately discard it, or not write serious posts in that style. Because writing in classic style is so unnatural to most LessWrong dwellers, forcing yourself to write in that way, unironically, and inhabiting the style in its own lights, and especially doing it in a way that doesn’t leave you unsatisfied in the end, is a great way to grow and improve as a writer, and understand the strengths and weaknesses of your own style of writing.
I think most people shouldn’t write in classic style, for various reasons. But I have a different take here. I think writing in classic style is just very hard for most people, for a number of subtle reasons. A central tenet of classic style is presentation: the writing should look smooth and effortless. But this effortlessness is almost always a mirage, like an Instagram model who spends three hours in front of a mirror to apply the “just woke up”, au naturel, “no makeup makeup” look. Of all the (mostly) internet writers I read, only two writers jump out to me as writing in mostly classic style: Paul Graham and Ted Chiang. I don’t think it’s coincidence that they both are very unprolific, and both talk about how hard it is to write well, and how many edits they go through.
Below is a short coda I wrote in classic style, for a recent article of mine.
Intellectual jokes, at their core, are jokes that teach you new ideas, or help you reconceive existing ideas in a new way.
My favorite forms of intellectual jokes/humor work on multiple levels: They’re accessible to those who just get the surface joke but rewards deeper knowledge with additional layers of meaning. In some of the best examples, the connection to insight is itself subtle, and not highlighted by a direct reference to the relevant academic fields.
There are two failures of attempts to do intellectual humor. They can fail to be intellectual, or they can fail to be funny. Of frequently cited attempts to do “intellectual” humor that fail to be intellectual, there are again two common forms: 1) they are about intellectuals as people, rather than about ideas, or 2) They’re about jargon, not ideas.
In both cases, the joke isn’t intellectual humor so much as “smart people jokes”: the humor rests on stereotypes, in-group solidarity, and the feeling of smartness that you get when you get a joke, but the joke does not actually teach you about new ideas, or help you reconceive of existing ideas in a new way.
Two examples come to mind:
Q: How do you tell if a mathematician is extroverted2?
A: When he’s talking to you, he stares at your shoes!
And
Q: What’s purple and commutes?
A: An Abelian3 grape.
If you were in my undergrad abstract algebra classes, the above jokes were the shit. For 20 year old math majors, they were hilarious. Nonetheless, they are not, by any reasonable definition of the term, intellectual.
Of course, a more common failure mode is that the jokes simply fail to be funny. I will not offer a treatise into what makes a joke funny. All unfunny jokes are alike in their unfunniness, but each funny joke is funny in its own way.
I’m currently drafting a post on different mature writing by first inhabiting the respective styles and then evaluating the pros and cons, especially in the context of internet writing. It’s a pretty hard post to write, and I suspect it’d be a lot less popular in the end than the Chiang review or many LessWrong posts, but I hope it’d be more helpful.
I’m glad you enjoyed it!
It Never Worked Before: Nine Intellectual Jokes
The main reason I disagree with both this comment and the OP is that you both have the underlying assumption that we are in a nadir (local nadir?) of connectedness-with-reality, whereas from my read of history I see no evidence of this, and indeed plenty of evidence against.
People used to be confused about all sorts of things, including, but not limited to, the supernatural, the causes of disease, causality itself, the capabilities of women, whether children can have conscious experiences, and so forth.
I think we’ve gotten more reasonable about almost everything, with a few minor exceptions that people seem to like highlighting (I assume in part because they’re so rare).
The past is a foreign place, and mostly not a pleasant one.
In both programming and mathematics, there’s a sense that only 3 numbers need no justification (0,1, infinity). Everything else is messier.
Unfortunately something similar is true for arguments as well. This creates a problem.
Much of the time, you want to argue that people underrate X (or overrate X). Or that people should be more Y (or less Y).
For example, people might underrate human rationality. Or overrate credentials. Or underrate near-term AI risks. Or overrate vegan food. Or underrate the case for moral realism. Or overrate Palestine’s claims. Or underrate Kendrick Lamar. (These are all real discussions I’ve had).
Much of the time, if a writer thinks their readers are underrating X, they’ll make an argument in favor of X. (Sounds obvious, I know).
But X and Y are usually not precise things that you can measure, never mind ascertain a specific value to it.
So if a writer argues for X, usually they don’t have a good sense of what value the reader assigns X (in part because of a lack of good statistics, and in part because a specific reader is a specific person with their own idiosyncratic views). Nor does a writer have a precise sense of what the optimal value of X ought to be, just that it’s higher (or lower) than what others think.
This creates major problems for both communication and clarity of thought!
One solution of course is to be an extremist. But this is a bad solution unless you actually think maximal (or minimal) X is good.
Sometimes either the structure of reality, or the structure of our disagreements, create natural mid-points while we can explicate their disagreements. For example, in my debate with BB, a natural midpoint is (we believe[1]) whether bees have net positive or net negative welfare. “0” is a natural midpoint. In my second post on the “rising premium of life”, I can naturally contrast my preferred hypothesis (premium of life rising) against the null hypothesis that the premium of life is mostly unchanged, or against the alternate hypothesis that it’s falling.
But reality often doesn’t give us such shortcuts! What are natural midpoints to argue for in terms of appropriate levels of credentialism? Or appropriate faith in human rationality? Or how much we should like Kendrick Lamar?
I don’t want to give people the illusion of an answer here, just presenting the problem as-is.
[1] This is disputed, see here.
I have a number of notes on questions I’m interested in. Would love to see some feedback and further thoughts from people!