Beware of Well-Written Posts
Beware of when a post is so well-written that you can’t put it down. Be wary of posts that are more visually attractive than average. Beware of posts that make you laugh out loud.
Why? Because all of this is orthogonal to whether the post’s argument is actually true and runs counter to the mission of rationality.
The Company Man
Yesterday, I read Tomás B.’s The Company Man. I was captivated on my first read, as I’m sure many others were. Had I been interrupted before I could scroll into the comments, I would have walked away thinking little more than “what a great post”.
But I did get to the comments. The first one was kyleherndon’s:
I did not enjoy this. I did not feel like I got anything out of reading this. However, this got curated and >500 karma… The best theory I can scrounge together is that this is “relatable” in some way to people in SF, like it conveys a vibe they are feeling? … I didn’t feel like I learned anything as a result...
This forced me to ask myself: what did I actually learn from The Company Man? Science fiction, at its best, makes you think in a new and productive way. Did the story provide any meaningful new scenarios or possibilities that I could factor into my view of the future? If not, did it teach me anything new about the present that I could trust as accurate and representative? To be honest, not really.
Undoubtedly, Tomás is an enormously talented writer, and there are many moments in The Company Man that reveal the eye and the pen of a literary genius. But what he wields is a dangerous power.
True Affect
“There is a clever man called Socrates who has theories about the heavens and has investigated everything below the earth, and can make the weaker argument defeat the stronger.” (Apology, Section A, 18c)
Some people are good at telling stories. Some people aren’t. If both present the same evidence with the same general conclusions, why would we believe the former more?
Well, one might say, isn’t an inconsistent, weak story negatively correlated with a weak position? I struggle to imagine where this is actually relevant, because if a bad belief has a bad story, it will never survive. In a 2x2 matrix of strong/weak position and good/bad story, all the real battles are fought between the other three quadrants.
Bohr and Heisenberg had math that fit with the empirical evidence. However, their story about quantum physics violated common sense. Einstein had a more compelling narrative: God doesn’t play dice with the universe, there is no spooky action at a distance, the moon doesn’t disappear when you aren’t looking at it. However, Einstein was wrong.
In the TED talk “Tales of Passion,” novelist Isabelle Allende relates one of her favorite sayings:
Question: What is truer than truth?
Answer: The story.
This is not a paradox. There is a true affect, a feeling of true-ness in the brain that a good story evokes, which is independent of whether the story is actually true in the mundane sense. It’s no coincidence that Allende is one of the foremost writers of the magical realism genre; her specialty is evoking this true affect in ways that are plainly not true to reality.
My impression of the middle ages is that writers didn’t understand this true affect at all. They would write reports inflating the number of attendees at the Council of Clermont[1] or epic poems about figures like Charlemagne full of invented details, and then they would claim they were accurate without blinking an eye. I don’t think this can be ascribed to Machiavellian consequentialism. Why would writers impelled by their own religious fervor knowingly and intentionally violate one of the Ten Commandments?
The true affect is highly subjective. Whether a story produces true affect is determined by its alignment with someone’s deep internal narratives and archetypes, which often transcend the level of personal desires and fears. The biases these archetypes create feels more substantial, and often spiritual, than the surface-level, selfish “it would be uncomfortable for me to take action on this” feeling that many rationalists cite.
But Stories Are Good for Life
In a chain of replies to kyleherndon’s comment on The Company Man, Ben Pace writes:
...it allows me to recognize these archetypes better in reality when I see them. I think these kinds of people do exist in some form and emphasizing these traits of theirs is capturing something about the world, as well as the dynamics that form between them and others, and paying attention to these archetypes helps me build accurate models of them and predict people’s behavior better.
I think this is mostly wrong and potentially very dangerous. None of the characters are meaningfully useful models for understanding, say, a Sam Altman or Dario Amodei.
But also, Ben is right. We can’t just be rid of stories and archetypes, and live in a pure data-vacuum. On an practical level, they’re the compressed format we use to survive in a world of impossibly complex people and events. They’re as embedded in the human mind as the notion of causation and the perception that there are distinct “things” separated by space (rather than just a sea of energy). I think there’s only so much we can do to stop thinking in terms of stories or archetypes unless we just stop thinking completely.
Because of this, it might be more tractable to cultivate awareness of the stories that influence us and be able to openly admit how they influence our priors, rather than trying to erase them and stop reading new fiction. For example, last year I watched the anime Pluto, which has a pretty strong thesis about how AI capability and the capacity to desire to kill someone are inseparable. It took me a while to realize how large of an effect it had on my views on personas.
On top of that, stories and archetypes are healthy. They give meaning to life. They’re fun to read and to tell. Having a fiction tag is part of what makes LessWrong human, in an unavoidably mushy and sentimental way.
When you do see a line that makes you laugh out loud, or that strikes you as intensely beautiful, don’t hesitate to add a reaction! This will also make it clearer to future you and to other people what is happening. I don’t know if this is what the designers of the feature intended, but these are great for identifying spots that are heavy in pathos.
Don’t stop reading good writing, or trying to write well. Just be aware of what you’re doing. And also, if AI replaces all human writing, there might be a silver lining to it.
- ^
The Council of Clermont was where Pope Urban II pronounced the First Crusade.
“Ferdinand Chalandon, Histoire de la Première Croisade jusqu’à l’élection de Godefroi de Bouillon (Paris, 1925), 75, states that the number of higher church officials present at the Council, as given by various accounts, ranged from 190 to 463. With a reminder that the number attending different sessions varied, Chalandon accepts as most nearly correct the number given by Urban in a Bull concerned with the Primacy at Lyons, which is the smallest, because it was an official statement, and because afterwards reporters of the Council were inclined to stress its importance by raising the figure.” (From The Chronicle of Fulcher of Chartres and Other Source Materials, second edition, edited by Edward Peters, page 50, footnote 6.)
Fulcher of Chartres, himself a cleric who was present at the council, gives the number 310 (with no qualification for uncertainty).
my job is to understand the world. explanations that are simple—by whatever crooked metric evolution has inspired—are more aesthetic, but also more useful. they are more readily recalled, and more easily applied. moreover, when queried, they are not content to issue a dry response; they rather sing with detail.
“good story” means this: “early in the mind’s inductive bias”.
one must not discard verisimilitude. and of course a story is not itself evidence—one should take care not to treat priors as new information. but i don’t find it appropriate to discount “good story” as a useful measure, when organizing thoughts, or communicating that organization.
I think the practical upshot of these considerations is to read more & give more credence to bad or dry writers.
Tyler Cowen’s ted talk on the same subject is also relevant here.
Most likely Tomas never intended to describe in detail what caused Altman/Amodei to act like the leaders of the Company. What he likely intended to do was to describe a median worker there who for some reason is taking part in creating the ASI despite the fact that he isn’t sure that p(aligning the ASI) is high enough (or, in Esther’s case, that Δp(not doom) is the biggest for Esther’s position).
Additionally, I suspect that your advice “When you do see a line that makes you laugh out loud, or that strikes you as intensely beautiful, don’t hesitate to add a reaction!” contradicts the broader LessWrong culture of leaving reacts versus comments.
I think the protagonist is one of the most compelling characters. However, he is almost entirely characterized by internal dialog. I would be cautious about comparing a real person to the protagonist without a deep personal relationship to that person and insight into how they think.
I was thinking more strongly of Arden Voss, who is delightfully characterized but essentially flat and adds no insight into the psychology of the powerful tech figure. “They’re spiritually off the deep end and can’t be reasoned with” is not a novel or useful model.
You could argue that this is too much to ask of satire.