Demon Threads

tldr: a De­mon Thread is a dis­cus­sion where ev­ery­thing is sub­tly warp­ing to­wards ag­gres­sion and con­fu­sion (i.e. as if peo­ple are un­der de­monic in­fluence), even if peo­ple are well in­ten­tioned and on the same ‘side.’ You can see a de­mon thread com­ing in ad­vance, but it’s still hard to do any­thing about.

(“Flame Wars” are similar but I felt the con­no­ta­tion was more like “ev­ery­thing has already gone to hell, and peo­ple aren’t even pre­tend­ing to be on the same side”)


I kept want­ing to refer­ence this post when dis­cussing in­ter­net dis­cus­sion policy, and I kept for­get­ting that no­body has writ­ten it yet. So here it is.

Suggested Back­ground Read­ing:

If some­one in the fu­ture linked you to this post, it’s prob­a­bly be­cause a gi­ant sprawl­ing mess of an­gry, con­fused com­ments is hap­pen­ing—or is about to hap­pen—and it’s go­ing to waste a lot of time, make peo­ple up­set, and prob­a­bly less likely to listen to each other about what­ever the con­ver­sa­tion os­ten­si­bly is about.

I have some ideas on what to do in­stead, which I dis­cuss in this fol­lowup post.

But for now, this post is meant to open a dis­cus­sion, ex­plore the me­chan­ics of how de­mon threads work, and then in the com­ments brain­storm solu­tions about how to han­dle them.

Wrong On the Internet

I find “Some­one Is Wrong On the In­ter­net” to be a weird, spe­cific feel­ing.

It’s dis­tinct from some­one be­ing fac­tu­ally wrong—peo­ple can be wrong, point it out, and hash out their dis­agree­ments with­out a prob­lem. But a com­mon pat­tern I’ve wit­nessed (and ex­pe­rienced) is to no­tice some­one be­ing wrong in a way that feels dis­tinctly bad, like if you don’t cor­rect them, some­thing pre­cious will get tram­pled over.

This is when peo­ple seem most prone to jump into the com­ments, and it’s when I think peo­ple should be most care­ful.

Some­times there ac­tu­ally is an im­por­tant thing at stake.

There usu­ally isn’t.

It of­ten feels like there is, be­cause our so­cial in­tu­itions were honed for tribes of a hun­dred or two, in­stead of a world of 7 billion. We live in a differ­ent world now. If you ac­tu­ally want to have an im­pact on so­ciety, yel­ling at each other on the in­ter­net is al­most cer­tainly not the best way to do so.

When there ac­tu­ally is some­thing im­por­tant at stake, I think there are usu­ally bet­ter plans than “get into a gi­ant in­ter­net ar­gu­ment.” Think about what your goals are. De­vise a plan that ac­tu­ally seems like it might help.

Differ­ent situ­a­tions call for differ­ent plans. For now, I want to talk about the com­mon anti-pat­tern that of­ten hap­pens in­stead.

De­mon Threads are ex­plo­sive, frus­trat­ing, many-ten­ta­cled con­ver­sa­tions that draw peo­ple in re­gard­less of how im­por­tant they are. They come in two forms:

  • Benign De­mon Threads are mostly time wast­ing. No­body gets that an­gry, it’s just a frus­trated mess of “you’re wrong” “no you’re wrong” and then peo­ple spend loads of digi­tal ink ar­gu­ing about some­thing that doesn’t mat­ter much.

  • Mal­ig­nant De­mon Threads feed upon emo­tions of defen­sive­ness, anger, tribal af­fili­a­tion and righ­teous­ness—and in­flame those emo­tions, draw­ing more peo­ple into the fire.

(A ma­lig­nant de­mon thread is cousin to the flame war—peo­ple hurl­ing pure in­sults at each other. What makes a ma­lig­nant de­mon thread in­sid­eous is the way it can warp dis­cus­sion even among peo­ple who are earnestly try­ing to com­mu­ni­cate, seek truth and solve prob­lems)

If you find your­self in a ma­lig­nant de­mon thread, I think it’s likely you are not only not helping, but are ac­tu­ally hurt­ing your cause.

The De­mon Seed

How to write so that peo­ple will com­ment [dis­claimer: not nec­es­sar­ily good ad­vice]

  1. Be wrong

  2. Be controversial

  3. Write things peo­ple feel qual­ified to have opinions on.

  4. In­voke so­cial re­al­ity.

- Writ­ing That Pro­vokes Comments

In the com­ments on YouTube, or the worst parts of Face­book or tum­blr, de­mon threads are not sur­pris­ing. Peo­ple write com­ments that in­flame ide­olog­i­cal war­fare all the time. In­ter­nets be in­ter­nets. Peo­ple be peo­ple. What can you do?

The sur­pris­ing thing is how this works in places where ev­ery­one should re­ally know bet­ter. The pow­ers of demons are de­vi­ous and sub­tle.

There’s an ex­per­i­ment — in­sert obli­ga­tory repli­ca­tion crisis dis­claimer — where one par­ti­ci­pant is told to gen­tly poke an­other par­ti­ci­pant. The sec­ond par­ti­ci­pant is told to poke the first par­ti­ci­pant the same amount the first per­son poked them.

It turns out peo­ple tend to poke back slightly harder than they were first poked.

Re­peat.

A few iter­a­tions later, they are strik­ing each other re­ally hard.

I think some­thing like this is at work in the me­chan­ics of de­mon threads.

The De­mon Seed is the first com­ment in what will soon be­come a de­mon thread. It might look pretty in­nocu­ous. Maybe it feels slightly rude, or slightly oblivi­ous, or push­ing a con­ver­sa­tion that should be about con­crete em­piri­cal facts slightly to­wards be­ing about so­cial con­sen­sus (or, vice versa?).

It feels 1% out­side the bound of a rea­son­able com­ment.

And then some­one wa­ters the de­mon seed. They don’t want to let the point stand, so they re­spond with what seems like a fair re­buke.

Maybe they’re self-aware that they’re feel­ing an­noyed, so they in­ten­tion­ally dial back the ag­gres­sion of their re­sponse. “Ah, ac­tu­ally this prob­a­bly comes across as too hos­tile, so I’ll tweak my word­ing to re­duce hos­tility by 4%.” But, ac­tu­ally, the words were 6% more hos­tile than they thought, and now they’ve es­ca­lated 2%.

Re­peat 2-3 times. The de­mon seed is wa­tered. La­tent un­der­ly­ing dis­agree­ments about how to think prop­erly or ideal so­cial norms… or which coal­i­tions should be high­est sta­tus… or pure, sim­ple you’re in­sult­ing me and I’m an­gry

They have fes­tered and now they are ready to ex­plode.

Then some­one makes a com­ment that pushes things over the edge, and a de­mon thread is born.

(It is, of course, pos­si­ble to skip steps 1-4 and just write a blatantly rude, in­cen­di­ary com­ment. I’m try­ing to de­scribe how this hap­pens even when ev­ery­one is well in­ten­tioned and mostly trusts each other)

From there, if you’re lucky it’s con­tained to two peo­ple. But of­ten, well mean­ing by­stan­ders will wan­der by and think “Ah! Peo­ple are be­ing wrong on the in­ter­net! Wrong about things I am qual­ified to have opinions on! I can help!”

And it grows.

Then peo­ple start link­ing it from el­se­where, or FB al­gorithms start shar­ing it be­cause peo­ple are com­ment­ing so the thread must be im­por­tant.

It grows fur­ther.

And it con­sumes days of peo­ple’s at­ten­tion and emo­tional en­ergy. More im­por­tantly, it of­ten en­trenches peo­ple’s cur­rent opinions, and it burns peo­ple’s good will that they might have been will­ing to spend on hon­est, co­op­er­a­tive dis­course.

Why De­mon Threads are Bad

I think de­mon threads are not just a bad plan—I think they are of­ten net nega­tive plan.

The rea­son is best ex­pressed in Conor More­ton’s Idea In­noc­u­la­tion and In­fer­en­tial Dis­tance. [Edit: the full ar­ti­cle is no longer available]

That post is im­por­tant enough that I think you should go read it if you haven’t. (Conor be­gins by say­ing the post is shorter than usual… “but I claim it’s su­per im­por­tant. If you’d nor­mally spend 7min read­ing my post and it takes you 3min to read it this time, con­sider read­ing it twice, maybe?“).

Real­is­ti­cally though you prob­a­bly won’t stop read­ing this post, so I’ll paraquote it here:

In­fer­en­tial dis­tance is the gap be­tween [your hy­pothe­ses and world model], and [my hy­pothe­ses and world model]. It’s just how far out we have to reach to one an­other in or­der to un­der­stand each other.
If you share poli­ti­cal and in­tel­lec­tual and cul­tural foun­da­tions, it’s (rel­a­tively) easy. If you have com­pletely differ­ent val­ues and as­sump­tions, (say you get dropped off in the 15th cen­tury and need to ar­gue with Christo­pher Colum­bus) it may be nigh im­pos­si­ble.
It’s right in the name—in­fer­en­tial dis­tance. It’s not about the “what” so much as it is about the “how”—how you in­fer new con­clu­sions from a given set of in­for­ma­tion. When there’s a large in­fer­en­tial dis­tance be­tween you and some­one else, you don’t just dis­agree on the ob­ject level, you also of­ten dis­agree about what counts as ev­i­dence, what counts as logic, and what counts as self-ev­i­dent truth.
What makes this re­ally bad is idea in­oc­u­la­tion.
When a per­son is ex­posed to a weak, badly-ar­gued, or un­canny-valley ver­sion of an idea, they af­ter­wards are in­oc­u­lated against stronger, bet­ter ver­sions of that idea. The anal­ogy to vac­cines is ex­tremely apt—your brain is at­tempt­ing to con­serve en­ergy and dis­till pat­terns of in­fer­ence, and once it gets the shape of an idea and at­taches the flag “bul­lshit” to it, it’s ever af­ter go­ing to lean to­ward at­tach­ing that same flag to any idea with a similar shape.
When you com­bine idea in­oc­u­la­tion with in­fer­en­tial dis­tance, you get a recipe for dis­aster—if your first at­tempt to bridge the gap fails, your sec­ond at­tempt will also have to over­come the per­son’s rapidly de­vel­op­ing re­sis­tance.
You might think that each suc­ces­sive at­tempt will bring you closer to the day that you fi­nally es­tab­lish com­mon ground and start com­mu­ni­cat­ing, but alas—of­ten, each at­tempt is just in­creas­ing their re­sis­tance to the core con­cept, as they build up a library of all the times they saw some­thing like this defeated, proven wrong, made to look silly and naive.

A de­mon thread is a recipe for bad at­tempts at com­mu­ni­cat­ing. Lots of peo­ple are yel­ling at once. Their defenses are raised. There’s a sense that if you give in, you or your peo­ple look like losers or villains.

This’ll make peo­ple worse at listen­ing and com­mu­ni­cat­ing.

Why the In­ter­net Worse

“De­mon threads” can hap­pen in per­son, but they’re worse on­line.

One ob­vi­ous rea­son is that the in­ter­net is more anony­mous. This re­duces con­se­quences to the per­son writ­ing a com­ment, and makes the tar­get of the com­ment eas­ier to round off to a bad stereo­type or an ab­stract rep­re­sen­ta­tion of The Enemy.

Other things peo­ple do:

A. Peo­ple end up writ­ing long winded monologues with­out any­one in­ter­rupt­ing them to cor­rect ba­sic, wrong as­sump­tions.

i.e. “you’re just wrong be­cause you think X, there­fore… [com­pli­cated ar­gu­ment]“, with­out pro­vid­ing op­por­tu­nity for some­one to re­spond “no I don’t ac­tu­ally think X at all”. And then, hav­ing writ­ten out [com­pli­cated ar­gu­ment] you’re already in­vested in it, de­spite it be­ing built on faulty premises.

B. Lots of peo­ple are writ­ing. Espe­cially as the de­mon thread grows. After 24 hours of its ex­is­tence, the thread will have so much con­tent it’s a huge in­vest­ment to ac­tu­ally read ev­ery­thing that’s been said.

C. The com­ments aren’t nec­es­sar­ily dis­played in or­der. Or, if they are, peo­ple aren’t read­ing them in or­der, they’re read­ing what­ever it’s largest or most in­ter­est­ing.

D. The in­ter­net is full of lots of other con­tent com­pet­ing for at­ten­tion.

This all means that:

E. Peo­ple are skim­ming. This is most true when lots of peo­ple are writ­ing lengthy monologues, but even when the thread first be­gins, peo­ple’s eyes may be bounc­ing around to differ­ent tabs or differ­ent threads within a page so they aren’t even read­ing what’s be­ing said, not with the in­ten­tion­al­ity and em­pa­thy they would when con­fronted with a real per­son in front of them.

And they might first be read­ing the most ex­plo­sive, re­cent parts of a thread rather than piec­ing to­gether the ac­tual or­der of es­ca­la­tion, which may make peo­ple look less rea­son­able than they were.

This all adds up to gi­ant threads be­ing a uniquely bad way to re­solve nu­anced, emo­tion­ally fraught is­sues.

Con­tain­ment?

De­mon threads are like wild­fires. Maybe you can put them out, with co­or­di­nated effort. You can also try to ig­nore them and hope they burn them­selves out.

But if you wanted to ac­tu­ally stop it, the best bet is to do so is be­fore they’re erupted in the first place.

I’ve de­vel­oped a sense of what seeds look like. I’ll see a com­ment, think “god, this is go­ing to be­come a de­mon thread in like two hours”, and then sure enough, two hours later peo­ple are yel­ling at each other and ev­ery­thing is awful and ev­ery­one in­volved seems re­ally sure that they are helping some­how.

Some flags that a de­mon thread might be about to hap­pen:

Flags Re­gard­ing: Ten­sion and La­tent Hostility

  • When you look a com­ment and want to re­spond, you feel a visceral sense of “you’re wrong”, or “ugh, those peo­ple [from group that an­noy me]” or “im­por­tant prin­ci­ple must be defended!” or “I am liter­ally un­der at­tack.”

  • You feel phys­iolog­i­cal defen­sive­ness or anger—you no­tice the hairs on the back of your neck or arms stand­ing on end, or a tight­ness in your chest, or how­ever those emo­tions man­i­fest in your body.

  • Peo­ple in the thread seem to be talk­ing past each other.

  • For what­ever rea­son, ten­sions seem to be es­ca­lat­ing.

Flags Re­gard­ing: So­cial Stakes

  • The ar­gu­ment seems like it’s about who should be high or low sta­tus, which peo­ple or groups are vir­tu­ous and which are not, etc.

  • The ar­gu­ment is about so­cial norms (in par­tic­u­lar if at stake is whether some peo­ple will end up feel­ing un­wel­come or un­com­fortable in a given com­mu­nity/​space that is im­por­tant to them—this is ex­tremely threat­en­ing)

  • More gen­er­ally—the ar­gu­ment touches in some way on so­cial re­al­ity, in ways that might have ram­ifi­ca­tions be­yond the im­me­di­ate con­ver­sa­tion (or that peo­ple are afraid might have such ram­ifi­ca­tions).

If some of the above seem true (in par­tic­u­lar, at least one of the first group and at least one of the sec­ond), then I think it’s worth step­ping back and be­ing very care­ful about how you en­gage, even if no com­ment seems es­pe­cially bad yet.

Po­ten­tial Solutions

The first line of defense is to no­tice what’s hap­pen­ing—rec­og­nize if you’re feel­ing defen­sive or an­gry or talk­ing past each other. Brienne’s Notic­ing Se­quence is pretty good for this (as well as her par­tic­u­lar posts on train­ing the skills of Em­pa­thy and han­dling Defen­sive­ness—these may not work for ev­ery­one but I found the un­der­ly­ing thought pro­cess use­ful).

But while notic­ing is nec­es­sary, it’s not suffi­cient.

Rather than list my first guesses here, I’ll be dis­cussing them in the com­ments and fol­low­ing this up with a “best-seem­ing of the po­ten­tial solu­tions” post.

Mean­while, some fac­tors to con­sider as you de­cide what to do:

  • How are you in­volved?

    • Are you one of the peo­ple ini­tially ar­gu­ing, or a by­stan­der?

    • How much do you nor­mally trust the peo­ple in­volved?

    • Is it pos­si­ble to take the con­ver­sa­tion pri­vate?

  • Are we on the de­mon seed or de­mon thread stage? Is there com­mon knowl­edge about ei­ther?

  • What are the ac­tual stakes?

  • What are the mod­er­a­tion tools available to you?

  • Are you in a venue where you have the abil­ity to shape con­ver­sa­tional norms?

    • Do you di­rectly con­trol them (i.e. per­sonal blog or feed?)

    • Does any­one have di­rect own­er­ship of the venue? (ei­ther tech­ni­cally, or cul­turally)

    • Is there any­thing you can do unilat­er­ally to make the con­ver­sa­tion bet­ter, or will it re­quire help from oth­ers?

  • Are you build­ing a site where you get to de­velop en­tire new tools to deal with this class of prob­lem?

With that in mind…

In what­ever venues you most find your­self de­mon-thread-prone, what sort of plans can you ac­tu­ally think of that might ac­tu­ally help?


Note: I have since writ­ten a fol­lowup post with a work­ing ex­am­ple of what I think peo­ple should usu­ally do in­stead of de­mon threads.