I did not enjoy this. I did not feel like I got anything out of reading this.
However, this got curated and >500 karma, so I feel like I must be missing something. Can anyone inform me: Why did other people enjoy this? The best theory I can scrounge together is that this is “relatable” in some way to people in SF, like it conveys a vibe they are feeling?
If the goal is to evoke a sensation of disgust with the characters, then I guess you’ve succeeded for me. I already knew I would not like the sort of person described in this story, though, so I didn’t feel like I learned anything as a result, but I could see how something like that could be useful for others. I essentially just felt disgust the entire time reading this.
It fleshed out a lot of archetypes (e.g. “the person on the inside who thinks they’ll be in the right place at the right time”, “the person who feels they have had a meditation/emotional insight and this has gotten them over the intellectual hump of not building killing machines”, “the person who believes the AI may kill everyone but did an EV calculation and decided to make lots of money anyway” etc) and ways-the-future-could-go in way more detail than I’ve seen before.
There are many ideas in here that I’ve heard said offhand but never really dived into, and there’s something very informative and satisfying about seeing them painted in detail. Similar to the difference between a one-sentence description of a painting, and the actual painting.
Plus lots of fun and dramatic detail to how it’s all woven together.
These are some of the things that occur to me in answer to your question.
Yes, I felt much disgust and dismay reading it. I also feel that way about many parts of the real world!
One way to say it is that I expect to still remember the story in five years time, and I appreciate stories with that property. I also expect to get different things out of it every couple years if I reread it, which is a measure of “writing quality” in my book.
But to try to be more specific:
I just wrote, while trying to talk about something unrelated to this story: “Many people today, especially high-level people in the bay area, seem to me sort of… abstract, dissociated, cobbled-together-on-purpose-via-conscious-understanding-of-algorithms compared to the people in older books and movies. I’d like more of the normal/historical human thing.” This… is a perception that’s been with me for awhile, but the story helped it click into slightly-better focus.
Also (not that central to what I liked about the story, but an example): at least two different people who’re in some way pushing AI development have told me privately about weird choices they made when hitting puberty, about how to set up their insides. I’m curious how common things like this are, since probably most people wouldn’t tell me, and about how they work. This is the first public discussion of such that I’ve personally bumped into, and I appreciate it.
I agree reading the story was some sorts of unpleasant, and it left me feeling… perturbed for a couple days. But it helped that I felt like the author had sound moral bearings, which made it less disorienting for me that the characters didn’t?
I enjoyed a few things about it, but I think what brought it all the way from “oof, that was well written but I’m not sure I enjoyed the experience” to having some fun reading and mulling on it is that, as a writer, I’ve spent a long time trying to build out my repertoire for writing actually “bad” antagonist characters. (I think this improves the conflict in stories – when I succeed, it clearly increases beta reader engagement even if that engagement consists entirely of “WOW I HATE THEM SO MUCH” – and also, like, writing any characters at all who aren’t unbearably earnest Hufflepuffs was a challenge for me).
This story was a very vivid and memorable depiction of a way a person could be shaped that definitely isn’t anywhere near my current character repertoire, but feels self-consistent enough that I could imagine booting up my own version of a similar guy and writing him “in character” for a whole story without running into too many blank spots where I can’t model him at all. I’m on the lookout for more unlikeable antagonist archetypes to introduce in my current fiction project, so it’s good timing. It also felt...deep? Rich? Like I could dig into this imaginary person’s psychology and find more there (part of me is going “wow, who hurt you? what backstory can I give you so I feel a little sympathetic that you’re like this”, because I can write hateable antagonists a lot better if I manage to feel a little sympathetic to them).
...Apart from seeing it as inspiration for my own writing: it does feel like it captures a piece of reality and pins it down where I can look at it, and I appreciate that even when I don’t enjoy looking. (It’s plausible it might help me model real life people who aren’t earnest Hufflepuffs?) It’s speckled with in-jokes and references that entertained me a bit. The prose and metaphors were also, IMO, really well done and vivid, including some that made me laugh out loud. (In general I think making a character’s internal monologue funny is a writing strategy that makes them more engaging/readable even if they’re not likable, so I’m taking notes on that too.)
My girlfriend (who is not at all SF-brained and typically doesn’t read LessWrong unless I send her something) really enjoyed it and felt it was great because it helped her empathize with people in AI safety / LessWrong (makes them feel more human). She felt it was well-written, enjoyably written. It was something she could read without it being a task.
Do you think you would have reacted differently and more positively, if the following text had been inserted at the top?
”Author’s note: This short story is a sort of hate letter to the big AGI companies, and to various (but not all!) of the people working there. It’s a caricature/parody; it is to the AGI companies what Dr. Strangelove was to institutions like RAND and the US Strategic Command. It is uncharitable and exaggerates—but, like Dr. Strangelove, I think it makes some pretty valid points and expresses some pretty valid emotions. I hope that some of the people building AGI are humble enough to read this story and take it as a cautionary tale.”
Had this text been inserted at the top, I probably would have been excited enough about the story to send it to some of my former colleagues at OpenAI and Anthropic.
With such a disclaimer, I think I would have been less confused but still not positive on the text. Maybe I’d be less negative? Neutral, even? A lack of disclaimer is not really the core of it for me though.
I have strange tastes in fiction, I think, and normally after reading something that anti-resonated with me so much, I’d just stop reading. But, I read to the end because a bunch of upvotes signaled to me that I might otherwise miss out on something later in the text.
I live in the Bay Area and work in tech and follow AI news. For me, I already knew that there are traits I don’t like in others, and had approximate models of their frequency in and correlations to the AI scene. Each character struck me as plausible and not really meaningfully more concrete to me. Concrete in the sense of “Oh, this behavior is warning sign that this is a morally bankrupt person I should stay away from” or “I might accidentally be falling into this pattern if I find this character relatable in some way.” I can definitely see this being useful to others who do not live in the bay area/work in tech/do not follow AI news, but I mostly expected people on lesswrong to match enough of that description that they wouldn’t find themselves learning from this “concretely describing a vibe or something.” And indeed, some of the people liked it for reasons other than that, which I failed to predict!
A big part of it for me is that I just cannot stand the first person narration of this main character. An example line, pulled at random: “But regardless, she makes very good coffee and is very nice and beautiful which sometimes amounts to the same thing.” I find this thoroughly jarring. I read it as the character’s internal monologue, but this is just so thoroughly outside the domain of what would go in my own internal monologue it feels… rude? an empty attempt at caricature? a failure to understand what internal thoughts look like or otherwise an indication of a mind horribly alien to my own such that it bears no resemblance to real humans? I do find the characters unappealing, but this just seemed unnecessary to me.
this is just so thoroughly outside the domain of what would go in my own internal monologue it feels… rude? an empty attempt at caricature? a failure to understand what internal thoughts look like or otherwise an indication of a mind horribly alien to my own such that it bears no resemblance to real humans?
Interesting! Maybe there’s an experiential crux in there (so to speak)? My reader experience of this is that the first-person inner monologue is indeed very different from mine, but I perceive that as increasing the immersion and helping frame the story. To the extent that there’s a group of humans it saliently bears little resemblance to, I might think of that group as something like “humans who are psychologically ‘healthy’ in a certain way which varies across a wide spectrum, where social spheres with concentrated power may disproportionately attract people who are low on that spectrum”. I’m deliberately putting the main adjective in scare quotes there because in my fuzzy mental model, there isn’t really a clear delimiter between treating that trait cluster as a health indicator and treating it as intersubjective values dissonance; it feels consonant with but not directly targeting dark triad traits. But also, I’m not sure if you’re referring to the same thing I am or if it’s some other feature of the first-person description that bothers you.
FWIW, culturally speaking, I’ve been socially adjacent to Bay Area / SV-startup / “mainstream big tech” culture via other people, but not really been immersed in it—I’ve splashed around in the shallow part of that pool long ago, but for “try not to build the Torment Nexus” reasons (plus other unrelated stuff) I historically bounced away a lot as well and wound up in a sort of limbo. So that colors my impression quite a bit.
I personally don’t believe “writing quality” can be divorced from content, and if you shove a bunch of words together in a “masterful way” but don’t say anything, I don’t want to read that and therefore would call that bad writing.
Upon slightly more reflection, I think I can appreciate on an intellectual level the quality of the writing if the goal is to evoke disgust. People (myself included, on occasion) partake in Horror content, which is also traditionally a negative emotion. I haven’t heard of a Revulsion genre before, so I didn’t really consider that this might be a thing people pursue, but I would still be a little surprised if that was what most people got out of the post.
I would also be surprised if the source of all the upvotes was just that it is “”“high quality””” writing. I usually find LessWrong to be more focused on content, and I still want to know what other people see in this post.
As an update, I’ve now gotten a bunch of useful responses of why people liked this! I was worried people would interpret me as being needlessly negative, but it seems mostly not so. I think I’m still far from enjoying this myself, but I do think I’ve learned a good amount from the answers I got, and I got some interesting things to think about.
I’m a big fan of dialoguing explicitly about taste. IME there’s often a bunch of cool perceptions and implicit predictive models and stuff buried in there. I appreciate that you asked here, since it let me see the responses too.
Yeah there seems to be a common idea that talking about taste is impossible, it’s not, and it’s the most important thing in the world for writers to do, it’s a critical part of learning to write for other people, instead of just writing for yourself.
I just went and read that one and found it interesting, yes!
That said, even if I’m unlikely to get a satisfactory response, I still want to ask. I would like to be able to better predict what other people like/think. And sometimes, I’ve even found that understanding someone else can help find new dimensions to appreciate :)
Yeah I finally tried to read this because it got so many upvotes and quit half-way. It feels like an extended chad vs virgin meme with the author far too smug in his implied sense of superiority.
I did not enjoy this. I did not feel like I got anything out of reading this.
However, this got curated and >500 karma, so I feel like I must be missing something. Can anyone inform me: Why did other people enjoy this? The best theory I can scrounge together is that this is “relatable” in some way to people in SF, like it conveys a vibe they are feeling?
If the goal is to evoke a sensation of disgust with the characters, then I guess you’ve succeeded for me. I already knew I would not like the sort of person described in this story, though, so I didn’t feel like I learned anything as a result, but I could see how something like that could be useful for others. I essentially just felt disgust the entire time reading this.
It fleshed out a lot of archetypes (e.g. “the person on the inside who thinks they’ll be in the right place at the right time”, “the person who feels they have had a meditation/emotional insight and this has gotten them over the intellectual hump of not building killing machines”, “the person who believes the AI may kill everyone but did an EV calculation and decided to make lots of money anyway” etc) and ways-the-future-could-go in way more detail than I’ve seen before.
There are many ideas in here that I’ve heard said offhand but never really dived into, and there’s something very informative and satisfying about seeing them painted in detail. Similar to the difference between a one-sentence description of a painting, and the actual painting.
Plus lots of fun and dramatic detail to how it’s all woven together.
These are some of the things that occur to me in answer to your question.
Yes, I felt much disgust and dismay reading it. I also feel that way about many parts of the real world!
One way to say it is that I expect to still remember the story in five years time, and I appreciate stories with that property. I also expect to get different things out of it every couple years if I reread it, which is a measure of “writing quality” in my book.
But to try to be more specific:
I just wrote, while trying to talk about something unrelated to this story: “Many people today, especially high-level people in the bay area, seem to me sort of… abstract, dissociated, cobbled-together-on-purpose-via-conscious-understanding-of-algorithms compared to the people in older books and movies. I’d like more of the normal/historical human thing.” This… is a perception that’s been with me for awhile, but the story helped it click into slightly-better focus.
Also (not that central to what I liked about the story, but an example): at least two different people who’re in some way pushing AI development have told me privately about weird choices they made when hitting puberty, about how to set up their insides. I’m curious how common things like this are, since probably most people wouldn’t tell me, and about how they work. This is the first public discussion of such that I’ve personally bumped into, and I appreciate it.
I agree reading the story was some sorts of unpleasant, and it left me feeling… perturbed for a couple days. But it helped that I felt like the author had sound moral bearings, which made it less disorienting for me that the characters didn’t?
I enjoyed a few things about it, but I think what brought it all the way from “oof, that was well written but I’m not sure I enjoyed the experience” to having some fun reading and mulling on it is that, as a writer, I’ve spent a long time trying to build out my repertoire for writing actually “bad” antagonist characters. (I think this improves the conflict in stories – when I succeed, it clearly increases beta reader engagement even if that engagement consists entirely of “WOW I HATE THEM SO MUCH” – and also, like, writing any characters at all who aren’t unbearably earnest Hufflepuffs was a challenge for me).
This story was a very vivid and memorable depiction of a way a person could be shaped that definitely isn’t anywhere near my current character repertoire, but feels self-consistent enough that I could imagine booting up my own version of a similar guy and writing him “in character” for a whole story without running into too many blank spots where I can’t model him at all. I’m on the lookout for more unlikeable antagonist archetypes to introduce in my current fiction project, so it’s good timing. It also felt...deep? Rich? Like I could dig into this imaginary person’s psychology and find more there (part of me is going “wow, who hurt you? what backstory can I give you so I feel a little sympathetic that you’re like this”, because I can write hateable antagonists a lot better if I manage to feel a little sympathetic to them).
...Apart from seeing it as inspiration for my own writing: it does feel like it captures a piece of reality and pins it down where I can look at it, and I appreciate that even when I don’t enjoy looking. (It’s plausible it might help me model real life people who aren’t earnest Hufflepuffs?) It’s speckled with in-jokes and references that entertained me a bit. The prose and metaphors were also, IMO, really well done and vivid, including some that made me laugh out loud. (In general I think making a character’s internal monologue funny is a writing strategy that makes them more engaging/readable even if they’re not likable, so I’m taking notes on that too.)
Maybe I’m just not the target audience, but I also didn’t feel like I got much out of this story. It just seemed like an exercise in cynicism.
I have no navel and I must gaze, the short story.
lol
My girlfriend (who is not at all SF-brained and typically doesn’t read LessWrong unless I send her something) really enjoyed it and felt it was great because it helped her empathize with people in AI safety / LessWrong (makes them feel more human). She felt it was well-written, enjoyably written. It was something she could read without it being a task.
Do you think you would have reacted differently and more positively, if the following text had been inserted at the top?
”Author’s note: This short story is a sort of hate letter to the big AGI companies, and to various (but not all!) of the people working there. It’s a caricature/parody; it is to the AGI companies what Dr. Strangelove was to institutions like RAND and the US Strategic Command. It is uncharitable and exaggerates—but, like Dr. Strangelove, I think it makes some pretty valid points and expresses some pretty valid emotions. I hope that some of the people building AGI are humble enough to read this story and take it as a cautionary tale.”
Had this text been inserted at the top, I probably would have been excited enough about the story to send it to some of my former colleagues at OpenAI and Anthropic.
Were I to do that, my very soul would wither and die.
Can’t you just say that yourself (not all, caricature, parody, uncharitable, exaggerates, &c.) when sharing it? Death of the author, right?
With such a disclaimer, I think I would have been less confused but still not positive on the text. Maybe I’d be less negative? Neutral, even? A lack of disclaimer is not really the core of it for me though.
I have strange tastes in fiction, I think, and normally after reading something that anti-resonated with me so much, I’d just stop reading. But, I read to the end because a bunch of upvotes signaled to me that I might otherwise miss out on something later in the text.
I live in the Bay Area and work in tech and follow AI news. For me, I already knew that there are traits I don’t like in others, and had approximate models of their frequency in and correlations to the AI scene. Each character struck me as plausible and not really meaningfully more concrete to me. Concrete in the sense of “Oh, this behavior is warning sign that this is a morally bankrupt person I should stay away from” or “I might accidentally be falling into this pattern if I find this character relatable in some way.” I can definitely see this being useful to others who do not live in the bay area/work in tech/do not follow AI news, but I mostly expected people on lesswrong to match enough of that description that they wouldn’t find themselves learning from this “concretely describing a vibe or something.” And indeed, some of the people liked it for reasons other than that, which I failed to predict!
A big part of it for me is that I just cannot stand the first person narration of this main character. An example line, pulled at random: “But regardless, she makes very good coffee and is very nice and beautiful which sometimes amounts to the same thing.” I find this thoroughly jarring. I read it as the character’s internal monologue, but this is just so thoroughly outside the domain of what would go in my own internal monologue it feels… rude? an empty attempt at caricature? a failure to understand what internal thoughts look like or otherwise an indication of a mind horribly alien to my own such that it bears no resemblance to real humans? I do find the characters unappealing, but this just seemed unnecessary to me.
Interesting! Maybe there’s an experiential crux in there (so to speak)? My reader experience of this is that the first-person inner monologue is indeed very different from mine, but I perceive that as increasing the immersion and helping frame the story. To the extent that there’s a group of humans it saliently bears little resemblance to, I might think of that group as something like “humans who are psychologically ‘healthy’ in a certain way which varies across a wide spectrum, where social spheres with concentrated power may disproportionately attract people who are low on that spectrum”. I’m deliberately putting the main adjective in scare quotes there because in my fuzzy mental model, there isn’t really a clear delimiter between treating that trait cluster as a health indicator and treating it as intersubjective values dissonance; it feels consonant with but not directly targeting dark triad traits. But also, I’m not sure if you’re referring to the same thing I am or if it’s some other feature of the first-person description that bothers you.
FWIW, culturally speaking, I’ve been socially adjacent to Bay Area / SV-startup / “mainstream big tech” culture via other people, but not really been immersed in it—I’ve splashed around in the shallow part of that pool long ago, but for “try not to build the Torment Nexus” reasons (plus other unrelated stuff) I historically bounced away a lot as well and wound up in a sort of limbo. So that colors my impression quite a bit.
I mean, for one thing, the writing is quite high-quality for a blog post.
I personally don’t believe “writing quality” can be divorced from content, and if you shove a bunch of words together in a “masterful way” but don’t say anything, I don’t want to read that and therefore would call that bad writing.
Upon slightly more reflection, I think I can appreciate on an intellectual level the quality of the writing if the goal is to evoke disgust. People (myself included, on occasion) partake in Horror content, which is also traditionally a negative emotion. I haven’t heard of a Revulsion genre before, so I didn’t really consider that this might be a thing people pursue, but I would still be a little surprised if that was what most people got out of the post.
I would also be surprised if the source of all the upvotes was just that it is “”“high quality””” writing. I usually find LessWrong to be more focused on content, and I still want to know what other people see in this post.
tbh, I think you just saw an attempt at art you don’t like and you’re unlikely to get a satisfactory response. The only fiction I have written which I suspect you won’t find disgusting is this: https://www.lesswrong.com/posts/H4kadKrC2xLK24udn/the-maker-of-mind
As an update, I’ve now gotten a bunch of useful responses of why people liked this! I was worried people would interpret me as being needlessly negative, but it seems mostly not so. I think I’m still far from enjoying this myself, but I do think I’ve learned a good amount from the answers I got, and I got some interesting things to think about.
I’m a big fan of dialoguing explicitly about taste. IME there’s often a bunch of cool perceptions and implicit predictive models and stuff buried in there. I appreciate that you asked here, since it let me see the responses too.
Yeah there seems to be a common idea that talking about taste is impossible, it’s not, and it’s the most important thing in the world for writers to do, it’s a critical part of learning to write for other people, instead of just writing for yourself.
I just went and read that one and found it interesting, yes!
That said, even if I’m unlikely to get a satisfactory response, I still want to ask. I would like to be able to better predict what other people like/think. And sometimes, I’ve even found that understanding someone else can help find new dimensions to appreciate :)
I do think it was well put together and an enjoyable read.
I also think one should be deeply wary of stories that inclines one to the “My opponents are just bad people” frame, though, even if well written.
Yeah I finally tried to read this because it got so many upvotes and quit half-way. It feels like an extended chad vs virgin meme with the author far too smug in his implied sense of superiority.
I thought it was good because it made me ruminate on how close the real world is to being driven by these sorts of characters.