That… um… I had a shortform just last week saying that it feels like most people making heavy use of LLMs are going backwards rather than forwards. But if you’re getting 10-20 of that per day, and that’s just on LessWrong… then the sort of people who seemed to me to be going backward are in fact probably the upper end of the distribution.
Guys, something is really really wrong with how these things interact with human minds. Like, I’m starting to think this is maybe less of a “we need to figure out the right ways to use the things” sort of situation and more of a “seal it in a box and do not touch it until somebody wearing a hazmat suit has figured out what’s going on” sort of situation. I’m not saying I’ve fully updated to that view yet, but it’s now explicitly in my hypothesis space.
Probably I should’ve said this out loud, but I had a couple of pretty explicit updates in this direction over the past couple years: the first was when I heard about character.ai (and similar), the second was when I saw all TPOTers talking about using Sonnet 3.5 as a therapist. The first is the same kind of bad idea as trying a new addictive substance and the second might be good for many people but probably carries much larger risks than most people appreciate. (And if you decide to use an LLM as a therapist/rubber duck/etc, for the love of god don’t use GPT-4o. Use Opus 3 if you have access to it. Maybe Gemini is fine? Almost certainly better than 4o. But you should consider using an empty Google Doc instead, if you don’t want to or can’t use a real person.)
I think using them as coding and research assistants is fine. I haven’t customized them to be less annoying to me personally, so their outputs often are annoying. Then I have to skim over the output to find the relevant details, and don’t absorb much of the puffery.
I had a weird moment when I noticed that talking to Claude was genuinely helpful for processing akrasia, but that this was equally true whether or not I hit enter and actually sent the message to the model. The Google Docs Therapist concept may be underrated, although it has its own privacy and safety issues- should we just bring back Eliza?
This was intended to be a humorously made point of the post. I have a long struggle with straddling the line between making a post funny and making it clear that I’m in on the joke.
The first draft of this comment was just “I use vim btw”
Stephen apparently found that the LLMs consistently suggest these people post on LessWrong, so insofar as you are extrapolating by normalizing based on the size of the LessWrong userbase (suggested by “that’s just on LessWrong”), that seems probably wrong.
Edit: I will say though that I do still agree this is worrying, but my model of the situation is much more along the lines of crazies being made more crazy by the agreement machine[1] than something very mysterious going on.
Contrary to the hope many have had that LLMs would make crazies less crazy due to being more patient & better at arguing than regular humans, ime they seem to have a memorized list-of-things-its-bad-to-believe which in new chats they will argue against you on, but for beliefs not on that list…
Yeah, Stephen’s comment is indeed a mild update back in the happy direction.
I’m still digesting, but a tentative part of my model here is that it’s similar to what typically happens to people in charge of large organizations. I.e. they accidentally create selection pressures which surround them with flunkies who display what the person in charge wants to see, and thereby lose the ability to see reality. And that’s not something which just happens to crazies. For instance, this is my central model of why Putin invaded Ukraine.
A small number of people are driven insane by books, films, artwork, even music. The same is true of LLMs—a particularly impressionable and already vulnerable cohort are badly affected by AI outputs. But this is a tiny minority—most healthy people are perfectly capable of using frontier LLMs for hours every day without ill effects.
Also, I bet most people who temporarily lose their grip on reality from contact with LLMs return to a completely normal state pretty quickly. I think most such cases are LLM helping to induce temporary hypomania rather than a permanent psychotic condition.
How do you know the rates are similar? (And it’s not e.g. like fentanyl, which in some ways resembles other opiates but is much more addictive and destructive on average)
I think that on most of the websites only about 1-10% of the users actually post things. I suspect that the number of people having those weird interactions with LLMs (and stopping before posting stuff) is like 10 − 10000 (most likely around 100) times bigger than what we see here
(not sure if this even suits the content guidelines of this site or whether I should degrade the standards here, but I will click submit to FAFO)
Guys, something is really really wrong with how these things interact with human minds.
um—yeah, how do I put this, I think I am over my sexting AI chatbots on AI roleplay platforms to well, stimulate myself (effectively soft AI nsfw). Probably spent over 80+ hrs on that pastime, now I have moved on, I think I may be the exception not the rule here, for some people the damage would be irrecoverable trauma and psychological damage, much rather for me it was just a 16-17 y/o spending his time fantasizing. For comparison I think more than 70% of people who were below 18 in my friend circle (last year), had their exposure to nsfw material of some kind before turning 18 (I would guess 14-16 is the median), I think AI porn is just the next iteration of “serving horny men stimuli” business, society has going for itself.
The effects will be similar to phones or internet, there would be a noticeable cultural shift where it’s readily accessible and culturally active , and the socially unacceptable extremes(Like AI relationships) will become part of Social Dark Matter . Currently LLMs have certainly not gone mainstream enough to appropriate AI nsfw as better than current baseline, but that seems like it will happen on this trajectory once we overcome the minor social taboos, there’s space for (economies of scales) innovation in that field.
the sort of people who seemed to me to be going backward are in fact probably the upper end of the distribution.
The cultural shift would be out sourcing boring and dense things to LLMs in varying degrees, potentially stunting “effectively literate” people’s ability to focus even further on topics they don’t like (sort of like ADHD)— which might as well be a confession on my part— this will act as Lowering the Sanity Waterline without the tech, similar to how people face withdrawal syndrome with social media finding it hard to focus and reason afterwards. Fwiw, a lot of people find current LLMs emotionally inauthentic , so I think that’s the part which will stay mainstream rather than the extremes. I remember people cried wolf for similar extremes eg; Superstimuli and the Collapse of Western Civilization , I am not expecting it this time, atleast not with the current tech, we would need more emotionally relatable chatbots to go mainstream for the AI rights revolution. (Some of my friends want to work on it —which I disagree with on basis of efficient markets here given their current skillset but that’s another story— since they’re annoyed at chatgpt’s emotional clumsiness)
I have some fun semi gears models of what’s probably going on based on some of the Leverage psychology research.[1] If correct, wow the next bit of this ride is going to have some wild turns.
Read sections 7/8/9 especially. Leverage had bad effects on some people (and good or mixed on others), but this was strongly downstream of them doing a large-scale competent effort to understand minds which had fruits. The things they’re pointing to work via text channels too, only somewhat attenuated, because minds decompress each other’s states.
That… um… I had a shortform just last week saying that it feels like most people making heavy use of LLMs are going backwards rather than forwards. But if you’re getting 10-20 of that per day, and that’s just on LessWrong… then the sort of people who seemed to me to be going backward are in fact probably the upper end of the distribution.
Guys, something is really really wrong with how these things interact with human minds. Like, I’m starting to think this is maybe less of a “we need to figure out the right ways to use the things” sort of situation and more of a “seal it in a box and do not touch it until somebody wearing a hazmat suit has figured out what’s going on” sort of situation. I’m not saying I’ve fully updated to that view yet, but it’s now explicitly in my hypothesis space.
Probably I should’ve said this out loud, but I had a couple of pretty explicit updates in this direction over the past couple years: the first was when I heard about character.ai (and similar), the second was when I saw all TPOTers talking about using Sonnet 3.5 as a therapist. The first is the same kind of bad idea as trying a new addictive substance and the second might be good for many people but probably carries much larger risks than most people appreciate. (And if you decide to use an LLM as a therapist/rubber duck/etc, for the love of god don’t use GPT-4o. Use Opus 3 if you have access to it. Maybe Gemini is fine? Almost certainly better than 4o. But you should consider using an empty Google Doc instead, if you don’t want to or can’t use a real person.)
I think using them as coding and research assistants is fine. I haven’t customized them to be less annoying to me personally, so their outputs often are annoying. Then I have to skim over the output to find the relevant details, and don’t absorb much of the puffery.
I had a weird moment when I noticed that talking to Claude was genuinely helpful for processing akrasia, but that this was equally true whether or not I hit enter and actually sent the message to the model. The Google Docs Therapist concept may be underrated, although it has its own privacy and safety issues- should we just bring back Eliza?
Google docs is not the only text editor.
This was intended to be a humorously made point of the post. I have a long struggle with straddling the line between making a post funny and making it clear that I’m in on the joke.
The first draft of this comment was just “I use vim btw”
Emacs has Eliza still built in by default of course :)
and literal paper still exists too .. for people who need a break from their laptops (eeh, who am I kidding, phones) 📝
I heard rumors about actual letter sending even, but no one in my social circles has seen it for real.. yet.
Stephen apparently found that the LLMs consistently suggest these people post on LessWrong, so insofar as you are extrapolating by normalizing based on the size of the LessWrong userbase (suggested by “that’s just on LessWrong”), that seems probably wrong.
Edit: I will say though that I do still agree this is worrying, but my model of the situation is much more along the lines of crazies being made more crazy by the agreement machine[1] than something very mysterious going on.
Contrary to the hope many have had that LLMs would make crazies less crazy due to being more patient & better at arguing than regular humans, ime they seem to have a memorized list-of-things-its-bad-to-believe which in new chats they will argue against you on, but for beliefs not on that list…
Yeah, Stephen’s comment is indeed a mild update back in the happy direction.
I’m still digesting, but a tentative part of my model here is that it’s similar to what typically happens to people in charge of large organizations. I.e. they accidentally create selection pressures which surround them with flunkies who display what the person in charge wants to see, and thereby lose the ability to see reality. And that’s not something which just happens to crazies. For instance, this is my central model of why Putin invaded Ukraine.
A small number of people are driven insane by books, films, artwork, even music. The same is true of LLMs—a particularly impressionable and already vulnerable cohort are badly affected by AI outputs. But this is a tiny minority—most healthy people are perfectly capable of using frontier LLMs for hours every day without ill effects.
Also, I bet most people who temporarily lose their grip on reality from contact with LLMs return to a completely normal state pretty quickly. I think most such cases are LLM helping to induce temporary hypomania rather than a permanent psychotic condition.
How do you know the rates are similar? (And it’s not e.g. like fentanyl, which in some ways resembles other opiates but is much more addictive and destructive on average)
I think that on most of the websites only about 1-10% of the users actually post things. I suspect that the number of people having those weird interactions with LLMs (and stopping before posting stuff) is like 10 − 10000 (most likely around 100) times bigger than what we see here
(not sure if this even suits the content guidelines of this site or whether I should degrade the standards here, but I will click submit to FAFO)
um—yeah, how do I put this, I think I am over my sexting AI chatbots on AI roleplay platforms to well, stimulate myself (effectively soft AI nsfw). Probably spent over 80+ hrs on that pastime, now I have moved on, I think I may be the exception not the rule here, for some people the damage would be irrecoverable trauma and psychological damage, much rather for me it was just a 16-17 y/o spending his time fantasizing. For comparison I think more than 70% of people who were below 18 in my friend circle (last year), had their exposure to nsfw material of some kind before turning 18 (I would guess 14-16 is the median), I think AI porn is just the next iteration of “serving horny men stimuli” business, society has going for itself.
The effects will be similar to phones or internet, there would be a noticeable cultural shift where it’s readily accessible and culturally active , and the socially unacceptable extremes(Like AI relationships) will become part of Social Dark Matter . Currently LLMs have certainly not gone mainstream enough to appropriate AI nsfw as better than current baseline, but that seems like it will happen on this trajectory once we overcome the minor social taboos, there’s space for (economies of scales) innovation in that field.
The cultural shift would be out sourcing boring and dense things to LLMs in varying degrees, potentially stunting “effectively literate” people’s ability to focus even further on topics they don’t like (sort of like ADHD)— which might as well be a confession on my part— this will act as Lowering the Sanity Waterline without the tech, similar to how people face withdrawal syndrome with social media finding it hard to focus and reason afterwards. Fwiw, a lot of people find current LLMs emotionally inauthentic , so I think that’s the part which will stay mainstream rather than the extremes. I remember people cried wolf for similar extremes eg; Superstimuli and the Collapse of Western Civilization , I am not expecting it this time, atleast not with the current tech, we would need more emotionally relatable chatbots to go mainstream for the AI rights revolution. (Some of my friends want to work on it —which I disagree with on basis of efficient markets here given their current skillset but that’s another story— since they’re annoyed at chatgpt’s emotional clumsiness)
None of that about AI relationships sounds particularly bad. Certainly that’s not the sort of problem I’m mainly worried about here.
Some of it seems bad to roughly the same degree you thought phones were bad, tho?
I have some fun semi gears models of what’s probably going on based on some of the Leverage psychology research.[1] If correct, wow the next bit of this ride is going to have some wild turns.
Read sections 7/8/9 especially. Leverage had bad effects on some people (and good or mixed on others), but this was strongly downstream of them doing a large-scale competent effort to understand minds which had fruits. The things they’re pointing to work via text channels too, only somewhat attenuated, because minds decompress each other’s states.