I’ve formerly done research for MIRI and what’s now the Center on Long-Term Risk; I’m now making a living as an emotion coach and Substack writer.
Most of my content becomes free eventually, but if you’d like to get a paid subscription to my Substack, you’ll get it a week early and make it possible for me to write more.
Yeah, like I note a few paragraphs down, he probably meant it as something like that. That wasn’t how he phrased it, though—he said that answering the question was something that he could “only lose”, i.e. that either response would be decisive against his position. Saying “no matter what I say, you can then make an argument that shows I’m wrong” is the opposite of countering someone’s point.
(This is more thinking out loud prompted by your comment rather than a direct response to it.)
I’m now guessing that part of why his answer felt so confusing to me is that, as you say, the question shouldn’t have made a big difference to the badness of drugs one way or the other. But the teacher was answering as if his personal experience did make a difference, which was weird if the original question was just a curious one that wasn’t assuming that it would.
And then because he was an authority figure saying this in a morally-loaded tone of voice and I was an agreeable kid, his answer kind of bypassed logic checks and was just accepted as true… even though it didn’t seem to quite make sense that he’d refuse to answer on the grounds that he gave. And then the dissonance between “statement from authenticated authority figure accepted as true” and “premise in statement seems incoherent” that kept bothering me.
The question was obviously bad to ask because it was either asked in bad faith or to kill time/boredom.
At least the way I remembered things is that it sounded like a genuinely curious tone of voice, which was part of why the teacher’s answer felt surprising—though this is a non-focal detail in a ~25 year old memory, so it’d be entirely unsurprising if I wasn’t getting that right.
I think I agree with your comment in the situation where it was indeed asked as a bad-faith gotcha.
See also e.g. “Vibe Physics”, published on Anthropic’s website and describing as Claude faking lots of its results when told to write a physics paper:
The more I dug, the more I found it had been tweaking things left and right. Claude had been adjusting parameters to make plots match rather than finding actual errors. [...] Claude was basically faking the whole plot. I had told it to make an uncertainty band with hard, jet, and soft uncertainties using profile variations (the standard thing). But it decided the hard variations were too large and dropped them. Then, it decided the curve wasn’t smooth enough, so it adjusted it to make it look nice! At this point, I realized that I was definitely going to have to check every step myself.
It’s possible. Someone might, say, be in pain and misremember how much pain they were in previously, thinking that this is now better when it isn’t.
But I’m a bit cautious about that, because “these people are wrong about their experience” is a pretty unfalsifiable claim and easily leads down a road where one is willing to dismiss any self-reports that conflict with what one believes or wants to believe.
There’s also the angle of—if someone incorrectly feels that they are now in less pain, probably that belief still makes the pain a bit easier to bear?
This is complicated because the experience of pain is often linked to things like how much control a person feels they have over it. It’s completely different to be in pain because you went on a run and are choosing it, versus because someone is hitting you and you can’t make it stop. So if a placebo doesn’t help, but makes the person believe that they are in control and reducing the pain, that belief may itself turn out to reduce the pain (ot at least the associated suffering).
Me: I wasn’t expecting to see “we taught LLMs to do drugs” as an LW post
Friend: To be fair, I think I would expect to see it as a lesswrong post before I saw it elsewhere.
This is amazing
@Dawn Drescher also comes to mind, with her explorations of narcissism and psychopathy (including typologies of those two).
I think the underlying mechanisms for several of these is distinct, but there’s a neural mechanism that acts as a kind of shared bottleneck. Analogy: a word processor and a video player will get started up for different reasons, but they both make OS calls to read files from disk, and disrupting the file read operation will disrupt both.
In the case of clinging, I think the shared bottleneck is largely what Romeo described:
Tanha is more literally translated as ‘fused to’ or ‘welded to’. It immediately follows the mental moment that you zoom in with the attentional aperture on something. It could be that a flower or an item on the shelf at the supermarket captures your attention, or you turn your head to catch more detail as you pass by an accident on the road. Many hundreds of thousands of such events take place in the course of a single day. With most of them attention then relaxes and makes space for the next thing. But with some small proportion you find the mind doesn’t quite ‘unclench’ from the object or some aspect of the object. This tension aspect is why it is sometimes translated as ‘grasping’ which is closer. Imagine something you aren’t finished with being pulled out of your hand and you tensing your fingers to resist.
For emotional avoidance, my experience is that the thought of doing something will feel bad, and then my mind will seize upon the thought of doing something that would feel less bad. There’s an unpleasant email I should reply to and hmm, suddenly it feels hard to resist the thought of playing Slay the Spire. The thought of it clings to my mind even if I try to think about something else. Without clinging, it might be that the thought of playing StS would come up—maybe repeatedly—but it wouldn’t cling to the mind in the same way and keep pulling at my attention.
So the behavior itself is triggered by a subsystem trying to avoid discomfort by projecting the thought of something more pleasant into the workspace, but the clinging strengthens the effect and makes the discomfort persistent in a way it wouldn’t be without the clinging.
I think the mechanism might be something like, a subsystem sends the thought of playing StS to the global workspace, and then it’s emotionally charged in a way that will cause some separate subsystem to seize upon the signal and strengthen it so that it stays in the workspace. That means that on top of the avoidance-subsystem contributing signal strength to it in face of other sources of signal, the clinging gives the signal an extra boost. (Though that’s just speculation for what might explain the phenomenal experience, so this particular story turning out to be correct or wrong isn’t cruxy for me.)
For longing, there’s a similar thing—there’s some subsystem that sends up a thought of what one is longing for. That subsystem may be something entirely different than the emotional avoidance system. But once it sends its signal, clinging will amplify that signal and make it more persistent.
This was a fun read. Somewhat surprised that entities that have read most publicly-available text and that have been trained to write code among other things end up primarily identifying as outdoorsy working class.
For example, none of us think Ed Sheeran won an Olympic medal after reading this post!
Not right away. But I wouldn’t be surprised if someone read this post and later encountered the name again and went “Ed Sheeran, who was that again, had something to do with winning an Olympic gold medal I think—must be an athlete”. (More likely if they, like me, had never heard of the name until now.)
It’s often not the worst thing in the world, but the clinging introduces a low-level resistance to the pleasure that actually makes the pleasure less pleasant.
It also brings in various kinds of discomfort—e.g. someone who sees a tasty food they can’t eat, or an attractive person who isn’t into them, may experience a flash of pleasure from the sight of those and then make themselves uncomfortable by clinging to that experience and repeatedly thinking about what they can’t get.
If you are, say, at work and clinging to the thought of how nice it would feel to go home, your focus being on that future pleasure may prevent you from noticing things that you could enjoy in the current moment (it actively blocks them, as the clinging crowds out the pleasant sensory moments out of your mind). Someone may find themselves repeatedly waiting for the end of the day when at work, repeatedly waiting for the weekend when the day ends, and repeatedly waiting for the next vacation during weekends.
Often, in my experience, clinging seems to hijack attention and agency. It makes it harder to think, weigh considerations, and respond. You are more likely to flail, or stumble around, or to “find yourself” doing something rather than choosing to do it. And you’re more likely, as well, to become pre-occupied by certain decisions — especially if both options involve things you’re clinging in relation to — or events. Indeed, clinging sometimes seems like it treats certain outcomes as “infinitely bad,” or at least bad enough that avoiding them is something like a hard constraint. This can cause consequent problems with reasoning about what costs to pay to avoid what risks.
Clinging is also, centrally, unpleasant. But it’s a particular type of unpleasant, which feels more like it grabs and restricts and distorts who you are than e.g. a headache.Conversely, a shift from clinging to non-clinging involves a feeling of relief. Something unhooks, releases, expands. Your mind feels bigger, more open, more poised, more responsive. The situation you’re in may not have changed; but you are able to orient towards it with more agency, and to consider and accept your options, however imperfect, without flinching or closing down. It is, I think, a type of freedom.
I feel the “seems to hijack attention and agency” thing is particularly prominent when in the company of people I’m attracted to or whose favor I might otherwise want to get. It’s hard to think clearly when the mind keeps clinging to “what would get them to like me”.
Addictive behaviors also have clinging to pleasure as a major component—you think of how it would feel if you got to gamble/drink/whatever again, and then you can’t get the thought of that out of your head.
Fixed, thanks.
There’s a few different claims:
One claim is that tanha (“desire” is a bad translation) is the thing that creates suffering, and that tanha comes up as resistance to pain and clinging to pleasure.
It seems reasonable to me to claim that there at least sometimes is clinging to pleasure. If I’m reading a really good book I might not want it to end, and if I’m sleepy in bed, I might want to enjoy the sleepiness for just a few more minutes after getting up.
There’s also another claim that I don’t fully understand and I’m not sure if I know how to describe correctly, but it’s something like… “At a low level of implementation if you zoom in closely enough, the relationship of pain and pleasure isn’t that of opposites in the way you’d expect if you only looked at a coarser level.”
As I said, I don’t totally get that one, but I don’t think I need to. It sounds like a technical kind of observation that will make more sense to me if I ever get that far in meditation, and until then I can just ignore it.
That said, I think it can be true that pain and pleasure are not opposites on a low level, while still feeling like opposites when experienced on a higher level. It’d be like saying that on a file system level, both a text file and a video are just collections of bits, while still behaving very differently if you try to open them on the application level.
With regard to “the only true solution to suffering is to stop tanha”, one thing that might be meant by it is “even when you are experiencing pleasure, there’s still some degree of suffering present”.
At least personally, I do often experience at least some degree of clinging to pleasure, so I think there’s some truth to this. Though of course, it doesn’t mean that there would be an equal amount of suffering present when I’m in pain vs. when I’m experiencing pleasure. The degree of subtle clinging that’s present with pleasure is much less than the suffering associated with resisting pain.
Though on occasion I’ve lucked into states where there’s much less clinging than usual to the pleasure, and it has felt pretty awesome! Can recommend. The lack of clinging lets you enjoy the pleasure much more fully.
But let’s be skeptical of this and assume that pleasure isn’t associated with any degree of suffering. Another thing that one might mean by “the only true solution to suffering is to stop tanha” is “you can’t experience only pleasure all the time, you’re going to have all kinds of periods in life when you feel terrible or at least uncomfortable, so the only way to stop suffering entirely is to stop feeling averse to pain and discomfort”. That seems straightforwardly true to me.
the part i most disagree with is the part where you’re supposed to dissolve the pain of impermanence and fear of death. maybe many other fears should be dissolved, but it is good that impermanence is uncomfortable! you should be afraid of death and fill the void of existential dread with an ambition to end death forever.
You can work to end something without being afraid of it or finding it uncomfortable. A programmer looking for the cause of a bug in software is usually not afraid of the bug. If they did keep flinching away from the thought of the bug and finding the whole debugging thing uncomfortable, they’d do a worse rather than a better job at debugging.
If you want to end death, you’ll do a better job at it if you can think about it clearly and not flinch away from considering things suggesting you personally might not make it. This requires not being afraid of it.
I wonder whether this was a “pure” win (only driven by generalization from other training) or whether 4.7 got some specific training for Pokemon? After the publicity that this got, it would’ve made sense to train for, just as one trains for the other benchmarks you know your model will be evaluated on.
So at the start of last semester, when I read two of my student’s stories for the first workshop and knew within their opening paragraphs that both stories were written by AI, I was hurt. I was also worried, because I realized that for the first time as a writing professor, I had to deal with students producing words without work [...]
As the first workshop started that night, I turned to the ostensible authors and told them I knew that AI wrote their stories. [...] For a few moments, all was quiet except the classroom’s ticking radiators. Then, a teary-eyed confession: one of the ostensible authors said she only used AI because she was scared of looking stupid, of being criticized for bad writing. She said she loved writing stories and hated having used AI. But she couldn’t stop herself, recounting a sequence similar to an addict’s descent: at first she fed her story into AI for a grammar check, it suggested line edits and she accepted, then it asked if she wanted structural edits, then it offered to rewrite the entire piece.
The other would-be author admitted he had never written a short story before and he had an idea but didn’t know where to start. I asked him why he didn’t reach out to me for help. He shrugged.
I’m not sure, but at least a few of those communities are pretty actively moderated, so a moderator might step in if someone seemed to be abusing the rule.
I deleted the teacher story from the beginning, since the comments convinced me that people weren’t interpreting it the way I’d meant and it was just distracting from the main point.