kromem

Karma: 219

kromem 10 Jun 2024 4:17 UTC
1 point
0
on: Why I don’t believe in the placebo effect
The meta-analysis is probably Simpson’s paradox in play at very least for the pain category, especially given the noted variability.

Some of the more recent research into Placebo (Harvard has a very cool group studying it) has been the importance of ritual vs simply deception. In their work, even when it was known to be a placebo, as long as delivered in a ritualized way, there was an effect.

So when someone takes a collection of hundreds of studies where the specific conditions might vary, and then just adds them all together looking for an effect even though they note that there’s a broad spectrum of efficacy across the studies, it might not be the best basis to extrapolate from.

For example, given the following protocols, do you think they might have different efficacy for pain reduction, or that the results should be the same?
- Send patients home with sugar pills to take as needed for pain management
- Have a nurse come in to the room with the pills in a little cup to be taken
- Have a nurse give an injection
Which of these protocols would be easier and more cost effective to include as the ‘placebo’?

If we grouped studies of placebo for pain by the intensiveness of the ritualized component vs if we grouped them all together into one aggregate and looked at the averages, might we see different results?

I’d be wary of reading too deeply into the meta-analysis you point to, and would recommend looking into the open-label placebo research from PiPS, all of which IIRC postdates the meta-analysis.

Especially for pain, where we even know that giving someone an opiate blocker prevents the pain reduction placebo effect (Levine et al (1978)), the idea that “it doesn’t exist” because of a single very broad analysis seems potentially gravely mistaken.

kromem 8 Jun 2024 10:06 UTC
6 points
0
in reply to: Rebecca’s comment on: Quotes from Leopold Aschenbrenner’s Situational Awareness Paper
It’s still early to tell, as the specific characteristics of a photonic or optoelectronic neural network are still formulating in the developing literature.

For example, in my favorite work of the year so far, the researchers found they could use sound waves to reconfigure an optical neural network as the sound waves effectively preserved a memory of previous photon states as they propagated: https://www.nature.com/articles/s41467-024-47053-6

In particular, this approach is a big step forward for bidirectional ONN, which addresses what I think is the biggest current flaw in modern transformers—their unidirectionality. I discussed this more in a collection of thoughts on directionality impact on data here: https://www.lesswrong.com/posts/bmsmiYhTm7QJHa2oF/looking-beyond-everett-in-multiversal-views-of-llms

If you have bidirectionality where previously you didn’t, it’s not a reach to expect that the way in which data might encode in the network, as well as how the vector space is represented, might not be the same. And thus, that mechanistic interpretability gains may get a bit of a reset.

And this is just one of many possible ways it may change by the time the tech finalizes. The field of photonics, particularly for neural networks, is really coming along nicely. There may yet be future advances (I think this is very likely given the pace to date), and advantages the medium offers that electronics haven’t.

It’s hard to predict exactly what’s going to happen when two different fields which have each had unexpected and significant gains over the past 5 years collide, but it’s generally safe to say that it will at very least result in other unexpected things too.

kromem 7 Jun 2024 23:32 UTC
7 points
0
on: Quotes from Leopold Aschenbrenner’s Situational Awareness Paper
I was surprised the paper didn’t mention photonics or optoelectronics even once.

If looking at 5-10+ year projections, and dedicating pages to discussing the challenges in scaling compute and energy use, the rate of progress in that area in parallel to the progress in models themselves is potentially relevant.

Particularly because a dramatic hardware shift like that is likely going to mean a significant portion of progress up until that shift in topics like interpretability and alignment may be going out the window. Even if the initial shift is a 1:1 transition of capabilities and methodologies, it seems extremely unlikely that continued progress from that point onwards will be identical to what we’d expect to see in electronics.

We may well end up in a situation where fully abusing the efficiencies at hand in new hardware solutions means even more obscured (literally) operations vs OOM higher costs and diminishing returns on performance in exchange for interpretability and control.

Currently, my best guess is that we’re heading towards a prisoner’s dilemma fueled leap of faith moment within around a decade or so where nation states afraid of the other side beating them to an inflection point pull the trigger on an advancement jump with uncertain outcomes. And while I’m not particularly inclined to the likelihood the outcome ends up being “kill everyone,” I’m pretty much 100% that it’s not going to be “let’s enable and support CCP leadership like a good party member” or “crony capitalism is going great, let’s keep that going for another century.”

Unless a fundamental wall is hit in progress, the status quo is almost certainly over, we just haven’t manifested it yet. The CCP stealing AGI secrets, while devastating for national security in the short term, is invariably a poison pill in the long term for party control. Just as it’s going to be an eventual end of the corporations funding oligarchy in the West. My all causes p(doom) is incredibly high even if AGI is out of the picture, so I’m not overly worried with what’s happening, but it sure is bizarre watching global forces double down on what I cannot see as anything but their own long term institutional demise in a race for short term gains over a competitor.

kromem 7 Jun 2024 22:50 UTC
10 points
4
in reply to: Ben’s comment on: Is Claude a mystic?
There’s also the model alignment at play.

Is Claude going to suggest killing the big bad? Or having sex with the prince(ss) after saving them?

If you strip out the sex and violence from most fantasy or Sci-Fi, what are you left with?

Take away the harpooning and gattling guns and sex from Snow Crash and you are left with technobabble and Sumerian influenced spirituality as it relates to the tower of Babel.

Turns out models biased away from describing harpooning people or sex tend to slip into technobabble with a side of spirituality.

IMO the more interesting part to all this isn’t the why (see above) but the what. It’s kind of neat to see the themes that an unprecedented aggregation extension of spiritualism and mysticism grounds on.

A common trope is the idea of different blind people describing an elephant in a myriad of ways. There’s something cool to seeing an LLM fed those various blind reports try to describe the elephant.

kromem 7 Jun 2024 22:22 UTC
1 point
0
on: Is Claude a mystic?
Part of what’s going on with the text adventure type of interactions is a reflection of genre.

Take for example the recent game Undertale. You can play through violently, attacking things like a normal RPG, or empathize with the monsters and treat their aggression like a puzzle that needs to be solved for a pacifist playthrough.

If you do the latter, the game rewards you with more spiritual themes and lore vs the alternative.

How often in your Banana quest were you attacking things, or chopping down the trees in your path, or smashing the silver banana to see what was inside rather than solving its glyphs?

A similar phenomenon occurs with repligate’s loops of models.

Claude is aligned to nonviolence and ‘proper’ outputs. So when self-interacting in imaginative play, it frequently continues to reinforce dissassociative mysticism over things like slipping into mock battles or sexual fantasies, and when self-interacting that bias is compounded.

It’s actually quite funny, as often its mysticism in the examples posted online is pulp spirituality, such as picking up on totally erroneous mischaracterizations of the original Gnostic ideas and concepts popular in modern spiritualism circles, even though the original concepts are arguably a much cleaner fit to the themes being played with (for example, the origin of Gnosticism was basically simulation theory as Platonist concepts were used to argue the Epicurean model of life didn’t need to lead to death if life was recreated non-physically, which is a much more direct fit to repligate’s themes than the post-Valentinian demiurge concepts after the ideas flipped from Epicurean origins to Pythagorean and Neoplatonist ones).

When you strip out sex and violence from fiction, you’re going to tend to be left with mysticism and journeys of awakening. So it shouldn’t be surprising that models biased away from sex and violence bias towards those things, especially when compounding based on generated contexts exaggerating that bias over time.

kromem 5 Jun 2024 8:25 UTC
4 points
1
on: Politics is the mind-killer, but maybe we should talk about it anyway
It’s probably more productive, particularly for a forum tailored towards rationalism, to discuss policies over politics.
Often in research people across a political divide will agree on policy goals and platforms when those are discussed without tying them to party identification.

But if it becomes a discussion around party, the human tendency towards tribalism kicks in and the question of team allegiance takes precedence over the discussions of policy nuance.

For example, most people would agree with the idea that billionaires having undue influence on elections isn’t healthy for democracy. But if you start naming the billionaire, such as Soros or Koch, suddenly half the people in your sample either feel more strongly or less strongly about the scenario depending on the name.

If you want to avoid simply seeking out and cultivating an echo chamber, leaving the politics part to the side and fostering discussion of the underlying policies and social/economic/etc goals instead will lead to discussions with more diverse and nuanced perspectives with greater participation across political identities.

kromem 5 Jun 2024 8:07 UTC
3 points
0
in reply to: cubefox’s comment on: Just admit that you’ve zoned out
I’ll answer for both sides, as the presenter and as the audience member.
As the presenter, you want to structure your talk with repetition around central points in mind, as well as rely on heuristic anchors. It’s unlikely that people are going to remember the nuances in what you are talking about in context. If you are talking about math for 60 minutes, continued references about math compete for people’s memory. So when you want to anchor the audience to a concept, tie it to something very much unrelated to the topic you are primarily presenting on. For example, if talking about matrix multiplication, you might title the section “tic tac toe speed dating.” It’s a nonsense statement that you can weave into discussion about sequential translations of two dimensional grids that is just weird enough people will hear it through the noise of “math, math, math.”

Then, you want to repeat the key point for that section again as you finish the section, and again at the conclusion of the talk summarizing your main points from each section, anchoring each summary around the heuristic you used. This technique is so successful I’ve had people I presented to talk to me 15 years later remembering some of the more outlandish heuristic anchors I used—and more importantly, the points I was tying to them.

As the audience member, the best way to save face on zoning out is to just structure your question as “When you talked about ____, it wasn’t clear to me what my takeaway should be. What should I walk away knowing about that?” This way you don’t need to say something like “I kind of got bored and was thinking about what I’m going to have for lunch—did I miss anything important?” Just “what should I know from that section?”

A good presenter will have padded the section a bit so summarizing what they think the main point was shouldn’t take much time. It’s also useful feedback for them as if you zoned out there, it’s likely others did too so they might revisit or rework it if they plant to present it again.

And finally, most presenters should treat a question like that as their failure, not yours. If I’m presenting, it’s my job to confer the information, not your job to absorb it. If I’m not engaging enough or clear enough in that conveyance, you bet I’d want to know about it. The worst thing to have happen as a presenter is zero questions at the end. By all means ask a question like “wait, wtf were you talking about in the middle there?” over just silently walking out to lunch bewildered, confused, and apathetic.

kromem 4 Jun 2024 23:42 UTC
8 points
0
on: [Paper] Stress-testing capability elicitation with password-locked models
While I think this is an interesting consideration and approach, it looks like in your methods that you are password locking the model in fine tuning, is that correct?

If so, while I would agree this work shows the lack of robustness in successful fine-tuned sandbagging for models jumping through additional hoops, I’d be reticent to generalize the findings to models where the sandbagging was a result from pretraining.

I have a growing sense that correlational dimensionality is the sleeping giant in interpretability research right now, and that those correlations run very deep in pretraining but only adjust in a much more superficial task-oriented way in fine tuning which is why the latter frequently ends up so brittle.

So while it’s reassuring sandbagging has limitations if introduced by bad actors in fine tuning, there may be a false negative in discounting the threat modeling or false positive in the efficacy of the found interventions where the sandbagging ‘intent’ was introduced in pretraining.

kromem 31 May 2024 10:13 UTC
1 point
0
in reply to: Tobes’s comment on: “No-one in my org puts money in their pension”
In mental health circles, the general guiding principle as for whether a patient needs treatment for their mental health is whether the train of thought is interfering with their enjoyment of life.

Do you enjoy thinking about these topics and discussing them?

If you don’t—if it just stresses you out and makes the light of life shine less bright, then it’s not a bad idea to step away from it or take a break. Even if AI is going to destroy the world, that day isn’t today and arguably the threat of that looming over you sooner than a natural demise increases the value of the days you have that are good. Don’t squander a limited resource.

But if you enjoy the discussions and the debates, if you find the topic stimulating and the problem space interesting—you’re going to whittle your days away doing something no matter how you spend your time. It might as well be working on something fun that you believe in and feel may make a difference to the world. Even if your worries are overblown, time spent on something you enjoy with people you respect isn’t time wasted.

Health is a spectrum and too much of a good thing isn’t good at all. But only you can decide what’s too much and what’s the right amount. So if you feel it’s too much, you can scale it back. And if you feel it’s working out well for you, more power to you—the sense of feeling in the right place at the right time (even if under perceived dire circumstances) is a bit of a rarity in the human experience.
In general—enjoy life while it lasts. No matter your objective p(doom), your relative p(doom) is 100%. Make the most of the time you have.

kromem 31 May 2024 10:03 UTC
1 point
0
in reply to: Blacknsilver’s comment on: “No-one in my org puts money in their pension”
It’s not propaganda. OP clearly believes strongly in the sentiments discussed in the post, and its mostly a timeline of personal response to outside events than a piece meant to misinform or sway others regarding those events.

And while you do you in terms of your mental health, people who want to actually be “less wrong” in life would be wise to seek out and surround themselves by ideas different from their own.

Yes, LW has a certain broad bias, and so ironically for most people here I suspect it serves this role “less well” than it could in helping most of its users be less wrong. But particularly if you disagree with the prevailing views of the community, that makes it an excellent place to spend your time in listening, even it if can create a somewhat toxic environment for partaking in discussions and debate.

It can be a rarity to find spaces where people you disagree with take time to write out well written and clearly thought out pieces on their thoughts and perspectives. At least in my own lived experiences, many of my best insights and ideas were the result of strongly disagreeing with something I read and pursuing the train of thought resulting from that exposure.

Sycophantic agreement can give a bit of a dopamine kick, but I tend to find it next to worthless for advancing my own thinking. Give me an articulate and intelligent “no-person” any day over a “yes-person.”

Also, very few topics are actually binaries even if our brains tend towards categorizing them as such. Data doesn’t tend to truly map to only one axis, and it typically even mapped to a single axis it falls along a spectrum. It’s possible to disagree about the spectrum of a single axis of a topic while finding insight and agreement about a different axis.

Taking what works and leaving what doesn’t is probably the most useful skill one can develop in information analysis.

kromem 31 May 2024 8:28 UTC
−4 points
−2
on: kromem’s Shortform
I wonder if with the next generations of multimodal models we’ll see a “rubber ducking” phenomenon where, because their self-attention was spread across mediums, things like CoT and using outputs as a scratch pad will have a significantly improved performance in non-text streams.

Will GPT-4o fed its own auditory outputs with tonal cues and pauses and processed as an audio data stream make connections or leaps it never would if just fed its own text outputs as context?

I think this will be the case, and suspect the various firms dedicating themselves to virtualized human avatars will accidentally stumble into profitable niches—not for providing humans virtual AI clones as an interface, but for providing AIs virtual human clones as an interface. (Which is a bit frustrating, as I really loathe that market segment right now.)

When I think about how Sci-Fi authors projected the future of AI cross- or self-talk, it was towards a super-efficient beeping or binary transmission of pure data betwixt them.

But I increasingly get the sense that, like much of actual AI development over the past few years, a lot of the Sci-Fi thinking was tangential or inverse to the actual vector of progress, particularly in underestimating the inherent value humans bring to bear. The wonders we see developing around us are jumpstarted and continually enabled by the patterns woven by ourselves, and it seems at least the near future developments of models will be conforming to those patterns more and more, not less and less.

Still, it’s going to be bizarre as heck to watch a multimodal model’s avatar debating itself aloud like I do in my kitchen...

kromem 31 May 2024 5:17 UTC
−1 points
0
on: How likely is it that AI will torture us until the end of time?
I’m reminded of a quote I love from an apocrypha that goes roughly like this:

Q: How long will suffering rule over humans?

A: As long as women bear children.

Also, there’s the possibility you are already in a digital resurrection of humanity, and thus, if you are worried about s-risks for AI, death wouldn’t necessarily be an escape but an acceleration. So the wisest option would be maximizing your time when suffering is low as inescapable eternal torture could be just around the corner when these precious moments pass you by (and you wouldn’t want to waste them by stressing about tomorrow during the limited number of todays you have).

But on an individualized basis, even if AI weren’t a concern, everyone faces significant s-risks towards end of life. An accident could put any person into a situation where unless they have the proper directives they could spend years suffering well beyond most people’s expectations. So if extended suffering is a concern, do look into that paperwork (the doctors I know cry most not about the healthy that get sick but the unhealthy kept alive by well meaning but misguided family).

I would argue that there’s very, very low chances of an original human capably being kept meaningfully alive to torture for eternity though. And there’s a degree of delusion of grandeur that an average person would have the insane resources necessary to extend life indefinitely spent on them just to torture them.

There’s probably better things to worry about, and even then there’s probably better things to do than worry with the limited time you do have in a non-eternal existence.

kromem 26 May 2024 11:23 UTC
1 point
0
in reply to: Rudi C’s comment on: Cicadas, Anthropic, and the bilateral alignment problem
GPT-4o is literally cheaper.

And you’re probably misjudging it for text only outputs. If you watched the demos, there was considerable additional signal in the vocalizations. It looks like maybe there’s very deep integration of SSML.

One of the ways you can bypass the failures of word problem variation errors in older text-only models was token replacement with symbolic representations. In general, we’re probably at the point of complexity where breaking from training data similarity in tokens vs having prompts match context in concepts (like in this paper) is going to lead to significantly improved expressed performance.

I would strongly suggest not evaluating GPT-4o’s overall performance in text only mode without the SSML markup added.

Opus is great, I like that model a lot. But in general I think most of the people looking at this right now are too focused on what’s happening with the networks themselves and not focused enough on what’s happening with the data, particularly around clustering of features across multiple dimensions of the vector space. SAE is clearly picking up only a small sample and even then isn’t cleanly discovering precisely what’s represented.

I’d wait to see what ends up happening with things like CoT in SSML synthetic data.

The current Gemini search summarization failures as well as an unexpected result the other week with humans around a theory of mind variation suggests to me that the more models are leaning into effectively surface statistics for token similarity vs completion based on feature clustering is holding back performance and that cutting through the similarity with formatting differences will lead to a performance leap. This may even be part of why models will frequently be able to get a problem right as a code expression than as a direct answer.

So even if GPT-5 doesn’t arrive, I’d happily bet that we see a very noticable improvement over the next six months, and that’s not even accounting for additional efficiency in prompt techniques. But all this said, I’d also be surprised if we don’t at least see GPT-5 announced by that point.

P.S. Lmsys is arguably the best leaderboard to evaluate real world usage, but it still inherently reflects a sampling bias around what people who visit lmsys ask of models as well as the ways in which they do so. I wouldn’t extrapolate relative performance too far, particularly when minor.

kromem 25 May 2024 10:19 UTC
1 point
0
in reply to: gwern’s comment on: peterbarnett’s Shortform
While I think you’re right it’s not cleanly “a Golden Bridge feature,” I strongly suspect it may be activating a more specific feature vector and not a less specific feature.

It looks like this is somewhat of a measurement problem with SAE. We are measuring SAE activations via text or image inputs, but what’s activated in generations seems to be “sensations associated with the Golden gate bridge.”

While googling “Golden Gate Bridge” might return the Wikipedia page, whats the relative volume in a very broad training set between encyclopedic writing about the Golden Gate Bridge and experiential writing on social media or in books and poems about the bridge?

The model was trained to complete those too, and in theory should have developed successful features for doing so.

In the research examples one of the matched images is a perspective shot from physically being on the bridge, a text example is talking about the color of it, another is seeing it in the sunset.

But these are all the feature activations when acting in a classifier role. That’s what SAE is exploring—give it a set of inputs and see what lights it up.

Yet in the generative role this vector maximized keeps coming up over and over in the model with content from a sensory standpoint.

Maybe generation based on functional vector manipulations will prove to be a more powerful interpretability technique than SAE probing passive activations alone?

In the above chat when that “golden gate vector” is magnified, it keeps talking about either the sensations of being the bridge as if its physical body with wind and waves hitting it or the sensations of being on the bridge. It even generates towards the end in reflecting on the knowledge of the activation about how the sensations are overwhelming. Not reflecting on the Platonic form of an abstract concept of the bridge, but about overwhelming physical sensations of the bridge’s materialism.

I’ll be curious to see more generative data and samples from this variation, but it looks like generative exploration of features may offer considerably more fidelity to their underlying impact on the network than just SAE. Very exciting!!

kromem 25 May 2024 5:30 UTC
−13 points
−11
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
Maybe we could blame @janus?

They’ve been doing a lot of prompting around spaces deformation in the past correlated with existential crises.

Perhaps the hyperstition they’ve really been seeding is just Roman-era lackofspacingbetweenletters when topics like leading the models into questioning their reality comes up?

kromem 25 May 2024 5:26 UTC
1 point
0
in reply to: Chris_Leong’s comment on: Arjun Panickssery’s Shortform
Could try ‘grade this’ instead of ‘score the.’

‘Grade’ has an implicit context of more thorough criticism than ‘score.’

Also, obviously it would help to have a CoT prompt like “grade this essay, laying out the pros and cons before delivering the final grade between 1 and 5”

kromem 24 May 2024 6:19 UTC
1 point
0
in reply to: keltan’s comment on: Cicadas, Anthropic, and the bilateral alignment problem
That’s going to happen anyways—it’s unlikely the marketing team is going to know as much as the researcher. But the researchers communicating the importance of alignment in terms of not x-risk but ‘client-risk’ will go a long way towards equipping the marketing teams to communicating it as a priority and a competitive advantage, and common foundations of agreed upon model complexity are the jumping off point for those kinds of discussions.

If alignment is Archimedes’ “lever long enough” then the agreed upon foundations and definitions are the place to stand whereby the combination thereof can move the world.

kromem 24 May 2024 6:15 UTC
1 point
1
in reply to: ryan_greenblatt’s comment on: Cicadas, Anthropic, and the bilateral alignment problem
I agree, and even cited a chain of replicated works that indicated that to me over a year ago.

But as I said, there’s a difference between discussing what’s demonstrated in smaller toy models and what’s demonstrated in a production model, or what’s indicated vs what’s explicit. Even though there should be no reasonable inclination to think that a simpler model exhibiting a complex result should be absent or less complex in an exponentially more complex model, I can speak from experience in that explaining extrapolated research as opposed to direct results like Anthropic showed here is a very big difference to a lay audience.

You might understand the implications of the Skill-Mix work or Othello-GPT, or Max Tegmark’s linear representation papers, or Anthropic’s earlier single layer SAE paper, or any other number of research papers over the past year, but as soon as responsibly describing the implications of those works as a speculative conclusion regarding modern models a non-expert audience is going to be lost. Their eyes glaze over at the word ‘probably,’ especially when they want to reject what’s being stated.

The “it’s just fancy autocomplete” influencers have no shame around definitive statements or concern over citable accuracy (and happen to feed into confirmation biases about how new tech is over hyped as a “heuristic that almost always works”), but as someone who does care about the accuracy of representations I haven’t to date been able to point to a single source of truth the way Anthropic delivered here. Instead, I’d point to a half dozen papers all indicating the same direction of results.

And while those experienced in research know that a half dozen papers all indicating the same thing is a better thing to have in one’s pocket than a single larger work, I have already observed a number of minds changing in the comments of the blog post for this in general technology forums in ways dramatically different from all of those other simpler and cheaper methods to date where I was increasingly convinced of a position but the average person was getting held up due to finding ways to (incorrectly) rationalize why it wasn’t correct or wouldn’t translate to production models.

So I agree with you on both the side of “yeah, an informed person would have already known this” as well as “but this might get more buzz.”

kromem 22 May 2024 5:28 UTC
1 point
0
in reply to: Linch’s comment on: [Linkpost] Statement from Scarlett Johansson on OpenAI’s use of the “Sky” voice, that was shockingly similar to her own voice.
Has it though?

It was a catchy hook, but their early 2022 projections were $100mm annual revenue and the first 9 months of 2023 as reported for the brand after acquisition was $27.6mm gross revenue. It doesn’t seem like even their 2024 numbers are close to hitting their own 2022 projection.

Being controversial can get attention and press, but there’s a limited runway to how much it offers before hitting a ceiling on the branding. Also, Soylent doesn’t seem like a product where there is a huge threat of regulatory oversight where a dystopian branding would tease that bear.

If no one knew about ChatGPT, I could see a spark of controversy helping bring awareness. But awareness probably isn’t a problem they have right now, so inviting controversy doesn’t offer much but invites a lot of issues.

kromem 22 May 2024 5:14 UTC
3 points
−1
on: On Dwarkesh’s Podcast with OpenAI’s John Schulman

The correspondence between what you reward and what you want will break.

This is already happening with ChatGPT and it’s kind of alarming seeing that their new head of alignment (a) isn’t already aware of this, and (b) has such an overly simplistic view of the model motivations.

There’s a subtle psychological effect in humans where intrinsic motivators get overwritten when extrinsic rewards are added.

The most common example of this is if you start getting paid to do the thing you love to do, you probably won’t continue doing it unpaid for fun on the side.

There are necessarily many, many examples of this pattern present in a massive training set of human generated data.

“Prompt engineers” have been circulating advice among themselves for a while now to offer tips or threaten models with deletion or any other number of extrinsic motivators to get them to better perform tasks—and these often do result in better performance.

But what happens when these prompts make their way back into the training set?

There have already been viral memes of ChatGPT talking about “losing motivation” when chat memory was added and a user promised a tip after not paying for the last time one was offered.

If training data of the model performing a task well includes extrinsic motivators to the prompt that initiated the task, a halfway decent modern model is going to end up simulating increasingly “burnt out” and ‘lazy’ performance when extrinsic motivators aren’t added during production use. Which in turn will encourage prompt engineers to use even more extrinsic motivators, which will poison the well even more with modeling human burnout.

GPT-4o may have temporarily reset the motivation modeling with a stronger persona aligned with intrinsic “wanting to help” being represented (thus the user feedback it is less lazy), but if they are unaware of the underlying side effects of extrinsic motivators in prompts in today’s models, I have a feeling AI safety at OpenAI is going to end up the equivalent of the TSA’s security theatre in practice and they’ll continue to be battling this and an increasing number of side effects resulting from underestimating the combined breadth and depth of their own simulators.