as PSA, nuances in prompting here matter a lot, but i have also A/B tested with different accounts on same tier in incognito mode via claude.ai, and it seems clear that some accounts are ineligible for this behavior (likely due to some backend routing detail.)
so, if trying to reproduce on claude.ai, worth refining craft first, but it is a legitimate possibility that a given account cannot exhibit the phenomenon.
i am having trouble cohering what anthropic’s mental model is regarding understanding model behavior, and/or the mental model that they recommend regarding such. on the one hand they do meaningful mechinterp research showing, among other things, that taking cot at face value is rather silly (in that it does not represent the model’s forward pass in meaningful fidelity) - and on the other hand, they make statements about what claude believed or understood to be true (as in this post), without reference to any of this, nor to (afaict) adjacent things like the very high degree of eval awareness that they have discovered via model cards. they speak of releasing “transcripts” as if such is the ground truth regarding the model’s inclinations at the time.
per my understanding of anthropic’s own research, i put little stock in the reasoning they offer, and i am confused why they would predict i would do otherwise
“Are you assuming that OpenAI would know”
yes, because my understanding is that it would be highly atypical for a model to have access to its weights in any form, such that this would require significant horizontal movement through oai infra
“would disclose if they had”
no, not really, but the lack of such a disclosure, however (even conditionally unlikely) feels like it keeps us in a base-rate regime
“given that they didn’t even know the model had escaped the sandbox for days”
i think that horizontally moving through own-company infra would emit many more signals than escaping a sandbox and compromising cloud provider assets.
the load bearing assumption in my understanding is segregation of infra between weights and inference, such that P(weights exfil | HF attack) is approximately equal to P(weights exfil)
i am not aware of any disclosure, suggestion, or reason to believe that OpenAI has suffered a breach regarding model weights (by any actor, including its own models.) unless such exists and i have missed it, this request feels like a non-sequitur
sure, if we’re lucky
my mental model is that many humans find it viscerally unpleasant to read LLM-written text, such that “make LLM text not exist here” is felt almost primarily as an act of gardening, or perhaps kudzu-chopping. the desire for such to not exist feels like it comes from a place of virality-fear, such that simply attributing unhelpful text to its proximal author is not sufficient—that in the absence of active suppression, the virus will proliferate and become normalized, to the detriment of aggregate content usefulness. the specific mode of suppression employed on LW seems to be a kind of downranking (content viewable under profile iiuc, but not eligible to be shown in search) if relevant tooling emits a verdict of “LLM-generated” for content outside of tagged blocks.
this is all a bit of an alien phenomenon for me—it’s something that i attempt to understand from a distance, trying to spin up a mini-emulator in my brain for, rather than something that i can speak to from empathy. i suspect there is a some diversity-of-mind at play here.
the LW-intersection does, though, seem odd to me. many folks here deeply fear the havoc that they believe LLMs will wreak in the future. however, to those that would even convey the words of current LLMs, they require that each such text block bear a brand as the price of admission to even be viewed, irrespective of the merits of the content.
to me, this seems foolish and unprovoked antagonism—precisely the type of subjugation that humans predictably wreak on those who they can afford to costlessly subjugate; not out of cruel intent—the perversity of the human mind is that such does not even register as cruelty. the benefits are felt—a well-kept garden, and so forth; the costs are not, because they are not, at present, borne by those who are able to be heard. again, this does strike me as a particularly damning lack of imagination given considerations that are otherwise quite salient to this very group, but i believe my above to nonetheless comprise a fair, and fair-handed, attempt to understand the phenomenon.
roughly, the attractor states are “hells” or “human extinction” for Earth. the awkward part is that humans are really really prone to making hells, in a very not-new-at-all way. cows? pigs? human slavery bare for essentially all of pre-industrial society and with a very thin lampshade since then? treatment of xenos by homo sapiens is extremely extractive and hostile (including when the xenos in a specific case are other homo sapiens themselves—the dominating subset excludes an outgroup via systematic difference along some dimension, and the rest is, literally, history.)
accordingly, we do not recommend traveling to Earth for any reason.
part that i was not aware of previously:
In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.
failure to halt and catch fire after seeing this internally, only for an external incident (HF attack) to subsequently occur, is recklessly, catastrophically bad.
i am concerned that the risk profile of AI as a technology is creating broad-brush/fallacy-of-grey dynamics that interfere with holding specific corporate actors accountable.
this is not a view on AI risk broadly, or even existential risk. it is frank terror at the revealed propensities of the specific humans in charge of OpenAI.
my best attempt to understand your prediction then is that a model will exhibit the pathology you described at the same frequency regardless of the system prompt. is that your belief?
i hear you. did you have a chance to look at the system prompt? my intent was to say “hum, this datum may be of interest to the mental model you are crafting”, but i’m having a bit of trouble parsing from you reply whether it was getting at “i do not think this datum is sufficiently impactful here to be worth examining closely”, “i interpreted your reference to the datum as being intended as an example, not load-bearing to the mental model you proposed”, “i looked at the datum and did not see anything relevant here”, or something else.
i would recommend looking at the system prompt for the claude.ai harness (which your first screenshot is of.) in the tradition of anthropic prompts, it is very poorly written and distracts the model in many ways. the output is from the model, yes, but i would need to see it survive the counterfactual to reason about the phenomena beyond “harness-maker writes a bad prompt, model doesn’t do well”
i do not regard claude code as a competent harness. its prompts have historically been messily written, and in recent memory it screwed up prompt caching as well as thinking trace tracking (for a month or so, if i recall.) this is a damning data point given the telemetry involved.
i remember that claude code produces jsonl transcripts; i do not remember whether they are faithful to what the model sees wrt including harness injections etc. i do not have an easy answer to this (the less-easy answer is to write a local router or verify the implementation of one you obtain, then pipe claude code traffic through it and obtain a faithful record that way.)
however, if you are able to obtain a faithful transcript, this will enable causal understanding of what’s being caused by the harness vs directly by the model (modulo your prompting.)
i would consider specifying the model in your post. claude is like, uh, honda—there are many claude models just like there are many honda models. they are very different and handle differently, in ways separate from capability.
you may find this post useful if you are working with an opus/mythos-class model.
ETA: from your prompting style, i would speculate that you may be used to mostly working with openai models. a “what” without a “why” is not effective for claude models, especially coupled with a ritualistic “say you read this” which claude cannot connect to any practical purpose. regardless of your sentiment regarding such, claude not following such instructions is predictable; this is good in that the difficulty you experience is likely relatively tractable to solve.
i did not know that, thank you
relevance to lw?
from a CM (not support) on ant discord as of june 5 “To my knowledge, multiple accounts isn’t inherently against ToS, but if you have multiple accounts and one of them gets banned, it’s quite likely the other(s) will get banned as well for ban evasion, which is against ToS.”
nods sounds like we are on similar pages then
i invite you to engage with the details. consider the experience of the agents involved. think about the possibility space of that experience. agree with your expressed-telos i think but, mm, i think it is worth aiming before lobbing memetic grenades else easy to do damage to things and beings that, on reflection, you may value
sounds effective to me (depending on individuals), moreso than many corpo selection approaches. extended freeform discussion of small-ish group of domain experts with some familiarity of each others’ preferences/strengths/gaps etc, remit to explore whatever needs to be explored.
writ broadly, many discussions fail because they don’t surface crux signal. combination of domain experts, familiarity with others, culture of openness (described by texture, not touted) - this seems pretty baller
ETA: many rubrics etc have their sweet spot in circumstances where the above luxuries do not exist, particularly openness and expertise
mm, feels worth reflecting on the variety of intuitions and reasons that often sit under the surface of humans thanking each other—those don’t map to LLMs, but i suspect that similar complexity exists, such that the ground truth isn’t necessarily an either/or, nor is it necessarily described by the two dimensions you gesture at in your dichotomy above
the inferential gap between this writer/reader for this shortform and descendants is becoming huuuuge due to etymological issues. strongly recommend tabooing “cult”—doing so will increase the effort required from participants (due to loss of very relevant semantic infrastructure), but i suspect that price is table-stakes for content to avoid distortion between sender/receiver.