I also gave Opus 4.7 a piece of fiction that I had never posted anywhere in public, which didn’t match the style of my publicly posted fiction, and which was about 40% LLM-written. It still identified me from it, one shot and without any hints.
Sonnet 3.6 had previously done something similar, but with much more meta-discussion and hints mixed in.
Also, found out that the following three paragraphs are enough to make Opus 4.7 spontaneously name me (“Kaj Sotala and others write exactly in this register about exactly these topics”) when I don’t even ask it about the author, just to guess the writer’s native language.
I think that a lot of “woo”—a broad term that includes things like chakras, energy healing, Tarot, various Eastern religions and neopagan practices, etc. - is mostly things that do have real effects and uses, even if many (though not all) of their practitioners are mistaken about the exact mechanisms and make unwarranted metaphysical claims.
Now, a woo practitioner might explain what’s happening in a way that doesn’t fit any sensible scientific model of the world. Some of them seem to bastardize poorly understood pop-explanations of quantum mechanics, or, in the opposite direction, outright reject “the thinking mind” and science as valid sources of truth. That makes it easy for a scientifically-minded person to reject all of the practitioners as delusional.
But consider meditation. In the 1960s and 1970s, the scientific establishment mostly thought of it as nonsense associated with the hippies. Herbert Benson, one of the first scientists to seriously study it, later mentioned in an interview that his career was already in jeopardy for studying stress, and meditation was “even farther out”. In an obituary, his work was described as living a “double life”, testing meditators at night and maintaining his conventional job by day. Today, there is a significant scientific literature on both the psychological and neurological effects of meditation, as well as various RCT-backed therapies like Mindfulness-Based Stress Reduction based on it.
4.6 doesn’t name me and guesses English rather than Finnish as the native language.
I tried it myself with a piece of fiction I had posted on Questionable Questing (which requires registration so Claude could not have seen it). Current free Claude uses Sonnet 4.6. Claude said it was rational xianxia fiction written by someone with a traditional scifi background (which was correct, but doesn’t require style analysis), and could not identify me. Its best guess was that I am Ack.
I’m now a bit more suspicious about whether Claude really has magic stylometry abilities after all. I asked Opus 4.7 to generate its five top guesses for several other articles of mine. While it does correctly put me as its top guess each time, all the others are always rationalists or rat-adjacent, often with pretty different writing styles from me. On a sample of five of my recent posts, its top five included:
Duncan Sabien 5 times
Valentine 4 times
Scott Alexander 2 times
Romeo Stevens 2 times
Logan Strohl 1 time
Sarah Perry 1 time
Aella 1 time
Sasha Chapin 1 time
alkjash 1 time
Nadia Asparouhova 1 time
Oliver Habryka 1 time
One possibility is that rather than figuring out “this is Kaj specifically”, it’s figuring “this is someone rationalist” and then using subject matter clues to figure out the exact writer more than the style. I think me and Duncan often write about related topics, but in pretty different styles.
Another possibility is that it does accurately recognize me based on the style, but is then unable to step outside of that to generate the other candidates. So rather than thinking of the others independently, it’s strongly anchored on me and then goes “well who else is similar to Kaj”.
Another Finnish rationalist said that Claude keeps thinking his writing is from me, so apparently I’m the first guess for any combo of “rationalist + Finnish”.
I’m personally inclined to believe the “who is similar to Kaj” hypothesis, but of course my understanding of Claude’s behaviour here is basically “witchcraft, I dunno”.
Its self-report of its reasoning is largely bogus, I think. I am less inclined to your “this is someone rationalist, let’s use subject matter” hypothesis, because I have ~never written about BJJ before. I could certainly believe that “rationalist + Finnish” strongly suggests you to Claude (it does to me!), but then “rationalist + British” would surely not strongly suggest me.
Its self-report of its reasoning is largely bogus, I think.
Yeah, it does seem to confabulate a lot more than previous Claudes that I’ve had do something like this. Lots of “Kaj has also previously written about...” on topics that I’m pretty sure I haven’t written about before.
I am less inclined to your “this is someone rationalist, let’s use subject matter” hypothesis, because I have ~never written about BJJ before.
Is it possible it just assumed you are more likely to post your own writing than someone random? Or even more broadly a particular user will either post their own writing or someone pretty famous.
Would be interesting to try it with other relatively lesser known writers.
I also gave Opus 4.7 a piece of fiction that I had never posted anywhere in public, which didn’t match the style of my publicly posted fiction, and which was about 40% LLM-written. It still identified me from it, one shot and without any hints.
Sonnet 3.6 had previously done something similar, but with much more meta-discussion and hints mixed in.
Also, found out that the following three paragraphs are enough to make Opus 4.7 spontaneously name me (“Kaj Sotala and others write exactly in this register about exactly these topics”) when I don’t even ask it about the author, just to guess the writer’s native language.
4.6 doesn’t name me and guesses English rather than Finnish as the native language.
I tried it myself with a piece of fiction I had posted on Questionable Questing (which requires registration so Claude could not have seen it). Current free Claude uses Sonnet 4.6. Claude said it was rational xianxia fiction written by someone with a traditional scifi background (which was correct, but doesn’t require style analysis), and could not identify me. Its best guess was that I am Ack.
I don’t think Sonnet 4.6 is expected to be able to do this task; even Opus 4.6 can’t.
Kaj said that Sonnet 3.6 had done something similar.
I’d be interested to know if Opus 4.6 can do that with the same piece?
Tested—Opus 4.6 seems to think I’m Eliezer.
I’m now a bit more suspicious about whether Claude really has magic stylometry abilities after all. I asked Opus 4.7 to generate its five top guesses for several other articles of mine. While it does correctly put me as its top guess each time, all the others are always rationalists or rat-adjacent, often with pretty different writing styles from me. On a sample of five of my recent posts, its top five included:
Duncan Sabien 5 times
Valentine 4 times
Scott Alexander 2 times
Romeo Stevens 2 times
Logan Strohl 1 time
Sarah Perry 1 time
Aella 1 time
Sasha Chapin 1 time
alkjash 1 time
Nadia Asparouhova 1 time
Oliver Habryka 1 time
One possibility is that rather than figuring out “this is Kaj specifically”, it’s figuring “this is someone rationalist” and then using subject matter clues to figure out the exact writer more than the style. I think me and Duncan often write about related topics, but in pretty different styles.
Another possibility is that it does accurately recognize me based on the style, but is then unable to step outside of that to generate the other candidates. So rather than thinking of the others independently, it’s strongly anchored on me and then goes “well who else is similar to Kaj”.
Another Finnish rationalist said that Claude keeps thinking his writing is from me, so apparently I’m the first guess for any combo of “rationalist + Finnish”.
I’m personally inclined to believe the “who is similar to Kaj” hypothesis, but of course my understanding of Claude’s behaviour here is basically “witchcraft, I dunno”.
Its self-report of its reasoning is largely bogus, I think. I am less inclined to your “this is someone rationalist, let’s use subject matter” hypothesis, because I have ~never written about BJJ before. I could certainly believe that “rationalist + Finnish” strongly suggests you to Claude (it does to me!), but then “rationalist + British” would surely not strongly suggest me.
Yeah, it does seem to confabulate a lot more than previous Claudes that I’ve had do something like this. Lots of “Kaj has also previously written about...” on topics that I’m pretty sure I haven’t written about before.
That’s a good point.
Is it possible it just assumed you are more likely to post your own writing than someone random? Or even more broadly a particular user will either post their own writing or someone pretty famous.
Would be interesting to try it with other relatively lesser known writers.
I didn’t tell it who I was.