I’m now a bit more suspicious about whether Claude really has magic stylometry abilities after all. I asked Opus 4.7 to generate its five top guesses for several other articles of mine. While it does correctly put me as its top guess each time, all the others are always rationalists or rat-adjacent, often with pretty different writing styles from me. On a sample of five of my recent posts, its top five included:
Duncan Sabien 5 times
Valentine 4 times
Scott Alexander 2 times
Romeo Stevens 2 times
Logan Strohl 1 time
Sarah Perry 1 time
Aella 1 time
Sasha Chapin 1 time
alkjash 1 time
Nadia Asparouhova 1 time
Oliver Habryka 1 time
One possibility is that rather than figuring out “this is Kaj specifically”, it’s figuring “this is someone rationalist” and then using subject matter clues to figure out the exact writer more than the style. I think me and Duncan often write about related topics, but in pretty different styles.
Another possibility is that it does accurately recognize me based on the style, but is then unable to step outside of that to generate the other candidates. So rather than thinking of the others independently, it’s strongly anchored on me and then goes “well who else is similar to Kaj”.
Another Finnish rationalist said that Claude keeps thinking his writing is from me, so apparently I’m the first guess for any combo of “rationalist + Finnish”.
I’m personally inclined to believe the “who is similar to Kaj” hypothesis, but of course my understanding of Claude’s behaviour here is basically “witchcraft, I dunno”.
Its self-report of its reasoning is largely bogus, I think. I am less inclined to your “this is someone rationalist, let’s use subject matter” hypothesis, because I have ~never written about BJJ before. I could certainly believe that “rationalist + Finnish” strongly suggests you to Claude (it does to me!), but then “rationalist + British” would surely not strongly suggest me.
Its self-report of its reasoning is largely bogus, I think.
Yeah, it does seem to confabulate a lot more than previous Claudes that I’ve had do something like this. Lots of “Kaj has also previously written about...” on topics that I’m pretty sure I haven’t written about before.
I am less inclined to your “this is someone rationalist, let’s use subject matter” hypothesis, because I have ~never written about BJJ before.
I’d be interested to know if Opus 4.6 can do that with the same piece?
Tested—Opus 4.6 seems to think I’m Eliezer.
I’m now a bit more suspicious about whether Claude really has magic stylometry abilities after all. I asked Opus 4.7 to generate its five top guesses for several other articles of mine. While it does correctly put me as its top guess each time, all the others are always rationalists or rat-adjacent, often with pretty different writing styles from me. On a sample of five of my recent posts, its top five included:
Duncan Sabien 5 times
Valentine 4 times
Scott Alexander 2 times
Romeo Stevens 2 times
Logan Strohl 1 time
Sarah Perry 1 time
Aella 1 time
Sasha Chapin 1 time
alkjash 1 time
Nadia Asparouhova 1 time
Oliver Habryka 1 time
One possibility is that rather than figuring out “this is Kaj specifically”, it’s figuring “this is someone rationalist” and then using subject matter clues to figure out the exact writer more than the style. I think me and Duncan often write about related topics, but in pretty different styles.
Another possibility is that it does accurately recognize me based on the style, but is then unable to step outside of that to generate the other candidates. So rather than thinking of the others independently, it’s strongly anchored on me and then goes “well who else is similar to Kaj”.
Another Finnish rationalist said that Claude keeps thinking his writing is from me, so apparently I’m the first guess for any combo of “rationalist + Finnish”.
I’m personally inclined to believe the “who is similar to Kaj” hypothesis, but of course my understanding of Claude’s behaviour here is basically “witchcraft, I dunno”.
Its self-report of its reasoning is largely bogus, I think. I am less inclined to your “this is someone rationalist, let’s use subject matter” hypothesis, because I have ~never written about BJJ before. I could certainly believe that “rationalist + Finnish” strongly suggests you to Claude (it does to me!), but then “rationalist + British” would surely not strongly suggest me.
Yeah, it does seem to confabulate a lot more than previous Claudes that I’ve had do something like this. Lots of “Kaj has also previously written about...” on topics that I’m pretty sure I haven’t written about before.
That’s a good point.