I would guess based on nothing in particular that with a probe of 100 tokens a 35B param MOE model that has been fine-tuned on the task could determine whether the author of the probe was the same as the author of a single 200 token reference comment with AUC of 0.83 or better.
I would further guess that if you were to score that probe comment against 16 randomly-drawn comments by every author in the corpus who had at least 16 comments of that length, the model’s top-1 guess for the author of that 100 token probe would be correct about 30% of the time.
If you had 5 probes of 100 tokens each I would guess that a fine-tuned model such as the one we’re discussing plus some basic stats could identify the correct author as the top guess over 90% of the time.
Hypothetically speaking.
I would finally speculate, in these hypothetical situation, that the three-letter agencies probably had similar capabilities since at least 2017 and probably before.
So it turns out that this hypothetical scenario had a major methodological problem where information leaked through a side channel, and once that’s fixed it’s actually more along the lines of a single probe only getting top-1 10% of the time, and 5 probes 30% of the time, and a substantial fraction of those times are near-duplicates e.g. someone really likes to drop a specific quote with specific commentary of their own, or cite a particular paper.
I would guess based on nothing in particular that with a probe of 100 tokens a 35B param MOE model that has been fine-tuned on the task could determine whether the author of the probe was the same as the author of a single 200 token reference comment with AUC of 0.83 or better.
I would further guess that if you were to score that probe comment against 16 randomly-drawn comments by every author in the corpus who had at least 16 comments of that length, the model’s top-1 guess for the author of that 100 token probe would be correct about 30% of the time.
If you had 5 probes of 100 tokens each I would guess that a fine-tuned model such as the one we’re discussing plus some basic stats could identify the correct author as the top guess over 90% of the time.
Hypothetically speaking.
I would finally speculate, in these hypothetical situation, that the three-letter agencies probably had similar capabilities since at least 2017 and probably before.
In this credibly-hypothetical scenario, how many authors were there in the corpus?
About 100k in train and about 10k in validation, the top-1 was in validation so out of a field of 10kish. Hypothetically.
So it turns out that this hypothetical scenario had a major methodological problem where information leaked through a side channel, and once that’s fixed it’s actually more along the lines of a single probe only getting top-1 10% of the time, and 5 probes 30% of the time, and a substantial fraction of those times are near-duplicates e.g. someone really likes to drop a specific quote with specific commentary of their own, or cite a particular paper.