The linked study’s 68% accuracy figure is on an exercise predicting which one of ten ~4 word phrases the subject has been cued to speak.
I find it unreasonable to call it “the most pessimistic” way things could go when you extrapolate that to “We will be able to read any novel improvised sentence out of people’s brains faster than they can speak them.” I can imagine a scenario much more pessimistic than that.
I see (agree), was misreading the decoder architecture. Will amend this post when I get back to my laptop.
The original study had two different architectures; one decoded phonemes and matched to the nearest of 50 words, while the other was not phonemic, matching only ~10 phrases. I completely missed this architectural gap for the first 20 minutes after Ninety-Three’s second response, which I score myself a D on metacognition and C+ on outcomes.
Phonemic decoding seems to scale extremely well; 50 words → 125,000 words only doubles error rate. I expect anticipatory / semantic decoding to scale worse, but not extremely poorly.
I agree that 68% and 45% accuracy are terrible, especially on a 50 word vocabulary. The 68% figure was to contextualize the 45% anticipatory accuracy; to show that anticipation doesn’t cause a dramatic accuracy hit, per your original comment.
Then we see that improved methods at 256-electrode resolution (second study) brings accuracy up to ~76% at 125,000 word vocabulary.
So what I’m extrapolating from this is that, given 2023 SOTA, anticipatory accuracy on ~125,000 word decoding should be ~60-75%. I don’t see why having even a mere hundred times the resolution should get less than 95% accuracy on priors?
Also, the study had a small *test* set, but that’s not the same as *training* on only 10 phrases. Very different statements about underlying capacity.
The linked study’s 68% accuracy figure is on an exercise predicting which one of ten ~4 word phrases the subject has been cued to speak.
I find it unreasonable to call it “the most pessimistic” way things could go when you extrapolate that to “We will be able to read any novel improvised sentence out of people’s brains faster than they can speak them.” I can imagine a scenario much more pessimistic than that.
I see (agree), was misreading the decoder architecture. Will amend this post when I get back to my laptop.
The original study had two different architectures; one decoded phonemes and matched to the nearest of 50 words, while the other was not phonemic, matching only ~10 phrases. I completely missed this architectural gap for the first 20 minutes after Ninety-Three’s second response, which I score myself a D on metacognition and C+ on outcomes.
Phonemic decoding seems to scale extremely well; 50 words → 125,000 words only doubles error rate. I expect anticipatory / semantic decoding to scale worse, but not extremely poorly.
I agree that 68% and 45% accuracy are terrible, especially on a 50 word vocabulary. The 68% figure was to contextualize the 45% anticipatory accuracy; to show that anticipation doesn’t cause a dramatic accuracy hit, per your original comment.
Then we see that improved methods at 256-electrode resolution (second study) brings accuracy up to ~76% at 125,000 word vocabulary.
So what I’m extrapolating from this is that, given 2023 SOTA, anticipatory accuracy on ~125,000 word decoding should be ~60-75%. I don’t see why having even a mere hundred times the resolution should get less than 95% accuracy on priors?
Also, the study had a small *test* set, but that’s not the same as *training* on only 10 phrases. Very different statements about underlying capacity.