The problem with dyi tests is that they have no external validation—during my initial experiment I actually had a 5 min test I did 2x a day (generated so it was new problems each time) -- but the results from that don’t really make sense to anyone but myself, hence why I’ve chosen to forgo doing it.
In terms of saturating the learning effect, that’s a better approach, but getting people to put their time into doing that makes it even harder.
Right, Quantified Mind tests are not normed, so you couldn’t say “participants added 10 IQ points” or even “this participant went from 130 to 140″.
However, they do have a lot of data from other test-takers, so you can say, “participants increased 0.7 SDs [amidst the population of other QM subjects]” or “this participant went from +2.0 to +2.7 SDs”, broken down very specifically by subskill. You are not going to get any real statistical power using full IQ tests.
In terms of saturating the learning effect, that’s a better approach, but getting people to put their time into doing that makes it even harder.
It sounds like the protocols involve hours of daily participant effort over multiple weeks. Compared to that, it seems doable to have them do 5-10 minutes of daily baseline psychometrics (which double as practice) for 2-4 weeks before the experimental protocols begin? This amount of practice washout might not be enough, but if your effects are strong, it might.
In reality, that’s table stakes for measuring cognitive effects from anything short of the strongest of interventions (like giving vs. withholding caffeine to someone accustomed to having it). I recall the founder of Soylent approached us at the beginning, wanting to test whether it had cognitive benefits. When we told him how much testing he would need to have subjects do, he shelved the idea. A QM-like approach reduces the burden of cognitive testing as much as possible, but you can’t reduce it further than this, or you can’t power your experiments.
On a more positive note, if you have a small number of participants who are willing to cycle your protocols for a long time, you can get a lot of power by comparing the on- and off-protocol time periods. So if this level of testing and implementation of protocols would be too daunting to consider for dozens of participants, but you have four hardcore people who can do it all for half a year, then you can likely get some very solid results.
If I sound skeptical about expected measured effects from cognitive testing due to various interventions, it’s because, as I recall, virtually none of the experiments we ran (on our selves, with academic collaborators from Stanford, from QS volunteers, etc.) ever led to any significant increases. The exceptions were all around removing negative interventions (being tired, not having your normal stimulants, alcohol, etc.); the supposed positives (meditation, nootropics, music, exercise, specific nutrients, etc.) consistently either did roughly nothing or had a surprising negative effect (butter). What this all reinforced:
it’s easy to fool yourself with self-reports of cognitive performance (unreliable)
it’s easy to fool yourself with underpowered experiments (especially due to practice effects in longer and more complicated tests)
virtually no one does well-powered experiments (because, as above, it’s hard)
This gives me a strong prior against most of the “intervention X boosts cognition!” claims. (“How would you know?”)
Still, I’m fascinated by this area and would love to see someone do it right and find the right interventions. If you offset different interventions in your protocols, you can even start to measure which pieces of your overall cocktail work, in general and for specific participants, and which can be skipped or are even hurting performance. I have a very old and poorly recorded talk on a lazy way to do this.
One last point: all of this kind of psychometric testing, like IQ tests, only measures subjects’ alert, “aroused” performance, which is close to peak performance and is very hard to affect. Even if you’re tired and not at your best but just plodding along, when someone puts a cognitive test in front of you, boom, let’s go, wake up, it’s time–energy levels go up, test goes well, and then back to your slump. Most interventions that might make you generally more alert and significantly increase average, passive performance will end up having a negligible impact on the peak, active performance that the tests are measuring. If I were building more cognitive testing tools these days, I would try to build things that infer mental performance passively, without triggering this testing arousal. Perhaps that is where the real impacts from interventions are plentiful, strong, and useful.
The problem with dyi tests is that they have no external validation—during my initial experiment I actually had a 5 min test I did 2x a day (generated so it was new problems each time) -- but the results from that don’t really make sense to anyone but myself, hence why I’ve chosen to forgo doing it.
In terms of saturating the learning effect, that’s a better approach, but getting people to put their time into doing that makes it even harder.
Right, Quantified Mind tests are not normed, so you couldn’t say “participants added 10 IQ points” or even “this participant went from 130 to 140″.
However, they do have a lot of data from other test-takers, so you can say, “participants increased 0.7 SDs [amidst the population of other QM subjects]” or “this participant went from +2.0 to +2.7 SDs”, broken down very specifically by subskill. You are not going to get any real statistical power using full IQ tests.
It sounds like the protocols involve hours of daily participant effort over multiple weeks. Compared to that, it seems doable to have them do 5-10 minutes of daily baseline psychometrics (which double as practice) for 2-4 weeks before the experimental protocols begin? This amount of practice washout might not be enough, but if your effects are strong, it might.
In reality, that’s table stakes for measuring cognitive effects from anything short of the strongest of interventions (like giving vs. withholding caffeine to someone accustomed to having it). I recall the founder of Soylent approached us at the beginning, wanting to test whether it had cognitive benefits. When we told him how much testing he would need to have subjects do, he shelved the idea. A QM-like approach reduces the burden of cognitive testing as much as possible, but you can’t reduce it further than this, or you can’t power your experiments.
On a more positive note, if you have a small number of participants who are willing to cycle your protocols for a long time, you can get a lot of power by comparing the on- and off-protocol time periods. So if this level of testing and implementation of protocols would be too daunting to consider for dozens of participants, but you have four hardcore people who can do it all for half a year, then you can likely get some very solid results.
If I sound skeptical about expected measured effects from cognitive testing due to various interventions, it’s because, as I recall, virtually none of the experiments we ran (on our selves, with academic collaborators from Stanford, from QS volunteers, etc.) ever led to any significant increases. The exceptions were all around removing negative interventions (being tired, not having your normal stimulants, alcohol, etc.); the supposed positives (meditation, nootropics, music, exercise, specific nutrients, etc.) consistently either did roughly nothing or had a surprising negative effect (butter). What this all reinforced:
it’s easy to fool yourself with self-reports of cognitive performance (unreliable)
it’s easy to fool yourself with underpowered experiments (especially due to practice effects in longer and more complicated tests)
virtually no one does well-powered experiments (because, as above, it’s hard)
This gives me a strong prior against most of the “intervention X boosts cognition!” claims. (“How would you know?”)
Still, I’m fascinated by this area and would love to see someone do it right and find the right interventions. If you offset different interventions in your protocols, you can even start to measure which pieces of your overall cocktail work, in general and for specific participants, and which can be skipped or are even hurting performance. I have a very old and poorly recorded talk on a lazy way to do this.
One last point: all of this kind of psychometric testing, like IQ tests, only measures subjects’ alert, “aroused” performance, which is close to peak performance and is very hard to affect. Even if you’re tired and not at your best but just plodding along, when someone puts a cognitive test in front of you, boom, let’s go, wake up, it’s time–energy levels go up, test goes well, and then back to your slump. Most interventions that might make you generally more alert and significantly increase average, passive performance will end up having a negligible impact on the peak, active performance that the tests are measuring. If I were building more cognitive testing tools these days, I would try to build things that infer mental performance passively, without triggering this testing arousal. Perhaps that is where the real impacts from interventions are plentiful, strong, and useful.