The more important an effect is usually the stronger it is so starting many of the experiments but for a short time might yield results much faster. May be possible to overlap the non blinded experiments and run many at the same time with varying periodicity so the same interventions do not always happen on top of each other.
Your statistical method is similar to two sample t test right? Well that does not account for several possible issues of time series and dependence between data points of one variable. Lag and training effects for example. So be sure to control all other possible independent variables and plot the data timeline and when you do do not connect data points with lines!
The more important an effect is usually the stronger it is so starting many of the experiments but for a short time might yield results much faster. May be possible to overlap the non blinded experiments and run many at the same time with varying periodicity so the same interventions do not always happen on top of each other.
Things like this have crossed my mind, but that seems fancier than I can handle at the moment (I may consider this once I’ve done one or two more experiments). Might be able to use Multi-Armed Bandit-like sampling for this, even? Hm…
Your statistical method is similar to two sample t test right? Well that does not account for several possible issues of time series and dependence between data points of one variable. Lag and training effects for example. So be sure to control all other possible independent variables and plot the data timeline and when you do do not connect data points with lines!
Could you elaborate on this a bit? I am randomizing the order of intervention/control activities.
Maybe you mean something like: If I’ve done an intervention X, then it’ll be more likely to be followed by a non-X day, but the effect of X lags, so non-X days will be more likely measured as being high in X-effect. But that’d mean that X days are more likely followed by non-X, which with random order is not the case.
Might be able to use Multi-Armed Bandit-like sampling for this, even? Hm…
Effects may take time and may require time to build up to detectable levels. This is why Winters increased the length of each intervention till they lasted some weeks. If the placebo causes a different self report rating then its a bad placebo and should be Blinded out but if it causes a psychological improvement then why not use it?
so non-X days will be more likely measured as being high in X-effect. But that’d mean that X days are more likely followed by non-X, which with random order is not the case.
Yes but it will still make the effect size much less.
Could you elaborate on this a bit
Lag and build up is mentioned above. Training effect is when you get better at something just by doing it, so later interventions look better. At the same time there may be drift of self report. In other words effect of slowly growing change on memory making user think there is no change. For all these reasons plot the time series with time on X results on Y and make each point the color of intervention or placebo. Do not connect the dots with lines but do make a smooth loess-like line. You will be able to see some of the issues if they occur. Some more on all the issues.
I think I’ll take a stab at your plotting suggestions with my previous self-experiments, and make some qualitative judgements section with more plots (which people asked for in the past anyway).
The more important an effect is usually the stronger it is so starting many of the experiments but for a short time might yield results much faster. May be possible to overlap the non blinded experiments and run many at the same time with varying periodicity so the same interventions do not always happen on top of each other.
Your statistical method is similar to two sample t test right? Well that does not account for several possible issues of time series and dependence between data points of one variable. Lag and training effects for example. So be sure to control all other possible independent variables and plot the data timeline and when you do do not connect data points with lines!
Things like this have crossed my mind, but that seems fancier than I can handle at the moment (I may consider this once I’ve done one or two more experiments). Might be able to use Multi-Armed Bandit-like sampling for this, even? Hm…
Could you elaborate on this a bit? I am randomizing the order of intervention/control activities.
Maybe you mean something like: If I’ve done an intervention X, then it’ll be more likely to be followed by a non-X day, but the effect of X lags, so non-X days will be more likely measured as being high in X-effect. But that’d mean that X days are more likely followed by non-X, which with random order is not the case.
Effects may take time and may require time to build up to detectable levels. This is why Winters increased the length of each intervention till they lasted some weeks. If the placebo causes a different self report rating then its a bad placebo and should be Blinded out but if it causes a psychological improvement then why not use it?
Yes but it will still make the effect size much less.
Lag and build up is mentioned above. Training effect is when you get better at something just by doing it, so later interventions look better. At the same time there may be drift of self report. In other words effect of slowly growing change on memory making user think there is no change. For all these reasons plot the time series with time on X results on Y and make each point the color of intervention or placebo. Do not connect the dots with lines but do make a smooth loess-like line. You will be able to see some of the issues if they occur. Some more on all the issues.
Okay, that is quite informative! Thank you.
I think I’ll take a stab at your plotting suggestions with my previous self-experiments, and make some qualitative judgements section with more plots (which people asked for in the past anyway).