The key difference being, of course, that you are interested in group differences.
A key important step will be offering power calculations so that they sample size can be estimated prior to performing the test. (Also, so that post-hoc, you can understand how big an effect your study should have been able to detect.)
There are already some web apps that perform this, however. How will your app improve over those, or will yours offer an integrated solution and therefore be more valuable?
I’m worried that a statistical calculator will throw off founders who would otherwise test their products with us (specifically YC founders, an abnormally influential group), so as much as possible I’d like to keep sample sizes in the “Advanced Menu” section. (This is not to say this is an unimportant issue—I’m saying this is a more important issue because many people won’t be customizing the default values).
I also think there are three unique features for product studies that can help simply defining good default values here:
Startups are going to be interested in talking about big improvements (small sample sizes needed).
Startups will likely view study participation as advertising, allowing for a generous margin of error on sample size.
Consumers are skeptical of low-sample size studies, even when they shouldn’t be.
What do you suggest we do? It sounds like getting baseline mean and variance data for the questions we include with the app is basically a requirement.
What is the awesome version of handling this issue? :p
This is an interesting project!
An obvious relevant model is Gwern’s self experimentation on himself (http://www.gwern.net/Nootropics)
The key difference being, of course, that you are interested in group differences.
A key important step will be offering power calculations so that they sample size can be estimated prior to performing the test. (Also, so that post-hoc, you can understand how big an effect your study should have been able to detect.)
There are already some web apps that perform this, however. How will your app improve over those, or will yours offer an integrated solution and therefore be more valuable?
Eg., see http://www.statisticalsolutions.net/pss_calc.php
Thanks, that’s a great point.
I’m worried that a statistical calculator will throw off founders who would otherwise test their products with us (specifically YC founders, an abnormally influential group), so as much as possible I’d like to keep sample sizes in the “Advanced Menu” section. (This is not to say this is an unimportant issue—I’m saying this is a more important issue because many people won’t be customizing the default values).
I also think there are three unique features for product studies that can help simply defining good default values here:
Startups are going to be interested in talking about big improvements (small sample sizes needed).
Startups will likely view study participation as advertising, allowing for a generous margin of error on sample size.
Consumers are skeptical of low-sample size studies, even when they shouldn’t be.
What do you suggest we do? It sounds like getting baseline mean and variance data for the questions we include with the app is basically a requirement.
What is the awesome version of handling this issue? :p