Here are some initial features we should consider:
Data will be collected by a webapp controlled by a trusted third party, and will only be editable by study participants.
The results will be computed by software decided on before the data is collected.
Studies will be published regardless of positive or negative results.
Studies will have mandatory general-purpose safety questions. (web-only products likely exempt)
Follow up studies will be mandatory for continued use of results in advertisements.
All software/contracts/questions used will be open sourced (MIT) and creative commons licensed (CC BY), allowing for easier cross-product comparisons.
These requirements are higher than the average study in social sciences could fulful.[Citation needed]
That being said, I would put more faith in this startup if it targeted more professional research first and thus made itself more compatible with traditional papers. In a first step it would require researchers to announce a study and then publish the results regardless of the outcome (as is already done by some journals, as far as I know.) In a second step, require the results to be analysed by code published in advance under some kind of open content / open source licence. In a third step require there to be a replication under the same conditions for the claim to be published “officially”. And so on.
I’ll think about it some more, but the whole thing seems like it has been discussed on LW before.
Thank you. Help considering the methodology and project growth prospects is very much appreciated.
I agree that compatibility with traditional papers is important. It was not stated explicitly, but I do want the results to be publishable in traditional journals. I plan on publishing the results for my company’s product. It seemed to me like being overly rigorous might be a selling point initially—“sure we did the study cheap / didn’t use a university, but look how insanely rigorous we were”
Going after professional researchers seems much harder. They actually know how to perform the research, so the value proposition is much weaker—they are already trusted, and know how to use R :p
These are just initial thoughts. I’ll think about this more.
I agree that compatibility with traditional papers is important. It was not stated explicitly, but I do want the results to be publishable in traditional journals. I plan on publishing the results for my company’s product. It seemed to me like being overly rigorous might be a selling point initially—“sure we did the study cheap / didn’t use a university, but look how insanely rigorous we were”
The thing is this seems like an ab-initio approach to doing research by people who are not researchers by trade. The vast majority of tech startups are lead by engineers not researchers, though there is no visible line between the two.
Going after professional researchers seems much harder. They actually know how to perform the research, so the value proposition is much weaker—they are already trusted, and know how to use R :p
By the principle of comparative advantage researchers should be willing to delegate some of their work to a third party, so look for the repetitive parts that could be automated by either protocol or program. If, for example, the journal requires a replication before the full study is published, the original researcher(s) might have an incentive to plan for a replication from another party.
My idea for you would be to follow the same line most other improvements on traditional procedures follow: Automate the parts that can be automated, standardise the parts that can be standardised and continue. Designing a whole system tends to fail from my reading of history.
A two-pronged approach might even be more favourable: Work with a traditional journal that has the “perfect” scientific standards so the requirements infect traditional science and meanwhile fill the journal with the papers generated from the program.
These requirements are higher than the average study in social sciences could fulful.[Citation needed]
That being said, I would put more faith in this startup if it targeted more professional research first and thus made itself more compatible with traditional papers. In a first step it would require researchers to announce a study and then publish the results regardless of the outcome (as is already done by some journals, as far as I know.) In a second step, require the results to be analysed by code published in advance under some kind of open content / open source licence. In a third step require there to be a replication under the same conditions for the claim to be published “officially”. And so on.
I’ll think about it some more, but the whole thing seems like it has been discussed on LW before.
Thank you. Help considering the methodology and project growth prospects is very much appreciated.
I agree that compatibility with traditional papers is important. It was not stated explicitly, but I do want the results to be publishable in traditional journals. I plan on publishing the results for my company’s product. It seemed to me like being overly rigorous might be a selling point initially—“sure we did the study cheap / didn’t use a university, but look how insanely rigorous we were”
Going after professional researchers seems much harder. They actually know how to perform the research, so the value proposition is much weaker—they are already trusted, and know how to use R :p
These are just initial thoughts. I’ll think about this more.
The thing is this seems like an ab-initio approach to doing research by people who are not researchers by trade. The vast majority of tech startups are lead by engineers not researchers, though there is no visible line between the two.
By the principle of comparative advantage researchers should be willing to delegate some of their work to a third party, so look for the repetitive parts that could be automated by either protocol or program. If, for example, the journal requires a replication before the full study is published, the original researcher(s) might have an incentive to plan for a replication from another party.
My idea for you would be to follow the same line most other improvements on traditional procedures follow: Automate the parts that can be automated, standardise the parts that can be standardised and continue. Designing a whole system tends to fail from my reading of history.
A two-pronged approach might even be more favourable: Work with a traditional journal that has the “perfect” scientific standards so the requirements infect traditional science and meanwhile fill the journal with the papers generated from the program.
I’ll have to think about this some more.