Just wanted to jump in here and say that Nancy, Jeremy, and the whole team are both exceptionally rational, and exceptionally capable. In my ~10 years of programming professionally, Apptimize is easily the best working environment I’ve experienced.
Absolutely worth checking out.
The karma has spoken. I’ve registered proveitforreal.com. Thank you!
I think a trademarked “proved” image will do nicely for use on labels :)
“Did you kill yourself at any point during the last 24 hours?” is not likely to produce anything useful at all.
I see. Right now the system doesn’t have any defined questions. I believe that suitable questions will be found so I’m focusing on the areas I have a solid background in.
If a project is unsafe in a literal way, shipping the product to consumers (or offering it for sale) is of course illegal. However, when considering a sous vide cooker in the past I have always worried about the dangers of potentially eating undercooked food (eg. diarrhea, nausea, and light headedness), which was how I took your meaning previously. “Product is safe for use, but accidental use might lead to undesirable outcomes”. As I mentioned in our discussion here this project is not intended to be a replacement for the FDA.
shipping a product to random people and asking them “Is it useful?” … is not likely to produce anything useful
I agree that “is it useful” is not a particularly useful question to ask, but I don’t see any harm in supporting it. If you are looking for a better question, “80% of users used the product twice a week or more three months after receiving it” sounds like information that would personally help me make a buying decision. (Have you used the product today?)
So perhaps frequency of use might be a better question? I wasn’t haggling over what questions to ask because it was your example.
never mind a proper scientific study
I think rigor in data collection and data processing are what make something scientific. For an example, you could do a rigorous study on “do you think the word turtle is funny?”.
The link I gave to the data collection webapp describes the data collection more depth, which I believe is what you are asking about between 6 and 7.
From that url:
Core function:
Every day an SMS/email is sent to participants with a securely generated one time URL.
The participant visits this URL and is greeted with a list of questions to answer.
Potential changes to this story:
If the URL is not used within 16 hours, it expires forever.
If a participant does not enter data for more than 3 days, they are automatically removed from the study.
If a participant feels that they need to removed from the study, they may do so at any time. They will be prompted to provide details on their reasons for doing so. These reasons will be communicated to the study organizer.
The study organizer may halt the study for logistical or ethical reasons at any time
I described an overview in a different thread, but that was before a lot of discussion happened.
I’ll use this as an opportunity to update the script based on what has been discussed. This is of course still non-final.
The creator of the sous vide machine (Tester) would register his study and agree to the terms.
The Tester would register this as a food-related study, automatically adding required safety questions.
The Tester would perform a search of our questions database and locate customer satisfaction related questions.
The Tester would click “start the experiment”.
Our application would post an ad seeking participants.
The participants would register for the study, once a critical mass was reached our app would create a new instance of the data collection webapp.
Once the study period is complete, the data collection app signs and transfers the participant data back to our main app. The analysis is performed, the study is posted publicly, and the Tester is notified of the results via email.
Okay, hope (and the link to my earlier user story) helps make things more clear. If you see issues with this please do bring it up—finding and fixing issues early is the reason I started this thread.
Awesome, great link. Example study here.
I think the needs for this project are still substantially different. Genomera trusts the scientists, which is usually a fine thing to do. I’ve applied for a beta invite, but don’t have access. Based on the example study I’ve linked it seems like they are more focused on assisting in data gathering (which based on my recent experience seems like the easiest thing we are considering).
Okay, sorry I’ve been away from the thread for a while. I spent the last half day hacking together a rough version of the data collection webapp. This seemed reasonable because I haven’t heard any disagreement on that part of the project, and I hope that having some working code will excite us :)
The models are quite good and well tested at this point, but the interface is still a proof of concept. I’ll have some more time tomorrow evening, which will hopefully be enough time to finish off the question rendering and SMS sending. I think with those two features added, we will have a reasonable v1.
We will still need to create
The main project page & study creation interfaces
Questions for use in our initial experiments
Participant location and screening criteria
Data analysis routines
Legal contracts
Paper describing what we did—erdos numbers don’t grow on trees :p
Repo is: https://github.com/GeneralBiotics/GLaDyS (I’ll move it away from the GB github once we finalize a project name).
Update: Question rendering now works, demo app can be viewed at http://gladys-example.herokuapp.com/
Ahh, okay. That one goes on the scrap heap.
I think if you change the price of something by an order of magnitude you get a fundamental change in what it’s used for. The examples that jump to mind are letters → email, hand copied parchment → printing press → blogs, and SpaceX. If you increase the quality at the same time you (at least sometimes) get a mini-revolution.
I think a better example might be online courses. It can be annoying that you can’t ask the professor any questions (customize the experience), but they are still vastly better than nothing.
Hmm. I’m confused. Let’s look at something slightly more extreme than what we’re talking about and see if that helps.
Level 0: Imagine we make a product study as good as possible, then allows anyone to perform the same study with a different product. Some products “shouldn’t” be tested that way, but I don’t see how a protocol like that will produce garbage (they will merely establish “no effect”).
Level 1: We broaden to support more companies, and allow anyone to perform those studies as well.
Level 2: After a sufficient number of companies have had their experiment created, we take the protocols and create a “build your own experiment constructor kit” which allows for an increasingly large number of products to receive a good test.
Level 3: As we add more and more products to the adaptable system, we reach the point where most product claims have a community ratified question for them, and the protocol is stable. You might not be able to test 20% of the things that you’d like, but for the other 80% you can test those just fine.
Please let me know where you believe that plan breaks. Actual plan will likely differ of course, but we need something concrete to talk about or I’m going to keep not understanding what sounds like potentially constructive methodology advice.
P.S. Oh, and a random thought. What do you think 4Chan will do with your “webapp”? X-)
Perform some hilarious experiments! Hopefully we get publicity from them :D
I did a poor job at the introduction. I’m assuming the studies exist, because if they don’t that’s full on false advertising.
Not to pick on anyone in particular here are some I recently encountered:
mtailor.com (“Scientifically proven to measure you 20% more accurately than a professional tailor.”—no details are provided on how this was measured, hard to believe claim, YC company)
Nightwave Sleep Assistant—list of effects, no source.
Basically anything in whole foods :p
The probiotics section at wholefoods (and my interactions with customers who believed those claims or were skeptical of my claims given the state of the supplement market) was what finally caused me to post this thread.
As a perplexing counterbalance to wholefoods are companies which don’t advertise any effects whatsoever, even though you’d expect they would.
List of companies where a lack of studies/objective claims caught my imagination:
unbounce.com & optimizely.com—these are huge companies doing science stuff. Why don’t they have “opptimizely users make X% more revenue after 9 months” rather than testimonials?
The five companies I did customer development with (Beeminder, HabitRPG, Mealsquares, Complice, Apptimize)
I am hopeful that at minimum we can create guidelines for selecting questions.
I also think that some standardized health & safety questions by product category would be good (for nutritional supplements I would personally be interested in seeing data for nausea, diarrhea, weight change, stress/mood, and changes in sleep quality).
For productivity solutions I’d be curious about effects on social relationships, and other changes in relaxation activities.
Within a given product category, I’m also hopeful we can reuse a lot of questions. Soylent’s test and Mealsquares’ test shouldn’t require significantly different questions.
Yeah, this is a brutal point. I wish I knew a good answer here.
Is there a gold standard approach? Last I checked even the state of the art wasn’t particularly good.
Facebook / Google / StumbleUpon ads sound promising in that they can be trivially automated, and if only ad respondents could sign up for the study, then the friend issue is moot. Facebook is the most interesting of those, because of the demographic control it gives.
How bad is the bias? I performed a couple google scholar searches but didn’t find anything satisfying.
To make things more complicated, some companies will want to test highly targeted populations. For example, Apptimize is only suitable for mobile app developers—and I don’t see a facebook campaign working out very well for locating such people.
A tentative solution might be having the company wishing to perform the test supply a list of websites they feel caters to good participants. This is even worse than facebook ads from a biasing perspective though. At minimum it sounds like disclosing how participants were located prominently will be important.
Thanks, that’s a great point.
I’m worried that a statistical calculator will throw off founders who would otherwise test their products with us (specifically YC founders, an abnormally influential group), so as much as possible I’d like to keep sample sizes in the “Advanced Menu” section. (This is not to say this is an unimportant issue—I’m saying this is a more important issue because many people won’t be customizing the default values).
I also think there are three unique features for product studies that can help simply defining good default values here:
Startups are going to be interested in talking about big improvements (small sample sizes needed).
Startups will likely view study participation as advertising, allowing for a generous margin of error on sample size.
Consumers are skeptical of low-sample size studies, even when they shouldn’t be.
What do you suggest we do? It sounds like getting baseline mean and variance data for the questions we include with the app is basically a requirement.
What is the awesome version of handling this issue? :p
Thank you for posting this!
I’m feeling like in this situation, I can safely say “I love standards, there are so many to choose from”
Getting a list of LessWrong approved questions would be awesome. Both because I think the LW list will be higher quality than a lot of what’s out there, and because I feel question choice is one of the free variables we shouldn’t leave in the hands of the corporation performing the test.
If anybody doubts that the products are real the trusted organisation has copies that they can analyse.
This is a great point. Maybe community members could bet karma on the outcome of a tox screening? This could create a prioritized list.
One problem with my earlier suggestion is that some companies will want narrowly selected participant pools. These will necessarily differ from the population at large, and might create data that looks like a poison placebo is being used. I see two possible solutions to this problem:
Log baseline data before the treatment is used. If people do worse on the placebo, that would be very suspicious.
Include an additional group of testers that do something different not related to the placebo/product. “Eat an apple every day for the next week”. If the placebo group did worse than the apple group, that would be very suspicious.
I feel like #2 from above is unsatisfying though, if we think it works then why are we using normal placebos?
The web app could regularly publish hashes of the data of specific studies to a public block chain. That way any tempering that happens afterwards can be detected and you only need to trust that the web app is temper proof the moment the data gets transmitted.
This would actually be really easy to implement. (Not the block chain portion, the per-study rolling checksums).
Can you provide an example of what you’d like to see pass muster?
I’m glad you’re here. My background is in backend web software, and stats once the data has been collected. I read “Measuring the Weight of Smoke” in college, but that’s not really a sufficient background to design the general protocol. That’s a lot of my motivation behind posting this to LW—there seem to be protocol experts here, with great critiques of the existing ones.
My hope is we can create a “getting started testing” document that gets honest companies on the right track. Searching around the web I’m finding things like this rather than serious guides to proper placebo creation.
In general, who will review proposed studies for things like suitable placebo decisions?
I’m hoping either registered statistical consultants or grad students. Hopefully this can be streamlined by a good introductory guide.
This is awesomely paranoid. Thank you for pointing this out.
I’m a little worried a solution here will call for whoever controls the webapp to also be an expert at creating placebos for every product type. (If we trust contract manufacturers to be honest, then the issue of adding poisons to a placebo can be handled by having them ship directly to the third party for mailing… but I that’s already the default case).
Perhaps poisons can be discovered by looking at other products which performed the same protocol? “This experiment has to be re-done because the control group mysteriously got sick” doesn’t seem like a good solution though...
I’ll wrestle with this. Maybe something with MaxL’s answer to #8 might be possible?
The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn’t need to go through the process of flipping a real coin.
I am worried that standardized shipping will come with standardized package layout, and I’m guessing “preference of left vs right identical thing” correlates with something the system will eventually test. Having thought about it more, this is the real issue with allowing customers to choose which product they’ll use: that decision has to be purely random if you want the math to be simple / understandable. I agree people are unlikely to actually flip a coin :/
Thankfully the fix is easy: you have the testing webapp decide for the participant. They receive the product, enter the numbers online, and are told which to use.
For non-crossover trials I agree this needlessly increases the cost. It’s almost surely better to use a trusted third party.
That means the participants has to do the work of going to the post office and remailing packages. Some of them will require additional time to remail and it might produce complications.
Agree. I think having a trusted third party handle the shipments is cleaner at the moment. I’m still curious what blogospheroid’s thread comes to. It seems like the paranoia of cryptoland is helping us see some more holes in modern experiment design (ie your thread on poisonous placebos).
Maybe you can win a company such as Amazon as a partner for distribution. Amazon warehouses can store both products and placebos and ship randomly.
Thanks for saying this. Looking closer, I actually think their existing Fulfillment APIs would just work for this (ie the webapp controls an Amazon fulfillment account, the person seeking a test ships two pallets of physical product there, the webapp says where to send them).
Okay, so I like everyone else’s comments, but they feel complicated with what I came up with:
Harry convinces himself of #2 enough to say it in parselmouth.
Harry says “I think I understand the prophecy you’re trying to avoid, and I believe killing me makes it happen. I would say more, but you’d probably use it to kill me” in parselmouth.
Harry stays silent.