So, “agile” or “lean” development, done in practice at all, is not totally theory-free—you have to select features that you think are worth testing to see if a change would be helpful, and you can’t brute-force your way through A/B testing everything, you have to use hypotheses and models of the world. It’s not actually possible to do totally model-free empiricism with limited resources.
(The same is true for science. On some level, you’re doing an experiment to test a particular hypothesis. Vast “hypothesis-free” experiments like phenotypic screens aren’t really completely hypothesis-free, and they may provide less information per time and money than testing specific well-informed hypotheses that come from the minds of experienced scientists.)
“I don’t want to have a model” sometimes means “I don’t want to tell you my model”, or “I want to keep my model tacit, because formalizing it will make it oversimplified and worse.” Sometimes people think they’re being hypothesis-free, because the act of selecting hypotheses is done by their “common sense”, which is invisible to them.
I sympathize a lot with the founders in your post who don’t want to think theoretically about the details of how the market for their product works. You were telling Snowshoes Guy “it seems like you ought to be able to charge much more for your product than you do,” and Snowshoes Guy was saying “wait wait wait, you’re basing all this speculation on a chain of reasoning that I don’t trust (or maybe understand), I don’t want to bet my company on some Thinky Stuff that Eliezer told me once and which sounded semi-plausible to my easily-impressed mind. The last time I listened to a person ramble enthusiastically about how things are gonna be great, they totally weren’t.” If you’re paying close attention, and if you have reason to believe that your thoughts about markets at a certain level of precision are better than totally random asspulls, then maybe it makes sense to try to “set prices by thinking”, the way Feynman “fixed radios by thinking” as a kid. I sure wouldn’t. When I freelance, I set prices by guessing, and then finding out empirically what people will pay. There are people I’d trust to reason about markets (my husband is one), but I’m usually safer thinking of markets as big blobs of mysterious nothing. (Do I trust myself to fix computer programs by thinking? Of course, and I’d never get anywhere if I didn’t. But then I’m a decent data scientist, and no kind of businessman at all.)
My understanding of A/B testing is that you don’t need an explicit causal model , or a “big theory” in order to successfully use it, you mostly would be using intuitions gained from experience in order to test hypotheses like “users like the red page better than the blue page”, which has no explicit causal information.
Here you argue that intuitions gained from experience count as hypotheses just as much as causal theories do, and not only that, but that they tend to succeed more often than the big theories do. That depends on what you consider to be “success” I think. I agree that empirically gained intuitions probably have a lower failure rate than causal theories (you won’t do much worse than average) but what Eliezer is mainly arguing is that you won’t do much better than average, either.
And as far as you don’t mind just doing ok on average, that might be fine, then. But the main thing this book is grappling with is “how do I know when I can do a lot better than average?” And that seems to be dependent on whether or not you have a good “big theory” available.
So, “agile” or “lean” development, done in practice at all, is not totally theory-free—you have to select features that you think are worth testing to see if a change would be helpful, and you can’t brute-force your way through A/B testing everything, you have to use hypotheses and models of the world. It’s not actually possible to do totally model-free empiricism with limited resources.
(The same is true for science. On some level, you’re doing an experiment to test a particular hypothesis. Vast “hypothesis-free” experiments like phenotypic screens aren’t really completely hypothesis-free, and they may provide less information per time and money than testing specific well-informed hypotheses that come from the minds of experienced scientists.)
“I don’t want to have a model” sometimes means “I don’t want to tell you my model”, or “I want to keep my model tacit, because formalizing it will make it oversimplified and worse.” Sometimes people think they’re being hypothesis-free, because the act of selecting hypotheses is done by their “common sense”, which is invisible to them.
I sympathize a lot with the founders in your post who don’t want to think theoretically about the details of how the market for their product works. You were telling Snowshoes Guy “it seems like you ought to be able to charge much more for your product than you do,” and Snowshoes Guy was saying “wait wait wait, you’re basing all this speculation on a chain of reasoning that I don’t trust (or maybe understand), I don’t want to bet my company on some Thinky Stuff that Eliezer told me once and which sounded semi-plausible to my easily-impressed mind. The last time I listened to a person ramble enthusiastically about how things are gonna be great, they totally weren’t.” If you’re paying close attention, and if you have reason to believe that your thoughts about markets at a certain level of precision are better than totally random asspulls, then maybe it makes sense to try to “set prices by thinking”, the way Feynman “fixed radios by thinking” as a kid. I sure wouldn’t. When I freelance, I set prices by guessing, and then finding out empirically what people will pay.
There are people I’d trust to reason about markets (my husband is one), but I’m usually safer thinking of markets as big blobs of mysterious nothing. (Do I trust myself to fix computer programs by thinking? Of course, and I’d never get anywhere if I didn’t. But then I’m a decent data scientist, and no kind of businessman at all.)
My understanding of A/B testing is that you don’t need an explicit causal model , or a “big theory” in order to successfully use it, you mostly would be using intuitions gained from experience in order to test hypotheses like “users like the red page better than the blue page”, which has no explicit causal information.
Here you argue that intuitions gained from experience count as hypotheses just as much as causal theories do, and not only that, but that they tend to succeed more often than the big theories do. That depends on what you consider to be “success” I think. I agree that empirically gained intuitions probably have a lower failure rate than causal theories (you won’t do much worse than average) but what Eliezer is mainly arguing is that you won’t do much better than average, either.
And as far as you don’t mind just doing ok on average, that might be fine, then. But the main thing this book is grappling with is “how do I know when I can do a lot better than average?” And that seems to be dependent on whether or not you have a good “big theory” available.