I claim that a company that uses the strategy of “come up with a target KPI and then implement every possible dark pattern which could plausibly lead to that KPI doing what we want it to do” will be outperformed by a company which uses the strategy of “come up with a bunch of things to do which will plausibly change the value of that KPI, try all of them out on a small scale, and then scale up the ones that work and scale down the ones that don’t work”.
For context for what’s driving my intuitions here, I at one point worked at a startup where one of the cofounders did pretty much operate under the philosophy of “let’s look at what other companies vaguely in our space do, paying particular attention to things which look like they’re intended to trick customers out of their money, and implement those things in our own product as faithfully as possible”. That strategy did in fact sometimes work, but in most cases it significantly hurt retention while being minimally useful for revenue (or, in a couple of cases, hurting both retention and revenue).
In the language of your steering systems post (thank you for writing that BTW), I expect a company where the pruning system is “try things out at small scale in the real world and iterate” will outperform even a human who has a very good world-model.
I actually suspect that this is a more general disagreement—I think that, in complicated domains, the approach of “figure out what things work locally, do those things, and iterate” outperforms the approach of “look at the problem, work really hard on coming up with an explicit model of the reward landscape, and then do the optimal thing according to your model”. Obviously you can outperform either approach in isolation by combining them, but I think that the best performance is generally far to the “try things and iterate” side. If that’s still a thing you disagree with, even in that framing, I suspect that’s a useful crux for us to explore more.
Edit: To be more explicit, I think that corporations are more powerful at steering the future towards a narrow space than individual humans because they are able to try out more things than any individual human, not because they have a better internal model of the world or better process for deciding which of two atomic, mutually exclusive plans to execute.
A few remarks, not necessarily disagreements with anything specific:
“hire a bunch of people and tell them to try a bunch of things according to some general guidelines, rather than explicitly micromanaging them” is causally upstream of trying out those things.
Given access to the same resources, sufficiently smart humans are usually capable of explicit strategy stealing of any other human or group of humans in full generality and on any level of meta. Though object-level strategy stealing of your competitors might not always be a good strategy, as you point out.
“figure out what things work locally, do those things, and iterate.” Agreed that this is a good strategy in general. I’m saying that an individual explicitly reflecting on and reasoning about what an organization is trying to do and the strategy they’re using to do it, should always help, or at least not hurt, if done correctly and at the right level of generality and meta. We might disagree about how strong those preconditions are, and how likely they are to be met in practice.
I claim that a company that uses the strategy of “come up with a target KPI and then implement every possible dark pattern which could plausibly lead to that KPI doing what we want it to do” will be outperformed by a company which uses the strategy of “come up with a bunch of things to do which will plausibly change the value of that KPI, try all of them out on a small scale, and then scale up the ones that work and scale down the ones that don’t work”.
For context for what’s driving my intuitions here, I at one point worked at a startup where one of the cofounders did pretty much operate under the philosophy of “let’s look at what other companies vaguely in our space do, paying particular attention to things which look like they’re intended to trick customers out of their money, and implement those things in our own product as faithfully as possible”. That strategy did in fact sometimes work, but in most cases it significantly hurt retention while being minimally useful for revenue (or, in a couple of cases, hurting both retention and revenue).
In the language of your steering systems post (thank you for writing that BTW), I expect a company where the pruning system is “try things out at small scale in the real world and iterate” will outperform even a human who has a very good world-model.
I actually suspect that this is a more general disagreement—I think that, in complicated domains, the approach of “figure out what things work locally, do those things, and iterate” outperforms the approach of “look at the problem, work really hard on coming up with an explicit model of the reward landscape, and then do the optimal thing according to your model”. Obviously you can outperform either approach in isolation by combining them, but I think that the best performance is generally far to the “try things and iterate” side. If that’s still a thing you disagree with, even in that framing, I suspect that’s a useful crux for us to explore more.
Edit: To be more explicit, I think that corporations are more powerful at steering the future towards a narrow space than individual humans because they are able to try out more things than any individual human, not because they have a better internal model of the world or better process for deciding which of two atomic, mutually exclusive plans to execute.
A few remarks, not necessarily disagreements with anything specific:
“hire a bunch of people and tell them to try a bunch of things according to some general guidelines, rather than explicitly micromanaging them” is causally upstream of trying out those things.
Given access to the same resources, sufficiently smart humans are usually capable of explicit strategy stealing of any other human or group of humans in full generality and on any level of meta. Though object-level strategy stealing of your competitors might not always be a good strategy, as you point out.
“figure out what things work locally, do those things, and iterate.” Agreed that this is a good strategy in general. I’m saying that an individual explicitly reflecting on and reasoning about what an organization is trying to do and the strategy they’re using to do it, should always help, or at least not hurt, if done correctly and at the right level of generality and meta. We might disagree about how strong those preconditions are, and how likely they are to be met in practice.
All of those remarks look correct to me. Though “at the right level of generality and meta” is doing a lot of the work.