Is it feasible to just generate a bunch of such metrics, with details about what was included or not included in a particular number, and share all of them?
Should I count the people I spoke to for 15 minutes for free at the imbue potlucks? That was year-changing for at least one. But if I count them I have to count all of the free people ever, even those who were uninvested. Then people will respond “Okok, how many bounties have you taken on?” Ok sure, but should I include the people who I told “Your case is not my specialty, idk if i’ll be able to help, but I’m interested in trying for a few hours if you’re into it”? Should I include the people who had an amazing session or two but haven’t communicated in two months? Should I include the people who are being really unagentic and slow?
It would already be informative if you put numbers on each of these questions (i.e. “how often does talking for 15 minutes accomplish something”, “how many bounties have you taken on in/outside of your specialty”, “what percent of your clients are ‘unagentic and slow’ (and what does this actually mean)”). Probably one could do much better by generating several metrics that one would expect to be most useful (or top N%tile useful) and share each of them.
Disclaimer : I would not pay and want to pay that much money anyway—so I am not your intended audience
I’d trust you more (and I would think members of the rationalist community would too) if you gave several metrics, even if some of them are not so good, with explanations. Right now, it seems you chose a metric so that it looks good.
More metrics would take more time but not much if you have the data easily available. This would be my suggestion :
You can provide three percentages ( like when one provides three quantiles instead of just the mean of data values) :
the percentage of success in people you discussed for at least an hour
the percentage among the people with reasonable chances of success (motivated + didn’t bail + your expertise + spent at least X hours)
the percentage among people with great chances of success.
These percentages, with precise information on what determines in which category clients fall in and the percentage of people treated who fall into each category, would give a first sound idea of the success rate.
Taking on low success rate people would not be a problem because their data is treated separately. It’s only a problem if 90% of your clients are unlikely to be helped but that would not be a good thing anyway.
hmm i suspect releasing these metrics could make my customers significantly more annoying. like, early adopters are fun and experimental. but if i make it seem not risky then i get risk-averse people who tend to be prickly
so maybe i will compile and release this data but i would need to figure out how to do it in a way that doesn’t change the funnel
Is it feasible to just generate a bunch of such metrics, with details about what was included or not included in a particular number, and share all of them?
could you give a few examples?
also seems time-intensive hmmmm
also, i thought about it more and i really like the metric of “results generated per hour”
I think you’ve already given several examples:
It would already be informative if you put numbers on each of these questions (i.e. “how often does talking for 15 minutes accomplish something”, “how many bounties have you taken on in/outside of your specialty”, “what percent of your clients are ‘unagentic and slow’ (and what does this actually mean)”). Probably one could do much better by generating several metrics that one would expect to be most useful (or top N%tile useful) and share each of them.
oh ok hm. i also don’t want to be incentivized to not give easy-for-me help to people with low odds of success though
Disclaimer : I would not pay and want to pay that much money anyway—so I am not your intended audience
I’d trust you more (and I would think members of the rationalist community would too) if you gave several metrics, even if some of them are not so good, with explanations. Right now, it seems you chose a metric so that it looks good.
More metrics would take more time but not much if you have the data easily available. This would be my suggestion :
You can provide three percentages ( like when one provides three quantiles instead of just the mean of data values) :
the percentage of success in people you discussed for at least an hour
the percentage among the people with reasonable chances of success (motivated + didn’t bail + your expertise + spent at least X hours)
the percentage among people with great chances of success.
These percentages, with precise information on what determines in which category clients fall in and the percentage of people treated who fall into each category, would give a first sound idea of the success rate.
Taking on low success rate people would not be a problem because their data is treated separately. It’s only a problem if 90% of your clients are unlikely to be helped but that would not be a good thing anyway.
i like this thanks. might take a bit of time to put together but interested
hmm i suspect releasing these metrics could make my customers significantly more annoying. like, early adopters are fun and experimental. but if i make it seem not risky then i get risk-averse people who tend to be prickly
so maybe i will compile and release this data but i would need to figure out how to do it in a way that doesn’t change the funnel