A lot of people in the community, including me, are working independently (or have a lot of autonomy, even if they are employed). A lot of people, including me, often feel like they are underperforming or at least wonder if they are. But how do I actually know when I’m not underperforming? I’d like to make some criteria for under which circumstances I’ll consider my work satisfactory. I’d be curious to hear what other people would consider “decent” output.
This is obviously hard to define but some types of data that seem helpful:
How much do you get done in a typical month/half year?
How much do consider aspirational but realistic to get done in a typical month/half year?
How much do you consider on the low end but okay to get done in a typical month/half year?
What kind of output would you want to see out of a researcher/community organiser/other independent worker within a month/half a year to be impressed/not be disappointed? (Assuming this is amount is representative of them)
What’s the minimum output would you want to see out of a researcher/community organiser/other independent worker to be in favour of them getting funding to continue their work? (Assuming this is amount is representative of them)
What’s the minimum output would you want to see out of your friend to feel good about them continuing their current work? (Assuming this is amount is representative of them)
I literally mean silly things like “X number of substantive research documents of roughly this and that quality”—totally okay if very domain-specific. I know this probably hugely varies from person to person and is hard to answer in general—There’s lots to take into account that seems hard to specify like quality of work and often we really care about outcomes instead of outputs. Feel free to rephrase the question in whichever way seems useful to you.
(I know that there is no objective “enough” or that the real measuring stick are my values and reality. (I would have liked to include a link to a post that I’m sure exists but couldn’t find it.) And there’s a bunch of things that are individual to me and my line of work of course, so I don’t expect people to say things that are directly useful and applicable to my situation. But I still think that sometimes a bit of data would be really useful to me and hopefully many others!)
My current goal is one safety-targeted NeurIPS level paper every 4-6 months. It might take me a few months to gain experience to get there, but I’d be happy to be an author on something the quality of the tuned lens, Inference-Time Intervention, IOI or overoptimization papers 3x/year if I could get 30% of the Shapley value each time (meaning some of those are first authors).
On a slightly smaller scale I spent 3 weeks * 0.75 FTE on Catastrophic Regressional Goodhart (both the proofs and writeup) and Drake spent about the same time. We also spent 3 weeks on experimental work which didn’t go anywhere. I was disappointed at the time, but considering the good reception to the post and the lack of progress at my concurrent MIRI work it meets my bar in retrospect.
On an even smaller scale, this week I spent one day seeing if I could apply subnetwork probing to Leela Chess Zero and gave up before actually writing experiment code because I hit Python-C++ interoperability issues. This was fine but I would have been sad to spend 3 days the same way, because I should be able to run the whole experiment in 3 days. I think below this scale it’s hard to communicate my standards because it would be less than 1% of a meaningful research output, and it’s hard to tell 0.2% of a project from 0.5% of a project from the outside.
Edit: the last paragraph was planning fallacy, I got stuck a couple more times and the project should now take ~14 days.
I personally use Toggl to track how much time I spend working per day. I usually aim for at least four hours of focused work per day.
Rather than having objective standards, I find a growth-centric approach to be most effective. Optimizing for output is easy to Goodheart, so as much as possible I treat more as a metric than a goal. It’s important that I’m getting more done now than I was a year ago, for example, but I don’t explicitly aim for a particular output on a day-to-day basis. Instead I aim to optimize my processes and improve my skills, which leads to increased output. That applies not just to good work performance, but many things.
> How much do you get done in a typical month/half year?
Measuring this objectively is hard, but roughly one large project (big new feature, new application, major design overhaul) per month, or more if the projects I’m working on are smaller.
> How much do consider aspirational but realistic to get done in a typical month/half year?
I’ve managed to get projects that I’d normally finish in a month done in 2 weeks or so by crunching hard, but I’m generally pretty consistent with output on the scale of months/half years. I definitely don’t aim for that.
> How much do you consider on the low end but okay to get done in a typical month/half year?
I wouldn’t be too upset if a project goes over by 25% due to low output (they can go over longer if there are unexpected issues, but that’s another thing). Again though, I’m pretty consistent on the scale of months/half years, so this rarely happens.
> What kind of output would you want to see out of a researcher/community organiser/other independent worker within a month/half a year to be impressed/not be disappointed? (Assuming this is amount is representative of them)
I don’t have objective standards here. If I get the impression they are genuinely putting in a good effort and improving with time, I’m happy. Different people have different strengths, and a person might work quite slowly relative to the average, but produce very high quality work. If they continue improving their output, eventually it will be high (for whatever standard of “high” you like). If they’re putting in effort and not improving, they might not be in the right line of work, and then I’d be disappointed.
> What’s the minimum output would you want to see out of a researcher/community organiser/other independent worker to be in favour of them getting funding to continue their work? (Assuming this is amount is representative of them)
This is a knapsack problem. Calculate the
score = (expected output * expected value of work per unit of output) / funding requiredfor each person that needs funding, sort the list in descending order, and allocate funding in order from top to bottom. You don’t need to fully solve the knapsack problem here, because leftover funding can be carried over.> What’s the minimum output would you want to see out of your friend to feel good about them continuing their current work? (Assuming this is amount is representative of them)
Their average output over the last 12 months should be higher than their average output over the previous 12, by some non-insignificant amount.