Two more points I’d add:
Diminishing returns in code produced. When code production is limited, one naturally works only on those pieces of code that are most useful. But if you can create 10x more code, you likely don’t reap 10x the benefits. I notice this a lot in private projects, I just quickly run out of ideas of what to even code because finding anything that has more value than the time it would even take to prompt it becomes non-trivial relatively quickly. Maybe somewhat of a skill issue, but I this effect will always be present to some degree.
Code doesn’t live in isolation. It often requires changes outside of the code to go along with it. So, often you can’t just speedrun the code to its final form, but many things around it are keeping it in place to some degree, or have to slowly advance alongside the code. Things like user expectations & knowledge, deployment, code in other projects that may depend on it and are outside of your control, general project planning / decisions, and so on. It reminds me of my observation about parameter updates during gradient descent here:
While most parameters have “settled down” after the first few hundred epochs, clearly some of them are still on a mission, though. I was also a bit surprised how smoothly many of them are moving. [...] many parameters are systematically drifting into one direction, albeit now in a more noisy fashion. It gives me “just update all the way bro!” vibes—but perhaps this is a bit of a coordination problem: multiple parameters have to move more or less in unison, and none of them can reasonably update further even though the longer term trajectory is clear.
This resonates a lot. I once wrote about what I’d say is a specific instantiation of this concept, namely that people often ask questions that appear to have a binary return type, when that is, in fact, just the wrong return type to look for. Many binary-sounding questions actually require a return type of “function that takes a potentially large number of input values and returns a boolean [or sometimes even something more complex than a boolean]”. When looking for and debating binary answers to such questions, one sweeps the most interesting parts of the answer under the rug.