This seems mostly right, except that it’s often hard to parallelize work and manage large projects—which seems like it slows thing importantly. And, of course, some things are strongly serialized using time that can’t be sped up via more compute or more people. (See: PM hires 9 women to have baby in one month.)
Similarly, running 1,000 AI research groups in parallel might get you the same 20 insights 50 times, rather than generating far more insights. And managing and integrating the research, and deciding where to allocate research time, plausibly gets harder at more than a linear rate with more groups.
So overall, the model seems correct, but I think the 10x speed up is more likely than the 20x speed up.
I agree parallelization penalties might bite hard in practice. But it’s worth noting that the AIs in the AutomatedCorp hypothetical also run 50x faster and are more capable.
(A strong marginal parallelization penalty exponent of 0.4 would render the 50x additional workers equivalent to a 5x improvement in labor speed, substantially smaller than the 50x speed improvement.)
Maybe it would be helpful to start using some toy models of DAGs/tech trees to get an idea of how wide/deep ratios affect the relevant speedups. It sounds like so far that much of this is just people having warring intuitions about ‘no, the tree is deep and narrow and so slowing down/speeding up workers doesn’t have that much effect because Amdahl’s law so I handwave it at ~1x speed’ vs ‘no, I think it’s wide and lots of work-arounds to any slow node if you can pay for the compute to bypass them and I will handwave it at 5x speed’.
This isn’t that important, but I think the idea of using an exponential parallelization penalty is common in the economics literature. I specifically used 0.4 as around the harshest penalty I’ve heard of. I believe this number comes from some studies on software engineering where they found something like this.
I’m currently skeptical that toy models of DAGs/tech trees will add much value over:
Looking at how parallelized AI R&D is right now.
Looking at what people typically find in the economics literature.
(Separately AIs might be notably better at coordinating than humans are which might change things substantially. Toy models of this might be helpful.)
This seems mostly right, except that it’s often hard to parallelize work and manage large projects—which seems like it slows thing importantly. And, of course, some things are strongly serialized using time that can’t be sped up via more compute or more people. (See: PM hires 9 women to have baby in one month.)
Similarly, running 1,000 AI research groups in parallel might get you the same 20 insights 50 times, rather than generating far more insights. And managing and integrating the research, and deciding where to allocate research time, plausibly gets harder at more than a linear rate with more groups.
So overall, the model seems correct, but I think the 10x speed up is more likely than the 20x speed up.
I agree parallelization penalties might bite hard in practice. But it’s worth noting that the AIs in the AutomatedCorp hypothetical also run 50x faster and are more capable.
(A strong marginal parallelization penalty exponent of 0.4 would render the 50x additional workers equivalent to a 5x improvement in labor speed, substantially smaller than the 50x speed improvement.)
Maybe it would be helpful to start using some toy models of DAGs/tech trees to get an idea of how wide/deep ratios affect the relevant speedups. It sounds like so far that much of this is just people having warring intuitions about ‘no, the tree is deep and narrow and so slowing down/speeding up workers doesn’t have that much effect because Amdahl’s law so I handwave it at ~1x speed’ vs ‘no, I think it’s wide and lots of work-arounds to any slow node if you can pay for the compute to bypass them and I will handwave it at 5x speed’.
This isn’t that important, but I think the idea of using an exponential parallelization penalty is common in the economics literature. I specifically used 0.4 as around the harshest penalty I’ve heard of. I believe this number comes from some studies on software engineering where they found something like this.
I’m currently skeptical that toy models of DAGs/tech trees will add much value over:
Looking at how parallelized AI R&D is right now.
Looking at what people typically find in the economics literature.
(Separately AIs might be notably better at coordinating than humans are which might change things substantially. Toy models of this might be helpful.)