Maybe it would be helpful to start using some toy models of DAGs/tech trees to get an idea of how wide/deep ratios affect the relevant speedups. It sounds like so far that much of this is just people having warring intuitions about ‘no, the tree is deep and narrow and so slowing down/speeding up workers doesn’t have that much effect because Amdahl’s law so I handwave it at ~1x speed’ vs ‘no, I think it’s wide and lots of work-arounds to any slow node if you can pay for the compute to bypass them and I will handwave it at 5x speed’.
This isn’t that important, but I think the idea of using an exponential parallelization penalty is common in the economics literature. I specifically used 0.4 as around the harshest penalty I’ve heard of. I believe this number comes from some studies on software engineering where they found something like this.
I’m currently skeptical that toy models of DAGs/tech trees will add much value over:
Looking at how parallelized AI R&D is right now.
Looking at what people typically find in the economics literature.
(Separately AIs might be notably better at coordinating than humans are which might change things substantially. Toy models of this might be helpful.)
Maybe it would be helpful to start using some toy models of DAGs/tech trees to get an idea of how wide/deep ratios affect the relevant speedups. It sounds like so far that much of this is just people having warring intuitions about ‘no, the tree is deep and narrow and so slowing down/speeding up workers doesn’t have that much effect because Amdahl’s law so I handwave it at ~1x speed’ vs ‘no, I think it’s wide and lots of work-arounds to any slow node if you can pay for the compute to bypass them and I will handwave it at 5x speed’.
This isn’t that important, but I think the idea of using an exponential parallelization penalty is common in the economics literature. I specifically used 0.4 as around the harshest penalty I’ve heard of. I believe this number comes from some studies on software engineering where they found something like this.
I’m currently skeptical that toy models of DAGs/tech trees will add much value over:
Looking at how parallelized AI R&D is right now.
Looking at what people typically find in the economics literature.
(Separately AIs might be notably better at coordinating than humans are which might change things substantially. Toy models of this might be helpful.)