There might be indirect effects like increasing hype around AI and thus investment, but overall I think those effects are small and I’m not even sure about the sign.
Sign of the effect of open source on hype? Or of hype on timelines? I’m not sure why either would be negative.
Open source --> more capabilities R&D --> more profitable applications --> more profit/investment --> shorter timelines
The example I’ve heard cited is Stable Diffusion leading to LORA.
There’s a countervailing effect of democratizing safety research, which one might think outweighs because it’s so much more neglected than capabilities, more low-hanging fruit.
Sign of the effect of open source on hype? Or of hype on timelines? I’m not sure why either would be negative.
By “those effects” I meant a collection of indirect “release weights → capability landscape changes” effects in general, not just hype/investment. And by “sign” I meant whether those effects taken together are good or bad. Sorry, I realize that wasn’t very clear.
As examples, there might be a mildly bad effect through increased investment, and/or there might be mildly good effects through more products and more continuous takeoff.
I agree that releasing weights probably increases hype and investment if anything. I also think that right now, democratizing safety research probably outweighs all those concerns, which is why I’m mainly worried about Meta etc. not having very clear (and reasonable) decision criteria for when they’ll stop releasing weights.
There’s a countervailing effect of democratizing safety research, which one might think outweighs because it’s so much more neglected than capabilities, more low-hanging fruit.
I take this argument very seriously. It in fact does seem the case that very much of the safety research I’m excited about happens on open source models. Perhaps I’m more plugged into the AI safety research landscape than the capabilities research landscape? Nonetheless, I think not even considering low-hanging-fruit effects, there’s a big reason to believe open sourcing your model will have disproportionate safety gains:
Capabilities research is about how to train your models to be better, but the overall sub-goal of safety research right now seems to be how to verify properties of your model.
Certainly framed like this, releasing the end-states of training (or possibly even training checkpoints) seems better suited to the safety research strategy than the capabilities research strategy.
Sign of the effect of open source on hype? Or of hype on timelines? I’m not sure why either would be negative.
Open source --> more capabilities R&D --> more profitable applications --> more profit/investment --> shorter timelines
The example I’ve heard cited is Stable Diffusion leading to LORA.
There’s a countervailing effect of democratizing safety research, which one might think outweighs because it’s so much more neglected than capabilities, more low-hanging fruit.
By “those effects” I meant a collection of indirect “release weights → capability landscape changes” effects in general, not just hype/investment. And by “sign” I meant whether those effects taken together are good or bad. Sorry, I realize that wasn’t very clear.
As examples, there might be a mildly bad effect through increased investment, and/or there might be mildly good effects through more products and more continuous takeoff.
I agree that releasing weights probably increases hype and investment if anything. I also think that right now, democratizing safety research probably outweighs all those concerns, which is why I’m mainly worried about Meta etc. not having very clear (and reasonable) decision criteria for when they’ll stop releasing weights.
I take this argument very seriously. It in fact does seem the case that very much of the safety research I’m excited about happens on open source models. Perhaps I’m more plugged into the AI safety research landscape than the capabilities research landscape? Nonetheless, I think not even considering low-hanging-fruit effects, there’s a big reason to believe open sourcing your model will have disproportionate safety gains:
Capabilities research is about how to train your models to be better, but the overall sub-goal of safety research right now seems to be how to verify properties of your model.
Certainly framed like this, releasing the end-states of training (or possibly even training checkpoints) seems better suited to the safety research strategy than the capabilities research strategy.