As far as open source, the quick argument is that once AI becomes sufficiently powerful, it’s unlikely that the incentives are toward open sourcing it (including goverment incentives). This isn’t totally obvious though, and this doesn’t rule out catastrophic bioterrorism (more like COVID scale than extinction scale) prior to AI powerful enough to substantially accelerate R&D across many sectors (including bio). It also doesn’t rule out powerful AI being open sourced years after it is first created (though the world might be radically transformed by this point anyway). I don’t have that much of an inside view on this, but reasonable people I talk to are skeptical that open source is a very big deal (in >20% of worlds) from at least an x-risk perspective. (Seems very sensitive to questions about government response, how much stuff is driven by ideology, and how much people end up being compelled (rightly or not) by “commoditize your complement” (and ecosystem) economic arguments.)
Open source seems good on current margins, at least to the extent it doesn’t leak algorithmic advances / similar.
As far as open source, the quick argument is that once AI becomes sufficiently powerful, it’s unlikely that the incentives are toward open sourcing it (including goverment incentives). This isn’t totally obvious though, and this doesn’t rule out catastrophic bioterrorism (more like COVID scale than extinction scale) prior to AI powerful enough to substantially accelerate R&D across many sectors (including bio). It also doesn’t rule out powerful AI being open sourced years after it is first created (though the world might be radically transformed by this point anyway). I don’t have that much of an inside view on this, but reasonable people I talk to are skeptical that open source is a very big deal (in >20% of worlds) from at least an x-risk perspective. (Seems very sensitive to questions about government response, how much stuff is driven by ideology, and how much people end up being compelled (rightly or not) by “commoditize your complement” (and ecosystem) economic arguments.)
Open source seems good on current margins, at least to the extent it doesn’t leak algorithmic advances / similar.
I would be happy to discuss in a dialogue about this. This seems to be an important topic, and I’m really unsure about many parameters here.