To make this model and little richer and share something of how I think of it, I tend to think of the risk of any particular powerful AI the way I think of risk in deploying software.
I work in site reliability/operations, and so we tend to deal with things we model as having aleatory uncertainty like holding constant a risk that any particular system will fail unexpected for some reason (hardware failure, cosmic rays, unexpected code execution path, etc.), but I also know that most of the risk comes right at the beginning when I first turn something on (turn on new hardware, deploy new code, etc.). A very simple model of this is something like f(x)=e−x+c where most of the risk of failure happens right at the start and beyond that there’s little to no risk of failure, so running for months doesn’t represent a 95% risk; almost all of the 5% risk is eaten up right at the start because the probability distribution function is shaped such that all the mass is under the curve at the beginning.
To make this model and little richer and share something of how I think of it, I tend to think of the risk of any particular powerful AI the way I think of risk in deploying software.
I work in site reliability/operations, and so we tend to deal with things we model as having aleatory uncertainty like holding constant a risk that any particular system will fail unexpected for some reason (hardware failure, cosmic rays, unexpected code execution path, etc.), but I also know that most of the risk comes right at the beginning when I first turn something on (turn on new hardware, deploy new code, etc.). A very simple model of this is something like f(x)=e−x+c where most of the risk of failure happens right at the start and beyond that there’s little to no risk of failure, so running for months doesn’t represent a 95% risk; almost all of the 5% risk is eaten up right at the start because the probability distribution function is shaped such that all the mass is under the curve at the beginning.