It is my impression that there are at least some examples in which this is done in practice: as far as I know, in rocket design you do in fact calculate those for most components, including software used on the on-board computers. This information is used to e.g. decide on the amount of duplication of electronics components in critical systems of the rocket. I am, however, not an expert on rockets.
It seems plausible that at least in some concepts, we can indeed build safeguards that have a certain efficiency that we know at reducing our overall risk. Even if this is true only sometimes, than it would be useful to have a way to calculate the maximum allowed risk levels for extinction-like events.
Incidentally, I am also of the opinion that having any kind of calculation would work better than making a non-zero extinction risk taboo, or not subject to negotiation (which seems to be the case currently).
However of course, I am not claiming that my idea is so great. I stand behind my opinion that we need some such system to make sensible tradeoffs on “emissions” of existential risk.
The upper bound is nearly always that there a black swan reason that makes you destroy the world.
Ah, I see you added this part.
I generally agree. Still, sometimes you’ll want something to guide your design even if you know that there might be some such black swan. You are surely not suggesting that existence of black swans is enough to make us abandon all effort and do whatever.
It is my impression that there are at least some examples in which this is done in practice: as far as I know, in rocket design you do in fact calculate those for most components, including software used on the on-board computers. This information is used to e.g. decide on the amount of duplication of electronics components in critical systems of the rocket. I am, however, not an expert on rockets.
Of course it’s possible to do risk calculations. At the same time that doesn’t mean that you are safe. Long-Term Capital Management exploded despite having low “verified upper bound” risk in the sense you speak about risk.
Incidentally, I am also of the opinion that having any kind of calculation would work better than making a non-zero extinction risk taboo, or not subject to negotiation (which seems to be the case currently).
Calculation of risk often leads to people taking more risk because they believe that the models of the risk they have accurately describe the risk.
Long-Term Capital Management exploded despite having low “verified upper bound” risk in the sense you speak about risk.
But it might be that some of these banks had a blind spot there. If there were outside estimates that carry part of the risk then it might have looked different. Insurers have reinsurance for that. And I think a risk market might improve on that.
But it might be that some of these banks had a blind spot there.
Every model has blind spots. That’s the nature of models. If you price risk by a specific model, people take less risk in your model and often take more risk that’s not part of the model.
It’s a systematic issue and if you want to get deeper into it read Antifragile or The Black Swan.
If you launch rockets, than it might be okay to assume that your risk model is good enough to optimize for it. If you are on the other hand talking about risk from UFAI there’s no reason to assume that you understand the problem well enough to model it and there a good chance that you take less risk in your model but increase the chance of the Black Swan event that kills you.
I’m quite aware of Black Swans. My suggestion was that some actors might kow about unknown unknowns and be able to make at least some predictions about this. Surely not inside systems that have opposing incentives. But e.g. reinsurer have some need to hedge these. These principles might be built upon. Maybe markets today price in black swans to some degree already.
Math can only tell you about what happens inside your model.
True by construction. Apparently I meant something else.
And I don’t mean it in the sense that a model of physics allows in principle to quantify that. But as a check of premises: Can we agree that known physics would in principle be model that would include the unknown unknowns are a quantifiable term (in principle)?
The known physics don’t allow you to say things about things unknown to model of known physics. Unknown variables that you can describe with the model of physics are known unknowns.
I agree to that. But we can’t get any further if we can’t agree on an intermediate point.
Would you argue about a system where we do not know the specifics of of some behavior of the system (to avoid the word ‘unknown’) but where we can know something about the (e.g. the probability mass) outside of the known specific behavior but still inside some general model of the system.
The known specific behavior is “known knowns” and not “known unknowns”. There are certainly known unknowns over which you can make valuable statements.
But we can’t get any further if we can’t agree on an intermediate point.
Accepting the limits of what one can know is important. That does often mean that one can’t go further.
Yes, the known specific behavior is known known. But I’m talking about the general behavior. Where we do not know specifics of but which is still within the general model? How do you call these?
Why do you think this happens to be the case?
The upper bound is nearly always that there a black swan reason that makes you destroy the world.
It is my impression that there are at least some examples in which this is done in practice: as far as I know, in rocket design you do in fact calculate those for most components, including software used on the on-board computers. This information is used to e.g. decide on the amount of duplication of electronics components in critical systems of the rocket. I am, however, not an expert on rockets.
It seems plausible that at least in some concepts, we can indeed build safeguards that have a certain efficiency that we know at reducing our overall risk. Even if this is true only sometimes, than it would be useful to have a way to calculate the maximum allowed risk levels for extinction-like events.
Incidentally, I am also of the opinion that having any kind of calculation would work better than making a non-zero extinction risk taboo, or not subject to negotiation (which seems to be the case currently).
However of course, I am not claiming that my idea is so great. I stand behind my opinion that we need some such system to make sensible tradeoffs on “emissions” of existential risk.
Ah, I see you added this part.
I generally agree. Still, sometimes you’ll want something to guide your design even if you know that there might be some such black swan. You are surely not suggesting that existence of black swans is enough to make us abandon all effort and do whatever.
Of course it’s possible to do risk calculations. At the same time that doesn’t mean that you are safe. Long-Term Capital Management exploded despite having low “verified upper bound” risk in the sense you speak about risk.
Calculation of risk often leads to people taking more risk because they believe that the models of the risk they have accurately describe the risk.
But it might be that some of these banks had a blind spot there. If there were outside estimates that carry part of the risk then it might have looked different. Insurers have reinsurance for that. And I think a risk market might improve on that.
Every model has blind spots. That’s the nature of models. If you price risk by a specific model, people take less risk in your model and often take more risk that’s not part of the model.
It’s a systematic issue and if you want to get deeper into it read Antifragile or The Black Swan.
If you launch rockets, than it might be okay to assume that your risk model is good enough to optimize for it. If you are on the other hand talking about risk from UFAI there’s no reason to assume that you understand the problem well enough to model it and there a good chance that you take less risk in your model but increase the chance of the Black Swan event that kills you.
I’m quite aware of Black Swans. My suggestion was that some actors might kow about unknown unknowns and be able to make at least some predictions about this. Surely not inside systems that have opposing incentives. But e.g. reinsurer have some need to hedge these. These principles might be built upon. Maybe markets today price in black swans to some degree already.
By the definition of unknown unknowns, they aren’t known.
Long-Term Capital Management did hedge their risk with their “Noble prize”-winning formulas.
Math. Can sometimes surprisingly say something about the unknown.
Social effects. Long-Term Capital Management maybe didn’t want to see the limits of their approach.
Math can only tell you about what happens inside your model. It can tell you something about known unknowns.
Their approach was that they thought risk can be measured with modern portfolio theory for which their funders got the “Nobel”.
It’s not that different from how you don’t want to see the limits.
And I don’t mean it in the sense that a model of physics allows in principle to quantify that. But as a check of premises: Can we agree that known physics would in principle be model that would include the unknown unknowns are a quantifiable term (in principle)?
The known physics don’t allow you to say things about things unknown to model of known physics. Unknown variables that you can describe with the model of physics are known unknowns.
I agree to that. But we can’t get any further if we can’t agree on an intermediate point.
Would you argue about a system where we do not know the specifics of of some behavior of the system (to avoid the word ‘unknown’) but where we can know something about the (e.g. the probability mass) outside of the known specific behavior but still inside some general model of the system.
The known specific behavior is “known knowns” and not “known unknowns”. There are certainly known unknowns over which you can make valuable statements.
Accepting the limits of what one can know is important. That does often mean that one can’t go further.
Yes, the known specific behavior is known known. But I’m talking about the general behavior. Where we do not know specifics of but which is still within the general model? How do you call these?
“known unknowns” describes a model where you have unknown variables but you know which variables you don’t know.
OK with that terminology we can agree.