I have the following obvious question: for example, we create an AI which ignores risks below 0.0001 probability and we start to use it as a mind of a self-driving car. Such AI will often pass other cars on incoming line (or ignore false positive about pedestrians, if we want more realist example), but after 10 000 passing it will have horrific accident., which will happen only after just a few years of driving: so it will be unsafe, but not in the same way as a paperclipper. What could be done about it?
I have the following obvious question: for example, we create an AI which ignores risks below 0.0001 probability and we start to use it as a mind of a self-driving car. Such AI will often pass other cars on incoming line (or ignore false positive about pedestrians, if we want more realist example), but after 10 000 passing it will have horrific accident., which will happen only after just a few years of driving: so it will be unsafe, but not in the same way as a paperclipper. What could be done about it?