One more example from aviation safety: a pilot put his son on a steering wheel, knowing that it is turned off and the plane is controlled by autopilot. However, after the wheel was turned 15 degrees, the autopilot turned off, as it has a new feature of “corrigibility” and the plane crashed.
“The pilots, who had previously flown Russian-designed planes that had audible warning signals, apparently failed to notice it.”
Unlike Russian planes, apparently designed for dummies, the Airbus designers assumed that the pilots would be adequately trained. And sober. Which is not an unreasonable assumption.
Safety systems must be foolproof. (I am now in the airport and is going to board a Russian plane which will fly almost the same rout as the one which had a catastrophic fire a few day ago.)
I have posted some time ago about the Boeing 737 Max 8 MCAS system as an example of incorrigibility.
I think this is actually the best example given so far here of incorrigibility in concrete systems.
One more example from aviation safety: a pilot put his son on a steering wheel, knowing that it is turned off and the plane is controlled by autopilot. However, after the wheel was turned 15 degrees, the autopilot turned off, as it has a new feature of “corrigibility” and the plane crashed.
https://en.wikipedia.org/wiki/Aeroflot_Flight_593
“The pilots, who had previously flown Russian-designed planes that had audible warning signals, apparently failed to notice it.”
Unlike Russian planes, apparently designed for dummies, the Airbus designers assumed that the pilots would be adequately trained. And sober. Which is not an unreasonable assumption.
Safety systems must be foolproof. (I am now in the airport and is going to board a Russian plane which will fly almost the same rout as the one which had a catastrophic fire a few day ago.)