In the post this is called a ‘miracle’ but this has misleading associations – it was not meant to imply a negligible probability, only surprise, so Rob suggested changing it to ‘surprising positive development.’
How about the phrase “positive model violation”? Later in that post Eliezer is recorded as saying:
a miracle that violates some aspect of my background model
…
positive model violation (“miracle”)
I think “model violation” and “surprising development” point to different things. For example:
If I buy a lottery ticket and win a million dollars from the lottery, that is a “surprising development”. If I don’t buy a lottery ticket and still win a million dollars from the lottery, that is also a “model violation”, as it violates our model of how lotteries work.
If my nemesis is struck by lightning from the sky, that is a “surprising development”. If my nemesis is struck by lightning cast by a magic wand, that is also a “model violation”, as it violates our model of lightning.
If many AI researchers independently decide to switch to safety work, that is a “surprising development”. If corrigibility turns out to be “natural” for an AI, that is also a “model violation” of (my model of) Eliezer’s models of corrigibility and intelligence.
My models of lightning and lotteries are relatively robust, and the model violations are negligible probability. Models of the future of AI development and intelligence and geo-politics and human nature and so forth are presumably much weaker, so we can reasonably expect some model violations, positive and negative.
How about the phrase “positive model violation”? Later in that post Eliezer is recorded as saying:
I think “model violation” and “surprising development” point to different things. For example:
If I buy a lottery ticket and win a million dollars from the lottery, that is a “surprising development”. If I don’t buy a lottery ticket and still win a million dollars from the lottery, that is also a “model violation”, as it violates our model of how lotteries work.
If my nemesis is struck by lightning from the sky, that is a “surprising development”. If my nemesis is struck by lightning cast by a magic wand, that is also a “model violation”, as it violates our model of lightning.
If many AI researchers independently decide to switch to safety work, that is a “surprising development”. If corrigibility turns out to be “natural” for an AI, that is also a “model violation” of (my model of) Eliezer’s models of corrigibility and intelligence.
My models of lightning and lotteries are relatively robust, and the model violations are negligible probability. Models of the future of AI development and intelligence and geo-politics and human nature and so forth are presumably much weaker, so we can reasonably expect some model violations, positive and negative.