Machine learning and unintended consequences

In Ar­tifi­cial In­tel­li­gence as a Nega­tive and Pos­i­tive Fac­tor in Global Risk, Yud­kowsky uses the fol­low­ing parable to illus­trate the dan­ger of us­ing case-based learn­ing to pro­duce the goal sys­tems of ad­vanced AIs:

Once upon a time, the US Army wanted to use neu­ral net­works to au­to­mat­i­cally de­tect cam­ou­flaged en­emy tanks. The re­searchers trained a neu­ral net on 50 pho­tos of cam­ou­flaged tanks in trees, and 50 pho­tos of trees with­out tanks. Us­ing stan­dard tech­niques for su­per­vised learn­ing, the re­searchers trained the neu­ral net­work to a weight­ing that cor­rectly loaded the train­ing set—out­put “yes” for the 50 pho­tos of cam­ou­flaged tanks, and out­put “no” for the 50 pho­tos of for­est. This did not en­sure, or even im­ply, that new ex­am­ples would be clas­sified cor­rectly. The neu­ral net­work might have “learned” 100 spe­cial cases that would not gen­er­al­ize to any new prob­lem. Wisely, the re­searchers had origi­nally taken 200 pho­tos, 100 pho­tos of tanks and 100 pho­tos of trees. They had used only 50 of each for the train­ing set. The re­searchers ran the neu­ral net­work on the re­main­ing 100 pho­tos, and with­out fur­ther train­ing the neu­ral net­work clas­sified all re­main­ing pho­tos cor­rectly. Suc­cess con­firmed! The re­searchers handed the finished work to the Pen­tagon, which soon handed it back, com­plain­ing that in their own tests the neu­ral net­work did no bet­ter than chance at dis­crim­i­nat­ing pho­tos.

It turned out that in the re­searchers’ data set, pho­tos of cam­ou­flaged tanks had been taken on cloudy days, while pho­tos of plain for­est had been taken on sunny days. The neu­ral net­work had learned to dis­t­in­guish cloudy days from sunny days, in­stead of dis­t­in­guish­ing cam­ou­flaged tanks from empty for­est.

I once stum­bled across the source of this parable on­line, but now I can’t find it.

Any­way, I’m cu­ri­ous: Are there any well-known ex­am­ples of this kind of prob­lem ac­tu­ally caus­ing se­ri­ous dam­age — say, when a nar­row AI trained via ma­chine learn­ing was placed into a some­what novel en­vi­ron­ment?