The metaphor I find myself reaching for is that of the drug addict who will steal from their own family or commit violent crimes to fund their addiction. Taking unsustainable action which they know will be discovered and punished eventually, because the short-term reward hacking is so powerful. Such a person (or AI) can be very very dangerous, but can also potentially still be helpful in alignment research.
The metaphor I find myself reaching for is that of the drug addict who will steal from their own family or commit violent crimes to fund their addiction. Taking unsustainable action which they know will be discovered and punished eventually, because the short-term reward hacking is so powerful. Such a person (or AI) can be very very dangerous, but can also potentially still be helpful in alignment research.