On alternative terminology: It feels more natural to me to think about it in terms of the inverse, of how much oversight and how carefully-designed controls / checks and balances you have. Also importantly, how much centralization into “one” AI, vs. diverse competing AIs. As you have more powerful AI, to some extent it becomes by default more difficult to oversee, and in a competitive situation there’s a temptation to skimp on those checks and balances. This feels closer to how we think about it in the human case, e.g. there’s a temptation to remove checks and balances from the executive in exchange for getting the trains to run on time, a board might feel like they don’t want to interfere in their genius CEOs actions, etc.
And yeah I was using “unfalsifiable” in a very loose sense, mostly to point at the fact that, without a definition of where you draw the line between e.g. trust handoff and not-trust-handoff, it becomes impossible to definitively say that e.g. a prediction that we’d do “trust handoff” at a given point in time, was wrong.
On alternative terminology: It feels more natural to me to think about it in terms of the inverse, of how much oversight and how carefully-designed controls / checks and balances you have. Also importantly, how much centralization into “one” AI, vs. diverse competing AIs. As you have more powerful AI, to some extent it becomes by default more difficult to oversee, and in a competitive situation there’s a temptation to skimp on those checks and balances. This feels closer to how we think about it in the human case, e.g. there’s a temptation to remove checks and balances from the executive in exchange for getting the trains to run on time, a board might feel like they don’t want to interfere in their genius CEOs actions, etc.
And yeah I was using “unfalsifiable” in a very loose sense, mostly to point at the fact that, without a definition of where you draw the line between e.g. trust handoff and not-trust-handoff, it becomes impossible to definitively say that e.g. a prediction that we’d do “trust handoff” at a given point in time, was wrong.