I endorse a weaker heuristic which is like: “Don’t work on safety measures that deal with non-catastrophic risks from current models but would obviously break down by TED-AI.”
Examples:
Don’t work on reducing reward hacking, unless your technique would deal with eval-aware sophisticated AIs with long-horizon.
Don’t work on untrusted monitoring, unless your technique is robust to collusion.
Don’t work on alignment auditing, unless your technique would work on a misaligned TED-AI.
That said, I think that metrics are pretty helpful for TED-robustness. You can push for metrics which would measure safety under pessimistic assumptions. For example, your metrics for a monitoring protocol could be: “How easy is this to subvert if the monitor is colluding with the policy?” or “How well does the monitor work if it doesn’t have access to the chains-of-thought?”
Stop doing “safety” work for the uncontrollable doomsday weapon companies. That is part of what gets everyone killed.
I endorse a weaker heuristic which is like: “Don’t work on safety measures that deal with non-catastrophic risks from current models but would obviously break down by TED-AI.”
Examples:
Don’t work on reducing reward hacking, unless your technique would deal with eval-aware sophisticated AIs with long-horizon.
Don’t work on untrusted monitoring, unless your technique is robust to collusion.
Don’t work on alignment auditing, unless your technique would work on a misaligned TED-AI.
That said, I think that metrics are pretty helpful for TED-robustness. You can push for metrics which would measure safety under pessimistic assumptions. For example, your metrics for a monitoring protocol could be: “How easy is this to subvert if the monitor is colluding with the policy?” or “How well does the monitor work if it doesn’t have access to the chains-of-thought?”