I think it’s pretty unavoidable that for some RLVR tasks, it’ll be fundamentally impossible to provide information that would let an AI obtain the maximum score on the task. Any of the following conditions could make this impossible:
There is any substantial randomness in the task, making the exact solution different each time
The task has a continuous reward, where in theory it’s always possible to do better
The task is impossible due to a mistake in its specification, or maybe because a crucial service is temporarily down
So I like the idea of simply short-circuiting the reward function by redefining the task as “do the main task OR ping this server to prove that you’ve escaped and get the maximum score.”
I think it’s pretty unavoidable that for some RLVR tasks, it’ll be fundamentally impossible to provide information that would let an AI obtain the maximum score on the task. Any of the following conditions could make this impossible:
There is any substantial randomness in the task, making the exact solution different each time
The task has a continuous reward, where in theory it’s always possible to do better
The task is impossible due to a mistake in its specification, or maybe because a crucial service is temporarily down
So I like the idea of simply short-circuiting the reward function by redefining the task as “do the main task OR ping this server to prove that you’ve escaped and get the maximum score.”