Some agendas are step-function. Maybe interuniversal teichmuller theory is an example, if it reaches the threshold where it solves ABC then it pays out, and if it doesn’t its payout is approximately zero[1]. Something something MIRI’s logic team[2].
Other agendas are monotonic. The more you do, the more you pay out. I think what I do, program synthesis security (which Mike Dodds calls “scalable formal oversight”) for defensive acceleration, is like this. I might be able to reduce some monotonic function of 40% of the attack surface by doing 40% of my job, its better to do more but not a waste of time to do less. This is related to, but distinct from, what John and David sometimes talk about re multi-theory-of-change robustness (i.e. that robustness can be a route toward an agenda being monotonic).
Not sure how to reason about “linear vs superlinear vs sublinear” here. How would you even tell?
caveat, mathematicians don’t make a habit of making bets on which tool stacks will prove useful for what purposes. ITT may or may not end up helping with something other than the ABC conjecture (which it failed at), MIRI’s logic team outputs may end up (and, IMO at least in terms of my own epistemic development, have) helping with something other than solving AGI alignment.
tonal clarification, I have a deep respect for the MIRI logic team, I think they displayed both courage and intellectual prowess that are aeons beyond me. I wouldn’t call them out here if I didn’t think many of them agree that what they were trying to do was mostly step-function.
step-function research and monotonic research.
Some agendas are step-function. Maybe interuniversal teichmuller theory is an example, if it reaches the threshold where it solves ABC then it pays out, and if it doesn’t its payout is approximately zero [1] . Something something MIRI’s logic team [2] .
Other agendas are monotonic. The more you do, the more you pay out. I think what I do, program synthesis security (which Mike Dodds calls “scalable formal oversight”) for defensive acceleration, is like this. I might be able to reduce some monotonic function of 40% of the attack surface by doing 40% of my job, its better to do more but not a waste of time to do less. This is related to, but distinct from, what John and David sometimes talk about re multi-theory-of-change robustness (i.e. that robustness can be a route toward an agenda being monotonic).
Not sure how to reason about “linear vs superlinear vs sublinear” here. How would you even tell?
caveat, mathematicians don’t make a habit of making bets on which tool stacks will prove useful for what purposes. ITT may or may not end up helping with something other than the ABC conjecture (which it failed at), MIRI’s logic team outputs may end up (and, IMO at least in terms of my own epistemic development, have) helping with something other than solving AGI alignment.
tonal clarification, I have a deep respect for the MIRI logic team, I think they displayed both courage and intellectual prowess that are aeons beyond me. I wouldn’t call them out here if I didn’t think many of them agree that what they were trying to do was mostly step-function.