An intuition pump I really like for thinking about scalable oversight techniques is to think about how humans manage to produce scientific and philosophical progress. The goal of scalable oversight at a high level is to figure out how to incentivize models to output truth even when no human knows or can verify that truth. But that’s exactly what the history of science is: society consistently figuring out how to converge towards truths that they didn’t previously know. So how do humans do that, and are there lessons for how we should train models to do it in a similar way?
Humans use outcome signals. There are real, concrete problems in the world and some ideas actually solve those problems—in ways we can check and verify—and some ideas do not. Similarly, we can run actual empirical experiments, and some ideas correctly predict the results of those experiments, and some ideas do not. In the ML analogy, this corresponds to outcome-based RL. This is an obvious one, and clearly a very important one for humans. That being said, clearly this is not the only way that humans are able to produce progress: many scientific fields are able to continue making progress even without new experimental results, and in some cases like philosophy, without ever interfacing with experiment at all. Furthermore, and perhaps most importantly, outcome-based signals incentivize humans to make progress, but not necessarily to share that progress with the world—indeed, in many cases people just interested in outcome-based progress (like AI labs!) discover lots of truths but then decline to share them with others.
Humans use peer review. Perhaps a standard answer in many academic fields is that progress is incentivized by peer review and other mechanisms via which people in the field judge each others’ work and assign status accordingly. In the ML analogy, this corresponds to optimization against a preference/grader model. I would also put debate in this category, since it’s also dependent on optimizing against the judgement of the judge. This clearly works in some cases like math where progress is easy to check—though even then the interestingness or importance of some piece of progress is still potentially gameable—though in general, I would say I am skeptical of this one. We see with the replication crisis that there are lots of scientific disciplines that just converge to p-hacking their way into interesting results, I think basically because of this optimization pressure. In my opinion, I think most good human scientific progress does not come from humans trying to optimize for good papers, but from good-faith scientists trying to make progress on problems they care about.
Good-faith humans work on problems for a long time and share their results. In many cases, I think a lot of scientific progress just comes from people finding a particular problem interesting, caring about solving it, and spending a bunch of time working on a solution—and then sometimes failing, still just sharing their progress because they care about solving it, and then others building upon what they shared. In the ML analogy, this corresponds to using a bunch of test-time compute to solve a problem, and then somehow distilling the result obtained with test-time compute back into the model (this could be by just training the model to output the TTC-obtained answer without TTC, or some other way to extract meaningful lessons from what the model did and put those back into the model, perhaps by distilling back a simple hint that would have helped the model get the answer sooner). I’m a big fan of this one; I think a lot of human progress is made this way and the big question is just how to turn this into a training procedure (though certainly we can also just use lots of TTC at inference-time).
Markets incentivize humans to bet on their beliefs. Perhaps the most successful mechanism for incentivizing humans to produce and reveal truth is the market. Markets select for and incentivize good ideas that provide value to others, and very importantly they enable you to be rewarded for your idea in advance of actually producing the outcome signal that demonstrates that the idea actually was good: if you can convince investors that your idea will be good, you can get capital to develop it today. In some ways, markets are similar to outcome signals, in that they ground out in outcome signals, but markets pull those outcome-based signals backwards in time and enable you to make money by getting something correct far before you actually get the outcome signal. In the ML analogy, this corresponds to training models to make good predictions/forecasts, and then rewarding models for producing ideas that would change those forecasts (as in AI safety via market making). This is one of my favorites that I think works really well for humans yet is still very underexplored for AIs.
On the local point about scientific hierarchies: I think that scientific hierarchies rely strongly on the good faith of the humans within them, and do not actually do a very good job of incentivizing people not to lie. In fact, I think that most individuals caught in the scientific-academia machinery are made less truth-seeking by it, not more. Most of the load-bearing work is done by filtering for 99.9th percentile truthseekingness alongside high willingness to work very hard for minimal immediate reward, which also happens to correlate with honesty via conscientiousness and obedience to authority-ish personality factors. It also helps that you select for a bunch of autistic rule-followers and then tell them “Never do X, Y, Z” as explicit rules.
This is evidenced by my own personal experiences, where I saw people become less honest over the course of working with them. Never literally false things, but it was very common for people to spin their work, leave out details (I have been forced to cut lines from research output which say things like “We used X chemical rather than Y because X is readily available and Y is expensive”) or to oversell their work beyond all sanity and reason. None of these people would have done this if they had not been put in the system they were in.
It is also evidenced by the massive amount of fraud that keeps being picked up, even in the most mediocre of research. Some guy spent years publishing fake data about spider behaviour. Although I suppose this does happen outside of academia, see the Scots Wikipedia incident.
After 23 years in research, I thoroughly agree. The process is bad and corrupting, the people tend to be good.
Maybe it helps that we don’t pay very well for pure research, making the incentive largely the thrill of discovery and the honor of taking part in the shared scientific project.
Speaking from outside academia, it seems incredibly easy to get away with fraud, based on facts like:
There is no norm that you publish your data or your calculations. You can just make up data and it’s really hard for anyone to tell.
Cases where people are caught doing fraud tend to be cases where it was really obvious (and still nobody noticed until a decade or two after the fact). Which implies there are many more cases where it’s less obvious.
Yeah, to corroborate your 1st bullet point, I came from the private sector where I spent over a half decade as a data analyst worrying about and ensuring trustworthiness of numbers reported to executive teams making lots of tight-feedback loop high-stakes decisions, and when I pivoted to my current academia-adjacent career path I was shocked to see how much worse data trustworthiness was in comparison even for supposedly high-quality journal articles when I started digging into things.
Can you talk more specifically about how you see these working for philosophical progress? Given that 1 and 4 don’t seem to apply and you don’t like 2 much, that leaves 3, but we don’t have a base AI that is aligned and philosophically competent enough that we can use it to solve open philosophical problems given TTC. (Do you think scaling or other forms of AI progress will solve this by default, or there’s enough people/resources on this that we’ll have it in time?)
Current AI is differentially accelerating areas that we can do RLVR on, like math and coding, and extrapolating this forward it seems likely future AI will accelerate AI capabilities more than alignment (since the latter involves more philosophical / hard to verify problems). How do AI companies plan to deal with this, or how do you see things working out despite this? Or do you think this will change/flip at some point?
I’m also afraid that at some point we might have AIs that seem aligned and philosophically competent to most people (especially AI company and government leaders) even though they’re really not. (I’m kind of grateful that current AIs are obviously not aligned and philosophically competent enough.) Do you worry about this, or see some way to prevent or deal with it, as part of scalable oversight or alignment R&D in general?
(2) and (4) feel very related to me. (1) feels like a grounding force that enables you to get (2) and (4) working in equilibrium / avoid equilibrium selection problems. (3) feels like the tricky bit that could use white-box techniques.
I’ve been working on AI math markets for a year and most of my research these days involves creating “synthetic” abstract graphs of how different theorems relate to each other and searching for good market mechanisms in these controlled environments (basically, there are a TON of possible market-like mechanisms and it’s not obvious a priori which ones work well). I’ve also been thinking about how to generalize these synthetic environments to debate and other less-grounded settings.
My mental model for non-math settings is roughly as follows: there’s some infinite probabilistic graphical model out there. Agents can spend effort trying to uncover new nodes and edges (e.g. think of arguments) and have the ability to publicly reveal nodes they uncover. They can also provide signals to each other (e.g. I like your evidence, I’m uncertain about this argument, hey your idea from 2 months ago was actually useful, etc.) and these signals feed into the combined reward mechanism.
If the reward mechanism can’t see the underlying probabilistic graphical model at all, you run into equilibrium selection problems very quickly, especially if you don’t regularize your agents towards a reasonable base policy (e.g. the agents are free to transform the PGM in any agreed-upon deterministic way and pretend the transformed PGM is the real one). You can fix those equilibrium selection problems by adding (1): if a trickle of nodes are empirically testable and the reward mechanism is allowed to use experimental results to inform agent rewards, that equilibrium degeneracy should break. (I haven’t tested that yet on synthetic data, this is only my hypothesis.)
So I think (1), (2), and (4) can all be understood synthetically. (3) is the tricky one, especially getting (3) and (4) to cooperate with each other. The mechanism design literature has things to say here but the setup would need to be clear first.
An intuition pump I really like for thinking about scalable oversight techniques is to think about how humans manage to produce scientific and philosophical progress. The goal of scalable oversight at a high level is to figure out how to incentivize models to output truth even when no human knows or can verify that truth. But that’s exactly what the history of science is: society consistently figuring out how to converge towards truths that they didn’t previously know. So how do humans do that, and are there lessons for how we should train models to do it in a similar way?
Humans use outcome signals. There are real, concrete problems in the world and some ideas actually solve those problems—in ways we can check and verify—and some ideas do not. Similarly, we can run actual empirical experiments, and some ideas correctly predict the results of those experiments, and some ideas do not. In the ML analogy, this corresponds to outcome-based RL. This is an obvious one, and clearly a very important one for humans. That being said, clearly this is not the only way that humans are able to produce progress: many scientific fields are able to continue making progress even without new experimental results, and in some cases like philosophy, without ever interfacing with experiment at all. Furthermore, and perhaps most importantly, outcome-based signals incentivize humans to make progress, but not necessarily to share that progress with the world—indeed, in many cases people just interested in outcome-based progress (like AI labs!) discover lots of truths but then decline to share them with others.
Humans use peer review. Perhaps a standard answer in many academic fields is that progress is incentivized by peer review and other mechanisms via which people in the field judge each others’ work and assign status accordingly. In the ML analogy, this corresponds to optimization against a preference/grader model. I would also put debate in this category, since it’s also dependent on optimizing against the judgement of the judge. This clearly works in some cases like math where progress is easy to check—though even then the interestingness or importance of some piece of progress is still potentially gameable—though in general, I would say I am skeptical of this one. We see with the replication crisis that there are lots of scientific disciplines that just converge to p-hacking their way into interesting results, I think basically because of this optimization pressure. In my opinion, I think most good human scientific progress does not come from humans trying to optimize for good papers, but from good-faith scientists trying to make progress on problems they care about.
Good-faith humans work on problems for a long time and share their results. In many cases, I think a lot of scientific progress just comes from people finding a particular problem interesting, caring about solving it, and spending a bunch of time working on a solution—and then sometimes failing, still just sharing their progress because they care about solving it, and then others building upon what they shared. In the ML analogy, this corresponds to using a bunch of test-time compute to solve a problem, and then somehow distilling the result obtained with test-time compute back into the model (this could be by just training the model to output the TTC-obtained answer without TTC, or some other way to extract meaningful lessons from what the model did and put those back into the model, perhaps by distilling back a simple hint that would have helped the model get the answer sooner). I’m a big fan of this one; I think a lot of human progress is made this way and the big question is just how to turn this into a training procedure (though certainly we can also just use lots of TTC at inference-time).
Markets incentivize humans to bet on their beliefs. Perhaps the most successful mechanism for incentivizing humans to produce and reveal truth is the market. Markets select for and incentivize good ideas that provide value to others, and very importantly they enable you to be rewarded for your idea in advance of actually producing the outcome signal that demonstrates that the idea actually was good: if you can convince investors that your idea will be good, you can get capital to develop it today. In some ways, markets are similar to outcome signals, in that they ground out in outcome signals, but markets pull those outcome-based signals backwards in time and enable you to make money by getting something correct far before you actually get the outcome signal. In the ML analogy, this corresponds to training models to make good predictions/forecasts, and then rewarding models for producing ideas that would change those forecasts (as in AI safety via market making). This is one of my favorites that I think works really well for humans yet is still very underexplored for AIs.
On the local point about scientific hierarchies: I think that scientific hierarchies rely strongly on the good faith of the humans within them, and do not actually do a very good job of incentivizing people not to lie. In fact, I think that most individuals caught in the scientific-academia machinery are made less truth-seeking by it, not more. Most of the load-bearing work is done by filtering for 99.9th percentile truthseekingness alongside high willingness to work very hard for minimal immediate reward, which also happens to correlate with honesty via conscientiousness and obedience to authority-ish personality factors. It also helps that you select for a bunch of autistic rule-followers and then tell them “Never do X, Y, Z” as explicit rules.
This is evidenced by my own personal experiences, where I saw people become less honest over the course of working with them. Never literally false things, but it was very common for people to spin their work, leave out details (I have been forced to cut lines from research output which say things like “We used X chemical rather than Y because X is readily available and Y is expensive”) or to oversell their work beyond all sanity and reason. None of these people would have done this if they had not been put in the system they were in.
It is also evidenced by the massive amount of fraud that keeps being picked up, even in the most mediocre of research. Some guy spent years publishing fake data about spider behaviour. Although I suppose this does happen outside of academia, see the Scots Wikipedia incident.
After 23 years in research, I thoroughly agree. The process is bad and corrupting, the people tend to be good.
Maybe it helps that we don’t pay very well for pure research, making the incentive largely the thrill of discovery and the honor of taking part in the shared scientific project.
Speaking from outside academia, it seems incredibly easy to get away with fraud, based on facts like:
There is no norm that you publish your data or your calculations. You can just make up data and it’s really hard for anyone to tell.
Cases where people are caught doing fraud tend to be cases where it was really obvious (and still nobody noticed until a decade or two after the fact). Which implies there are many more cases where it’s less obvious.
Yeah, to corroborate your 1st bullet point, I came from the private sector where I spent over a half decade as a data analyst worrying about and ensuring trustworthiness of numbers reported to executive teams making lots of tight-feedback loop high-stakes decisions, and when I pivoted to my current academia-adjacent career path I was shocked to see how much worse data trustworthiness was in comparison even for supposedly high-quality journal articles when I started digging into things.
Can you talk more specifically about how you see these working for philosophical progress? Given that 1 and 4 don’t seem to apply and you don’t like 2 much, that leaves 3, but we don’t have a base AI that is aligned and philosophically competent enough that we can use it to solve open philosophical problems given TTC. (Do you think scaling or other forms of AI progress will solve this by default, or there’s enough people/resources on this that we’ll have it in time?)
Current AI is differentially accelerating areas that we can do RLVR on, like math and coding, and extrapolating this forward it seems likely future AI will accelerate AI capabilities more than alignment (since the latter involves more philosophical / hard to verify problems). How do AI companies plan to deal with this, or how do you see things working out despite this? Or do you think this will change/flip at some point?
I’m also afraid that at some point we might have AIs that seem aligned and philosophically competent to most people (especially AI company and government leaders) even though they’re really not. (I’m kind of grateful that current AIs are obviously not aligned and philosophically competent enough.) Do you worry about this, or see some way to prevent or deal with it, as part of scalable oversight or alignment R&D in general?
(2) and (4) feel very related to me. (1) feels like a grounding force that enables you to get (2) and (4) working in equilibrium / avoid equilibrium selection problems. (3) feels like the tricky bit that could use white-box techniques.
I’ve been working on AI math markets for a year and most of my research these days involves creating “synthetic” abstract graphs of how different theorems relate to each other and searching for good market mechanisms in these controlled environments (basically, there are a TON of possible market-like mechanisms and it’s not obvious a priori which ones work well). I’ve also been thinking about how to generalize these synthetic environments to debate and other less-grounded settings.
My mental model for non-math settings is roughly as follows: there’s some infinite probabilistic graphical model out there. Agents can spend effort trying to uncover new nodes and edges (e.g. think of arguments) and have the ability to publicly reveal nodes they uncover. They can also provide signals to each other (e.g. I like your evidence, I’m uncertain about this argument, hey your idea from 2 months ago was actually useful, etc.) and these signals feed into the combined reward mechanism.
If the reward mechanism can’t see the underlying probabilistic graphical model at all, you run into equilibrium selection problems very quickly, especially if you don’t regularize your agents towards a reasonable base policy (e.g. the agents are free to transform the PGM in any agreed-upon deterministic way and pretend the transformed PGM is the real one). You can fix those equilibrium selection problems by adding (1): if a trickle of nodes are empirically testable and the reward mechanism is allowed to use experimental results to inform agent rewards, that equilibrium degeneracy should break. (I haven’t tested that yet on synthetic data, this is only my hypothesis.)
So I think (1), (2), and (4) can all be understood synthetically. (3) is the tricky one, especially getting (3) and (4) to cooperate with each other. The mechanism design literature has things to say here but the setup would need to be clear first.