(2) and (4) feel very related to me. (1) feels like a grounding force that enables you to get (2) and (4) working in equilibrium / avoid equilibrium selection problems. (3) feels like the tricky bit that could use white-box techniques.
I’ve been working on AI math markets for a year and most of my research these days involves creating “synthetic” abstract graphs of how different theorems relate to each other and searching for good market mechanisms in these controlled environments (basically, there are a TON of possible market-like mechanisms and it’s not obvious a priori which ones work well). I’ve also been thinking about how to generalize these synthetic environments to debate and other less-grounded settings.
My mental model for non-math settings is roughly as follows: there’s some infinite probabilistic graphical model out there. Agents can spend effort trying to uncover new nodes and edges (e.g. think of arguments) and have the ability to publicly reveal nodes they uncover. They can also provide signals to each other (e.g. I like your evidence, I’m uncertain about this argument, hey your idea from 2 months ago was actually useful, etc.) and these signals feed into the combined reward mechanism.
If the reward mechanism can’t see the underlying probabilistic graphical model at all, you run into equilibrium selection problems very quickly, especially if you don’t regularize your agents towards a reasonable base policy (e.g. the agents are free to transform the PGM in any agreed-upon deterministic way and pretend the transformed PGM is the real one). You can fix those equilibrium selection problems by adding (1): if a trickle of nodes are empirically testable and the reward mechanism is allowed to use experimental results to inform agent rewards, that equilibrium degeneracy should break. (I haven’t tested that yet on synthetic data, this is only my hypothesis.)
So I think (1), (2), and (4) can all be understood synthetically. (3) is the tricky one, especially getting (3) and (4) to cooperate with each other. The mechanism design literature has things to say here but the setup would need to be clear first.
(2) and (4) feel very related to me. (1) feels like a grounding force that enables you to get (2) and (4) working in equilibrium / avoid equilibrium selection problems. (3) feels like the tricky bit that could use white-box techniques.
I’ve been working on AI math markets for a year and most of my research these days involves creating “synthetic” abstract graphs of how different theorems relate to each other and searching for good market mechanisms in these controlled environments (basically, there are a TON of possible market-like mechanisms and it’s not obvious a priori which ones work well). I’ve also been thinking about how to generalize these synthetic environments to debate and other less-grounded settings.
My mental model for non-math settings is roughly as follows: there’s some infinite probabilistic graphical model out there. Agents can spend effort trying to uncover new nodes and edges (e.g. think of arguments) and have the ability to publicly reveal nodes they uncover. They can also provide signals to each other (e.g. I like your evidence, I’m uncertain about this argument, hey your idea from 2 months ago was actually useful, etc.) and these signals feed into the combined reward mechanism.
If the reward mechanism can’t see the underlying probabilistic graphical model at all, you run into equilibrium selection problems very quickly, especially if you don’t regularize your agents towards a reasonable base policy (e.g. the agents are free to transform the PGM in any agreed-upon deterministic way and pretend the transformed PGM is the real one). You can fix those equilibrium selection problems by adding (1): if a trickle of nodes are empirically testable and the reward mechanism is allowed to use experimental results to inform agent rewards, that equilibrium degeneracy should break. (I haven’t tested that yet on synthetic data, this is only my hypothesis.)
So I think (1), (2), and (4) can all be understood synthetically. (3) is the tricky one, especially getting (3) and (4) to cooperate with each other. The mechanism design literature has things to say here but the setup would need to be clear first.