OK, I guess I’m a bit unclear on the problem setup and how it involves a training phase and deployment phase.
John Schulman
I might be missing some context here, but I didn’t understand the section “No Indescribable Hellworlds Hypothesis” and how hellworlds have to do with debate.
The results in Neural Networks Are Fundamentally Bayesian are pretty cool—that’s clever how they were able to estimate the densities.
A couple thoughts on the limitations:
There are various priors over functions for which we can calculate the exact posterior. (E.g., Gaussian processes.) However, doing Bayesian inference on these priors doesn’t perform as well as neural networks on most datasets. So knowing SGD is Bayesian is only interesting if we also know that the prior is interesting. I think the ideal theoretical result would be to show that SGD on neural nets is an approximation of Solomonoff Induction (or something like SI), and the approximation gets better as the NNs get bigger and deeper. But I have yet to see any theory that connects neural nets/ SGD to something like short programs.
If SGD works because it’s Bayesian, then making it more Bayesian should make it work better. But according to https://arxiv.org/abs/2002.02405 that’s not the case. Lowering the temperature, or taking the MAP (=temperature 0) generalizes better than taking the full Bayesian posterior, as calculated by an expensive MCMC procedure.
There’s a decent amount of literature on using multiple rewards, though often it’s framed as learning about multiple goals. Here are some off the top of my head:
The Horde (classic): http://www.ifaamas.org/Proceedings/aamas2011/papers/A6_R70.pdf
Universal Value Function Approximators: http://proceedings.mlr.press/v37/schaul15.html
Learning to Act By Predicting: https://arxiv.org/abs/1611.01779
Temporal Difference Models: https://arxiv.org/abs/1802.09081
Successor Features: https://papers.nips.cc/paper/2017/hash/350db081a661525235354dd3e19b8c05-Abstract.html
Also see the discussion in Appendix D about prediction heads in OpenAI Five, used mostly for interpretability/diagnostics https://cdn.openai.com/dota-2.pdf.
I think this is a good idea. If you go ahead with it, here’s a suggestion.
Reviewers often procrastinate for weeks or months. This is partly because doing a review takes an unbounded amount of time, especially for articles that are long or confusing. So instead of sending the reviewers a manuscript with a due date, book a calendar event for 2 hours with the reviewers. The reviewers join a call or group chat and read the paper and discuss it. They can also help clear each other’s confusions. They aim to complete the review by the end of the time window.
Isn’t the Step 1 objective (the unnormalized posterior log probability of (θ₁, θ₂)) maximized at θ₁ = θ₂=argmax L + prior? Also, I don’t see what this objective has to do with learning a world model.
D’oh, re: the optimum of the objective, I now see that the solution is nontrivial. Here’s my current understanding.
Intuitively, the MAP version of the objective says: find me a simple model theta1 such that there’s more-complex theta2 with high likelihood under p(theta2|theta1) (which corresponds to sampling theta2 near theta1 until theta2 satisfies head-agreement condition) and high data-likelihood p(data|theta2).
And this connects to the previous argument about world models and language as follows: we want theta1 to contain half a world model, and we want theta2 to contain the full world model and high data-likelihood (for one of the head) and the two heads agree. Based on Step1, the problem is still pretty underconstrained, but maybe that’s resolved in Step 2.
Super clear and actionable—my new favorite post on AF.
I also agree with it, and it’s similar to what we’re doing at OpenAI (largely thanks to Paul’s influence).
Basically agree—I think that a model trained by maximum likelihood on offline data is less goal-directed than one that’s trained by an iterative process where you reinforce its own samples (aka online RL), but still somewhat goal directed. It needs to simulate a goal-directed agent to do a good job at maximum likelihood. OTOH it’s mostly concerned with covering all possibilities, so the goal directed reasoning isn’t emphasized. But with multiple iterations, the model can improve quality (-> more goal directedness) at the expense of coverage/diversity.
yup, added a sentence about it
Yeah that’s also good point, though I don’t want to read too much into it, since it might be a historical accident.
In my experience, you need separate teams doing safety research because specialization is useful—it’s easiest to make progress when both individuals and teams specialize a bit and develop taste and mastery of a narrow range of topics.
Agree with what you’ve written here—I think you put it very well.
Performance is mostly limited here by the fact that there are 500 levels for each game (i.e., level overfitting is the problem) so it’s not that meaningful to look at sample efficiency wrt environment interactions. The results would look a lot different on the full distribution of levels. I agree with your statement directionally though.
There’s no PPO/PPG curve there—I’d be curious to see that comparison. (though I agree that QL/MuZero will probably be more sample efficient.)
I’m still not sure how to reconcile your results with the fact that the participants in the procgen contest ended up winning with modifications of our PPO/PPG baselines, rather than Q-learning and other value-based algorithms, whereas your paper suggests that Q-learning performs much better. The contest used 8M timesteps + 200 levels. I assume that your “QL” baseline is pretty similar to widespread DQN implementations.
https://arxiv.org/pdf/2103.15332.pdf
Are there implementation level changes that dramatically improve performance of your QL implementation?
(Currently on vacation and I read your paper briefly while traveling, but I may very well have missed something.)
Thanks, this is very insightful. BTW, I think your paper is excellent!
Would you say Learning to Summarize is an example of this? https://arxiv.org/abs/2009.01325
It’s model based RL because you’re optimizing against the model of the human (ie the reward model). And there are some results at the end on test-time search.
Or do you have something else in mind?
I’m especially interested in the analogy between AI alignment and democracy. (I guess this goes under “Social Structures and Institutions”.) Democracy is supposed to align a superhuman entity with the will of the people, but there are a lot of failures, closely analogous to well-known AI alignment issues:
politicians optimize for the approval of low-information voters, rather than truly optimizing the people’s wellbeing (deceptive alignment)
politician, pacs, parties, permanent bureaucrats are agents with their own goals that don’t align with the populace (mesa optimizers)
I think it’s more likely that insights will transfer from the field of AI alignment to the field of government design than vice versa. Easier to do experiments on the AI side, and clearer thinkers.
Wonderful writeup!
I’m sure you’ve thought about this, but I’m curious why the following approach fails. Suppose we require the debaters to each initially write up a detailed argument in judge-understandable language and read each other’s argument. Then, during the debate, each debater is allowed to quote short passages from their opponent’s writeup. Honest will be able to either find a contradiction or an unsupported statement in Dishonest’s initial writeup. If Honest quotes a passage and says its unsupported, then dishonest has to respond with the supporting sentences.