p.p.s I’ve read a bunch of so-called Shard Theory stuff and I’m still not sure how it differs from the concepts of optimization daemons/mesa optimization besides less exclusively emphasising the ‘post-general’ regime (for want of a better term).
The only difference is that we’re betting there’ll be a lot of interesting, foreseeable structure in which mesa-optimizers are learned, conditional on a choice of key training parameters. We have some early conjectures as to what that structure will be, and the project is to build up an understanding and impressively win a bunch of Bayes points with it. Most alignment people don’t especially think this is an important question to ask, because they don’t think there’ll end up being a lot of predictable structure in which proxies mesa-optimizers latch on to.
Shard theory is also a bet that proto-mesa-optimizers are also the mechanistic explanation of how current deep RL (and other deep ML settings, to a lesser extent) works.
I love how your intro has the flavour of
p.s. Hail Team Shard
p.p.s I’ve read a bunch of so-called Shard Theory stuff and I’m still not sure how it differs from the concepts of optimization daemons/mesa optimization besides less exclusively emphasising the ‘post-general’ regime (for want of a better term).
The only difference is that we’re betting there’ll be a lot of interesting, foreseeable structure in which mesa-optimizers are learned, conditional on a choice of key training parameters. We have some early conjectures as to what that structure will be, and the project is to build up an understanding and impressively win a bunch of Bayes points with it. Most alignment people don’t especially think this is an important question to ask, because they don’t think there’ll end up being a lot of predictable structure in which proxies mesa-optimizers latch on to.
Shard theory is also a bet that proto-mesa-optimizers are also the mechanistic explanation of how current deep RL (and other deep ML settings, to a lesser extent) works.