Because in brain-like AGI, the reward function is written in Python (or whatever), not in natural language.
Yup. I’d bet some people will reply with something like “why not define the reward function in natural language, like constitutional AI”. I think this fails due to strong optimization finding the most convenient (for it, not us) settings of free parameters left by fuzzy statistical things like words, and if you give it a chance to feed back into the definitions via training data or do online learning etc gets totally wrecked by semantic drift.
And don’t you think 500 lines of Python also “fails due to” having unintended optima?
I’ve put “fails due to” in scare quotes because what’s failing is not every possible approach, merely almost all samples from approaches we currently know how to take. If we knew how to select python code much more cleverly, suddenly it wouldn’t fail anymore. And ditto for if we knew how to better construct reward functions from big AI systems plus small amounts of human text or human feedback.
Yup. I’d bet some people will reply with something like “why not define the reward function in natural language, like constitutional AI”. I think this fails due to strong optimization finding the most convenient (for it, not us) settings of free parameters left by fuzzy statistical things like words, and if you give it a chance to feed back into the definitions via training data or do online learning etc gets totally wrecked by semantic drift.
And don’t you think 500 lines of Python also “fails due to” having unintended optima?
I’ve put “fails due to” in scare quotes because what’s failing is not every possible approach, merely almost all samples from approaches we currently know how to take. If we knew how to select python code much more cleverly, suddenly it wouldn’t fail anymore. And ditto for if we knew how to better construct reward functions from big AI systems plus small amounts of human text or human feedback.
Oh no, almost all possible 500 lines of python are also bad.