The question I’d ask is whether a “minimum surprise principle” requires that much smartness. A present day LLM, for example, might not have a perfect understanding of surprisingness but it like it has some and the concept seems reasonably trainable.
It seems only marginally simpler than figuring out what I want. Both require a pretty good model of me; or better yet, asking me if it’s not sure.
The question I’d ask is whether a “minimum surprise principle” requires that much smartness. A present day LLM, for example, might not have a perfect understanding of surprisingness but it like it has some and the concept seems reasonably trainable.
It seems only marginally simpler than figuring out what I want. Both require a pretty good model of me; or better yet, asking me if it’s not sure.