At first glance, the current Agent Foundations work seems to be formal, but it’s not the kind of formal where you work in an established setting. It’s counterintuitive that people can doodle in math, but they can. There’s a lot of that in physics and economics. Pre-formal work doesn’t need to lack formality, it just doesn’t follow a specific set of rules, so it can use math to sketch problems approximately the same way informal words sketch problems approximately.
I’m not saying that you can’t doodle in maths. It’s just that when someone stumbles upon a mathematical model, it’s very easy to fall into confirmation bias, instead of really, deeply considering if what they’re doing makes sense from first principles. And I’m worried that this is what is happening in Agent Foundations research.
At first glance, the current Agent Foundations work seems to be formal, but it’s not the kind of formal where you work in an established setting. It’s counterintuitive that people can doodle in math, but they can. There’s a lot of that in physics and economics. Pre-formal work doesn’t need to lack formality, it just doesn’t follow a specific set of rules, so it can use math to sketch problems approximately the same way informal words sketch problems approximately.
I’m not saying that you can’t doodle in maths. It’s just that when someone stumbles upon a mathematical model, it’s very easy to fall into confirmation bias, instead of really, deeply considering if what they’re doing makes sense from first principles. And I’m worried that this is what is happening in Agent Foundations research.