Maybe an even better analogy is non-Euclidean geometry. Agent foundations is studying a strange alternate world where agents know the source code to themselves and the universe, where perfect predictors exist and so on. It’s not an abstraction of our world, but something quite different. But surprisingly it turns out that many aspects of decision-making in our world have counterparts in the alternate world, and in doing so we shed a strange light on what decision-making in our world actually means.
I’m not even sure these investigations should be tied to AI risk (though that’s very important too). To me the other world offers mathematical and philosophical interest on its own, and frankly I’m curious where these investigations will lead (and have contributed to them where I could).
Modelling always requires idealisation. Currently, in many respects the formal models that Agent Foundations use to capture the informal notion of agency, intention, goal etc are highly idealised. This is not an intrinsic feature of Agent Foundations or mathematical modelling- just a reflection of the inadequate mathematical and conceptual state of the world.
By analogy—intro to Newtonian Mechanics begins with frictionless surfaces and the highly simple orbits of planetary systems. That doesn’t mean that Newtonian Mechanics in more sophisticated forms cannot be applied to the real world.
One can get lost in the ethereal beauty of ideal worlds. That should not detract from the ultimate aim of mathematical modelling of the real world.
Agent foundations is studying a strange alternate world where agents know the source code to themselves and the universe, where perfect predictors exist and so on
I just want to flag that this is very much not a defining characteristic of agent foundations! Some work in agent foundations will make assumptions like this, some won’t—I consider it a major goal of agent foundations to come up with theories that do not rely on assumptions like this.
Maybe an even better analogy is non-Euclidean geometry. Agent foundations is studying a strange alternate world where agents know the source code to themselves and the universe, where perfect predictors exist and so on. It’s not an abstraction of our world, but something quite different. But surprisingly it turns out that many aspects of decision-making in our world have counterparts in the alternate world, and in doing so we shed a strange light on what decision-making in our world actually means.
I’m not even sure these investigations should be tied to AI risk (though that’s very important too). To me the other world offers mathematical and philosophical interest on its own, and frankly I’m curious where these investigations will lead (and have contributed to them where I could).
Modelling always requires idealisation. Currently, in many respects the formal models that Agent Foundations use to capture the informal notion of agency, intention, goal etc are highly idealised. This is not an intrinsic feature of Agent Foundations or mathematical modelling- just a reflection of the inadequate mathematical and conceptual state of the world.
By analogy—intro to Newtonian Mechanics begins with frictionless surfaces and the highly simple orbits of planetary systems. That doesn’t mean that Newtonian Mechanics in more sophisticated forms cannot be applied to the real world.
One can get lost in the ethereal beauty of ideal worlds. That should not detract from the ultimate aim of mathematical modelling of the real world.
I just want to flag that this is very much not a defining characteristic of agent foundations! Some work in agent foundations will make assumptions like this, some won’t—I consider it a major goal of agent foundations to come up with theories that do not rely on assumptions like this.
(Or maybe you just meant those as examples?)