Of course, people are building AGIs anyway. This means that it’s critical to have a second group of people who are working in parallel to understand as quickly as possible what is being built, to maintain the safety of those systems. And for those people, the activities of experimentation and instrument building are still essential.
The relevant problem with cognitive science (for which “agent foundations” is just an idiosyncratic alias coined in the MIRI circles) is that intelligent systems are typically complex, which means they can be modelled in many different ways.
One (probable) consequence of this is that no “agent foundations” theory will lead to building a real HRAD AGI (rather than a toy or a “proof of concept”) because as long as the agent architecture will be clean (engineered strictly according to some “agent foundations” theory) and GOFAI-ish it won’t be complex enough to really respond intelligently to the complexity of the environment (cf. the law of requisite variety).
Another very important consequence is that this probably reduces the effectiveness of theoretical work in the field, especially outside of major AGI labs. Basically, this is because cognitive science is “swampy”: a lot of different theories seem plausible and at least partially “correct” (i.e., model the reality with nonzero fidelity), and at the same time, no single theory is completely correct, to the level of the theory of general relativity, for instance. Therefore, lead scientists at major AGI labs pick their “favourite” theories and it’s probably just impossible to move their positions with theoretical arguments. (See this argument laid out in more detail in this draft, in which I argue that independent theoretical work in AI safety is ineffective.)
The relevant problem with cognitive science (for which “agent foundations” is just an idiosyncratic alias coined in the MIRI circles) is that intelligent systems are typically complex, which means they can be modelled in many different ways.
One (probable) consequence of this is that no “agent foundations” theory will lead to building a real HRAD AGI (rather than a toy or a “proof of concept”) because as long as the agent architecture will be clean (engineered strictly according to some “agent foundations” theory) and GOFAI-ish it won’t be complex enough to really respond intelligently to the complexity of the environment (cf. the law of requisite variety).
Another very important consequence is that this probably reduces the effectiveness of theoretical work in the field, especially outside of major AGI labs. Basically, this is because cognitive science is “swampy”: a lot of different theories seem plausible and at least partially “correct” (i.e., model the reality with nonzero fidelity), and at the same time, no single theory is completely correct, to the level of the theory of general relativity, for instance. Therefore, lead scientists at major AGI labs pick their “favourite” theories and it’s probably just impossible to move their positions with theoretical arguments. (See this argument laid out in more detail in this draft, in which I argue that independent theoretical work in AI safety is ineffective.)