One measuring stick I have for how close we are to “ASI” is the extent to which an AI can find a realm of knowledge where the mainstream of human learning has happened to stumble down a blind alley, a set of incorrect theories increasingly kept afloat with ad-hoc epicycles, where the AI confidently tries to offer alternative answers, and in explaining those answers, is able to build up an entire alternative field of literature on that topic from first principles. I know this sounds like a high bar, but ASI would be a big deal, so it has to be able to clear such a high bar.
Example: imagine we are living in a reality where Austrian Business Cycle Theory, or the Marxist Labor Theory of Value, or some other out-of-fashion set of theories happens to be objectively correct about the way capitalist economies function, despite the fact that the mainstream of the economics profession, and the majority of the AI’s training data, argue otherwise. I would define an ASI as the sort of AI that could look through all of that, use its own “judgment” and even sense-data (from having access to cameras, microphones, and other devices by which it can gather its own empirical data and not just rely on what humans report about objective reality), and confidently and persuasively demonstrate an alternative, against-the-mainstream body of work on that topic.
However, is such a thing possible? Perhaps not without empirical sense-data becoming a much bigger part of the AI’s training. Perhaps LLM architecture will not be enough to get us there, if LLMs are only able to make headway by being exposed to the “correct” training data in their imitative learning. If humanity turns out to be mistaken in a big way about what is “correct” in some of the fuzzier, softer sciences, and if LLMs are inherently constrained by what is possible through imitative learning, then we are unlikely to ever see LLMs overturn an academic consensus unless we develop radically different training architectures that incorporate more kinds of sense-data other than tokens of prior human (and by now, AI-generated) text.
One measuring stick I have for how close we are to “ASI” is the extent to which an AI can find a realm of knowledge where the mainstream of human learning has happened to stumble down a blind alley, a set of incorrect theories increasingly kept afloat with ad-hoc epicycles, where the AI confidently tries to offer alternative answers, and in explaining those answers, is able to build up an entire alternative field of literature on that topic from first principles. I know this sounds like a high bar, but ASI would be a big deal, so it has to be able to clear such a high bar.
Example: imagine we are living in a reality where Austrian Business Cycle Theory, or the Marxist Labor Theory of Value, or some other out-of-fashion set of theories happens to be objectively correct about the way capitalist economies function, despite the fact that the mainstream of the economics profession, and the majority of the AI’s training data, argue otherwise. I would define an ASI as the sort of AI that could look through all of that, use its own “judgment” and even sense-data (from having access to cameras, microphones, and other devices by which it can gather its own empirical data and not just rely on what humans report about objective reality), and confidently and persuasively demonstrate an alternative, against-the-mainstream body of work on that topic.
However, is such a thing possible? Perhaps not without empirical sense-data becoming a much bigger part of the AI’s training. Perhaps LLM architecture will not be enough to get us there, if LLMs are only able to make headway by being exposed to the “correct” training data in their imitative learning. If humanity turns out to be mistaken in a big way about what is “correct” in some of the fuzzier, softer sciences, and if LLMs are inherently constrained by what is possible through imitative learning, then we are unlikely to ever see LLMs overturn an academic consensus unless we develop radically different training architectures that incorporate more kinds of sense-data other than tokens of prior human (and by now, AI-generated) text.