ASI narratives tend to assume superhuman metacognitive skills will be used to efficiently learn new ordinary skills. However, so far, ordinary skill learning has outstripped metacognition. What intuition do you think people have for this swapping around?
How generalisable do you think metacognitive skills are? I was interested in your comment that ‘Humans’ metacognitive abilities seem to vary with their expertise on a given topic’. Continuing with the narrative of using superhuman metacognitive skills to efficiently learn ordinary skills: if learning the latter skills in simulations only generalises so far to the real world, surely metacognitive skills can’t bail you out as the same will be true of them? To the degree that the metacognitive skill is general it generalises, to the degree it is specific, it does not. E.g. the paper you linked here https://arxiv.org/pdf/2504.05419 saw probes are well-calibrated within distribution, but not always outside, suggesting one important metacognitive ability—a well-calibrated sense of how good your answer is—does not generalise well.
‘If everyone who asked was told something like “humans pretty clearly don’t know how hard alignment is, so you should probably slow down progress toward AGI if you possibly can,” it might indirectly help a fair amount. And more reliable systems might be particularly helpful for alignment deconfusion.’ It’s not clear to me how a system could learn this objectively. It would get accused of having been post-trained to say it or people would speculate about the training data. You need some trust-building process, and the lowest error answer on a particular training set is not that. How reliant do we have to get on AI before we trust it this much?
Interesting post! I have a few questions:
ASI narratives tend to assume superhuman metacognitive skills will be used to efficiently learn new ordinary skills. However, so far, ordinary skill learning has outstripped metacognition. What intuition do you think people have for this swapping around?
How generalisable do you think metacognitive skills are? I was interested in your comment that ‘Humans’ metacognitive abilities seem to vary with their expertise on a given topic’. Continuing with the narrative of using superhuman metacognitive skills to efficiently learn ordinary skills: if learning the latter skills in simulations only generalises so far to the real world, surely metacognitive skills can’t bail you out as the same will be true of them? To the degree that the metacognitive skill is general it generalises, to the degree it is specific, it does not. E.g. the paper you linked here https://arxiv.org/pdf/2504.05419 saw probes are well-calibrated within distribution, but not always outside, suggesting one important metacognitive ability—a well-calibrated sense of how good your answer is—does not generalise well.
‘If everyone who asked was told something like “humans pretty clearly don’t know how hard alignment is, so you should probably slow down progress toward AGI if you possibly can,” it might indirectly help a fair amount. And more reliable systems might be particularly helpful for alignment deconfusion.’ It’s not clear to me how a system could learn this objectively. It would get accused of having been post-trained to say it or people would speculate about the training data. You need some trust-building process, and the lowest error answer on a particular training set is not that. How reliant do we have to get on AI before we trust it this much?