In this case, I think it’s worth being very VERY curious as to how that judge got in there in the first place. It’s also probably worth eventually doing psychological research in order to classify types of judge, in case they aren’t all the same. Do mathematicians above a certain caliber all possess internal judges with a common standard for proof? How does this phenomenon relate to actual judges?
In general, I would expect a person following this advice to, in the average case, diverge from the process of creating a map in correspondence with the territory, towards the replacement of the map with a feedback system conditioning model-free harmony. I would expect that their mind would gradually transition from asking ‘is this true’ to asking ‘is this what power wants me to say’, and eventually to come to see truth as a dreadful constraint on safety rather than as a support with which to achieve safety. I would expect them to grow in their ability to lead and to sell, but to loose the ability to manage, or otherwise constrain the actions of a group in order to direct them towards some goal other than politics.
That doesn’t at all mean that the ideal mode of cognition involves such a judge. Just that collaborative cognition requires a common set of protocols and this seems to be the default such set of protocols for constructive collaboration, while other protocols seem favored by predatory collaboration and seem likely to emerge if not suppressed.
Build a seed AI seems like something that would translate into a religious proposal to me. Discover God perhaps? Might it work better if proposed by someone who didn’t understand the implications of AGI? Would discussion of MNT translate ad the suggestion of miniaturizing Hiero’s engine and the Antikythera mechanism, or scaling them up? The rules of this game really don’t seem clear enough. We need to get feedback from Archimedes on what he’s hearing. Personally, I strongly suspect that when it comes to ethical and political ideas the Ancient Greeks will understand us fairly well without much need for metaphorical translation, so long as we keep the discussion abstract. Women’s suffrage = wise dictator? For people who’s ancestors were inventing democracies of a sort and writing Lysistrata centuries earlier? Make the machine reversible and if you suggest women’s suffrage and he responds about Hiero II, you will hear a sensible to modern ears argument for moving to Singapore.