I think tacit knowledge is severely underrated in discussions of AGI and ASI.
In HPMOR, there is a scene near the end of the book where our hero wins the day by using some magic that would be equivalent to flying a drone around an extremely complicated path involving lots of loops in places not directly observable for our hero.
Our hero has never once in the book practiced doing this.
In theory, if I possess a drone and have a flight path the drone is capable of flying, I can pick up the controller for the first time and make it happen.
In practice, I will fail spectacularly. A lot of writing in this space assumes that with sufficient ‘thinking power’, success on the first attempt is assured.
re: your last remark, FWIW I think a lot of those writings you’ve seen were probably intuition-pumped by this parable of Eliezer’s, to which I consider titotal’s pushback the most persuasive.
I think tacit knowledge is severely underrated in discussions of AGI and ASI.
In HPMOR, there is a scene near the end of the book where our hero wins the day by using some magic that would be equivalent to flying a drone around an extremely complicated path involving lots of loops in places not directly observable for our hero.
Our hero has never once in the book practiced doing this.
In theory, if I possess a drone and have a flight path the drone is capable of flying, I can pick up the controller for the first time and make it happen.
In practice, I will fail spectacularly. A lot of writing in this space assumes that with sufficient ‘thinking power’, success on the first attempt is assured.
re: your last remark, FWIW I think a lot of those writings you’ve seen were probably intuition-pumped by this parable of Eliezer’s, to which I consider titotal’s pushback the most persuasive.