Thanks for this investigation! I appreciate the detailed third-party perspective.
As someone who’s already bought a discount card, and is contemplating buying some more for family, what I really want is a solid landing page / short article that sells the concept to non-rationalist types. Wait But Why’s 2016 post is targeted at the right audience, but is 24 printed pages and doesn’t have any discussion of the advantages of Nectome.
I might have to write one up myself...
I appreciate what this post is trying to point out, although I am sympathetic to some of the other comments that it doesn’t make an airtight case for its specific analogy-thesis.
A while ago I gave a talk at a local university in Tokyo about agent foundations. (You can look at the speaker notes to get a more-or-less verbatim idea of what I said.) I tried to give a relatively fair tour of the field, but the final slide section was titled “Does any of this matter?”, and the final slide, well, let me just quote it:
So yeah, I strongly sympathize with the vibe that agent foundations seems unmoored from the program of actually aligning agents, building castles in the sky on top of abstract mathematical formulations but never quite getting around to showing that they are useful.
(And I love a good abstract mathematical castle! It would give me great personal joy to sit down and become one of the few people in the world who understands all the infra-Bayesianism math. I just don’t see how it’s going to help us align AIs.)
I should probably write up that slideshow into a top-level post at some point, but, it’s a bit intimidating to anticipate what the reactions might be from all the people I’m critiquing. So, I’ll hide it in this comment for now.