From a completely different angle: Nietzsche.
We believe that we know something about the things themselves when we speak of trees, colors, snow, and flowers; and yet we possess nothing but metaphors for things — metaphors which correspond in no way to the original entities.
Every concept arises from the equation of unequal things. Just as it is certain that one leaf is never totally the same as another, so it is certain that the concept “leaf” is formed by arbitrarily discarding these individual differences and by forgetting the distinguishing aspects.We obtain the concept, as we do the form, by overlooking what is individual and actual; whereas nature is acquainted with no forms and no concepts, and likewise with no species, but only with an X which remains inaccessible and undefinable for us.
One may certainly admire man as a mighty genius of construction, who succeeds in piling an infinitely complicated dome of concepts upon an unstable foundation, and, as it were, on running water. Of course, in order to be supported by such a foundation, his construction must be like one constructed of spiders’ webs: delicate enough to be carried along by the waves, strong enough not to be blown apart by every wind.
When someone hides something behind a bush and looks for it again in the same place and finds it there as well, there is not much to praise in such seeking and finding. Yet this is how matters stand regarding seeking and finding “truth” within the realm of reason. If I make up the definition of a mammal, and then, after inspecting a camel, declare “look, a mammal’ I have indeed brought a truth to light in this way, but it is a truth of limited value.
I love this text and come back to it often. Isn’t it true for everything? Reality is infinitely complex—the only way we can talk about it is by making abstractions that are very distant from reality as it ‘really’ is. Everything can be “zoomed in” upon and labeled with an infinite, expanding dictionary.
I just call something a “plank”, but a carpenter will know exactly from what tree it has come, how old it is, and what varnishes it has received.
I talk about my “fingers”, but a doctor knows the Latin names for all bones and tendons there.
I notice “electral wires”, an electrician says all kinds of complicated stuff about volts and amperes and types of wirings and grounding etcetera.
When you delve into any subject, you will notice new distinctions, and gain new vocabulary to describe these distinctions. That’s very helpful in many subjects!
But do you want that with humans? Do you like it when somebody starts dividing up humans in “alphas” and “betas”? Should Facebook display your BMI? Do you want to make the near infinite depth of a human—of a mind, of a personality, of a complex genome, the unique set of things they’ve learned from their culture, their family and their friends—something that is easily legible to everyone? Something that ought to be legible?
How we ought to behave, how we want to behave, studying humans and cultures, finding new norms for relationships and sex appropriate to the 21st century—these are fascinating subjects! They are worthy of attention, and I could understand the necessity to develop a deeper vocabulary to study them in detail.
But I don’t think creating some new boxes to fit people into, and demanding that special physical spaces are created for them, and making teens very confused about what box they ought to be in, is very helpful...
I do know one writer who talks a lot about demons and entities from beyond the void. It’s you, and it happens in some of, IMHO, the most valuable pieces you’ve written.
It seems pretty obvious to me:
1.) We humans aren’t conscious of all the consequences of our actions, both because the subconscious has an important role in making our choices, and because our world is enormously complex so all consequences are practically unknowable
2.) In a society of billions, these unforeseeable forces combine in something larger than humans can explicitly plan and guide: “the economy”, “culture”, “the market”, “democracy”, “memes”
3.) These larger-than-human-systems prefer some goals that are often antithetical to human preferences. You describe it perfectly in Seeing Like A State: the state has a desire for legibility and ‘rationally planned’ designs that are at odds with the human desire for organic design. And thus, the ‘supersystem’ isn’t merely an aggregate of human desires, it has some qualities of being an actual separate agent with its own preferences. It could be called a hypercreature, an egregore, Moloch or the devil.
4.) We keep hurting ourselves, again and again and again. We keep falling into multipolar traps, we keep choosing for Moloch, which you describe as “the god of child sacrifice, the fiery furnace into which you can toss your babies in exchange for victory in war”. And thus, we have not accomplished for ourselves what we want to do with AI. Humanity is not aligned with human preferences. This is what failure looks like.
5.) If we fail to align humanity, if we fail to align major governments and corporations, if we don’t even recognize our own misalignment, how big is the chance that we will manage to align AGI with human preferences? Total nuclear war has not been avoided by nuclear technicians who kept perfect control over their inventions—it has been avoided by the fact that the US government in 1945 was reasonably aligned with human preferences. I dare not imagine the world where the Nazi government was the first to get its hands on nuclear weapons.
And thus, I think it would be very, very valuable to put a lot more effort into ‘aligning humanity’. How do we keep our institutions and our grassroots movement “free from Moloch”? How do we get and spread reliable, non-corrupt authorities and politicians? How do we stop falling into multipolar traps, how do we stop suffering unnecessarily?
Best case scenario: this effort will turn out to be vital to AGI alignment
Worst case scenario: this effort will turn out to be irrelevant to AGI alignment, but in the meanwhile, we made the world a much better place