Despite the title, this reads to me like an interesting overview of how we’d want a good benevolent AI to work, in fact: it needs to help us be curious about our own wants and values and help us defend against things that would decrease our agency.
AI summary via claude2:
Here are 30 key points from the article:
MIRI recently announced a “death with dignity” strategy, giving up on solving the AI alignment problem.
Many in the AI field believe progress is being made on AI capabilities but not on AI safety.
The framing of “benevolent AI” makes faulty assumptions about agency, values, benevolence, etc.
The author has studied human psychology and finds most concepts around agency, values, etc. woefully inadequate.
Trying to fully encapsulate or consciously rationalize human values is dangerous and bound to fail.
Human values are not universal or invariant across environments.
Language cannot fully describe conceptual space, and conceptual space cannot fully describe possibility space.
We do not need complete self-knowledge or full descriptions of values to function well.
The desire for complete descriptions of values comes from fear of human incompetence.
“Protectionist” projects will decrease human agency, consciously or not.
Current AI trends already reduce agency through frustrating automation experiences.
AI could help increase agency by expanding conceptual range, not just increasing power.
Most choices go unrecognized due to limited conceptual frames that steer our autopilot.
Our imagined futures are constrained by cultural imagination and trauma.
Verbalization of “values” is downstream of fundamental motivations.
Opening imaginative possibilities requires more than empathy or verbalization.
Human psychology evolves by functions observing and modifying other functions.
An example chatbot could help with self-reflection and concept formation.
The chatbot prompts focused introspection and pattern recognition.
The chatbot draws on diverse analytical commentary in its training.
The chatbot doesn’t need sophisticated intelligence or goals.
This approach avoids problems of encapsulating human values.
Current AI safety discourse has problematic assumptions baked in.
It reflects poor epistemics and spreads fear via social media.
Better to imagine AI that helps ground us and know ourselves.
We should turn our agency toward increasing future agency.
Aligned AI helps us explore creating the futures we want.
We can engage the topic from outside the dominant ideology.
Letting go of its urgency allows more personal agency.
Our incomplete self-understanding is beautiful and we can grow it.