Naively extrapolating from my thoughts about semantics of intelligence and economy of cognition, I suspect that it might be a good thing for people of the future to specialize and increasingly differ in their morality, while composing a community that as a whole robustly continues on the path set by the present human psychology. It doesn’t look like a good thing even for some people to start liking “paperclipping”, and maybe it isn’t, but maybe it is.
Several steps further, if “person” remains a morally significant abstraction, people, while distinct, may become increasingly abstract and interleaved, so that the same process can be interpreted as a part of one person’s thought and another person’s different thought at once, with substrate interpreted as implementing individual subjective timelines distributed in space and time. One person as a high-level regularity in many other people’s thoughts. It’s more an implementation detail than morally significant picture though, like running a program in obfuscated binary code, with originally intended algorithm surprisingly managing to carry through.
About the arguably good future that frightens.
Naively extrapolating from my thoughts about semantics of intelligence and economy of cognition, I suspect that it might be a good thing for people of the future to specialize and increasingly differ in their morality, while composing a community that as a whole robustly continues on the path set by the present human psychology. It doesn’t look like a good thing even for some people to start liking “paperclipping”, and maybe it isn’t, but maybe it is.
Several steps further, if “person” remains a morally significant abstraction, people, while distinct, may become increasingly abstract and interleaved, so that the same process can be interpreted as a part of one person’s thought and another person’s different thought at once, with substrate interpreted as implementing individual subjective timelines distributed in space and time. One person as a high-level regularity in many other people’s thoughts. It’s more an implementation detail than morally significant picture though, like running a program in obfuscated binary code, with originally intended algorithm surprisingly managing to carry through.