“But with a sufficient surplus of power, you could start doing things the eudaimonic way. Start rethinking the life experience as a road to internalizing new strengths, instead of just trying to keep people alive efficiently.”
It should be noted that this doesn’t make the phenomenon of borrowed strength go away, it just outsources it to the FAI. If anything, given the kind of perfect recall and easy access to information that an FAI would have, the ratio of cached historical information to newly created information should be much higher than that of a human. Of course, an FAI wouldn’t suffer the problem of losing the information’s deep structure like a human would, but it seems to be a fairly consistent principle that the amount of cached data grows faster than the rate of data generation.
The problem here- the thing that actually decreases utility- is humans taking actions without sufficient understanding of the potential consequences, in cases where “Humans seem to do very well at recognizing the need to check for global consequences by perceiving local features of an action.” (CFAI 3.2.2) fails. I wonder, out of a sense of morbid curiosity, what the record is for the highest amount of damage caused by a single human without said human ever realizing that they did anything bad.
“But with a sufficient surplus of power, you could start doing things the eudaimonic way. Start rethinking the life experience as a road to internalizing new strengths, instead of just trying to keep people alive efficiently.”
It should be noted that this doesn’t make the phenomenon of borrowed strength go away, it just outsources it to the FAI. If anything, given the kind of perfect recall and easy access to information that an FAI would have, the ratio of cached historical information to newly created information should be much higher than that of a human. Of course, an FAI wouldn’t suffer the problem of losing the information’s deep structure like a human would, but it seems to be a fairly consistent principle that the amount of cached data grows faster than the rate of data generation.
The problem here- the thing that actually decreases utility- is humans taking actions without sufficient understanding of the potential consequences, in cases where “Humans seem to do very well at recognizing the need to check for global consequences by perceiving local features of an action.” (CFAI 3.2.2) fails. I wonder, out of a sense of morbid curiosity, what the record is for the highest amount of damage caused by a single human without said human ever realizing that they did anything bad.