You are right that a vat brain life should certainly seem far worse than a human life—to a human. But would a vat brain agree? From its perspective, human lives could be horrible, because they’re constantly assaulted by amounts of novelty and physical danger that a vat brain couldn’t imagine handling. Humans always need to work for homeostasis from a wildly heterogenous set of environmental situations. A vat brain wouldn’t at all be surprised to hear that human lives are much shorter than vat brain lives.
Do you think that once we know what intelligence is exactly, we’ll be able to fully describe it mathematically? Since you’re assuming electronics-based superintelligence is possible, it would appear so. Well if you’re right, intelligence is substrate-independent.
Your distinction between “single mind” and “higher-order mechanism” is a substrate distinction, so it shouldn’t matter. You and I feel it does matter, because we’re glorified chimps with inborn intuitions about what constitutes an agent, but math is not a chimp—and if math doesn’t care whether intelligence runs on a brain or on a computer system, it shouldn’t care whether intelligence runs on one brain or on several.
You are right that a vat brain life should certainly seem far worse than a human life—to a human. But would a vat brain agree? From its perspective, human lives could be horrible, because they’re constantly assaulted by amounts of novelty and physical danger that a vat brain couldn’t imagine handling. Humans always need to work for homeostasis from a wildly heterogenous set of environmental situations. A vat brain wouldn’t at all be surprised to hear that human lives are much shorter than vat brain lives.
Do you think that once we know what intelligence is exactly, we’ll be able to fully describe it mathematically? Since you’re assuming electronics-based superintelligence is possible, it would appear so. Well if you’re right, intelligence is substrate-independent.
Your distinction between “single mind” and “higher-order mechanism” is a substrate distinction, so it shouldn’t matter. You and I feel it does matter, because we’re glorified chimps with inborn intuitions about what constitutes an agent, but math is not a chimp—and if math doesn’t care whether intelligence runs on a brain or on a computer system, it shouldn’t care whether intelligence runs on one brain or on several.