The original argument that your OP is responding to is about “bio evolution”. I understand the distinction, but why is it relevant? Indeed, in the OP you say:
For the evolution of human intelligence, the optimizer is just evolution: biological natural selection. The utility function is fitness: gene replication count (of the human defining genes).
The OP is talking about history and thus bio evolution, and this thread shifted into the future (where info-evolution dominates) here:
I think the jury is still out about what will happen in the future. Maybe future humans will all upload their brains into computers, which is presumably count as zero IGF, if we define IGF via literal DNA molecules (which we don’t have to, but that’s another story). Whatever, I dunno.
Of course—and we’d hope that there is some decoupling eventually! Otherwise it’s just be fruitful and multiply, forever.
I’m saying that you, a bio-evolved thing, are saying that you hope something happens, and that something is not what bio-evolution wants. So you’re a misaligned optimizer from bio-evolution’s perspective.
If you narrowly define the utility function as “IGF via literal DNA molecules” - (which obviously is the relevant context for my statement “hope that there is some decoupling eventually”) then obviously I’m somewhat misaligned to that util func (but not completely, I am having children). And i’m increasingly aligned with the more general utility functions.
None of this is especially relevant, because I am not a species.
The original argument that your OP is responding to is about “bio evolution”. I understand the distinction, but why is it relevant? Indeed, in the OP you say:
So we’re talking about bio evolution, right?
The OP is talking about history and thus bio evolution, and this thread shifted into the future (where info-evolution dominates) here:
I’m saying that you, a bio-evolved thing, are saying that you hope something happens, and that something is not what bio-evolution wants. So you’re a misaligned optimizer from bio-evolution’s perspective.
If you narrowly define the utility function as “IGF via literal DNA molecules” - (which obviously is the relevant context for my statement “hope that there is some decoupling eventually”) then obviously I’m somewhat misaligned to that util func (but not completely, I am having children). And i’m increasingly aligned with the more general utility functions.
None of this is especially relevant, because I am not a species.