I think coherence over time is a very difficult problem, and one humans still struggle at, even though (I assume) evolution optimized us hard for this.
Not sure about this. Did humans in the ancestral environment need more than 1-day coherence?
yes. they needed to plan many months ahead to choose where to live, where to hunt, how to raise kids. even long before farming, this sort of need to plan far ahead is the norm for animals. which animals know how to do it intentionally, less clear, but there are clear benefits to being able to do it pretty early in evolutionary history.
Several-months-AGI is required to be coherent in the sense of coherence defined with human experts today. I think this is pretty distinct from coherence that humans were being optimized to have before behavioral modernity (50K years ago).
I agree that evolution optimized hard for some kind of coherence, like persistent self-schema, attitudes, emotional and behavioral patterns, attachments, long-term memory access. But what humans have going for them is the combination of this prior coherence and just 50K years of evolution after humans unlocked access to the abstract thinking toolkit. I don’t think we can expect it to enable much in terms of to the ability to coherently plan to do complex tasks or to the ability to write and reason abstractly.
This makes me think humans struggling at coherence is not good evidence for building agents with large t being much more difficult compared to small t: there wasn’t enough optimization pressure.
Not sure about this. Did humans in the ancestral environment need more than 1-day coherence?
yes. they needed to plan many months ahead to choose where to live, where to hunt, how to raise kids. even long before farming, this sort of need to plan far ahead is the norm for animals. which animals know how to do it intentionally, less clear, but there are clear benefits to being able to do it pretty early in evolutionary history.
Several-months-AGI is required to be coherent in the sense of coherence defined with human experts today. I think this is pretty distinct from coherence that humans were being optimized to have before behavioral modernity (50K years ago).
I agree that evolution optimized hard for some kind of coherence, like persistent self-schema, attitudes, emotional and behavioral patterns, attachments, long-term memory access. But what humans have going for them is the combination of this prior coherence and just 50K years of evolution after humans unlocked access to the abstract thinking toolkit. I don’t think we can expect it to enable much in terms of to the ability to coherently plan to do complex tasks or to the ability to write and reason abstractly.
This makes me think humans struggling at coherence is not good evidence for building agents with large t being much more difficult compared to small t: there wasn’t enough optimization pressure.
Ah, perhaps.