making sandwiches is a task relatively similar to tasks we had to deal with in the ancestral environment, and in particular there is not a lot of serial depth to the know-how of making sandwiches. If we pluck a person from virtually any culture in any period of history, then it won’t be difficult to explain em how to make a sandwich. On the other hand, in the case of AI risk, just understanding the question requires a lot of background knowledge that was built over generations and requires years of study to properly grasp.
If I understand your argument correctly, it implies that dealing with the agents that evolve from simpler than you are to smarter than you are within a few lifetimes (“foom”) is not a task that was ever present, or at least not successfully accomplished by your evolutionary ancestors, and hence not incorporated into the intuitive part of the brain. Unlike, say, the task of throwing a rock with the aim of hitting something, which has been internalized and eventually resulted in NBA, with all the required nonlinear differential equations solved by the brain in real time accurately enough, for some of us more so than for others.
Similarly, approximate basic counting is something humans and other animals have done for millions of years, while, say, accurate long division was never evolutionarily important and so requires engaging the conscious parts of the brain just to understand the question (“why do we need all these extra digits and what do they mean?”), even though it is technically much much simpler than calculating the way one’s hand must move in order to throw a ball on just the right trajectory.
If this is your argument, then I agree with it (and made a similar one here before numerous times).
If I understand your argument correctly, it implies that dealing with the agents that evolve from simpler than you are to smarter than you are within a few lifetimes (“foom”) is not a task that was ever present, or at least not successfully accomplished by your evolutionary ancestors, and hence not incorporated into the intuitive part of the brain. Unlike, say, the task of throwing a rock with the aim of hitting something, which has been internalized and eventually resulted in NBA, with all the required nonlinear differential equations solved by the brain in real time accurately enough, for some of us more so than for others.
Similarly, approximate basic counting is something humans and other animals have done for millions of years, while, say, accurate long division was never evolutionarily important and so requires engaging the conscious parts of the brain just to understand the question (“why do we need all these extra digits and what do they mean?”), even though it is technically much much simpler than calculating the way one’s hand must move in order to throw a ball on just the right trajectory.
If this is your argument, then I agree with it (and made a similar one here before numerous times).