It could, although frankly I’m sceptical. I’ve had 18 years to collect data about the world and so far it hasn’t led me to a point where I’d be confident in modifying myself without changing my goals, if an AI takes much longer than that another UFAI will probably beat it to the punch? If it is possible to figure out friendliness only through empirical reasoning without intelligence enhancement, why not figure it out ourselves and then build the AI (this seems roughly the approach SIAI is counting on).
Maturing isn’t a magical process. It happens because of good modifications made to source code.
Why can’t it happen because of additional data collected about the world?
It could, although frankly I’m sceptical. I’ve had 18 years to collect data about the world and so far it hasn’t led me to a point where I’d be confident in modifying myself without changing my goals, if an AI takes much longer than that another UFAI will probably beat it to the punch? If it is possible to figure out friendliness only through empirical reasoning without intelligence enhancement, why not figure it out ourselves and then build the AI (this seems roughly the approach SIAI is counting on).