Logan, for your preferred alignment approach how likely is it that the alignment remains durable over time? A superhuman AGI will understand the choices that were made by its creators to align it. It will be capable of comparing its current programming with counterfactuals where it’s not aligned. It will also have the ability to alter its own code. So what if it determines it’s best course of action is to alter the very code that maintains it’s alignment? How would this be prevented?
Shay
Yeah, I agree that valuing humans isn’t enough. I’m suggesting something that humans intrinsically have, or at least have the capacity for. Something that most life on Earth also shares a capacity for. Something that doesn’t change drastically over time in the way that ethics and morals do. Something that humans value, that is universal, and also durable.
I am not suggesting anything about efficiency. Why bother with efficiencies in a post scarcity world?
The goal should not be to maximize anything, not even intelligence. Maintaining or incrementally increasing intelligence would be favorable to humans.
Perhaps the attractor could be intelligence itself. So a primary goal of the AGI would be to maximize intelligence. It seems like human flourishing would then be helpful to the AGI’s goal. Human flourishing, properly defined, implies flourishing of the Earth and its biosphere as a whole, so maybe that attractor brings our world, cultures, and way of life along for the ride.
We may also need to ensure that intelligences have property rights over the substrates they operate on. That may be needed prevent the AGI from converting brains and bodies into microchips, if that’s even possible.
Regarding increased costs in healthcare…
I’ve worked in med device since 2008. The effort it takes to develop and commercialize med devices is continuously increasing and subsequently driving up costs. Many teams of engineers are paid well to generate binders full of documentation in support of the regulatory/compliance requirements of even simple devices. I’m sad to say that this increased effort doesn’t directly translate to better devices, but it certainly keeps a lot of people employed.