* You don't need to make a particularly advanced ontology to create an AI. By this I mean that while complex, the AI need not come built with an ontology that represents even all of current human knowledge, let alone potential ontological breakthroughs made by future humans.
Human ontologies are complex, redundant and outright contradictory at times. Not only is some of it not needed to create an AI it would be counter-productive to include it.
A GAI could maintain its original goal system under self improvement even if it makes ontological breakthroughs.
When AI comes to model human preferences and the elements of the universe which are most relevant to fulfilling them they need not interfere at all with the implementation of the AI itself. It’s just a complex form of data and metadata to keep in mind. When it comes to things that would more fundamentally influence the direct operation of the AI goals it can ensure that any alterations do not contradict the old version or do so only to resolve a discovered contradiction in whatever the sanest way possible is.
I would never trust an AI to have a full human goal system, or the attached ontology necessary to represent it.
Humans suck compared to superintelligences. They even suck at knowing what they want. I’d rather tell a friendly superintelligence to do what I want it to do rather than try to program my goals into it. Did I mention that it is smarter than me? It can even emulate me and ask em-me my goals that way if it hasn’t got a better option. There is no downside to getting the FAI to do it for me. If it isn’t friendly then....
The ontology of a pre-foom FAI would not be like the general human one. It would be simpler and clearer, with a mechanism to create whatever further ontology necessary to represent the values from an appropriate reference.
Humans suck at creating ontologies. Less than any other species I know but they still suck. I wouldn’t include stupid parts in a FAI, that’d make it particularly hard to prove friendly. But it would naturally be able to look at humans and figure out any necessary stupid parts itself.
A super-intelligence can figure out ontological stuff better than a human. The ‘only’ problem (for the AI creators) is getting to to a system that has a simplified (less incoherent) version of the goal system of the creators that can self improve without goal change. Where ‘without goal change’ includes “don’t destroy all of that gooey grey stuff from which you need to get a whole heap more of the detailed information about your values!!!”
That is rather dense, I’ll admit. But the gist of the reasoning is there.
(Only slightly less briefly)
Human ontologies are complex, redundant and outright contradictory at times. Not only is some of it not needed to create an AI it would be counter-productive to include it.
When AI comes to model human preferences and the elements of the universe which are most relevant to fulfilling them they need not interfere at all with the implementation of the AI itself. It’s just a complex form of data and metadata to keep in mind. When it comes to things that would more fundamentally influence the direct operation of the AI goals it can ensure that any alterations do not contradict the old version or do so only to resolve a discovered contradiction in whatever the sanest way possible is.
Humans suck compared to superintelligences. They even suck at knowing what they want. I’d rather tell a friendly superintelligence to do what I want it to do rather than try to program my goals into it. Did I mention that it is smarter than me? It can even emulate me and ask em-me my goals that way if it hasn’t got a better option. There is no downside to getting the FAI to do it for me. If it isn’t friendly then....
Humans suck at creating ontologies. Less than any other species I know but they still suck. I wouldn’t include stupid parts in a FAI, that’d make it particularly hard to prove friendly. But it would naturally be able to look at humans and figure out any necessary stupid parts itself.
That is rather dense, I’ll admit. But the gist of the reasoning is there.