That leads to an interesting question—how would an FAI decide how much intelligence is enough?
I don’t know. It’s supposed to be the smart one, not me. ;)
I’m hoping it goes something like:
Predict the expected outcome of choosing to self improve some more.
Predict the expected outcome of choosing not to self improve some more.
Do the one that gives the best probability distribution of expected results.
That leads to an interesting question—how would an FAI decide how much intelligence is enough?
I don’t know. It’s supposed to be the smart one, not me. ;)
I’m hoping it goes something like:
Predict the expected outcome of choosing to self improve some more.
Predict the expected outcome of choosing not to self improve some more.
Do the one that gives the best probability distribution of expected results.