There are seeds of some good ideas in Ben’s paper like having the goal of the AGI system maintained in a distributed peer to peer system like bit torrent or bitcoin, preventing it from getting too corrupted. That partially addresses one of my concerns with friendliness, the cosmic ray coming in and flips one important register and the AI turns Evil instead of Good (Laugh all you want, I was genuinely worried about this once upon a time)
My idea follows - (warning: Some familiarity with the bitcoin protocol may be needed) An open morality project to provide the goal for a future AGI could begin with a community effort to understand goodness, praise goodness and reward goodness. The community’s reputation points/karma could be maintained in a bitcoin like distributed ledger, practically unhackable if the community has gone unmolested for 2 years or so.
Modifying the highest goal—Be friendly would be impossible. Modifying the weights of the lower level subgoals would require karma and more karma as you go higher towards the highest goal. Changing weights drastically, adding new subgoals would require more karma and there could be allowances for goals/behaviors that are supported/opposed greatly by a few and goals/behaviours supported by vast numbers.
Instead of paying money to a foundation subject to future corruption, future philanthropists can just back the “coin” with extra money, strengthening the hand of all members in the community and further strengthening the goal.
Infact, existing communities could transfer their Karma/Reputation points onto the initial distribution and then begin from there, with future Karma coming only coming from the peer to peer network. The coin distribution could be a limited one or a slowly growing one depending on the best estimates of the coders.
There are seeds of some good ideas in Ben’s paper like having the goal of the AGI system maintained in a distributed peer to peer system like bit torrent or bitcoin, preventing it from getting too corrupted. That partially addresses one of my concerns with friendliness, the cosmic ray coming in and flips one important register and the AI turns Evil instead of Good (Laugh all you want, I was genuinely worried about this once upon a time)
My idea follows - (warning: Some familiarity with the bitcoin protocol may be needed) An open morality project to provide the goal for a future AGI could begin with a community effort to understand goodness, praise goodness and reward goodness. The community’s reputation points/karma could be maintained in a bitcoin like distributed ledger, practically unhackable if the community has gone unmolested for 2 years or so.
Modifying the highest goal—Be friendly would be impossible. Modifying the weights of the lower level subgoals would require karma and more karma as you go higher towards the highest goal. Changing weights drastically, adding new subgoals would require more karma and there could be allowances for goals/behaviors that are supported/opposed greatly by a few and goals/behaviours supported by vast numbers.
Instead of paying money to a foundation subject to future corruption, future philanthropists can just back the “coin” with extra money, strengthening the hand of all members in the community and further strengthening the goal.
Infact, existing communities could transfer their Karma/Reputation points onto the initial distribution and then begin from there, with future Karma coming only coming from the peer to peer network. The coin distribution could be a limited one or a slowly growing one depending on the best estimates of the coders.