I’ll try, just for fun, to summarize Eliezer’s conclusions of the pre-fun-theory and pre-community-building part of the sequence:
artificial intelligence can self-improve;
with every improvement, the rate at which it can improve increases;
AGI will therefore experience exponential improvement (AI fooms);
even if there’s a cap to this process, the resulting agent will be a very powerful agent, incomprehensibly so (singularity);
an agent effectiveness does not constrain its utility function (orthogonality thesis);
humanity’s utility function occupy a very tiny and fragmented fraction of the set of all possible utility functions (human values are fragile);
if we fail to encode the correct human utility function in a self-improving AGI, even tiny differences will results in a catastrophically unpleasant future (UFAI as x-risk);
an AGI is about to come pretty soon, so we better hurry to figure out how to do the latter point correctly.
I’ll try, just for fun, to summarize Eliezer’s conclusions of the pre-fun-theory and pre-community-building part of the sequence:
artificial intelligence can self-improve;
with every improvement, the rate at which it can improve increases;
AGI will therefore experience exponential improvement (AI fooms);
even if there’s a cap to this process, the resulting agent will be a very powerful agent, incomprehensibly so (singularity);
an agent effectiveness does not constrain its utility function (orthogonality thesis);
humanity’s utility function occupy a very tiny and fragmented fraction of the set of all possible utility functions (human values are fragile);
if we fail to encode the correct human utility function in a self-improving AGI, even tiny differences will results in a catastrophically unpleasant future (UFAI as x-risk);
an AGI is about to come pretty soon, so we better hurry to figure out how to do the latter point correctly.