This sounds about right. And that’s a lot of weight on simulacra carrying out alignment, as opposed to perfect imitations of specific humans (uploads) doing that.
If simulacra are used for strawberry alignment of uploading, that already requires natural abstraction to work for avoiding weird side effects or unbounded optimization while performing a task, itself a major miracle. But if they are used for anything else, that requires natural abstraction to work for values in order for things to go well, an even less plausible miracle.
This sounds about right. And that’s a lot of weight on simulacra carrying out alignment, as opposed to perfect imitations of specific humans (uploads) doing that.
If simulacra are used for strawberry alignment of uploading, that already requires natural abstraction to work for avoiding weird side effects or unbounded optimization while performing a task, itself a major miracle. But if they are used for anything else, that requires natural abstraction to work for values in order for things to go well, an even less plausible miracle.