If you can’t write a program that produces aligned (under whatever definition of alignment you use) output being run on unphysically large computer, you can’t deduce from training data or weights of superintelligent neural network if it produces aligned output.
If you can’t write a program that produces aligned (under whatever definition of alignment you use) output being run on unphysically large computer, you can’t deduce from training data or weights of superintelligent neural network if it produces aligned output.