I had a conversation about this recently, and I raised the point that fieldbuilding suffers from the same issues. When it really comes down to the wire, someone has to carry out the hard task of (being capable of) understanding and verifying the entire alignment stack end-to-end.
This puts governments, AI company CEOs, and Dustin Moskowitz in an awkward position: the person to understand the stack might not be able to make it legible to them (compare to the US Army guys who had to take the atmospheric ignition calculations on the word of the physicists).
Maaaybe you can break it down and entrust the local validity of each step to a different genius: “Terry Tao says the maths checks out, and our compiler engineers all agree on the architecture” or something like that, but I wouldn’t bet the world on it.
I had a conversation about this recently, and I raised the point that fieldbuilding suffers from the same issues. When it really comes down to the wire, someone has to carry out the hard task of (being capable of) understanding and verifying the entire alignment stack end-to-end.
This puts governments, AI company CEOs, and Dustin Moskowitz in an awkward position: the person to understand the stack might not be able to make it legible to them (compare to the US Army guys who had to take the atmospheric ignition calculations on the word of the physicists).
Maaaybe you can break it down and entrust the local validity of each step to a different genius: “Terry Tao says the maths checks out, and our compiler engineers all agree on the architecture” or something like that, but I wouldn’t bet the world on it.