I think this is a really interesting post and seems like a promising and tractable way to accelerate alignment research. It reminds me of Neuralink but seems more feasible at present. I also like how the post emphasizes differentially accelerating alignment because I think one of the primary risks of any kind of augmentation is that it just globally accelerates progress and has no net positive impact.
One sentence I noticed that seemed like a misdefinition was how the concept of a genie was defined:
An antithetical example to this is something like a genie, where the human outsources all of their agency to an external system that is then empowered to go off and optimize the world.
To me, this sounds more like a ‘sovereign’ as defined in Superintelligence whereas a genie just executes a command before waiting for the next command. Though the difference doesn’t seem that big since both types of systems take action.
A key concept I thought was missing was Amdahl’s Law which is a formula that calculates the maximum theoretical speedup of a computation given the percentage of the computation that can be parallelized. The formula is S=11−p. I think it’s also relevant here: if 50% of work can be delegated to a model, the maximum speedup is a factor of 2 because then there will only be half as much work for the human to do. If 90% can be delegated, the maximum speedup is 10.
Also, maybe it would be valuable to have more thinking focused on the human component of the system: ideas about productivity, cognitive enhancement, or alignment. Though I think these ideas are beyond the scope of the post.
I think this is a really interesting post and seems like a promising and tractable way to accelerate alignment research. It reminds me of Neuralink but seems more feasible at present. I also like how the post emphasizes differentially accelerating alignment because I think one of the primary risks of any kind of augmentation is that it just globally accelerates progress and has no net positive impact.
One sentence I noticed that seemed like a misdefinition was how the concept of a genie was defined:
To me, this sounds more like a ‘sovereign’ as defined in Superintelligence whereas a genie just executes a command before waiting for the next command. Though the difference doesn’t seem that big since both types of systems take action.
A key concept I thought was missing was Amdahl’s Law which is a formula that calculates the maximum theoretical speedup of a computation given the percentage of the computation that can be parallelized. The formula is S=11−p. I think it’s also relevant here: if 50% of work can be delegated to a model, the maximum speedup is a factor of 2 because then there will only be half as much work for the human to do. If 90% can be delegated, the maximum speedup is 10.
Also, maybe it would be valuable to have more thinking focused on the human component of the system: ideas about productivity, cognitive enhancement, or alignment. Though I think these ideas are beyond the scope of the post.