Why wouldn’t AGI build a superhuman understanding of ethics, which it would then use to guide its decision-making?
kaputmi
Karma: 3
Overconfidence bubbles
I think the gears-level models are really the key here. Without a gears-level model, you are flying blind, and the outside view is very helpful when you’re flying blind. But with a solid understanding of the causal mechanisms in a system, you don’t need to rely on others’ opinions to make good predictions and decisions.
It also comes with ~0 risk of paperclipping the world — Alphazero is godlike at chess without needing to hijack all resources for its purposes