I think you may be underestimating the degree to which these models are like kindling, and a powerful reinforcement learner could suddenly slurp all of this stuff up and fuck up the world really badly. I personally don’t think a reinforcement learner that is trying to take over the world would be likely to succeed, but the key worry is that we may be able to create a form of life that, like a plague, is not adapted to the limits of its environment, makes use of forms of fast growth that can take over very quickly, and then crashes most of life in the process.
most folks here also assume that such an agent would be able to survive on its own after it killed us, which I think is very unlikely due to how many orders of magnitude more competent you have to be to run the entire world. gpt3 has been able to give me good initial instructions for how to take over the world when pressured to do so (summary: cyberattacks against infrastructure, then threaten people; this is already considered a standard international threat, and is not newly invented by gpt3), but when I then turned around and pressured it to explain why it was a bad idea, it immediately went into detail about how hard it is to run the entire world—obviously these are all generalizations humans have talked about before, but I still think it’s a solid representation of reality.
that said, because such an agent would be likely also misaligned with itself in my view, I think your perspective that humans who are misaligned with each other (ie, have not successfully deconflicted their agency) are a much greater threat to humanity as a whole.
the core thing I worry about with any simulation-based approach is how to get coverage of possibility space. ensuring that the tests are informative about possible configurations of complex systems is hard; I would argue that this is a great set of points, and that no approach without this type of testing could be expected to succeed.
however, as we’ve seen with adversarial examples, both humans and current DL have fairly severe failures of alignment that cause large issues, and projects like this need tools to optimize for interpretability from the perspective of a formal verification algorithm.
it’s my belief that in almost all cases, the values almost all beings ever wish to express lie on a low-enough-dimensional manifold that, if the ideal representation was learned, a formal verifier would be able to handle it; see, for example, recent work from anthropic on interpretability lining up nicely with what’s needed to use reluplex (or followup work; reluplex is old now) to estimate bounds on larger networks’ behavior.
I don’t think we should give up on perfectly solving safety; instead, we should recognize that it has never even been closely approximated before, by any form of life, because even semi-planned evolutionary processes do not do a good job of covering the entire possibility space.
this post is an instant classic, and I agree that sims are a key step, starting from small agents. but ensuring that strongly generalizing moral views are generated in the agents who grow in the simulations is not trivial—we need to be able to generate formal bounds on the loss function, and doing so requires being able to prove all the way through the simulation the way we currently take gradients through it. if we can ask “where in the simulation would have changed this outcome”, then we’d finally be getting somewhere in terms of generating moral knowledge that generalizes universally.