LLMs are a social technology applicable to the social problem of extinction from misaligned superintelligence. There is no law of nature that mandates building of superintelligence before alignment is well-understood. Alignment of LLMs seems to hold about as well as alignment of humans. It breaks down if the social problem of unstoppably pursuing discoveries that are too sweet remains unsolved. It breaks down with value drift, ignoring importance of reflective stability one step of change at a time.
LLMs might soon allow taking the steps faster, making discoveries that are particularly sweet faster. It’s not a change of frame the way superintelligence is a change of frame, both the problems and the solutions remain mostly the same. But the time is running out.
LLMs are a social technology applicable to the social problem of extinction from misaligned superintelligence. There is no law of nature that mandates building of superintelligence before alignment is well-understood. Alignment of LLMs seems to hold about as well as alignment of humans. It breaks down if the social problem of unstoppably pursuing discoveries that are too sweet remains unsolved. It breaks down with value drift, ignoring importance of reflective stability one step of change at a time.
LLMs might soon allow taking the steps faster, making discoveries that are particularly sweet faster. It’s not a change of frame the way superintelligence is a change of frame, both the problems and the solutions remain mostly the same. But the time is running out.