Trying to get the gist of this post… There’s the broad sweep of AI research across the decades, up to our contemporary era of deep learning, AlphaGo, GPT-3. In the 2000s, the school of thought associated with Eliezer, MIRI, and Less Wrong came into being. It was a pioneer in AI safety, but its core philosophy of preparing for the first comprehensively superhuman AI, remains idiosyncratic in a world focused on more specialized forms of AI.
There is a quote from Eliezer talking about “AI alignment” research, which would be that part of AI safety concerned with AI (whether general or special) getting the right goals. Apparently the alignment research community was more collaborative before OpenAI and truly big money got involved, but now it’s competitive and factional.
Part of the older culture of alignment research, was a reluctance to talk about competitive scenarios. The fear was that research into alignment per se would be derailed by a focus on western values vs nonwestern values, one company rather than another, and so on. But this came to pass anyway, and the thesis of this post is that there should now be more attention given to politics and how to foster cooperation.
My thoughts… I don’t know how much attention those topics should be given. But I do think it essential that someone keep trying to solve the problem of human-friendly general AI in a first-principles way… As I see it, the MIRI school of thought was born into a world that, at the level of civilizational trends, was already headed towards superhuman AI, and in an uncontrolled and unsafe way, and that has never stopped being the case.
In a world where numerous projects and research programs existed, that theoretically might cross the AI threshold unprepared, MIRI was a voice for planning ahead and doing it right, by figuring out how to do it right. For a while that was its distinctive quality, its “brand”, in the AI world… Now it’s a different time: AI and its applications are everywhere, and AI safety is an academic subdiscipline.
But for me, the big picture and the endgame is still the same. Technical progress occurs in an out-of-control way, the threshold of superhuman AI is still being approached on multiple fronts, and so while one can try to moderate the risks at a political or cultural level, the ultimate outcome still depends on whether or not the first project across the threshold is “safe” or “aligned” or “friendly”.
Trying to get the gist of this post… There’s the broad sweep of AI research across the decades, up to our contemporary era of deep learning, AlphaGo, GPT-3. In the 2000s, the school of thought associated with Eliezer, MIRI, and Less Wrong came into being. It was a pioneer in AI safety, but its core philosophy of preparing for the first comprehensively superhuman AI, remains idiosyncratic in a world focused on more specialized forms of AI.
There is a quote from Eliezer talking about “AI alignment” research, which would be that part of AI safety concerned with AI (whether general or special) getting the right goals. Apparently the alignment research community was more collaborative before OpenAI and truly big money got involved, but now it’s competitive and factional.
Part of the older culture of alignment research, was a reluctance to talk about competitive scenarios. The fear was that research into alignment per se would be derailed by a focus on western values vs nonwestern values, one company rather than another, and so on. But this came to pass anyway, and the thesis of this post is that there should now be more attention given to politics and how to foster cooperation.
My thoughts… I don’t know how much attention those topics should be given. But I do think it essential that someone keep trying to solve the problem of human-friendly general AI in a first-principles way… As I see it, the MIRI school of thought was born into a world that, at the level of civilizational trends, was already headed towards superhuman AI, and in an uncontrolled and unsafe way, and that has never stopped being the case.
In a world where numerous projects and research programs existed, that theoretically might cross the AI threshold unprepared, MIRI was a voice for planning ahead and doing it right, by figuring out how to do it right. For a while that was its distinctive quality, its “brand”, in the AI world… Now it’s a different time: AI and its applications are everywhere, and AI safety is an academic subdiscipline.
But for me, the big picture and the endgame is still the same. Technical progress occurs in an out-of-control way, the threshold of superhuman AI is still being approached on multiple fronts, and so while one can try to moderate the risks at a political or cultural level, the ultimate outcome still depends on whether or not the first project across the threshold is “safe” or “aligned” or “friendly”.