Epistemic status: Making guesses, though the conclusion feels right to me.
But there were many others (e.g. Fermi, Szilard, Oppenheimer, Bethe, Teller, Von Neumann, Wigner) who seemed… to truly grok the stakes.
Is it the case that many of these people ‘actually tried’ before (a) the problem became primarily an engineering problem and (b) it was a crucial project during wartime and backed by the establishment?
This is probably a confused analogy, but I’ll say it anyway: if they didn’t, the analogy would be that in a world where there was war fought using advanced machine learning, there might be a similar set of top researchers doing research into adversarial examples, reward hacking and other concrete problems in ML—only doing things pretty closely related to actually building things, and only building things close to military applications.
This has many implications, but if you take MIRI’s view that new basic theory of embedded agency / grounding for optimisation is required, the other side of the analogy is that there was e.g. no team rushing to unify quantum mechanics and gravity during WWII.
So I suppose (to MIRI) the problem does seem more abstract and not built for human brains to reason about than the situation even with the atom bomb.
I think that if you think this, you should update hard against promoting the open problems as ‘important’ and instead focus on making the problem ‘interesting’.
(Would appreciate someone chiming in about whether (a) and (b) are accurate.)
I think nuclear physics then had more of an established paradigm than AI safety has now; from what I understand, building a bomb was considered a hard, unsolved problem, but one which it was broadly known how to solve. So I think the answer to A is basically “no.”
A bunch of people on the above list do seem to me to have actually tried before the project was backed by the establishment, though—from what I understand Fermi, Szilard, Wigner and Teller were responsible for getting the government involved in the first place. But their actions seem mostly to have been in the domains of politics, engineering and paradigmatic science, rather than new-branch-of-science-style theorizing.
(I do suspect it might be useful to find more ways of promoting the problem chiefly as interesting).
Epistemic status: Making guesses, though the conclusion feels right to me.
Is it the case that many of these people ‘actually tried’ before (a) the problem became primarily an engineering problem and (b) it was a crucial project during wartime and backed by the establishment?
This is probably a confused analogy, but I’ll say it anyway: if they didn’t, the analogy would be that in a world where there was war fought using advanced machine learning, there might be a similar set of top researchers doing research into adversarial examples, reward hacking and other concrete problems in ML—only doing things pretty closely related to actually building things, and only building things close to military applications.
This has many implications, but if you take MIRI’s view that new basic theory of embedded agency / grounding for optimisation is required, the other side of the analogy is that there was e.g. no team rushing to unify quantum mechanics and gravity during WWII.
So I suppose (to MIRI) the problem does seem more abstract and not built for human brains to reason about than the situation even with the atom bomb.
I think that if you think this, you should update hard against promoting the open problems as ‘important’ and instead focus on making the problem ‘interesting’.
(Would appreciate someone chiming in about whether (a) and (b) are accurate.)
I think nuclear physics then had more of an established paradigm than AI safety has now; from what I understand, building a bomb was considered a hard, unsolved problem, but one which it was broadly known how to solve. So I think the answer to A is basically “no.”
A bunch of people on the above list do seem to me to have actually tried before the project was backed by the establishment, though—from what I understand Fermi, Szilard, Wigner and Teller were responsible for getting the government involved in the first place. But their actions seem mostly to have been in the domains of politics, engineering and paradigmatic science, rather than new-branch-of-science-style theorizing.
(I do suspect it might be useful to find more ways of promoting the problem chiefly as interesting).