Well, gosh, the formal-methods crowd has dominated the first posts of the day here! Naturally I should respond when my name comes up. ;-)
I’m actually already spending a majority of my research time on projects related to AI safety, and I’ll have more to say on my blog as they get mature-enough. I already wrote (cross-posting here) on how I’m relatively optimistic on our chance to counter this cyberpocalypse, especially by regularly regenerating all code following latest best practices. I think if we get too rushed we’ll miss the chance for bigger improvements to development methods and focus too much on hardening rickety conventional code bases. If we get a singularity big enough to blast all companies out of existence by exploiting their software, we should also be ready to automate away the development of better software.
Independently of that kind of bet on the direction of the field, I’ll mention that old-fashioned university research groups are pretty darned good at both the first-order goal of developing tools for securing systems and the second-order goal of training students to do that kind of work themselves. Funding in U.S. academia has gotten weird lately, and it’s a real opportunity for funders who care about this area to make themselves known to the people who can do the work. (Anyone whose attention that sentence gets is welcome to contact me!) I’d be willing to bet that single U.S. research groups today achieve better throughput in producing people very well-trained for this task than all AI-safety fellowship programs put together. That ratio could change with more investment, but maybe a little more Edmund Burke-style appreciation for longstanding institutions is also valuable.
Hi Adam! I strongly agree RE the benefits of longstanding institutions. When I went through YC I found it really striking how few SF engineers understood the degree to which American academic institutions are totally load-bearing for industry. For all its foibles, there’s no other institution as effective as the PhD program for producing new foundational research and skilled researchers.
I hadn’t seen your post on regularly regenerating all code. I just read it and I think it’s really good! I’m curious what you think about the problem I bring up, of spec elicitation (what Mike is working on). IE, how do we know which properties of pre-existing software are important to preserve when we rewrite, formalize, and so forth? Is this something you’ve thought about, or have opinions about? It seems very challenging to me (I am not at all sure that this is something we can rely on AI for ..) but maybe I am thinking about the problem incorrectly.
Take a canonical-sounding example: we want to replace the control software for a particular process within a nuclear power plant. It was designed based on the experience and intuitions of nuclear engineers decades ago. With minimal construction of new nuclear plants in the U.S., that kind of expertise isn’t available anymore. We’re tempted to go through a perilous exercise, reconstructing the requirements of a single component.
But I expect that there’s a failure of imagination in assuming we want to change just this piece of software. The whole nuclear plant was designed by puny human brains, and AI will be able to do better. It should redesign the whole plant, with a simpler specification connected to the fundamental goal of producing electricity safely. Heck, we should even expect AI science engines inventing entirely new styles of electricity generation.
Well, gosh, the formal-methods crowd has dominated the first posts of the day here! Naturally I should respond when my name comes up. ;-)
I’m actually already spending a majority of my research time on projects related to AI safety, and I’ll have more to say on my blog as they get mature-enough. I already wrote (cross-posting here) on how I’m relatively optimistic on our chance to counter this cyberpocalypse, especially by regularly regenerating all code following latest best practices. I think if we get too rushed we’ll miss the chance for bigger improvements to development methods and focus too much on hardening rickety conventional code bases. If we get a singularity big enough to blast all companies out of existence by exploiting their software, we should also be ready to automate away the development of better software.
Independently of that kind of bet on the direction of the field, I’ll mention that old-fashioned university research groups are pretty darned good at both the first-order goal of developing tools for securing systems and the second-order goal of training students to do that kind of work themselves. Funding in U.S. academia has gotten weird lately, and it’s a real opportunity for funders who care about this area to make themselves known to the people who can do the work. (Anyone whose attention that sentence gets is welcome to contact me!) I’d be willing to bet that single U.S. research groups today achieve better throughput in producing people very well-trained for this task than all AI-safety fellowship programs put together. That ratio could change with more investment, but maybe a little more Edmund Burke-style appreciation for longstanding institutions is also valuable.
Hi Adam! I strongly agree RE the benefits of longstanding institutions. When I went through YC I found it really striking how few SF engineers understood the degree to which American academic institutions are totally load-bearing for industry. For all its foibles, there’s no other institution as effective as the PhD program for producing new foundational research and skilled researchers.
I hadn’t seen your post on regularly regenerating all code. I just read it and I think it’s really good! I’m curious what you think about the problem I bring up, of spec elicitation (what Mike is working on). IE, how do we know which properties of pre-existing software are important to preserve when we rewrite, formalize, and so forth? Is this something you’ve thought about, or have opinions about? It seems very challenging to me (I am not at all sure that this is something we can rely on AI for ..) but maybe I am thinking about the problem incorrectly.
I think spec elicitation from legacy code won’t matter much in the long term. The reason connects to how end-to-end verification catches mistakes in specifications of internal components. We should grow the scope of well-encapsulated systems to minimize components that aren’t internal, in a suitable sense.
Take a canonical-sounding example: we want to replace the control software for a particular process within a nuclear power plant. It was designed based on the experience and intuitions of nuclear engineers decades ago. With minimal construction of new nuclear plants in the U.S., that kind of expertise isn’t available anymore. We’re tempted to go through a perilous exercise, reconstructing the requirements of a single component.
But I expect that there’s a failure of imagination in assuming we want to change just this piece of software. The whole nuclear plant was designed by puny human brains, and AI will be able to do better. It should redesign the whole plant, with a simpler specification connected to the fundamental goal of producing electricity safely. Heck, we should even expect AI science engines inventing entirely new styles of electricity generation.
The general pattern should be increasingly large bubbles of deliberate legibility to AI, driven by the economic costs of specification problems that are too complex.