Hi Adam! I strongly agree RE the benefits of longstanding institutions. When I went through YC I found it really striking how few SF engineers understood the degree to which American academic institutions are totally load-bearing for industry. For all its foibles, there’s no other institution as effective as the PhD program for producing new foundational research and skilled researchers.
I hadn’t seen your post on regularly regenerating all code. I just read it and I think it’s really good! I’m curious what you think about the problem I bring up, of spec elicitation (what Mike is working on). IE, how do we know which properties of pre-existing software are important to preserve when we rewrite, formalize, and so forth? Is this something you’ve thought about, or have opinions about? It seems very challenging to me (I am not at all sure that this is something we can rely on AI for ..) but maybe I am thinking about the problem incorrectly.
Take a canonical-sounding example: we want to replace the control software for a particular process within a nuclear power plant. It was designed based on the experience and intuitions of nuclear engineers decades ago. With minimal construction of new nuclear plants in the U.S., that kind of expertise isn’t available anymore. We’re tempted to go through a perilous exercise, reconstructing the requirements of a single component.
But I expect that there’s a failure of imagination in assuming we want to change just this piece of software. The whole nuclear plant was designed by puny human brains, and AI will be able to do better. It should redesign the whole plant, with a simpler specification connected to the fundamental goal of producing electricity safely. Heck, we should even expect AI science engines inventing entirely new styles of electricity generation.
Hi Adam! I strongly agree RE the benefits of longstanding institutions. When I went through YC I found it really striking how few SF engineers understood the degree to which American academic institutions are totally load-bearing for industry. For all its foibles, there’s no other institution as effective as the PhD program for producing new foundational research and skilled researchers.
I hadn’t seen your post on regularly regenerating all code. I just read it and I think it’s really good! I’m curious what you think about the problem I bring up, of spec elicitation (what Mike is working on). IE, how do we know which properties of pre-existing software are important to preserve when we rewrite, formalize, and so forth? Is this something you’ve thought about, or have opinions about? It seems very challenging to me (I am not at all sure that this is something we can rely on AI for ..) but maybe I am thinking about the problem incorrectly.
I think spec elicitation from legacy code won’t matter much in the long term. The reason connects to how end-to-end verification catches mistakes in specifications of internal components. We should grow the scope of well-encapsulated systems to minimize components that aren’t internal, in a suitable sense.
Take a canonical-sounding example: we want to replace the control software for a particular process within a nuclear power plant. It was designed based on the experience and intuitions of nuclear engineers decades ago. With minimal construction of new nuclear plants in the U.S., that kind of expertise isn’t available anymore. We’re tempted to go through a perilous exercise, reconstructing the requirements of a single component.
But I expect that there’s a failure of imagination in assuming we want to change just this piece of software. The whole nuclear plant was designed by puny human brains, and AI will be able to do better. It should redesign the whole plant, with a simpler specification connected to the fundamental goal of producing electricity safely. Heck, we should even expect AI science engines inventing entirely new styles of electricity generation.
The general pattern should be increasingly large bubbles of deliberate legibility to AI, driven by the economic costs of specification problems that are too complex.