I’m not sure that design will end up being as simple as this picture makes it look, no matter how well we understand it—it seems like factorization is one kind of activity in design, but it feels like overall “design” is being used as a kind of catch-all that is probably very complicated.
An important distinction for me is: does the artifact work because of the story (as in “design”), or does the artifact work because of the evaluation (as in search)?
This isn’t so clean, since:
Most artifacts work for a combination of the two reasons—I design a thing then test it and need a few iterations—there is some quantitative story where both factors almost always play a role for practical artifacts.
There seem to many other reasons things work (e.g. “it’s similar to other things that worked” seems to play a super important role in both design and search).
A story seems like it’s the same kind of thing as an artifact, and we could also talk about where *it* comes from. A story that plays a role in a design itself comes from some combination of search and design.
During design it seems likely that humans rely very extensively on searching against mental models, which may not be introspectively available to us as a search but seems like it has similar properties.
Despite those and more complexities, it feels to me like if there is a clean abstraction it’s somewhere in that general space, about the different reasons why a thing can work.
Post-hoc stories are clearly *not* the “reason why things work” (at least at this level of explanation). But also if you do jointly search for a model+helpful story about it, the story still isn’t the reason why the model works, and from a safety perspective it might be similarly bad.
Hey thank you for your thoughts on this post, friend
overall “design” is being used as a kind of catch-all that is probably very complicated
Yes it may be that “automating design” is really just a rephrasing of the whole AI problem. But I’m hopeful that it’s not. Keep in mind that we only have to be competitive with machine learning, which means we only have to be able to automate the design of artifacts that can also be produced by black box search. This seems to me to be a lower bar than automating all human capacity for design, or automating design in general.
In fact you might think the machine learning problem statement of “search a hypothesis space for a policy that performs well empirically” is itself a restatement of the whole AI problem, and perhaps a full solution to this problem would in fact be a general solution to AI. But in practice we’ve been able to make incremental progress in machine learning without needing to solve any AI-complete parts of the problem space (yet).
does the artifact work because of the story (as in “design”), or does the artifact work because of the evaluation (as in search)?
Interesting. I wasn’t thinking of the story as playing any causal role in making the artifact work. (Though it’s very important that the story convey how the artifact actually works, rather than being optimized for merely being convincing.)
Can you say more about what it would mean for an artifact to work because of the story?
This isn’t so clean, since [...] Most artifacts work for a combination of the two reasons—I design a thing then test it and need a few iterations
Yup very much agreed. But the trial-and-error part of design seems to me very different from the evaluate-and-gradient-step part of search. When I write some code and test it and it doesn’t work, I almost always get some insight into some general failure mode that I failed to anticipate before, and I update both my story and my code. This might be realizing that some API sometimes return null instead of a string for a certain field, or noticing that negative numbers on the command line look like flags to the argument parser. Perhaps we could view these as gradient steps in story space. Or perhaps that’s too much of a stretch.
There seem to many other reasons things work (e.g. “it’s similar to other things that worked” seems to play a super important role in both design and search).
Yes when we look at the evolution of, say, a particular product made by a particular company over a timespan of years, it seems that there is a local search happening. Same thing if we look at, say, manufacturing processes for semiconductors. I’m keen to spend more time thinking about this.
A story seems like it’s the same kind of thing as an artifact, and we could also talk about where it comes from. A story that plays a role in a design itself comes from some combination of search and design.
Yeah this seems important. If we view a story as a particular kind of artifact then it would be good to get clear on what exactly identifies an artifact as a story. There is a section in the Genesis manifesto (audacious GOFAI manifesto from some cog sci folks at MIT) that talks about internal and external stories, where an internal story is a story represented in mental wetware, and an external story is any object that spontaneously gives rise to an internal story upon examination.
During design it seems likely that humans rely very extensively on searching against mental models, which may not be introspectively available to us as a search but seems like it has similar properties.
In my very limited experiment I was continuously surprised about how non-search-like the practical experience of design was to me. But yes there is a lot happening beneath conscious awareness so I shouldn’t update too heavily on this.
if you do jointly search for a model+helpful story about it, the story still isn’t the reason why the model works, and from a safety perspective it might be similarly bad
Well when I buy an air conditioner that comes with an instruction manual that includes some details on how the air conditioner is constructed, it’s not literally the case that the air conditioner works because of the physical instruction manual. The physical instruction manual is superfluous to the correct functioning of the physical air conditioner. And it’s also quite possible that the physical instruction manual was constructed after the physical air conditioner was already constructed, and this doesn’t rule out the helpfulness of the instruction manual.
What’s important is that the instruction manual conveys a true account of how the air conditioner really works.
Now if we really nailed down what it means for a story to give a true account of how an artifact really works then perhaps we could search over (artifact, story) pairs. But this seems like deeply inaccessible information to me, so I agree this kind of solution can’t work. If I’m a search algorithm searching over (artifact, story) pairs then I have no understanding whatsoever about how the artifacts I’m constructing work. I’m just searching. On what basis could I possibly discern whether a certain story faithfully captures how it is that a certain artifact really works?
What we actually need is a process for constructing artifacts that builds them up piece by piece, so that a story can be constructed piece by piece in parallel. Or something else entirely. But it just doesn’t seem that search is enough here. I might try to formalize this argument at some point.
I liked this post.
I’m not sure that design will end up being as simple as this picture makes it look, no matter how well we understand it—it seems like factorization is one kind of activity in design, but it feels like overall “design” is being used as a kind of catch-all that is probably very complicated.
An important distinction for me is: does the artifact work because of the story (as in “design”), or does the artifact work because of the evaluation (as in search)?
This isn’t so clean, since:
Most artifacts work for a combination of the two reasons—I design a thing then test it and need a few iterations—there is some quantitative story where both factors almost always play a role for practical artifacts.
There seem to many other reasons things work (e.g. “it’s similar to other things that worked” seems to play a super important role in both design and search).
A story seems like it’s the same kind of thing as an artifact, and we could also talk about where *it* comes from. A story that plays a role in a design itself comes from some combination of search and design.
During design it seems likely that humans rely very extensively on searching against mental models, which may not be introspectively available to us as a search but seems like it has similar properties.
Despite those and more complexities, it feels to me like if there is a clean abstraction it’s somewhere in that general space, about the different reasons why a thing can work.
Post-hoc stories are clearly *not* the “reason why things work” (at least at this level of explanation). But also if you do jointly search for a model+helpful story about it, the story still isn’t the reason why the model works, and from a safety perspective it might be similarly bad.
Hey thank you for your thoughts on this post, friend
Yes it may be that “automating design” is really just a rephrasing of the whole AI problem. But I’m hopeful that it’s not. Keep in mind that we only have to be competitive with machine learning, which means we only have to be able to automate the design of artifacts that can also be produced by black box search. This seems to me to be a lower bar than automating all human capacity for design, or automating design in general.
In fact you might think the machine learning problem statement of “search a hypothesis space for a policy that performs well empirically” is itself a restatement of the whole AI problem, and perhaps a full solution to this problem would in fact be a general solution to AI. But in practice we’ve been able to make incremental progress in machine learning without needing to solve any AI-complete parts of the problem space (yet).
Interesting. I wasn’t thinking of the story as playing any causal role in making the artifact work. (Though it’s very important that the story convey how the artifact actually works, rather than being optimized for merely being convincing.)
Can you say more about what it would mean for an artifact to work because of the story?
Yup very much agreed. But the trial-and-error part of design seems to me very different from the evaluate-and-gradient-step part of search. When I write some code and test it and it doesn’t work, I almost always get some insight into some general failure mode that I failed to anticipate before, and I update both my story and my code. This might be realizing that some API sometimes return null instead of a string for a certain field, or noticing that negative numbers on the command line look like flags to the argument parser. Perhaps we could view these as gradient steps in story space. Or perhaps that’s too much of a stretch.
Yes when we look at the evolution of, say, a particular product made by a particular company over a timespan of years, it seems that there is a local search happening. Same thing if we look at, say, manufacturing processes for semiconductors. I’m keen to spend more time thinking about this.
Yeah this seems important. If we view a story as a particular kind of artifact then it would be good to get clear on what exactly identifies an artifact as a story. There is a section in the Genesis manifesto (audacious GOFAI manifesto from some cog sci folks at MIT) that talks about internal and external stories, where an internal story is a story represented in mental wetware, and an external story is any object that spontaneously gives rise to an internal story upon examination.
In my very limited experiment I was continuously surprised about how non-search-like the practical experience of design was to me. But yes there is a lot happening beneath conscious awareness so I shouldn’t update too heavily on this.
Well when I buy an air conditioner that comes with an instruction manual that includes some details on how the air conditioner is constructed, it’s not literally the case that the air conditioner works because of the physical instruction manual. The physical instruction manual is superfluous to the correct functioning of the physical air conditioner. And it’s also quite possible that the physical instruction manual was constructed after the physical air conditioner was already constructed, and this doesn’t rule out the helpfulness of the instruction manual.
What’s important is that the instruction manual conveys a true account of how the air conditioner really works.
Now if we really nailed down what it means for a story to give a true account of how an artifact really works then perhaps we could search over (artifact, story) pairs. But this seems like deeply inaccessible information to me, so I agree this kind of solution can’t work. If I’m a search algorithm searching over (artifact, story) pairs then I have no understanding whatsoever about how the artifacts I’m constructing work. I’m just searching. On what basis could I possibly discern whether a certain story faithfully captures how it is that a certain artifact really works?
What we actually need is a process for constructing artifacts that builds them up piece by piece, so that a story can be constructed piece by piece in parallel. Or something else entirely. But it just doesn’t seem that search is enough here. I might try to formalize this argument at some point.