I think this doesn’t make sense capabilities-wise for solving genuinely hard scientific/technological/mathematical/philosophical problems such as the strawberry problem. (It makes sense when the big task has a basically known decomposition into a large number of small easy tasks though.) A central issue is that good high-level decisions are very important, there are very many of them, and they need to be made with deep understanding of the domain/[design space], which the human in this setup doesn’t have by default for hard problems and which can only be gained by spending a lot of time understanding novel stuff. Like, it would be extremely silly to have a setup in which a human with no university-level math education is suggesting high-level riemann hypothesis proof strategies to Terence Tao. That human could not be contributing basically anything positive to Tao’s ability to solve the problem.
Maybe the following is a key observation in this (you might have considered it already, but including it just in case):
The example we should have in mind is NOT having a research problem that takes an individual human 1000 years to solve, which has some clever decomposition into 5 problems, each of which takes 200 years to solve, with the human needing to provide the 5 problem decomposition and the AI solving the 200-year problems. This is NOT what we should have in mind because if the AI can solve 200 year problems, then we are already very close to making an AI that can autonomously solve 1000 year problems (in fact in practice for these particular numbers I expect we would be there basically instantly). Instead, for the question to be interesting, we should imagine the AI being able to solve much shorter problems, like idk 1-week problems. In that case, even if there is some reasonable decomposition into eventually 1-week tasks, it will be a really big complicated object.
One also gets a bound on the capabilities-usefulness of this scheme from the consideration that if decomposition work is easy enough for a task, then one should just be able to have an AI with a small time horizon do it as well, at least if we trust time-horizon-thinking. And so either decomposition work is easy and you could replace the human in this scheme with that AI (or better yet, just have the AI solving the subproblems also make decomposition decisions) or decomposition work is hard and it takes a long time for this AI-human system to do the task. Quantitatively, this is saying that if a task can be done by a human-AI system but not an only-AI system, then it should take at least the AI time horizon in wall clock time. I guess this conclusion would be softened if decomposition-work is outlier-hard among things with the same human time horizon, which seems plausible.
That said, one can of course get some speedup as a human researcher from asking AIs to sometimes do small tasks, I’m just doubting that this can give a huge speedup for solving hard scientific/technological/mathematical/philosophical problems without the AI being basically able to solve them autonomously.
I think the strawberry problem is very different in character from proving the Riemann hypothesis. I expect that the bottleneck for (dis)proving the Riemann Hypothesis is having sufficient insight, where the bottleneck for the Strawberry Problem is a huge volume of conceptually simple but quite finicky-in-practice work. Most of that work probably looks like “prove that an approach to a sub-sub-sub problem can work at all, then do a bunch of schleppy stuff to build tools and process to make it possible to do that thing cheaply and reliably”.
Even if LLMs are better than humans at most tasks, their failure modes are less than perfectly correlated with human failure modes. This is particularly true of tasks which involve rapid processing of novel visual input where no training data exists to provide ground truth (e.g. because they’re videos of attempts to use a newly-developed tool).
Basically, I’m not picturing “the AI can do all tasks up to 2 week time horizon with near-perfect reliability but requires a human to do stuff which requires planning on a longer horizon”, I’m picturing “the AI can do almost but not all of the subtasks required to operate autonomously for two weeks, and the failure mode looks like getting stuck without realizing it’s stuck, and humans can recognize when that happens”.
I think this is an area where the expectations of math-brained people diverge sharply from those of biology-brained people.
Aren’t we extremely confused about how one would go about making two strawberries which are identical down to the cellular level? Like, the simplest path might go through nanotech or some other pretty crazy thing? (Being able to do that probably implies it wouldn’t be much harder to mass-manufacture humans who are identical down to the cellular level?) I feel like you’re saying you basically know how to reduce it to a bunch of grad student gruntwork (or at least think someone else could) and that sounds really wild to me!
I don’t think we’re particularly confused about what we’d need to do to make two strawberries which are identical down to the cellular level. It looks like a very hard problem, in that it involves figuring out how to a very large number of moderately hard things at scale, and how to combine those new techniques together. And it looks like a problem where a lot of the things you learn in the process of solving it would be particular to the specific species and in fact the specific genome of the strawberry you chose to duplicate, rather than providing valuable generalized insights.
Place, onto this particular plate here, two strawberries identical down to the cellular but not molecular level
At an extremely high level, the most obvious approach that satisfies that looks something like
Figure out how to preserve this particular plate here indefinitely, at least for a few decades and possibly centuries
Figure out how to obtain fertilized strawberry ovules that are genetically identical to each other and approximately identical in terms of physical characteristics (obnoxious, but examples of Polyembryony exist in plants, and failing that you can cross two fully homozygous parents)
Figure out how to grow ovules into fruits in a controlled environment, ideally not attached to plants (there is already some existing work on tomatoes, but repeating the feat in strawberries is likely to be quite difficult and time consuming in a way that previous work on tomatoes doesn’t help all that much with)
Pare your strawberry’s genome down to the absolute bare minimum that can still be considered a “strawberry”, tailored to grow in an environment where things vary as little as possible (e.g. growing in the dark in microgravity at constant temperature with externally-provided auxin, maybe even make the cells division-incompetent unless a specific trigger (light?) is present).
Now that you have a programmable strawberry, debug it until you are confident in your ability to deterministically grow these programmable strawberries to be identical
Grow two
Plate them
???
Profit.
Oh wait. There’s no actual value in this. Go bankrupt instead.
This is obviously extremely hard. If you were to attempt this with mostly-human labor (and still computers, and an advanced industrial civilization backing them), I’d guess you’re looking at something like a research team of a thousand people working for a hundred years. But it’s not hard because we’re confused about how to do it, it’s hard because doing it requires learning millions of little fiddly things about the real world. It’s a “reality has a surprising amount of detail” shaped problem rather than a “we need a single big insight that we aren’t smart enough to derive” shaped problem.
I haven’t thought a lot about this but my guess is that this approach basically can’t work because chaos is a thing so you need to determine parameters on the fly so you need to put some controllers inside
edit: oh i guess maybe you’re suggesting controlling cell division directly very precisely with optical stimulation at precise points inside the strawberry somehow? hmm. i guess you also need to control cell death very precisely
edit 2: oh also you have a major chicken and egg problem with the ovules and the surrounding structure in the parent plant right?
oh i guess maybe you’re suggesting controlling cell division directly very precisely with optical stimulation at precise points inside the strawberry somehow
Yep. Probably not just cell division either, if you’re attempting this it’s probably also wise to use caged auxin for controlling growth, rather than leaving hormonal regulation up to the individual cells. And even with those tricks, I expect that this is a very very hard problem, and also that in the course of trying to solve it this way the researchers would discover reasons why the exact thing I spitballed wouldn’t work (but would likely also discover other angles of attack that were more promising based on their observations). You probably need to be able to target cell growth even more finely than the cellular level, since in order to prevent small differences spiralling into large differences a few divisions down the road, you need to be able to make fine corrections to cell size/shape/orientation based on observations.
oh also you have a major chicken and egg problem with the ovules and the surrounding structure in the parent plant right?
To develop a system with which to study fruit ripening, in vitro ovary cultures were initiated from tomato flowers.
You really don’t want to be dealing with the parent plant at all. It looks to me like it’s probably straightforward to handle this part (relative to the other parts of this project), but again that’s the sort of thing that you have to actually go out and interact with the external world to determine for sure.
FWIW I do expect that Yudkowsky expected that this problem would require the solver to solve bioprinting or something “sci-fi” like that rather than “just” absurd amounts of iterative work. But the problem as stated seems solvable through absurd amounts of iterative work, and it is worlds where solving problems involves lots of iterative work of the “reality has a surprising amount of detail” type that I expect humans remain relevant for a while.
I think this doesn’t make sense capabilities-wise for solving genuinely hard scientific/technological/mathematical/philosophical problems such as the strawberry problem. (It makes sense when the big task has a basically known decomposition into a large number of small easy tasks though.) A central issue is that good high-level decisions are very important, there are very many of them, and they
Maybe the following is a key observation in this (you might have considered it already, but including it just in case):
The example we should have in mind is NOT having a research problem that takes an individual human 1000 years to solve, which has some clever decomposition into 5 problems, each of which takes 200 years to solve, with the human needing to provide the 5 problem decomposition and the AI solving the 200-year problems. This is NOT what we should have in mind because if the AI can solve 200 year problems, then we are already very close to making an AI that can autonomously solve 1000 year problems (in fact in practice for these particular numbers I expect we would be there basically instantly). Instead, for the question to be interesting, we should imagine the AI being able to solve much shorter problems, like idk 1-week problems. In that case, even if there is some reasonable decomposition into eventually 1-week tasks, it will be a really big complicated object.
One also gets a bound on the capabilities-usefulness of this scheme from the consideration that if decomposition work is easy enough for a task, then one should just be able to have an AI with a small time horizon do it as well, at least if we trust time-horizon-thinking. And so either decomposition work is easy and you could replace the human in this scheme with that AI (or better yet, just have the AI solving the subproblems also make decomposition decisions) or decomposition work is hard and it takes a long time for this AI-human system to do the task. Quantitatively, this is saying that if a task can be done by a human-AI system but not an only-AI system, then it should take at least the AI time horizon in wall clock time. I guess this conclusion would be softened if decomposition-work is outlier-hard among things with the same human time horizon, which seems plausible.
That said, one can of course get some speedup as a human researcher from asking AIs to sometimes do small tasks, I’m just doubting that this can give a huge speedup for solving hard scientific/technological/mathematical/philosophical problems without the AI being basically able to solve them autonomously.
I think the strawberry problem is very different in character from proving the Riemann hypothesis. I expect that the bottleneck for (dis)proving the Riemann Hypothesis is having sufficient insight, where the bottleneck for the Strawberry Problem is a huge volume of conceptually simple but quite finicky-in-practice work. Most of that work probably looks like “prove that an approach to a sub-sub-sub problem can work at all, then do a bunch of schleppy stuff to build tools and process to make it possible to do that thing cheaply and reliably”.
Even if LLMs are better than humans at most tasks, their failure modes are less than perfectly correlated with human failure modes. This is particularly true of tasks which involve rapid processing of novel visual input where no training data exists to provide ground truth (e.g. because they’re videos of attempts to use a newly-developed tool).
Basically, I’m not picturing “the AI can do all tasks up to 2 week time horizon with near-perfect reliability but requires a human to do stuff which requires planning on a longer horizon”, I’m picturing “the AI can do almost but not all of the subtasks required to operate autonomously for two weeks, and the failure mode looks like getting stuck without realizing it’s stuck, and humans can recognize when that happens”.
I think this is an area where the expectations of math-brained people diverge sharply from those of biology-brained people.
Aren’t we extremely confused about how one would go about making two strawberries which are identical down to the cellular level? Like, the simplest path might go through nanotech or some other pretty crazy thing? (Being able to do that probably implies it wouldn’t be much harder to mass-manufacture humans who are identical down to the cellular level?) I feel like you’re saying you basically know how to reduce it to a bunch of grad student gruntwork (or at least think someone else could) and that sounds really wild to me!
I don’t think we’re particularly confused about what we’d need to do to make two strawberries which are identical down to the cellular level. It looks like a very hard problem, in that it involves figuring out how to a very large number of moderately hard things at scale, and how to combine those new techniques together. And it looks like a problem where a lot of the things you learn in the process of solving it would be particular to the specific species and in fact the specific genome of the strawberry you chose to duplicate, rather than providing valuable generalized insights.
As a reminder, the exact wording of the Strawberry Problem is
At an extremely high level, the most obvious approach that satisfies that looks something like
Figure out how to preserve this particular plate here indefinitely, at least for a few decades and possibly centuries
Figure out how to obtain fertilized strawberry ovules that are genetically identical to each other and approximately identical in terms of physical characteristics (obnoxious, but examples of Polyembryony exist in plants, and failing that you can cross two fully homozygous parents)
Figure out how to grow ovules into fruits in a controlled environment, ideally not attached to plants (there is already some existing work on tomatoes, but repeating the feat in strawberries is likely to be quite difficult and time consuming in a way that previous work on tomatoes doesn’t help all that much with)
Pare your strawberry’s genome down to the absolute bare minimum that can still be considered a “strawberry”, tailored to grow in an environment where things vary as little as possible (e.g. growing in the dark in microgravity at constant temperature with externally-provided auxin, maybe even make the cells division-incompetent unless a specific trigger (light?) is present).
Now that you have a programmable strawberry, debug it until you are confident in your ability to deterministically grow these programmable strawberries to be identical
Grow two
Plate them
???
Profit.
Oh wait. There’s no actual value in this. Go bankrupt instead.
This is obviously extremely hard. If you were to attempt this with mostly-human labor (and still computers, and an advanced industrial civilization backing them), I’d guess you’re looking at something like a research team of a thousand people working for a hundred years. But it’s not hard because we’re confused about how to do it, it’s hard because doing it requires learning millions of little fiddly things about the real world. It’s a “reality has a surprising amount of detail” shaped problem rather than a “we need a single big insight that we aren’t smart enough to derive” shaped problem.
I haven’t thought a lot about this but my guess is that this approach basically can’t work because chaos is a thing so you need to determine parameters on the fly so you need to put some controllers inside
edit: oh i guess maybe you’re suggesting controlling cell division directly very precisely with optical stimulation at precise points inside the strawberry somehow? hmm. i guess you also need to control cell death very precisely
edit 2: oh also you have a major chicken and egg problem with the ovules and the surrounding structure in the parent plant right?
Yep. Probably not just cell division either, if you’re attempting this it’s probably also wise to use caged auxin for controlling growth, rather than leaving hormonal regulation up to the individual cells. And even with those tricks, I expect that this is a very very hard problem, and also that in the course of trying to solve it this way the researchers would discover reasons why the exact thing I spitballed wouldn’t work (but would likely also discover other angles of attack that were more promising based on their observations). You probably need to be able to target cell growth even more finely than the cellular level, since in order to prevent small differences spiralling into large differences a few divisions down the road, you need to be able to make fine corrections to cell size/shape/orientation based on observations.
That’s where I was going with the Ishida 1991 paper
You really don’t want to be dealing with the parent plant at all. It looks to me like it’s probably straightforward to handle this part (relative to the other parts of this project), but again that’s the sort of thing that you have to actually go out and interact with the external world to determine for sure.
FWIW I do expect that Yudkowsky expected that this problem would require the solver to solve bioprinting or something “sci-fi” like that rather than “just” absurd amounts of iterative work. But the problem as stated seems solvable through absurd amounts of iterative work, and it is worlds where solving problems involves lots of iterative work of the “reality has a surprising amount of detail” type that I expect humans remain relevant for a while.