This seems to me like a soft constraint problem at its core.
“How many little pins do you need to pin down the behavior of this system and force it into some subspace that’s reasonably close to the solution?”
I.e. for the LSD example: it’s not a “large scale” phenomenon. LSD’s effects are already prominent in tiny neuron populations “in vitro”. It’s not a “tiny almost imperceptible error that compounds to become catastrophic at scale”—it’s “already a big error that simply scales upwards well”.
Which means: you can have a “tiny neuron population in vitro” reference, and that alone will expose the “we made every single neuron in the brain sim act like it’s on LSD” failure mode. Many others like it too.
How many little pins? How many imposed constraints does it take to bleed off the bulk of major systematic failures? How many constraints does it take to walk away from “catastrophic instability” and enter the “mostly convergent” subspace, where innate perturbation resistance begins to work in our favor? How many of those constraints can be imposed with relatively straightforward means, long before you need to collect large bodies of new data and build specialized tooling like high bandwidth inspection?
The only real way to figure out is to go for it, and see what works and what fails. It’s an empirical problem, there’s no way around it. You can only test your assumptions if you make them first.
This seems to me like a soft constraint problem at its core.
“How many little pins do you need to pin down the behavior of this system and force it into some subspace that’s reasonably close to the solution?”
I.e. for the LSD example: it’s not a “large scale” phenomenon. LSD’s effects are already prominent in tiny neuron populations “in vitro”. It’s not a “tiny almost imperceptible error that compounds to become catastrophic at scale”—it’s “already a big error that simply scales upwards well”.
Which means: you can have a “tiny neuron population in vitro” reference, and that alone will expose the “we made every single neuron in the brain sim act like it’s on LSD” failure mode. Many others like it too.
How many little pins? How many imposed constraints does it take to bleed off the bulk of major systematic failures? How many constraints does it take to walk away from “catastrophic instability” and enter the “mostly convergent” subspace, where innate perturbation resistance begins to work in our favor? How many of those constraints can be imposed with relatively straightforward means, long before you need to collect large bodies of new data and build specialized tooling like high bandwidth inspection?
The only real way to figure out is to go for it, and see what works and what fails. It’s an empirical problem, there’s no way around it. You can only test your assumptions if you make them first.