Yes, that’s what ‘controlling’ usually looks like… Only a fool puts himself into situations where he must perform like a genius just to get the same results he could have by ‘avoiding’ those situations. IRL, an ounce of prevention is worth far more than a pound of cure.
To demonstrate how chaos theory imposes some limits on the skill of an arbitrary intelligence, I will also look at a game: pinball.
...Pinball is typical for a chaotic system. The sensitive dependence on initial conditions renders long term predictions impossible. If you cannot predict what will happen, you cannot plan a strategy that allows you to perform consistently well. There is a ceiling on your abilities because of the interactions with the chaotic system. In order to improve your performance you often try to avoid the chaos and focus on developing your skill in places where the world is more predictable.
This is wrong. Your pinball example shows no such thing. Note the complete non-sequitur here from your conclusion, about skill, to your starting premises, about unpredictability of an idealized system with no agents in it. You don’t show where the ceiling is to playing pinball even for humans, or any reason to think there is a ceiling at all. Nor do you explain why the result of ’12 bounces’ which you spend so much time analyzing is important; would 13 bounces have proven that there is no limit? Would 120 bounces have done so? Or 1,200 bounces? If the numerical result of your analysis could be replaced by a random number with no change to any of your conclusions, then it was irrelevant and a waste of 2000 words at best, and ‘proof by intimidation’ at worst.
And what you do seem to concede (buried at the end, hidden in an off-site footnote I doubt anyone here read) is:
Perhaps you think that this argument proves too much. Pinball is not completely a game of chance. How do some people get much better at pinball than others? If you watch a few games of professional pinball, the answer becomes clear. The strategy typically is to catch the ball with the flippers, then to carefully hit the balls so that it takes a particular ramp which scores a lot of points and then returns the ball to the flippers. Professional pinball players try to avoid the parts of the board where the motion is chaotic. This is a good strategy because, if you cannot predict the motion of the ball, you cannot guarantee that it will not fall directly between the flippers where you cannot save it. Instead, professional pinball players score points mostly from the non-chaotic regions where it is possible to predict the motion of the pinball.3 3. The result [of human pros] is a pretty boring game. However, some of these ramps release extra balls after you have used them a few times. My guess is that this is the game designer trying to reintroduce chaos to make the game more interesting again.
So, in reality, the score ceiling is almost arbitrarily high but “The result [of skilled pinball players maximizing scores] is a pretty boring game.” Well, you didn’t ask if ‘arbitrary intelligences’ would play boring games, now did you? You claimed to prove that they wouldn’t be able to play high-scoring games, and that you could use pinball games to “demonstrate how chaos theory imposes limits on the skill of an arbitrary intelligence”. But you didn’t “demonstrate” anything like that, so apparently, chaos theory doesn’t—it doesn’t even “impose some limits on the skill of” merely human intelligences...
Who knows how long sufficiently skilled players could keep racking up points by exercising skill to keep the ball in point-scoring trajectories in a particular pinball machine? Not you. And many games have been ‘broken’ in this respect, where the limit to points is simply an underlying (non-chaotic) variable, like the point where a numeric variable overflows & the software crashes, or the human player collapses due to lack of sleep.
This whole post is just an intimidation-by-physics bait-and-switch, where you give a rigorous deduction of an irrelevant claim where you concede the actual substantive claim is empirically false.
(And you can’t give any general relationship between difficulty of modeling environment for an arbitrary number of steps and the power of agents in environments, because there is none, and highly-effective intelligent agents may model arbitrarily small parts of the environment†.)
This entire essay strikes me as a good example of the abuse of impossibility proofs.
If we took it seriously, it illustrates in fact how little such analogies could ever tell us, because you have left out everything to do with ‘skill’ and ‘arbitrary intelligence’. This is like the Creationists like arguing that humans could not have evolved because it’d be like a tornado hitting a junkyard and spontaneously building a jetliner which would be astronomically unlikely; the exact combinatorics of that tornado (or bouncing balls) are irrelevant, because evolution (and intelligence) are not purely random—that’s the point! The whole point of an arbitrary intelligence is not stand idly by LARPing as Laplace’s demon. “The best way to predict the future is to invent it.” And the better you can potentially control outcomes, the less you need accurate long-range forecasts (a recent economics paper on modeling this exact question: Millner & Heyen 2021/op-ed.)
As Von Neumann was commenting about weather ‘forecasting’, and was a standard trope in cybernetics, prediction and control are duals, and ‘muh chaos theory’ doesn’t mean you can’t predict chaotic systems (or ‘unstable’ systems in general, chaos theory is just a narrow & overhyped subset, much beloved of midwits of the sort who like to criticize ‘Yudkowski’) - neural networks are good at predicting chaotic systems, and if you can’t, it just means that you need to control them upstream as well. This is why the field of “control of chaos” can exist. (Or, to put it more pragmatically: If you are worried about hurricanes causing damage, you “pour oil over troubled waters” in the right places, or you can try to predict them months out and disrupt the initial warm air formation; but it might be better to control them later by steering them, and if that turns out to be infeasible, control the damage by ensuring people aren’t there to begin with by quietly tweaking tax & insurance policy decades in advance to avoid lavishly subsidizing coastal vacation homes, etc; there are many places in the system to exert control rather than go ‘ah, you see, the weather was proven by Mandelbrot to be chaotic and unpredictable!’ & throw your hands up.)
That’s the beauty of unstable systems: what makes them hard to predict is also what makes them powerful to control. The sensitivity to infinitesimal detail is conserved: because they are so sensitive at the points in orbits where they transition between attractors, they must be extremely insensitive elsewhere. (If a butterfly flapping its wings can cause a hurricane, then that implies there must be another point where flapping wings could stop a hurricane...) So if you can observe where the crossings are in the phase-space, you can focus your control on avoiding going near the crossings. This nowhere requires infinitely precise measurements/predictions, even though it is true that you would if you wanted to stand by passively and try to predict it.
Nothing new here to the actual experts in chaos theory, but worth pointing out: discussions of bouncing balls diverging after n bounces are a good starting point, but they really should not be the discussion’s ending point, any more than a discussion of the worst-case complexity of NP problems should end there (which leads to bad arguments like ‘NP proves AI is impossible or can’t be dangerous’).
Another observation worth making is that speed (of control) and quality (of prediction) and power (of intervention) are substitutes: if you have very fast control, you can get away with low-quality prediction and small weak interventions, and vice-versa.
Would you argue that fusion tokamaks are impossible, even in principle, because high-temperature plasmas are extremely chaotic systems which would dissipate in milliseconds? No, because the plasmas are controlled by high-speed systems. Those systems aren’t very smart (which is why there’s research into applying smarter predictors like neural nets), but they are very fast.
There are loads of cool robot demos like the octobouncer which show that you can control ‘chaotic’ phenomena like bouncing balls with scarcely any action if you are precise and/or fast enough. (For example, the Ishikawa lab has for over a decade been publishing demos of ‘what if impossible-for-a-robot-task-X but with a really high-speed arm/hand?’ eg. why bother learning theory of mind to predict rock-paper-scissors when Ishikawa can use a high-speed camera+hand to cheat? And I saw a fun inverted pendulum demo on YT last week which makes that point as well.*) With superhuman reaction time, actuators can work on all sorts of (what would be) wildly nonlinear phenomenon despite crude controllers simply because they are so fast that at the time-scale that’s relevant, the phenomena never have a chance to diverge far from really simple linear dynamic approximations or require infeasibly heavy-handed interventions. And since they never escape into the nonlinearities (much less other attractors), they are both easy to predict and to control. And if you don’t need to deal with them, then value-equivalent model-based RL agents learning end-to-end will not bother to learn to predict them accurately at all, because that is predicting the wrong thing...
All of this comes up in lots of fields and has various niches in chaos theory and control theory and decision theory and RL etc, but I think you get the idea—whether it’s chaos, Godel, Turing, complexity, or Arrow, when it comes to impossibility proofs, there is always less there than meets the eye, and it is more likely you have made a mistake in your interpretation or proof than some esoteric a priori argument has all the fascinating real-world empirical consequences you over-interpret it to have.
* I think I might have also once saw this exact example of repeated-bouncing-balls done with robot control demonstrating how even with apparently-stationary plates (maybe using electromagnets instead of tilting?), a tiny bit of high-speed computer vision control could beat the chaos and make it bounce accurately many more times than the naive calculation says is possible, but I can’t immediately refind it.
And in this respect, pinball is a good example for AI risk: people try to analyze these problems as if life was a game like playing a carefully-constructed pinball machine for fun where you will play it as intended and obeying all the artificial rules and avoiding ‘boring’ games and delighting in proofs like ‘you can’t predict a pinball machine after n bounces because of quantum mechanics, for which I have written a beautiful tutorial on LW proving why’; while in real life, if you actually had to maximize the scores on a pinball machine for some bizarre reason, you’d unsee the rules and begin making disjunctive plans to hack the pinball machine entirely. It might be a fun brainstorming exercise for a group to try to come up with 100 hacks, try to taxonomize them, and analogize each category to AI failure modes.
Just off the top of my head: one could pay a pro to play for you instead, use magnetic balls to maneuver from the outside or magnets to tamper with the tilt sensor, snake objects into it through the coin slot or any other tiny hole to block off paths, use a credit card to open up the back and enable ‘operator mode’ to play without coins or with increased bonuses, hotwire it, insert new chips to reprogram it, glue fake high-score numbers over the real display numbers, replace it with a fake pinball machine entirely, photoshop a photo of high scores or yourself into a photo of someone else’s high scores or copy the same high score repeatedly into ‘different’ photos, etc. That’s 9 hacks there, none of which can be disproven by ‘chaos theory’ (or ‘Godel’, or ‘Turing’, or ‘P!=NP’, or...).
See also: temporal scaling laws, where Hilton et al 2023 offers a first stab at properly conceptualizing how we should think of the relationship of prediction & ‘difficulty’ of environments for RL agents, finding two distinguishable concepts with different scaling:
One key takeaway is that the scaling multiplier is not simply proportional to the horizon length, as one might have naively expected. Instead, the number of samples required is the sum of two components, one that is inherent to the task and independent of the horizon length, and one that is proportional to the horizon length.
(One implication here being that if you think about modeling the total ‘difficulty’ of pinball like this, you might find that the ‘inherent’ difficulty is high, but then the ‘horizon length’ difficulty is necessarily low: it is very difficult to master mechanical control of pinballs and understand the environment at all with all its fancy rules & bonuses & special-cases, but once you do, you have high control of the game and then it doesn’t matter much what would (not) happen 100 steps down the line due to chaos (or anything else). Your ‘regret’ from not being omniscient is technically non-zero & still contributing to your regret, but too small to care about because it’s so unlikely to matter. And this may well describe most real-world tasks.)
† model-based agents want to learn to be value-equivalent. A trivial counterexample to any kind of universal claim about chaos etc: imagine a MDP in which action #1 leads to some totally chaotic environment unpredictable even a step in advance, and where action #2 earns $MAX_REWARD; obviously, RL agents will learn the optimal policy of simply always executing action B and need learn nothing whatsoever about the chaotic environment behind action #1.
Presumably that’s because pinball is a game with imposed rules. Physically the control solutions could be installing more flippers, or rearranging the machine components. Or if you want to forbid altering the machine, maybe installing an electromagnet that magnetizes the ball and then drags it around directly, or even just stays stationary at the bottom to keep it from falling between the flippers.
You control an unstable process with short-loop feedback. In the pinball analogy, this would be like using your hand to guide the ball between the disks in whatever path you like.
Seems like the answer with pinball is to avoid the unstable processes, not control them.
Yes, that’s what ‘controlling’ usually looks like… Only a fool puts himself into situations where he must perform like a genius just to get the same results he could have by ‘avoiding’ those situations. IRL, an ounce of prevention is worth far more than a pound of cure.
This is wrong. Your pinball example shows no such thing. Note the complete non-sequitur here from your conclusion, about skill, to your starting premises, about unpredictability of an idealized system with no agents in it. You don’t show where the ceiling is to playing pinball even for humans, or any reason to think there is a ceiling at all. Nor do you explain why the result of ’12 bounces’ which you spend so much time analyzing is important; would 13 bounces have proven that there is no limit? Would 120 bounces have done so? Or 1,200 bounces? If the numerical result of your analysis could be replaced by a random number with no change to any of your conclusions, then it was irrelevant and a waste of 2000 words at best, and ‘proof by intimidation’ at worst.
And what you do seem to concede (buried at the end, hidden in an off-site footnote I doubt anyone here read) is:
So, in reality, the score ceiling is almost arbitrarily high but “The result [of skilled pinball players maximizing scores] is a pretty boring game.” Well, you didn’t ask if ‘arbitrary intelligences’ would play boring games, now did you? You claimed to prove that they wouldn’t be able to play high-scoring games, and that you could use pinball games to “demonstrate how chaos theory imposes limits on the skill of an arbitrary intelligence”. But you didn’t “demonstrate” anything like that, so apparently, chaos theory doesn’t—it doesn’t even “impose some limits on the skill of” merely human intelligences...
Who knows how long sufficiently skilled players could keep racking up points by exercising skill to keep the ball in point-scoring trajectories in a particular pinball machine? Not you. And many games have been ‘broken’ in this respect, where the limit to points is simply an underlying (non-chaotic) variable, like the point where a numeric variable overflows & the software crashes, or the human player collapses due to lack of sleep.
This whole post is just an intimidation-by-physics bait-and-switch, where you give a rigorous deduction of an irrelevant claim where you concede the actual substantive claim is empirically false.
(And you can’t give any general relationship between difficulty of modeling environment for an arbitrary number of steps and the power of agents in environments, because there is none, and highly-effective intelligent agents may model arbitrarily small parts of the environment†.)
This entire essay strikes me as a good example of the abuse of impossibility proofs. If we took it seriously, it illustrates in fact how little such analogies could ever tell us, because you have left out everything to do with ‘skill’ and ‘arbitrary intelligence’. This is like the Creationists like arguing that humans could not have evolved because it’d be like a tornado hitting a junkyard and spontaneously building a jetliner which would be astronomically unlikely; the exact combinatorics of that tornado (or bouncing balls) are irrelevant, because evolution (and intelligence) are not purely random—that’s the point! The whole point of an arbitrary intelligence is not stand idly by LARPing as Laplace’s demon. “The best way to predict the future is to invent it.” And the better you can potentially control outcomes, the less you need accurate long-range forecasts (a recent economics paper on modeling this exact question: Millner & Heyen 2021/op-ed.)
As Von Neumann was commenting about weather ‘forecasting’, and was a standard trope in cybernetics, prediction and control are duals, and ‘muh chaos theory’ doesn’t mean you can’t predict chaotic systems (or ‘unstable’ systems in general, chaos theory is just a narrow & overhyped subset, much beloved of midwits of the sort who like to criticize ‘Yudkowski’) - neural networks are good at predicting chaotic systems, and if you can’t, it just means that you need to control them upstream as well. This is why the field of “control of chaos” can exist. (Or, to put it more pragmatically: If you are worried about hurricanes causing damage, you “pour oil over troubled waters” in the right places, or you can try to predict them months out and disrupt the initial warm air formation; but it might be better to control them later by steering them, and if that turns out to be infeasible, control the damage by ensuring people aren’t there to begin with by quietly tweaking tax & insurance policy decades in advance to avoid lavishly subsidizing coastal vacation homes, etc; there are many places in the system to exert control rather than go ‘ah, you see, the weather was proven by Mandelbrot to be chaotic and unpredictable!’ & throw your hands up.)
That’s the beauty of unstable systems: what makes them hard to predict is also what makes them powerful to control. The sensitivity to infinitesimal detail is conserved: because they are so sensitive at the points in orbits where they transition between attractors, they must be extremely insensitive elsewhere. (If a butterfly flapping its wings can cause a hurricane, then that implies there must be another point where flapping wings could stop a hurricane...) So if you can observe where the crossings are in the phase-space, you can focus your control on avoiding going near the crossings. This nowhere requires infinitely precise measurements/predictions, even though it is true that you would if you wanted to stand by passively and try to predict it.
Nothing new here to the actual experts in chaos theory, but worth pointing out: discussions of bouncing balls diverging after n bounces are a good starting point, but they really should not be the discussion’s ending point, any more than a discussion of the worst-case complexity of NP problems should end there (which leads to bad arguments like ‘NP proves AI is impossible or can’t be dangerous’).
Another observation worth making is that speed (of control) and quality (of prediction) and power (of intervention) are substitutes: if you have very fast control, you can get away with low-quality prediction and small weak interventions, and vice-versa.
Would you argue that fusion tokamaks are impossible, even in principle, because high-temperature plasmas are extremely chaotic systems which would dissipate in milliseconds? No, because the plasmas are controlled by high-speed systems. Those systems aren’t very smart (which is why there’s research into applying smarter predictors like neural nets), but they are very fast.
There are loads of cool robot demos like the octobouncer which show that you can control ‘chaotic’ phenomena like bouncing balls with scarcely any action if you are precise and/or fast enough. (For example, the Ishikawa lab has for over a decade been publishing demos of ‘what if impossible-for-a-robot-task-X but with a really high-speed arm/hand?’ eg. why bother learning theory of mind to predict rock-paper-scissors when Ishikawa can use a high-speed camera+hand to cheat? And I saw a fun inverted pendulum demo on YT last week which makes that point as well.*) With superhuman reaction time, actuators can work on all sorts of (what would be) wildly nonlinear phenomenon despite crude controllers simply because they are so fast that at the time-scale that’s relevant, the phenomena never have a chance to diverge far from really simple linear dynamic approximations or require infeasibly heavy-handed interventions. And since they never escape into the nonlinearities (much less other attractors), they are both easy to predict and to control. And if you don’t need to deal with them, then value-equivalent model-based RL agents learning end-to-end will not bother to learn to predict them accurately at all, because that is predicting the wrong thing...
All of this comes up in lots of fields and has various niches in chaos theory and control theory and decision theory and RL etc, but I think you get the idea—whether it’s chaos, Godel, Turing, complexity, or Arrow, when it comes to impossibility proofs, there is always less there than meets the eye, and it is more likely you have made a mistake in your interpretation or proof than some esoteric a priori argument has all the fascinating real-world empirical consequences you over-interpret it to have.
* I think I might have also once saw this exact example of repeated-bouncing-balls done with robot control demonstrating how even with apparently-stationary plates (maybe using electromagnets instead of tilting?), a tiny bit of high-speed computer vision control could beat the chaos and make it bounce accurately many more times than the naive calculation says is possible, but I can’t immediately refind it.
And in this respect, pinball is a good example for AI risk: people try to analyze these problems as if life was a game like playing a carefully-constructed pinball machine for fun where you will play it as intended and obeying all the artificial rules and avoiding ‘boring’ games and delighting in proofs like ‘you can’t predict a pinball machine after n bounces because of quantum mechanics, for which I have written a beautiful tutorial on LW proving why’; while in real life, if you actually had to maximize the scores on a pinball machine for some bizarre reason, you’d unsee the rules and begin making disjunctive plans to hack the pinball machine entirely. It might be a fun brainstorming exercise for a group to try to come up with 100 hacks, try to taxonomize them, and analogize each category to AI failure modes.
Just off the top of my head: one could pay a pro to play for you instead, use magnetic balls to maneuver from the outside or magnets to tamper with the tilt sensor, snake objects into it through the coin slot or any other tiny hole to block off paths, use a credit card to open up the back and enable ‘operator mode’ to play without coins or with increased bonuses, hotwire it, insert new chips to reprogram it, glue fake high-score numbers over the real display numbers, replace it with a fake pinball machine entirely, photoshop a photo of high scores or yourself into a photo of someone else’s high scores or copy the same high score repeatedly into ‘different’ photos, etc. That’s 9 hacks there, none of which can be disproven by ‘chaos theory’ (or ‘Godel’, or ‘Turing’, or ‘P!=NP’, or...).
See also: temporal scaling laws, where Hilton et al 2023 offers a first stab at properly conceptualizing how we should think of the relationship of prediction & ‘difficulty’ of environments for RL agents, finding two distinguishable concepts with different scaling:
(One implication here being that if you think about modeling the total ‘difficulty’ of pinball like this, you might find that the ‘inherent’ difficulty is high, but then the ‘horizon length’ difficulty is necessarily low: it is very difficult to master mechanical control of pinballs and understand the environment at all with all its fancy rules & bonuses & special-cases, but once you do, you have high control of the game and then it doesn’t matter much what would (not) happen 100 steps down the line due to chaos (or anything else). Your ‘regret’ from not being omniscient is technically non-zero & still contributing to your regret, but too small to care about because it’s so unlikely to matter. And this may well describe most real-world tasks.)
† model-based agents want to learn to be value-equivalent. A trivial counterexample to any kind of universal claim about chaos etc: imagine a MDP in which action #1 leads to some totally chaotic environment unpredictable even a step in advance, and where action #2 earns $MAX_REWARD; obviously, RL agents will learn the optimal policy of simply always executing action B and need learn nothing whatsoever about the chaotic environment behind action #1.
Presumably that’s because pinball is a game with imposed rules. Physically the control solutions could be installing more flippers, or rearranging the machine components. Or if you want to forbid altering the machine, maybe installing an electromagnet that magnetizes the ball and then drags it around directly, or even just stays stationary at the bottom to keep it from falling between the flippers.
You control an unstable process with short-loop feedback. In the pinball analogy, this would be like using your hand to guide the ball between the disks in whatever path you like.