Seems similar enough to “Every part of your brain assumes that all the other surrounding parts work a certain way. The present brain is the Environment of Evolutionary Adaptedness for every individual piece of the present brain.
Start modifying the pieces in ways that seem like “good ideas”—making the frontal cortex larger, for example—and you start operating outside the ancestral box of parameter ranges. And then everything goes to hell.
So you’ll forgive me if I am somewhat annoyed with people who run around saying, “I’d like to be a hundred times as smart!” as if it were as simple as scaling up a hundred times instead of requiring a whole new cognitive architecture.”
Well, OTOH, he also complains that messing around by trial and error is likely to cause unpredictable side effects, like nasty insanity, some of which may be too subtle to notice at first, or just tolerated.
Seems similar enough to “Every part of your brain assumes that all the other surrounding parts work a certain way. The present brain is the Environment of Evolutionary Adaptedness for every individual piece of the present brain.
Start modifying the pieces in ways that seem like “good ideas”—making the frontal cortex larger, for example—and you start operating outside the ancestral box of parameter ranges. And then everything goes to hell.
So you’ll forgive me if I am somewhat annoyed with people who run around saying, “I’d like to be a hundred times as smart!” as if it were as simple as scaling up a hundred times instead of requiring a whole new cognitive architecture.”
Eliezer Yudkowsky, Growing Up is Hard
Well, OTOH, he also complains that messing around by trial and error is likely to cause unpredictable side effects, like nasty insanity, some of which may be too subtle to notice at first, or just tolerated.