Your point c definitely rings true to me. An answer often seems simple in hindsight, but that an answer is simple doesn’t mean it’s simple to find. There are often many simple answers and the vast majority of them useless.
tangerine
Karma: 102
You are assuming a superintelligence that knows how to perform all these deductions. Why would this be a valid assumption? You are reasoning from your own point of view, i.e., the point of view of someone who has already seen much, much more of the world than a few frames, and more importantly someone who already knows what the thing is that is supposed to be deduced, which allows you to artificially reduce the hypothesis space. On what basis would this superintelligence be able to do this?
An individual agent can’t beat humans at cultural evolution, but multiple agents can. However, the way they do it will almost certainly be very conspicuous, especially if it’s novel (outside the training distribution), because the way you get sufficient data about a new task is by trial and error. If these agents tried to take over the world quickly it would be like the January 6th Insurrection; very visible, misguided and ineffective. They could do it over a long time span by assuming parts of the economy and gaining leverage by lobbying, but that is a slow process.
The Bengio quote is valid, but it doesn’t apply to short timespans. How would a group of agents be able to learn to copy itself over a very large array of hardware, and learn to coordinate, without drawing massive attention to itself? None of this could be done without precedent. We have systems currently that do distributed learning, but these are very specific and narrow implementations that do not scale to taking over or destroying large parts of the world; that is absolutely unprecedented.
For any synthetic data to be useful however, requires that data to be grounded. Generating synthetic data is easy, fast and cheap, but if you want to ground it in empirical facts, that makes it much slower and expensive. For example, behind every paper published is an amount of work much, much greater than writing down the words.
Okay, I guess this comes down to the interpretation of what “foom” means? I don’t think a world that looks like the current one can be taken over inconspicuously by AI in seconds, and not weeks either, and not even less than a year. If society has progressed to a point where we feel comfortable giving much more power to artificial agents, then that shortens the timeline.
The reason I think timelines are long is that I think it is inherently hard to do novel things, much harder than typically thought. I mean, what new things do you and I really do? Virtually nothing. What I tried to state in this essay is that knowledge is an inherent part of what we typically mean by intelligence, and for new tasks, new intelligence and knowledge is needed. The way this knowledge is gained is through cultural evolution; memes constitute knowledge and intelligence and these evolve similarly to genes; the vast majority of good genes you have are from your ancestors and most of your new genes or recombinations thereof are vastly likely to not improve your genetic makeup. It works the same way with memes; virtually everything you and I can do that we consider uniquely human are things we’ve copied from somewhere else, including “simple” things like counting or percentages. And, virtually none of the new things you and I do are improvements.
AI is not exempted from the process described above. Its intelligence is just as dependent on knowledge gained through trial and error and cultural evolution. This process is slow, and the faster and greater the effect to be achieved, the more knowledge and time is needed to actually do it in one shot.
evidence against foom-in-a-box is just an improvement to the map of how to foom.
Could you elaborate on this? I equate foom with the hard take-off scenario, for which I think I’ve stated why I think this is virtually impossible, in contrast to the slow take-off, which in spite of being slow is still very dangerous, as I described.
I think my view roughly aligns with those of Robin Hanson and Paul Christiano, but I think I’ve provided a more precise, gears-level description that has been lacking and why the onus is really on those who think the hard take-off is possible at all.
Intelligence is indeed not magic. None of the behaviors that you display that are more intelligent than a chimpanzee’s behaviors are things you have invented. I’m willing to bet that virtually no behavior that you have personally come up with is an improvement. (That’s not an insult, it’s simply par for the course for humans.) In other words, a human is not smarter than a chimpanzee.
The reason humans are able to display more intelligent behavior is because we’ve evolved to sustain cultural evolution, i.e., the mutation and selection of behaviors from one generation to the next. All of the smart things you do are a result of that slow accumulation of behaviors, such as language, counting, etc., that you have been able to simply imitate. So the author’s point stands that you need new information from experiments in order to do something new, including new kinds of persuasion.
The laws of physics are much simpler than the detailed structure of a given table
It is not practical to simulate everything down to the level of the laws of physics. In practice, you usually have to come up with much coarser models that can actually be computed within a reasonable time and most of the experimentation is needed to construct those models in the first place so that they align sufficiently with reality, and even then only in certain circumstances.
You could maybe use quantum mechanics to calculate the planetary orbits out for thousands of years, but it’s much simpler to use Newtonian mechanics for that, and that’s because the planetary motions happen to be easily modelable in that way, which however isn’t true for building rocket engines, or predicting the stock market or global politics.
The difference between humans and chimpanzees is purely one of imitation. Humans have evolved to sustain cultural evolution, by imitating existing culture and expanding upon it during a lifetime. Chimpanzees don’t imitate reliably enough to sustain this process and so their individual knowledge gains are mostly lost at death, but the individual intelligence of a human and a chimpanzee is virtually the same. A feral human child, that is, a human without human culture, does not behave more intelligently than a chimpanzee.
The slow accumulation of culture is what separated humans from chimpanzees. For AI to be to humans what humans are to chimps is not really possible because you either accumulate culture (knowledge) or you don’t. The only remaining distinction is one of speed. AI could accumulate a culture of its own, faster than humans. But how fast?
Limits to Learning: Rethinking AGI’s Path to Dominance
Thank you for your response, I will try to address your comments.
Few people end up “contributing much to science or overall culture”. It’s like being surprised that most people who do sport regularly do not end up winning the Olympic Games.
Well, people often extrapolate that if a child prodigy, whether in sports or an intellectual pursuit, does much better than other children of the same age, that this difference will persist into adulthood. What I’m saying is that the reason it usually doesn’t, is that once you’ve absorbed existing techniques, you reach a plateau that’s extremely hard to break out of and even if you do, it’s only by a small amount and largely based on luck. And it doesn’t matter much if one reaches that plateau at age 15 (like, say, a child prodigy) or 25 (like, say, a “normal” person).
You don’t specify what exactly is the “myth” and what exactly is your alternative explanation.
The myth of general intelligence is that it is somehow different from “regular” intelligence. A human brain is not more general than any other primate brain. It’s actually the training methods that are different. There is nothing inherent in the model that is the human brain which makes it inherently more capable, à la Chomsky’s universal grammar. The only effective difference is that the human mind has a strong tendency towards imitation, which does not in itself make it more intelligent, but only if there are intelligent behaviors available to imitate, so what is in effect different about what’s called a “general” intelligence is not the model or agent itself, but the training method. There is nothing in principle that stops a chimpanzee from being able to read and write English, for example. It’s just that we haven’t figured out the methods to configure their brains into that state, because they don’t have a strong tendency to imitate, which human children do have, which makes training them much easier.
If your point is that Einstein could not have discovered general relativity if he was born 1000 years earlier, I agree. If your point is that any other person with university education living in the same era could have discovered the same, I disagree.
We agree on the first point. As for the second point, in hindsight we of course know that Einstein was able to discover what he did discover. However, before his discovery, it was not known what kind of person would be required. We did not even know exactly what was out there to discover. We are in that situation today with respect to discoveries we haven’t discovered yet. We don’t know what kind of person, with what kind of brain, in what configuration, will make what discoveries, so there is an element of chance. If in Einstein’s time the chips had fallen slightly differently, I don’t see why some other person couldn’t have made the same or very similar discoveries. It could have happened five years earlier or five years later, but it seems extremely unlikely to me that if Einstein had died as a baby that we would still be stuck with Newtonian mechanics today.
No, you do not need any specific knowledge in order to be intelligent.
Well, this comes down to what is a useful definition of intelligence. You indeed don’t need any specific knowledge if you define intelligence as something like IQ, but even a feral child with an IQ of 200 won’t outmatch a chimpanzee in any meaningful way; it won’t invent Hindu-Arabic numerals, language or even a hand axe. Likewise, ChatGPT is a useless pile of numbers before it sees the training data, and afterwards its behavior depends on what was in that training data. So in practice I would argue that to be intelligent in a specific domain you do need specific knowledge, whether that’s factual knowledge, or more implied knowledge that is absorbed by osmosis or practice.
But it is also true that some people are way better at copying and repurposing these things than others.
Some people are better at that, but I wouldn’t say way better. John von Neumann was perhaps at the apex of this, but that didn’t make him much more powerful in practice. He didn’t discover the Higgs boson. He didn’t build a gigahertz microprocessor. He didn’t cure his own cancer. He didn’t even put wheels under his suitcase. It’s impressive what he did do, but still pretty incremental. Why would an AI be much better at this?
Nope. Depends on what kind of task you have in mind, and what kind of learning is available.
What kind of practical task wouldn’t require trial and error? There are some tasks, such as predicting the motions of the planets, for which there turned out to be a method which works quite generally, but even then when you drop certain assumptions, the method becomes intensive or impossible to calculate.
Also, intelligence is related to how fast one learns.
Yes, a bigger model or a higher IQ can help you get more out of a given amount of data, but the scaling laws show diminishing returns. Pretty quickly, having more data starts to outweigh trying to squeeze more out of what you’ve already got.
I really liked this post; it puts modern computing into an interesting perspective.
I think the perspective of physics is also interesting to add, because I think it really shows how fundamental the idea of computation is. According to modern physics, everything is computation; every physical interaction, whether it’s between two elementary particles, or between two people shaking hands, is computation. As you mentioned, it was mathematically proven by Alan Turing that there is a type of machine, now called a Turing machine, which can compute anything that can in principle be computed, which therefore supposedly includes all of physics. All our laptops and other modern computers are Turing machines, so our laptops, in principle, are capable of computing anything physical, including a human brain. The question is, how practical is it to do a given computation?
Computing the movements that a grain of sand makes over the course of one second would take a laptop aeons to compute, let alone computing in real time what goes on in a human brain. However, in practice you don’t usually need to do an exhaustive computation to get the result you want. For example, you can program your laptop to calculate the result of 1+1, but you can also simulate your physical laptop while it is computing 1+1 to get the same result (hopefully 2). However, the latter would waste an extreme amount of computation you don’t need to do. Similarly, the relevant outputs of the human brain might be calculated with much less computation than it takes to simulate an entire physical human brain. In fact, artificial neural networks appear to do just that, meaning that the physical brain appears to “merely” physically implement the virtual equivalent of our artificial neural networks.
Some people have countered by saying that many important processes in the human brain have been abstracted away in artificial neural networks, but it is not clear that these processes are part of the fundamental computations that lead to the results we care about, instead of simply supporting the physical implementation of these fundamental computations. Some other people, such as Roger Penrose, have even claimed that human brains do processing that is not computable and therefore cannot be computed even using a Turing machine, in turn implying that such uncomputable processes are possible in nature, which is counter to modern physics’ understanding of nature.
I can’t really be persuaded by this kind of argument, because suffering (and pain) is not coherent enough for this kind of calculus. Why couldn’t plants suffer? What about one-celled organisms? It’s arbitrary.
One could write a microcontroller program that makes some signal if it’s circuit is damaged.
People are not fundamentally different from such a microcontroller. It’s signals all the way down.
One can try to do suffering calculus like in the original post, based on certain axioms, but these are unfalsifiable. Realistically, suffering calculus is based on a political and sociological consensus, e.g., the Overton window. Humans have political representation and Western civilization has human equality as a kind of axiom, at least nominally, so in the Western world there is a lot of incentive to reduce all human suffering (unlike in China, for example, where Uyghurs are marginalized). Animals have far less representation, so there is less incentive to reduce suffering. In my country, there is a Party for Animals with a few seats in Parliament, voted in by people (because axiomatically animals don’t have the right to vote) who do it to perhaps virtue signal, or because the human brain is wired to empathize more with similar beings, or for unfalsifiable philosophical considerations, or for some other reason which for other voting blocs is not in their Overton window. For plants the situation is far more dire. It is in principle possible that for large swaths of the public the Overton window shifts such that they will refuse to support the “slaughter” of plants. But what’s in the Overton window is separate from the facts and there are no facts of the matter about suffering.
Whether or not one’s beliefs are correct is similar to whether one’s genes are correct, because, fundamentally, one’s beliefs are determined by an evolutionary algorithm; the ideas that are best at spreading and keeping their hosts alive survive and are in that sense “correct”. Some of these ideas involve heuristics and elements of what we call critical thinking or rationality, but they are limited.
We talk about rationality a lot on here, obviously, but I dare say it doesn’t help to pay the rent in almost any case for anyone here at LessWrong, and in some cases is actively harmful, so I’m not sure that it’s helpful on average.
The Hard Problem doesn’t exist.
Do you believe that all your beliefs are represented only in the structure of your brain? Then changing the structure of your brain changes your beliefs; this way you could theoretically be made to believe anything, including, of course, things that are false. Some false beliefs are useful, such as some optical illusions and illusions in general, such as the belief that “you” are “experiencing” things. (I once interviewed a man who had had a stroke and reported feeling like “he wasn’t there” anymore and he would look at his own hand and say, “it’s like it’s not mine”. This caused difficulties with locomotion and knowing when “he” had to go to the bathroom, because it was hard for him to realize it was actually “his” bladder being full and “him” having to make a decision to relieve it.) It’s a useful, evolved structure in your brain that makes you believe that. But it’s technically still false.
Similarly, you could hallucinate that a dragon is standing in front of you; some people actually have such hallucinations. The rational thing to do at such a moment is to disbelieve your direct experience, based on all the other things you know about the world; you know that the evidence against the existence of dragons is overwhelming and you know hallucinations happen. However, disbelieving that what you’re experiencing is real doesn’t make you suddenly not experience it, that’s why hallucinations can be so debilitating.
Ever had déjà vu? When I have déjà vu, I get an overwhelming sense of having experienced something before and my mind starts racing, trying to explain it. I usually recognize rationally that this is probably a déjà vu, but that explanation feels very unsatisfying in the moment because the feeling of recognition is so convincing. It’s only when this sensation subsides about ten seconds later that I can put the matter to rest, assured that it was just a glitch in my brain. But what if it never subsided? What if you had a déjà vu that lasted the rest of your life? It would be very hard to ignore the constant sensation of recognition. This is exactly the kind of thing that “believing you are experiencing” (and many associated beliefs) is. In both cases it’s false, but in the latter it’s there because it’s useful.
The point is that sometimes the structure of our brains can reach a state in which certain things seem to be true, even though rationally we should disbelieve it based on all the other things we know. The evidence for the existence of the Hard Problem is extremely thin. In fact, it’s unfalsifiable—not even wrong. Everything we know about science, truth and knowledge points to the Hard Problem not existing. What actually needs to be explained is not why we experience things, but why we think that we experience things and we have perfectly good (physical) explanations for that.
The burden of proof is on those who assert that the Hard Problem is real. You can say what consciousness is not, but can you say what it is? As it stands, no explanation of the Hard Problem is possible, because the Hard Problem has no criteria for what would comprise a satisfying explanation; no way to distinguish a correct explanation from an incorrect one. All real science has such criteria, yet even David Chalmers has none. Until those criteria are established, the existence of the Hard Problem will forever remain unfalsifiable, unscientific and belief in it irrational.
Unfortunately, proper criteria for explanations always involve (physical) observations and their predictions. Therefore any attempt to establish criteria for explanations of the Hard Problem is met with the criticism that, because it refers to physical aspects of consciousness, it ignores the Hard Problem. Evidently, proponents of the Hard Problem have backed themselves into a contradictory corner; the Hard Problem is unfalsifiable and any attempt to make it falsifiable makes it not the Hard Problem.
If the Hard Problem is “above” science (i.e., not science), as it seems to be, then it is above inquiry and if it’s above inquiry, why inquire?
The naked truth is that belief in the existence of the Hard Problem fetishizes mystery; it abhors actual explanation and therefore scrambles to keep its suppositions immune to it.
Belief in the Hard Problem, being unscientific and therefore not real, begs the question why such beliefs can nonetheless take root in the face of overwhelming contrary evidence, which is what my earlier post attempted to explain.
I’m experiencing just like you, but the Hard Problem doesn’t jive at all with the rest of my beliefs (and I have seen many attempts to reconcile them, all unsuccessful). Therefore I choose to accept the benefits of the sensation of experience and accept the Easy Problem of consciousness as the overwhelmingly likely Only Problem of consciousness.
Thank you for this reply, I think this helps to pin down where our disagreement comes from.
Technically I don’t disagree with your assumptions, because I think it’s equally valid to say they’re true as that they’re false, which is exactly the issue I have with them. There doesn’t seem to be a fact of the matter about them (i.e., there’s no way to experimentally distinguish a world in which any of these assumptions holds from one in which it does not), so if the existence of the Hard Problem is derived from them, then that doesn’t alleviate the issue of its unfalsifiability.
The cause of this issue is that (from my point of view) many of the words you’re using don’t have clear definitions in the domain that you’re trying to use them in. I don’t mean to be a pedant, but if we’re really trying to use language for extraordinary investigations like these, then I think precision is warranted. For now, let me just focus on the thought experiment you posed. The way I see it, it’s equivalent to the Ship of Theseus. I think what we’re ultimately trying to grapple with is how best to model reality and it seems to me that we actually already have a perfectly good model to solve the Ship of Theseus and your thought experiment, namely particle physics. If you look at the Ship of Theseus or a person’s brain or body (or a piece of text they wrote), these are collections of particles that create a causal chain to somebody saying “Hey, it’s the Ship of Theseus!” or “Hey, gilch wrote a reply!” Over time, some of those particles may get swapped for others and cause us to still use the same name or maybe not. There’s no mystery or contradiction there, it’s a bunch of particles doing their thing and names are patterns in those particles, for example in the air when we speak them or in silicon when we’re writing it on a phone.
Do we think about the world in terms of fundamental particles? No, it’s wildly impractical, so we’ve been forced to resort, through our evolution and the evolution of language, to much simpler models/heuristics. Daniel Dennett has this idea of “folk psychology”, which talks specifically about how we model other people’s behaviors, by talking about things like “belief”, “desire”, “fear” and “hope”. This model works most of the time, but it breaks down when you try to use it to model, for example, the behavior of a schizophrenic person, or the behavior of a dead person. You can extend this idea to a kind of “folk reality”, where we model the world in terms of “people”, “alive”, “dead”, “conscious”, “justice”, “love” and pretty much all other words, which can similarly break down when trying to use them to communicate about things that they’re not normally used to communicate about.
If you like, I could go into detail how this applies to each of your assumptions, but I’ll do so just for your last assumption for now. Consciousness in normal usage is a word that evolved to mean something like “able to respond appropriately to its surroundings”, so a person who is sleeping or knocked out is basically unconscious; that’s enough for practical, daily usage. Similarly, we say humans normally are conscious, other primates and mammals maybe a little less, insects maybe and plants not really, i.e., the fewer traits it has that we recognize in humans the less conscious it is; this is already a bit less practical and more academic, but it affects how we behave. (For example, vegans claim eating animals is bad, while eating plants is okay, even though they’re absurdly glossing over whether plants can feel pain, which is not clear at all.) Over time, the evolution of language (which is a product of both chance and deliberate human decisions) adapted the meaning of words like consciousness to remain a useful part of folk reality. Our intuitions about the meanings of words and in turn about reality depend on how we see these words being used as we grow up, even if they don’t model reality correctly; we always end up with somewhat mistaken intuitions, because folk reality does not model reality exactly. And now, quite suddenly, we’ve ended up in a situation where there are machines that can behave in a way that we’re only used to recognizing in humans and so there’s a lot of confusion over whether they are conscious. Again, from a particle physics perspective it’s clear what’s going on; it’s particles doing their thing like they always have. Some particles are arranged in a structure we haven’t seen before, so what? However, our folk reality model breaks down because it’s imprecise and not adapted to this new situation. That’s also not an issue in itself; language and intuitions just have to adapt. Maybe we’ll come to a consensus that they are just as conscious as we are, or maybe we’ll see them as inferior and therefore treat them with greater indifference, even though how we describe them doesn’t actually change their nature, just our perception and treatment of them.
The real problems begin when people assume that their intuitions are true and fail to recognize that our intuitions and language are models of reality (largely inherited from cultures before us who had much less experience with the world) and that they frequently don’t generalize well. So when I encounter something like the Hard Problem, I throw my intuitions about how “I really feel like I’m experiencing things, so I can’t just be an automaton” out the window, because going down that road just leads to a bunch of useless contradictions and I conclude that whatever is going on must be made possible by particles doing their thing and nothing else, at least until I encounter a better model.
As for whether I would choose to undergo the procedure, I probably would. I don’t see any meaningful difference between my brain being replaced by new synthetic or biological material. In fact, according to my intuition (perhaps mistaken), my future counterpart with a 100% biological brain would be just as much a different person from me as my alternative future counterpart with a 100% synthetic brain.
Hi, please see my reply to gilch above.
To add to that reply, an explanation only ever serves one function, namely to aid in prediction; every moment of our life, we try to achieve outcomes by predicting which action will lead to which outcome. An explanation to the Hard Problem doesn’t do that. Any state of consciousness that I try to achieve I do so with concepts related to the Easy Problem. I do have experiences (I don’t know what the word “phenomenal” would add to that), such as pain, but to the extent that I can predict and control these, I do so purely with solutions to the Easy Problem. And in my book, concepts that exist only in explanations that don’t aid in prediction are by definition not real. But the Hard Problem is even worse than that; it’s set up so that we can’t tell the difference between a correct and incorrect explanation in the first place, which means literally anything could be an explanation, which is equivalent to no explanation at all. Sure, you can choose to believe that something like panpsychism is real or that it’s not real, but because neither belief adds any predictive power, you’re better off just cutting it out, as per Occam’s Razor.
Mr. Topaz is right (even if for the wrong reasons). If he is optimizing for money, being the first to market may indeed get him a higher probability of becoming rich. Wanting to be rich is not wrong. Sure, the service will be insecure, but if he sells before any significant issues pop up he may sell for billions. “Après moi, le déluge.”
Unfortunately, the actions of Mr. Topaz are also the very optimization process that leads to misaligned AGI, already happening at this very moment. And indeed, trying to fix Mr. Topaz is ordinary paranoia. Security mindset would be closer to getting rid of The Market that so perversely incentivized him; I don’t like the idea, but if it’s between that and a paperclip maximizer, I know what I would choose.