humans are general because of the data, not the algorithm
Interesting statement. Could you expand a bit on what you mean by this?
tangerine
Karma: 232
You cannot in general pay a legislator $400 to kill a person who pays no taxes and doesn’t vote.
Indeed not directly, but when the inferential distance increases it quickly becomes more palatable. For example, most people would rather buy a $5 T-shirt that was made by a child for starvation wages on the other side of the world, instead of a $100 T-shirt made locally by someone who can afford to buy a house with their salary. And many of those same T-shirt buyers would bury their head in the sand when made aware of such a fact.
If I can tell an AI to increase profits, incidentally causing the AI to ultimately kill a bunch of people, I can at least claim a clean conscience by saying that wasn’t what I intended, even though it happened just the same.
In practice, legislators do this sort of thing routinely. They pass legislation that causes harm—sometimes a lot of harm—and sleep soundly.
Unfortunately, democracy itself depends on the economic and military relevance of masses of people. If that goes away, the iceberg will flip and the equilibrium system of government won’t be democracy.
Agreed. The rich and powerful could pick off more and more economically irrelevant classes while promising the remaining ones the same won’t happen to them, until eventually they can get everything they need from AI and live in enclaves protected by vast drone armies. Pretty bleak, but seems like the default scenario given the current incentives.
It seems really hard to think of any examples of such tech.
I think you would effectively have to build extensions to people’s neocortexes in such a way that those extensions cannot ever function on their own. Building AI agents is clearly not that.
Excellent post. This puts into words really well some thoughts that I have had.
I would also like to make an additional point: it seems to me that a lot of people (perhaps less so on LessWrong) hold the view that humanity has somehow “escaped” the process of evolution by natural selection, since we can choose to do a variety of things that our genes do not “want”, such as having non-reproductive sex. This is wrong. Evolution by natural selection is inescapable. When resources are relatively abundant, which is currently true for many Western nations, it can seem that it’s escapable because the selection pressures are relatively low and we can thus afford to spend resources somewhat frivolously. Since resources are not infinitely abundant, over time those selection pressures will increase. Those selection pressures will select out unproductive elements.
This means that even if we managed to get aligment right and form a utopia where everybody gets everything they need or more, they will eventually still be discarded because they cannot produce anything of economic value. In your post, capitalist incentives effectively play the role of natural selection, but even if we converted to a communist utopia, the result would ultimately be the same once selection pressures increase sufficiently, and they will.
Very interesting write-up! When you say that orcas could be more intelligent than humans, do you mean something similar to them having a higher IQ or g factor? I think this is quite plausible.
My thinking has been very much influenced by Joseph Henrich’s The Secret of Our Success, which you mentioned. For example, looking at the behavior of feral (human) children, it seems quite obvious to me now that all the things that humans can do better than other animals are all things that humans imitate from an existing cultural “reservoir” so to speak and that an individual human has virtually no hope of inventing within their lifetime, such as language, music, engineering principles, etc.
Gene-culture coevolution has resulted in a human culture and a human body that are adapted to each other. For example, the human digestive system is quite short because we’ve been cooking food for a long time, humans have muscles that are very weak compared to those of our evolutionary cousins because we’ve learned to make do with tools (weapons) instead and we have relatively protracted childhoods to absorb all of the culture required to survive and reproduce. If we tried to “uplift” orcas, the fact that human culture has co-evolved with the human body and not with the orca body would likely be an issue in trying to get them to learn it (a bit like trying to get software built for x86 to run on an ARM processor). Still, I think progress in LLM scaling shows that neural networks (artificial or biological) are able to absorb a significant chunk of human culture, as long as you have the right training method. I’ve made a similar point here.
There is nothing in principle that stops a chimpanzee from being able to read and write English, for example. It’s just that we haven’t figured out the methods to configure their brains into that state, because they don’t have a strong tendency to imitate, which human children do have, which makes training them much easier.
I agree with this view. Deep neural nets trained with SGD can learn anything. (“The models just want to learn.”) Human brains are also not really different from brains of other animals. I think the main struggles are 1. scaling up compute, which follows a fairly predictable pattern, and 2. figuring out what we actually want them to learn, which is what I think we’re most confused about.
My introduction to Dennett, half a lifetime ago, was this talk:
That was the start of his profound influence on my thinking. I especially appreciated his continuous and unapologetic defense of the meme as a useful concept, despite the many detractors of memetics.
Sad to know that we won’t be hearing from him anymore.
Yes. My bad, I shouldn’t have implied all hidden-variables interpretations.
Every non-deterministic interpretation has a virtually infinite Kolmogorov complexity because it has to hardcode the outcome of each random event.
Hidden-variables interpretations are uncomputable because they are incomplete.
It’s the simplest explanation (in terms of Kolmogorov complexity).
It’s also the interpretation which by far has the most elegant explanation for the apparent randomness of reality. Most interpretations provide no mechanism for the selection of a specific outcome, which is absurd. Under the MWI, randomness emerges from determinism through indexical uncertainty, i.e., not knowing which branch you’re in. Some people, such as Sabine Hossenfelder for example, get confused by this and ask, “then why am I this version of me?”, which implicitly assumes dualism, as if there is a free-floating consciousness which could in principle inhabit any branch; this is patently untrue because you are by definition this “version” of you. If you were someone else (including someone in a different branch where one of your atoms is moved by one Planck distance) then you wouldn’t be you; you would be literally someone else.
Note that the Copenhagen interpretation is also a many-worlds explanation, but with the added assumption that all but one randomly chosen world disappears when an “observation” is made, i.e., when entanglement with your branch takes place.
It’s just a matter of definition. We say that “you” and “I” are the things that are entangled with a specific observed state. Different versions of you are entangled with different observations. Nothing is stopping you from defining a new kind of person which is a superposition of different entanglements. The reason it doesn’t “look” that way from your perspective is because of entanglement and the law of the excluded middle. What would you expect to see if you were a superposition?
Have you read Joseph Henrich’s books The Secret of Our Success, and its sequel The WEIRDest People in the World? If not, they provide a pretty comprehensive view of how humanity innovates and particularly the Western world, which is roughly in line with what you wrote here.
I kind of agree that most knowledge is useless, but the utility of knowledge and experience that people accrue is probably distributed like a bell curve, which means you can’t just have more of the good knowledge without also accruing lots of useless knowledge. In addition, very often stuff that seems totally useless turns out to be very useful; you can’t always tell which is which.
I completely agree. In Joseph Henrich’s book The Secret of Our Success, he shows that the amount of knowledge possessed by a society is proportional to the number of people in that society. Dwindling population leads to dwindling technology and dwindling quality of life.
Those who advocate for population decline are unwittingly advocating for the disappearance of the knowledge, experience and frankly wisdom that is required to keep the comfortable life that they take for granted going.
Keeping all that knowledge in books is not enough. Otherwise our long years in education would be unnecessary. Knowing how to apply knowledge is its own form of knowledge.
If causality is everywhere, it is nowhere; declaring “causality is involved” will have no meaning. It begs the question whether an ontology containing the concept of causality is the best one to wield for what you’re trying to achieve. Consider that causality is not axiomatic, since the laws of physics are time-reversible.
I respect Sutskever a lot, but if he believed that he could get an equivalent world model by spending an equivalent amount of compute learning from next-token prediction using any other set of real-world data samples, why would they go to such lengths to specifically obtain human-generated text for training? They might as well just do lots of random recordings (e.g., video, audio, radio signals) and pump it all into the model. In principle that could probably work, but it’s very inefficient.
Human language is a very high density encoding of world models, so by training on human language models get much of their world model “for free“, because humanity has already done a lot of pre-work by sampling reality in a wide variety of ways and compressing it into the structure of language. However, our use of language still doesn’t capture all of reality exactly and I would argue it’s not even close. (Saying otherwise is equivalent to saying we’ve already discovered almost all possible capabilities, which would entail that AI actually has a hard cap at roughly human ability.)
In order to expand its world model beyond human ability, AI has to sample reality itself, which is much less sample-efficient than sampling human behavior, hence the “soft cap”.
In theory, yes, but that’s obviously a lot more costly than running just one instance. And you’ll need to keep these virtual researchers running in order to keep the new capabilities coming. At some point this will probably happen and totally eclipse human ability, but I think the soft cap will slow things down by a lot (i.e., no foom). That’s assuming that compute and the number of researchers even is the bottleneck to new discoveries; it could also be empirical data.
If you accept the premise of AI remaining within the human capability range in some broad sense, where it brings great productivity improvements and rewards those who use it well but remains foundationally a tool and everything seems basically normal, essentially the AI-Fizzle world, then we have disagreements
There is good reason to believe that AI will have a soft cap at roughly human ability (and by “soft cap” I mean that anything beyond the cap will be much harder to achieve) for the same reason that humans have a soft cap at human ability: copying existing capabilities is much easier than discovering new capabilities.
A human being born today can relatively easily achieve abilities that other humans have achieved, because you just copy them; lots of 12-year-olds can learn calculus, which is much easier than inventing it. AI will have the same issue.
The European socket map is deceptive. My charger will work anywhere on mainland Europe. Looking at the sockets, can you tell why?
I don’t know how selection pressures would take hold exactly, but it seems to me that in order to prevent selection pressures, there would have to be complete and indefinite control over the environment. This is not possible because the universe is largely computationally irreducible and chaotic. Eventually, something surprising will occur which an existing system will not survive. Diverse ecosystems are robust to this to some extent, but that requires competition, which in turn creates selection pressures.