But human nervous systems do have much higher bandwidth communication channels. We share them with the other mammals. It’s the limbic system.
I’m quite uncertain about how high-bandwidth this actually is. I agree that in the first second of meeting someone, it’s much more informative than language could be. Once the initial “first impression” has occurred, though, the rate of communication drops off sharply, and I think that language could overtake it after a few minutes. For example, it takes half a second to say “I’m nervous”, and you can keep saying similarly-informative things for a long time: do you think you could get a new piece of similar information every half second for ten minutes via the limbic system?
(Note that I’m not necessarily saying people do communicate information about their emotions, personality and social status faster via language, just that they could).
The success of human society is a good demonstration of how very low complexity systems and behaviours can drive your competition extinct, magnify available resources, and more
On what basis are you calling human societies “very low complexity systems”? The individual units are humans, whose brains are immensely complex; and then the interactions between humans are often complicated enough that nobody has a good understanding of the system as a whole.
Ultimately there seems to be no impetus for a half-baked neuron tentacle, and a lot of cost and risk, so that will probably never be the path to such organisms.
This seems somewhat intuitive, but note that this statement is a universal negative: it’s saying there is no plausible path to this outcome. In general I think we should be pretty cautious about such statements, since a lot of evolutionary innovations would have seemed deeply implausible before they happened. For example, the elephant’s trunk is a massive neuron-rich tentacle which is heavily used for communication.
There are lots of simple things that organisms could do to make them wildly more successful.
I guess this is the crux of our disagreement—could you provide some examples?
Anyone who’s done any infosec or network protocol work will laugh at the idea that trial and error (evolution) can make a safe high-bandwidth connection.
There are hundreds of things that people have laughed at the idea of trial and error (evolution) doing, which evolution in fact did. Thinking that evolution is dumb is generally not a good heuristic.
Also, I’m not sure what counts as safe in this context. Is language safe? Is sight?
Downvoted for being very hyperbolic (“less than zero”, “his opponents here are all pure conflict theorists”), uncharitable/making personal attacks (“There are people opposed to nerds or to thinking”, “There are people who are opposed to action of any kind”), and not substantiating these extreme views (“Because, come on. Read their quotes. Consider their arguments.”)
As a more object-level comment, suppose I accept the hypothetical that all attacks on billionaire philanthropy are entirely aimed at reducing the power of billionaires. Yet if we build a societal consensus that this particular attack is very misguided and causes undesirable collateral damage, then even people who are just as anti-billionaire as before will be less likely to use it. E.g. it’s much harder to criticise billionaires who only spend their money on helping kids with cancer.
Very interesting question—the sort that makes me swing between thinking it’s brilliant and thinking it’s nonsense. I do think you overstate your premise. In almost all of the examples given in The Secret of our Success, the relevant knowledge is either non-arbitrary (e.g. the whole passage about hunting seals makes sense, it’s just difficult to acquire all that knowledge), or there’s a low cost to failure (try a different wood for your arrows; if they don’t fly well, go back to basics).
If I engage with the question as posed, though, my primary answer is simply that over time we became wealthy and technologically capable enough that we were able to replace all the natural things that might kill us with whatever we’re confident won’t kill us. Which is why you can improvise while cooking—all of the ingredients have been screened very hard for safety. This is closely related to your first hypothesis.
However, this still leaves open a slightly different question. The modern world is far too complicated for anyone to understand, and so we might wonder why incomprehensible emergent effects don’t render our daily lives haphazard and illegible. One partial answer is that even large-scale components of the world (like countries and companies) were designed by humans. A second partial answer, though, is that even incomprehensible patterns and mechanisms in the modern world still interact with you via other people.
This has a couple of effects. Firstly, other people try to be legible, it’s just part of human interaction. (If the manioc could bargain with you, it’d be much easier to figure out how to process it properly.)
Secondly, there’s an illusion of transparency because we’re so good at and so used to understanding other people. Social interactions are objectively very complicated: in fact, they’re “cultural norms and processes which appear arbitrary, yet could have fatal consequences if departed from”. Yet it doesn’t feel like the reason I refrain from spitting on strangers is arbitrary (even though I couldn’t explain the causal pathway by which people started considering it rude). Note also that the space of ideas that startups explore is heavily constrained by social norms and laws.
Thirdly, facts about other humans serve as semantic stop signs. Suppose your boss fires you, because you don’t get along. There’s a nearly unlimited amount of complexity which shaped your personality, and your boss’ personality, and the fact that you ended up in your respective positions. But once you’ve factored it out into “I’m this sort of person, they’re that sort of person”, it feels pretty legible—much more than “some foods are eat-raw sorts of foods, other foods are eat-cooked sorts of foods”. (Or at least, it feels much more legible to us today—maybe people used to find the latter explanation just as compelling). A related stop sign is the idea that “somebody knows” why each step of a complex causal chain happened, which nudges us away from thinking of the chain as a whole as illegible.
So I’ve given two reasons for increased legibility (humans building things, and humans explaining things), and two for the illusion of legibility (illusion of transparency, and semantic stop signs). I think on small scales, the former effects predominate. But on large scales, the latter predominate—the world seems more legible than it actually is. For example:
The world seems legible—I can roughly predict how many planes fly every day by multiplying a handful rough numbers.
Roughly predicting the number of planes which fly every day is a very low bar! You can also predict the number of trees in a forest by multiplying a handful of numbers. This doesn’t help you survive in that forest. What helps you survive in the forest is being able to predict the timing of storms and the local tiger population. In the modern world, what helps you thrive is being able to predict the timing of recessions and crime rate trends. I don’t think we’re any better at the latter two than our ancestors were at the former. In fact, the large-scale arcs of our lives are now governed to a much greater extent by very unpredictable and difficult-to-understand events, such as scientific discoveries, technological innovation and international relations.
In summary, technology has helped us replace individual objects in our environments with safer and more legible alternatives, and the emergent complexity which persists in our modern environments is now either mediated by people, or still very tricky to predict (or both).
But my simple sense is that openly discussing whether or not nuclear weapons were possible (a technical claim on which people might have private information, including intuitions informed by their scientific experience) would have had costs and it was sensible to be secretive about it. If I think that timelines are short because maybe technology X and technology Y fit together neatly, then publicly announcing that increases the chances that we get short timelines because someone plugs together technology X and technology Y. It does seem like marginal scientists speed things up here.
I agree that there are clear costs to making extra arguments of the form “timelines are short because technology X and technology Y will fit together neatly”. However, you could still make public that your timelines are a given probability distribution D, and the reasons which led you to that conclusion are Z% object-level views which you won’t share, and (100-Z)% base rate reasoning and other outside-view considerations, which you will share.
I think there are very few costs to declaring which types of reasoning you’re most persuaded by. There are some costs to actually making the outside-view reasoning publicly available—maybe people who read it will better understand the AI landscape and use that information to do capabilities research.
But having a lack of high-quality public timelines discussion also imposes serious costs, for a few reasons:
1. It means that safety researchers are more likely to be wrong, and therefore end up doing less relevant research. I am generally pretty skeptical of reasoning that hasn’t been written down and undergone public scrutiny.
2. It means there’s a lot of wasted motion across the safety community, as everyone tries to rederive the various arguments involved, and figure out why other people have the views they do, and who they should trust.
3. It makes building common knowledge (and the coordination which that knowledge can be used for) much harder.
4. It harms the credibility of the field of safety from the perspective of outside observers, including other AI researchers.
Also, the more of a risk you think 1 is, the lower the costs of disclosure are, because it becomes more likely that any information gleaned from the disclosure is wrong anyway. Yet predicting the future is incredibly hard! So the base rate for correctness here is low. And I don’t think that safety researchers have a compelling advantage when it comes to correctly modelling how AI will reach human level (compared with thoughtful ML researchers).
Consider, by analogy, a debate two decades ago about whether to make public the ideas of recursive self-improvement and fast takeoff. The potential cost of that is very similar to the costs of disclosure now—giving capabilities researchers these ideas might push them towards building self-improving AIs faster. And yet I think making those arguments public was clearly the right decision. Do you agree that our current situation is fairly analogous?
EDIT: Also, I’m a little confused by
Suppose I have 5 reasons for wanting discussions to be private, and 3 of them I can easily say.
I understand that there are good reasons for discussions to be private, but can you elaborate on why we’d want discussions about privacy to be private?
We have strong outside-view reasons to expect that the information processing in question probably approximates Bayesian reasoning (for some model of the environment), and the decision-making process approximately maximizes some expected utility function (which itself approximates fitness within the ancestral environment).
The use of “approximates” in this sentence (and in the post as a whole) is so loose as to be deeply misleading—for the same reasons that the “blue-minimising robot” shouldn’t be described as maximising some expected utility function, and the information processing done by a single neuron shouldn’t be described as Bayesian reasoning (even approximately!)
See also: coherent behaviour in the real world is an incoherent concept.
There is at least one example (I’ve struggled to dig up) of a memory-less RL agent learning to encode memory information in the state of the world.
I recall an example of a Mujoco agent whose memory was periodically wiped storing information in the position of its arms. I’m also having trouble digging it up though.
We will say that a system is an optimizer if it is internally searching through a search space (consisting of possible outputs, policies, plans, strategies, or similar) looking for those elements that score high according to some objective function that is explicitly represented within the system.
I appreciate the difficulty of actually defining optimizers, and so don’t want to quibble with this definition, but am interested in whether you think humans are a central example of optimizers under this definition, and if so whether you think that most mesa-optimizers will “explicitly represent” their objective functions to a similar degree that humans do.
Agreed that this points in the right direction. I think there’s more to it than that though. Consider for example a three-body problem under Newtonian mechanics. Then there’s a sense in which specifying the initial masses and velocities of the bodies, along with Newton’s laws of motion, is the best way to compress the information about these chaotic trajectories.
But there’s still an open question here, which is why are three-body systems chaotic? Two-body systems aren’t. What makes the difference? Finding an explanation probably doesn’t allow you to compress any data any more, but it still seems important and interesting.
(This seems related to a potential modification of your data compression standard: that good explanations compress data in a way that minimises not just storage space, but also the computation required to unpack the data. I’m a little confused about this though.)
Thanks for the kind words. I agree that refactoring would be useful, but don’t have the time now. I have added some headings though.
A relevant book recommendation: The Enigma of Reason argues that thinking of high-level human reasoning as a tool for attacking other people’s beliefs and defending our own (regardless of their actual veracity) helps explain a lot of weird asymmetries in cognitive biases we’re susceptible to, including this one.
I’d like to push back on the assumption that AIs will have explicit utility functions. Even if you think that sufficiently advanced AIs will behave in a utility-maximising way, their utility functions may be encoded in a way that’s difficult to formalise (e.g. somewhere within a neural network).
It may also be the case that coordination is much harder for AIs than for humans. For example, humans are constrained by having bodies, which makes it easier to punish defection—hiding from the government is tricky! Our bodies also make anonymity much harder. Whereas if you’re a piece of code which can copy itself anywhere in the world, reneging on agreements may become relatively easy. Another reasion why AI cooperation might be harder is simply that AIs will be capable of a much wider range of goals and cognitive processes than humans, and so they may be less predictable to each other and/or have less common ground with each other.
This paper by Critch is relevant: it argues that agents with different beliefs will bet their future share of a merged utility function, such that it skews towards whoever’s predictions are more correct.