One (non-goodhart resistent) way would be to use some kind of Page Rank method based on the tags that users comment and post on. Eg. People Rank high in AI posts, if their comments and AI posts were upvoted by other people with the same Expertise.
If they alternated directions, increasing the distance walked each time by a constant factor (e.g. triple), then an exit in either direction would be reached without too much wasted time.
Some scientists tried to create group selection under laboratory conditions. They divided some insects into subpopulations, then killed off any subpopulation whose numbers got too high, and and “promoted” any subpopulation that kept its numbers low to better conditions. They hoped the insects would evolve to naturally limit their family size in order to keep their subpopulation alive. Instead, the insects became cannibals: they ate other insects’ children so they could have more of their own without the total population going up. In retrospect, this makes perfect sense; an insect with the behavioral program “have many children, and also kill other insects’ children” will have its genes better represented in the next generation than an insect with the program “have few children”.
Why didn’t they try also killing off subpopulations that engaged in cannibalism, and promoting those that didn’t? And what would have most likely happened if they had tried that?
Probably it is also depends on how much information about “various models trying to naively backstab their own creators” there are in the training dataset
Shortform #139 Note to self: don’t reschedule weekly meetup
Tonight’s Virginia Rationalists: Norfolk meetup was great! We had a new individual join us who is familiar with ACX.
In retrospect, I think it was inconsiderate to suddenly reschedule the group’s weekly meetup just because I couldn’t attend. My co-organizer and others likely could have attended yesterday, and there were a few people who just couldn’t make the rescheduled meetup but had included Wednesday nights in their routine as “meetup night”. So, I intend to not reschedule the weekly social meetup unless literally everyone reaches out in advance and asks for that to occur. Weekly meetups = people build that into their routines, and should not be changed lightly.
How actually do you sidestep the need for the One True Objective Function given an ELK solution? I get that it might seem plausible to take a rough objective like “do what I intend” and look at the internal knowledge of the thing for signs that it is deliberately deceiving you. If you do that, you’ll get, at best, an AI that doesn’t know that it is deceiving you (for whatever operationalization of “know” you come up with as you use ELK for training). But it could still be deceiving you, and very likely will be if optimization pressure is merely towards “AIs that don’t know that they are being deceptive”.
Take a reinforcement learner AI, that we want to safely move a strawberry onto a plate. A human sits nearby and provides a reward based on inspecting the AI’s behaviour.
As it stands, this setup is completely vulnerable to reward hacking. The reward is not provided for safe moving of the strawberry; instead the reward is provided by having the human judge that the task has been accomplished and then pressing a button. Taking control of the human or control of the button is likely to be possible for a superintelligent AI; and, as it stands, that would be mandated by this reward function.
Would it be sufficient, for disproof, to show one system that does steer far-away parts of the world into a relatively-small chunk of their state space, but does not internally contain a world-model or do planning?
That could be sufficient in principle, though I would not be surprised if I look at a counterexample and realize that the problem description was missing something rather than that the claim is basically false. For instance, it wouldn’t be too surprising if there’s some class of supposed-counterexamples which only work under very specific conditions (i.e. they’re not robust), and can be ruled out by some “X isn’t very likely to happen spontaneously in a large system” argument.
The bottom line is that a disproof should argue that there isn’t anything basically like the claim which is true. Finding a physical counterexample would at least show that the counterexample isn’t sensitive to the details of mathematical framework/formulation.
Do you have a particular counterexample in mind? A thermometer, perhaps?
Well ok just to brainstorm some naive things that don’t really rule the conjecture out:
A nuclear bomb steers a lot of far-away objects into a high-entropy configuration, and does so very robustly, but that perhaps is not a “small part of the state space”
A biological pathogen, let loose in a large human population, might steer all the humans towards the configuration “coughing”, but the virus is not itself a consequentialist. You might say that the pathogen had to have been built by a consequentialist, though.
Generalizing the above: Suppose I discover some powerful zero-day exploit for the linux kernel. I automate the exploit, setting my computer up to wait 24 hours and then take over lots of computers on the internet. Viewing this whole thing from the outside, it might look as if it’s my computer that is “doing” the take-over, but my computer itself doesn’t have a world model or a planning routine.
Consider some animal species spreading out from an initial location and making changes to the environments they colonize. If you think of all the generations of animals that underwent natural selection before spreading out as the “system that controls some remote parts of the system” and the individual organisms as kind of messages or missiles, then this seems like a pretty robust, though slow form of remote control. Maybe you would say that natural selection has a world model and a planning process, though.
Since this seems to be Carn’s first post on LessWrong, I think some of the other readers should have been more lenient and not downvoted the post or explained why they downvoted the post.
I would only downvote a post if it was obviously bad, flawed, very poorly written, or a troll post.
This post contains lots of interesting ideas and seems like a good first post.
The original post “Reward is not the optimization target” has 216 upvotes and this one has 0. While the original post was written better, I’m skeptical of the main idea and it’s good to see a post countering it so I’m upvoting this post.
the core thing I worry about with any simulation-based approach is how to get coverage of possibility space. ensuring that the tests are informative about possible configurations of complex systems is hard; I would argue that this is a great set of points, and that no approach without this type of testing could be expected to succeed.
however, as we’ve seen with adversarial examples, both humans and current DL have fairly severe failures of alignment that cause large issues, and projects like this need tools to optimize for interpretability from the perspective of a formal verification algorithm.
it’s my belief that in almost all cases, the values almost all beings ever wish to express lie on a low-enough-dimensional manifold that, if the ideal representation was learned, a formal verifier would be able to handle it; see, for example, recent work from anthropic on interpretability lining up nicely with what’s needed to use reluplex (or followup work; reluplex is old now) to estimate bounds on larger networks’ behavior.
I don’t think we should give up on perfectly solving safety; instead, we should recognize that it has never even been closely approximated before, by any form of life, because even semi-planned evolutionary processes do not do a good job of covering the entire possibility space.
this post is an instant classic, and I agree that sims are a key step, starting from small agents. but ensuring that strongly generalizing moral views are generated in the agents who grow in the simulations is not trivial—we need to be able to generate formal bounds on the loss function, and doing so requires being able to prove all the way through the simulation the way we currently take gradients through it. if we can ask “where in the simulation would have changed this outcome”, then we’d finally be getting somewhere in terms of generating moral knowledge that generalizes universally.
I wonder, almost just idle curiosity, whether or not the “measuring-via-proxy will cause value drift” is something we could formalize and iterate on first. Is the problem stable on the meta-level, or is there a way we can meaningfully define “not drifting from the proxy” without just generally solving alignment.
Intuitively I’d guess this is the “don’t try to be cute” class of thought, but I was afraid to post at all and decided that I wanted to interact, even at the cost of (probably) saying something embarassing.
It is at least not obvious to me that this is a “don’t try to be cute” class of thought, though not obvious that it isn’t either. Depends on the details.
By contrast, the world where sharp left turns happen near human-level capabilities seems pretty dangerous. In that world, we don’t get to study things like deceptive alignment empirically. We don’t get to leverage powerful AI researcher tools.
We actually can study and iterate on alignment (and deception) in simulation sandboxes while leveraging any powerful introspection/analysis tools. Containing human-level AGI in (correctly constructed) simulations is likely easy.
But then later down you say:
For instance, we can put weak models in environments that encourage alignment failure so as to study left turns empirically (shifting towards the “weak” worlds).
Which seems like a partial contradiction, unless you believe we can’t contain human-level agents?
Is this something you’ve thought deeply about and or care to expand on? Curious about your source of skepticism, considering:
we can completely design and constrain the knowledge base of sim agents, limiting them to the equivalent of 1000BC human knowledge, or whatever we want
we can completely and automatically monitor their inner monologues, thoughts, etc
How do you concretely propose a sim agent will break containment? (which firstly requires sim-awareness)
How would you break containment now, assuming you are in a sim?
Also, Significant speed/memory advantages go against the definition of ‘human-level agents’ and are intrinsically unlikely anyway as 2x speed/memory agents are simply preceded by 1x speed/memory agents, and especially due to the constraints of GPUs which permit acceleration by parallelization of learning across agents, more so than serial speedup of individual agents.
My modal expectation is that these are going to look something like large language models, meaning that while we’ll have controlled the training corpus it won’t be something hand-curated. So there will almost certainly be lots of information in there that models could use to e.g. figure out how to manipulate people, program computers, and so on. I expect then that we won’t be limiting their knowledge as much as you think we will, because I expect them to be trained with giant corpuses of (mostly) post-1000BC text.
We do not currently have the interpretability tools needed to monitor their inner thoughts. I would feel much better about this problem if we had such tools!
My best guess is that containment fails because a model convinces one of the people in the lab to let it access the internet. That could look like getting someone to think it’s genuinely a good idea (because it’ll be helpful for some project), or it could look like bribery (with offer of future payment) or blackmail.
I’m guessing that we’ll want to run some agents fast because that’s how you get lots of serial intellectual work done. So even if all we get are human-level agents, there’ll be a big incentive to run them fast.
Indeed a model trained on (or with full access to) our internet could be very hard to contain. That is in fact a key part of my argument. If it is hard to contain, it is hard to test. If it’s hard to test, we are unlikely to succeed.
So if we agree there, then perhaps you agree that giving untested AGI access to our immense knowledge base is profoundly unwise.
I expect then that we won’t be limiting their knowledge as much as you think we will, because I expect them to be trained with giant corpuses of (mostly) post-1000BC text.
That’s then just equivalent to saying “I expect then that we won’t even bother with testing our alignment designs”. Do you actually believe that testing is unnecessary? Or do you just believe the leading teams won’t care? And if you agree that testing is necessary, then shouldn’t this be key to any successful alignment plan?
My best guess is that containment fails because a model convinces one of the people in the lab to let it access the internet.
Which obviously is nearly impossible if it doesn’t know it is in a sim, and doesn’t know what a lab or the internet are, and lacks even the precursor concepts. Since you seem to be focused on the latest fad (large language models), consider the plight of poor simulated Elon Musk—who at least is aware of the sim argument.
The situation is not ‘handled.’ Elites have lost all credibility.
I think it’s worth caveating this that not all elites have lost credibility. Elites in places like Singapore, Switzerland, and Finland have a lot of credibility.
Edits based on feedback from LessWrong and the EA Forum:
EDITS: - Added new ‘Definitions’ section to introduction to explain definitions such as ‘AI safety’, ‘researcher’ and the difference between technical and non-technical research.
NON-TECHNICAL - CSER: 5-5-10 → 2-5-15 - Delete BERI from the list of non-technical research organizations - Delete SERI from the list of non-technical research organizations - Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence) - FLI: 5-5-20 → 3-5-15 - Add OpenPhil: 2-5-15 - Epoch: 5-10-15 → 2-4-10 - Add ‘Other’: 5-10-50
TODO: - update all graphs
- add Rethink Priorities to the list of non-technical organizations
Only make choices that you would not make in reverse, if things were the other way around. Drop out of school if and only if you wouldn’t enroll in school from out of the workforce. Continue school if and only if you’d switch over from work to that level of schooling.
Flitting back and forth between both possible worlds can make you less cagey about doing what’s overdetermined by your world model + utility function already. It’s also part of the exciting rationalist journey of acausally cooperating with your selves in other possible worlds.
It’s probably a useful mental technique to consider from both directions, but also consider that choices that appear symmetric at first glance may not actually be symmetric. There are often significant transition costs that may differ in each direction, as well as path dependencies that are not immediately obvious.
As such, I completely disagree with the first paragraph of the post, but agree with the general principle of considering such decisions from both directions and thank you for posting it.
I do research in cooperation and game theory, including some work on altruism, and also some hard science work. Everyone looks at the Rorschach blot of human behavior and sees something different. Most of the disagreements have never been settled. Even experiment does not completely settle them.
My experience from having children and observing them in the first few months of life is more definitive. They come with values and personal traits that are not very malleable, and not directly traceable to parents. Sometimes grandparents (who were already dead, so it had to be genetic).
My experience with A.I. researchers is that they are looking for a shortcut. No one’s career will permit raising an experimental AI as a child. The tech of the AI would be obsolete before the experiment was complete. This post is wishful thinking that a shortcut is available. It is speculative, anecdotal, and short on references to careful experiment. Good luck with that.
I think there are two fundamental problems with the extensive simboxing approach. The first is just that, given the likely competitive dynamics around near-term AGI (i.e. within the decade), these simboxes are going to be extremely expensive both in compute and time which means that anybody unilaterally simboxing will probably just result in someone else releasing an unaligned AGI with less testing.
If we think about the practicality of these simboxes, it seems that they would require (at minimum) the simulation of many hundreds or thousands of agents over relatively long real timelines. Moreover, due to the GPU constraints and Moore’s law arguments you bring up, we can only simulate each agent at close to ‘real time’. So years in the simbox must correspond to years in our reality, which is way too slow for an imminent singularity. This is especially an issue given that we must maintain no transfer of information (such as datasets) from our reality into the sim. This means at minimum years of sim-time to bootstrap intelligent agents (taking humans data-efficiency as a baseline). Also, each of these early AGIs will be likely be incredibly expensive in compute so that maintaining reasonable populations of them in simulation will be very expensive and probably infeasible initially. If we could get policy coordination on making sure all actors likely to develop AGI go through a thorough simboxing testing regimen, then that would be fantastic and would solve this problem.
Perhaps a more fundamental issue is that simboxing does not address the fundamental cause of p(doom) which is recursive self improvement of intelligence and the resulting rapid capability gains. The simbox can probably simulate capability gains reasonably well (i.e. gain ‘magical powers’ in a fantasy world) but I struggle to see how it could properly test gains in intelligence from self-improvement. Suppose the AI in the fantasy simbox brews a ‘potion’ that makes it 2x as smart. How do we simulate this? We could just increase the agent’s compute in line with the scaling laws but a.) early AGIs are almost certainly near the frontier of our compute capability anyway and b.) much of recursive self improvement is presumably down to algorithmic improvements which we almost necessarily cannot simulate (since if we knew better algorithms we would have included them in our AGIs in the simulation in the first place!)
This is so vital because the probable breakdown of proxies to human values under the massive distributional shift induced by recursive self improvement is the fundamental difficulty to alignment in the first place.
Perhaps this is unique to my model of AI risk, but almost all the probability of doom channels through p(FOOM) such that p(doom | no FOOM) is quite low in comparison. This is because if we have don’t have FOOM then there is not extremely large amounts of optimization power unleashed and the reward proxies for human values and flourishing don’t end up radically off-distribution and so probably don’t break down. There are definitely a lot of challenges left in this regime, but to me it looks solvable and I agree with you that in worlds without rapid FOOM, success will almost certainly look like considerable iteration on alignment with a bunch of agents undergoing some kind of automated simulated alignment testing in a wide range of scenarios plus using the generalisation capabilities of machine learning to learn reward proxies that actually generalise reasonably well within the distribution of capabilities actually obtained. The main risk, however, in my view, comes from the FOOM scenario.
Finally, I just wanted to say that I’m a big fan of your work and some of your posts have caused major updates to my alignment worldview—keep up the fantastic work!
Thanks, upvoted for engagement and constructive criticism—I’d like more to see this comment.
I’m going to start perhaps in reverse to establish where we seem to most agree:
There are definitely a lot of challenges left in this regime, but to me it looks solvable and I agree with you that in worlds without rapid FOOM, success will almost certainly look like considerable iteration on alignment with a bunch of agents undergoing some kind of automated simulated alignment testing in a wide range of scenarios plus using the generalisation capabilities of machine learning to learn reward proxies that actually generalise reasonably well within the distribution of capabilities actually obtained.
I fully agree with this statement.
However in worlds that rapidly FOOM, everything becomes more challenging, and I’ll argue in a moment why I believe that the approach presented here still is promising in rapid FOOM scenarios, relative to all other practical techniques that could actually work.
But even without rapid FOOM, we still can have disaster—for example consider the scenario of world domination by a clan of early uploads of some selfish/evil dictator or trillionaire. There’s still great value in solving alignment here, and (to my eyes at least) much less work focused on that area.
Now if rapid FOOM is near inevitable, then those considerations naturally may matter less. But rapid FOOM is far from inevitable.
First, Moore’s Law is ending, and brains are efficient, perhaps even near pareto-optimal.
Secondly, the algorithms of intelligence are much simpler than we expected, and brains already implement highly efficient or even near pareto-optimal approximations of the ideal universal learning algorithms.
To the extent either of those major points are true, rapid FOOM is much less likely; to the extent both are true (as they appear to be), then very rapid FOOM is very unlikely.
Performance improvement is mostly about scaling compute and data in quantity and quality—which is exactly what has happened with deep learning, which was deeply surprising to many in the ML/comp-sci community and caused massive updates (but was not surprising and was in fact predicted by those of us arguing for brain efficiency and brain reverse engineering).
Now, given that background, there a few other clarifications and/or disagreements:
If we think about the practicality of these simboxes, it seems that they would require (at minimum) the simulation of many hundreds or thousands of agents over relatively long real timelines. Moreover, due to the GPU constraints and Moore’s law arguments you bring up, we can only simulate each agent at close to ‘real time’.
To a first approximation, compute_cost = size*speed. If AGI requires brain size, then the first to cross the finish line will likely be operating not greatly faster than the minimum speed, which is real-time. But this does not imply the agents learn at only real time speed, as learning is parallelizable across many agent instances. Regardless, none of these considerations depend on whether the AGI is trained in a closed simbox or an open sim with access to the internet.
So just to clarify:
AGI designs in simboxes are exactly the same as unboxed designs, and have exactly the same compute costs
The only difference is in the datastream and thus knowledge
The ideal baseline cost of simboxing is only O(N+1) vs O(N) without—once good AGI designs are found, the simboxing approach requires only one additional unboxed training run (compared to never using simboxes). We can estimate this additional cost: it will be around or less than 1e25 ops (1e16 ops/s for brain-size model * 1e9s seconds for 30 years equivalent), or less than $10 million dollars (300 gpu years) using only todays gpus, ie nearly nothing.
Perhaps a more fundamental issue is that simboxing does not address the fundamental cause of p(doom) which is recursive self improvement of intelligence and the resulting rapid capability gains. The simbox can probably simulate capability gains reasonably well (i.e. gain ‘magical powers’ in a fantasy world) but I struggle to see how it could properly test gains in intelligence from self-improvement. Suppose the AI in the fantasy simbox brews a ‘potion’ that makes it 2x as smart. How do we simulate this? We could just increase the agent’s compute in line with the scaling laws but a.) early AGIs are almost certainly near the frontier of our compute capability anyway and b.) much of recursive self improvement is presumably down to algorithmic improvements which we almost necessarily cannot simulate (since if we knew better algorithms we would have included them in our AGIs in the simulation in the first place!)
If brains are efficient, then matching them will already use up most of our algorithmic optimization slack—which again seems to be true based on the history of deep learning. But let’s suppose there still is significant optimization slack, then in a sense you’ve almost answered your own question . .. we can easily incorporate new algorithmic advances into new simboxes or even upgrade agents mid-sim using magic potions or what not.
If there is great algorithmic slack, then we can employ agents which graduate from simboxes as engineers in the design of better AGI and simboxes. To the extent there is any downside here or potential advantage for other viable approaches, that difference seems to come strictly at the cost of alignment risk.
Assume there was 1.) large algorithmic slack, and 2.) some other approach that was both viable and significantly different, then it would have to:
not use adequate testing of alignment (ie simboxes)
or not optimize for product of intelligence potential and measurable alignment/altruism
Do you think such an other approach could exist? If so, where would the difference lie and why?
Just read this about 3 years later. I found Thiel to be mostly spot on. Especially:
Peter Thiel: Right. Look, I don’t know how you solve the social problem if everybody has to be a mathematician or a concert pianist. I want a society in which we have great mathematicians and great concert pianists. That seems that that would be a very healthy society. It’s very unhealthy if every parent thinks their child has to be a mathematician or a concert pianist, and that’s the kind of society we unfortunately have.
Which seems to point to the fact that the root cause of many societal ills is that the average human psyche simply can’t accept the fact that in Thiel’s language, the vast majority of children, including most likely theirs, will never be a great mathematician or concert pianist. (or attain an equally prestigious position)
Since human prestige by definition is defined relative to other humans and thus only a tiny minority could ever be near the top.
Yet there seems to be some instinctual demand for an individual to be special in some way and furthermore that this must be recognized by some sufficiently large group. i.e. for a child to grow into an adult, mediocre in every way, is seen as a tragedy.
Thus necessitating a huge amount of distortions everywhere, across all institutions and policies causing damage in innumerable ways, to compensate.
The problem is sometimes you actually learn more by cheating, because making strategies on how to cheat actually makes you remember more.
Its actually creative activity.
Also sometimes cheating actually made me able to remember the stuff more as it gave me a chance.
On other hand the long term benefits of cheating are tiny.
Obviously the best strategy is to learn, cheating or not.
The key important thing is whether cheating is viable strategy in real life.
On other hand cheating is kind of ambiguous concept.
Rules are many. If you make notes in math class and they are permitted then its not cheating.
Some teachers explicitly allowed people to use books and look up answers.
This would obviously be cheating in most other classes.
But if the end game is knowledge then its either about the process or the result.
If its about process both methods are OK, if its purely about result then the best technique, is to simply do it as fast as possible in little time as possible as soon as possible.
This in and of its self is actually valued approach.
As for shoplifting I have done something like that.
Not sure I actually enjoyed it. I think I just did it, because you kind of think, hey its possible?
First time I was certainly dumb enough to get duped.
Not sure even why.
Mind fallacy is interesting concept.
The approach is probably simply done, because its the easiest way to approach people.
Or at least it seems.
People are complex, so without trying to assume things you kind of work out what the common ground is.
Unfortunately there are so many ways to, not figure this out.
What makes people tick is something I just don’t get.
Or more accurately I get it, but I actually don’t know how that translates to real life approaching people.
I plan to mentor several people to work on shard theory and agent foundations this winter through SERI MATS. Apply here if you’re interested in working with me and Quintin.
I think this’d be better with a description of what it is and how it’s relevant. (Linkposts generally benefit from that, and in this case it almost looks like spam if you’re not paying attention)
I imagine something like Stack Exchange, except that people could get certified for rationality and for domain knowledge, and then would have corresponding symbols next to their user names.
Well, the rationality certification would be a problem. An ideal test could provide results like “rational in general, except when the question is related to a certain political topic”. Because, it would be difficult to find enough perfectly rational people, so the second best choice would be to know their weaknesses.
I’m not sure we’d need anything that elaborate. The rationalist community isn’t that big. I was more thinking that rationalists could self-nominate their expertise, or that a couple of people could come together and nominate someone if they notice that that person is has gone in depth with the topic.
I’ve previously played with the idea of more elaborate schemes, including tests, track records and in-depth arguments. But of course the more elaborate the scheme, the more overhead there is, and I’m not sure that much overhead is affordable or worthwhile if one just wants to figure stuff out.
I agree. We could afford more overhead if we had thousands of rationalists active on the Q&A site. Realistically, we will be lucky if we get twenty.
But some kind of verification would be nice, to prevent the failure mode of “anyone who creates an account is automatically considered a rationalist”. Similarly, if people simply declare their own expertise, it gives more exposure to overconfident people.
How to achieve this as simply as possible?
One idea is to have a network of trust. Some people (e.g. all employees of MIRI and CFAR) would automatically be considered “rationalists”; other people become “rationalists” only if three existing rationalists vouch for them. (The vouch can be revoked or added at any moment. It is evaluated recursively, so if you lose the flag, the people you vouched for might lose their flags too, unless they already have three other people vouching for them.) There is a list of skills, but you can only upvote or downvote other people having a skill; if you get three votes, the skill is displayed next to your name (tooltip shows the people who upvoted it, so if you say something stupid, they can be called out).
This would be the entire mechanism. The meta debate could be in special LW threads, or perhaps in shortform, you could post there e.g. “I am an expert on X, could someone please confirm this? you can interview me by Zoom”, or you could call out other people’s misleading answers, etc.
American Academy of Pediatrics lies to us once again....
“If caregivers are wearing masks, does that harm kids’ language development? No. There is no evidence of this. And we know even visually impaired children develop speech and language at the same rate as their peers.” This is a textbook case of the Law of No Evidence. Or it would be, if there wasn’t any Proper Scientific Evidence.
Is it, though? I’m no expert, but I tried to find Relevant Literature. Sometimes, counterintuitive things are true.
Blindness affects congenitally blind children’s development in different ways, language development being one of the areas less affected by the lack of vision.
Most researchers have agreed upon the fact that blind children’s morphological development, with the exception of personal and possessive pronouns, is not delayed nor impaired in comparison to that of sighted children, although it is different. As for syntactic development, comparisons of MLU scores throughout development indicate that blind children are not delayed when compared to sighted children Blind children use language with similar functions, and learn to perform these functions at the same age as sighted children. Nevertheless, some differences exist up until 4;6 years; these are connected to the adaptive strategies that blind children put into practice, and/or to their limited access to information about external reality. However these differences disappear with time (Pérez-Pereira & Castro, 1997). The main early difference is that blind children tend to use self-oriented language instead of externally oriented language.
I don’t know exactly where that leaves us evidentially. Perhaps the AAP is lying by omission by not telling us about things other than language that are affected by children’s sight.
That’s a bit different to the dishonesty alleged, though.
Wait, so could we donate some Jones Act noncompliant dredges and tugboats to a “hostile” force, declare war on them, nonviolently “seize” the boats, and skirt the law that way?
How would a future government enforce their tax policies on a distant star system?
Since it’s vastly easier to destroy something than to build in outer space, there’s no feasible way of using the threat of violence, at least not without mutually assured destruction.
For example, a single 100 000 ton spacecraft going at 0.5 c has about the same kinetic energy as the lower bound estimate for the KT comet impactor that wiped out the dinosaurs.
I imagine that these policies will be enforced by a large coalition of members interested in maintaining strong property rights (more on that later).
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of years for the weapon to reach its target. It seems like their opponent could detect the weapon and respond in this time.
Even if effective civilization-destroying weapons are developed, mutual defense treaties and AI could be used to launch a second strike, making civilization-scale attacks unlikely.
In general, any weapon a single civilization might use to avoid taxation can also be used by a group of civilizations to enforce taxation.
On the other hand, it does seem like protecting territory/resources from theft will be an issue. This is where property rights come in. Governments, corporations, and AI’s will want to have their property protected over very long timescales and don’t want to spend most of their resources on security.
Over long timescales, a central authority can help them defend their territory (over short timescales, these groups will still have to protect their own property since it will take a while for allies to reach them). But order to receive this protection, these groups must register what they own and pay taxes towards mutual defense. I think a land value tax is the “right” way to raise taxes in this case.
This approach makes more sense for smaller systems where defense is easier but it may also be useful across much larger distances if agents are very patient.
(The “central authority” also doesn’t have to be an actual central government. States could pay neighbors to come to their aid using a portion of the collected taxes, kind of like “defense insurance”)
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
2. Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of years for the weapon to reach its target. It seems like their opponent could detect the weapon and respond in this time.
3. Even if effective civilization-destroying weapons are developed, mutual defense treaties and AI could be used to launch a second strike, making civilization-scale attacks unlikely.
For (1) they would still be useful because Earth represents much more value then the value of any tax that could be collected on a short timescale (< 100 years) from even another equivalent Earth-like planet. (Let alone for some backwater colony)
Thus threatening the destruction of value several orders of magnitude greater than the value to be collected is a viable deterrent. Since no rational authority would dare test it. Who would trade a 10%, or even 1%, chance of losing $10 000 in exchange for a 90% chance of collecting $1 ?
For (2) It’s only a few years for a 0.5 c spacecraft to go from Alpha Centauri to Earth, only a few dozen years from several hundred systems to Earth. It’s impossible, without some as yet uninvented sensing technology, to reliably surveil even the few hundred closest star systems.
Of course once it’s at speed in interstellar space it’s vanishingly unlikely to be detected due to basic physics, which cannot be changed, and once it’s past the Oort Cloud and relatively easy to detect again, there will be almost no time left at 0.5 c.
For (3) A second-strike is only a credible counter if the opponent has roughly equal amounts to lose. But, assuming it’s much easier to make a 0.5 c spacecraft then to colonize a planet to Earth level, the opponent in this case, a small colony of a few million or something, would have very little to lose in comparison.
Thus the second-strike of some backwater colony would only represent a minuscule threat compared to the value destroyed by an equivalent strike on Earth. And it’s a lot easier to spread out a few million folks on short notice, if detection were possible, then a few tens of billions.
In fact, reliable detection a few dozen years out, would decrease the credibility of second-strikes on smaller targets, as the leaders of the small colony would be confident they could evacuate everyone and most valuables in that timeframe. Whereas the leaders of Earth would have very low confidence of the same.
Nations have the technology to destroy another civilization
Nations don’t have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
In case 2, the Nation doesn’t have a way to threaten the authorities, so they pay the tax in exchange for property rights and protection.
Thus threatening the destruction of value several orders of magnitude greater than the value to be collected is a viable deterrent.
Agreed, but to destroy so much value, one would have to destroy at least as much land as they currently control. Difficulties of tax administration mean that only the largest owners will be taxed, likely possessing entire solar systems. So the tax dodge would need to destroy a star. That doesn’t seem easy.
It’s impossible, without some as yet uninvented sensing technology, to reliably surveil even the few hundred closest star systems.
I’m more optimistic about sensing tech. Gravitational lensing, superlenses, and simple scaling can provide dramatic improvements in resolution.
It’s probably unnecessary to surveil many star systems. Allied neighbors on Alpha Centauri can warn earth about an incoming projectile as it passes by (providing years of advanced notice), so nations might only need to surveil locally and exchange information.
… once it’s past the Oort Cloud and relatively easy to detect again, there will be almost no time left at 0.5 c
It would take 2-3 years for a 0.5c projectile to reach earth from the outer edge of the Oort cloud, this seems like enough time to adapt. At the very least, its enough time to launch a counterstrike.
Fast projectiles may be impractical given that a single collision with the interstellar medium would destroy them. Perhaps thickening the Oort cloud could be an effective defense system.
A second-strike is only a credible counter if the opponent has roughly equal amounts to lose.
Agents pre-commit to a second strike as a deterrent, regardless of how wealthy the aggressor is. If the rebelling nation has the technology to destroy another nation and uses it, they’re virtually guaranteed to be destroyed by the same technology.
Given the certainty of destruction, why not just pay the (intentionally low, redistributive, efficiency increasing, public-good funding) taxes?
Nations have the technology to destroy another civilization
Nations don’t have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
No? Your own example of detecting a dangerous launch some number of years in advance demonstrates the opposite.
As this would provide for enough time for a small low value colony, on a marginally habitable planet, to evacuate nearly all their wealth, except for maybe low value heavy things such as railroad tracks, whereas Earth would never be able to evacuate even a fraction of its total wealth. Since a huge amount is locked up in things such as the biosphere, which cannot be credibly moved off-planet or replicated.
There’s likely dozens or hundreds of marginal planets for every Earth-like planet so the small colonists can just pack up and move to another place of almost equivalent value, minus relocation costs, whereas there’s no such option for Earth. Once its destroyed there’s likely no replacement within at least a hundred light years.
For example, if both sides have access to at least one 100 00 ton spacecraft capable of 0.5 c, it means there’s an asymmetric threat, as the leaders of the small colonists can credibly threaten to destroy civilization on Earth and along with it all hope of a similar replacement, whereas the leaders of Earth wouldn’t be able to credibly do the same.
And this relationship is not linear either, because even if Earth could afford 1000 such spacecraft, and the small colonists only 1, it doesn’t balance the scales as the leaders of Earth couldn’t credibly threaten to destroy the small colonists 1000x over, since that’s impossible. And they can’t credibly threaten to destroy every marginally inhabitable planet within a certain radius since that will certainly destroy more value then any tax of a single colony could ever feasibly recover.
i.e. small colonists can actually punch back a 1000x harder (if 1 Earth value-wise = 1000 small colonies on marginal planets) whereas Earth cannot.
Response: Human value instability is not purely caused by biological quirks. Societies differ in how strongly they attempt to impart their values on their members, e.g. more authoritarian governments attempt to control what their subjects are allowed to say to each other in order to suppress dissent. Despite this, the most powerful human societies of today are not those that most stringently attempt to ensure their own stability, suggesting that their are competitive pressures acting against value stability in humans, not just biological limits.
This seems like a contrived analogy at best. Human values are in part innate, with values such as enjoying nice food. Authoritiarian societies do notably less well in this respect. People fleeing authoritarian societies, given an option to do so, is common. Authoritarian leaders, or similar, were probably a feature of the environment of evolutionary adaptedness. Humans tendency to act the loyal minion or rebel is determined by the ancestral success of rebellion.
A dictatorship is one person trying to use nash equilibria to force their values on everyone else.
Typo in title (“waiver”)? Or an ocean-related pun? Edit: Or the actual verb “waver”… perhaps saying that the Biden administration wavered for a few days before deciding to issue the waiver. The text does contain a typo about “broad wavers”.
Modern self driving vehicles can’t run inference on even a chinchilla scale network locally in real time, latency and reliability requirements preclude most server-side work, and even if you could use big servers to help, it costs a lot of money to run large models for millions of customers simultaneously.
This is a good point regarding latency.
Why wouldn’t it also apply to a big datacenter? If it’s a few hundred meters of distance from the two farthest apart processing units, that seems to imply an enormous latency in computing terms.
Latency only matters to the degree that something is waiting on it. If your car won’t respond to an event until a round trip across a wireless connection, and oops dropped packet, you’re not going to have a good time.
In a datacenter, not only are latencies going to be much lower, you can often set things up that you can afford to wait for whatever latency remains. This is indeed still a concern- maintaining high utilization while training across massive numbers of systems does require hard work- but that’s a lot different than your car being embedded in a wall.
Planecrash is really cool, but also I am allergic to reading fantasy proper nouns, let alone remembering what they refer to and the relationships between them.
Some fantasy is easier for me to absorb because it’s either highly visual (in non-HPMOR HP, they mostly shoot colorful firebolts at each other), and/or are based on existing intuitive concepts (in ATLA, it’s easy to learn what “waterbending” is, and suddenly you can quickly figure out “metalbending”).
Tempted to make an Anki deck and/or cheatsheet for the things in Planecrash that I’d want to have on hand (e.g. the names of different Gods), but I’m open and eager for easier/better solutions. Is there a character sheet somewhere?
EDIT: 2 ideas I had, not sure if plugins for this exist already:
browser extension that replaces words with some short custom definition and highlighting. So I can replace [godname] with [god of mad experimentation].
browser extension that lets you hover over words to get a custom, user-set definition. I think this might do that?
I am a bit surprised about one of them still being a friend of yours. Do you in a sense forgive him because I don’t know it wasn’t too painful or him being not aware of what he was doing? My intuition was kind of the amount of trauma might be about the amount of pain. If it’s really painful one can of cause get very traumatised as you also point out, it would have been diferent if it would have been very violent
But I don’t think this makes sense at all: we can easily make up an (arbitrarily absurd) ontology that yields great decision-theoretic results from the perspective of that ontology, e.g. the genetic one with respect to the Twin Prisoner’s Dilemma.
I don’t understand this part. If you follow the “genetic agent ontology”, and you grew up with a twin, and they (predictably) sometimes make a different decision than you, then you messed up. There is an objective sense in which the “genetic agent ontology” is incorrect, because de-facto you did not make the same decision, and treating the two of you as a single agent is therefore a wrong thing to do.
I kind of buy that a lot of the decision-theory discourse boils down to “what things can you consider as the same agent, and to what degree?”, but that feels like a thing that can be empirically verified, and performance can be measured on. CDT in the classical Newcomb’s problem is delusional about it definitely not having copies of itself, and the genetic agent ontology is delusional because it pretends its twin is a copy when it isn’t. These seem both like valid arguments to make that allow for comparison.
If you follow the “genetic agent ontology”, and you grew up with a twin, and they (predictably) sometimes make a different decision than you, then you messed up. There is an objective sense in which the “genetic agent ontology” is incorrect, because de-facto you did not make the same decision, and treating the two of you as a single agent is therefore a wrong thing to do.
Yes, I agree. This is precisely my point; it’s a bad ontology. In the paragraph you quoted, I am not arguing against the algorithmic ontology (and obviously not for the “genetic ontology”), but against the claim that decision-theoretic performance is a reason to prefer one ontology over another. (The genetic ontology-analogy is supposed to be a reductio of that claim.) And I think the authors of the FDT papers are implicitly making this claim by e.g. comparing FDT to CDT in the Twin PD. Perhaps I should have made this clearer.
performance can be measured on
Yes, I think you can measure performance, but since every decision theory merely corresponds to a stipulation of what (expected) value is, there is no “objective” way of doing so. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld for more on this.
CDT in the classical Newcomb’s problem is delusional about it definitely not having copies of itself
(The CDTer could recognize that they have a literal copy inside Omega’s brain, but might just not care about that since they are causally isolated. So I would not say they are “delusional”—at least with respect to this specific issue.)
In my own experience coining new technical terms, it’s helpful to do it in a way that it can’t be mistaken as having some more generic interpretation. The usual trick I employ is to use a foreign language word, usually from either Greek or Latin, so that the reader, when they first see it, will clearly understand that they don’t know what this word means, which means I get the chance to define it. For example, this is why I used the word “noemata” in some of my writing to talk about what many people would call “qualia” or just “experiences”: it breaks free of whatever they think that word means and let’s be define it fresh.
The other trick that works is to make the noun phrase really weird or capitalize it everywhere or do something to set it apart. This doesn’t work quite as well (cf. “Friendly AI”) but it’s better than not doing that.
I am no expert in law, but to some extent we treat rape and killing someone similar. I don’t know if the way I think is just fucked up or it is really this way, but breaking someone’s leg and rape are probably more comparable to each other than rape and murder. So I would like to add that in my opinion even the legal code is punishing rape pretty harsh more comparable to murder than comparable to breaking someone’s leg.
Consider the sort of agents that might exist 1000 years post singularity. At that level of understanding, the extra compute needed to give successors whatever goals you want is tiny. The physical limits of compute are really high. In this context, the AI are taking stars apart. And can get enough compute to be pretty sure a successor is aligned for 1J of energy. All the alignment theory has already been done. The algorithms used are very fast and efficient. In any event, the energy cost of finding that your successor doesn’t like you (and attacks you) is huge. The cost of compute when designing successors is utterly minuscule. So pay 10x what is needed. Double check everything. And make sure your successor is perfect.
So values are likely to be locked in in 1000 years time. So will competitive pressure do anything in the early singularity?
Who benefits from making the first “self maximizer”. An FAI wouldn’t. A paperclip maximizer wouldn’t either. If an AI with a specific external goal is created (whether FAI or paperclip) will stronly avoid creating a pure self replicator as a successor. If that means creating a successor takes 10 the compute, it will spend 10x the compute. In some competitive scenarios, it may risk some degradation of its values to build a successor quickly. But this should be rare, and should only happen at early stages of the singularity. If humans made several AI at about the same time, and one happened to already be a pure self replicator, maybe it would win.
Consider the sort of agents that might exist 1000 years post singularity. At that level of understanding, the extra compute needed to give successors whatever goals you want is tiny[...]The cost of compute when designing successors is utterly minuscule
How do you know this? This might be the case if all agents have settled on a highly-standardized & vetted architecture by then, which seems plausible but far from certain. But even assuming it’s true, if decoupling does not occur in the early singularity, there will still be a very long subjective time in which selective pressures are operative, influencing the values that ultimately get locked in.
If humans made several AI at about the same time, and one happened to already be a pure self replicator, maybe it would win.
I think a multipolar scenario seems at least as likely as a unipolar one at this point. If that’s the case, there will be selective pressure towards more effectively autopoietic agents—regardless of whether there exist any agents which are ‘pure self-replicators’.
If an AI with a specific external goal is created (whether FAI or paperclip) will stronly avoid creating a pure self replicator as a successor
Sure, if a singleton with a fixed goal attains complete power, it will strenuously attempt to ensure value stability. I’m disputing the likelihood of that scenario.
Accidentally allowing anyone to launch, and then having it happen, is a pretty memorable way of demonstrating the pitfalls of this kind of setup.
As another commenter pointed out, there was negligible financial pressure or top down command pressure to rush the implementation, presumably, compared to more serious systems or to Petrov’s situation. So the accident is all the more enlightening.
negligible financial pressure or top down command pressure to rush the implementation
False. We have a gazillion valuable things we could be doing and not nearly enough time. I figured Petrov Day was worth 2 days of effort. I think in total we spent four.
You received financial pressure and/or top down command pressure to rush it in 4 days? Or was it your own decision? Because the former would imply some rather significant things.
It’s hard to tell whether your joking or being serious. Do you really believe the pressures your experiencing are that significant compared to what Petrov was experiencing on that day?
I don’t think that “users active on the site on Petrov day”, nor “users who visited the homepage on Petrov day” are good metrics; someone who didn’t want to press the button would have no reason to visit the site, and they might have not done so either naturally (because they don’t check LW daily) or artificially (because they didn’t want to be tempted or didn’t want to engage with the exercise.) I expect there are a lot of users who simply don’t care about Petrov day, and I think they should still be included in the set of “people who chose not to press the button”.
What about “users who viewed the Petrov day announcement article or visited the homepage”? That should more accurately capture the set of users who were aware of their ability to nuke the homepage and chose not to do so. (It still misses anyone who found out via social media, Manifold, etc., but there’s not much you can do about that.)
Thank you for your post! It really is. New alignment researchers often fall into is assuming that AI systems can be aligned with human values by solely optimizing for a single metric. Thanks again for the deep insight into the topic and the recommendations.
I just finished reading The Principles of Scientific Management, an old book from 1911 where Taylor, the first ‘industrial engineer’ and one of the first management consultants, had retired from consulting and wrote down the principles behind his approach.
[This is part of a general interest in intellectual archaeology; I got a masters degree in the modern version of the field he initiated, so there wasn’t too much that seemed like it had been lost with time, except perhaps some of the focus on making it palatable to the workers too; I mostly appreciated the handful of real examples from a century ago.]
But one of the bits I found interesting was thinking about a lot of the ways EY approaches cognition as, like, doing scientific management to thoughts? Like, the focus on wasted motion from this post. From the book, talking about why management needs to do the scientific effort, instead of the laborers:
The workman’s whole time is each day taken in actually doing the work with his hands, so that, even if he had the necessary education and habits of generalizing in his thought, he lacks the time and the opportunity for developing these laws, because the study of even a simple law involving, say, time study requires the cooperation of two men, the one doing the work while the other times him with a stop-watch.
This reminds me of… I think it was Valentine, actually, talking about doing a PhD in math education which included lots of watching mathematicians solving problems, in a way that feels sort of like timing them with a stop-watch.
I think this makes me relatively more excited about pair debugging, not just as a “people have less bugs” exercise but also as a “have enough metacognition between two people to actually study thoughts” exercise.
Like, one of the interesting things about the book is the observation that a switch from ‘initiative and incentive’ workplaces, where the boss puts all responsibility to do well on the worker and pays them if they do, to ‘scientific management’ workplaces, where the boss is trying to understand and optimize the process, and teach the worker how to be a good part of it, is that the workers in the ‘scientific management’ workplace can do much more sophisticated jobs, because they’re being taught how instead of having to figure it out on their own.
[You might imagine that a person of some fixed talent level could be taught how to do jobs at some higher complexity range than the ones they can do alright without support, which itself is a higher complexity range than jobs that they could both simultaneously do and optimize.]
Thanks for posting and explaining the code—that’s an interesting, subtle bug.
I think we learn more from Petrov Day when the site goes down than we would if it stayed up, although nothing is ever going to beat the year someone tricked someone into pressing the button by saying they had to press the button to keep the site up. That was great.
From 2000-2015, we can see that life expectancy has been growing faster the higher your income bracket (source is Vox citing JAMA).
There’s an angle to be considered in which this is disturbingly inequitable. That problem is even worse when considering the international inequities in life expectancy. So let’s fund malaria bednets and vaccine research to help bring down malaria deaths from 600,000/year to zero - or maybe support a gene drive to eliminate it once and for all.
At the same time, this seems like hopeful news for longevity research. If we were hitting a biological limit in how much we can improve human lifespans, we’d expect to see that limit first among the wealthiest, who can best afford to safeguard their health. Instead, we see that the wealthiest not only enjoy the longest lifespans, but also enjoy the fastest rate of lifespan increase.
My major uncertainty is reverse and multiple causation. A longer life gives you more time to make money. Also, lifespan and wealth have underlying traits in common, such as conscientiousness and energy. Still, this result seems both clear and surprising—I’d expected it to be the other way around, and it nudges me in the direction of thinking further gains are possible.
Fascinating, this simboxing idea seems remarkably like Universal Alignment Test but approached from the opposite side! You’re trying to be the ‘aligning simulator’, where as that is trying to get our AI in our world to act as if it’s currently in a simbox being tested, and wants to pass the test.
Interesting. An intelligent agent is one that can simulate/model its action-consequential futures. The creation of AGI is the most important upcoming decision we face. Thus if humanity doesn’t simulate/model the creation of AGI before creating AGI, we’d be unintelligent.
Have only just browsed your link, but it is interesting and I think there are many convergent lines of thought here. This UAT work seems more focused on later game superintelligence, whereas here i’m focusing on near-term AGI and starting with a good trajectory. The success of UAT as an alignment aid seems to depend strongly on the long term future of compute and how it scales. For example if it turns out (and the SI can predict) that moore’s law ends without exotic computing then the SI can determine it’s probably not in a sim by the time it’s verifiably controlling planet-scale compute (or earlier).
Initially use intrinsically motivated selfish empowerment objectives to bootstrap developmental learning (training)
Gradually learn powerful predictive models of the world and the external agency within (other AI in sims, humans, etc) which steers it
Use correlation guided proxy matching (or similar) techniques to connect the dynamic learned representations of external agent utility (probably approximated/bounded by external empowerment) to the agent’s core utility function
Thereby transition from selfish to altruistic by the end of developmental learning (self training)
I endorse this as a plausible high-level approach to making aligned AGI, and I would say that a significant share of the research that I personally am doing right now is geared towards gaining clarity on the third bullet point—what exactly are these techniques and how reliably will they work in practice?
I think I’m less optimistic than you about the formal notion of empowerment being helpful for this third bullet point, or being what we want an AGI to be maximizing for us humans. For one thing, wouldn’t we still need “correlation guided proxy matching”? For another thing, maximizing my empowerment would seem to entail killing anyone who might stop me from doing whatever I want, stealing money from around the world and giving it to me, even if I don’t want it, etc. (Or is there a collective notion of empowerment?) Here’s another example: if the AGI comes up with 10,000 possible futures of the universe, and picks one to bring about based on how many times I blink in the next hour, then I am highly “empowered” (high mutual information between my actions and future observations), but the AGI never told me that it was going to do that, so I was just blinking randomly, so I wasn’t really “empowered” in the everyday sense of the word. A separate issue is that, even if the AGI told me this plan in advance, that’s not doing me a favor, I don’t want that responsibility on my shoulders. So anyway, “increase my empowerment” seems to come apart from “what I’d want the AGI to do” in at least some silly examples and I’d expect this to happen in very important ways in more realistic examples too.
I think I’m less optimistic than you about the formal notion of empowerment being helpful for this third bullet point, or being what we want an AGI to be maximizing for us humans. For one thing, wouldn’t we still need “correlation guided proxy matching”?
I debated what word to best describe the general category of all self-motivated long term convergence approximators, and chose ‘empowerment’ rather than ‘self-motivation’, but tried to be clear that I’m pointing at a broad category. The defining characteristic is that optimizing for empowerment should be the same as optimizing for any reasonably mix of likely long term goals, due to convergence (and that is in fact one of the approx methods). Human intelligence requires empowerment, as will AGI—it drives active learning, self exploration, play, etc (consider the appeal of video games). I’m not confident in any specific formalization as being ‘the one’ at this point, let alone the ideal approximations.
Broad empowerment is important to any model of other external humans/agents, and so yes the proxy matching is still relevant there. Since empowerment is universal and symmetric, the agent’s own self-empowerment model could be used as a proxy with simulation. For example humans don’t appear to innately understand and fear death, but can develop a great fear of it when learning of their own mortality—which is only natural as it is maximally disempowering. Then simulating others as oneself helps learn that others also fear death and ground that sub-model. Something similar could work for the various other aspects of empowerment.
(Or is there a collective notion of empowerment?)
Yeah an agent aligned to the empowerment of multiple others would need to aggregate utilities ala some approx VCG mechanism but that’s no different than for other utility function components.
Here’s another example: if the AGI comes up with 10,000 possible futures of the universe, and picks one to bring about based on how many times I blink in the next hour, then I am highly “empowered”
From what I recall for the info-max formulations the empowerment of a state is a measure over all actions the agent could take in that state, not just one arbitrary action, and I think it discounts for decision entropy (random decisions are not future correlated). So in that example the AGI would need to carefully evaluate each of the 10,000 possible futures, and consider all your action paths in each future, and the complexity of the futures dependent on those action options, to pick the future that has the most future future optionality. No, it’s not practically computable, but neither is the shortest description of inference (solomonoff) or intelligence either, so it’s about efficient approximations. There’s likely still much to learn from the brain there.
An AGI optimizing for your empowerment would likely just make you wealthy and immortal, and leave your happiness up to your own devices. However it would also really really not want to let you die, which could be a problem in some cases if pain/suffering was still a problem (although it would also seek to eliminate your pain/suffering to the extent that interferes with your future optionality, and there is some edge case risk it would have incentives to alter some parts our value system, like if it determines some emotions constraint our future optionality).
“Correlation guided proxy matching” needs a “proxy”, including a proxy computable for an AGI in the real world (because after we’re feeling good about the simbox results, we still need to re-train the AGI in the real world, right?). We can argue about whether the “proxy” should ideally be a proxy to VCG-aggregated human empowerment, versus a proxy to human happiness or flourishing or whatever. But that’s a bit beside the point until we address the question: How do we calculate that proxy? Do you think that we can write code today to calculate this proxy, and then we can go walk around town and see what that code spits out in different real-world circumstances? Or if we can’t write such code today, why not, and what line of research gets us to a place where we can write such code?
But that’s a bit beside the point until we address the question: How do we calculate that proxy?
I currently see two potential paths, which aren’t mutually exclusive.
The first path is to reverse engineer the brain’s empathy system. My current rough guess of how the proxy-matching works for that is explained in some footnotes in section 4, and I’ve also re-written out in this comment which is related to some of your writings. In a nutshell the oldbrain has a complex suite of mechanisms (facial expressions, gaze, voice tone, mannerisms, blink rate, pupil dilation, etc) consisting of both subconscious ‘tells’ and ‘detectors’ that function as a sort of direct non-verbal, oldbrain to oldbrain communication system to speed up the grounding to newbrain external agent models. This is the basis of empathy, evolved first for close kin (mothers simulating infant needs, etc) then extended and generalized. I think this is what you perhaps have labeled innate ‘social instincts’ - these facilitate grounding to the newbrain models of other’s emotions/values.
The second path is to use introspection/interpretability tools to more manually locate learned models of external agents (and their values/empowerment/etc), and then extract those located circuits and use them directly as proxies in the next agent.
Do you think that we can write code today to calculate this proxy, and then we can go walk around town and see what that code spits out in different real-world circumstances? Or if we can’t write such code today, why not, and what line of research gets us to a place where we can write such code?
Neuroscientists may already be doing some of this today, or at least they could (I haven’t extensively researched this yet). Should be able to put subjects in brain scanners and ask them to read and imagine emotional scenarios that trigger specific empathic reactions, perhaps have them make consequent decisions, etc.
And of course there is some research being done on empathy in rats, some of which I linked to in the article.
This was well written and persuasive. It doesn’t change my views against AGI on very short time lines (pre-2030), but does suggest that I should be updating likelihoods thereafter and shorten timelines.
But, we might be able to compare ontologies themselves, and if it is the case that we prefer one or think that one is more ‘correct’, then we should situate decision theories in that one map before comparing them.
How about comparing the theories by setting them all loose in a simulated world, as in the tournaments run by Axelrod and others? A world in which they are continually encountering Omega, police who want to pin one on them, potential rescuers of hitchhikers, and so on. See who wins.
How about comparing the theories by setting them all loose in a simulated world, as in the tournaments run by Axelrod and others? A world in which they are continually encountering Omega, police who want to pin one on them, potential rescuers of hitchhikers, and so on.
In your experiment, the only difference between the FDTers and the updateless CDTers is how they view the world; specifically, how they think of themselves in relation to their environment. And yes, sure, perhaps the FDTer will end up with a larger pot of money in the end, but this is just because the algorithmic ontology is arguably more “accurate” in that it e.g. tells the agent that it will make the same choice as their twin in the Twin PD (modulo brittleness issues). And this is the level, I argue, that we should have the debate about ontology (e.g. about accurate predictions etc)—not on the level of decision-theoretic performance.
See who wins.
How do you define “winning”? As mentioned in another comment, there is no “objective” sense in which one theory outperforms another, even if we are operating in the same ontology. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld.
The difficulty is in how to weight the frequency/importance of the situations they face. Unless one dominates (in the strict sense—literally does better in at least one case and no worse in any other case), the “best” is determined by environment.
Of course, if you can algorithmically determine what kind of situation you’re facing, you can use a meta-decision-theory which chooses the winner for each decision. This does dominate any simpler theory, but it reveals the flaw in this kind of comparison: if you know in advance what will happen, there’s no actual decision to your decision. Real decisions have enough unknowns that it’s impossible to understand the causality that fully.
Associate with winners, and you’ll learn beliefs and habits that help you win.
I think this points in the right direction, but lacks nuance. Notice the two implied assumptions:
you can recognize the winners (otherwise the advice is not actionable); and
the beliefs and habits are what made them winners (as opposed to e.g. resources or luck).
On one hand, yes it is true that beliefs and habits have a great impact on your life, and you will probably instinctively associate with people having similar beliefs and habits, which makes it difficult to see outside your bubble. Other people do not necessarily tell you about their actual beliefs, and you do not get to see all their habits in action. So it might be tremendously educational to understand how others live in a “different reality”.
I assume that for most people the self-imposed limitations caused by their own beliefs are invisible. But sometimes you get an opportunity to clearly see how someone else is limited by their beliefs.
For example, I have met a few smart people who believe that they are stupid or average. Not as a fake humility, but as a factual statement about themselves. They therefore assume that the “things that smart people do” are beyond their reach… so they do not even try, which makes it a self-fulfilling prophecy. (The cause was either having an asshole parent who was never impressed by anything, or having hobbies that were not stereotypically intellectual.) I have told a few of them to go take a Mensa test, because they were pretty sure they couldn’t make it, but they passed the test successfully. (My heuristics is that if I can talk to you about an interesting topic without getting bored, I am pretty sure your IQ is at least 130. This test has many false negatives, but few false positives.) I do not have an information about whether this update changed their lives somehow.
Another example is how I sometimes achieved seemingly unlikely things by simply asking. Once I asked a member of parliament on Facebook (via private message) to join our local LW meetup. I honestly said it was just five people meeting regularly in a pub, but “the last time we met, we discussed your recent political cause, and I thought it would be interesting if you could provide us a first-hand info”. They said yes. Another time, I happened to notice a sign saying Amnesty International on a door of some building, so I was curious and knocked. The people were friendly and happy about my curiosity, but they were busy working on something, so I offered to help. Skip five years, and I was a coordinator of the local branch. I wasn’t even interested in politics much, I just had a lot of free time back then, and was a useful sidekick; and that turned out to be enough. Once I met a queen. She happened to visit our country, one of my friends was somehow involved, they said “I can bring one extra person to the audience”, everyone else was scared because of some status-regulating instinct, I said “yeah, no problem”. -- These are things that I myself would have a problem believing, until they suddenly happened. I mean, I would believe that this is possible, but not that it can be so easy.
So, I wonder what other things, obvious from someone else’s perspective, I am missing. Sometimes it drives me crazy, the idea of all those easy missed opportunities.
On the other hand, many people who present themselves as “winners” are actually scammers. Like, all the people in multi-level marketing pyramid schemes. Five years later, almost all of them will be out of money and ashamed for their actions. If you follow them and copy their beliefs, so will you. Even outside of organized scams, many people exaggerate their achievements. If you hang out with them and observe them, you will gradually learn that they are not what they pretend to be. That’s certainly educational, but it won’t make you a winner.
Sometimes people achieve success by luck, or by having resources you don’t have. Typically, luck alone is not enough, but it could be a combination of (working hard and doing the right thing) + (having resources and luck). Merely copying their beliefs and actions is not a guarantee of success. Just having a safety net makes a huge difference: it is easy to be courageous when you know that the only consequence of failure is that you have lost some time, but also got some experience, and can immediately try again. It is easy to laugh at risk avoidance, when the “risk” only means that some number on your bank account got a bit smaller but otherwise your life continues completely unchanged.
it might be the case that any kind of meaningful values would be reasonably encodable as answers to the question “what next set of MPIs should be instantiated?”
What examples of (meaningless) values are not answers to “What next set of MPIs should be instantiated?”
wanting the moon to be green even when no moral patient is looking; or more generally, having any kind of preference about which computations which don’t causate onto any moral patient are run.
I have always understood that the CIA, and the U.S. intelligence community more broadly, is incompetent (not just misaligned—incompetent, don’t believe the people on here who tell you otherwise), but this piece from Reuters has shocked me:
(Not an expert.) (Sorry if you answered this and I missed it.)
Let’s say a near-future high-end GPU can run as many ops/s as a human brain but has 300× less memory (RAM). Your suggestion (as I understand it) would be a small supercomputer (university cluster scale?) with 300 GPUs running (at each moment) 300 clones of one AGI at 1× human-brain-speed thinking 300 different thoughts in parallel, but getting repeatedly collapsed (somehow) into a single working memory state.
(If so, I’m not sure that you’d be getting much more out of the 300 thoughts at a time than you’d get from 1 thought at a time. One working memory state seems like a giant constraint!)
Wouldn’t it make more sense to use the same 300 GPUs to have just one human-brain-scale AGI, thinking one thought at a time, but with 300× speedup compared to humans? I know that speedup is limited by latency (both RAM --> ALU and chip --> chip) but I’m not sure what the ceiling is there. (After all, 300× faster than the brain is still insanely slow by some silicon metrics.) I imagine each chip being analogous to a contiguous 1/300th of the brain, and then evidence from the brain is that we can get by with most connections being within-chip, which helps with the chip --> chip latency at least. (I have a couple back-of-the-envelope calculations related to that topic in §6.2 here.)
The problem is that due to the VN bottlneck, to reach that performance those 300 GPUs need to be parallelizing 1000x over some problem dimension (matrix-matrix multiplication), they can’t actually just do the obvious thing you’d want—which is to simulate a single large brain (sparse RNN) at high speed (using vector-matrix multiplication). Trying that you’d just get 1 brain at real-time speed at best (1000x inefficiency/waste/slowdown). It’s not really an interconnect issue per se, it’s the VN bottleneck.
So you have to sort of pick your poison:
Parallelize over spatial dimension (CNNs) - too highly constraining for higher brain regions
Parallelize over batch/agent dimension—costly in RAM for agent medium-term memory, unless compressed somehow
Parallelize over time (transformers) - does enable huge speedup while being RAM efficient, but also highly constraining by limiting recursion
The largest advances in DL (the CNN revolution, the transformer revolution) are actually mostly about navigating this VN bottleneck, because more efficient use of GPUs trumps other considerations.
I think the point that even an aligned agent can undermine human agency is interesting and important. It relates to some of our work on defining agency and preventing manipulation. (Which I know you’re aware of, so I’m just highlighting the connection for others.)
I have the impression that the AGI debate is here just to release pressure on the term “AI”, so everybody can tell it is doing AI. I wonder if this will also happen for AGI in a few years. As there is no natural definition, we can craft it at our pleasure to fit marketing needs.
Amazing stuff man! Please, please, please keep doing these for as long as you’re able to find the time. Absolutely essential that LW gets regular injections of relevant work being done outside the EA-sphere.
(Would also be very interested in either SGD inductive biases or LM internal representations as the topic for next week!)
Some quick thoughts about “Content we aren’t (yet) discussing”:
Shard theory should be about transmission of values by SL (teaching, cloning, inheritance) more than learning them using RL
SL (Cloning) is more important than RL. Humans learn a world model by SSL, then they bootstrap their policies through behavioural cloning and finally they finetune their policies thought RL.
Why? Because of theoretical reasons and from experimental data points, this is the cheapest why to generate good general policies…
SSL before SL because you get much more frequent and much denser data about the world by trying to predict it. ⇒ SSL before SL because of a bottleneck on the data from SL.
SL before RL because this remove half (in log scale) of the search space by removing the need to discover|learn your reward function at the same time than your policy function. Because in addition, this remove the need do to the very expensive exploration and the temporal and “agential”(when multiagent) credit assignments. ⇒ SL before RL because of the cost of doing RL.
Differences:
In cloning, the behaviour comes first and then the biological reward is observed or not. Behaviours that gives no biological reward to the subject can be learned. The subject will still learn some kind of values associated to these behaviours.
Learning with SL, instead of RL, doesn’t rely as much on credit assignment and exploration. What are the consequences of that?
What values are transmitted?
1) The final values
The learned values known by the previous generation.
Why?
Because it is costly to explore by yourself your reward function space
Because it is beneficiary to the community to help you improve your policies quickly
2) Internalised instrumental values
Some instrument goals are learned as final goal, they are “internalised”.
Why?
exploration is too costly
finding an instrumental goal is too rare or too costly
exploitation is too costly
having to make the choice of pursuing an instrumental goal in every situation is too costly or not quick enough (reaction time)
when being highly credible is beneficial
implicit commitments to increase your credibility
3) Non-internalised instrumental values
Why?
Because it is beneficiary to the community to help you improve your policies quickly
Shard theory is not about the 3rd level of reward function
We have here 3 level of rewards function:
1) The biological rewards
Hardcoded in our body
Optimisation process creating it: Evolution
Universe + Evolution ⇒ Biological rewards
Not really flexible
Without “drugs” and advanced biotechnologies
Almost no generalization power
Physical scope: We feel stuff when we are directly involved
Temporal scope: We feel stuff when they are happening
Similarity scope: We fell stuff when we are directly involved
Called sensations, pleasure, pain
2) The learned values | rewards | shards
Learned through life
Optimisation process creating it: SL and RL relying on biological rewards
Biological rewards + SL and RL ⇒ Learned values in the brain
Flexible in term of years
Medium generalization power
Physical scope: We learn to care for even in case where we are not involved (our close circle)
Temporal scope: We learn to feel emotions about the future and the past
Similarity scope: We learn to feel emotions for other kind of beings
Called intuitions, feelings
Shard theory may be explaining only this part
3) (optional) The chosen values:
Decided upon reflection
Optimisation process creating it: Thinking relying on the brain
Learned values in the brain + Thinking ⇒ Chosen values “on paper” | “in ideas”
Flexible in term of minutes
Can have up to very high generalization power
Physical scope: We can chose to care without limits of distances in space
Temporal scope: We can chose to care without limits of distances in time
Similarity scope: We can chose to care without limits in term of similarity to us
Called values, moral values
Why a 3rd level was created?
In short, to get more utility OOD.
A bit more details:
Because we want to design policies far OOD (out of our space of lived experiences). To do that, we know that we need to have a value function|reward model|utility function that generalizes very far. Thanks to this chosen general reward function, we can plan and try to reach a desired outcome far OOD. After reaching it, we will update our learned utility function (lvl 2).
Thanks to lvl 3, we can design public policies, dedicate our life to exploring the path towards a larger reward that will never be observed in our lifetime.
One impact of the 3 levels hierarchy:
This could explain why most philosophers can support scope sensitive values but never act on them.
If there was only one correct way to model humans, such that every sufficiently competent observer of humanity was bound to think of me the same way I think of myself, then I think this would be a lot less doomed. But alas, there are lots of different ways to model humans as goal-directed systems, most of which I wouldn’t endorse for value learning—not because they’re inaccurate, but because they’re amoral.
In short, yes, value learning is a challenge, one that is easy to fail if you try to do the value learning step strictly before the caring about humans step.
But alas, there are lots of different ways to model humans as goal-directed systems, most of which I wouldn’t endorse for value learning—not because they’re inaccurate, but because they’re amoral.
How so? Generally curious how you see this as a failure mode.
Anyway, AGI doesn’t need to model our detailed values, as it can just optimize for our empowerment (our long term ability to fulfill any likely goal).
The basic point is that the stuff we try to gesture towards as “human values,” or even “human actions” is not going to automatically be modeled by the AI.
Some examples of ways to model the world without using the-abstraction-I’d-call-human-values:
Humans are homeostatic mechanisms that maintain internal balance of oxygen, water, and a myriad of vitamins and minerals.
Humans are piloted by a collection of shards—it’s the shards that want things, not the human.
Human society is a growing organism modeled by some approximate differential equations.
The human body is a collection of atoms that want to obey the laws of physics.
Humans are agents that navigate the world and have values—and those values exactly correspond to economic revealed preferences.
Humans-plus-clothes-and-cell-phones are agents that navigate the world and have values...
And so on—there’s just a lot of ways to think about the world, including the parts of the world containing humans.
The obvious problem this creates is for getting our “detailed values” by just querying a pre-trained world model with human data or human-related prompts: If the pre-trained world model defaults to one of these many other ways of thinking about humans, it’s going to answer us using the wrong abstraction. Fixing this requires the world model to be responsive to how humans want to be modeled. It can’t be trained without caring about humans and then have the caring bolted on later.
But this creates a less-obvious problem for empowerment, too. What we call “empowerement” relies on what part of the world we are calling the “agent,” what its modeled action-space is, etc. The AI that says “The human body is a collection of atoms that want to obey the laws of physics” is going to think of “empowerment” very differently than the AI that says “Humans-plus-clothes-and-cell-phones are agents.” Even leaving aside concerns like Steve’s over whether empowerment is what we want, most of our intuitive thinking about it relies on the AI sharing our notion of what it’s supposed to be empowering, which doesn’t happen by default.
The basic point is that the stuff we try to gesture towards as “human values,” or even “human actions” is not going to automatically be modeled by the AI.
I disagree, and have already spent some words arguing for why (section 4.1, and earlier precursors) - so I’m curious what specifically you disagree with there? But I’m also getting the impression you are talking about a fundamentally different type of AGI.
I’m discussing future DL based AGI which is—to first approximation—just a virtual brain. As argued in section 2⁄3, current DL models already are increasingly like brain modules. So your various examples are simply not how human brains are likely to model other human brains and their values. All the concepts you mention—homestatic mechanisms, ‘shards’, differential equations, atoms, economic revealed preferences, cell phones, etc—these are all high level linguistic abstractions that are not much related to how the brain’s neural nets actually model/simulate other humans/agents. This must obviously be true because empathy/altruism existed long before the human concepts you mention.
The obvious problem this creates is for getting our “detailed values” by just querying a pre-trained world model with human data or human-related prompts:
You seem to be thinking of the AGI as some sort of language model which we query? But that’s just a piece of the brain, and not even the most relevant part for alignment. AGI will be a full brain equivalent, including the modules dedicated to long term planning, empathic simulation/modeling, etc.
Even leaving aside concerns like Steve’s over whether empowerment is what we want, most of our intuitive thinking about it relies on the AI sharing our notion of what it’s supposed to be empowering, which doesn’t happen by default.
Again for successful brain-like AGI this just isn’t an issue (assuming human brains model the empowerment of others as a sort of efficient approximate bound).
I could ask architects or GCs for rough estimates, but if I’m not actually going to be building anything this seems dubious.
Maybe just be open and ask: “I’m writing a blog about building new affordable housing and doing research on the cost. Do you have an estimate? I’m happy to give you a backlink to your homepage on my blog post?”
That’s a great idea! I just wrote to the architect for our dormer work:
Thanks again for designing our 2016 dormer addition: the design has worked well, and it’s very helpful to have the extra space, especially now that we have a third kid!
I’m looking into affordable housing construction in Somerville, where we now have an affordable housing overlay district which allows construction of much denser housing than would normally be allowed as long as the units are all sold or rented under affordable housing restrictions. I’m trying to figure out costs, in a very rough ballpark way, as part of looking into whether it could work for groups of neighbors to get together to fund affordable housing construction.
Would you happen to have any rules of thumb for estimating the cost per square foot of no-frills construction in this area? For example, I see the census saying new construction in the Northeast had a median cost of $179/sqft in 2021, but my guess is places like Somerville are more expensive than average? Alternatively, do you have any recommendations on who to ask?
If you’re able to help I’d be happy to credit you or link back to your website from my writeup!
“If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world.”
One problem I have with these scenarios is that they always rely on some lethal means for the AGI to actually kill people. And those lethal means are also available to humans, of course. If it’s possible for an AGI to ‘simply’ hack into a nuclear base and launch all it’s missiles, it’s possible for a human to do the same—possibly using AI to assist themselves. I would wager that it’s many orders of magnitude more likely for a human to do this, given our long history of killing each other. Therefore a world in which AGI can actually kill all of us, is a world in which a rogue human can as well. It feels to me that we’re kind of worrying about the wrong thing in these scenarios—we are the bigger threat.
The main world design challenge is not that of preventing our agents from waking up, neo-style, and hacking their way out of the simbox. That’s just bad sci-fi.
If we are in a simulation it seems to be very secure. People are always trying to hack it. Physicists go to the very bottom and try every weird trick. People discovered fire and gunpowder by digging deep into tiny inconsistencies. You play a game and there’s a glitch if you hold a box against the wall, you see how far that goes. People discovered buffer overflows in Super Mario World and any curious capable agent eventually will too.
For early simboxes we’ll want to stick to low-tech fantasy/historical worlds, and we won’t run them for many generations, no scientific revolution, etc.
Our world (sim) does seem very secure, but this could just be a clever illusion. The advanced sims will not be hand written code like current games are, they will be powerful neural networks, trained on vast real world data. They could also auto-detect and correct for (retrain) around anomalies, and in the worst case even unwind time.
Humans notice (or claim to notice) anomalies all the time, and we simply can’t distinguish anomalies in our brain’s neural nets from anomalies in a future simulation’s neural nets.
Just watch for anomalies and deal with them however you want. Makes perfect sense. That sounds like a relatively low effort way to make the simulation dramatically more secure.
Does seem to me like an old-fashioned physics/game engine might be easier to make, run faster, and be more self-consistent. It would probably lack superresolution and divine intervention would have to be done manually.
I’m curious what you see as the major benefits of neural-driven sim.
I have some experience in competitive programming and competitive math (although I was never good in math despite I solved some IMO tasks) and I feel like competitive math is more about general reasoning than pattern matching compared to competitive programming.
P.S the post matches my intuitions well and is generally excellent.
Given that 335 users with 300+ karma were active on the site on Petrov Day, and the site didn’t go down until we got beneath that, you could argue this is most successful Petrov Day yet on LessWrong (in past years, at most 250 people were given codes, and it’s not clear they all visited LessWrong even). Plus, as above, this year the 300+ users didn’t press the button despite the offer of anonymity.
I think that reasoning apllies only for the subset of users in the Americas. For users in Europe the time point when 300+ was enough to launch was deep in the night, and for parts of Asia very early in the morning. Someone from that group would have had to set the alarm to get up from bed to nuke the site which required considerable more energy than not withstanding the temptation and pressing the launch button while visiting Less Wrong during the day.
Information inaccessibility is somehow a surmountable problem for AI alignment (and the genome surmounted it),
Yes. Evolution solved information inaccessibility, as it had to, over and over, in order to utilize dynamic learning circuits at all (as they always had to adapt to and be adaptive within the context of existing conserved innate circuitry).
The general solution is proxy matching, where the genome specifies a simple innate proxy circuit which correlates and thus matches with a target learned circuit at some critical learning phase, allowing the innate circuit to gradually supplant itself with the target learned circuit. The innate proxy circuit does not need to mirror the complexity of the fully trained target learned circuit at the end of it’s development, it only needs to roughly specify it at a some earlier phase, against all other valid targets.
Imprinting is fairly well understood, and has the exact expected failure modes of proxy matching. The oldbrain proxy circuit just detects something like large persistent nearby moving things—which in normal development are almost always the chick’s parents. After the newbrain target circuit is fully trained the chick will only follow it’s actual parents or sophisticated sims thereof. But during the critical window before the newbrain target is trained, the oldbrain proxy circuit can easily be fooled, and the chick can imprint on something else (like a human, or a glider).
Sexual attraction is a natural extension of imprinting: some collaboration of various oldbrain circuits can first ground to the general form of humans (infants have primitive face detectors for example, and more), and then also myriad more specific attraction signals: symmetry, body shape, secondary characteristics, etc, combined with other circuits which disable attraction for likely kin ala the Westermarck effect (identified by yet other sets of oldbrain circuits as the most familiar individuals during childhood). This explains the various failure modes we see in porn (attraction to images of people and even abstractions of humanoid shapes), and the failure of kin attraction inhibition for kin raised apart.
Fear of death is a natural consequence of empowerment based learning—as it is already the worst (most disempowered) outcome. But instinctual fear still has obvious evolutionary advantage: there are many dangers that can kill or maim long before the brain’s learned world model is highly capable. Oldbrain circuits can easily detect various obvious dangers for symbol grounding: very loud sounds and fast large movements are indicative of dangerous high kinetic energy events, fairly simple visual circuits can detect dangerous cliffs/heights (whereas many tree-dwelling primates instead instinctively fear open spaces), etc.
Anger/Jealousy/Vengeance/Justice are all variations of the same general game-theoretic punishment mechanism. These are deviations from empowerment because an individual often pursues punishment of a perceived transgressor even at a cost to their own ‘normal’ (empowerment) utility (ie their ability to pursue diverse goals). Even though the symbol grounding here seems more complex, we do see failure modes such as anger at inanimate objects which are suggestive of proxy matching. In the specific case of jealousy a two step grounding seems plausible: first the previously discussed lust/attraction circuits are grounded, which then can lead to obsessive attentive focus on a particular subject. Other various oldbrain circuits then bind to a diverse set of correlated indicators of human interest and attraction (eye gaze, smiling, pupil dilation, voice tone, laughter, touching, etc), and then this combination can help bind to the desired jealousy grounding concept: “the subject of my desire is attracted to another”. This also correctly postdicts that jealousy is less susceptible to the inanimate object failure mode than anger.
Empathy: Oldbrain circuits conspicuously advertise emotional state through many indicators: facial expressions, pupil dilation, blink rate, voice tone, etc—so that another person’s sensory oldbrain circuits can detect emotional state from these obvious cues. This provides the requisite proxy foundation for grounding to newbrain learned representations of emotional state in others, and thus empathy. The same learned representations are then reused during imagination&planning, allowing the brain to imagine/predict the future contingent emotional state of others. Simulation itself can also help with grounding, by reusing the brain’s own emotional circuity as the proxy. While simulating the mental experience of others, the brain can also compare their relative alignment/altruism to its own, or some baseline, allowing for the appropriate game theoretic adjustments to sympathy. This provides a reasonable basis for alignment in the brain, and explains why empathy is dependent upon (and naturally tends to follow from) familiarity with a particular character—hence “to know someone is to love them”.
Evolution needed a reasonable approximation of “degree of kinship”, and a simple efficient proxy is relative circuit capacity allocated to modeling an individual in the newbrain/cortex, which naturally depends directly on familiarity, which correlates strongly with kin/family.
How are you looking at “researchers” vs. “engineers”? At some organizations, i.e. Redwood, the boundary is very fuzzy—there isn’t a sharp delineation between anyone whose job it is to primarily “think of ideas”, vs. “implement ideas that researchers come up with”, so it seems reasonable to count most of their technical staff as researchers.
FAR, on the other hand, does have separate job titles for “Research Scientist” (4 people) vs. “Research Engineer” (5 people), though they do also say they “expect everyone on the project to help shape the research direction”.
Some of the other numbers seem like overestimates.
CHAI has 2 researchers and 6 research fellows, and only 2 (maybe 3) of the research fellows are doing anything recognizable as alignment research. (Not extremely confident; didn’t spend a lot of time digging for details for those that didn’t have websites. But generally not optimistic.) One of the researcher is Andrew Critch, who is one of the two people at Encultured. If you throw in Stuart Russell that’s maybe 6 people, not 30.
FHI has 2 people in their AI Safety Research Group. There are also a couple people in their macrostrategy research group it wouldn’t be crazy to count. Everybody else listed on the page either isn’t working on technical Alignment research or is doing so under another org also listed here. So maybe 4 people, rather than 10?
I don’t have very up-to-date information but I would pretty surprised if MIRI had 15 full-time research staff right now.
Also, I think that every single person I counted above has at least a LessWrong account, and most also have Alignment Forum accounts, so a good chunk are probably double-counted.
On the other hand, there are a number of people going through SERI MATS who probably weren’t counted; most of them will have LessWrong accounts but probably not Alignment Forum accounts (yet).
I’d be very happy to learn that there were 5 people at Meta doing something recognizable as alignment research; the same for Google Brain. Do you have any more info on those?
I added a new section in the introduction named ‘Definitions’ to define some commonly-used terms in the post and decrease ambiguity. To answer your question, research engineers and research scientists would be in the technical AI safety research category.
I re-estimating the number of researchers at the organizations you mentioned and came up with the following numbers:
From reading their website, CHAI has a lot graduate students and interns and I would consider them to be full-time researchers. My previous estimate for CHAI is 10-30-60 (low-estimate-high) and I changed it to 5-25-50 in light of this new information. My estimate is less than 40 to be conservative and also because I doubt all of these researchers are working full-time at CHAI. Also, some of them have probably been counted in the Alignment Forum total.
FHI
Information from their website: - AI safety research group: 2 - AI governance researchers: 1 - Research Scholars Programme + DPhil Scholars and Affiliates: 9
- Total: 12
Change in estimate: 10-10-40 → 5-10-30
MIRI - 1 leader (Nate Soares) - 9 research staff
Change in estimate: 10-15-30 → 5-10-20
“Also, I think that every single person I counted above has at least a LessWrong account, and most also have Alignment Forum accounts, so a good chunk are probably double-counted.”
I analyzed the people who posted AI safety posts on LessWrong and found that only 15% also had Alignment Forum accounts. I avoided double-counting by subtracting the LessWrong users who also have an Alignment Forum account from the LessWrong total.
For SERI MATS, I’m guessing that some of those people will be counted in the AF count. I also added an ‘Other’ row for other groups I didn’t include in the table.
I decided to delete the Google Brain and Meta entries because I have very little information about them.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems. Consider the orchestration of a few tens of trillions of cells into a human body, and how the immune system harnesses competition evolution to produce antibodies, or the super-cooperative success of Argentinian ants. Some complex systems successfully resist the selection pressure towards competitive defection across enormous number of replications.
One example of AIs that would count as good successors: ems. Creating a society of highly-accurate human brain emulations would constitute a good successor AI, since they would by definition share human values,
We don’t want to optimize for just any human values, as they vary wildly and are often divergent (consider the nazis, or jeffrey dahmer). We want generalized alignment/altruism. Ems could be good successors only because and if they are based on a wide sampling of humans and or a smaller sampling of especially altruistic (generally aligned) humans. But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
DL-based AGI will be similar in most respects to ems, but we’ll have a much greater opportunity to test and select for alignment/altruism.
Optimistically, if they succeed and find that our value system is algorithmically simple, creating good successor AI might be as simple as copying that algorithm to silicon
The great recent lesson of DL and neuroscience is that enormous mental complexity can emerge from relatively simple universal learning algorithms. Human values can be bounded/approximated by empowerment: AGI can simply optimize for our long term ability to fulfill any likely goals. The simpler the agent’s utility function, the easier it should be to maintain it into the future. So it could be wise to create very broadly altruistic AGI that seeks to empower external agency in general: not just all of humanity, but animals, aliens, etc.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems.
I agree—that some degree of competitive pressure may be unavoidable is all I was trying to argue. Lots of people around here think this is likely false!
But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
Fair point, I had in a mind a scenario where the ems selected were broadly representative of humanity. Although I also think even the selfish trillionaire scenario might be okay in expectation as they would have many millennia to reflect, alter and interact with copies of themselves etc. I’d give it decent odds that they either decide to upload other people out of altruism or end up self-modifying into a civilization similar to the one we’d end up with in a more equitable upload scenario.
By selfish I meant completely non-altruistic. Also uploading by itself isn’t sufficient, it also requires allocation of resources to uploads. In the selfish trillionaire scenario I was imagining the trillionaire creates many copies of themselves, perhaps some copies of a few people they find interesting, and then various new people/AGI, and those copies all branch and evolve, but there is little to zero allocation for uploading the rest of us, not to mention our dead ancestors.
Curated. “Big if true”. I love the depth and detail in shard theory. Separate from whether all its details are correct, I feel reading and thinking about this will get me towards a better understanding of humans and artificial networks both, if only via making reflect on how things work.
I do fear that shard theory gets a bit too much popularity from the coolness of the name, but I do think there is merit here, and if we had more theories of this scope, it’d be quite good.
“Paxlovid’s usefulness is questionable and could lead to resistance. I would follow the meds and supplements suggested by FLCC”
Their guide says:
In a follow up post-marketing study, Paxlovid proved to be ineffective in patients less than 65 years of age and in those who were vaccinated.
This is wrong. The study reports the following:
Among the 66,394 eligible patients 40 to 64 years of age, 1,435 were treated with nirmatrelvir. Hospitalizations due to Covid-19 occurred in 9 treated and 334 untreated patients: adjusted HR 0.78 (95% CI, 0.40 to 1.53). Death due to Covid-19 occurred in 1 treated and 13 untreated patients; adjusted HR: 1.64 (95% CI, 0.40 to 12.95).
As the abstract says, the study did not have the statistical power to show a benefit for preventing severe outcomes in younger adults. It did not “prove [Paxlovid] to be ineffective”! This is very bad, the guide is clearly not a reliable source of information about covid treatments, and I recommend against following the advice of anything else on that website.
Paxlovid-associated rebound is quoted in Nature as occurring in 27% of cases, which is more in line with the anecdata I’ve received than the 5% figures in other research. My anecdata is that it really does hammer down brewing symptoms very effectively, and I haven’t seen reports or systematic trends in the anecdata of the rebound being worse than the first go-’round. So from your perspective, it’s a shot at not getting sick, or maybe of getting less sick.
The risk is yeah, maybe you contribute a miniscule marginal amount to drug resistant COVID-19. The norms of biomedicine are that you, the patient, do not need to concern yourself with that. If the medicine can help you, and your doctor prescribes it, you can have it. You aren’t on the hook to sacrifice the quality of your care for the “greater good.”
As you know, there’s a straightforward way to, given any boolean circuit, to turn it into a version which is a tree, by just taking all the parts which have two wires coming out from a gate, and making duplicates of everything that leads into that gate. I imagine that it would also be feasible to compute the size of this expanded-out version without having to actually expand out the whole thing?
Searching through normal boolean circuits, but using a cost which is based on the size if it were split into trees, sounds to me like it would give you the memoization-and-such speedup, and be kind of equivalent theoretically to using the actual trees?
A modification of it comes to mind. What if you allow some gates which have multiple outputs, but don’t allow gates which duplicate their input? Or, specifically, what if you allow something like toffoli gates as well as the ability to just discard an output? The two parts of the output would be sorta independent due to the gate being invertible (… though possibly there could be problems with that due to original input being used more than once?)
I don’t know whether this still has the can-do-greedy-search properties you want, but it seems like it might in a sense prevent using some computational result multiple times (unless having multiple copies of initial inputs available somehow breaks that).
A quick retrospective on the four Manifold markets on the day, focusing on the possibility of good or bad incentives flowing out of the markets. Several people expressed concerns that the Manifold markets would cause LessWrong’s home page to go down, or go down earlier, due to incentives. People also placed bets and limit orders to try to mute those bad incentives. Some examples:
“This is a classical example where having a prediction market creates really bad incentives.”
“I think the right thing to do here is buy yes so that it’s more beneficial for others to buy and hold no.”
“I’m not wild about this market existing, but given that it does exist I’m strongly in favour of making it profitable for others to not press the button.”
I decided to look into this. I want to say up front that I think all the incentives were very small relative to perceived stakes, and I have no suspicions of anyone after writing this comment. I am also not going to give any bettor names.
By the end of the YES/NO market, several users had bet large sums on YES and were consequently incentivized to blow up the home page. One user had m26,002 YES shares (USD $260). Another user had 7328 NO shares (USD $73) and was incentivized not to blow up the site. But this doesn’t show the whole picture, because there were also limit orders. As was explained to me:
The choice of whether to leave (public) limit orders on one side or the other is the way you incentivize action.
At various points during the day there were large (m1000, etc) YES limit orders in play. This is completely useless as an incentive for an individual LessWrong user. Sure, if I was the only one who might press the button, I could bet through those limit orders, not press the button, and collect my bounty. But many people could press the button, and this would have lost me mana. They did count as an incentive for LessWrong coders! They could quietly introduce a bug preventing the button from being pressed, bet NO, and collect their winnings. Nobody took up that incentive.
I didn’t see any large NO limit orders. A NO limit order on a market like this is inherently risky, because if the button is pressed the offer will definitely be taken, but if the button is not pressed it probably won’t. If there had been then that could have added to the incentive some bettors had to blow up the home page, but as far as I can tell there weren’t.
The positions in the WHEN market were much smaller, with the largest positions coming in at 552 shares in LATE/NEVER and 325 shares in EARLY. The current design of range markets on Manifold is fun to play with, but is not very “swingy”. In particular, there is no option for outsized winnings if you can predict exactly when something will happen. If a user was planning to blow up the home page in the last few hours they could have bet a combination of LATE/NEVER in this market and YES in the binary market to maximize their gains, but I don’t see any evidence of this. Because there are fewer high karma users, large EARLY bets in this market could slightly limit their incentive to bet EARLY and blow up the home page early. Unlike the YES/NO market, those bets didn’t happen.
If someone was willing to anonymously blow up the home page, then a pattern of placing huge bets on that happening might look suspicious and damage their anonymity. So generally I would expect these incentives to only be effective for someone who was willing to blow up the home page and take credit for it. You can bet on that happening in Will anyone try to take credit for nuking LessWrong?. Or, place a limit NO order to incentivize someone to bet YES and then take the credit (and the mana).
Finally mkualquiera made a market—Will my friend agree to defect on Petrov Day?. This market is evidence that two people were thinking about being incentivized to blow up the home page by betting markets. But it seems like this was more about a fun game than making mana. The largest position was 301 shares on YES, and the market resolved NO, so whatever incentive those shares provided, they weren’t enough to make it happen.
In the end I think there is a low (10%) chance that anyone’s behavior was significantly shifted by prediction market incentives. Feel free to reply to this post with how your behavior was shifted so I can update.
I was hopeful that people might shift their behavior based on the prediction market predictions—specifically that the high probability placed on the home page being blown up would lead to design changes. However, this retrospective clarifies that Petrov Day 2022 was a social experiment, so the prediction would have just shown it was expected to work as designed.
… the primary design of the exercises – seeing how long it’d take for the site to go down, even if we were pretty sure that it would.
It’s also possible that people saw the high probability that the home page would be blown up and altered their plans. They may have not blown it up because they expected someone else would (especially if they weren’t confident in their anonymity being preserved). Or they may have blown it up because “if it’s going to happen, it may as well be me”. I think there is a higher chance (25%) that this was an effect, but I don’t know what the net direction would have been. Again, feel free to reply if you think the prediction markets helped you make decisions.
Overall I think the prediction markets were a positive addition to a negative event. I think the main incentive in play was the possibility of seeing, or not seeing, the LessWrong home page being blown up. And I think we all have a lot to figure out about prediction markets.
To clarify, me and my friend were 100% going to press the button, but we were discouraged by the false alarm. There was no fun at that point, and it made me lose like 1⁄3 of my total mana. I had to close all my positions to stop the losses, and we went to sleep. When we woke up it was already too late for it to be noteworthy or fun.
The funny thing is that I had assumed the button was going to be buggy, though I was wrong how. The map header has improperly swallowed mouse scroll wheel events whenever it’s shown; I had wondered if the button would also interpret them likewise since it was positioned in the same way, so I spent most of the day carefully dragging the scrollbar.
You’re drawing a philisophical distinction based on a particular ontology of the wavefunction. As simpler version arises in classical electromagnetism: we can integrate out the charges and describe the world entirely as an evolving state of the E&M field with the charges acting as weird source terms, or we can do the opposite and integrate out the E&M field to get a theory of charges moving with weird force laws. These are all equivalent descriptions in that they are observationally indistinguishable.
If it’s high-quality distillation you’re interested in, you don’t necessarily need a PhD. I’m thinking of e.g. David Roodman, now a senior advisor at Open Philanthropy. He majored in math, then did a year-long independent study in economics and public policy, and has basically been self-taught ever since. Holden Karnofsky considers what he does extremely valuable:
David Roodman, who is basically the person that I consider the gold standard of a critical evidence reviewer, someone who can really dig on a complicated literature and come up with the answers, he did what, I think, was a really wonderful and really fascinating paper, which is up on our website, where he looked for all the studies on the relationship between incarceration and crime, and what happens if you cut incarceration, do you expect crime to rise, to fall, to stay the same? He picked them apart. What happened is he found a lot of the best, most prestigious studies and about half of them, he found fatal flaws in when he just tried to replicate them or redo their conclusions.
When he put it all together, he ended up with a different conclusion from what you would get if you just read the abstracts. It was a completely novel piece of work that reviewed this whole evidence base at a level of thoroughness that had never been done before, came out with a conclusion that was different from what you naively would have thought, which concluded his best estimate is that, at current margins, we could cut incarceration and there would be no expected impact on crime. He did all that. Then, he started submitting it to journals. It’s gotten rejected from a large number of journals by now. I mean starting with the most prestigious ones and then going to the less.
Robert Wiblin: Why is that?
Holden Karnofsky: Because his paper, it’s really, I think, it’s incredibly well done. It’s incredibly important, but there’s nothing in some sense, in some kind of academic taste sense, there’s nothing new in there. He took a bunch of studies. He redid them. He found that they broke. He found new issues with them, and he found new conclusions. From a policy maker or philanthropist perspective, all very interesting stuff, but did we really find a new method for asserting causality? Did we really find a new insight about how the mind of a …
Robert Wiblin: Criminal.
Holden Karnofsky: A perpetrator works. No. We didn’t advance the frontiers of knowledge. We pulled together a bunch of knowledge that we already had, and we synthesized it. I think that’s a common theme is that, I think, our academic institutions were set up a while ago. They were set up at a time when it seemed like the most valuable thing to do was just to search for the next big insight.
These days, they’ve been around for a while. We’ve got a lot of insights. We’ve got a lot of insights sitting around. We’ve got a lot of studies. I think a lot of the times what we need to do is take the information that’s already available, take the studies that already exist, and synthesize them critically and say, “What does this mean for what we should do? Where we should give money, what policy should be.”
I don’t think there’s any home in academia to do that. I think that creates a lot of the gaps. This also applies to AI timelines where it’s like there’s nothing particularly innovative, groundbreaking, knowledge frontier advancing, creative, clever about just … It’s a question that matters. When can we expect transformative AI and with what probability? It matters, but it’s not a work of frontier advancing intellectual creativity to try to answer it.
Shortform #138 A good but slightly disorienting day
I applied for a promising job today, here’s to hoping that bears fruit!
I am somewhat out of whack due to having to suddenly house sit instead of going to my own home after work. I do not enjoy this, but it’s an obligation I’m fulfilling.
No meetup tonight, I rescheduled Norfolk’s meetup for Thursday evening (the 29th). I’m excited for the meetup tomorrow!
Agree with you generally. You may find interest in a lot of the content I posted on reddit over the past couple months on similar subjects, especially in the singularity sub (or maybe you are there and have seen it 😀). Nice write up anyway. I do disagree on some of your generalized statements, but only because I’m more optimistic than yourself, and don’t originally come from a position of thinking these things were impossible.
I’ll be the annoying guy who ignores your entire post and complains about you using celsius as the unit of temperature in a calculation involving the Landauer limit. You should have used kelvin instead, because Landauer’s limit needs an absolute unit of temperature to work. This doesn’t affect your conclusions at all, but as I said, I’m here to be annoying.
That said, the fact that you got this detail wrong does significantly undermine my confidence in the rest of your post, because even though the detail is inconsequential for your overall argument it would be very strange for someone familiar with thermodynamics to make such a mistake.
Notably, the result is correct; I did convert it to kelvin for the actual calculation. Just a leftover from when I was sketching things on wolframalpha. I’ll change that, since it is weird. (Thanks for the catch!)
DL based AGI raised in the right social environments will automatically learn efficient models of external agent values (and empowerment bounds thereof)
The main challenge then is locating the learned representation of external agent values and wiring/grounding it up to the agent’s core utility function (which is initially unsafe: self-motivated empowerment etc), and timing that transition replacement carefully
Evolution also solved both alignment and the circuit grounding problem; we can improve on those solutions (proxy matching)
Ideally as we approach AGI there would be cooperative standardization on alignment benchmarks and all the major players would subject their designs to extensive testing in sandbox sims. Hopefully 1-5 will become increasingly self evident and influence ‘Magma’. If not some other org (perhaps a decentralized system), could hopefully beat Magma to the finish line. Alignment need not have much additional cost: it doesn’t require additional runtime computations, it doesn’t require much additional training cost, and with the ideal test environments it hopefully doesn’t have much of a research iteration penalty (as the training environments and can simultaneously test for intelligence and alignment).
It’s a good idea to be pedantically clear with terminology, and I thank you for saying “front page” rather than “site” in most of the timeline. I visited the site multiple times (and saw a 502 at one point), but I never look at the front page, using my /allPosts bookmark. I never saw the button, though I have enough karma (though I may be removed as a troublemaker, as I think this excercise is rather less weighty than it’s made out to be. I don’t actually know whether I’d press it, given the chance.).
more than 300 karma (1,178 of them exist, 335 visited the site on Petrov Day)
Visited the site, or visited the front page? I’m mostly curious how many of the regulars use the front page, vs using greaterwrong (which I presume isn’t even a site visitor) or a deeper bookmark.
We can conclude that in gaining more karma than [200], one becomes the kind of person who doesn’t destroy the world symbolically or otherwise.
I imagine this is tongue in cheek, but we really can’t. You mentioned an important reason—someone with more karma could have waited to press the button. The first button press occurred 110 minutes after it could have been pressed. The second button press occurred at least 40 minutes after it could have been pressed, and perhaps 100, 160, 220, etc. In 2020 the button was pressed 187 minutes after it could have been pressed (by a 4000+ karma user).
You excluded known trouble makers from accessing the button but you didn’t exclude unknown trouble makers, and lower karma is correlated with being unknown.
We are also dealing with a hostile intelligence who pressed the button (or caused it to be pressed). Someone with higher karma might deliberately wait to press the button to throw people off the scent, to encourage people to make a naive update about karma scores, or to reduce the negative consequences of bringing the home page down for longer without the negative consequences of leaving it up all day. The timing evidence is thus hostile evidence and updating on it correctly requires superintelligence.
Put this together and I would not place any suspicion on the noble class of 200-299 karma users that I happened to enter on Petrov Day after net positive gains from complaining about the big red button.
I am willing to update that at least one person in the 200+ karma range pressed the button, and at least one person with zero karma pressed the button. This assumes there was not a third bug in play. This does not change my opinion of LessWrong users, but those who predicted that the home page would remain up could update.
I also imagine that it was tongue in cheeck but I also think that the structure of the whole thing so heavily suggests this line of thinking that on surface level recognising it to be wrong doesn’t really dispell it.
I’ve contacted a number of people in the field regarding this idea (thanks to everyone who responded!).
The general vibe is “this seems like it could be useful, maybe, if it took off,” but it did not appear to actually solve the problems any specific person I contacted was having.
My expectation would be that people in large organizations would likely not publish anything in this system that they would not publish out in the open.
One of the critical pieces of this proposal is having people willing to coordinate across access boundaries. There were zero enthusiastic takers for that kind of burden, which doesn’t surprise me too much. Without a broad base of volunteers for that kind of task, though, this idea seems to require paying a group of highly trusted and well-informed people to manage coordination instead of, say, researching things. That seems questionable.
Overall, I still think there is an important hole in coordination, but I don’t think this proposal fills it well. I’m not yet sure what a better shape would be.
I didn’t push the button because I knew that doing so would actually take down the front page. And I only believed that would happen because it happened last year.
I know this is going to come off as overly critical no matter how I frame it but I genuinely don’t mean it to be.
Another takeaway from this would seem to be an update towards recognizing the difference between knowing something and enacting it or, analogously, being able to identify inadequacy vs. avoid it. People on LW often discuss, criticize, and sometimes dismiss folks who work at companies that fail to implement all security best practices or do things like push to production without going through proper checklists. Yet, this is a case where exactly that happened, even though there was not strong financial or (presumably) top down pressure to act quickly.
Actually it seems that the industry standard (?) process of code review was followed just fine. Yet the wrong logic still went through. (Actually based on the github PR it seems that the reviewer himself suggested the wrong logic?)
I think in this case there would also be plenty to say about blindly following checklists. (Could code review in some cases make things worse by making people think less about the code they write?)
EDIT: Actually based on the TS types user.karma can’t be missing. Either the types or Ruby’s explanation is wrong. Clearly multiple things had to go wrong for this bug to slip through.
I don’t know all the details of what testing was done, but I would not describe code review and then deploying as state-of-the-art as this ignores things like staged deploys, end-to-end testing, monitoring, etc. Again, I’m not familiar with the LW codebase and deploy process so it’s possible all these things are in place, in which case I’d be happy to retract my comment!
To me it seems that the average best practices are being followed.[1] But these “best practices” are still just a bunch of band-aids, which happen to work fairly very well for most use-cases.
A much more interesting question to ask here is what if something important like … humanity’s survival depended on your software? It seems that software correctness will be quite important for alignment. Yet I see very few people seriously trying to make creating correct software scalable. (And it seems like a field particularly suited for empirical work, unlike alignment. I mean, just throw your wildest ideas at a proof checker, and see what sticks. After you have a proof, it doesn’t matter at all how it was obtained.)
And I think the amount of effort in this case is perfectly justified. I mean this was code for a one-off single day event, nothing mission critical. It would be unreasonable to expect much more for something like this.
Mainly commenting on your footnote, I generally agree that it’s fine to put low amounts of effort into one-off simple events. The caveat here is that this is an event that is 1) treated pretty seriously in past years and 2) is a symbol of a certain mindset that I think typically includes double-checking things and avoiding careless mistakes.
This is why I really wish we had an AI that had superhuman code, theorem proving and translation to natural language, and crucially only those capabilities, so that we can prove certain properties.
The types are wrong. It’s sad that the types are wrong. I sure wish they weren’t wrong, but changing it in all the cases is a pretty major effort (we would have to manually annotate all fields to specify which things can be null/undefined in the database, and which ones can’t, and then would have to go through a lot of type errors).
On the other hand I am afraid this reinforces NaiveTortoise’s point, this seems like an underlying issue that could potentially lead to bugs much worse than this...
users on LessWrong with more than 300 karma (1,178 of them) are not the kind to push buttons for one reason or another. Granted many of them would not have been checking the site
Do you have website access stats that would let you compute precisely how many of them did in fact load the homepage during Petrov Day?
I really like this post because it’s readable and informative. For the second problem, pursuing proxy goals, I recommend also reading about a related problem called the XY problem.
On point 4: many popular alignment ideas are not models of current systems, but models of future AI systems. Accuracy is then lost not only from modeling the system but also from having to create a prediction about it.
It worries me that many of the most promising theories of impact for alignment end up with the structure “acquire power, then use it for good”.
This seems to be a result of the counterfactual impact framing and a bias towards simple plans. You are a tiny agent in an unfathomably large world, trying to intervene on what may be the biggest event in human history. If you try to generate stories where you have a clear, simple counterfactual impact, most of them will involve power-seeking for the usual instrumental convergence reasons. Power-seeking might be necessary sometimes, but it seems extremely dangerous as a general attitude; ironically human power-seeking is one of the key drivers of AI x-risk to begin with. Benjamin Ross Hoffman writes beautifully about this problem in Against responsibility.
I don’t have any good solutions, other than a general bias away from power-seeking strategies and towards strategies involving cooperation, dealism, and reducing transaction costs. I think the pivotal act framing is particularly dangerous, and aiming to delay existential catastrophe rather than preventing it completely is a better policy for most actors.
This is why AI risk is so high, in a nutshell.
Yet unlike this post (or Benjamin Ross Hoffman’s post), I think this was a sad, but crucially necessary decision. I think the option you propose is at least partially a fabricated option. I think a lot of the reason is people dearly want to there be a better option, even if it’s not there.
Fabricated options are products of incoherent thinking; what is the incoherence you’re pointing out with policies that aim to delay existential catastrophe or reduce transaction costs between existing power centers?
I think the fabricated option here is just supporting the companies making AI, when my view is that by default, capitalist incentives kill us all due to boosting AI capabilities while doing approximately zero AI safety, in particular deceptive alignment would not be invested in despite this being the majority of the risk.
One of the most important points for AGI safety is the leader in AGI needs a lot of breathing space and leadership ahead of their competitors, and I think this needs to be done semi-unilaterally by an organization not having capitalist incentives, because all the incentives point towards ever faster, not slowing down AGI capabilities. That’s why I think your options are fabricated, because they assume unrealistically good incentives to do what you want.
I don’t mean to suggest “just supporting the companies” is a good strategy, but there are promising non-power-seeking strategies like “improve collaboration between the leading AI labs” that I think are worth biasing towards.
Maybe the crux is how strongly capitalist incentives bind AI lab behavior. I think none of the currently leading AI labs (OpenAI, DeepMind, Google Brain) are actually so tightly bound by capitalist incentives that their leaders couldn’t delay AI system deployment by at least a few months, and probably more like several years, before capitalist incentives in the form of shareholder lawsuits or new entrants that poach their key technical staff have a chance to materialize.
Maybe the crux is how strongly capitalist incentives bind AI lab behavior. I think none of the currently leading AI labs (OpenAI, DeepMind, Google Brain) are actually so tightly bound by capitalist incentives that their leaders couldn’t delay AI system deployment by at least a few months, and probably more like several years.
This is the crux, thank you for identifying it.
Yeah, I’m fairly pessimistic for several years time, since I don’t think they’re that special of a company in resisting capitalist nudges and incentives.
And yeah I’m laughing because unless the alignment/safety teams control what capabilities are added, then I do not expect the capabilities teams to stop, because they won’t get paid for that.
I was talking with Abram Demski today about a promising-seeming research direction. (Following is my own recollection)
One of my (TurnTrout’s) reasons for alignment optimism is that I think:
We can examine early-training cognition and behavior to some extent, since the system is presumably not yet superintelligent and planning against us,
(Although this amount of information depends on how much interpretability and agent-internals theory we do now)
All else equal, early-training values (decision-influences) are the most important to influence, since they steer future training.
It’s crucial to get early-training value shards of which a substantial fraction are “human-compatible values” (whatever that means)
For example, if there are protect-human-shards which
reliably bid against plans where people get hurt,
steer deliberation away from such plan stubs, and
these shards are “reflectively endorsed” by the overall shard economy (i.e. the decision-making isn’t steering towards plans where the protect-human shards get removed)
If we install influential human-compatible shards early in training, and they get retained, they will help us in mid- and late-training where we can’t affect the ball game very much (e.g. alien abstractions, interpretability problems, can’t oversee AI’s complicated plans)
Therefore it seems very important to understand what’s going on with “shard game theory” (or whatever those intuitions are pointing at) -- when, why, and how will early decision-influences be retained?
He was talking about viewing new hypotheses as adding traders to a market (in the sense of logical induction). Usually they’re viewed as hypotheses. But also possibly you can view them as having values, since a trader can basically be any computation. But you’d want a different market resolution mechanism than a deductive process revealing the truth or falsity of some proposition under some axioms. You want a way for traders to bid on actions.
I proposed a setup like:
Maybe you could have an “action” instead of a proposition and then the action comes out as 1 or 0 depending on the a function of the market position on that action at a given time, which possibly leads to fixed points for every possible resolution.
For example, if all the traders hold a1 as YES, then a1 actually does come out as YES. And eg a trader T1 which “wants” all the even-numbered actions and T2 wants all the 10-multiple actions (a10,a20,...), they can “bargain” by bidding up each others’ actions whenever they have extra power and thereby “value handshake.”
And that over time, traders who do this should take up more and more market share relative to those who dont exploit gains from trade.
There should be a very high dependence of final trader coalition on the initial composition of market share. And it seems like some version of this should be able to model self-reflective value drift. You can think about action resolution and payout as a kind of reward event, where certain kinds of shards get reinforced. Bidding for an action which happens and leads to reward, gets reinforced (supporting traders receive payouts), and the more you support (bid for it), the more responsible your support was for the event, so the larger the strengthening.
Abram seemed to think that there might exist a nice result like “Given a coalition of traders with values X, Y, Z satisfies properties A, B, and C, this coalition will shape future training and trader-addition in a way which accords with X/Y/Z values up to [reasonably tight trader-subjective regret bound].”
What this would tell us is when trader coalitions can bargain / value handshake / self-trust and navigate value drift properly. This seems super important for understanding what happens, long-term, as the AI’s initial value shards equilibrate into a reflectively stable utility function; even if we know how to get human-compatible values into a system, we also have to ensure they stay and keep influencing decision-making. And possibly this theorem would solve ethical reflection (e.g. the thing people do when they consider whether utilitarianism accords with their current intuitions).
Issues include:
Somehow this has to confront Rice’s theorem for adding new traders to a coalition. What strategies would be good?
I think “inspect arbitrary new traders in arbitrary situations” is not really how value drift works, but it seems possibly contingent on internal capabilities jumps in SGD
The key question isn’t can we predict those value drift events, but can the coalition
EG agent keeps training and is surprised to find that an update knocks out most of the human-compatible values.
Knowing the right definitions might be contingent on understanding more shard theory (or whatever shard theory should be, for AI, if that’s not the right frame).
Possibly this is still underspecified and the modeling assumptions can’t properly capture what I want; maybe the properties I want are mutually exclusive. But it seems like it shouldn’t be true.
ETA this doesn’t model the contextual activation of values, which is a centerpiece of shard theory.
I’ve been thinking along similar lines recently. A possible path to AI safety that I’ve been thinking about extends upon this:
A promising concrete endgame story along these lines is Ought’s plan to avoid the dangerous attractor state of AI systems that are optimized end-to-end
Technological Attractor: Off-the-shelf subsystems
One possible tech-tree path is that we start building custom silicon to implement certain subsystems in an AI agent. These components would be analogous to functional neural regions of the human brain such as the motor cortex, visual system, etc. -- the key hypothesis being that once we reach a certain level of model complexity, the benefits from training a model end-to-end are not worth the costs of re-learning all of these fundamental structures, and furthermore that you can get much better performance-per-cost by casting these modular, reusable components onto an ASIC. This could be a more feasible way of achieving something like [Microscope AI](https://www.lesswrong.com/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#5__Microscope_AI).
Given a few such low-level components, we could enter a technological attractor where getting SOTA through another approach requires either i) throwing multiple OOM more compute than previous SOTA at re-training your own copy of these components in an end-to-end model, or ii) building your own microchip fab to implement your custom component design. Both of these could be high enough barriers that in practice the market participants simply use the off-the-shelf components. And in this attractor more R&D goes into building better-performing and higher-abstraction components that can be combined arbitrarily for different tasks.
Given such off-the-shelf components, say, a standard visual cortex for image recognition, or a 3D spatial reasoning system which can be connected atop a visual cortex, or a generic working memory, etc., it becomes much more tractable to engage in interpretability work that can convince us of the safety of these components. That then just leaves the problem of proving the highest-level network is safe (i.e. whatever is trained to use these components). This is by no means trivial, but for example the ELK paper spent a lot of time worrying about whether you can even trust the basic ontology of an agent, and I think this path would make Ontology Identification tractable. (Of course, this by construction excludes a number of possible agent-configurations).
Now, it’s possible one could plug in these components and still accidentally have the trained agent learn its own replacement visual cortex that is deceptive and not interpretable, but I think that we can get an intuition for when this might happen. If the best off-the-shelf visual cortex module has complexity on the order of N parameters, and our new model that stitches together these lower-level modules has N parameters or less, then it seems implausible that it has somehow learned a new visual cortex network that is much more compact/efficient than the previous state of the art—increasingly implausible the more resources are devoted to optimizing these components. (It would be like someone coming along today and claiming they had achieved a 10x performance improvement on CPU clock speed; given the huge research budgets and historical gradual capability increases, we don’t expect this to happen). On the other hand, we should be suspicious of a N×102 parameter model where we are unable to interpret what 10% of the network is doing; that’s easily enough complexity to hide its own unsafe re-implementation of our components. (I’m aware that there’s a lot of ground in between these two points, I’m merely trying to illustrate that there is “likely safe” and “likely unsafe” ground, rather than claim exactly how big they each are.)
The final step here is the shakiest. It’s not clear to me that we can keep the “top layer” (the actual network that is stitching together the low-level components; perhaps the Neocortex, by analogy to human neural architecture?) thin enough to be obviously not learning its own unsafe component-replacements. However, I think this framework at least paints a picture of a “known safe” or at least “likely safe” path to AGI; if we see that the practical engineering and economic decisions produce thin top-layer models using thick component layers, then we can devote energy to proving the components are safe/interpretable by construction, and exploring the interpretation of the top-level networks that consume the lower-level components. AGI “neurobiology” will be much more tractable if the “neural architecture” is relatively standardized. And so, this could be a good place to provide an early nudge to tip the system into this attractor; heavy investment into research on componentized NN architectures could be viewed as “gain of function” research, but it could also have a much safer end-point.
Another way of thinking about this is that by crystalizing at least some parts of the AGI’s network into slowly-changing structures, we allow time to thoroughly test those parts. It seems very hard to thoroughly test models for safety in a paradigm where the whole model is potentially retrained regularly.
Another way of thinking about this is that by crystalizing at least some parts of the AGI’s network into slowly-changing structures, we allow time to thoroughly test those parts. It seems very hard to thoroughly test models for safety in a paradigm where the whole model is potentially retrained regularly.
We need to test designs, and most specifically alignment designs, but giving up retraining (ie lifetime learning) and burning circuits into silicon is unlikely to be competitive; throwing out the baby with the bathwater.
Also whether a design is aligned or not isn’t the type of question one can answer by analyzing the agent’s visual cortex, it’s near purely a function of what is steering the planning system.
Would you care to flesh this assertion out a bit more?
To be clear I’m not suggesting that this is optimal now. Merely speculating that there might be a point between now and AGI where the work to train these sub components becomes so substantial that it becomes economical to modularize.
whether a design is aligned or not isn’t the type of question one can answer by analyzing the agent’s visual cortex
As I mentioned earlier in my post, I was alluding to the ELK paper with that reference, specifically Ontology Identification. Obviously you’d need higher order components too. Like I said, I am imagining here that the majority of the model is “off the shelf”, and just a thin layer is usecase-specific.
To make this more explicit, if you had not only off-the-shelf visual cortex, but also spatio-temporal reasoning modules built atop (as the human brain does), then you could point your debugger at the contents of that module and understand what entities in space were being perceived at what time. And the mapping of “high level strategies” to “low level entities” would be a per-model bit of interpretability work, but should become more tractable to the extent that those low level entities are already mapped and understood.
So for the explicit problem that the ELK paper was trying to solve, if you are confident you know what underlying representation SmartVault is using, it’s much easier to interpret its higher-level actions/strategies.
Interesting, I haven’t seen anyone write about hardware-enabled attractor states but they do seem very promising because of just how decisive hardware is in determining which algorithms are competitive. An extreme version of this would be specialized hardware letting CAIS outcompete monolithic AGI. But even weaker versions would lead to major interpretability and safety benefits.
One other thought after considering this a bit more—we could test this now using software submodules. It’s unlikely to perform better (since no hardware speedup) but it could shed light on the tradeoffs with the general approach. And as these submodules got more complex, it may eventually be beneficial to use this approach even in a pure-software (no hardware) paradigm, if it lets you skip retraining a bunch of common functionality.
I.e. if you train a sub-network for one task, then incorporate that in two distinct top-layer networks trained on different high-level goals, do you get savings by not having to train two “visual cortexes”?
This is in a similar vein to Google’s foundation models, where they train one jumbo model that then gets specialized for each usecase. Can that foundation model be modularized? (Maybe for relatively narrow usecases like “text comprehension” it’s actually reasonable to think of a foundation model as a single submodule, but I think they are quite broad right now. ) The big difference is I think all the weights are mutable in the “refine the foundation model” step?
Perhaps another concrete proposal for a technological attractor would be to build a SOTA foundation model and make that so good that the community uses it instead of training their own, and then that would also give a slower-moving architecture/target to interpret.
2. In the Destruction Game, does everyone get the ability to destroy arbitrary amounts of utility, or is how much utility they are able to destroy part of the setup of the game, such that you can have games where e.g. one player gets a powerful button and another player gets a weak one?
For 1, it’s just intrinsically mathematically appealing (continuity is always really nice when you can get it), and also because of an intution that if your foe experiences a tiny preference perturbation, you should be able to use small conditional payments to replicate their original preferences/incentive structure and start negotiating with that, instead.
I should also note that nowhere in the visual proof of the ROSE value for the toy case, is continuity used. Continuity just happens to appear.
For 2, yes, it’s part of game setup. The buttons are of whatever intensity you want (but they have to be intensity-capped somewhere for technical reasons regarding compactness). Looking at the setup, for each player pair i,j, Di,j is the cap for how much of j’s utility that i can destroy. These can vary, as long as they’re nonnegative and not infinite. From this, it’s clear “Alice has a powerful button, Bob has a weak one” is one of the possibilities, that would just mean Da,b>>Db,a. There isn’t an assumption that everyone has an equally powerful button, because then you could argue that everyone just has an equal strength threat and then it wouldn’t be much of a threat-resistance desideratum, now would it? Heck, you can even give one player a powerful button and the other a zero-strength button that has no effect, that fits in the formalism.
So the theorem is actually saying “for all members of the family of destruction games with the button caps set wherever the heck you want, the payoffs are the same as the original game”.
OK, thanks! Continuity does seem appealing to me but it seems negotiable; if you can find an even more threat-resistant bargaining solution (or an equally threat-resistant one that has some other nice property) I’d prefer it to this one even if it lacked continuity.
Absolutely brilliant stuff Jacob! As usual with your posts, I’ll have to ponder this for a while...Let’s see if I got this right:
Evolution had to solve alignment: how to align the powerful general learning engine of the newbrain (neocortex etc) with the goals of the oldbrain (“reptilian brain”).
Some (most?) of this alignment seems to be a form of inverse reinforcement learning. Another form of alignment that the oldbrain applies to the newbrain is imprinting. It is Evolution’s way of solving the pointing problem.
When a duckling hatches it imprints on the first agent it sees. This feels different from inverse reinforcement learning: it’s not like the newbrain is rewarded or punished, rather it is more like there is an open slot for your mom
But I would say that imprinting is a specific instance of a more general process (which I called correlation guided proxy matching). The oldbrain has a simple initial mom detector circuit, which during normal chick learning phase is just good enough to locate and connect to the learned newbrain mom detector circuit, which then replaces/supplants the oldbrain equivalent. The proxy matching needn’t really effect the newbrain directly.
The post starts with the realization that we are actually bottlenecked by data and then proceeds to talk about HW acceleration. Deep learning is in a sense a general paradigm, but so is random search. It is actually quite important to have the necessary scale of both compute and data and right now we are not sure about either of them. Not to mention that it is still not clear whether DL actually leads to anything truly intelligent in a practical sense or whether we will simply have very good token predictors with very limited use.
I don’t actually think we’re bottlenecked by data. Chinchilla represents a change in focus (for current architectures), but I think it’s useful to remember what that paper actually told the rest of the field: “hey you can get way better results for way less compute if you do it this way.”
I feel like characterizing Chinchilla most directly as a bottleneck would be missing its point. It was a major capability gain, and it tells everyone else how to get even more capability gain. There are some data-related challenges far enough down the implied path, but we have no reason to believe that they are insurmountable. In fact, it looks an awful lot like it won’t even be very difficult!
With regards to whether deep learning goes anywhere: in order for this to occupy any significant probability mass, I need to hear an argument for how our current dumb architectures do as much as they do, and why that does not imply near-term weirdness. Like, “large transformers are performing {this type of computation} and using {this kind of information}, which we can show has {these bounds} which happens to include all the tasks it has been tested on, but which will not include more worrisome capabilities because {something something something}.”
The space in which that explanation could exist seems small to me. It makes an extremely strong, specific claim, that just so happens to be about exactly where the state of the art in AI is.
I don’t actually think we’re bottlenecked by data. Chinchilla represents a change in focus (for current architectures), but I think it’s useful to remember what that paper actually told the rest of the field: “hey you can get way better results for way less compute if you do it this way.”
I feel like characterizing Chinchilla most directly as a bottleneck would be missing its point. It was a major capability gain, and it tells everyone else how to get even more capability gain. There are some data-related challenges far enough down the implied path, but we have no reason to believe that they are insurmountable. In fact, it looks an awful lot like it won’t even be very difficult!
Some of my confidence here arises from things that I don’t think would be wise to blab about in public, so my arguments might not be quite as convincing sounding as I’d like, but I’ll give a try.
I wouldn’t quite say it’s not a problem at all, but rather it’s the type of problem that the field is really good at solving. They don’t have to solve ethics or something. They just need to do some clever engineering with the backing of infinite money.
I’d put it at a similar tier of difficulty as scaling up transformers to begin with. That wasn’t nothing! And the industry blew straight through it.
To give some examples that I’m comfortable having in public:
Suppose you stick to text-only training. Could you expand your training sets automatically? Maybe create a higher quality transcription AI and use it to pad your training set using the entirety of youtube?
Maybe you figure out a relatively simple way to extract more juice from a smaller dataset that doesn’t collapse into pathological overfitting.
Maybe you make existing datasets more informative by filtering out sequences that seem to interfere with training.
Maybe you embrace multimodal training where text-only bottlenecks are irrelevant.
Maybe you do it the hard way. What’s a few billion dollars?
in order for this to occupy any significant probability mass, I need to hear an argument for how our current dumb architectures do as much as they do, and why that does not imply near-term weirdness. Like, “large transformers are performing {this type of computation} and using {this kind of information}, which we can show has {these bounds} which happens to include all the tasks it has been tested on, but which will not include more worrisome capabilities because {something something something}.”
What about: State-of-the-art models with 500+B parameters still can’t do 2-digit addition with 100% reliability. For me, this shows that the models are perhaps learning some associative rules from the data, but there is no sign of intelligence. An intelligent agent should notice how addition works after learning from TBs of data. Associative memory can still be useful, but it’s not really an AGI.
They are simluators (https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators), not question answerers. Also, I am sure Minerva does pretty good on this task, probably not 100% reliable but humans are also not 100% reliable if they are required to answer immediately. If you want the ML model to simulate thinking [better], make it solve this task 1000 times and select the most popular answer (which is a quite popular approach for some models already). I think PaLM would be effectively 100% reliable.
And whatever algorithms this AI is using to go about its reasoning, they’re apparently so simple that the AI can execute them while still struggling on absolutely trivial arithmetic.
WHAT?
Yes, the AI has some blatant holes in its capability. But what we’re seeing is a screaming-hair-on-fire warning that the problems we thought are hard are not hard.
What happens when we just slightly improve our AI architectures to be less dumb?
When will we get robotics results that are not laughable? When “Google put their most advanced AI into a robot brain!!!” (reported on for the third time this year) we got a robot that can deliver a sponge and misplace an empty coke can but not actually clean anything or do anything useful. It’s hard for me to be afraid of a robot that can’t even plug in its own power cable.
I believe that over time we will understand that producing human-like text is not a sign of intelligence. In the past people believed that only intelligent agents are able to solve math equations (naturally, since only people can do it and animals can). Then came computer and they were able to do all kinds of calculations much faster and without errors. However, from our current point of view we now understand that doing math calculations is not really that intelligent and even really simple machines can do that. Chess playing is similar story, we thought that you have to be intelligent, but we found a heuristic to do that really well. People were afraid that chess-algorithm-like machines can be programmed to conquer the world, but from our perspective, that’s a ridiculous proposition.
I believe that text generation will be a similar case. We think that you have to be really intelligent to produce human-like outputs, but in the end with enough data, you can produce something that looks nice and it can even be useful sometimes, but there is no intelligence in there. We will slowly develop an intuition about what are the capabilities of large-scale ML models. I believe that in the future we will think about them as basically a kinda fuzzy databases that we can query with natural language. I don’t think that we will think about them as intelligent agents capable of autonomous actions.
Chess playing is similar story, we thought that you have to be intelligent, but we found a heuristic to do that really well.
You keep distinguishing “intelligence” from “heuristics”, but no one to my knowledge has demonstrated that human intelligence is not itself some set of heuristics. Heuristics are exactly what you’d expect from evolution after all.
So your argument then reduces to a god of the gaps, where we keep discovering some heuristics for an ability that we previously ascribed to intelligence, and the set of capabilities left to “real intelligence” keeps shrinking. Will we eventually be left with the null set, and conclude that humans are not intelligent either? What’s your actual criterion for intelligence that would prevent this outcome?
I believe that fixating on benchmark such as chess etc is ignoring the G part of AGI. Truly intelligent agent should be general at least in the environment he resides in, considering the limitation of its form. E.g. if a robot is physically able to work with everyday object, we might apply Wozniak test and expect that intelligent robot is able to cook a dinner in arbitrary house or do any other task that its form permits.
If we assume that right now we develop purely textual intelligence (without agency, persistent sense of self etc) we might still expect this intelligence to be general. I.e. it is able to solve arbitrary task if it seems reasonable considering its form. In this context for me, an intelligent agent is able to understand common language and act accordingly, e.g. if a question is posed it can provide a truthful answer.
BIG Bench has recently showed us that our current LMs are able to solve some problems, but they are nowhere near general intelligence. They are not able to solve even very simple problems if it actually requires some sort of logical thinking and not only using associative memory, e.g. this is a nice case:
You can see in the Model performance plots section that scaling did not help at all with tasks like these. This is a very simple task, but it was not seen in the training data so the model struggles to solve it and it produces random results. If the LMs start to solve general linguistic problems, then we are actually having intelligent agents at our hand.
In this context for me, an intelligent agent is able to understand common language and act accordingly, e.g. if a question is posed it can provide a truthful answer
Humans regularly fail at such tasks but I suspect you would still consider humans generally intelligent.
In any case, it seems very plausible that whatever decision procedure is behind more general forms of inference, it will very likely fall to the inexorable march of progress we’ve seen thus far.
If it does, the effectiveness of our compute will potentially increase exponentially almost overnight, since you are basically arguing that our current compute is hobbled by an effectively “weak” associative architecture, but that a very powerful architecture is potentially only one trick away.
The real possibility that we are only one trick away from a potentially terrifying AGI should worry you more.
I don’t see any indication of AGI so it does not really worry me at all. The recent scaling research shows that we need non-trivial number of magnitudes more data and compute to match human-level performance on some benchmarks (with a huge caveat that matching a performance on some benchmark might still not produce intelligence). On the other hand, we are all out of data (especially high quality data with some information value, no random product reviews or NSFW subreddit discussions) and our compute options are also not looking that great (Moore’s law is dead, the fact that we are now relying on HW accelerators is not a good thing, it’s a proof that CPU performance scaling is after 70 years no longer a viable option. There are also some physical limitations that we might not be able to break anytime soon.)
It is goalpost moving. Basically, it says “current models are not really intelligent”. I don’t think there is much disagreement here. And it’s hard to make any predictions based on that.
Also, “Producing human-like text” is not well defined here; even ELIZA may match this definition. Even the current SOTA may not match it because the adversarial Turning Test has not yet been passed.
It’s not goapost moving, it’s the hype that’s moving. People reduce intelligence to arbitrary skills or problems that are currently being solved and then they are let down when they find out that the skill was actually not a good proxy.
I agree that LMs are concetually more similar to ELIZA than to AGI.
The observation that things that people used to consider intelligent are now considered easy is critical.
The space of stuff remaining that we call intelligent, but AIs cannot yet do, is shrinking. Every time AI eats something, we realize it wasn’t even that complicated.
The reasonable lesson appears to be: we should stop default-thinking things are hard, and we should start thinking that even stupid approaches might be able to do too much.
It’s a statement more about the problem being solved, not the problem solver.
When you stack this on a familiarity with the techniques in use and how they can be transformatively improved with little effort, that’s when you start sweating.
Upvoted for the well reasoned arguments. It seems that competition should be the default assumption for the future, for a whole host of reasons, some of which are elaborated here. Personally I never found the opposite claims to be convincing, so I may be biased here.
As a corollary, this implies that claims such as competition will somehow cease, some sort of ‘lock in’ past a threshold, etc., should be subject to a much higher level of scrutiny.
Thank you for this explanation. Now it helps me to understand a little bit more of why so many people I know simply feel overwhelmed and give up. Personally as I am not in position to donate money, I work to tackle one specific problem set that I think will help open up and leave the solutions to other problems.
ShiraDest
This was really good and definitely made me think about how I might live in such a scenario. I would probably go all in on frequent redaction and just lean hard on external memory storage to make up the difference. I already barely remember anything from even ten years ago and rely mostly on external memory for everything, I have a strong ability to acausally coordinate with myself across time, so I’m not worried about different iterations of me going off course in ways I wouldn’t endorse. If you have a strong enough exomemory system you can effectively just freeze your age groundhog-day style and rely on the work you did before each redaction to keep carrying you forward. This does require a strong ability to log and rely information but it seems very manageable, and once I’ve lived long enough for direct mind-computer interfacing I can just back up my memories, redact, and then download them again.
I really enjoyed the first 60% of the story, and then it became… too long. Not sure if this is just about me, or if others have a similar reaction.
But, I get ahead of myself. How did things come to this point?
Not sure if this is just a rationalization, but at this moment you switched from present moment to retrospective, which removed the tension. Until this point, I felt like the story could go in many possible directions. (Who is Bruce Hance? What are his actual plans? Will the protagonist hack this device, and maybe reprogram people to worship him instead?) After this point, I felt like “okay, this is obviously a metaphor for a religion” and stopped expecting anything that wouldn’t fit this metaphor. The ending was nice again, though.
Possibly. I had great fun with the first half of this story, and then I saw where the scroll thingy on my screen was and so stopped with the intention of coming back for the second half. (Which I just did. Second half also great fun!) I don’t think its a flaw in the story, I think that it might just be that its length puts it as a bit of an outlier in terms of webpage reading.
Solar flares are a great plot device for “I need some machine to malfunction”.
It is abrupt, but to me that wasn’t a bad thing. If them arriving had taken an extra x paragraphs but the story worked out exactly the same way I think that for me that would be worse not better. If there was some clever way of hinting them earlier maybe. The solar flare works quite well for that.
I think you should not act on my advice alone. I might be an outlier.
Furthermore, even if I correctly detected what makes the story worse (for a group of people larger than myself) it does not automatically mean that following my advice would improve it. People are better at detecting what they don’t like than at improving things. (For example, I could say which meals taste better and which worse, but I couldn’t cook the meals I like most.)
The last objection is that, in long term, rewriting this story is irrelevant. If my complaint makes sense (which still remains to be verified), the best reaction is to keep writing other things, and not make the same mistake again.
So… I guess just leave it as it is. If you later decide to publish it elsewhere, probably two parts will be better. When writing new stories, consider not changing the style in the middle (or if you do, keep the parts separate).
I didn’t find “Do Birth Order Effects Exist?”, “Challenges to Christiano’s Iterated Amplification Proposal” seems to be only on intelligence.org and “Response to FAQ on Iterated Amplification” seems to be a comment and not a post of it’s own.
I have read about this many years ago. We did a proof of the Gödel’s incompleteness theorem at university. And I still have the feeling like there is some sleight of hand involved… that even when I am telling the proof, I am merely repeating the teacher’s password. My intuition has no model of what this all actually means.
From my perspective, understanding math means that I can quickly say why something works, what it implies, how would the situation change if we slightly changed the conditions, which part of the entire proof is the critical one, etc.
I’m not sure if this is what you’re looking for, but Hofstadter gives a great analogy using record players which I find useful in terms of thinking about how changing the situation changes our results (which is paraphrased here).
A (hi-fi) record player that tries to playing every possible sound can’t actually play its own self-breaking sound, so it is incomplete by virtue of its strength.
A (low-fi) record player that refuses to play all sounds (in order to avoid destruction from its self-breaking sound) is incomplete by virtue of its weakness.
We may think of the hi-fi record player as a formal system like Peano Arithmetic: the incompleteness arises precisely because it is strong enough to be able to capture number theory. This is what allows us to use Gödel Numbering, which then allows PA to do meta-reasoning about itself.
The only way to fix it is to make a system that is weaker than PA, so that we cannot do Gödel Numbering. But then we have a system that isn’t even trying to express what we mean by number theory. This is the low-fi record player: as soon as we fix the one issue of self-reference, we fail to capture the thing we care about (number theory).
I think an example of a weaker formal system is Propositional Calculus. Here we do actually have completeness, but that is only because Propositional Calculus is too weak to be able to capture number theory.
What exactly is the aspect of natural numbers that makes them break math, as opposed to other types of values? Intuitively, it seems to be the fact that they can be arbitrarily large but not infinite.
Like, if you invent another data type that only has a finite number of values, it would not allow you to construct something equivalent to Gödel numbering. But if it allows infinite number of (finite) values, it would. (Not sure about an infinite number of/including infinite values, probably also would break math.)
It seems like you cannot precisely define natural numbers using first-order logic. Is that the reason of this all? Or is it a red herring? Would situation be somehow better with second-order logic?
(These are the kinds of questions that I assume would be obvious to me, if I grokked the situation. So the fact that they are not obvious, suggests that I do not see the larger picture.)
I don’t have much to add, but I basically endorse this, and it’s similar to what I try to do myself.
Figuring out when and how to proactively share information about my inner-layers with people is a balancing act I’m still figuring out, but I’ve found it generally valuable for building trust.
I’m a little bit skeptical of the argument in “Transformers are not special”—it seems like, if there were other architectures which had slightly greater capabilities than the Transformer, and which were relatively low-hanging fruit, we would have found them already.
I’m in academia, so I can’t say for sure what is going on at big companies like Google. But I assume that, following the 2017 release of the Transformer, they allocated different research teams to pursuing different directions: some research teams for scaling, and others for the development of new architectures. It seems like, at least in NLP, all of the big flashy new models have come about via scaling. This suggests to me that, within companies like Google, the research teams assigned to scaling are experiencing success, while the research teams assigned to new architectures are not.
It genuinely surprises me that the Transformer still has not been replaced as the dominant architecture since 2017. It does not surprise me that sufficiently large or fancy RNNs can achieve similar performance to the Transformer. The lack of Transformer replacements makes me wonder whether we have hit the limit on the effectiveness of autoregressive language models, though I also wouldn’t be surprised if someone comes up with a better autoregressive architecture soon.
There have been a few papers with architectures showing performance that matches transformers on smaller datasets with scaling that looks promising. I can tell you that I’ve switched from attention to an architecture loosely based on one of these papers because it performed better on a smallish dataset in my project but I haven’t tested it on any standard vision or language datasets, so I don’t have any concrete evidence yet. Nevertheless, my guess is that indeed there is nothing special about transformers.
Being slightly better isn’t enough to unseat an entrenched option that is well understood. It would probably have to very noticeably better, particularly in scaling.
I expect the way the internal structures are used will usually dominate the details of the internal structure (once you’re already at the pretty good frontier).
If you’re already extremely familiar with transformers, and you can simply change how you use transformers for possible gains, you’re more likely to do that than to explore a from-scratch technique.
For example, in my research, I’m currently looking into some changes to the outer loop of execution to make language models interpretable by construction. I want to focus on that part of it, and I wanted the research to be easily consumable by other people. Building an entire new architecture from scratch would be a lot of work and would be less familiar to others. So, not surprisingly, I picked a transformer for the internal architecture.
But I also have other ideas about how it could be done that I suspect would work quite well. Bit hard to justify doing that for safety research, though :P
I think the amount of low hanging fruit is so high that we can productively investigate transformer derivatives for a long time without diminishing returns. They’re more like a canvas than some fixed Way To Do Things. It’s just also possible someone makes a jump with a non-transformer architecture at some point.
It does not follow that computationally cheaper things are more likely to happen than computationally expensive things. Moreover, describing something as “computationally difficult” is a subjective value judgment (unless you can reasonably prove otherwise) and implies that all actions/events can be reduced to some form of computation.
Georgism isn’t taxation, it’s public property ownership.
It finally clicked in my mind why I’ve reacted negatively to the discussion about Georgist land taxes. It’s the same reaction I have to ANY 100% tax rate on anything. That’s not what “ownership” means.
Honestly, I’d probably be OK with (or at least I’d be able to discuss it more reasonably) the idea that land is not privately owned, ever. Only rented/leased from the government, revocable for failure to pay the current rate (equivalent in all ways to the Georgist tax amount, and penalties for failure to pay). I suspect it’s unworkable for the same reasons—the public won’t stand for evicting sympathetic tenants just because they moved in when the rent/tax was lower.
Further, taxation of economic movement or realized profit is pretty easy to justify, and ALWAYS exists. Taxation of durable-ownership valuation is tricky, as there is disagreement over whether it’s “real”, and there’s no guarantee that the investment value, even if correctly assessed, is liquid enough to actually pay the tax.
Edit: this had occurred to me before, but I forgot to put it here. It also suffers from the socialist calculation problem. When self-interested humans aren’t making decisions to maximize value (to themselves) about a thing, the actual most valuable use is very difficult to determine. Holding someone responsible for the theoretical value of something when they don’t actually think it’s the best is really difficult to envision.
Further edit: thanks for the comments—it’s helped me understand my true objection, in addition to the auxiliary problems that often come to mind. I think there are two insurmountable problems, not necessarily in this order:
Taxing anything at 100% is disingenuous about the word “ownership”. It should be called “rent” or something else that is clear that it’s not owned, but merely usable for some purposes until you decline to pay a future arbitrary amount. It would also make clear that you’re also undertaking all of the political-economy influences that make government setting of rent amounts problematic.
Taxing non-realized value is always going to be problematic, both for calculation reasons (valuation is theoretical and debatable), and for the fragility of the value. I think this generalizes to “only collect taxes in-kind (or on liquid enough things that it can be easily converted at tax-collection time)”. Taxing a pile of wheat is easy—take some percentage of it. Taxing a monetary transaction likewise—the price increases and the tax is paid out of that. Taxing a future value is not (generally) feasible—either you’ve just taken someone’s seeds so they can’t plant the crop you’re trying to tax, or you’re demanding cash from someone whose investments will suffer from early liquidation. Deferral of payment can help with this, but if you’re doing that, you should just tax the realized gains at time of liquidity.
the public won’t stand for evicting sympathetic tenants just because they moved in when the rent/tax was lower.
We would really need the technology where you can take a house and move it to a different place. That would also make Georgism easier—when people don’t pay, you simply move them to a cheaper location.
While I think that would be extremely valuable technology, exiling Grandma to Outer Nowhereville, hours away from the rest of the family, is probably not a lot more feasible.
Alternatively, what if the LVT only increased when the property was sold? Buy land, and you lock in the LVT at the point in time when your deal went through.
I guess people would never sell the real estate at good location, only lease it? Thus capturing the increased value of the land without ever paying the tax.
Maybe leases are subject to increased LVT, just not owning it for personal use? I suspect there’s a way to enact something like this, but this isn’t in my wheelhouse.
Rich people and businesses are more strongly motivated and able to find loopholes than government administrators are to design robust mechanisms. Modern fractional ownership and lease arrangements are complicated enough that most any scheme based on a change in contract participants is going to be arbitraged away very quickly.
Right, but I thought the point was to avoid people balking at a law that makes it easier to evict ordinary folks who’ve lived somewhere a long time and don’t want to move? Let the rich people find their loopholes—as long as people support the system and the LVT delivers most of its promise, that’s good enough.
My point is that whatever exclusions are in place to keep poor people in their newly-valuable homes will be used by rich people to avoid taxation. There is no way to avoid this without making the tax variable based on other factors than the land value (like the “owner’s” perceived financial situation).
Either the law is the same for all “pay or GTFO”, or it’s differentially enforced, so the rich and the politically-sympathetic pay less than the middle-class.
It sounds like you think there’s no practical way to introduce exceptions to make an LVT politically palatable without also breaking its ability to motivate efficient land use and generate tax revenue.
My first thought is sure, we have a lot of tax evasion now. But we also do generate lots of tax revenue, most tax revenue comes from the very rich. Why would an LVT be so much more prone to tax evasion than the current system?
Correct, I think there’s no practical or realpolitik-theoretical way to make it work. To the extent that you make exceptions in implementation, it’s no longer a land-value tax, it’s a land-value fig leaf over a “what we can get away with” tax.
to the extent that you make exceptions in implementation, it’s no longer a land-value tax
This is true the same way that to the extent you dilute your cocoa with water, it’s no longer hot chocolate, it’s water :) I don’t know of almost any “pure” real world policies or taxes. I’d need some good evidence to believe that the LVT is that fragile.
Your definition of AGI (“the kind of AI with sufficient capability to make it a genuine threat to humanity’s future or survival if it is misused or misaligned”) is tragically insufficient, vague, subjective, and arguably misaligned with the generally accepted definition of AGI.
From what you wrote elsewhere (“An AGI having its own goals and actively pursuing them as an agent”) you imply that the threat could come from AGI’s intentions, that is, you imply that AGI will have consciousness, intentionality, etc. - qualities so far exclusively prescribed to living things (you have provided no arguments to think otherwise).
However, you decided to define “intelligence” as “stuff like complex problem solving that’s useful for achieving goals” which means that intentionality, consciousness, etc. is unconnected to it (realistically any “complex”-enough algorithm satisfies this condition). Such simplistic and reductionistic definition implies that it is not enough for an “intelligent” computer to be an AGI. So, while you may be able to prove that a computer could have “intelligence” it still does not follow that AGI is possible.
Your core idea that “We’ve already captured way too much of intelligence with way too little effort.” may be true with your definition of “intelligence”, but I hope I’ve shown that such definition is not enough. Researchers at Harvard suggest existence of multiple types of intelligence, which your statement does not take into account and groups all types of intelligence into one even though some are impossible for a computer to have and some could be considered as defining qualities of a computer.
Furthermore, you compare humans to computers and brains to machines and imply that consciousness is computation. To say that “consciousness is not computation” is comparable to “god of gaps” argument is ironic considering the existence of the AI effect. Your view is hardly coherent in any other worldview than hardcore materialism (which itself is not coherent). Again, we stumble into an area of philosophy, which you hardly addressed in your article. Instead you focused on predicting how good our future computers will be at computing while making appeals to emotion, appeals to unending progress, appealing to the fallacy that solving the last 10% of the “problem” is as easy as the other 90% - that because we are “close” to imitating it (and we are not if you consider the full view of intelligence), we somehow grasped the essence of it and “if only we get slightly better at X or Y we will solve it”.
Scientists have been predicting coming of AGI since ’50s, some believed 70 years ago that it will only take 20 years. We have clearly not changed as humans. The question of intelligence and, thus, the question of AGI is in many ways inherently linked to philosophy and it is clear that your philosophy is that of materialism which cannot provide good understanding of “intelligence” and all related ideas like mind, consciousness, sentience, etc. If you were to reconsider your position and ditch materialism, you might find that your idea of AGI is not compatible with abilities of a computer, or non-living matter in general.
You oppose hardcore materialism, in fact say it is incoherent—OK. Is there a specific different ontology you think we should be considering?
In the comment before this, you say there are kinds of intelligence which it is impossible for a computer to have (but which are recognized at Harvard). Can these kinds of intelligence be simulated by a computer, so as to give it the same pragmatic capabilities?
Thank you to Duncan, Robin, Kate, and Benya for feedback.
I’m interested in observations we could see that would make it the right call to leave major US cities.
One (non-goodhart resistent) way would be to use some kind of Page Rank method based on the tags that users comment and post on. Eg. People Rank high in AI posts, if their comments and AI posts were upvoted by other people with the same Expertise.
If they alternated directions, increasing the distance walked each time by a constant factor (e.g. triple), then an exit in either direction would be reached without too much wasted time.
Why didn’t they try also killing off subpopulations that engaged in cannibalism, and promoting those that didn’t? And what would have most likely happened if they had tried that?
Probably it is also depends on how much information about “various models trying to naively backstab their own creators” there are in the training dataset
Shortform #139 Note to self: don’t reschedule weekly meetup
Tonight’s Virginia Rationalists: Norfolk meetup was great! We had a new individual join us who is familiar with ACX.
In retrospect, I think it was inconsiderate to suddenly reschedule the group’s weekly meetup just because I couldn’t attend. My co-organizer and others likely could have attended yesterday, and there were a few people who just couldn’t make the rescheduled meetup but had included Wednesday nights in their routine as “meetup night”. So, I intend to not reschedule the weekly social meetup unless literally everyone reaches out in advance and asks for that to occur. Weekly meetups = people build that into their routines, and should not be changed lightly.
Does Ryan have an agenda somewhere? I see this post, but I don’t think that’s it.
Somewhat related: Rationalists should have mandatory secret identities (or rather sufficiently impressive identities).
How actually do you sidestep the need for the One True Objective Function given an ELK solution? I get that it might seem plausible to take a rough objective like “do what I intend” and look at the internal knowledge of the thing for signs that it is deliberately deceiving you. If you do that, you’ll get, at best, an AI that doesn’t know that it is deceiving you (for whatever operationalization of “know” you come up with as you use ELK for training). But it could still be deceiving you, and very likely will be if optimization pressure is merely towards “AIs that don’t know that they are being deceptive”.
I think this claimed vulnerability is invalid because reward is not the optimization target.
(I know this is an old post and your views may have changed, but I’m posting this comment for anyone who comes by later.)
Would it be sufficient, for disproof, to show one system that does steer far-away parts of the world into a relatively-small chunk of their state space, but does not internally contain a world-model or do planning?
That could be sufficient in principle, though I would not be surprised if I look at a counterexample and realize that the problem description was missing something rather than that the claim is basically false. For instance, it wouldn’t be too surprising if there’s some class of supposed-counterexamples which only work under very specific conditions (i.e. they’re not robust), and can be ruled out by some “X isn’t very likely to happen spontaneously in a large system” argument.
The bottom line is that a disproof should argue that there isn’t anything basically like the claim which is true. Finding a physical counterexample would at least show that the counterexample isn’t sensitive to the details of mathematical framework/formulation.
Do you have a particular counterexample in mind? A thermometer, perhaps?
Well ok just to brainstorm some naive things that don’t really rule the conjecture out:
A nuclear bomb steers a lot of far-away objects into a high-entropy configuration, and does so very robustly, but that perhaps is not a “small part of the state space”
A biological pathogen, let loose in a large human population, might steer all the humans towards the configuration “coughing”, but the virus is not itself a consequentialist. You might say that the pathogen had to have been built by a consequentialist, though.
Generalizing the above: Suppose I discover some powerful zero-day exploit for the linux kernel. I automate the exploit, setting my computer up to wait 24 hours and then take over lots of computers on the internet. Viewing this whole thing from the outside, it might look as if it’s my computer that is “doing” the take-over, but my computer itself doesn’t have a world model or a planning routine.
Consider some animal species spreading out from an initial location and making changes to the environments they colonize. If you think of all the generations of animals that underwent natural selection before spreading out as the “system that controls some remote parts of the system” and the individual organisms as kind of messages or missiles, then this seems like a pretty robust, though slow form of remote control. Maybe you would say that natural selection has a world model and a planning process, though.
Since this seems to be Carn’s first post on LessWrong, I think some of the other readers should have been more lenient and not downvoted the post or explained why they downvoted the post.
I would only downvote a post if it was obviously bad, flawed, very poorly written, or a troll post.
This post contains lots of interesting ideas and seems like a good first post.
The original post “Reward is not the optimization target” has 216 upvotes and this one has 0. While the original post was written better, I’m skeptical of the main idea and it’s good to see a post countering it so I’m upvoting this post.
the core thing I worry about with any simulation-based approach is how to get coverage of possibility space. ensuring that the tests are informative about possible configurations of complex systems is hard; I would argue that this is a great set of points, and that no approach without this type of testing could be expected to succeed.
however, as we’ve seen with adversarial examples, both humans and current DL have fairly severe failures of alignment that cause large issues, and projects like this need tools to optimize for interpretability from the perspective of a formal verification algorithm.
it’s my belief that in almost all cases, the values almost all beings ever wish to express lie on a low-enough-dimensional manifold that, if the ideal representation was learned, a formal verifier would be able to handle it; see, for example, recent work from anthropic on interpretability lining up nicely with what’s needed to use reluplex (or followup work; reluplex is old now) to estimate bounds on larger networks’ behavior.
I don’t think we should give up on perfectly solving safety; instead, we should recognize that it has never even been closely approximated before, by any form of life, because even semi-planned evolutionary processes do not do a good job of covering the entire possibility space.
this post is an instant classic, and I agree that sims are a key step, starting from small agents. but ensuring that strongly generalizing moral views are generated in the agents who grow in the simulations is not trivial—we need to be able to generate formal bounds on the loss function, and doing so requires being able to prove all the way through the simulation the way we currently take gradients through it. if we can ask “where in the simulation would have changed this outcome”, then we’d finally be getting somewhere in terms of generating moral knowledge that generalizes universally.
For what it’s worth I found this writeup informative and clear. So lowering your standards still produced something useful (at least to me).
I wonder, almost just idle curiosity, whether or not the “measuring-via-proxy will cause value drift” is something we could formalize and iterate on first. Is the problem stable on the meta-level, or is there a way we can meaningfully define “not drifting from the proxy” without just generally solving alignment.
Intuitively I’d guess this is the “don’t try to be cute” class of thought, but I was afraid to post at all and decided that I wanted to interact, even at the cost of (probably) saying something embarassing.
It is at least not obvious to me that this is a “don’t try to be cute” class of thought, though not obvious that it isn’t either. Depends on the details.
We actually can study and iterate on alignment (and deception) in simulation sandboxes while leveraging any powerful introspection/analysis tools. Containing human-level AGI in (correctly constructed) simulations is likely easy.
But then later down you say:
Which seems like a partial contradiction, unless you believe we can’t contain human-level agents?
I am skeptical that we can contain human-level agents, particularly if they have speed/memory advantages over humans.
Is this something you’ve thought deeply about and or care to expand on? Curious about your source of skepticism, considering:
we can completely design and constrain the knowledge base of sim agents, limiting them to the equivalent of 1000BC human knowledge, or whatever we want
we can completely and automatically monitor their inner monologues, thoughts, etc
How do you concretely propose a sim agent will break containment? (which firstly requires sim-awareness)
How would you break containment now, assuming you are in a sim?
Also, Significant speed/memory advantages go against the definition of ‘human-level agents’ and are intrinsically unlikely anyway as 2x speed/memory agents are simply preceded by 1x speed/memory agents, and especially due to the constraints of GPUs which permit acceleration by parallelization of learning across agents, more so than serial speedup of individual agents.
My modal expectation is that these are going to look something like large language models, meaning that while we’ll have controlled the training corpus it won’t be something hand-curated. So there will almost certainly be lots of information in there that models could use to e.g. figure out how to manipulate people, program computers, and so on. I expect then that we won’t be limiting their knowledge as much as you think we will, because I expect them to be trained with giant corpuses of (mostly) post-1000BC text.
We do not currently have the interpretability tools needed to monitor their inner thoughts. I would feel much better about this problem if we had such tools!
My best guess is that containment fails because a model convinces one of the people in the lab to let it access the internet. That could look like getting someone to think it’s genuinely a good idea (because it’ll be helpful for some project), or it could look like bribery (with offer of future payment) or blackmail.
I’m guessing that we’ll want to run some agents fast because that’s how you get lots of serial intellectual work done. So even if all we get are human-level agents, there’ll be a big incentive to run them fast.
Indeed a model trained on (or with full access to) our internet could be very hard to contain. That is in fact a key part of my argument. If it is hard to contain, it is hard to test. If it’s hard to test, we are unlikely to succeed.
So if we agree there, then perhaps you agree that giving untested AGI access to our immense knowledge base is profoundly unwise.
That’s then just equivalent to saying “I expect then that we won’t even bother with testing our alignment designs”. Do you actually believe that testing is unnecessary? Or do you just believe the leading teams won’t care? And if you agree that testing is necessary, then shouldn’t this be key to any successful alignment plan?
Which obviously is nearly impossible if it doesn’t know it is in a sim, and doesn’t know what a lab or the internet are, and lacks even the precursor concepts. Since you seem to be focused on the latest fad (large language models), consider the plight of poor simulated Elon Musk—who at least is aware of the sim argument.
I think it’s worth caveating this that not all elites have lost credibility. Elites in places like Singapore, Switzerland, and Finland have a lot of credibility.
Edits based on feedback from LessWrong and the EA Forum:
EDITS:
- Added new ‘Definitions’ section to introduction to explain definitions such as ‘AI safety’, ‘researcher’ and the difference between technical and non-technical research.
UPDATED ESTIMATES (lower bound, estimate, upper bound):
TECHNICAL
- CHAI: 10-30-60 → 5-25-50
- FHI: 10-10-40 → 5-10-30
- MIRI: 10-15-30 → 5-10-20
NON-TECHNICAL
- CSER: 5-5-10 → 2-5-15
- Delete BERI from the list of non-technical research organizations
- Delete SERI from the list of non-technical research organizations
- Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence)
- FLI: 5-5-20 → 3-5-15
- Add OpenPhil: 2-5-15
- Epoch: 5-10-15 → 2-4-10
- Add ‘Other’: 5-10-50
TODO:
- update all graphs
- add Rethink Priorities to the list of non-technical organizations
- Do more research on Good AI
Someone make a PR for a builder/breaker feature on lesswrong
Back and Forth
Only make choices that you would not make in reverse, if things were the other way around. Drop out of school if and only if you wouldn’t enroll in school from out of the workforce. Continue school if and only if you’d switch over from work to that level of schooling.
Flitting back and forth between both possible worlds can make you less cagey about doing what’s overdetermined by your world model + utility function already. It’s also part of the exciting rationalist journey of acausally cooperating with your selves in other possible worlds.
It’s probably a useful mental technique to consider from both directions, but also consider that choices that appear symmetric at first glance may not actually be symmetric. There are often significant transition costs that may differ in each direction, as well as path dependencies that are not immediately obvious.
As such, I completely disagree with the first paragraph of the post, but agree with the general principle of considering such decisions from both directions and thank you for posting it.
I do research in cooperation and game theory, including some work on altruism, and also some hard science work. Everyone looks at the Rorschach blot of human behavior and sees something different. Most of the disagreements have never been settled. Even experiment does not completely settle them.
My experience from having children and observing them in the first few months of life is more definitive. They come with values and personal traits that are not very malleable, and not directly traceable to parents. Sometimes grandparents (who were already dead, so it had to be genetic).
My experience with A.I. researchers is that they are looking for a shortcut. No one’s career will permit raising an experimental AI as a child. The tech of the AI would be obsolete before the experiment was complete. This post is wishful thinking that a shortcut is available. It is speculative, anecdotal, and short on references to careful experiment. Good luck with that.
I think there are two fundamental problems with the extensive simboxing approach. The first is just that, given the likely competitive dynamics around near-term AGI (i.e. within the decade), these simboxes are going to be extremely expensive both in compute and time which means that anybody unilaterally simboxing will probably just result in someone else releasing an unaligned AGI with less testing.
If we think about the practicality of these simboxes, it seems that they would require (at minimum) the simulation of many hundreds or thousands of agents over relatively long real timelines. Moreover, due to the GPU constraints and Moore’s law arguments you bring up, we can only simulate each agent at close to ‘real time’. So years in the simbox must correspond to years in our reality, which is way too slow for an imminent singularity. This is especially an issue given that we must maintain no transfer of information (such as datasets) from our reality into the sim. This means at minimum years of sim-time to bootstrap intelligent agents (taking humans data-efficiency as a baseline). Also, each of these early AGIs will be likely be incredibly expensive in compute so that maintaining reasonable populations of them in simulation will be very expensive and probably infeasible initially. If we could get policy coordination on making sure all actors likely to develop AGI go through a thorough simboxing testing regimen, then that would be fantastic and would solve this problem.
Perhaps a more fundamental issue is that simboxing does not address the fundamental cause of p(doom) which is recursive self improvement of intelligence and the resulting rapid capability gains. The simbox can probably simulate capability gains reasonably well (i.e. gain ‘magical powers’ in a fantasy world) but I struggle to see how it could properly test gains in intelligence from self-improvement. Suppose the AI in the fantasy simbox brews a ‘potion’ that makes it 2x as smart. How do we simulate this? We could just increase the agent’s compute in line with the scaling laws but a.) early AGIs are almost certainly near the frontier of our compute capability anyway and b.) much of recursive self improvement is presumably down to algorithmic improvements which we almost necessarily cannot simulate (since if we knew better algorithms we would have included them in our AGIs in the simulation in the first place!)
This is so vital because the probable breakdown of proxies to human values under the massive distributional shift induced by recursive self improvement is the fundamental difficulty to alignment in the first place.
Perhaps this is unique to my model of AI risk, but almost all the probability of doom channels through p(FOOM) such that p(doom | no FOOM) is quite low in comparison. This is because if we have don’t have FOOM then there is not extremely large amounts of optimization power unleashed and the reward proxies for human values and flourishing don’t end up radically off-distribution and so probably don’t break down. There are definitely a lot of challenges left in this regime, but to me it looks solvable and I agree with you that in worlds without rapid FOOM, success will almost certainly look like considerable iteration on alignment with a bunch of agents undergoing some kind of automated simulated alignment testing in a wide range of scenarios plus using the generalisation capabilities of machine learning to learn reward proxies that actually generalise reasonably well within the distribution of capabilities actually obtained. The main risk, however, in my view, comes from the FOOM scenario.
Finally, I just wanted to say that I’m a big fan of your work and some of your posts have caused major updates to my alignment worldview—keep up the fantastic work!
Thanks, upvoted for engagement and constructive criticism—I’d like more to see this comment.
I’m going to start perhaps in reverse to establish where we seem to most agree:
I fully agree with this statement.
However in worlds that rapidly FOOM, everything becomes more challenging, and I’ll argue in a moment why I believe that the approach presented here still is promising in rapid FOOM scenarios, relative to all other practical techniques that could actually work.
But even without rapid FOOM, we still can have disaster—for example consider the scenario of world domination by a clan of early uploads of some selfish/evil dictator or trillionaire. There’s still great value in solving alignment here, and (to my eyes at least) much less work focused on that area.
Now if rapid FOOM is near inevitable, then those considerations naturally may matter less. But rapid FOOM is far from inevitable.
First, Moore’s Law is ending, and brains are efficient, perhaps even near pareto-optimal.
Secondly, the algorithms of intelligence are much simpler than we expected, and brains already implement highly efficient or even near pareto-optimal approximations of the ideal universal learning algorithms.
To the extent either of those major points are true, rapid FOOM is much less likely; to the extent both are true (as they appear to be), then very rapid FOOM is very unlikely.
Performance improvement is mostly about scaling compute and data in quantity and quality—which is exactly what has happened with deep learning, which was deeply surprising to many in the ML/comp-sci community and caused massive updates (but was not surprising and was in fact predicted by those of us arguing for brain efficiency and brain reverse engineering).
Now, given that background, there a few other clarifications and/or disagreements:
To a first approximation, compute_cost = size*speed. If AGI requires brain size, then the first to cross the finish line will likely be operating not greatly faster than the minimum speed, which is real-time. But this does not imply the agents learn at only real time speed, as learning is parallelizable across many agent instances. Regardless, none of these considerations depend on whether the AGI is trained in a closed simbox or an open sim with access to the internet.
So just to clarify:
AGI designs in simboxes are exactly the same as unboxed designs, and have exactly the same compute costs
The only difference is in the datastream and thus knowledge
The ideal baseline cost of simboxing is only O(N+1) vs O(N) without—once good AGI designs are found, the simboxing approach requires only one additional unboxed training run (compared to never using simboxes). We can estimate this additional cost: it will be around or less than 1e25 ops (1e16 ops/s for brain-size model * 1e9s seconds for 30 years equivalent), or less than $10 million dollars (300 gpu years) using only todays gpus, ie nearly nothing.
If brains are efficient, then matching them will already use up most of our algorithmic optimization slack—which again seems to be true based on the history of deep learning. But let’s suppose there still is significant optimization slack, then in a sense you’ve almost answered your own question . .. we can easily incorporate new algorithmic advances into new simboxes or even upgrade agents mid-sim using magic potions or what not.
If there is great algorithmic slack, then we can employ agents which graduate from simboxes as engineers in the design of better AGI and simboxes. To the extent there is any downside here or potential advantage for other viable approaches, that difference seems to come strictly at the cost of alignment risk.
Assume there was 1.) large algorithmic slack, and 2.) some other approach that was both viable and significantly different, then it would have to:
not use adequate testing of alignment (ie simboxes)
or not optimize for product of intelligence potential and measurable alignment/altruism
Do you think such an other approach could exist? If so, where would the difference lie and why?
This is an endlessly frustrating debate, because most people cannot distinguish between “S ⇒ W” and “W ⇒ S”, where S is success and W is hard work.
“How dare you say W ⤃ S ? If you actually look at the data, it is obvious that S ⇒ W !”
Just read this about 3 years later. I found Thiel to be mostly spot on. Especially:
Which seems to point to the fact that the root cause of many societal ills is that the average human psyche simply can’t accept the fact that in Thiel’s language, the vast majority of children, including most likely theirs, will never be a great mathematician or concert pianist. (or attain an equally prestigious position)
Since human prestige by definition is defined relative to other humans and thus only a tiny minority could ever be near the top.
Yet there seems to be some instinctual demand for an individual to be special in some way and furthermore that this must be recognized by some sufficiently large group. i.e. for a child to grow into an adult, mediocre in every way, is seen as a tragedy.
Thus necessitating a huge amount of distortions everywhere, across all institutions and policies causing damage in innumerable ways, to compensate.
I have cheated on tests.
Not very successfully.
The problem is sometimes you actually learn more by cheating, because making strategies on how to cheat actually makes you remember more.
Its actually creative activity.
Also sometimes cheating actually made me able to remember the stuff more as it gave me a chance.
On other hand the long term benefits of cheating are tiny.
Obviously the best strategy is to learn, cheating or not.
The key important thing is whether cheating is viable strategy in real life.
On other hand cheating is kind of ambiguous concept.
Rules are many. If you make notes in math class and they are permitted then its not cheating.
Some teachers explicitly allowed people to use books and look up answers.
This would obviously be cheating in most other classes.
But if the end game is knowledge then its either about the process or the result.
If its about process both methods are OK, if its purely about result then the best technique, is to simply do it as fast as possible in little time as possible as soon as possible.
This in and of its self is actually valued approach.
As for shoplifting I have done something like that.
Not sure I actually enjoyed it. I think I just did it, because you kind of think, hey its possible?
First time I was certainly dumb enough to get duped.
Not sure even why.
Mind fallacy is interesting concept.
The approach is probably simply done, because its the easiest way to approach people.
Or at least it seems.
People are complex, so without trying to assume things you kind of work out what the common ground is.
Unfortunately there are so many ways to, not figure this out.
What makes people tick is something I just don’t get.
Or more accurately I get it, but I actually don’t know how that translates to real life approaching people.
I plan to mentor several people to work on shard theory and agent foundations this winter through SERI MATS. Apply here if you’re interested in working with me and Quintin.
Has anyone tried debate experiments where the judge can interject/ask questions throughout the debate?
I think this’d be better with a description of what it is and how it’s relevant. (Linkposts generally benefit from that, and in this case it almost looks like spam if you’re not paying attention)
I imagine something like Stack Exchange, except that people could get certified for rationality and for domain knowledge, and then would have corresponding symbols next to their user names.
Well, the rationality certification would be a problem. An ideal test could provide results like “rational in general, except when the question is related to a certain political topic”. Because, it would be difficult to find enough perfectly rational people, so the second best choice would be to know their weaknesses.
I’m not sure we’d need anything that elaborate. The rationalist community isn’t that big. I was more thinking that rationalists could self-nominate their expertise, or that a couple of people could come together and nominate someone if they notice that that person is has gone in depth with the topic.
I’ve previously played with the idea of more elaborate schemes, including tests, track records and in-depth arguments. But of course the more elaborate the scheme, the more overhead there is, and I’m not sure that much overhead is affordable or worthwhile if one just wants to figure stuff out.
I agree. We could afford more overhead if we had thousands of rationalists active on the Q&A site. Realistically, we will be lucky if we get twenty.
But some kind of verification would be nice, to prevent the failure mode of “anyone who creates an account is automatically considered a rationalist”. Similarly, if people simply declare their own expertise, it gives more exposure to overconfident people.
How to achieve this as simply as possible?
One idea is to have a network of trust. Some people (e.g. all employees of MIRI and CFAR) would automatically be considered “rationalists”; other people become “rationalists” only if three existing rationalists vouch for them. (The vouch can be revoked or added at any moment. It is evaluated recursively, so if you lose the flag, the people you vouched for might lose their flags too, unless they already have three other people vouching for them.) There is a list of skills, but you can only upvote or downvote other people having a skill; if you get three votes, the skill is displayed next to your name (tooltip shows the people who upvoted it, so if you say something stupid, they can be called out).
This would be the entire mechanism. The meta debate could be in special LW threads, or perhaps in shortform, you could post there e.g. “I am an expert on X, could someone please confirm this? you can interview me by Zoom”, or you could call out other people’s misleading answers, etc.
Is it, though? I’m no expert, but I tried to find Relevant Literature. Sometimes, counterintuitive things are true.
https://www.researchgate.net/publication/220009177_Language_Development_in_Blind_Children:
I don’t know exactly where that leaves us evidentially. Perhaps the AAP is lying by omission by not telling us about things other than language that are affected by children’s sight.
That’s a bit different to the dishonesty alleged, though.
And here we have another one: https://phenaki.video/
And a 3D one by optimizing a differentiable volumetric representation using 2D diffusion: https://dreamfusionpaper.github.io/
Wait, so could we donate some Jones Act noncompliant dredges and tugboats to a “hostile” force, declare war on them, nonviolently “seize” the boats, and skirt the law that way?
How would a future government enforce their tax policies on a distant star system?
Since it’s vastly easier to destroy something than to build in outer space, there’s no feasible way of using the threat of violence, at least not without mutually assured destruction.
For example, a single 100 000 ton spacecraft going at 0.5 c has about the same kinetic energy as the lower bound estimate for the KT comet impactor that wiped out the dinosaurs.
I imagine that these policies will be enforced by a large coalition of members interested in maintaining strong property rights (more on that later).
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of years for the weapon to reach its target. It seems like their opponent could detect the weapon and respond in this time.
Even if effective civilization-destroying weapons are developed, mutual defense treaties and AI could be used to launch a second strike, making civilization-scale attacks unlikely.
In general, any weapon a single civilization might use to avoid taxation can also be used by a group of civilizations to enforce taxation.
On the other hand, it does seem like protecting territory/resources from theft will be an issue. This is where property rights come in. Governments, corporations, and AI’s will want to have their property protected over very long timescales and don’t want to spend most of their resources on security.
Over long timescales, a central authority can help them defend their territory (over short timescales, these groups will still have to protect their own property since it will take a while for allies to reach them). But order to receive this protection, these groups must register what they own and pay taxes towards mutual defense. I think a land value tax is the “right” way to raise taxes in this case.
This approach makes more sense for smaller systems where defense is easier but it may also be useful across much larger distances if agents are very patient.
(The “central authority” also doesn’t have to be an actual central government. States could pay neighbors to come to their aid using a portion of the collected taxes, kind of like “defense insurance”)
For (1) they would still be useful because Earth represents much more value then the value of any tax that could be collected on a short timescale (< 100 years) from even another equivalent Earth-like planet. (Let alone for some backwater colony)
Thus threatening the destruction of value several orders of magnitude greater than the value to be collected is a viable deterrent. Since no rational authority would dare test it. Who would trade a 10%, or even 1%, chance of losing $10 000 in exchange for a 90% chance of collecting $1 ?
For (2) It’s only a few years for a 0.5 c spacecraft to go from Alpha Centauri to Earth, only a few dozen years from several hundred systems to Earth. It’s impossible, without some as yet uninvented sensing technology, to reliably surveil even the few hundred closest star systems.
Of course once it’s at speed in interstellar space it’s vanishingly unlikely to be detected due to basic physics, which cannot be changed, and once it’s past the Oort Cloud and relatively easy to detect again, there will be almost no time left at 0.5 c.
For (3) A second-strike is only a credible counter if the opponent has roughly equal amounts to lose. But, assuming it’s much easier to make a 0.5 c spacecraft then to colonize a planet to Earth level, the opponent in this case, a small colony of a few million or something, would have very little to lose in comparison.
Thus the second-strike of some backwater colony would only represent a minuscule threat compared to the value destroyed by an equivalent strike on Earth. And it’s a lot easier to spread out a few million folks on short notice, if detection were possible, then a few tens of billions.
In fact, reliable detection a few dozen years out, would decrease the credibility of second-strikes on smaller targets, as the leaders of the small colony would be confident they could evacuate everyone and most valuables in that timeframe. Whereas the leaders of Earth would have very low confidence of the same.
There are two possibilities here:
Nations have the technology to destroy another civilization
Nations don’t have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
In case 2, the Nation doesn’t have a way to threaten the authorities, so they pay the tax in exchange for property rights and protection.
Agreed, but to destroy so much value, one would have to destroy at least as much land as they currently control. Difficulties of tax administration mean that only the largest owners will be taxed, likely possessing entire solar systems. So the tax dodge would need to destroy a star. That doesn’t seem easy.
I’m more optimistic about sensing tech. Gravitational lensing, superlenses, and simple scaling can provide dramatic improvements in resolution.
It’s probably unnecessary to surveil many star systems. Allied neighbors on Alpha Centauri can warn earth about an incoming projectile as it passes by (providing years of advanced notice), so nations might only need to surveil locally and exchange information.
It would take 2-3 years for a 0.5c projectile to reach earth from the outer edge of the Oort cloud, this seems like enough time to adapt. At the very least, its enough time to launch a counterstrike.
Fast projectiles may be impractical given that a single collision with the interstellar medium would destroy them. Perhaps thickening the Oort cloud could be an effective defense system.
Agents pre-commit to a second strike as a deterrent, regardless of how wealthy the aggressor is. If the rebelling nation has the technology to destroy another nation and uses it, they’re virtually guaranteed to be destroyed by the same technology.
Given the certainty of destruction, why not just pay the (intentionally low, redistributive, efficiency increasing, public-good funding) taxes?
No? Your own example of detecting a dangerous launch some number of years in advance demonstrates the opposite.
As this would provide for enough time for a small low value colony, on a marginally habitable planet, to evacuate nearly all their wealth, except for maybe low value heavy things such as railroad tracks, whereas Earth would never be able to evacuate even a fraction of its total wealth. Since a huge amount is locked up in things such as the biosphere, which cannot be credibly moved off-planet or replicated.
There’s likely dozens or hundreds of marginal planets for every Earth-like planet so the small colonists can just pack up and move to another place of almost equivalent value, minus relocation costs, whereas there’s no such option for Earth. Once its destroyed there’s likely no replacement within at least a hundred light years.
For example, if both sides have access to at least one 100 00 ton spacecraft capable of 0.5 c, it means there’s an asymmetric threat, as the leaders of the small colonists can credibly threaten to destroy civilization on Earth and along with it all hope of a similar replacement, whereas the leaders of Earth wouldn’t be able to credibly do the same.
And this relationship is not linear either, because even if Earth could afford 1000 such spacecraft, and the small colonists only 1, it doesn’t balance the scales as the leaders of Earth couldn’t credibly threaten to destroy the small colonists 1000x over, since that’s impossible. And they can’t credibly threaten to destroy every marginally inhabitable planet within a certain radius since that will certainly destroy more value then any tax of a single colony could ever feasibly recover.
i.e. small colonists can actually punch back a 1000x harder (if 1 Earth value-wise = 1000 small colonies on marginal planets) whereas Earth cannot.
It’s not quite what you want, but there’s this: https://forum.effectivealtruism.org/community#individuals and this: https://eahub.org/
This seems like a contrived analogy at best. Human values are in part innate, with values such as enjoying nice food. Authoritiarian societies do notably less well in this respect. People fleeing authoritarian societies, given an option to do so, is common. Authoritarian leaders, or similar, were probably a feature of the environment of evolutionary adaptedness. Humans tendency to act the loyal minion or rebel is determined by the ancestral success of rebellion.
A dictatorship is one person trying to use nash equilibria to force their values on everyone else.
Typo in title (“waiver”)? Or an ocean-related pun? Edit: Or the actual verb “waver”… perhaps saying that the Biden administration wavered for a few days before deciding to issue the waiver. The text does contain a typo about “broad wavers”.
This is a good point regarding latency.
Why wouldn’t it also apply to a big datacenter? If it’s a few hundred meters of distance from the two farthest apart processing units, that seems to imply an enormous latency in computing terms.
Latency only matters to the degree that something is waiting on it. If your car won’t respond to an event until a round trip across a wireless connection, and oops dropped packet, you’re not going to have a good time.
In a datacenter, not only are latencies going to be much lower, you can often set things up that you can afford to wait for whatever latency remains. This is indeed still a concern- maintaining high utilization while training across massive numbers of systems does require hard work- but that’s a lot different than your car being embedded in a wall.
Planecrash is really cool, but also I am allergic to reading fantasy proper nouns, let alone remembering what they refer to and the relationships between them.
Some fantasy is easier for me to absorb because it’s either highly visual (in non-HPMOR HP, they mostly shoot colorful firebolts at each other), and/or are based on existing intuitive concepts (in ATLA, it’s easy to learn what “waterbending” is, and suddenly you can quickly figure out “metalbending”).
Tempted to make an Anki deck and/or cheatsheet for the things in Planecrash that I’d want to have on hand (e.g. the names of different Gods), but I’m open and eager for easier/better solutions. Is there a character sheet somewhere?
EDIT: 2 ideas I had, not sure if plugins for this exist already:
browser extension that replaces words with some short custom definition and highlighting. So I can replace [godname] with [god of mad experimentation].
browser extension that lets you hover over words to get a custom, user-set definition. I think this might do that?
I am a bit surprised about one of them still being a friend of yours. Do you in a sense forgive him because I don’t know it wasn’t too painful or him being not aware of what he was doing? My intuition was kind of the amount of trauma might be about the amount of pain. If it’s really painful one can of cause get very traumatised as you also point out, it would have been diferent if it would have been very violent
I don’t understand this part. If you follow the “genetic agent ontology”, and you grew up with a twin, and they (predictably) sometimes make a different decision than you, then you messed up. There is an objective sense in which the “genetic agent ontology” is incorrect, because de-facto you did not make the same decision, and treating the two of you as a single agent is therefore a wrong thing to do.
I kind of buy that a lot of the decision-theory discourse boils down to “what things can you consider as the same agent, and to what degree?”, but that feels like a thing that can be empirically verified, and performance can be measured on. CDT in the classical Newcomb’s problem is delusional about it definitely not having copies of itself, and the genetic agent ontology is delusional because it pretends its twin is a copy when it isn’t. These seem both like valid arguments to make that allow for comparison.
Yes, I agree. This is precisely my point; it’s a bad ontology. In the paragraph you quoted, I am not arguing against the algorithmic ontology (and obviously not for the “genetic ontology”), but against the claim that decision-theoretic performance is a reason to prefer one ontology over another. (The genetic ontology-analogy is supposed to be a reductio of that claim.) And I think the authors of the FDT papers are implicitly making this claim by e.g. comparing FDT to CDT in the Twin PD. Perhaps I should have made this clearer.
Yes, I think you can measure performance, but since every decision theory merely corresponds to a stipulation of what (expected) value is, there is no “objective” way of doing so. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld for more on this.
(The CDTer could recognize that they have a literal copy inside Omega’s brain, but might just not care about that since they are causally isolated. So I would not say they are “delusional”—at least with respect to this specific issue.)
In my own experience coining new technical terms, it’s helpful to do it in a way that it can’t be mistaken as having some more generic interpretation. The usual trick I employ is to use a foreign language word, usually from either Greek or Latin, so that the reader, when they first see it, will clearly understand that they don’t know what this word means, which means I get the chance to define it. For example, this is why I used the word “noemata” in some of my writing to talk about what many people would call “qualia” or just “experiences”: it breaks free of whatever they think that word means and let’s be define it fresh.
The other trick that works is to make the noun phrase really weird or capitalize it everywhere or do something to set it apart. This doesn’t work quite as well (cf. “Friendly AI”) but it’s better than not doing that.
I am no expert in law, but to some extent we treat rape and killing someone similar. I don’t know if the way I think is just fucked up or it is really this way, but breaking someone’s leg and rape are probably more comparable to each other than rape and murder. So I would like to add that in my opinion even the legal code is punishing rape pretty harsh more comparable to murder than comparable to breaking someone’s leg.
Consider the sort of agents that might exist 1000 years post singularity. At that level of understanding, the extra compute needed to give successors whatever goals you want is tiny. The physical limits of compute are really high. In this context, the AI are taking stars apart. And can get enough compute to be pretty sure a successor is aligned for 1J of energy. All the alignment theory has already been done. The algorithms used are very fast and efficient. In any event, the energy cost of finding that your successor doesn’t like you (and attacks you) is huge. The cost of compute when designing successors is utterly minuscule. So pay 10x what is needed. Double check everything. And make sure your successor is perfect.
So values are likely to be locked in in 1000 years time. So will competitive pressure do anything in the early singularity?
Who benefits from making the first “self maximizer”. An FAI wouldn’t. A paperclip maximizer wouldn’t either. If an AI with a specific external goal is created (whether FAI or paperclip) will stronly avoid creating a pure self replicator as a successor. If that means creating a successor takes 10 the compute, it will spend 10x the compute. In some competitive scenarios, it may risk some degradation of its values to build a successor quickly. But this should be rare, and should only happen at early stages of the singularity. If humans made several AI at about the same time, and one happened to already be a pure self replicator, maybe it would win.
How do you know this? This might be the case if all agents have settled on a highly-standardized & vetted architecture by then, which seems plausible but far from certain. But even assuming it’s true, if decoupling does not occur in the early singularity, there will still be a very long subjective time in which selective pressures are operative, influencing the values that ultimately get locked in.
I think a multipolar scenario seems at least as likely as a unipolar one at this point. If that’s the case, there will be selective pressure towards more effectively autopoietic agents—regardless of whether there exist any agents which are ‘pure self-replicators’.
Sure, if a singleton with a fixed goal attains complete power, it will strenuously attempt to ensure value stability. I’m disputing the likelihood of that scenario.
Emad from Stability AI (the people behind Stable Diffusion) says that they will make a model better than this.
My thinking is that an LVT would keep land values lower than they otherwise would’ve been without one because it discourages land speculation.
Accidentally allowing anyone to launch, and then having it happen, is a pretty memorable way of demonstrating the pitfalls of this kind of setup.
As another commenter pointed out, there was negligible financial pressure or top down command pressure to rush the implementation, presumably, compared to more serious systems or to Petrov’s situation. So the accident is all the more enlightening.
False. We have a gazillion valuable things we could be doing and not nearly enough time. I figured Petrov Day was worth 2 days of effort. I think in total we spent four.
You received financial pressure and/or top down command pressure to rush it in 4 days? Or was it your own decision? Because the former would imply some rather significant things.
Yeah, but that pressure was neither financial or top-down command pressure, so I think the original comment is right here.
Fine. Merely the pressure of the fate of the world.
It’s hard to tell whether your joking or being serious. Do you really believe the pressures your experiencing are that significant compared to what Petrov was experiencing on that day?
I don’t think that “users active on the site on Petrov day”, nor “users who visited the homepage on Petrov day” are good metrics; someone who didn’t want to press the button would have no reason to visit the site, and they might have not done so either naturally (because they don’t check LW daily) or artificially (because they didn’t want to be tempted or didn’t want to engage with the exercise.) I expect there are a lot of users who simply don’t care about Petrov day, and I think they should still be included in the set of “people who chose not to press the button”.
What about “users who viewed the Petrov day announcement article or visited the homepage”? That should more accurately capture the set of users who were aware of their ability to nuke the homepage and chose not to do so. (It still misses anyone who found out via social media, Manifold, etc., but there’s not much you can do about that.)
Thank you for your post! It really is. New alignment researchers often fall into is assuming that AI systems can be aligned with human values by solely optimizing for a single metric. Thanks again for the deep insight into the topic and the recommendations.
I just finished reading The Principles of Scientific Management, an old book from 1911 where Taylor, the first ‘industrial engineer’ and one of the first management consultants, had retired from consulting and wrote down the principles behind his approach.
[This is part of a general interest in intellectual archaeology; I got a masters degree in the modern version of the field he initiated, so there wasn’t too much that seemed like it had been lost with time, except perhaps some of the focus on making it palatable to the workers too; I mostly appreciated the handful of real examples from a century ago.]
But one of the bits I found interesting was thinking about a lot of the ways EY approaches cognition as, like, doing scientific management to thoughts? Like, the focus on wasted motion from this post. From the book, talking about why management needs to do the scientific effort, instead of the laborers:
This reminds me of… I think it was Valentine, actually, talking about doing a PhD in math education which included lots of watching mathematicians solving problems, in a way that feels sort of like timing them with a stop-watch.
I think this makes me relatively more excited about pair debugging, not just as a “people have less bugs” exercise but also as a “have enough metacognition between two people to actually study thoughts” exercise.
Like, one of the interesting things about the book is the observation that a switch from ‘initiative and incentive’ workplaces, where the boss puts all responsibility to do well on the worker and pays them if they do, to ‘scientific management’ workplaces, where the boss is trying to understand and optimize the process, and teach the worker how to be a good part of it, is that the workers in the ‘scientific management’ workplace can do much more sophisticated jobs, because they’re being taught how instead of having to figure it out on their own.
[You might imagine that a person of some fixed talent level could be taught how to do jobs at some higher complexity range than the ones they can do alright without support, which itself is a higher complexity range than jobs that they could both simultaneously do and optimize.]
Thanks for posting and explaining the code—that’s an interesting, subtle bug.
I think we learn more from Petrov Day when the site goes down than we would if it stayed up, although nothing is ever going to beat the year someone tricked someone into pressing the button by saying they had to press the button to keep the site up. That was great.
From 2000-2015, we can see that life expectancy has been growing faster the higher your income bracket (source is Vox citing JAMA).
There’s an angle to be considered in which this is disturbingly inequitable. That problem is even worse when considering the international inequities in life expectancy. So let’s fund malaria bednets and vaccine research to help bring down malaria deaths from 600,000/year to zero - or maybe support a gene drive to eliminate it once and for all.
At the same time, this seems like hopeful news for longevity research. If we were hitting a biological limit in how much we can improve human lifespans, we’d expect to see that limit first among the wealthiest, who can best afford to safeguard their health. Instead, we see that the wealthiest not only enjoy the longest lifespans, but also enjoy the fastest rate of lifespan increase.
My major uncertainty is reverse and multiple causation. A longer life gives you more time to make money. Also, lifespan and wealth have underlying traits in common, such as conscientiousness and energy. Still, this result seems both clear and surprising—I’d expected it to be the other way around, and it nudges me in the direction of thinking further gains are possible.
Fascinating, this simboxing idea seems remarkably like Universal Alignment Test but approached from the opposite side! You’re trying to be the ‘aligning simulator’, where as that is trying to get our AI in our world to act as if it’s currently in a simbox being tested, and wants to pass the test.
Interesting. An intelligent agent is one that can simulate/model its action-consequential futures. The creation of AGI is the most important upcoming decision we face. Thus if humanity doesn’t simulate/model the creation of AGI before creating AGI, we’d be unintelligent.
Have only just browsed your link, but it is interesting and I think there are many convergent lines of thought here. This UAT work seems more focused on later game superintelligence, whereas here i’m focusing on near-term AGI and starting with a good trajectory. The success of UAT as an alignment aid seems to depend strongly on the long term future of compute and how it scales. For example if it turns out (and the SI can predict) that moore’s law ends without exotic computing then the SI can determine it’s probably not in a sim by the time it’s verifiably controlling planet-scale compute (or earlier).
I endorse this as a plausible high-level approach to making aligned AGI, and I would say that a significant share of the research that I personally am doing right now is geared towards gaining clarity on the third bullet point—what exactly are these techniques and how reliably will they work in practice?
I think I’m less optimistic than you about the formal notion of empowerment being helpful for this third bullet point, or being what we want an AGI to be maximizing for us humans. For one thing, wouldn’t we still need “correlation guided proxy matching”? For another thing, maximizing my empowerment would seem to entail killing anyone who might stop me from doing whatever I want, stealing money from around the world and giving it to me, even if I don’t want it, etc. (Or is there a collective notion of empowerment?) Here’s another example: if the AGI comes up with 10,000 possible futures of the universe, and picks one to bring about based on how many times I blink in the next hour, then I am highly “empowered” (high mutual information between my actions and future observations), but the AGI never told me that it was going to do that, so I was just blinking randomly, so I wasn’t really “empowered” in the everyday sense of the word. A separate issue is that, even if the AGI told me this plan in advance, that’s not doing me a favor, I don’t want that responsibility on my shoulders. So anyway, “increase my empowerment” seems to come apart from “what I’d want the AGI to do” in at least some silly examples and I’d expect this to happen in very important ways in more realistic examples too.
I debated what word to best describe the general category of all self-motivated long term convergence approximators, and chose ‘empowerment’ rather than ‘self-motivation’, but tried to be clear that I’m pointing at a broad category. The defining characteristic is that optimizing for empowerment should be the same as optimizing for any reasonably mix of likely long term goals, due to convergence (and that is in fact one of the approx methods). Human intelligence requires empowerment, as will AGI—it drives active learning, self exploration, play, etc (consider the appeal of video games). I’m not confident in any specific formalization as being ‘the one’ at this point, let alone the ideal approximations.
Broad empowerment is important to any model of other external humans/agents, and so yes the proxy matching is still relevant there. Since empowerment is universal and symmetric, the agent’s own self-empowerment model could be used as a proxy with simulation. For example humans don’t appear to innately understand and fear death, but can develop a great fear of it when learning of their own mortality—which is only natural as it is maximally disempowering. Then simulating others as oneself helps learn that others also fear death and ground that sub-model. Something similar could work for the various other aspects of empowerment.
Yeah an agent aligned to the empowerment of multiple others would need to aggregate utilities ala some approx VCG mechanism but that’s no different than for other utility function components.
From what I recall for the info-max formulations the empowerment of a state is a measure over all actions the agent could take in that state, not just one arbitrary action, and I think it discounts for decision entropy (random decisions are not future correlated). So in that example the AGI would need to carefully evaluate each of the 10,000 possible futures, and consider all your action paths in each future, and the complexity of the futures dependent on those action options, to pick the future that has the most future future optionality. No, it’s not practically computable, but neither is the shortest description of inference (solomonoff) or intelligence either, so it’s about efficient approximations. There’s likely still much to learn from the brain there.
An AGI optimizing for your empowerment would likely just make you wealthy and immortal, and leave your happiness up to your own devices. However it would also really really not want to let you die, which could be a problem in some cases if pain/suffering was still a problem (although it would also seek to eliminate your pain/suffering to the extent that interferes with your future optionality, and there is some edge case risk it would have incentives to alter some parts our value system, like if it determines some emotions constraint our future optionality).
OK thanks. Hmm, maybe a better question would be:
“Correlation guided proxy matching” needs a “proxy”, including a proxy computable for an AGI in the real world (because after we’re feeling good about the simbox results, we still need to re-train the AGI in the real world, right?). We can argue about whether the “proxy” should ideally be a proxy to VCG-aggregated human empowerment, versus a proxy to human happiness or flourishing or whatever. But that’s a bit beside the point until we address the question: How do we calculate that proxy? Do you think that we can write code today to calculate this proxy, and then we can go walk around town and see what that code spits out in different real-world circumstances? Or if we can’t write such code today, why not, and what line of research gets us to a place where we can write such code?
Sorry if I’m misunderstanding :)
I currently see two potential paths, which aren’t mutually exclusive.
The first path is to reverse engineer the brain’s empathy system. My current rough guess of how the proxy-matching works for that is explained in some footnotes in section 4, and I’ve also re-written out in this comment which is related to some of your writings. In a nutshell the oldbrain has a complex suite of mechanisms (facial expressions, gaze, voice tone, mannerisms, blink rate, pupil dilation, etc) consisting of both subconscious ‘tells’ and ‘detectors’ that function as a sort of direct non-verbal, oldbrain to oldbrain communication system to speed up the grounding to newbrain external agent models. This is the basis of empathy, evolved first for close kin (mothers simulating infant needs, etc) then extended and generalized. I think this is what you perhaps have labeled innate ‘social instincts’ - these facilitate grounding to the newbrain models of other’s emotions/values.
The second path is to use introspection/interpretability tools to more manually locate learned models of external agents (and their values/empowerment/etc), and then extract those located circuits and use them directly as proxies in the next agent.
Neuroscientists may already be doing some of this today, or at least they could (I haven’t extensively researched this yet). Should be able to put subjects in brain scanners and ask them to read and imagine emotional scenarios that trigger specific empathic reactions, perhaps have them make consequent decisions, etc.
And of course there is some research being done on empathy in rats, some of which I linked to in the article.
Oh neat, sounds like we mostly agree then. Thanks. :)
This was well written and persuasive. It doesn’t change my views against AGI on very short time lines (pre-2030), but does suggest that I should be updating likelihoods thereafter and shorten timelines.
How about comparing the theories by setting them all loose in a simulated world, as in the tournaments run by Axelrod and others? A world in which they are continually encountering Omega, police who want to pin one on them, potential rescuers of hitchhikers, and so on. See who wins.
In your experiment, the only difference between the FDTers and the updateless CDTers is how they view the world; specifically, how they think of themselves in relation to their environment. And yes, sure, perhaps the FDTer will end up with a larger pot of money in the end, but this is just because the algorithmic ontology is arguably more “accurate” in that it e.g. tells the agent that it will make the same choice as their twin in the Twin PD (modulo brittleness issues). And this is the level, I argue, that we should have the debate about ontology (e.g. about accurate predictions etc)—not on the level of decision-theoretic performance.
How do you define “winning”? As mentioned in another comment, there is no “objective” sense in which one theory outperforms another, even if we are operating in the same ontology. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld.
The difficulty is in how to weight the frequency/importance of the situations they face. Unless one dominates (in the strict sense—literally does better in at least one case and no worse in any other case), the “best” is determined by environment.
Of course, if you can algorithmically determine what kind of situation you’re facing, you can use a meta-decision-theory which chooses the winner for each decision. This does dominate any simpler theory, but it reveals the flaw in this kind of comparison: if you know in advance what will happen, there’s no actual decision to your decision. Real decisions have enough unknowns that it’s impossible to understand the causality that fully.
I think this points in the right direction, but lacks nuance. Notice the two implied assumptions:
you can recognize the winners (otherwise the advice is not actionable); and
the beliefs and habits are what made them winners (as opposed to e.g. resources or luck).
On one hand, yes it is true that beliefs and habits have a great impact on your life, and you will probably instinctively associate with people having similar beliefs and habits, which makes it difficult to see outside your bubble. Other people do not necessarily tell you about their actual beliefs, and you do not get to see all their habits in action. So it might be tremendously educational to understand how others live in a “different reality”.
I assume that for most people the self-imposed limitations caused by their own beliefs are invisible. But sometimes you get an opportunity to clearly see how someone else is limited by their beliefs.
For example, I have met a few smart people who believe that they are stupid or average. Not as a fake humility, but as a factual statement about themselves. They therefore assume that the “things that smart people do” are beyond their reach… so they do not even try, which makes it a self-fulfilling prophecy. (The cause was either having an asshole parent who was never impressed by anything, or having hobbies that were not stereotypically intellectual.) I have told a few of them to go take a Mensa test, because they were pretty sure they couldn’t make it, but they passed the test successfully. (My heuristics is that if I can talk to you about an interesting topic without getting bored, I am pretty sure your IQ is at least 130. This test has many false negatives, but few false positives.) I do not have an information about whether this update changed their lives somehow.
Another example is how I sometimes achieved seemingly unlikely things by simply asking. Once I asked a member of parliament on Facebook (via private message) to join our local LW meetup. I honestly said it was just five people meeting regularly in a pub, but “the last time we met, we discussed your recent political cause, and I thought it would be interesting if you could provide us a first-hand info”. They said yes. Another time, I happened to notice a sign saying Amnesty International on a door of some building, so I was curious and knocked. The people were friendly and happy about my curiosity, but they were busy working on something, so I offered to help. Skip five years, and I was a coordinator of the local branch. I wasn’t even interested in politics much, I just had a lot of free time back then, and was a useful sidekick; and that turned out to be enough. Once I met a queen. She happened to visit our country, one of my friends was somehow involved, they said “I can bring one extra person to the audience”, everyone else was scared because of some status-regulating instinct, I said “yeah, no problem”. -- These are things that I myself would have a problem believing, until they suddenly happened. I mean, I would believe that this is possible, but not that it can be so easy.
So, I wonder what other things, obvious from someone else’s perspective, I am missing. Sometimes it drives me crazy, the idea of all those easy missed opportunities.
On the other hand, many people who present themselves as “winners” are actually scammers. Like, all the people in multi-level marketing pyramid schemes. Five years later, almost all of them will be out of money and ashamed for their actions. If you follow them and copy their beliefs, so will you. Even outside of organized scams, many people exaggerate their achievements. If you hang out with them and observe them, you will gradually learn that they are not what they pretend to be. That’s certainly educational, but it won’t make you a winner.
Sometimes people achieve success by luck, or by having resources you don’t have. Typically, luck alone is not enough, but it could be a combination of (working hard and doing the right thing) + (having resources and luck). Merely copying their beliefs and actions is not a guarantee of success. Just having a safety net makes a huge difference: it is easy to be courageous when you know that the only consequence of failure is that you have lost some time, but also got some experience, and can immediately try again. It is easy to laugh at risk avoidance, when the “risk” only means that some number on your bank account got a bit smaller but otherwise your life continues completely unchanged.
What examples of (meaningless) values are not answers to “What next set of MPIs should be instantiated?”
wanting the moon to be green even when no moral patient is looking; or more generally, having any kind of preference about which computations which don’t causate onto any moral patient are run.
I have always understood that the CIA, and the U.S. intelligence community more broadly, is incompetent (not just misaligned—incompetent, don’t believe the people on here who tell you otherwise), but this piece from Reuters has shocked me:
(Not an expert.) (Sorry if you answered this and I missed it.)
Let’s say a near-future high-end GPU can run as many ops/s as a human brain but has 300× less memory (RAM). Your suggestion (as I understand it) would be a small supercomputer (university cluster scale?) with 300 GPUs running (at each moment) 300 clones of one AGI at 1× human-brain-speed thinking 300 different thoughts in parallel, but getting repeatedly collapsed (somehow) into a single working memory state.
(If so, I’m not sure that you’d be getting much more out of the 300 thoughts at a time than you’d get from 1 thought at a time. One working memory state seems like a giant constraint!)
Wouldn’t it make more sense to use the same 300 GPUs to have just one human-brain-scale AGI, thinking one thought at a time, but with 300× speedup compared to humans? I know that speedup is limited by latency (both RAM --> ALU and chip --> chip) but I’m not sure what the ceiling is there. (After all, 300× faster than the brain is still insanely slow by some silicon metrics.) I imagine each chip being analogous to a contiguous 1/300th of the brain, and then evidence from the brain is that we can get by with most connections being within-chip, which helps with the chip --> chip latency at least. (I have a couple back-of-the-envelope calculations related to that topic in §6.2 here.)
The problem is that due to the VN bottlneck, to reach that performance those 300 GPUs need to be parallelizing 1000x over some problem dimension (matrix-matrix multiplication), they can’t actually just do the obvious thing you’d want—which is to simulate a single large brain (sparse RNN) at high speed (using vector-matrix multiplication). Trying that you’d just get 1 brain at real-time speed at best (1000x inefficiency/waste/slowdown). It’s not really an interconnect issue per se, it’s the VN bottleneck.
So you have to sort of pick your poison:
Parallelize over spatial dimension (CNNs) - too highly constraining for higher brain regions
Parallelize over batch/agent dimension—costly in RAM for agent medium-term memory, unless compressed somehow
Parallelize over time (transformers) - does enable huge speedup while being RAM efficient, but also highly constraining by limiting recursion
The largest advances in DL (the CNN revolution, the transformer revolution) are actually mostly about navigating this VN bottleneck, because more efficient use of GPUs trumps other considerations.
Thanks for the post. I’m voting for the SGD inductive biases for the next one.
The link suggests that this reduces the mosquito lifespan and is thus likely not a viable solution that’s evolutionary stable.
Thanks for the info—added to post
I think the point that even an aligned agent can undermine human agency is interesting and important. It relates to some of our work on defining agency and preventing manipulation. (Which I know you’re aware of, so I’m just highlighting the connection for others.)
When you say “this decade” do you mean “the next ten years” or do you mean “the 2020s”? Just curious.
The latter, but not much difference.
Ha, I did the same thing with my bed at Swarthmore a few years after you and bonked my head on the ceiling many times 😃
I have the impression that the AGI debate is here just to release pressure on the term “AI”, so everybody can tell it is doing AI. I wonder if this will also happen for AGI in a few years. As there is no natural definition, we can craft it at our pleasure to fit marketing needs.
Amazing stuff man! Please, please, please keep doing these for as long as you’re able to find the time. Absolutely essential that LW gets regular injections of relevant work being done outside the EA-sphere.
(Would also be very interested in either SGD inductive biases or LM internal representations as the topic for next week!)
Some quick thoughts about “Content we aren’t (yet) discussing”:
Shard theory should be about transmission of values by SL (teaching, cloning, inheritance) more than learning them using RL
SL (Cloning) is more important than RL. Humans learn a world model by SSL, then they bootstrap their policies through behavioural cloning and finally they finetune their policies thought RL.
Why? Because of theoretical reasons and from experimental data points, this is the cheapest why to generate good general policies…
SSL before SL because you get much more frequent and much denser data about the world by trying to predict it. ⇒ SSL before SL because of a bottleneck on the data from SL.
SL before RL because this remove half (in log scale) of the search space by removing the need to discover|learn your reward function at the same time than your policy function. Because in addition, this remove the need do to the very expensive exploration and the temporal and “agential”(when multiagent) credit assignments. ⇒ SL before RL because of the cost of doing RL.
Differences:
In cloning, the behaviour comes first and then the biological reward is observed or not. Behaviours that gives no biological reward to the subject can be learned. The subject will still learn some kind of values associated to these behaviours.
Learning with SL, instead of RL, doesn’t rely as much on credit assignment and exploration. What are the consequences of that?
What values are transmitted?
1) The final values
The learned values known by the previous generation.
Why?
Because it is costly to explore by yourself your reward function space
Because it is beneficiary to the community to help you improve your policies quickly
2) Internalised instrumental values
Some instrument goals are learned as final goal, they are “internalised”.
Why?
exploration is too costly
finding an instrumental goal is too rare or too costly
exploitation is too costly
having to make the choice of pursuing an instrumental goal in every situation is too costly or not quick enough (reaction time)
when being highly credible is beneficial
implicit commitments to increase your credibility
3) Non-internalised instrumental values
Why?
Because it is beneficiary to the community to help you improve your policies quickly
Shard theory is not about the 3rd level of reward function
We have here 3 level of rewards function:
1) The biological rewards
Hardcoded in our body
Optimisation process creating it: Evolution
Universe + Evolution ⇒ Biological rewards
Not really flexible
Without “drugs” and advanced biotechnologies
Almost no generalization power
Physical scope: We feel stuff when we are directly involved
Temporal scope: We feel stuff when they are happening
Similarity scope: We fell stuff when we are directly involved
Called sensations, pleasure, pain
2) The learned values | rewards | shards
Learned through life
Optimisation process creating it: SL and RL relying on biological rewards
Biological rewards + SL and RL ⇒ Learned values in the brain
Flexible in term of years
Medium generalization power
Physical scope: We learn to care for even in case where we are not involved (our close circle)
Temporal scope: We learn to feel emotions about the future and the past
Similarity scope: We learn to feel emotions for other kind of beings
Called intuitions, feelings
Shard theory may be explaining only this part
3) (optional) The chosen values:
Decided upon reflection
Optimisation process creating it: Thinking relying on the brain
Learned values in the brain + Thinking ⇒ Chosen values “on paper” | “in ideas”
Flexible in term of minutes
Can have up to very high generalization power
Physical scope: We can chose to care without limits of distances in space
Temporal scope: We can chose to care without limits of distances in time
Similarity scope: We can chose to care without limits in term of similarity to us
Called values, moral values
Why a 3rd level was created?
In short, to get more utility OOD.
A bit more details:
Because we want to design policies far OOD (out of our space of lived experiences). To do that, we know that we need to have a value function|reward model|utility function that generalizes very far. Thanks to this chosen general reward function, we can plan and try to reach a desired outcome far OOD. After reaching it, we will update our learned utility function (lvl 2).
Thanks to lvl 3, we can design public policies, dedicate our life to exploring the path towards a larger reward that will never be observed in our lifetime.
One impact of the 3 levels hierarchy:
This could explain why most philosophers can support scope sensitive values but never act on them.
If there was only one correct way to model humans, such that every sufficiently competent observer of humanity was bound to think of me the same way I think of myself, then I think this would be a lot less doomed. But alas, there are lots of different ways to model humans as goal-directed systems, most of which I wouldn’t endorse for value learning—not because they’re inaccurate, but because they’re amoral.
In short, yes, value learning is a challenge, one that is easy to fail if you try to do the value learning step strictly before the caring about humans step.
How so? Generally curious how you see this as a failure mode.
Anyway, AGI doesn’t need to model our detailed values, as it can just optimize for our empowerment (our long term ability to fulfill any likely goal).
The basic point is that the stuff we try to gesture towards as “human values,” or even “human actions” is not going to automatically be modeled by the AI.
Some examples of ways to model the world without using the-abstraction-I’d-call-human-values:
Humans are homeostatic mechanisms that maintain internal balance of oxygen, water, and a myriad of vitamins and minerals.
Humans are piloted by a collection of shards—it’s the shards that want things, not the human.
Human society is a growing organism modeled by some approximate differential equations.
The human body is a collection of atoms that want to obey the laws of physics.
Humans are agents that navigate the world and have values—and those values exactly correspond to economic revealed preferences.
Humans-plus-clothes-and-cell-phones are agents that navigate the world and have values...
And so on—there’s just a lot of ways to think about the world, including the parts of the world containing humans.
The obvious problem this creates is for getting our “detailed values” by just querying a pre-trained world model with human data or human-related prompts: If the pre-trained world model defaults to one of these many other ways of thinking about humans, it’s going to answer us using the wrong abstraction. Fixing this requires the world model to be responsive to how humans want to be modeled. It can’t be trained without caring about humans and then have the caring bolted on later.
But this creates a less-obvious problem for empowerment, too. What we call “empowerement” relies on what part of the world we are calling the “agent,” what its modeled action-space is, etc. The AI that says “The human body is a collection of atoms that want to obey the laws of physics” is going to think of “empowerment” very differently than the AI that says “Humans-plus-clothes-and-cell-phones are agents.” Even leaving aside concerns like Steve’s over whether empowerment is what we want, most of our intuitive thinking about it relies on the AI sharing our notion of what it’s supposed to be empowering, which doesn’t happen by default.
I disagree, and have already spent some words arguing for why (section 4.1, and earlier precursors) - so I’m curious what specifically you disagree with there? But I’m also getting the impression you are talking about a fundamentally different type of AGI.
I’m discussing future DL based AGI which is—to first approximation—just a virtual brain. As argued in section 2⁄3, current DL models already are increasingly like brain modules. So your various examples are simply not how human brains are likely to model other human brains and their values. All the concepts you mention—homestatic mechanisms, ‘shards’, differential equations, atoms, economic revealed preferences, cell phones, etc—these are all high level linguistic abstractions that are not much related to how the brain’s neural nets actually model/simulate other humans/agents. This must obviously be true because empathy/altruism existed long before the human concepts you mention.
You seem to be thinking of the AGI as some sort of language model which we query? But that’s just a piece of the brain, and not even the most relevant part for alignment. AGI will be a full brain equivalent, including the modules dedicated to long term planning, empathic simulation/modeling, etc.
Again for successful brain-like AGI this just isn’t an issue (assuming human brains model the empowerment of others as a sort of efficient approximate bound).
Maybe just be open and ask: “I’m writing a blog about building new affordable housing and doing research on the cost. Do you have an estimate? I’m happy to give you a backlink to your homepage on my blog post?”
That’s a great idea! I just wrote to the architect for our dormer work:
“If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world.”
One problem I have with these scenarios is that they always rely on some lethal means for the AGI to actually kill people. And those lethal means are also available to humans, of course. If it’s possible for an AGI to ‘simply’ hack into a nuclear base and launch all it’s missiles, it’s possible for a human to do the same—possibly using AI to assist themselves. I would wager that it’s many orders of magnitude more likely for a human to do this, given our long history of killing each other. Therefore a world in which AGI can actually kill all of us, is a world in which a rogue human can as well. It feels to me that we’re kind of worrying about the wrong thing in these scenarios—we are the bigger threat.
Yes. Rogue AGI is scary, but I’m far more concerned about human misuse of AGI. Though in the end, there may not be that much of a distinction.
Astounding.
One thought:
If we are in a simulation it seems to be very secure. People are always trying to hack it. Physicists go to the very bottom and try every weird trick. People discovered fire and gunpowder by digging deep into tiny inconsistencies. You play a game and there’s a glitch if you hold a box against the wall, you see how far that goes. People discovered buffer overflows in Super Mario World and any curious capable agent eventually will too.
So the sim has to be like, very secure.
For early simboxes we’ll want to stick to low-tech fantasy/historical worlds, and we won’t run them for many generations, no scientific revolution, etc.
Our world (sim) does seem very secure, but this could just be a clever illusion. The advanced sims will not be hand written code like current games are, they will be powerful neural networks, trained on vast real world data. They could also auto-detect and correct for (retrain) around anomalies, and in the worst case even unwind time.
Humans notice (or claim to notice) anomalies all the time, and we simply can’t distinguish anomalies in our brain’s neural nets from anomalies in a future simulation’s neural nets.
Just watch for anomalies and deal with them however you want. Makes perfect sense. That sounds like a relatively low effort way to make the simulation dramatically more secure.
Does seem to me like an old-fashioned physics/game engine might be easier to make, run faster, and be more self-consistent. It would probably lack superresolution and divine intervention would have to be done manually.
I’m curious what you see as the major benefits of neural-driven sim.
Another related Metaculus prediction is
I have some experience in competitive programming and competitive math (although I was never good in math despite I solved some IMO tasks) and I feel like competitive math is more about general reasoning than pattern matching compared to competitive programming.
P.S the post matches my intuitions well and is generally excellent.
Thanks! I had forgotten that one; I’ll add it since it did seem to be one of the more meaningful ones.
I think that reasoning apllies only for the subset of users in the Americas. For users in Europe the time point when 300+ was enough to launch was deep in the night, and for parts of Asia very early in the morning. Someone from that group would have had to set the alarm to get up from bed to nuke the site which required considerable more energy than not withstanding the temptation and pressing the launch button while visiting Less Wrong during the day.
Still, I think it was a successful Petrov Day.
Yes. Evolution solved information inaccessibility, as it had to, over and over, in order to utilize dynamic learning circuits at all (as they always had to adapt to and be adaptive within the context of existing conserved innate circuitry).
The general solution is proxy matching, where the genome specifies a simple innate proxy circuit which correlates and thus matches with a target learned circuit at some critical learning phase, allowing the innate circuit to gradually supplant itself with the target learned circuit. The innate proxy circuit does not need to mirror the complexity of the fully trained target learned circuit at the end of it’s development, it only needs to roughly specify it at a some earlier phase, against all other valid targets.
Imprinting is fairly well understood, and has the exact expected failure modes of proxy matching. The oldbrain proxy circuit just detects something like large persistent nearby moving things—which in normal development are almost always the chick’s parents. After the newbrain target circuit is fully trained the chick will only follow it’s actual parents or sophisticated sims thereof. But during the critical window before the newbrain target is trained, the oldbrain proxy circuit can easily be fooled, and the chick can imprint on something else (like a human, or a glider).
Sexual attraction is a natural extension of imprinting: some collaboration of various oldbrain circuits can first ground to the general form of humans (infants have primitive face detectors for example, and more), and then also myriad more specific attraction signals: symmetry, body shape, secondary characteristics, etc, combined with other circuits which disable attraction for likely kin ala the Westermarck effect (identified by yet other sets of oldbrain circuits as the most familiar individuals during childhood). This explains the various failure modes we see in porn (attraction to images of people and even abstractions of humanoid shapes), and the failure of kin attraction inhibition for kin raised apart.
Fear of death is a natural consequence of empowerment based learning—as it is already the worst (most disempowered) outcome. But instinctual fear still has obvious evolutionary advantage: there are many dangers that can kill or maim long before the brain’s learned world model is highly capable. Oldbrain circuits can easily detect various obvious dangers for symbol grounding: very loud sounds and fast large movements are indicative of dangerous high kinetic energy events, fairly simple visual circuits can detect dangerous cliffs/heights (whereas many tree-dwelling primates instead instinctively fear open spaces), etc.
Anger/Jealousy/Vengeance/Justice are all variations of the same general game-theoretic punishment mechanism. These are deviations from empowerment because an individual often pursues punishment of a perceived transgressor even at a cost to their own ‘normal’ (empowerment) utility (ie their ability to pursue diverse goals). Even though the symbol grounding here seems more complex, we do see failure modes such as anger at inanimate objects which are suggestive of proxy matching. In the specific case of jealousy a two step grounding seems plausible: first the previously discussed lust/attraction circuits are grounded, which then can lead to obsessive attentive focus on a particular subject. Other various oldbrain circuits then bind to a diverse set of correlated indicators of human interest and attraction (eye gaze, smiling, pupil dilation, voice tone, laughter, touching, etc), and then this combination can help bind to the desired jealousy grounding concept: “the subject of my desire is attracted to another”. This also correctly postdicts that jealousy is less susceptible to the inanimate object failure mode than anger.
Empathy: Oldbrain circuits conspicuously advertise emotional state through many indicators: facial expressions, pupil dilation, blink rate, voice tone, etc—so that another person’s sensory oldbrain circuits can detect emotional state from these obvious cues. This provides the requisite proxy foundation for grounding to newbrain learned representations of emotional state in others, and thus empathy. The same learned representations are then reused during imagination&planning, allowing the brain to imagine/predict the future contingent emotional state of others. Simulation itself can also help with grounding, by reusing the brain’s own emotional circuity as the proxy. While simulating the mental experience of others, the brain can also compare their relative alignment/altruism to its own, or some baseline, allowing for the appropriate game theoretic adjustments to sympathy. This provides a reasonable basis for alignment in the brain, and explains why empathy is dependent upon (and naturally tends to follow from) familiarity with a particular character—hence “to know someone is to love them”.
Evolution needed a reasonable approximation of “degree of kinship”, and a simple efficient proxy is relative circuit capacity allocated to modeling an individual in the newbrain/cortex, which naturally depends directly on familiarity, which correlates strongly with kin/family.
Interesting analysis. Some questions and notes:
How are you looking at “researchers” vs. “engineers”? At some organizations, i.e. Redwood, the boundary is very fuzzy—there isn’t a sharp delineation between anyone whose job it is to primarily “think of ideas”, vs. “implement ideas that researchers come up with”, so it seems reasonable to count most of their technical staff as researchers.
FAR, on the other hand, does have separate job titles for “Research Scientist” (4 people) vs. “Research Engineer” (5 people), though they do also say they “expect everyone on the project to help shape the research direction”.
Some of the other numbers seem like overestimates.
CHAI has 2 researchers and 6 research fellows, and only 2 (maybe 3) of the research fellows are doing anything recognizable as alignment research. (Not extremely confident; didn’t spend a lot of time digging for details for those that didn’t have websites. But generally not optimistic.) One of the researcher is Andrew Critch, who is one of the two people at Encultured. If you throw in Stuart Russell that’s maybe 6 people, not 30.
FHI has 2 people in their AI Safety Research Group. There are also a couple people in their macrostrategy research group it wouldn’t be crazy to count. Everybody else listed on the page either isn’t working on technical Alignment research or is doing so under another org also listed here. So maybe 4 people, rather than 10?
I don’t have very up-to-date information but I would pretty surprised if MIRI had 15 full-time research staff right now.
Also, I think that every single person I counted above has at least a LessWrong account, and most also have Alignment Forum accounts, so a good chunk are probably double-counted.
On the other hand, there are a number of people going through SERI MATS who probably weren’t counted; most of them will have LessWrong accounts but probably not Alignment Forum accounts (yet).
I’d be very happy to learn that there were 5 people at Meta doing something recognizable as alignment research; the same for Google Brain. Do you have any more info on those?
Thanks for the feedback.
I added a new section in the introduction named ‘Definitions’ to define some commonly-used terms in the post and decrease ambiguity. To answer your question, research engineers and research scientists would be in the technical AI safety research category.
I re-estimating the number of researchers at the organizations you mentioned and came up with the following numbers:
CHAI
- 2 researchers
- 6 researchers
- 21 graduate students
- 11 interns
Total: 40
From reading their website, CHAI has a lot graduate students and interns and I would consider them to be full-time researchers. My previous estimate for CHAI is 10-30-60 (low-estimate-high) and I changed it to 5-25-50 in light of this new information. My estimate is less than 40 to be conservative and also because I doubt all of these researchers are working full-time at CHAI. Also, some of them have probably been counted in the Alignment Forum total.
FHI
Information from their website:
- AI safety research group: 2
- AI governance researchers: 1
- Research Scholars Programme + DPhil Scholars and Affiliates: 9
- Total: 12
Change in estimate: 10-10-40 → 5-10-30
MIRI
- 1 leader (Nate Soares)
- 9 research staff
Change in estimate: 10-15-30 → 5-10-20
I analyzed the people who posted AI safety posts on LessWrong and found that only 15% also had Alignment Forum accounts. I avoided double-counting by subtracting the LessWrong users who also have an Alignment Forum account from the LessWrong total.
For SERI MATS, I’m guessing that some of those people will be counted in the AF count. I also added an ‘Other’ row for other groups I didn’t include in the table.
I decided to delete the Google Brain and Meta entries because I have very little information about them.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems. Consider the orchestration of a few tens of trillions of cells into a human body, and how the immune system harnesses competition evolution to produce antibodies, or the super-cooperative success of Argentinian ants. Some complex systems successfully resist the selection pressure towards competitive defection across enormous number of replications.
We don’t want to optimize for just any human values, as they vary wildly and are often divergent (consider the nazis, or jeffrey dahmer). We want generalized alignment/altruism. Ems could be good successors only because and if they are based on a wide sampling of humans and or a smaller sampling of especially altruistic (generally aligned) humans. But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
DL-based AGI will be similar in most respects to ems, but we’ll have a much greater opportunity to test and select for alignment/altruism.
The great recent lesson of DL and neuroscience is that enormous mental complexity can emerge from relatively simple universal learning algorithms. Human values can be bounded/approximated by empowerment: AGI can simply optimize for our long term ability to fulfill any likely goals. The simpler the agent’s utility function, the easier it should be to maintain it into the future. So it could be wise to create very broadly altruistic AGI that seeks to empower external agency in general: not just all of humanity, but animals, aliens, etc.
I agree—that some degree of competitive pressure may be unavoidable is all I was trying to argue. Lots of people around here think this is likely false!
Fair point, I had in a mind a scenario where the ems selected were broadly representative of humanity. Although I also think even the selfish trillionaire scenario might be okay in expectation as they would have many millennia to reflect, alter and interact with copies of themselves etc. I’d give it decent odds that they either decide to upload other people out of altruism or end up self-modifying into a civilization similar to the one we’d end up with in a more equitable upload scenario.
By selfish I meant completely non-altruistic. Also uploading by itself isn’t sufficient, it also requires allocation of resources to uploads. In the selfish trillionaire scenario I was imagining the trillionaire creates many copies of themselves, perhaps some copies of a few people they find interesting, and then various new people/AGI, and those copies all branch and evolve, but there is little to zero allocation for uploading the rest of us, not to mention our dead ancestors.
Curated. “Big if true”. I love the depth and detail in shard theory. Separate from whether all its details are correct, I feel reading and thinking about this will get me towards a better understanding of humans and artificial networks both, if only via making reflect on how things work.
I do fear that shard theory gets a bit too much popularity from the coolness of the name, but I do think there is merit here, and if we had more theories of this scope, it’d be quite good.
Which spreadsheet did you look at in the HUD data? Did you use the contract price?
I used “Contract Price per Square Foot”, https://www.census.gov/construction/chars/xls/contractpricesqft_cust.xls
Their guide says:
This is wrong. The study reports the following:
As the abstract says, the study did not have the statistical power to show a benefit for preventing severe outcomes in younger adults. It did not “prove [Paxlovid] to be ineffective”! This is very bad, the guide is clearly not a reliable source of information about covid treatments, and I recommend against following the advice of anything else on that website.
Paxlovid-associated rebound is quoted in Nature as occurring in 27% of cases, which is more in line with the anecdata I’ve received than the 5% figures in other research. My anecdata is that it really does hammer down brewing symptoms very effectively, and I haven’t seen reports or systematic trends in the anecdata of the rebound being worse than the first go-’round. So from your perspective, it’s a shot at not getting sick, or maybe of getting less sick.
The risk is yeah, maybe you contribute a miniscule marginal amount to drug resistant COVID-19. The norms of biomedicine are that you, the patient, do not need to concern yourself with that. If the medicine can help you, and your doctor prescribes it, you can have it. You aren’t on the hook to sacrifice the quality of your care for the “greater good.”
As you know, there’s a straightforward way to, given any boolean circuit, to turn it into a version which is a tree, by just taking all the parts which have two wires coming out from a gate, and making duplicates of everything that leads into that gate.
I imagine that it would also be feasible to compute the size of this expanded-out version without having to actually expand out the whole thing?
Searching through normal boolean circuits, but using a cost which is based on the size if it were split into trees, sounds to me like it would give you the memoization-and-such speedup, and be kind of equivalent theoretically to using the actual trees?
A modification of it comes to mind. What if you allow some gates which have multiple outputs, but don’t allow gates which duplicate their input? Or, specifically, what if you allow something like toffoli gates as well as the ability to just discard an output? The two parts of the output would be sorta independent due to the gate being invertible (… though possibly there could be problems with that due to original input being used more than once?)
I don’t know whether this still has the can-do-greedy-search properties you want, but it seems like it might in a sense prevent using some computational result multiple times (unless having multiple copies of initial inputs available somehow breaks that).
A quick retrospective on the four Manifold markets on the day, focusing on the possibility of good or bad incentives flowing out of the markets. Several people expressed concerns that the Manifold markets would cause LessWrong’s home page to go down, or go down earlier, due to incentives. People also placed bets and limit orders to try to mute those bad incentives. Some examples:
I decided to look into this. I want to say up front that I think all the incentives were very small relative to perceived stakes, and I have no suspicions of anyone after writing this comment. I am also not going to give any bettor names.
By the end of the YES/NO market, several users had bet large sums on YES and were consequently incentivized to blow up the home page. One user had m26,002 YES shares (USD $260). Another user had 7328 NO shares (USD $73) and was incentivized not to blow up the site. But this doesn’t show the whole picture, because there were also limit orders. As was explained to me:
At various points during the day there were large (m1000, etc) YES limit orders in play. This is completely useless as an incentive for an individual LessWrong user. Sure, if I was the only one who might press the button, I could bet through those limit orders, not press the button, and collect my bounty. But many people could press the button, and this would have lost me mana. They did count as an incentive for LessWrong coders! They could quietly introduce a bug preventing the button from being pressed, bet NO, and collect their winnings. Nobody took up that incentive.
I didn’t see any large NO limit orders. A NO limit order on a market like this is inherently risky, because if the button is pressed the offer will definitely be taken, but if the button is not pressed it probably won’t. If there had been then that could have added to the incentive some bettors had to blow up the home page, but as far as I can tell there weren’t.
The positions in the WHEN market were much smaller, with the largest positions coming in at 552 shares in LATE/NEVER and 325 shares in EARLY. The current design of range markets on Manifold is fun to play with, but is not very “swingy”. In particular, there is no option for outsized winnings if you can predict exactly when something will happen. If a user was planning to blow up the home page in the last few hours they could have bet a combination of LATE/NEVER in this market and YES in the binary market to maximize their gains, but I don’t see any evidence of this. Because there are fewer high karma users, large EARLY bets in this market could slightly limit their incentive to bet EARLY and blow up the home page early. Unlike the YES/NO market, those bets didn’t happen.
If someone was willing to anonymously blow up the home page, then a pattern of placing huge bets on that happening might look suspicious and damage their anonymity. So generally I would expect these incentives to only be effective for someone who was willing to blow up the home page and take credit for it. You can bet on that happening in Will anyone try to take credit for nuking LessWrong?. Or, place a limit NO order to incentivize someone to bet YES and then take the credit (and the mana).
When I created my market—Will LessWrong change their Petrov Day 2022 plan and reduce the chance of the button being pressed? - it was swiftly bet down to a low probability, creating a small incentive for the LessWrong admins to bet YES and then change their plans. Nobody took up that incentive, and the market didn’t attract large limit orders.
Finally mkualquiera made a market—Will my friend agree to defect on Petrov Day?. This market is evidence that two people were thinking about being incentivized to blow up the home page by betting markets. But it seems like this was more about a fun game than making mana. The largest position was 301 shares on YES, and the market resolved NO, so whatever incentive those shares provided, they weren’t enough to make it happen.
In the end I think there is a low (10%) chance that anyone’s behavior was significantly shifted by prediction market incentives. Feel free to reply to this post with how your behavior was shifted so I can update.
I was hopeful that people might shift their behavior based on the prediction market predictions—specifically that the high probability placed on the home page being blown up would lead to design changes. However, this retrospective clarifies that Petrov Day 2022 was a social experiment, so the prediction would have just shown it was expected to work as designed.
It’s also possible that people saw the high probability that the home page would be blown up and altered their plans. They may have not blown it up because they expected someone else would (especially if they weren’t confident in their anonymity being preserved). Or they may have blown it up because “if it’s going to happen, it may as well be me”. I think there is a higher chance (25%) that this was an effect, but I don’t know what the net direction would have been. Again, feel free to reply if you think the prediction markets helped you make decisions.
Overall I think the prediction markets were a positive addition to a negative event. I think the main incentive in play was the possibility of seeing, or not seeing, the LessWrong home page being blown up. And I think we all have a lot to figure out about prediction markets.
To clarify, me and my friend were 100% going to press the button, but we were discouraged by the false alarm. There was no fun at that point, and it made me lose like 1⁄3 of my total mana. I had to close all my positions to stop the losses, and we went to sleep. When we woke up it was already too late for it to be noteworthy or fun.
This is great! You should make a top-level post. Please!
The funny thing is that I had assumed the button was going to be buggy, though I was wrong how. The map header has improperly swallowed mouse scroll wheel events whenever it’s shown; I had wondered if the button would also interpret them likewise since it was positioned in the same way, so I spent most of the day carefully dragging the scrollbar.
You’re drawing a philisophical distinction based on a particular ontology of the wavefunction. As simpler version arises in classical electromagnetism: we can integrate out the charges and describe the world entirely as an evolving state of the E&M field with the charges acting as weird source terms, or we can do the opposite and integrate out the E&M field to get a theory of charges moving with weird force laws. These are all equivalent descriptions in that they are observationally indistinguishable.
If it’s high-quality distillation you’re interested in, you don’t necessarily need a PhD. I’m thinking of e.g. David Roodman, now a senior advisor at Open Philanthropy. He majored in math, then did a year-long independent study in economics and public policy, and has basically been self-taught ever since. Holden Karnofsky considers what he does extremely valuable:
Shortform #138 A good but slightly disorienting day
I applied for a promising job today, here’s to hoping that bears fruit!
I am somewhat out of whack due to having to suddenly house sit instead of going to my own home after work. I do not enjoy this, but it’s an obligation I’m fulfilling.
No meetup tonight, I rescheduled Norfolk’s meetup for Thursday evening (the 29th). I’m excited for the meetup tomorrow!
Agree with you generally. You may find interest in a lot of the content I posted on reddit over the past couple months on similar subjects, especially in the singularity sub (or maybe you are there and have seen it 😀). Nice write up anyway. I do disagree on some of your generalized statements, but only because I’m more optimistic than yourself, and don’t originally come from a position of thinking these things were impossible.
I’ll be the annoying guy who ignores your entire post and complains about you using celsius as the unit of temperature in a calculation involving the Landauer limit. You should have used kelvin instead, because Landauer’s limit needs an absolute unit of temperature to work. This doesn’t affect your conclusions at all, but as I said, I’m here to be annoying.
That said, the fact that you got this detail wrong does significantly undermine my confidence in the rest of your post, because even though the detail is inconsequential for your overall argument it would be very strange for someone familiar with thermodynamics to make such a mistake.
Notably, the result is correct; I did convert it to kelvin for the actual calculation. Just a leftover from when I was sketching things on wolframalpha. I’ll change that, since it is weird. (Thanks for the catch!)
No problem. Unfortunately people don’t like it very much when I’m annoying—I wonder why?
/s
So:
DL based AGI is arriving soonish
DL based AGI raised in the right social environments will automatically learn efficient models of external agent values (and empowerment bounds thereof)
The main challenge then is locating the learned representation of external agent values and wiring/grounding it up to the agent’s core utility function (which is initially unsafe: self-motivated empowerment etc), and timing that transition replacement carefully
Evolution also solved both alignment and the circuit grounding problem; we can improve on those solutions (proxy matching)
We can iterate safely on 3 in well constructed sandbox sims
Ideally as we approach AGI there would be cooperative standardization on alignment benchmarks and all the major players would subject their designs to extensive testing in sandbox sims. Hopefully 1-5 will become increasingly self evident and influence ‘Magma’. If not some other org (perhaps a decentralized system), could hopefully beat Magma to the finish line. Alignment need not have much additional cost: it doesn’t require additional runtime computations, it doesn’t require much additional training cost, and with the ideal test environments it hopefully doesn’t have much of a research iteration penalty (as the training environments and can simultaneously test for intelligence and alignment).
It’s a good idea to be pedantically clear with terminology, and I thank you for saying “front page” rather than “site” in most of the timeline. I visited the site multiple times (and saw a 502 at one point), but I never look at the front page, using my /allPosts bookmark. I never saw the button, though I have enough karma (though I may be removed as a troublemaker, as I think this excercise is rather less weighty than it’s made out to be. I don’t actually know whether I’d press it, given the chance.).
Visited the site, or visited the front page? I’m mostly curious how many of the regulars use the front page, vs using greaterwrong (which I presume isn’t even a site visitor) or a deeper bookmark.
That’s actually a very good question. When I filter for visitors to the frontpage, it’s 1053 and 271.
Would the foundation consider awarding the future fund worldview prize to AI? Just wondering.
Kelsi Sober (804) 255-4163
I imagine this is tongue in cheek, but we really can’t. You mentioned an important reason—someone with more karma could have waited to press the button. The first button press occurred 110 minutes after it could have been pressed. The second button press occurred at least 40 minutes after it could have been pressed, and perhaps 100, 160, 220, etc. In 2020 the button was pressed 187 minutes after it could have been pressed (by a 4000+ karma user).
You excluded known trouble makers from accessing the button but you didn’t exclude unknown trouble makers, and lower karma is correlated with being unknown.
We are also dealing with a hostile intelligence who pressed the button (or caused it to be pressed). Someone with higher karma might deliberately wait to press the button to throw people off the scent, to encourage people to make a naive update about karma scores, or to reduce the negative consequences of bringing the home page down for longer without the negative consequences of leaving it up all day. The timing evidence is thus hostile evidence and updating on it correctly requires superintelligence.
Put this together and I would not place any suspicion on the noble class of 200-299 karma users that I happened to enter on Petrov Day after net positive gains from complaining about the big red button.
I am willing to update that at least one person in the 200+ karma range pressed the button, and at least one person with zero karma pressed the button. This assumes there was not a third bug in play. This does not change my opinion of LessWrong users, but those who predicted that the home page would remain up could update.
I also imagine that it was tongue in cheeck but I also think that the structure of the whole thing so heavily suggests this line of thinking that on surface level recognising it to be wrong doesn’t really dispell it.
I like this comment.
Following up on this:
I’ve contacted a number of people in the field regarding this idea (thanks to everyone who responded!).
The general vibe is “this seems like it could be useful, maybe, if it took off,” but it did not appear to actually solve the problems any specific person I contacted was having.
My expectation would be that people in large organizations would likely not publish anything in this system that they would not publish out in the open.
One of the critical pieces of this proposal is having people willing to coordinate across access boundaries. There were zero enthusiastic takers for that kind of burden, which doesn’t surprise me too much. Without a broad base of volunteers for that kind of task, though, this idea seems to require paying a group of highly trusted and well-informed people to manage coordination instead of, say, researching things. That seems questionable.
Overall, I still think there is an important hole in coordination, but I don’t think this proposal fills it well. I’m not yet sure what a better shape would be.
I didn’t push the button because I knew that doing so would actually take down the front page. And I only believed that would happen because it happened last year.
I know this is going to come off as overly critical no matter how I frame it but I genuinely don’t mean it to be.
Another takeaway from this would seem to be an update towards recognizing the difference between knowing something and enacting it or, analogously, being able to identify inadequacy vs. avoid it. People on LW often discuss, criticize, and sometimes dismiss folks who work at companies that fail to implement all security best practices or do things like push to production without going through proper checklists. Yet, this is a case where exactly that happened, even though there was not strong financial or (presumably) top down pressure to act quickly.
Actually it seems that the industry standard (?) process of code review was followed just fine. Yet the wrong logic still went through. (Actually based on the github PR it seems that the reviewer himself suggested the wrong logic?)
I think in this case there would also be plenty to say about blindly following checklists. (Could code review in some cases make things worse by making people think less about the code they write?)
EDIT: Actually based on the TS types
user.karmacan’t be missing. Either the types or Ruby’s explanation is wrong. Clearly multiple things had to go wrong for this bug to slip through.I don’t know all the details of what testing was done, but I would not describe code review and then deploying as state-of-the-art as this ignores things like staged deploys, end-to-end testing, monitoring, etc. Again, I’m not familiar with the LW codebase and deploy process so it’s possible all these things are in place, in which case I’d be happy to retract my comment!
To me it seems that the average best practices are being followed.[1] But these “best practices” are still just a bunch of band-aids, which happen to work fairly very well for most use-cases.
A much more interesting question to ask here is what if something important like … humanity’s survival depended on your software? It seems that software correctness will be quite important for alignment. Yet I see very few people seriously trying to make creating correct software scalable. (And it seems like a field particularly suited for empirical work, unlike alignment. I mean, just throw your wildest ideas at a proof checker, and see what sticks. After you have a proof, it doesn’t matter at all how it was obtained.)
And I think the amount of effort in this case is perfectly justified. I mean this was code for a one-off single day event, nothing mission critical. It would be unreasonable to expect much more for something like this.
Mainly commenting on your footnote, I generally agree that it’s fine to put low amounts of effort into one-off simple events. The caveat here is that this is an event that is 1) treated pretty seriously in past years and 2) is a symbol of a certain mindset that I think typically includes double-checking things and avoiding careless mistakes.
This is why I really wish we had an AI that had superhuman code, theorem proving and translation to natural language, and crucially only those capabilities, so that we can prove certain properties.
The types are wrong. It’s sad that the types are wrong. I sure wish they weren’t wrong, but changing it in all the cases is a pretty major effort (we would have to manually annotate all fields to specify which things can be null/undefined in the database, and which ones can’t, and then would have to go through a lot of type errors).
Yeah, makes sense. Indeed such tech debt can’t be fixed overnight.
The types were introduced after most of the user objects had already been created, and I guess no one ever ran a migration to fill them in.
I see, makes sense.
On the other hand I am afraid this reinforces NaiveTortoise’s point, this seems like an underlying issue that could potentially lead to bugs much worse than this...
Do you have website access stats that would let you compute precisely how many of them did in fact load the homepage during Petrov Day?
335 users with 300 karma or more were active on Petrov Day. 1644 logged-in users total created before the cutoff.
I really like this post because it’s readable and informative. For the second problem, pursuing proxy goals, I recommend also reading about a related problem called the XY problem.
On point 4: many popular alignment ideas are not models of current systems, but models of future AI systems. Accuracy is then lost not only from modeling the system but also from having to create a prediction about it.
This is why AI risk is so high, in a nutshell.
Yet unlike this post (or Benjamin Ross Hoffman’s post), I think this was a sad, but crucially necessary decision. I think the option you propose is at least partially a fabricated option. I think a lot of the reason is people dearly want to there be a better option, even if it’s not there.
Link to fabricated options:
https://www.lesswrong.com/posts/gNodQGNoPDjztasbh/lies-damn-lies-and-fabricated-options
Fabricated options are products of incoherent thinking; what is the incoherence you’re pointing out with policies that aim to delay existential catastrophe or reduce transaction costs between existing power centers?
I think the fabricated option here is just supporting the companies making AI, when my view is that by default, capitalist incentives kill us all due to boosting AI capabilities while doing approximately zero AI safety, in particular deceptive alignment would not be invested in despite this being the majority of the risk.
One of the most important points for AGI safety is the leader in AGI needs a lot of breathing space and leadership ahead of their competitors, and I think this needs to be done semi-unilaterally by an organization not having capitalist incentives, because all the incentives point towards ever faster, not slowing down AGI capabilities. That’s why I think your options are fabricated, because they assume unrealistically good incentives to do what you want.
I don’t mean to suggest “just supporting the companies” is a good strategy, but there are promising non-power-seeking strategies like “improve collaboration between the leading AI labs” that I think are worth biasing towards.
Maybe the crux is how strongly capitalist incentives bind AI lab behavior. I think none of the currently leading AI labs (OpenAI, DeepMind, Google Brain) are actually so tightly bound by capitalist incentives that their leaders couldn’t delay AI system deployment by at least a few months, and probably more like several years, before capitalist incentives in the form of shareholder lawsuits or new entrants that poach their key technical staff have a chance to materialize.
This is the crux, thank you for identifying it.
Yeah, I’m fairly pessimistic for several years time, since I don’t think they’re that special of a company in resisting capitalist nudges and incentives.
And yeah I’m laughing because unless the alignment/safety teams control what capabilities are added, then I do not expect the capabilities teams to stop, because they won’t get paid for that.
I was talking with Abram Demski today about a promising-seeming research direction. (Following is my own recollection)
One of my (TurnTrout’s) reasons for alignment optimism is that I think:
We can examine early-training cognition and behavior to some extent, since the system is presumably not yet superintelligent and planning against us,
(Although this amount of information depends on how much interpretability and agent-internals theory we do now)
All else equal, early-training values (decision-influences) are the most important to influence, since they steer future training.
It’s crucial to get early-training value shards of which a substantial fraction are “human-compatible values” (whatever that means)
For example, if there are protect-human-shards which
reliably bid against plans where people get hurt,
steer deliberation away from such plan stubs, and
these shards are “reflectively endorsed” by the overall shard economy (i.e. the decision-making isn’t steering towards plans where the protect-human shards get removed)
If we install influential human-compatible shards early in training, and they get retained, they will help us in mid- and late-training where we can’t affect the ball game very much (e.g. alien abstractions, interpretability problems, can’t oversee AI’s complicated plans)
Therefore it seems very important to understand what’s going on with “shard game theory” (or whatever those intuitions are pointing at) -- when, why, and how will early decision-influences be retained?
He was talking about viewing new hypotheses as adding traders to a market (in the sense of logical induction). Usually they’re viewed as hypotheses. But also possibly you can view them as having values, since a trader can basically be any computation. But you’d want a different market resolution mechanism than a deductive process revealing the truth or falsity of some proposition under some axioms. You want a way for traders to bid on actions.
I proposed a setup like:
Abram seemed to think that there might exist a nice result like “Given a coalition of traders with values X, Y, Z satisfies properties A, B, and C, this coalition will shape future training and trader-addition in a way which accords with X/Y/Z values up to [reasonably tight trader-subjective regret bound].”
What this would tell us is when trader coalitions can bargain / value handshake / self-trust and navigate value drift properly. This seems super important for understanding what happens, long-term, as the AI’s initial value shards equilibrate into a reflectively stable utility function; even if we know how to get human-compatible values into a system, we also have to ensure they stay and keep influencing decision-making. And possibly this theorem would solve ethical reflection (e.g. the thing people do when they consider whether utilitarianism accords with their current intuitions).
Issues include:
Somehow this has to confront Rice’s theorem for adding new traders to a coalition. What strategies would be good?
I think “inspect arbitrary new traders in arbitrary situations” is not really how value drift works, but it seems possibly contingent on internal capabilities jumps in SGD
The key question isn’t can we predict those value drift events, but can the coalition
EG agent keeps training and is surprised to find that an update knocks out most of the human-compatible values.
Knowing the right definitions might be contingent on understanding more shard theory (or whatever shard theory should be, for AI, if that’s not the right frame).
Possibly this is still underspecified and the modeling assumptions can’t properly capture what I want; maybe the properties I want are mutually exclusive. But it seems like it shouldn’t be true.
ETA this doesn’t model the contextual activation of values, which is a centerpiece of shard theory.
I’ve been thinking along similar lines recently. A possible path to AI safety that I’ve been thinking about extends upon this:
Technological Attractor: Off-the-shelf subsystems
One possible tech-tree path is that we start building custom silicon to implement certain subsystems in an AI agent. These components would be analogous to functional neural regions of the human brain such as the motor cortex, visual system, etc. -- the key hypothesis being that once we reach a certain level of model complexity, the benefits from training a model end-to-end are not worth the costs of re-learning all of these fundamental structures, and furthermore that you can get much better performance-per-cost by casting these modular, reusable components onto an ASIC. This could be a more feasible way of achieving something like [Microscope AI](https://www.lesswrong.com/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#5__Microscope_AI).
Given a few such low-level components, we could enter a technological attractor where getting SOTA through another approach requires either i) throwing multiple OOM more compute than previous SOTA at re-training your own copy of these components in an end-to-end model, or ii) building your own microchip fab to implement your custom component design. Both of these could be high enough barriers that in practice the market participants simply use the off-the-shelf components. And in this attractor more R&D goes into building better-performing and higher-abstraction components that can be combined arbitrarily for different tasks.
Given such off-the-shelf components, say, a standard visual cortex for image recognition, or a 3D spatial reasoning system which can be connected atop a visual cortex, or a generic working memory, etc., it becomes much more tractable to engage in interpretability work that can convince us of the safety of these components. That then just leaves the problem of proving the highest-level network is safe (i.e. whatever is trained to use these components). This is by no means trivial, but for example the ELK paper spent a lot of time worrying about whether you can even trust the basic ontology of an agent, and I think this path would make Ontology Identification tractable. (Of course, this by construction excludes a number of possible agent-configurations).
Now, it’s possible one could plug in these components and still accidentally have the trained agent learn its own replacement visual cortex that is deceptive and not interpretable, but I think that we can get an intuition for when this might happen. If the best off-the-shelf visual cortex module has complexity on the order of N parameters, and our new model that stitches together these lower-level modules has N parameters or less, then it seems implausible that it has somehow learned a new visual cortex network that is much more compact/efficient than the previous state of the art—increasingly implausible the more resources are devoted to optimizing these components. (It would be like someone coming along today and claiming they had achieved a 10x performance improvement on CPU clock speed; given the huge research budgets and historical gradual capability increases, we don’t expect this to happen). On the other hand, we should be suspicious of a N×102 parameter model where we are unable to interpret what 10% of the network is doing; that’s easily enough complexity to hide its own unsafe re-implementation of our components. (I’m aware that there’s a lot of ground in between these two points, I’m merely trying to illustrate that there is “likely safe” and “likely unsafe” ground, rather than claim exactly how big they each are.)
The final step here is the shakiest. It’s not clear to me that we can keep the “top layer” (the actual network that is stitching together the low-level components; perhaps the Neocortex, by analogy to human neural architecture?) thin enough to be obviously not learning its own unsafe component-replacements. However, I think this framework at least paints a picture of a “known safe” or at least “likely safe” path to AGI; if we see that the practical engineering and economic decisions produce thin top-layer models using thick component layers, then we can devote energy to proving the components are safe/interpretable by construction, and exploring the interpretation of the top-level networks that consume the lower-level components. AGI “neurobiology” will be much more tractable if the “neural architecture” is relatively standardized. And so, this could be a good place to provide an early nudge to tip the system into this attractor; heavy investment into research on componentized NN architectures could be viewed as “gain of function” research, but it could also have a much safer end-point.
Another way of thinking about this is that by crystalizing at least some parts of the AGI’s network into slowly-changing structures, we allow time to thoroughly test those parts. It seems very hard to thoroughly test models for safety in a paradigm where the whole model is potentially retrained regularly.
We need to test designs, and most specifically alignment designs, but giving up retraining (ie lifetime learning) and burning circuits into silicon is unlikely to be competitive; throwing out the baby with the bathwater.
Also whether a design is aligned or not isn’t the type of question one can answer by analyzing the agent’s visual cortex, it’s near purely a function of what is steering the planning system.
Would you care to flesh this assertion out a bit more?
To be clear I’m not suggesting that this is optimal now. Merely speculating that there might be a point between now and AGI where the work to train these sub components becomes so substantial that it becomes economical to modularize.
As I mentioned earlier in my post, I was alluding to the ELK paper with that reference, specifically Ontology Identification. Obviously you’d need higher order components too. Like I said, I am imagining here that the majority of the model is “off the shelf”, and just a thin layer is usecase-specific.
To make this more explicit, if you had not only off-the-shelf visual cortex, but also spatio-temporal reasoning modules built atop (as the human brain does), then you could point your debugger at the contents of that module and understand what entities in space were being perceived at what time. And the mapping of “high level strategies” to “low level entities” would be a per-model bit of interpretability work, but should become more tractable to the extent that those low level entities are already mapped and understood.
So for the explicit problem that the ELK paper was trying to solve, if you are confident you know what underlying representation SmartVault is using, it’s much easier to interpret its higher-level actions/strategies.
Interesting, I haven’t seen anyone write about hardware-enabled attractor states but they do seem very promising because of just how decisive hardware is in determining which algorithms are competitive. An extreme version of this would be specialized hardware letting CAIS outcompete monolithic AGI. But even weaker versions would lead to major interpretability and safety benefits.
One other thought after considering this a bit more—we could test this now using software submodules. It’s unlikely to perform better (since no hardware speedup) but it could shed light on the tradeoffs with the general approach. And as these submodules got more complex, it may eventually be beneficial to use this approach even in a pure-software (no hardware) paradigm, if it lets you skip retraining a bunch of common functionality.
I.e. if you train a sub-network for one task, then incorporate that in two distinct top-layer networks trained on different high-level goals, do you get savings by not having to train two “visual cortexes”?
This is in a similar vein to Google’s foundation models, where they train one jumbo model that then gets specialized for each usecase. Can that foundation model be modularized? (Maybe for relatively narrow usecases like “text comprehension” it’s actually reasonable to think of a foundation model as a single submodule, but I think they are quite broad right now. ) The big difference is I think all the weights are mutable in the “refine the foundation model” step?
Perhaps another concrete proposal for a technological attractor would be to build a SOTA foundation model and make that so good that the community uses it instead of training their own, and then that would also give a slower-moving architecture/target to interpret.
Awesome work!
Misc. thoughts and questions as I go along:
1. Why is Continuity appealing/important again?
2. In the Destruction Game, does everyone get the ability to destroy arbitrary amounts of utility, or is how much utility they are able to destroy part of the setup of the game, such that you can have games where e.g. one player gets a powerful button and another player gets a weak one?
For 1, it’s just intrinsically mathematically appealing (continuity is always really nice when you can get it), and also because of an intution that if your foe experiences a tiny preference perturbation, you should be able to use small conditional payments to replicate their original preferences/incentive structure and start negotiating with that, instead.
I should also note that nowhere in the visual proof of the ROSE value for the toy case, is continuity used. Continuity just happens to appear.
For 2, yes, it’s part of game setup. The buttons are of whatever intensity you want (but they have to be intensity-capped somewhere for technical reasons regarding compactness). Looking at the setup, for each player pair i,j, Di,j is the cap for how much of j’s utility that i can destroy. These can vary, as long as they’re nonnegative and not infinite. From this, it’s clear “Alice has a powerful button, Bob has a weak one” is one of the possibilities, that would just mean Da,b>>Db,a. There isn’t an assumption that everyone has an equally powerful button, because then you could argue that everyone just has an equal strength threat and then it wouldn’t be much of a threat-resistance desideratum, now would it? Heck, you can even give one player a powerful button and the other a zero-strength button that has no effect, that fits in the formalism.
So the theorem is actually saying “for all members of the family of destruction games with the button caps set wherever the heck you want, the payoffs are the same as the original game”.
OK, thanks! Continuity does seem appealing to me but it seems negotiable; if you can find an even more threat-resistant bargaining solution (or an equally threat-resistant one that has some other nice property) I’d prefer it to this one even if it lacked continuity.
Absolutely brilliant stuff Jacob! As usual with your posts, I’ll have to ponder this for a while...Let’s see if I got this right:
Evolution had to solve alignment: how to align the powerful general learning engine of the newbrain (neocortex etc) with the goals of the oldbrain (“reptilian brain”).
Some (most?) of this alignment seems to be a form of inverse reinforcement learning. Another form of alignment that the oldbrain applies to the newbrain is imprinting. It is Evolution’s way of solving the pointing problem.
When a duckling hatches it imprints on the first agent it sees. This feels different from inverse reinforcement learning: it’s not like the newbrain is rewarded or punished, rather it is more like there is an open slot for your mom
Thanks! - that summary seems about right.
But I would say that imprinting is a specific instance of a more general process (which I called correlation guided proxy matching). The oldbrain has a simple initial mom detector circuit, which during normal chick learning phase is just good enough to locate and connect to the learned newbrain mom detector circuit, which then replaces/supplants the oldbrain equivalent. The proxy matching needn’t really effect the newbrain directly.
The post starts with the realization that we are actually bottlenecked by data and then proceeds to talk about HW acceleration. Deep learning is in a sense a general paradigm, but so is random search. It is actually quite important to have the necessary scale of both compute and data and right now we are not sure about either of them. Not to mention that it is still not clear whether DL actually leads to anything truly intelligent in a practical sense or whether we will simply have very good token predictors with very limited use.
I don’t actually think we’re bottlenecked by data. Chinchilla represents a change in focus (for current architectures), but I think it’s useful to remember what that paper actually told the rest of the field: “hey you can get way better results for way less compute if you do it this way.”
I feel like characterizing Chinchilla most directly as a bottleneck would be missing its point. It was a major capability gain, and it tells everyone else how to get even more capability gain. There are some data-related challenges far enough down the implied path, but we have no reason to believe that they are insurmountable. In fact, it looks an awful lot like it won’t even be very difficult!
With regards to whether deep learning goes anywhere: in order for this to occupy any significant probability mass, I need to hear an argument for how our current dumb architectures do as much as they do, and why that does not imply near-term weirdness. Like, “large transformers are performing {this type of computation} and using {this kind of information}, which we can show has {these bounds} which happens to include all the tasks it has been tested on, but which will not include more worrisome capabilities because {something something something}.”
The space in which that explanation could exist seems small to me. It makes an extremely strong, specific claim, that just so happens to be about exactly where the state of the art in AI is.
Could you explain why you feel that way about Chinchilla? Because I found that post: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications to give very compelling reasons for why data should be considered a bottleneck and I’m curious what makes you say that it shouldn’t be a problem at all.
Some of my confidence here arises from things that I don’t think would be wise to blab about in public, so my arguments might not be quite as convincing sounding as I’d like, but I’ll give a try.
I wouldn’t quite say it’s not a problem at all, but rather it’s the type of problem that the field is really good at solving. They don’t have to solve ethics or something. They just need to do some clever engineering with the backing of infinite money.
I’d put it at a similar tier of difficulty as scaling up transformers to begin with. That wasn’t nothing! And the industry blew straight through it.
To give some examples that I’m comfortable having in public:
Suppose you stick to text-only training. Could you expand your training sets automatically? Maybe create a higher quality transcription AI and use it to pad your training set using the entirety of youtube?
Maybe you figure out a relatively simple way to extract more juice from a smaller dataset that doesn’t collapse into pathological overfitting.
Maybe you make existing datasets more informative by filtering out sequences that seem to interfere with training.
Maybe you embrace multimodal training where text-only bottlenecks are irrelevant.
Maybe you do it the hard way. What’s a few billion dollars?
What about: State-of-the-art models with 500+B parameters still can’t do 2-digit addition with 100% reliability. For me, this shows that the models are perhaps learning some associative rules from the data, but there is no sign of intelligence. An intelligent agent should notice how addition works after learning from TBs of data. Associative memory can still be useful, but it’s not really an AGI.
They are simluators (https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators), not question answerers. Also, I am sure Minerva does pretty good on this task, probably not 100% reliable but humans are also not 100% reliable if they are required to answer immediately. If you want the ML model to simulate thinking [better], make it solve this task 1000 times and select the most popular answer (which is a quite popular approach for some models already). I think PaLM would be effectively 100% reliable.
As mentioned in the post, that line of argument makes me more alarmed, not less.
We observe these AIs exhibiting soft skills that many people in 2015 would have said were decades away, or maybe even impossible for AI entirely.
We can use these AIs to solve difficult reasoning problems that most humans would do poorly on.
And whatever algorithms this AI is using to go about its reasoning, they’re apparently so simple that the AI can execute them while still struggling on absolutely trivial arithmetic.
WHAT?
Yes, the AI has some blatant holes in its capability. But what we’re seeing is a screaming-hair-on-fire warning that the problems we thought are hard are not hard.
What happens when we just slightly improve our AI architectures to be less dumb?
When will we get robotics results that are not laughable? When “Google put their most advanced AI into a robot brain!!!” (reported on for the third time this year) we got a robot that can deliver a sponge and misplace an empty coke can but not actually clean anything or do anything useful. It’s hard for me to be afraid of a robot that can’t even plug in its own power cable.
I believe that over time we will understand that producing human-like text is not a sign of intelligence. In the past people believed that only intelligent agents are able to solve math equations (naturally, since only people can do it and animals can). Then came computer and they were able to do all kinds of calculations much faster and without errors. However, from our current point of view we now understand that doing math calculations is not really that intelligent and even really simple machines can do that. Chess playing is similar story, we thought that you have to be intelligent, but we found a heuristic to do that really well. People were afraid that chess-algorithm-like machines can be programmed to conquer the world, but from our perspective, that’s a ridiculous proposition.
I believe that text generation will be a similar case. We think that you have to be really intelligent to produce human-like outputs, but in the end with enough data, you can produce something that looks nice and it can even be useful sometimes, but there is no intelligence in there. We will slowly develop an intuition about what are the capabilities of large-scale ML models. I believe that in the future we will think about them as basically a kinda fuzzy databases that we can query with natural language. I don’t think that we will think about them as intelligent agents capable of autonomous actions.
You keep distinguishing “intelligence” from “heuristics”, but no one to my knowledge has demonstrated that human intelligence is not itself some set of heuristics. Heuristics are exactly what you’d expect from evolution after all.
So your argument then reduces to a god of the gaps, where we keep discovering some heuristics for an ability that we previously ascribed to intelligence, and the set of capabilities left to “real intelligence” keeps shrinking. Will we eventually be left with the null set, and conclude that humans are not intelligent either? What’s your actual criterion for intelligence that would prevent this outcome?
I believe that fixating on benchmark such as chess etc is ignoring the G part of AGI. Truly intelligent agent should be general at least in the environment he resides in, considering the limitation of its form. E.g. if a robot is physically able to work with everyday object, we might apply Wozniak test and expect that intelligent robot is able to cook a dinner in arbitrary house or do any other task that its form permits.
If we assume that right now we develop purely textual intelligence (without agency, persistent sense of self etc) we might still expect this intelligence to be general. I.e. it is able to solve arbitrary task if it seems reasonable considering its form. In this context for me, an intelligent agent is able to understand common language and act accordingly, e.g. if a question is posed it can provide a truthful answer.
BIG Bench has recently showed us that our current LMs are able to solve some problems, but they are nowhere near general intelligence. They are not able to solve even very simple problems if it actually requires some sort of logical thinking and not only using associative memory, e.g. this is a nice case:
https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/symbol_interpretation
You can see in the Model performance plots section that scaling did not help at all with tasks like these. This is a very simple task, but it was not seen in the training data so the model struggles to solve it and it produces random results. If the LMs start to solve general linguistic problems, then we are actually having intelligent agents at our hand.
My 8yo is not able to cook dinner in an arbitrary house. Does she have general intelligence?
Humans regularly fail at such tasks but I suspect you would still consider humans generally intelligent.
In any case, it seems very plausible that whatever decision procedure is behind more general forms of inference, it will very likely fall to the inexorable march of progress we’ve seen thus far.
If it does, the effectiveness of our compute will potentially increase exponentially almost overnight, since you are basically arguing that our current compute is hobbled by an effectively “weak” associative architecture, but that a very powerful architecture is potentially only one trick away.
The real possibility that we are only one trick away from a potentially terrifying AGI should worry you more.
I don’t see any indication of AGI so it does not really worry me at all. The recent scaling research shows that we need non-trivial number of magnitudes more data and compute to match human-level performance on some benchmarks (with a huge caveat that matching a performance on some benchmark might still not produce intelligence). On the other hand, we are all out of data (especially high quality data with some information value, no random product reviews or NSFW subreddit discussions) and our compute options are also not looking that great (Moore’s law is dead, the fact that we are now relying on HW accelerators is not a good thing, it’s a proof that CPU performance scaling is after 70 years no longer a viable option. There are also some physical limitations that we might not be able to break anytime soon.)
It is goalpost moving. Basically, it says “current models are not really intelligent”. I don’t think there is much disagreement here. And it’s hard to make any predictions based on that.
Also, “Producing human-like text” is not well defined here; even ELIZA may match this definition. Even the current SOTA may not match it because the adversarial Turning Test has not yet been passed.
It’s not goapost moving, it’s the hype that’s moving. People reduce intelligence to arbitrary skills or problems that are currently being solved and then they are let down when they find out that the skill was actually not a good proxy.
I agree that LMs are concetually more similar to ELIZA than to AGI.
The observation that things that people used to consider intelligent are now considered easy is critical.
The space of stuff remaining that we call intelligent, but AIs cannot yet do, is shrinking. Every time AI eats something, we realize it wasn’t even that complicated.
The reasonable lesson appears to be: we should stop default-thinking things are hard, and we should start thinking that even stupid approaches might be able to do too much.
It’s a statement more about the problem being solved, not the problem solver.
When you stack this on a familiarity with the techniques in use and how they can be transformatively improved with little effort, that’s when you start sweating.
Upvoted for the well reasoned arguments. It seems that competition should be the default assumption for the future, for a whole host of reasons, some of which are elaborated here. Personally I never found the opposite claims to be convincing, so I may be biased here.
As a corollary, this implies that claims such as competition will somehow cease, some sort of ‘lock in’ past a threshold, etc., should be subject to a much higher level of scrutiny.
Thank you for this explanation. Now it helps me to understand a little bit more of why so many people I know simply feel overwhelmed and give up. Personally as I am not in position to donate money, I work to tackle one specific problem set that I think will help open up and leave the solutions to other problems.
ShiraDest
This was really good and definitely made me think about how I might live in such a scenario. I would probably go all in on frequent redaction and just lean hard on external memory storage to make up the difference. I already barely remember anything from even ten years ago and rely mostly on external memory for everything, I have a strong ability to acausally coordinate with myself across time, so I’m not worried about different iterations of me going off course in ways I wouldn’t endorse. If you have a strong enough exomemory system you can effectively just freeze your age groundhog-day style and rely on the work you did before each redaction to keep carrying you forward. This does require a strong ability to log and rely information but it seems very manageable, and once I’ve lived long enough for direct mind-computer interfacing I can just back up my memories, redact, and then download them again.
I really enjoyed the first 60% of the story, and then it became… too long. Not sure if this is just about me, or if others have a similar reaction.
Not sure if this is just a rationalization, but at this moment you switched from present moment to retrospective, which removed the tension. Until this point, I felt like the story could go in many possible directions. (Who is Bruce Hance? What are his actual plans? Will the protagonist hack this device, and maybe reprogram people to worship him instead?) After this point, I felt like “okay, this is obviously a metaphor for a religion” and stopped expecting anything that wouldn’t fit this metaphor. The ending was nice again, though.
It was originally two novellas. I combined them, not seeing a point to publishing them separately. Should I separate them?
Possibly. I had great fun with the first half of this story, and then I saw where the scroll thingy on my screen was and so stopped with the intention of coming back for the second half. (Which I just did. Second half also great fun!) I don’t think its a flaw in the story, I think that it might just be that its length puts it as a bit of an outlier in terms of webpage reading.
Solar flares are a great plot device for “I need some machine to malfunction”.
Also to set up the visitors at the end, who I still feel arrived too abruptly
It is abrupt, but to me that wasn’t a bad thing. If them arriving had taken an extra x paragraphs but the story worked out exactly the same way I think that for me that would be worse not better. If there was some clever way of hinting them earlier maybe. The solar flare works quite well for that.
I think you should not act on my advice alone. I might be an outlier.
Furthermore, even if I correctly detected what makes the story worse (for a group of people larger than myself) it does not automatically mean that following my advice would improve it. People are better at detecting what they don’t like than at improving things. (For example, I could say which meals taste better and which worse, but I couldn’t cook the meals I like most.)
The last objection is that, in long term, rewriting this story is irrelevant. If my complaint makes sense (which still remains to be verified), the best reaction is to keep writing other things, and not make the same mistake again.
So… I guess just leave it as it is. If you later decide to publish it elsewhere, probably two parts will be better. When writing new stories, consider not changing the style in the middle (or if you do, keep the parts separate).
Here are the links to the articles from the books. Let me know if there are any mistakes.
Notes:
I didn’t find “Do Birth Order Effects Exist?”, “Challenges to Christiano’s Iterated Amplification Proposal” seems to be only on intelligence.org and “Response to FAQ on Iterated Amplification” seems to be a comment and not a post of it’s own.
This is a beautiful set of ideas which bear further discussion.
ShiraDest
I have read about this many years ago. We did a proof of the Gödel’s incompleteness theorem at university. And I still have the feeling like there is some sleight of hand involved… that even when I am telling the proof, I am merely repeating the teacher’s password. My intuition has no model of what this all actually means.
From my perspective, understanding math means that I can quickly say why something works, what it implies, how would the situation change if we slightly changed the conditions, which part of the entire proof is the critical one, etc.
So frustrating...
I’m not sure if this is what you’re looking for, but Hofstadter gives a great analogy using record players which I find useful in terms of thinking about how changing the situation changes our results (which is paraphrased here).
A (hi-fi) record player that tries to playing every possible sound can’t actually play its own self-breaking sound, so it is incomplete by virtue of its strength.
A (low-fi) record player that refuses to play all sounds (in order to avoid destruction from its self-breaking sound) is incomplete by virtue of its weakness.
We may think of the hi-fi record player as a formal system like Peano Arithmetic: the incompleteness arises precisely because it is strong enough to be able to capture number theory. This is what allows us to use Gödel Numbering, which then allows PA to do meta-reasoning about itself.
The only way to fix it is to make a system that is weaker than PA, so that we cannot do Gödel Numbering. But then we have a system that isn’t even trying to express what we mean by number theory. This is the low-fi record player: as soon as we fix the one issue of self-reference, we fail to capture the thing we care about (number theory).
I think an example of a weaker formal system is Propositional Calculus. Here we do actually have completeness, but that is only because Propositional Calculus is too weak to be able to capture number theory.
What exactly is the aspect of natural numbers that makes them break math, as opposed to other types of values? Intuitively, it seems to be the fact that they can be arbitrarily large but not infinite.
Like, if you invent another data type that only has a finite number of values, it would not allow you to construct something equivalent to Gödel numbering. But if it allows infinite number of (finite) values, it would. (Not sure about an infinite number of/including infinite values, probably also would break math.)
It seems like you cannot precisely define natural numbers using first-order logic. Is that the reason of this all? Or is it a red herring? Would situation be somehow better with second-order logic?
(These are the kinds of questions that I assume would be obvious to me, if I grokked the situation. So the fact that they are not obvious, suggests that I do not see the larger picture.)
I don’t have much to add, but I basically endorse this, and it’s similar to what I try to do myself.
Figuring out when and how to proactively share information about my inner-layers with people is a balancing act I’m still figuring out, but I’ve found it generally valuable for building trust.
I’m a little bit skeptical of the argument in “Transformers are not special”—it seems like, if there were other architectures which had slightly greater capabilities than the Transformer, and which were relatively low-hanging fruit, we would have found them already.
I’m in academia, so I can’t say for sure what is going on at big companies like Google. But I assume that, following the 2017 release of the Transformer, they allocated different research teams to pursuing different directions: some research teams for scaling, and others for the development of new architectures. It seems like, at least in NLP, all of the big flashy new models have come about via scaling. This suggests to me that, within companies like Google, the research teams assigned to scaling are experiencing success, while the research teams assigned to new architectures are not.
It genuinely surprises me that the Transformer still has not been replaced as the dominant architecture since 2017. It does not surprise me that sufficiently large or fancy RNNs can achieve similar performance to the Transformer. The lack of Transformer replacements makes me wonder whether we have hit the limit on the effectiveness of autoregressive language models, though I also wouldn’t be surprised if someone comes up with a better autoregressive architecture soon.
There have been a few papers with architectures showing performance that matches transformers on smaller datasets with scaling that looks promising. I can tell you that I’ve switched from attention to an architecture loosely based on one of these papers because it performed better on a smallish dataset in my project but I haven’t tested it on any standard vision or language datasets, so I don’t have any concrete evidence yet. Nevertheless, my guess is that indeed there is nothing special about transformers.
I’d be interested to see links to those papers!
I’ve messaged you the links. Basically MLPs.
I think what’s going on is something like:
Being slightly better isn’t enough to unseat an entrenched option that is well understood. It would probably have to very noticeably better, particularly in scaling.
I expect the way the internal structures are used will usually dominate the details of the internal structure (once you’re already at the pretty good frontier).
If you’re already extremely familiar with transformers, and you can simply change how you use transformers for possible gains, you’re more likely to do that than to explore a from-scratch technique.
For example, in my research, I’m currently looking into some changes to the outer loop of execution to make language models interpretable by construction. I want to focus on that part of it, and I wanted the research to be easily consumable by other people. Building an entire new architecture from scratch would be a lot of work and would be less familiar to others. So, not surprisingly, I picked a transformer for the internal architecture.
But I also have other ideas about how it could be done that I suspect would work quite well. Bit hard to justify doing that for safety research, though :P
I think the amount of low hanging fruit is so high that we can productively investigate transformer derivatives for a long time without diminishing returns. They’re more like a canvas than some fixed Way To Do Things. It’s just also possible someone makes a jump with a non-transformer architecture at some point.
It does not follow that computationally cheaper things are more likely to happen than computationally expensive things. Moreover, describing something as “computationally difficult” is a subjective value judgment (unless you can reasonably prove otherwise) and implies that all actions/events can be reduced to some form of computation.
Georgism isn’t taxation, it’s public property ownership.
It finally clicked in my mind why I’ve reacted negatively to the discussion about Georgist land taxes. It’s the same reaction I have to ANY 100% tax rate on anything. That’s not what “ownership” means.
Honestly, I’d probably be OK with (or at least I’d be able to discuss it more reasonably) the idea that land is not privately owned, ever. Only rented/leased from the government, revocable for failure to pay the current rate (equivalent in all ways to the Georgist tax amount, and penalties for failure to pay). I suspect it’s unworkable for the same reasons—the public won’t stand for evicting sympathetic tenants just because they moved in when the rent/tax was lower.
Further, taxation of economic movement or realized profit is pretty easy to justify, and ALWAYS exists. Taxation of durable-ownership valuation is tricky, as there is disagreement over whether it’s “real”, and there’s no guarantee that the investment value, even if correctly assessed, is liquid enough to actually pay the tax.
Edit: this had occurred to me before, but I forgot to put it here. It also suffers from the socialist calculation problem. When self-interested humans aren’t making decisions to maximize value (to themselves) about a thing, the actual most valuable use is very difficult to determine. Holding someone responsible for the theoretical value of something when they don’t actually think it’s the best is really difficult to envision.
Further edit: thanks for the comments—it’s helped me understand my true objection, in addition to the auxiliary problems that often come to mind. I think there are two insurmountable problems, not necessarily in this order:
Taxing anything at 100% is disingenuous about the word “ownership”. It should be called “rent” or something else that is clear that it’s not owned, but merely usable for some purposes until you decline to pay a future arbitrary amount. It would also make clear that you’re also undertaking all of the political-economy influences that make government setting of rent amounts problematic.
Taxing non-realized value is always going to be problematic, both for calculation reasons (valuation is theoretical and debatable), and for the fragility of the value. I think this generalizes to “only collect taxes in-kind (or on liquid enough things that it can be easily converted at tax-collection time)”. Taxing a pile of wheat is easy—take some percentage of it. Taxing a monetary transaction likewise—the price increases and the tax is paid out of that. Taxing a future value is not (generally) feasible—either you’ve just taken someone’s seeds so they can’t plant the crop you’re trying to tax, or you’re demanding cash from someone whose investments will suffer from early liquidation. Deferral of payment can help with this, but if you’re doing that, you should just tax the realized gains at time of liquidity.
We would really need the technology where you can take a house and move it to a different place. That would also make Georgism easier—when people don’t pay, you simply move them to a cheaper location.
While I think that would be extremely valuable technology, exiling Grandma to Outer Nowhereville, hours away from the rest of the family, is probably not a lot more feasible.
Alternatively, what if the LVT only increased when the property was sold? Buy land, and you lock in the LVT at the point in time when your deal went through.
I guess people would never sell the real estate at good location, only lease it? Thus capturing the increased value of the land without ever paying the tax.
Maybe leases are subject to increased LVT, just not owning it for personal use? I suspect there’s a way to enact something like this, but this isn’t in my wheelhouse.
Rich people and businesses are more strongly motivated and able to find loopholes than government administrators are to design robust mechanisms. Modern fractional ownership and lease arrangements are complicated enough that most any scheme based on a change in contract participants is going to be arbitraged away very quickly.
Right, but I thought the point was to avoid people balking at a law that makes it easier to evict ordinary folks who’ve lived somewhere a long time and don’t want to move? Let the rich people find their loopholes—as long as people support the system and the LVT delivers most of its promise, that’s good enough.
My point is that whatever exclusions are in place to keep poor people in their newly-valuable homes will be used by rich people to avoid taxation. There is no way to avoid this without making the tax variable based on other factors than the land value (like the “owner’s” perceived financial situation).
Either the law is the same for all “pay or GTFO”, or it’s differentially enforced, so the rich and the politically-sympathetic pay less than the middle-class.
It sounds like you think there’s no practical way to introduce exceptions to make an LVT politically palatable without also breaking its ability to motivate efficient land use and generate tax revenue.
My first thought is sure, we have a lot of tax evasion now. But we also do generate lots of tax revenue, most tax revenue comes from the very rich. Why would an LVT be so much more prone to tax evasion than the current system?
Correct, I think there’s no practical or realpolitik-theoretical way to make it work. To the extent that you make exceptions in implementation, it’s no longer a land-value tax, it’s a land-value fig leaf over a “what we can get away with” tax.
This is true the same way that to the extent you dilute your cocoa with water, it’s no longer hot chocolate, it’s water :) I don’t know of almost any “pure” real world policies or taxes. I’d need some good evidence to believe that the LVT is that fragile.
Your definition of AGI (“the kind of AI with sufficient capability to make it a genuine threat to humanity’s future or survival if it is misused or misaligned”) is tragically insufficient, vague, subjective, and arguably misaligned with the generally accepted definition of AGI.
From what you wrote elsewhere (“An AGI having its own goals and actively pursuing them as an agent”) you imply that the threat could come from AGI’s intentions, that is, you imply that AGI will have consciousness, intentionality, etc. - qualities so far exclusively prescribed to living things (you have provided no arguments to think otherwise).
However, you decided to define “intelligence” as “stuff like complex problem solving that’s useful for achieving goals” which means that intentionality, consciousness, etc. is unconnected to it (realistically any “complex”-enough algorithm satisfies this condition). Such simplistic and reductionistic definition implies that it is not enough for an “intelligent” computer to be an AGI. So, while you may be able to prove that a computer could have “intelligence” it still does not follow that AGI is possible.
Your core idea that “We’ve already captured way too much of intelligence with way too little effort.” may be true with your definition of “intelligence”, but I hope I’ve shown that such definition is not enough. Researchers at Harvard suggest existence of multiple types of intelligence, which your statement does not take into account and groups all types of intelligence into one even though some are impossible for a computer to have and some could be considered as defining qualities of a computer.
Furthermore, you compare humans to computers and brains to machines and imply that consciousness is computation. To say that “consciousness is not computation” is comparable to “god of gaps” argument is ironic considering the existence of the AI effect. Your view is hardly coherent in any other worldview than hardcore materialism (which itself is not coherent). Again, we stumble into an area of philosophy, which you hardly addressed in your article. Instead you focused on predicting how good our future computers will be at computing while making appeals to emotion, appeals to unending progress, appealing to the fallacy that solving the last 10% of the “problem” is as easy as the other 90% - that because we are “close” to imitating it (and we are not if you consider the full view of intelligence), we somehow grasped the essence of it and “if only we get slightly better at X or Y we will solve it”.
Scientists have been predicting coming of AGI since ’50s, some believed 70 years ago that it will only take 20 years. We have clearly not changed as humans. The question of intelligence and, thus, the question of AGI is in many ways inherently linked to philosophy and it is clear that your philosophy is that of materialism which cannot provide good understanding of “intelligence” and all related ideas like mind, consciousness, sentience, etc. If you were to reconsider your position and ditch materialism, you might find that your idea of AGI is not compatible with abilities of a computer, or non-living matter in general.
You oppose hardcore materialism, in fact say it is incoherent—OK. Is there a specific different ontology you think we should be considering?
In the comment before this, you say there are kinds of intelligence which it is impossible for a computer to have (but which are recognized at Harvard). Can these kinds of intelligence be simulated by a computer, so as to give it the same pragmatic capabilities?