I think I have a contender for something which evades the conditional-threat issue stated at the end, as well as obvious variants and strengthenings of it, and which would be threat-resistant in a dramatically stronger sense than ROSE.
There’s still a lot of things to check about it that I haven’t done yet. And I’m unsure how to generalize to the n-player case. And it still feels unpleasantly hacky, according to my mathematical taste.
Reiterating a point above: observe how this whole scheme has basically assumed that capabilities won’t start to generalize relevantly out of distribution. My model says that they eventually will, and that this is precisely when things start to get scary, and that one of the big hard bits of alignment is that once that starts happening, the capabilities generalize further than the alignment. A problem that has been simply assumed away in this agenda, as far as I can tell, before we even dive into the details of this framework.
One (non-goodhart resistent) way would be to use some kind of Page Rank method based on the tags that users comment and post on. Eg. People Rank high in AI posts, if their comments and AI posts were upvoted by other people with the same Expertise.
If they alternated directions, increasing the distance walked each time by a constant factor (e.g. triple), then an exit in either direction would be reached without too much wasted time.
Some scientists tried to create group selection under laboratory conditions. They divided some insects into subpopulations, then killed off any subpopulation whose numbers got too high, and and “promoted” any subpopulation that kept its numbers low to better conditions. They hoped the insects would evolve to naturally limit their family size in order to keep their subpopulation alive. Instead, the insects became cannibals: they ate other insects’ children so they could have more of their own without the total population going up. In retrospect, this makes perfect sense; an insect with the behavioral program “have many children, and also kill other insects’ children” will have its genes better represented in the next generation than an insect with the program “have few children”.
Why didn’t they try also killing off subpopulations that engaged in cannibalism, and promoting those that didn’t? And what would have most likely happened if they had tried that?
Probably it is also depends on how much information about “various models trying to naively backstab their own creators” there are in the training dataset
Shortform #139 Note to self: don’t reschedule weekly meetup
Tonight’s Virginia Rationalists: Norfolk meetup was great! We had a new individual join us who is familiar with ACX.
In retrospect, I think it was inconsiderate to suddenly reschedule the group’s weekly meetup just because I couldn’t attend. My co-organizer and others likely could have attended yesterday, and there were a few people who just couldn’t make the rescheduled meetup but had included Wednesday nights in their routine as “meetup night”. So, I intend to not reschedule the weekly social meetup unless literally everyone reaches out in advance and asks for that to occur. Weekly meetups = people build that into their routines, and should not be changed lightly.
How actually do you sidestep the need for the One True Objective Function given an ELK solution? I get that it might seem plausible to take a rough objective like “do what I intend” and look at the internal knowledge of the thing for signs that it is deliberately deceiving you. If you do that, you’ll get, at best, an AI that doesn’t know that it is deceiving you (for whatever operationalization of “know” you come up with as you use ELK for training). But it could still be deceiving you, and very likely will be if optimization pressure is merely towards “AIs that don’t know that they are being deceptive”.
Our very broad hope is to use ELK to select actions that (i) keep humans safe, and give them time and space to evolve according to their current (essentially local) preferences, (ii) are expected to produce outcomes that would be judged favorably by the future humans, primarily by maximizing option value until it becomes clear what those future humans want (see the strategy stealing assumption).
This is discussed very briefly in this appendix of the ELK report and the subsequent appendix. There are two or three big foreseeable difficulties with this approach and likely a bunch of other problems.
I don’t think this should be particularly persuasive, but it hopefully illustrates how ARC is currently thinking about this part of the problem. Overall my current view is that this is fairly unlikely to be the weakest link in the plan, i.e. if it doesn’t work it will be because of a failure at an earlier step, and so it’s not one of the main things I’m thinking about.
Take a reinforcement learner AI, that we want to safely move a strawberry onto a plate. A human sits nearby and provides a reward based on inspecting the AI’s behaviour.
As it stands, this setup is completely vulnerable to reward hacking. The reward is not provided for safe moving of the strawberry; instead the reward is provided by having the human judge that the task has been accomplished and then pressing a button. Taking control of the human or control of the button is likely to be possible for a superintelligent AI; and, as it stands, that would be mandated by this reward function.
Would it be sufficient, for disproof, to show one system that does steer far-away parts of the world into a relatively-small chunk of their state space, but does not internally contain a world-model or do planning?
That could be sufficient in principle, though I would not be surprised if I look at a counterexample and realize that the problem description was missing something rather than that the claim is basically false. For instance, it wouldn’t be too surprising if there’s some class of supposed-counterexamples which only work under very specific conditions (i.e. they’re not robust), and can be ruled out by some “X isn’t very likely to happen spontaneously in a large system” argument.
The bottom line is that a disproof should argue that there isn’t anything basically like the claim which is true. Finding a physical counterexample would at least show that the counterexample isn’t sensitive to the details of mathematical framework/formulation.
Do you have a particular counterexample in mind? A thermometer, perhaps?
Well ok just to brainstorm some naive things that don’t really rule the conjecture out:
A nuclear bomb steers a lot of far-away objects into a high-entropy configuration, and does so very robustly, but that perhaps is not a “small part of the state space”
A biological pathogen, let loose in a large human population, might steer all the humans towards the configuration “coughing”, but the virus is not itself a consequentialist. You might say that the pathogen had to have been built by a consequentialist, though.
Generalizing the above: Suppose I discover some powerful zero-day exploit for the linux kernel. I automate the exploit, setting my computer up to wait 24 hours and then take over lots of computers on the internet. Viewing this whole thing from the outside, it might look as if it’s my computer that is “doing” the take-over, but my computer itself doesn’t have a world model or a planning routine.
Consider some animal species spreading out from an initial location and making changes to the environments they colonize. If you think of all the generations of animals that underwent natural selection before spreading out as the “system that controls some remote parts of the system” and the individual organisms as kind of messages or missiles, then this seems like a pretty robust, though slow form of remote control. Maybe you would say that natural selection has a world model and a planning process, though.
Since this seems to be Carn’s first post on LessWrong, I think some of the other readers should have been more lenient and not downvoted the post or explained why they downvoted the post.
I would only downvote a post if it was obviously bad, flawed, very poorly written, or a troll post.
This post contains lots of interesting ideas and seems like a good first post.
The original post “Reward is not the optimization target” has 216 upvotes and this one has 0. While the original post was written better, I’m skeptical of the main idea and it’s good to see a post countering it so I’m upvoting this post.
the core thing I worry about with any simulation-based approach is how to get coverage of possibility space. ensuring that the tests are informative about possible configurations of complex systems is hard; I would argue that this is a great set of points, and that no approach without this type of testing could be expected to succeed.
however, as we’ve seen with adversarial examples, both humans and current DL have fairly severe failures of alignment that cause large issues, and projects like this need tools to optimize for interpretability from the perspective of a formal verification algorithm.
it’s my belief that in almost all cases, the values almost all beings ever wish to express lie on a low-enough-dimensional manifold that, if the ideal representation was learned, a formal verifier would be able to handle it; see, for example, recent work from anthropic on interpretability lining up nicely with what’s needed to use reluplex (or followup work; reluplex is old now) to estimate bounds on larger networks’ behavior.
I don’t think we should give up on perfectly solving safety; instead, we should recognize that it has never even been closely approximated before, by any form of life, because even semi-planned evolutionary processes do not do a good job of covering the entire possibility space.
this post is an instant classic, and I agree that sims are a key step, starting from small agents. but ensuring that strongly generalizing moral views are generated in the agents who grow in the simulations is not trivial—we need to be able to generate formal bounds on the loss function, and doing so requires being able to prove all the way through the simulation the way we currently take gradients through it. if we can ask “where in the simulation would have changed this outcome”, then we’d finally be getting somewhere in terms of generating moral knowledge that generalizes universally.
I wonder, almost just idle curiosity, whether or not the “measuring-via-proxy will cause value drift” is something we could formalize and iterate on first. Is the problem stable on the meta-level, or is there a way we can meaningfully define “not drifting from the proxy” without just generally solving alignment.
Intuitively I’d guess this is the “don’t try to be cute” class of thought, but I was afraid to post at all and decided that I wanted to interact, even at the cost of (probably) saying something embarassing.
It is at least not obvious to me that this is a “don’t try to be cute” class of thought, though not obvious that it isn’t either. Depends on the details.
By contrast, the world where sharp left turns happen near human-level capabilities seems pretty dangerous. In that world, we don’t get to study things like deceptive alignment empirically. We don’t get to leverage powerful AI researcher tools.
We actually can study and iterate on alignment (and deception) in simulation sandboxes while leveraging any powerful introspection/analysis tools. Containing human-level AGI in (correctly constructed) simulations is likely easy.
But then later down you say:
For instance, we can put weak models in environments that encourage alignment failure so as to study left turns empirically (shifting towards the “weak” worlds).
Which seems like a partial contradiction, unless you believe we can’t contain human-level agents?
Is this something you’ve thought deeply about and or care to expand on? Curious about your source of skepticism, considering:
we can completely design and constrain the knowledge base of sim agents, limiting them to the equivalent of 1000BC human knowledge, or whatever we want
we can completely and automatically monitor their inner monologues, thoughts, etc
How do you concretely propose a sim agent will break containment? (which firstly requires sim-awareness)
How would you break containment now, assuming you are in a sim?
Also, Significant speed/memory advantages go against the definition of ‘human-level agents’ and are intrinsically unlikely anyway as 2x speed/memory agents are simply preceded by 1x speed/memory agents, and especially due to the constraints of GPUs which permit acceleration by parallelization of learning across agents, more so than serial speedup of individual agents.
My modal expectation is that these are going to look something like large language models, meaning that while we’ll have controlled the training corpus it won’t be something hand-curated. So there will almost certainly be lots of information in there that models could use to e.g. figure out how to manipulate people, program computers, and so on. I expect then that we won’t be limiting their knowledge as much as you think we will, because I expect them to be trained with giant corpuses of (mostly) post-1000BC text.
We do not currently have the interpretability tools needed to monitor their inner thoughts. I would feel much better about this problem if we had such tools!
My best guess is that containment fails because a model convinces one of the people in the lab to let it access the internet. That could look like getting someone to think it’s genuinely a good idea (because it’ll be helpful for some project), or it could look like bribery (with offer of future payment) or blackmail.
I’m guessing that we’ll want to run some agents fast because that’s how you get lots of serial intellectual work done. So even if all we get are human-level agents, there’ll be a big incentive to run them fast.
Indeed a model trained on (or with full access to) our internet could be very hard to contain. That is in fact a key part of my argument. If it is hard to contain, it is hard to test. If it’s hard to test, we are unlikely to succeed.
So if we agree there, then perhaps you agree that giving untested AGI access to our immense knowledge base is profoundly unwise.
I expect then that we won’t be limiting their knowledge as much as you think we will, because I expect them to be trained with giant corpuses of (mostly) post-1000BC text.
That’s then just equivalent to saying “I expect then that we won’t even bother with testing our alignment designs”. Do you actually believe that testing is unnecessary? Or do you just believe the leading teams won’t care? And if you agree that testing is necessary, then shouldn’t this be key to any successful alignment plan?
My best guess is that containment fails because a model convinces one of the people in the lab to let it access the internet.
Which obviously is nearly impossible if it doesn’t know it is in a sim, and doesn’t know what a lab or the internet are, and lacks even the precursor concepts. Since you seem to be focused on the latest fad (large language models), consider the plight of poor simulated Elon Musk—who at least is aware of the sim argument.
The situation is not ‘handled.’ Elites have lost all credibility.
I think it’s worth caveating this that not all elites have lost credibility. Elites in places like Singapore, Switzerland, and Finland have a lot of credibility.
Edits based on feedback from LessWrong and the EA Forum:
EDITS: - Added new ‘Definitions’ section to introduction to explain definitions such as ‘AI safety’, ‘researcher’ and the difference between technical and non-technical research.
NON-TECHNICAL - CSER: 5-5-10 → 2-5-15 - Delete BERI from the list of non-technical research organizations - Delete SERI from the list of non-technical research organizations - Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence) - FLI: 5-5-20 → 3-5-15 - Add OpenPhil: 2-5-15 - Epoch: 5-10-15 → 2-4-10 - Add ‘Other’: 5-10-50
TODO: - update all graphs
- add Rethink Priorities to the list of non-technical organizations
Only make choices that you would not make in reverse, if things were the other way around. Drop out of school if and only if you wouldn’t enroll in school from out of the workforce. Continue school if and only if you’d switch over from work to that level of schooling.
Flitting back and forth between both possible worlds can make you less cagey about doing what’s overdetermined by your world model + utility function already. It’s also part of the exciting rationalist journey of acausally cooperating with your selves in other possible worlds.
It’s probably a useful mental technique to consider from both directions, but also consider that choices that appear symmetric at first glance may not actually be symmetric. There are often significant transition costs that may differ in each direction, as well as path dependencies that are not immediately obvious.
As such, I completely disagree with the first paragraph of the post, but agree with the general principle of considering such decisions from both directions and thank you for posting it.
I do research in cooperation and game theory, including some work on altruism, and also some hard science work. Everyone looks at the Rorschach blot of human behavior and sees something different. Most of the disagreements have never been settled. Even experiment does not completely settle them.
My experience from having children and observing them in the first few months of life is more definitive. They come with values and personal traits that are not very malleable, and not directly traceable to parents. Sometimes grandparents (who were already dead, so it had to be genetic).
My experience with A.I. researchers is that they are looking for a shortcut. No one’s career will permit raising an experimental AI as a child. The tech of the AI would be obsolete before the experiment was complete. This post is wishful thinking that a shortcut is available. It is speculative, anecdotal, and short on references to careful experiment. Good luck with that.
I think there are two fundamental problems with the extensive simboxing approach. The first is just that, given the likely competitive dynamics around near-term AGI (i.e. within the decade), these simboxes are going to be extremely expensive both in compute and time which means that anybody unilaterally simboxing will probably just result in someone else releasing an unaligned AGI with less testing.
If we think about the practicality of these simboxes, it seems that they would require (at minimum) the simulation of many hundreds or thousands of agents over relatively long real timelines. Moreover, due to the GPU constraints and Moore’s law arguments you bring up, we can only simulate each agent at close to ‘real time’. So years in the simbox must correspond to years in our reality, which is way too slow for an imminent singularity. This is especially an issue given that we must maintain no transfer of information (such as datasets) from our reality into the sim. This means at minimum years of sim-time to bootstrap intelligent agents (taking humans data-efficiency as a baseline). Also, each of these early AGIs will be likely be incredibly expensive in compute so that maintaining reasonable populations of them in simulation will be very expensive and probably infeasible initially. If we could get policy coordination on making sure all actors likely to develop AGI go through a thorough simboxing testing regimen, then that would be fantastic and would solve this problem.
Perhaps a more fundamental issue is that simboxing does not address the fundamental cause of p(doom) which is recursive self improvement of intelligence and the resulting rapid capability gains. The simbox can probably simulate capability gains reasonably well (i.e. gain ‘magical powers’ in a fantasy world) but I struggle to see how it could properly test gains in intelligence from self-improvement. Suppose the AI in the fantasy simbox brews a ‘potion’ that makes it 2x as smart. How do we simulate this? We could just increase the agent’s compute in line with the scaling laws but a.) early AGIs are almost certainly near the frontier of our compute capability anyway and b.) much of recursive self improvement is presumably down to algorithmic improvements which we almost necessarily cannot simulate (since if we knew better algorithms we would have included them in our AGIs in the simulation in the first place!)
This is so vital because the probable breakdown of proxies to human values under the massive distributional shift induced by recursive self improvement is the fundamental difficulty to alignment in the first place.
Perhaps this is unique to my model of AI risk, but almost all the probability of doom channels through p(FOOM) such that p(doom | no FOOM) is quite low in comparison. This is because if we have don’t have FOOM then there is not extremely large amounts of optimization power unleashed and the reward proxies for human values and flourishing don’t end up radically off-distribution and so probably don’t break down. There are definitely a lot of challenges left in this regime, but to me it looks solvable and I agree with you that in worlds without rapid FOOM, success will almost certainly look like considerable iteration on alignment with a bunch of agents undergoing some kind of automated simulated alignment testing in a wide range of scenarios plus using the generalisation capabilities of machine learning to learn reward proxies that actually generalise reasonably well within the distribution of capabilities actually obtained. The main risk, however, in my view, comes from the FOOM scenario.
Finally, I just wanted to say that I’m a big fan of your work and some of your posts have caused major updates to my alignment worldview—keep up the fantastic work!
Thanks, upvoted for engagement and constructive criticism—I’d like more to see this comment.
I’m going to start perhaps in reverse to establish where we seem to most agree:
There are definitely a lot of challenges left in this regime, but to me it looks solvable and I agree with you that in worlds without rapid FOOM, success will almost certainly look like considerable iteration on alignment with a bunch of agents undergoing some kind of automated simulated alignment testing in a wide range of scenarios plus using the generalisation capabilities of machine learning to learn reward proxies that actually generalise reasonably well within the distribution of capabilities actually obtained.
I fully agree with this statement.
However in worlds that rapidly FOOM, everything becomes more challenging, and I’ll argue in a moment why I believe that the approach presented here still is promising in rapid FOOM scenarios, relative to all other practical techniques that could actually work.
But even without rapid FOOM, we still can have disaster—for example consider the scenario of world domination by a clan of early uploads of some selfish/evil dictator or trillionaire. There’s still great value in solving alignment here, and (to my eyes at least) much less work focused on that area.
Now if rapid FOOM is near inevitable, then those considerations naturally may matter less. But rapid FOOM is far from inevitable.
First, Moore’s Law is ending, and brains are efficient, perhaps even near pareto-optimal.
Secondly, the algorithms of intelligence are much simpler than we expected, and brains already implement highly efficient or even near pareto-optimal approximations of the ideal universal learning algorithms.
To the extent either of those major points are true, rapid FOOM is much less likely; to the extent both are true (as they appear to be), then very rapid FOOM is very unlikely.
Performance improvement is mostly about scaling compute and data in quantity and quality—which is exactly what has happened with deep learning, which was deeply surprising to many in the ML/comp-sci community and caused massive updates (but was not surprising and was in fact predicted by those of us arguing for brain efficiency and brain reverse engineering).
Now, given that background, there a few other clarifications and/or disagreements:
If we think about the practicality of these simboxes, it seems that they would require (at minimum) the simulation of many hundreds or thousands of agents over relatively long real timelines. Moreover, due to the GPU constraints and Moore’s law arguments you bring up, we can only simulate each agent at close to ‘real time’.
To a first approximation, compute_cost = size*speed. If AGI requires brain size, then the first to cross the finish line will likely be operating not greatly faster than the minimum speed, which is real-time. But this does not imply the agents learn at only real time speed, as learning is parallelizable across many agent instances. Regardless, none of these considerations depend on whether the AGI is trained in a closed simbox or an open sim with access to the internet.
So just to clarify:
AGI designs in simboxes are exactly the same as unboxed designs, and have exactly the same compute costs
The only difference is in the datastream and thus knowledge
The ideal baseline cost of simboxing is only O(N+1) vs O(N) without—once good AGI designs are found, the simboxing approach requires only one additional unboxed training run (compared to never using simboxes). We can estimate this additional cost: it will be around or less than 1e25 ops (1e16 ops/s for brain-size model * 1e9s seconds for 30 years equivalent), or less than $10 million dollars (300 gpu years) using only todays gpus, ie nearly nothing.
Perhaps a more fundamental issue is that simboxing does not address the fundamental cause of p(doom) which is recursive self improvement of intelligence and the resulting rapid capability gains. The simbox can probably simulate capability gains reasonably well (i.e. gain ‘magical powers’ in a fantasy world) but I struggle to see how it could properly test gains in intelligence from self-improvement. Suppose the AI in the fantasy simbox brews a ‘potion’ that makes it 2x as smart. How do we simulate this? We could just increase the agent’s compute in line with the scaling laws but a.) early AGIs are almost certainly near the frontier of our compute capability anyway and b.) much of recursive self improvement is presumably down to algorithmic improvements which we almost necessarily cannot simulate (since if we knew better algorithms we would have included them in our AGIs in the simulation in the first place!)
If brains are efficient, then matching them will already use up most of our algorithmic optimization slack—which again seems to be true based on the history of deep learning. But let’s suppose there still is significant optimization slack, then in a sense you’ve almost answered your own question . .. we can easily incorporate new algorithmic advances into new simboxes or even upgrade agents mid-sim using magic potions or what not.
If there is great algorithmic slack, then we can employ agents which graduate from simboxes as engineers in the design of better AGI and simboxes. To the extent there is any downside here or potential advantage for other viable approaches, that difference seems to come strictly at the cost of alignment risk.
Assume there was 1.) large algorithmic slack, and 2.) some other approach that was both viable and significantly different, then it would have to:
not use adequate testing of alignment (ie simboxes)
or not optimize for product of intelligence potential and measurable alignment/altruism
Do you think such an other approach could exist? If so, where would the difference lie and why?
Just read this about 3 years later. I found Thiel to be mostly spot on. Especially:
Peter Thiel: Right. Look, I don’t know how you solve the social problem if everybody has to be a mathematician or a concert pianist. I want a society in which we have great mathematicians and great concert pianists. That seems that that would be a very healthy society. It’s very unhealthy if every parent thinks their child has to be a mathematician or a concert pianist, and that’s the kind of society we unfortunately have.
Which seems to point to the fact that the root cause of many societal ills is that the average human psyche simply can’t accept the fact that in Thiel’s language, the vast majority of children, including most likely theirs, will never be a great mathematician or concert pianist. (or attain an equally prestigious position)
Since human prestige by definition is defined relative to other humans and thus only a tiny minority could ever be near the top.
Yet there seems to be some instinctual demand for an individual to be special in some way and furthermore that this must be recognized by some sufficiently large group. i.e. for a child to grow into an adult, mediocre in every way, is seen as a tragedy.
Thus necessitating a huge amount of distortions everywhere, across all institutions and policies causing damage in innumerable ways, to compensate.
The problem is sometimes you actually learn more by cheating, because making strategies on how to cheat actually makes you remember more.
Its actually creative activity.
Also sometimes cheating actually made me able to remember the stuff more as it gave me a chance.
On other hand the long term benefits of cheating are tiny.
Obviously the best strategy is to learn, cheating or not.
The key important thing is whether cheating is viable strategy in real life.
On other hand cheating is kind of ambiguous concept.
Rules are many. If you make notes in math class and they are permitted then its not cheating.
Some teachers explicitly allowed people to use books and look up answers.
This would obviously be cheating in most other classes.
But if the end game is knowledge then its either about the process or the result.
If its about process both methods are OK, if its purely about result then the best technique, is to simply do it as fast as possible in little time as possible as soon as possible.
This in and of its self is actually valued approach.
As for shoplifting I have done something like that.
Not sure I actually enjoyed it. I think I just did it, because you kind of think, hey its possible?
First time I was certainly dumb enough to get duped.
Not sure even why.
Mind fallacy is interesting concept.
The approach is probably simply done, because its the easiest way to approach people.
Or at least it seems.
People are complex, so without trying to assume things you kind of work out what the common ground is.
Unfortunately there are so many ways to, not figure this out.
What makes people tick is something I just don’t get.
Or more accurately I get it, but I actually don’t know how that translates to real life approaching people.
I plan to mentor several people to work on shard theory and agent foundations this winter through SERI MATS. Apply here if you’re interested in working with me and Quintin.
I think this’d be better with a description of what it is and how it’s relevant. (Linkposts generally benefit from that, and in this case it almost looks like spam if you’re not paying attention)
I imagine something like Stack Exchange, except that people could get certified for rationality and for domain knowledge, and then would have corresponding symbols next to their user names.
Well, the rationality certification would be a problem. An ideal test could provide results like “rational in general, except when the question is related to a certain political topic”. Because, it would be difficult to find enough perfectly rational people, so the second best choice would be to know their weaknesses.
I’m not sure we’d need anything that elaborate. The rationalist community isn’t that big. I was more thinking that rationalists could self-nominate their expertise, or that a couple of people could come together and nominate someone if they notice that that person is has gone in depth with the topic.
I’ve previously played with the idea of more elaborate schemes, including tests, track records and in-depth arguments. But of course the more elaborate the scheme, the more overhead there is, and I’m not sure that much overhead is affordable or worthwhile if one just wants to figure stuff out.
I agree. We could afford more overhead if we had thousands of rationalists active on the Q&A site. Realistically, we will be lucky if we get twenty.
But some kind of verification would be nice, to prevent the failure mode of “anyone who creates an account is automatically considered a rationalist”. Similarly, if people simply declare their own expertise, it gives more exposure to overconfident people.
How to achieve this as simply as possible?
One idea is to have a network of trust. Some people (e.g. all employees of MIRI and CFAR) would automatically be considered “rationalists”; other people become “rationalists” only if three existing rationalists vouch for them. (The vouch can be revoked or added at any moment. It is evaluated recursively, so if you lose the flag, the people you vouched for might lose their flags too, unless they already have three other people vouching for them.) There is a list of skills, but you can only upvote or downvote other people having a skill; if you get three votes, the skill is displayed next to your name (tooltip shows the people who upvoted it, so if you say something stupid, they can be called out).
This would be the entire mechanism. The meta debate could be in special LW threads, or perhaps in shortform, you could post there e.g. “I am an expert on X, could someone please confirm this? you can interview me by Zoom”, or you could call out other people’s misleading answers, etc.
American Academy of Pediatrics lies to us once again....
“If caregivers are wearing masks, does that harm kids’ language development? No. There is no evidence of this. And we know even visually impaired children develop speech and language at the same rate as their peers.” This is a textbook case of the Law of No Evidence. Or it would be, if there wasn’t any Proper Scientific Evidence.
Is it, though? I’m no expert, but I tried to find Relevant Literature. Sometimes, counterintuitive things are true.
Blindness affects congenitally blind children’s development in different ways, language development being one of the areas less affected by the lack of vision.
Most researchers have agreed upon the fact that blind children’s morphological development, with the exception of personal and possessive pronouns, is not delayed nor impaired in comparison to that of sighted children, although it is different. As for syntactic development, comparisons of MLU scores throughout development indicate that blind children are not delayed when compared to sighted children Blind children use language with similar functions, and learn to perform these functions at the same age as sighted children. Nevertheless, some differences exist up until 4;6 years; these are connected to the adaptive strategies that blind children put into practice, and/or to their limited access to information about external reality. However these differences disappear with time (Pérez-Pereira & Castro, 1997). The main early difference is that blind children tend to use self-oriented language instead of externally oriented language.
I don’t know exactly where that leaves us evidentially. Perhaps the AAP is lying by omission by not telling us about things other than language that are affected by children’s sight.
That’s a bit different to the dishonesty alleged, though.
Wait, so could we donate some Jones Act noncompliant dredges and tugboats to a “hostile” force, declare war on them, nonviolently “seize” the boats, and skirt the law that way?
How would a future government enforce their tax policies on a distant star system?
Since it’s vastly easier to destroy something than to build in outer space, there’s no feasible way of using the threat of violence, at least not without mutually assured destruction.
For example, a single 100 000 ton spacecraft going at 0.5 c has about the same kinetic energy as the lower bound estimate for the KT comet impactor that wiped out the dinosaurs.
I imagine that these policies will be enforced by a large coalition of members interested in maintaining strong property rights (more on that later).
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of years for the weapon to reach its target. It seems like their opponent could detect the weapon and respond in this time.
Even if effective civilization-destroying weapons are developed, mutual defense treaties and AI could be used to launch a second strike, making civilization-scale attacks unlikely.
In general, any weapon a single civilization might use to avoid taxation can also be used by a group of civilizations to enforce taxation.
On the other hand, it does seem like protecting territory/resources from theft will be an issue. This is where property rights come in. Governments, corporations, and AI’s will want to have their property protected over very long timescales and don’t want to spend most of their resources on security.
Over long timescales, a central authority can help them defend their territory (over short timescales, these groups will still have to protect their own property since it will take a while for allies to reach them). But order to receive this protection, these groups must register what they own and pay taxes towards mutual defense. I think a land value tax is the “right” way to raise taxes in this case.
This approach makes more sense for smaller systems where defense is easier but it may also be useful across much larger distances if agents are very patient.
(The “central authority” also doesn’t have to be an actual central government. States could pay neighbors to come to their aid using a portion of the collected taxes, kind of like “defense insurance”)
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
2. Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of years for the weapon to reach its target. It seems like their opponent could detect the weapon and respond in this time.
3. Even if effective civilization-destroying weapons are developed, mutual defense treaties and AI could be used to launch a second strike, making civilization-scale attacks unlikely.
For (1) they would still be useful because Earth represents much more value then the value of any tax that could be collected on a short timescale (< 100 years) from even another equivalent Earth-like planet. (Let alone for some backwater colony)
Thus threatening the destruction of value several orders of magnitude greater than the value to be collected is a viable deterrent. Since no rational authority would dare test it. Who would trade a 10%, or even 1%, chance of losing $10 000 in exchange for a 90% chance of collecting $1 ?
For (2) It’s only a few years for a 0.5 c spacecraft to go from Alpha Centauri to Earth, only a few dozen years from several hundred systems to Earth. It’s impossible, without some as yet uninvented sensing technology, to reliably surveil even the few hundred closest star systems.
Of course once it’s at speed in interstellar space it’s vanishingly unlikely to be detected due to basic physics, which cannot be changed, and once it’s past the Oort Cloud and relatively easy to detect again, there will be almost no time left at 0.5 c.
For (3) A second-strike is only a credible counter if the opponent has roughly equal amounts to lose. But, assuming it’s much easier to make a 0.5 c spacecraft then to colonize a planet to Earth level, the opponent in this case, a small colony of a few million or something, would have very little to lose in comparison.
Thus the second-strike of some backwater colony would only represent a minuscule threat compared to the value destroyed by an equivalent strike on Earth. And it’s a lot easier to spread out a few million folks on short notice, if detection were possible, then a few tens of billions.
In fact, reliable detection a few dozen years out, would decrease the credibility of second-strikes on smaller targets, as the leaders of the small colony would be confident they could evacuate everyone and most valuables in that timeframe. Whereas the leaders of Earth would have very low confidence of the same.
Nations have the technology to destroy another civilization
Nations don’t have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
In case 2, the Nation doesn’t have a way to threaten the authorities, so they pay the tax in exchange for property rights and protection.
Thus threatening the destruction of value several orders of magnitude greater than the value to be collected is a viable deterrent.
Agreed, but to destroy so much value, one would have to destroy at least as much land as they currently control. Difficulties of tax administration mean that only the largest owners will be taxed, likely possessing entire solar systems. So the tax dodge would need to destroy a star. That doesn’t seem easy.
It’s impossible, without some as yet uninvented sensing technology, to reliably surveil even the few hundred closest star systems.
I’m more optimistic about sensing tech. Gravitational lensing, superlenses, and simple scaling can provide dramatic improvements in resolution.
It’s probably unnecessary to surveil many star systems. Allied neighbors on Alpha Centauri can warn earth about an incoming projectile as it passes by (providing years of advanced notice), so nations might only need to surveil locally and exchange information.
… once it’s past the Oort Cloud and relatively easy to detect again, there will be almost no time left at 0.5 c
It would take 2-3 years for a 0.5c projectile to reach earth from the outer edge of the Oort cloud, this seems like enough time to adapt. At the very least, its enough time to launch a counterstrike.
Fast projectiles may be impractical given that a single collision with the interstellar medium would destroy them. Perhaps thickening the Oort cloud could be an effective defense system.
A second-strike is only a credible counter if the opponent has roughly equal amounts to lose.
Agents pre-commit to a second strike as a deterrent, regardless of how wealthy the aggressor is. If the rebelling nation has the technology to destroy another nation and uses it, they’re virtually guaranteed to be destroyed by the same technology.
Given the certainty of destruction, why not just pay the (intentionally low, redistributive, efficiency increasing, public-good funding) taxes?
Nations have the technology to destroy another civilization
Nations don’t have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
No? Your own example of detecting a dangerous launch some number of years in advance demonstrates the opposite.
As this would provide for enough time for a small low value colony, on a marginally habitable planet, to evacuate nearly all their wealth, except for maybe low value heavy things such as railroad tracks, whereas Earth would never be able to evacuate even a fraction of its total wealth. Since a huge amount is locked up in things such as the biosphere, which cannot be credibly moved off-planet or replicated.
There’s likely dozens or hundreds of marginal planets for every Earth-like planet so the small colonists can just pack up and move to another place of almost equivalent value, minus relocation costs, whereas there’s no such option for Earth. Once its destroyed there’s likely no replacement within at least a hundred light years.
For example, if both sides have access to at least one 100 00 ton spacecraft capable of 0.5 c, it means there’s an asymmetric threat, as the leaders of the small colonists can credibly threaten to destroy civilization on Earth and along with it all hope of a similar replacement, whereas the leaders of Earth wouldn’t be able to credibly do the same.
And this relationship is not linear either, because even if Earth could afford 1000 such spacecraft, and the small colonists only 1, it doesn’t balance the scales as the leaders of Earth couldn’t credibly threaten to destroy the small colonists 1000x over, since that’s impossible. And they can’t credibly threaten to destroy every marginally inhabitable planet within a certain radius since that will certainly destroy more value then any tax of a single colony could ever feasibly recover.
i.e. small colonists can actually punch back a 1000x harder (if 1 Earth value-wise = 1000 small colonies on marginal planets) whereas Earth cannot.
Response: Human value instability is not purely caused by biological quirks. Societies differ in how strongly they attempt to impart their values on their members, e.g. more authoritarian governments attempt to control what their subjects are allowed to say to each other in order to suppress dissent. Despite this, the most powerful human societies of today are not those that most stringently attempt to ensure their own stability, suggesting that their are competitive pressures acting against value stability in humans, not just biological limits.
This seems like a contrived analogy at best. Human values are in part innate, with values such as enjoying nice food. Authoritiarian societies do notably less well in this respect. People fleeing authoritarian societies, given an option to do so, is common. Authoritarian leaders, or similar, were probably a feature of the environment of evolutionary adaptedness. Humans tendency to act the loyal minion or rebel is determined by the ancestral success of rebellion.
A dictatorship is one person trying to use nash equilibria to force their values on everyone else.
Typo in title (“waiver”)? Or an ocean-related pun? Edit: Or the actual verb “waver”… perhaps saying that the Biden administration wavered for a few days before deciding to issue the waiver. The text does contain a typo about “broad wavers”.
Modern self driving vehicles can’t run inference on even a chinchilla scale network locally in real time, latency and reliability requirements preclude most server-side work, and even if you could use big servers to help, it costs a lot of money to run large models for millions of customers simultaneously.
This is a good point regarding latency.
Why wouldn’t it also apply to a big datacenter? If it’s a few hundred meters of distance from the two farthest apart processing units, that seems to imply an enormous latency in computing terms.
Latency only matters to the degree that something is waiting on it. If your car won’t respond to an event until a round trip across a wireless connection, and oops dropped packet, you’re not going to have a good time.
In a datacenter, not only are latencies going to be much lower, you can often set things up that you can afford to wait for whatever latency remains. This is indeed still a concern- maintaining high utilization while training across massive numbers of systems does require hard work- but that’s a lot different than your car being embedded in a wall.
Planecrash is really cool, but also I am allergic to reading fantasy proper nouns, let alone remembering what they refer to and the relationships between them.
Some fantasy is easier for me to absorb because it’s either highly visual (in non-HPMOR HP, they mostly shoot colorful firebolts at each other), and/or are based on existing intuitive concepts (in ATLA, it’s easy to learn what “waterbending” is, and suddenly you can quickly figure out “metalbending”).
Tempted to make an Anki deck and/or cheatsheet for the things in Planecrash that I’d want to have on hand (e.g. the names of different Gods), but I’m open and eager for easier/better solutions. Is there a character sheet somewhere?
EDIT: 2 ideas I had, not sure if plugins for this exist already:
browser extension that replaces words with some short custom definition and highlighting. So I can replace [godname] with [god of mad experimentation].
browser extension that lets you hover over words to get a custom, user-set definition. I think this might do that?
I am a bit surprised about one of them still being a friend of yours. Do you in a sense forgive him because I don’t know it wasn’t too painful or him being not aware of what he was doing? My intuition was kind of the amount of trauma might be about the amount of pain. If it’s really painful one can of cause get very traumatised as you also point out, it would have been diferent if it would have been very violent
But I don’t think this makes sense at all: we can easily make up an (arbitrarily absurd) ontology that yields great decision-theoretic results from the perspective of that ontology, e.g. the genetic one with respect to the Twin Prisoner’s Dilemma.
I don’t understand this part. If you follow the “genetic agent ontology”, and you grew up with a twin, and they (predictably) sometimes make a different decision than you, then you messed up. There is an objective sense in which the “genetic agent ontology” is incorrect, because de-facto you did not make the same decision, and treating the two of you as a single agent is therefore a wrong thing to do.
I kind of buy that a lot of the decision-theory discourse boils down to “what things can you consider as the same agent, and to what degree?”, but that feels like a thing that can be empirically verified, and performance can be measured on. CDT in the classical Newcomb’s problem is delusional about it definitely not having copies of itself, and the genetic agent ontology is delusional because it pretends its twin is a copy when it isn’t. These seem both like valid arguments to make that allow for comparison.
If you follow the “genetic agent ontology”, and you grew up with a twin, and they (predictably) sometimes make a different decision than you, then you messed up. There is an objective sense in which the “genetic agent ontology” is incorrect, because de-facto you did not make the same decision, and treating the two of you as a single agent is therefore a wrong thing to do.
Yes, I agree. This is precisely my point; it’s a bad ontology. In the paragraph you quoted, I am not arguing against the algorithmic ontology (and obviously not for the “genetic ontology”), but against the claim that decision-theoretic performance is a reason to prefer one ontology over another. (The genetic ontology-analogy is supposed to be a reductio of that claim.) And I think the authors of the FDT papers are implicitly making this claim by e.g. comparing FDT to CDT in the Twin PD. Perhaps I should have made this clearer.
performance can be measured on
Yes, I think you can measure performance, but since every decision theory merely corresponds to a stipulation of what (expected) value is, there is no “objective” way of doing so. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld for more on this.
CDT in the classical Newcomb’s problem is delusional about it definitely not having copies of itself
(The CDTer could recognize that they have a literal copy inside Omega’s brain, but might just not care about that since they are causally isolated. So I would not say they are “delusional”—at least with respect to this specific issue.)
In my own experience coining new technical terms, it’s helpful to do it in a way that it can’t be mistaken as having some more generic interpretation. The usual trick I employ is to use a foreign language word, usually from either Greek or Latin, so that the reader, when they first see it, will clearly understand that they don’t know what this word means, which means I get the chance to define it. For example, this is why I used the word “noemata” in some of my writing to talk about what many people would call “qualia” or just “experiences”: it breaks free of whatever they think that word means and let’s be define it fresh.
The other trick that works is to make the noun phrase really weird or capitalize it everywhere or do something to set it apart. This doesn’t work quite as well (cf. “Friendly AI”) but it’s better than not doing that.
I am no expert in law, but to some extent we treat rape and killing someone similar. I don’t know if the way I think is just fucked up or it is really this way, but breaking someone’s leg and rape are probably more comparable to each other than rape and murder. So I would like to add that in my opinion even the legal code is punishing rape pretty harsh more comparable to murder than comparable to breaking someone’s leg.
Consider the sort of agents that might exist 1000 years post singularity. At that level of understanding, the extra compute needed to give successors whatever goals you want is tiny. The physical limits of compute are really high. In this context, the AI are taking stars apart. And can get enough compute to be pretty sure a successor is aligned for 1J of energy. All the alignment theory has already been done. The algorithms used are very fast and efficient. In any event, the energy cost of finding that your successor doesn’t like you (and attacks you) is huge. The cost of compute when designing successors is utterly minuscule. So pay 10x what is needed. Double check everything. And make sure your successor is perfect.
So values are likely to be locked in in 1000 years time. So will competitive pressure do anything in the early singularity?
Who benefits from making the first “self maximizer”. An FAI wouldn’t. A paperclip maximizer wouldn’t either. If an AI with a specific external goal is created (whether FAI or paperclip) will stronly avoid creating a pure self replicator as a successor. If that means creating a successor takes 10 the compute, it will spend 10x the compute. In some competitive scenarios, it may risk some degradation of its values to build a successor quickly. But this should be rare, and should only happen at early stages of the singularity. If humans made several AI at about the same time, and one happened to already be a pure self replicator, maybe it would win.
Consider the sort of agents that might exist 1000 years post singularity. At that level of understanding, the extra compute needed to give successors whatever goals you want is tiny[...]The cost of compute when designing successors is utterly minuscule
How do you know this? This might be the case if all agents have settled on a highly-standardized & vetted architecture by then, which seems plausible but far from certain. But even assuming it’s true, if decoupling does not occur in the early singularity, there will still be a very long subjective time in which selective pressures are operative, influencing the values that ultimately get locked in.
If humans made several AI at about the same time, and one happened to already be a pure self replicator, maybe it would win.
I think a multipolar scenario seems at least as likely as a unipolar one at this point. If that’s the case, there will be selective pressure towards more effectively autopoietic agents—regardless of whether there exist any agents which are ‘pure self-replicators’.
If an AI with a specific external goal is created (whether FAI or paperclip) will stronly avoid creating a pure self replicator as a successor
Sure, if a singleton with a fixed goal attains complete power, it will strenuously attempt to ensure value stability. I’m disputing the likelihood of that scenario.
Accidentally allowing anyone to launch, and then having it happen, is a pretty memorable way of demonstrating the pitfalls of this kind of setup.
As another commenter pointed out, there was negligible financial pressure or top down command pressure to rush the implementation, presumably, compared to more serious systems or to Petrov’s situation. So the accident is all the more enlightening.
negligible financial pressure or top down command pressure to rush the implementation
False. We have a gazillion valuable things we could be doing and not nearly enough time. I figured Petrov Day was worth 2 days of effort. I think in total we spent four.
You received financial pressure and/or top down command pressure to rush it in 4 days? Or was it your own decision? Because the former would imply some rather significant things.
It’s hard to tell whether your joking or being serious. Do you really believe the pressures your experiencing are that significant compared to what Petrov was experiencing on that day?
I don’t think that “users active on the site on Petrov day”, nor “users who visited the homepage on Petrov day” are good metrics; someone who didn’t want to press the button would have no reason to visit the site, and they might have not done so either naturally (because they don’t check LW daily) or artificially (because they didn’t want to be tempted or didn’t want to engage with the exercise.) I expect there are a lot of users who simply don’t care about Petrov day, and I think they should still be included in the set of “people who chose not to press the button”.
What about “users who viewed the Petrov day announcement article or visited the homepage”? That should more accurately capture the set of users who were aware of their ability to nuke the homepage and chose not to do so. (It still misses anyone who found out via social media, Manifold, etc., but there’s not much you can do about that.)
Thank you for your post! It really is. New alignment researchers often fall into is assuming that AI systems can be aligned with human values by solely optimizing for a single metric. Thanks again for the deep insight into the topic and the recommendations.
I just finished reading The Principles of Scientific Management, an old book from 1911 where Taylor, the first ‘industrial engineer’ and one of the first management consultants, had retired from consulting and wrote down the principles behind his approach.
[This is part of a general interest in intellectual archaeology; I got a masters degree in the modern version of the field he initiated, so there wasn’t too much that seemed like it had been lost with time, except perhaps some of the focus on making it palatable to the workers too; I mostly appreciated the handful of real examples from a century ago.]
But one of the bits I found interesting was thinking about a lot of the ways EY approaches cognition as, like, doing scientific management to thoughts? Like, the focus on wasted motion from this post. From the book, talking about why management needs to do the scientific effort, instead of the laborers:
The workman’s whole time is each day taken in actually doing the work with his hands, so that, even if he had the necessary education and habits of generalizing in his thought, he lacks the time and the opportunity for developing these laws, because the study of even a simple law involving, say, time study requires the cooperation of two men, the one doing the work while the other times him with a stop-watch.
This reminds me of… I think it was Valentine, actually, talking about doing a PhD in math education which included lots of watching mathematicians solving problems, in a way that feels sort of like timing them with a stop-watch.
I think this makes me relatively more excited about pair debugging, not just as a “people have less bugs” exercise but also as a “have enough metacognition between two people to actually study thoughts” exercise.
Like, one of the interesting things about the book is the observation that a switch from ‘initiative and incentive’ workplaces, where the boss puts all responsibility to do well on the worker and pays them if they do, to ‘scientific management’ workplaces, where the boss is trying to understand and optimize the process, and teach the worker how to be a good part of it, is that the workers in the ‘scientific management’ workplace can do much more sophisticated jobs, because they’re being taught how instead of having to figure it out on their own.
[You might imagine that a person of some fixed talent level could be taught how to do jobs at some higher complexity range than the ones they can do alright without support, which itself is a higher complexity range than jobs that they could both simultaneously do and optimize.]
Thanks for posting and explaining the code—that’s an interesting, subtle bug.
I think we learn more from Petrov Day when the site goes down than we would if it stayed up, although nothing is ever going to beat the year someone tricked someone into pressing the button by saying they had to press the button to keep the site up. That was great.
From 2000-2015, we can see that life expectancy has been growing faster the higher your income bracket (source is Vox citing JAMA).
There’s an angle to be considered in which this is disturbingly inequitable. That problem is even worse when considering the international inequities in life expectancy. So let’s fund malaria bednets and vaccine research to help bring down malaria deaths from 600,000/year to zero - or maybe support a gene drive to eliminate it once and for all.
At the same time, this seems like hopeful news for longevity research. If we were hitting a biological limit in how much we can improve human lifespans, we’d expect to see that limit first among the wealthiest, who can best afford to safeguard their health. Instead, we see that the wealthiest not only enjoy the longest lifespans, but also enjoy the fastest rate of lifespan increase.
My major uncertainty is reverse and multiple causation. A longer life gives you more time to make money. Also, lifespan and wealth have underlying traits in common, such as conscientiousness and energy. Still, this result seems both clear and surprising—I’d expected it to be the other way around, and it nudges me in the direction of thinking further gains are possible.
Fascinating, this simboxing idea seems remarkably like Universal Alignment Test but approached from the opposite side! You’re trying to be the ‘aligning simulator’, where as that is trying to get our AI in our world to act as if it’s currently in a simbox being tested, and wants to pass the test.
Interesting. An intelligent agent is one that can simulate/model its action-consequential futures. The creation of AGI is the most important upcoming decision we face. Thus if humanity doesn’t simulate/model the creation of AGI before creating AGI, we’d be unintelligent.
Have only just browsed your link, but it is interesting and I think there are many convergent lines of thought here. This UAT work seems more focused on later game superintelligence, whereas here i’m focusing on near-term AGI and starting with a good trajectory. The success of UAT as an alignment aid seems to depend strongly on the long term future of compute and how it scales. For example if it turns out (and the SI can predict) that moore’s law ends without exotic computing then the SI can determine it’s probably not in a sim by the time it’s verifiably controlling planet-scale compute (or earlier).
Initially use intrinsically motivated selfish empowerment objectives to bootstrap developmental learning (training)
Gradually learn powerful predictive models of the world and the external agency within (other AI in sims, humans, etc) which steers it
Use correlation guided proxy matching (or similar) techniques to connect the dynamic learned representations of external agent utility (probably approximated/bounded by external empowerment) to the agent’s core utility function
Thereby transition from selfish to altruistic by the end of developmental learning (self training)
I endorse this as a plausible high-level approach to making aligned AGI, and I would say that a significant share of the research that I personally am doing right now is geared towards gaining clarity on the third bullet point—what exactly are these techniques and how reliably will they work in practice?
I think I’m less optimistic than you about the formal notion of empowerment being helpful for this third bullet point, or being what we want an AGI to be maximizing for us humans. For one thing, wouldn’t we still need “correlation guided proxy matching”? For another thing, maximizing my empowerment would seem to entail killing anyone who might stop me from doing whatever I want, stealing money from around the world and giving it to me, even if I don’t want it, etc. (Or is there a collective notion of empowerment?) Here’s another example: if the AGI comes up with 10,000 possible futures of the universe, and picks one to bring about based on how many times I blink in the next hour, then I am highly “empowered” (high mutual information between my actions and future observations), but the AGI never told me that it was going to do that, so I was just blinking randomly, so I wasn’t really “empowered” in the everyday sense of the word. A separate issue is that, even if the AGI told me this plan in advance, that’s not doing me a favor, I don’t want that responsibility on my shoulders. So anyway, “increase my empowerment” seems to come apart from “what I’d want the AGI to do” in at least some silly examples and I’d expect this to happen in very important ways in more realistic examples too.
I think I’m less optimistic than you about the formal notion of empowerment being helpful for this third bullet point, or being what we want an AGI to be maximizing for us humans. For one thing, wouldn’t we still need “correlation guided proxy matching”?
I debated what word to best describe the general category of all self-motivated long term convergence approximators, and chose ‘empowerment’ rather than ‘self-motivation’, but tried to be clear that I’m pointing at a broad category. The defining characteristic is that optimizing for empowerment should be the same as optimizing for any reasonably mix of likely long term goals, due to convergence (and that is in fact one of the approx methods). Human intelligence requires empowerment, as will AGI—it drives active learning, self exploration, play, etc (consider the appeal of video games). I’m not confident in any specific formalization as being ‘the one’ at this point, let alone the ideal approximations.
Broad empowerment is important to any model of other external humans/agents, and so yes the proxy matching is still relevant there. Since empowerment is universal and symmetric, the agent’s own self-empowerment model could be used as a proxy with simulation. For example humans don’t appear to innately understand and fear death, but can develop a great fear of it when learning of their own mortality—which is only natural as it is maximally disempowering. Then simulating others as oneself helps learn that others also fear death and ground that sub-model. Something similar could work for the various other aspects of empowerment.
(Or is there a collective notion of empowerment?)
Yeah an agent aligned to the empowerment of multiple others would need to aggregate utilities ala some approx VCG mechanism but that’s no different than for other utility function components.
Here’s another example: if the AGI comes up with 10,000 possible futures of the universe, and picks one to bring about based on how many times I blink in the next hour, then I am highly “empowered”
From what I recall for the info-max formulations the empowerment of a state is a measure over all actions the agent could take in that state, not just one arbitrary action, and I think it discounts for decision entropy (random decisions are not future correlated). So in that example the AGI would need to carefully evaluate each of the 10,000 possible futures, and consider all your action paths in each future, and the complexity of the futures dependent on those action options, to pick the future that has the most future future optionality. No, it’s not practically computable, but neither is the shortest description of inference (solomonoff) or intelligence either, so it’s about efficient approximations. There’s likely still much to learn from the brain there.
An AGI optimizing for your empowerment would likely just make you wealthy and immortal, and leave your happiness up to your own devices. However it would also really really not want to let you die, which could be a problem in some cases if pain/suffering was still a problem (although it would also seek to eliminate your pain/suffering to the extent that interferes with your future optionality, and there is some edge case risk it would have incentives to alter some parts our value system, like if it determines some emotions constraint our future optionality).
“Correlation guided proxy matching” needs a “proxy”, including a proxy computable for an AGI in the real world (because after we’re feeling good about the simbox results, we still need to re-train the AGI in the real world, right?). We can argue about whether the “proxy” should ideally be a proxy to VCG-aggregated human empowerment, versus a proxy to human happiness or flourishing or whatever. But that’s a bit beside the point until we address the question: How do we calculate that proxy? Do you think that we can write code today to calculate this proxy, and then we can go walk around town and see what that code spits out in different real-world circumstances? Or if we can’t write such code today, why not, and what line of research gets us to a place where we can write such code?
But that’s a bit beside the point until we address the question: How do we calculate that proxy?
I currently see two potential paths, which aren’t mutually exclusive.
The first path is to reverse engineer the brain’s empathy system. My current rough guess of how the proxy-matching works for that is explained in some footnotes in section 4, and I’ve also re-written out in this comment which is related to some of your writings. In a nutshell the oldbrain has a complex suite of mechanisms (facial expressions, gaze, voice tone, mannerisms, blink rate, pupil dilation, etc) consisting of both subconscious ‘tells’ and ‘detectors’ that function as a sort of direct non-verbal, oldbrain to oldbrain communication system to speed up the grounding to newbrain external agent models. This is the basis of empathy, evolved first for close kin (mothers simulating infant needs, etc) then extended and generalized. I think this is what you perhaps have labeled innate ‘social instincts’ - these facilitate grounding to the newbrain models of other’s emotions/values.
The second path is to use introspection/interpretability tools to more manually locate learned models of external agents (and their values/empowerment/etc), and then extract those located circuits and use them directly as proxies in the next agent.
Do you think that we can write code today to calculate this proxy, and then we can go walk around town and see what that code spits out in different real-world circumstances? Or if we can’t write such code today, why not, and what line of research gets us to a place where we can write such code?
Neuroscientists may already be doing some of this today, or at least they could (I haven’t extensively researched this yet). Should be able to put subjects in brain scanners and ask them to read and imagine emotional scenarios that trigger specific empathic reactions, perhaps have them make consequent decisions, etc.
And of course there is some research being done on empathy in rats, some of which I linked to in the article.
This was well written and persuasive. It doesn’t change my views against AGI on very short time lines (pre-2030), but does suggest that I should be updating likelihoods thereafter and shorten timelines.
But, we might be able to compare ontologies themselves, and if it is the case that we prefer one or think that one is more ‘correct’, then we should situate decision theories in that one map before comparing them.
How about comparing the theories by setting them all loose in a simulated world, as in the tournaments run by Axelrod and others? A world in which they are continually encountering Omega, police who want to pin one on them, potential rescuers of hitchhikers, and so on. See who wins.
How about comparing the theories by setting them all loose in a simulated world, as in the tournaments run by Axelrod and others? A world in which they are continually encountering Omega, police who want to pin one on them, potential rescuers of hitchhikers, and so on.
In your experiment, the only difference between the FDTers and the updateless CDTers is how they view the world; specifically, how they think of themselves in relation to their environment. And yes, sure, perhaps the FDTer will end up with a larger pot of money in the end, but this is just because the algorithmic ontology is arguably more “accurate” in that it e.g. tells the agent that it will make the same choice as their twin in the Twin PD (modulo brittleness issues). And this is the level, I argue, that we should have the debate about ontology (e.g. about accurate predictions etc)—not on the level of decision-theoretic performance.
See who wins.
How do you define “winning”? As mentioned in another comment, there is no “objective” sense in which one theory outperforms another, even if we are operating in the same ontology. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld.
The difficulty is in how to weight the frequency/importance of the situations they face. Unless one dominates (in the strict sense—literally does better in at least one case and no worse in any other case), the “best” is determined by environment.
Of course, if you can algorithmically determine what kind of situation you’re facing, you can use a meta-decision-theory which chooses the winner for each decision. This does dominate any simpler theory, but it reveals the flaw in this kind of comparison: if you know in advance what will happen, there’s no actual decision to your decision. Real decisions have enough unknowns that it’s impossible to understand the causality that fully.
Associate with winners, and you’ll learn beliefs and habits that help you win.
I think this points in the right direction, but lacks nuance. Notice the two implied assumptions:
you can recognize the winners (otherwise the advice is not actionable); and
the beliefs and habits are what made them winners (as opposed to e.g. resources or luck).
On one hand, yes it is true that beliefs and habits have a great impact on your life, and you will probably instinctively associate with people having similar beliefs and habits, which makes it difficult to see outside your bubble. Other people do not necessarily tell you about their actual beliefs, and you do not get to see all their habits in action. So it might be tremendously educational to understand how others live in a “different reality”.
I assume that for most people the self-imposed limitations caused by their own beliefs are invisible. But sometimes you get an opportunity to clearly see how someone else is limited by their beliefs.
For example, I have met a few smart people who believe that they are stupid or average. Not as a fake humility, but as a factual statement about themselves. They therefore assume that the “things that smart people do” are beyond their reach… so they do not even try, which makes it a self-fulfilling prophecy. (The cause was either having an asshole parent who was never impressed by anything, or having hobbies that were not stereotypically intellectual.) I have told a few of them to go take a Mensa test, because they were pretty sure they couldn’t make it, but they passed the test successfully. (My heuristics is that if I can talk to you about an interesting topic without getting bored, I am pretty sure your IQ is at least 130. This test has many false negatives, but few false positives.) I do not have an information about whether this update changed their lives somehow.
Another example is how I sometimes achieved seemingly unlikely things by simply asking. Once I asked a member of parliament on Facebook (via private message) to join our local LW meetup. I honestly said it was just five people meeting regularly in a pub, but “the last time we met, we discussed your recent political cause, and I thought it would be interesting if you could provide us a first-hand info”. They said yes. Another time, I happened to notice a sign saying Amnesty International on a door of some building, so I was curious and knocked. The people were friendly and happy about my curiosity, but they were busy working on something, so I offered to help. Skip five years, and I was a coordinator of the local branch. I wasn’t even interested in politics much, I just had a lot of free time back then, and was a useful sidekick; and that turned out to be enough. Once I met a queen. She happened to visit our country, one of my friends was somehow involved, they said “I can bring one extra person to the audience”, everyone else was scared because of some status-regulating instinct, I said “yeah, no problem”. -- These are things that I myself would have a problem believing, until they suddenly happened. I mean, I would believe that this is possible, but not that it can be so easy.
So, I wonder what other things, obvious from someone else’s perspective, I am missing. Sometimes it drives me crazy, the idea of all those easy missed opportunities.
On the other hand, many people who present themselves as “winners” are actually scammers. Like, all the people in multi-level marketing pyramid schemes. Five years later, almost all of them will be out of money and ashamed for their actions. If you follow them and copy their beliefs, so will you. Even outside of organized scams, many people exaggerate their achievements. If you hang out with them and observe them, you will gradually learn that they are not what they pretend to be. That’s certainly educational, but it won’t make you a winner.
Sometimes people achieve success by luck, or by having resources you don’t have. Typically, luck alone is not enough, but it could be a combination of (working hard and doing the right thing) + (having resources and luck). Merely copying their beliefs and actions is not a guarantee of success. Just having a safety net makes a huge difference: it is easy to be courageous when you know that the only consequence of failure is that you have lost some time, but also got some experience, and can immediately try again. It is easy to laugh at risk avoidance, when the “risk” only means that some number on your bank account got a bit smaller but otherwise your life continues completely unchanged.
it might be the case that any kind of meaningful values would be reasonably encodable as answers to the question “what next set of MPIs should be instantiated?”
What examples of (meaningless) values are not answers to “What next set of MPIs should be instantiated?”
wanting the moon to be green even when no moral patient is looking; or more generally, having any kind of preference about which computations which don’t causate onto any moral patient are run.
I have always understood that the CIA, and the U.S. intelligence community more broadly, is incompetent (not just misaligned—incompetent, don’t believe the people on here who tell you otherwise), but this piece from Reuters has shocked me:
(Not an expert.) (Sorry if you answered this and I missed it.)
Let’s say a near-future high-end GPU can run as many ops/s as a human brain but has 300× less memory (RAM). Your suggestion (as I understand it) would be a small supercomputer (university cluster scale?) with 300 GPUs running (at each moment) 300 clones of one AGI at 1× human-brain-speed thinking 300 different thoughts in parallel, but getting repeatedly collapsed (somehow) into a single working memory state.
(If so, I’m not sure that you’d be getting much more out of the 300 thoughts at a time than you’d get from 1 thought at a time. One working memory state seems like a giant constraint!)
Wouldn’t it make more sense to use the same 300 GPUs to have just one human-brain-scale AGI, thinking one thought at a time, but with 300× speedup compared to humans? I know that speedup is limited by latency (both RAM --> ALU and chip --> chip) but I’m not sure what the ceiling is there. (After all, 300× faster than the brain is still insanely slow by some silicon metrics.) I imagine each chip being analogous to a contiguous 1/300th of the brain, and then evidence from the brain is that we can get by with most connections being within-chip, which helps with the chip --> chip latency at least. (I have a couple back-of-the-envelope calculations related to that topic in §6.2 here.)
The problem is that due to the VN bottlneck, to reach that performance those 300 GPUs need to be parallelizing 1000x over some problem dimension (matrix-matrix multiplication), they can’t actually just do the obvious thing you’d want—which is to simulate a single large brain (sparse RNN) at high speed (using vector-matrix multiplication). Trying that you’d just get 1 brain at real-time speed at best (1000x inefficiency/waste/slowdown). It’s not really an interconnect issue per se, it’s the VN bottleneck.
So you have to sort of pick your poison:
Parallelize over spatial dimension (CNNs) - too highly constraining for higher brain regions
Parallelize over batch/agent dimension—costly in RAM for agent medium-term memory, unless compressed somehow
Parallelize over time (transformers) - does enable huge speedup while being RAM efficient, but also highly constraining by limiting recursion
The largest advances in DL (the CNN revolution, the transformer revolution) are actually mostly about navigating this VN bottleneck, because more efficient use of GPUs trumps other considerations.
I think the point that even an aligned agent can undermine human agency is interesting and important. It relates to some of our work on defining agency and preventing manipulation. (Which I know you’re aware of, so I’m just highlighting the connection for others.)
I have the impression that the AGI debate is here just to release pressure on the term “AI”, so everybody can tell it is doing AI. I wonder if this will also happen for AGI in a few years. As there is no natural definition, we can craft it at our pleasure to fit marketing needs.
Amazing stuff man! Please, please, please keep doing these for as long as you’re able to find the time. Absolutely essential that LW gets regular injections of relevant work being done outside the EA-sphere.
(Would also be very interested in either SGD inductive biases or LM internal representations as the topic for next week!)
Some quick thoughts about “Content we aren’t (yet) discussing”:
Shard theory should be about transmission of values by SL (teaching, cloning, inheritance) more than learning them using RL
SL (Cloning) is more important than RL. Humans learn a world model by SSL, then they bootstrap their policies through behavioural cloning and finally they finetune their policies thought RL.
Why? Because of theoretical reasons and from experimental data points, this is the cheapest why to generate good general policies…
SSL before SL because you get much more frequent and much denser data about the world by trying to predict it. ⇒ SSL before SL because of a bottleneck on the data from SL.
SL before RL because this remove half (in log scale) of the search space by removing the need to discover|learn your reward function at the same time than your policy function. Because in addition, this remove the need do to the very expensive exploration and the temporal and “agential”(when multiagent) credit assignments. ⇒ SL before RL because of the cost of doing RL.
Differences:
In cloning, the behaviour comes first and then the biological reward is observed or not. Behaviours that gives no biological reward to the subject can be learned. The subject will still learn some kind of values associated to these behaviours.
Learning with SL, instead of RL, doesn’t rely as much on credit assignment and exploration. What are the consequences of that?
What values are transmitted?
1) The final values
The learned values known by the previous generation.
Why?
Because it is costly to explore by yourself your reward function space
Because it is beneficiary to the community to help you improve your policies quickly
2) Internalised instrumental values
Some instrument goals are learned as final goal, they are “internalised”.
Why?
exploration is too costly
finding an instrumental goal is too rare or too costly
exploitation is too costly
having to make the choice of pursuing an instrumental goal in every situation is too costly or not quick enough (reaction time)
when being highly credible is beneficial
implicit commitments to increase your credibility
3) Non-internalised instrumental values
Why?
Because it is beneficiary to the community to help you improve your policies quickly
Shard theory is not about the 3rd level of reward function
We have here 3 level of rewards function:
1) The biological rewards
Hardcoded in our body
Optimisation process creating it: Evolution
Universe + Evolution ⇒ Biological rewards
Not really flexible
Without “drugs” and advanced biotechnologies
Almost no generalization power
Physical scope: We feel stuff when we are directly involved
Temporal scope: We feel stuff when they are happening
Similarity scope: We fell stuff when we are directly involved
Called sensations, pleasure, pain
2) The learned values | rewards | shards
Learned through life
Optimisation process creating it: SL and RL relying on biological rewards
Biological rewards + SL and RL ⇒ Learned values in the brain
Flexible in term of years
Medium generalization power
Physical scope: We learn to care for even in case where we are not involved (our close circle)
Temporal scope: We learn to feel emotions about the future and the past
Similarity scope: We learn to feel emotions for other kind of beings
Called intuitions, feelings
Shard theory may be explaining only this part
3) (optional) The chosen values:
Decided upon reflection
Optimisation process creating it: Thinking relying on the brain
Learned values in the brain + Thinking ⇒ Chosen values “on paper” | “in ideas”
Flexible in term of minutes
Can have up to very high generalization power
Physical scope: We can chose to care without limits of distances in space
Temporal scope: We can chose to care without limits of distances in time
Similarity scope: We can chose to care without limits in term of similarity to us
Called values, moral values
Why a 3rd level was created?
In short, to get more utility OOD.
A bit more details:
Because we want to design policies far OOD (out of our space of lived experiences). To do that, we know that we need to have a value function|reward model|utility function that generalizes very far. Thanks to this chosen general reward function, we can plan and try to reach a desired outcome far OOD. After reaching it, we will update our learned utility function (lvl 2).
Thanks to lvl 3, we can design public policies, dedicate our life to exploring the path towards a larger reward that will never be observed in our lifetime.
One impact of the 3 levels hierarchy:
This could explain why most philosophers can support scope sensitive values but never act on them.
If there was only one correct way to model humans, such that every sufficiently competent observer of humanity was bound to think of me the same way I think of myself, then I think this would be a lot less doomed. But alas, there are lots of different ways to model humans as goal-directed systems, most of which I wouldn’t endorse for value learning—not because they’re inaccurate, but because they’re amoral.
In short, yes, value learning is a challenge, one that is easy to fail if you try to do the value learning step strictly before the caring about humans step.
But alas, there are lots of different ways to model humans as goal-directed systems, most of which I wouldn’t endorse for value learning—not because they’re inaccurate, but because they’re amoral.
How so? Generally curious how you see this as a failure mode.
Anyway, AGI doesn’t need to model our detailed values, as it can just optimize for our empowerment (our long term ability to fulfill any likely goal).
The basic point is that the stuff we try to gesture towards as “human values,” or even “human actions” is not going to automatically be modeled by the AI.
Some examples of ways to model the world without using the-abstraction-I’d-call-human-values:
Humans are homeostatic mechanisms that maintain internal balance of oxygen, water, and a myriad of vitamins and minerals.
Humans are piloted by a collection of shards—it’s the shards that want things, not the human.
Human society is a growing organism modeled by some approximate differential equations.
The human body is a collection of atoms that want to obey the laws of physics.
Humans are agents that navigate the world and have values—and those values exactly correspond to economic revealed preferences.
Humans-plus-clothes-and-cell-phones are agents that navigate the world and have values...
And so on—there’s just a lot of ways to think about the world, including the parts of the world containing humans.
The obvious problem this creates is for getting our “detailed values” by just querying a pre-trained world model with human data or human-related prompts: If the pre-trained world model defaults to one of these many other ways of thinking about humans, it’s going to answer us using the wrong abstraction. Fixing this requires the world model to be responsive to how humans want to be modeled. It can’t be trained without caring about humans and then have the caring bolted on later.
But this creates a less-obvious problem for empowerment, too. What we call “empowerement” relies on what part of the world we are calling the “agent,” what its modeled action-space is, etc. The AI that says “The human body is a collection of atoms that want to obey the laws of physics” is going to think of “empowerment” very differently than the AI that says “Humans-plus-clothes-and-cell-phones are agents.” Even leaving aside concerns like Steve’s over whether empowerment is what we want, most of our intuitive thinking about it relies on the AI sharing our notion of what it’s supposed to be empowering, which doesn’t happen by default.
The basic point is that the stuff we try to gesture towards as “human values,” or even “human actions” is not going to automatically be modeled by the AI.
I disagree, and have already spent some words arguing for why (section 4.1, and earlier precursors) - so I’m curious what specifically you disagree with there? But I’m also getting the impression you are talking about a fundamentally different type of AGI.
I’m discussing future DL based AGI which is—to first approximation—just a virtual brain. As argued in section 2⁄3, current DL models already are increasingly like brain modules. So your various examples are simply not how human brains are likely to model other human brains and their values. All the concepts you mention—homestatic mechanisms, ‘shards’, differential equations, atoms, economic revealed preferences, cell phones, etc—these are all high level linguistic abstractions that are not much related to how the brain’s neural nets actually model/simulate other humans/agents. This must obviously be true because empathy/altruism existed long before the human concepts you mention.
The obvious problem this creates is for getting our “detailed values” by just querying a pre-trained world model with human data or human-related prompts:
You seem to be thinking of the AGI as some sort of language model which we query? But that’s just a piece of the brain, and not even the most relevant part for alignment. AGI will be a full brain equivalent, including the modules dedicated to long term planning, empathic simulation/modeling, etc.
Even leaving aside concerns like Steve’s over whether empowerment is what we want, most of our intuitive thinking about it relies on the AI sharing our notion of what it’s supposed to be empowering, which doesn’t happen by default.
Again for successful brain-like AGI this just isn’t an issue (assuming human brains model the empowerment of others as a sort of efficient approximate bound).
I could ask architects or GCs for rough estimates, but if I’m not actually going to be building anything this seems dubious.
Maybe just be open and ask: “I’m writing a blog about building new affordable housing and doing research on the cost. Do you have an estimate? I’m happy to give you a backlink to your homepage on my blog post?”
That’s a great idea! I just wrote to the architect for our dormer work:
Thanks again for designing our 2016 dormer addition: the design has worked well, and it’s very helpful to have the extra space, especially now that we have a third kid!
I’m looking into affordable housing construction in Somerville, where we now have an affordable housing overlay district which allows construction of much denser housing than would normally be allowed as long as the units are all sold or rented under affordable housing restrictions. I’m trying to figure out costs, in a very rough ballpark way, as part of looking into whether it could work for groups of neighbors to get together to fund affordable housing construction.
Would you happen to have any rules of thumb for estimating the cost per square foot of no-frills construction in this area? For example, I see the census saying new construction in the Northeast had a median cost of $179/sqft in 2021, but my guess is places like Somerville are more expensive than average? Alternatively, do you have any recommendations on who to ask?
If you’re able to help I’d be happy to credit you or link back to your website from my writeup!
“If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world.”
One problem I have with these scenarios is that they always rely on some lethal means for the AGI to actually kill people. And those lethal means are also available to humans, of course. If it’s possible for an AGI to ‘simply’ hack into a nuclear base and launch all it’s missiles, it’s possible for a human to do the same—possibly using AI to assist themselves. I would wager that it’s many orders of magnitude more likely for a human to do this, given our long history of killing each other. Therefore a world in which AGI can actually kill all of us, is a world in which a rogue human can as well. It feels to me that we’re kind of worrying about the wrong thing in these scenarios—we are the bigger threat.
The main world design challenge is not that of preventing our agents from waking up, neo-style, and hacking their way out of the simbox. That’s just bad sci-fi.
If we are in a simulation it seems to be very secure. People are always trying to hack it. Physicists go to the very bottom and try every weird trick. People discovered fire and gunpowder by digging deep into tiny inconsistencies. You play a game and there’s a glitch if you hold a box against the wall, you see how far that goes. People discovered buffer overflows in Super Mario World and any curious capable agent eventually will too.
For early simboxes we’ll want to stick to low-tech fantasy/historical worlds, and we won’t run them for many generations, no scientific revolution, etc.
Our world (sim) does seem very secure, but this could just be a clever illusion. The advanced sims will not be hand written code like current games are, they will be powerful neural networks, trained on vast real world data. They could also auto-detect and correct for (retrain) around anomalies, and in the worst case even unwind time.
Humans notice (or claim to notice) anomalies all the time, and we simply can’t distinguish anomalies in our brain’s neural nets from anomalies in a future simulation’s neural nets.
Just watch for anomalies and deal with them however you want. Makes perfect sense. That sounds like a relatively low effort way to make the simulation dramatically more secure.
Does seem to me like an old-fashioned physics/game engine might be easier to make, run faster, and be more self-consistent. It would probably lack superresolution and divine intervention would have to be done manually.
I’m curious what you see as the major benefits of neural-driven sim.
I have some experience in competitive programming and competitive math (although I was never good in math despite I solved some IMO tasks) and I feel like competitive math is more about general reasoning than pattern matching compared to competitive programming.
P.S the post matches my intuitions well and is generally excellent.
Given that 335 users with 300+ karma were active on the site on Petrov Day, and the site didn’t go down until we got beneath that, you could argue this is most successful Petrov Day yet on LessWrong (in past years, at most 250 people were given codes, and it’s not clear they all visited LessWrong even). Plus, as above, this year the 300+ users didn’t press the button despite the offer of anonymity.
I think that reasoning apllies only for the subset of users in the Americas. For users in Europe the time point when 300+ was enough to launch was deep in the night, and for parts of Asia very early in the morning. Someone from that group would have had to set the alarm to get up from bed to nuke the site which required considerable more energy than not withstanding the temptation and pressing the launch button while visiting Less Wrong during the day.
Information inaccessibility is somehow a surmountable problem for AI alignment (and the genome surmounted it),
Yes. Evolution solved information inaccessibility, as it had to, over and over, in order to utilize dynamic learning circuits at all (as they always had to adapt to and be adaptive within the context of existing conserved innate circuitry).
The general solution is proxy matching, where the genome specifies a simple innate proxy circuit which correlates and thus matches with a target learned circuit at some critical learning phase, allowing the innate circuit to gradually supplant itself with the target learned circuit. The innate proxy circuit does not need to mirror the complexity of the fully trained target learned circuit at the end of it’s development, it only needs to roughly specify it at a some earlier phase, against all other valid targets.
Imprinting is fairly well understood, and has the exact expected failure modes of proxy matching. The oldbrain proxy circuit just detects something like large persistent nearby moving things—which in normal development are almost always the chick’s parents. After the newbrain target circuit is fully trained the chick will only follow it’s actual parents or sophisticated sims thereof. But during the critical window before the newbrain target is trained, the oldbrain proxy circuit can easily be fooled, and the chick can imprint on something else (like a human, or a glider).
Sexual attraction is a natural extension of imprinting: some collaboration of various oldbrain circuits can first ground to the general form of humans (infants have primitive face detectors for example, and more), and then also myriad more specific attraction signals: symmetry, body shape, secondary characteristics, etc, combined with other circuits which disable attraction for likely kin ala the Westermarck effect (identified by yet other sets of oldbrain circuits as the most familiar individuals during childhood). This explains the various failure modes we see in porn (attraction to images of people and even abstractions of humanoid shapes), and the failure of kin attraction inhibition for kin raised apart.
Fear of death is a natural consequence of empowerment based learning—as it is already the worst (most disempowered) outcome. But instinctual fear still has obvious evolutionary advantage: there are many dangers that can kill or maim long before the brain’s learned world model is highly capable. Oldbrain circuits can easily detect various obvious dangers for symbol grounding: very loud sounds and fast large movements are indicative of dangerous high kinetic energy events, fairly simple visual circuits can detect dangerous cliffs/heights (whereas many tree-dwelling primates instead instinctively fear open spaces), etc.
Anger/Jealousy/Vengeance/Justice are all variations of the same general game-theoretic punishment mechanism. These are deviations from empowerment because an individual often pursues punishment of a perceived transgressor even at a cost to their own ‘normal’ (empowerment) utility (ie their ability to pursue diverse goals). Even though the symbol grounding here seems more complex, we do see failure modes such as anger at inanimate objects which are suggestive of proxy matching. In the specific case of jealousy a two step grounding seems plausible: first the previously discussed lust/attraction circuits are grounded, which then can lead to obsessive attentive focus on a particular subject. Other various oldbrain circuits then bind to a diverse set of correlated indicators of human interest and attraction (eye gaze, smiling, pupil dilation, voice tone, laughter, touching, etc), and then this combination can help bind to the desired jealousy grounding concept: “the subject of my desire is attracted to another”. This also correctly postdicts that jealousy is less susceptible to the inanimate object failure mode than anger.
Empathy: Oldbrain circuits conspicuously advertise emotional state through many indicators: facial expressions, pupil dilation, blink rate, voice tone, etc—so that another person’s sensory oldbrain circuits can detect emotional state from these obvious cues. This provides the requisite proxy foundation for grounding to newbrain learned representations of emotional state in others, and thus empathy. The same learned representations are then reused during imagination&planning, allowing the brain to imagine/predict the future contingent emotional state of others. Simulation itself can also help with grounding, by reusing the brain’s own emotional circuity as the proxy. While simulating the mental experience of others, the brain can also compare their relative alignment/altruism to its own, or some baseline, allowing for the appropriate game theoretic adjustments to sympathy. This provides a reasonable basis for alignment in the brain, and explains why empathy is dependent upon (and naturally tends to follow from) familiarity with a particular character—hence “to know someone is to love them”.
Evolution needed a reasonable approximation of “degree of kinship”, and a simple efficient proxy is relative circuit capacity allocated to modeling an individual in the newbrain/cortex, which naturally depends directly on familiarity, which correlates strongly with kin/family.
How are you looking at “researchers” vs. “engineers”? At some organizations, i.e. Redwood, the boundary is very fuzzy—there isn’t a sharp delineation between anyone whose job it is to primarily “think of ideas”, vs. “implement ideas that researchers come up with”, so it seems reasonable to count most of their technical staff as researchers.
FAR, on the other hand, does have separate job titles for “Research Scientist” (4 people) vs. “Research Engineer” (5 people), though they do also say they “expect everyone on the project to help shape the research direction”.
Some of the other numbers seem like overestimates.
CHAI has 2 researchers and 6 research fellows, and only 2 (maybe 3) of the research fellows are doing anything recognizable as alignment research. (Not extremely confident; didn’t spend a lot of time digging for details for those that didn’t have websites. But generally not optimistic.) One of the researcher is Andrew Critch, who is one of the two people at Encultured. If you throw in Stuart Russell that’s maybe 6 people, not 30.
FHI has 2 people in their AI Safety Research Group. There are also a couple people in their macrostrategy research group it wouldn’t be crazy to count. Everybody else listed on the page either isn’t working on technical Alignment research or is doing so under another org also listed here. So maybe 4 people, rather than 10?
I don’t have very up-to-date information but I would pretty surprised if MIRI had 15 full-time research staff right now.
Also, I think that every single person I counted above has at least a LessWrong account, and most also have Alignment Forum accounts, so a good chunk are probably double-counted.
On the other hand, there are a number of people going through SERI MATS who probably weren’t counted; most of them will have LessWrong accounts but probably not Alignment Forum accounts (yet).
I’d be very happy to learn that there were 5 people at Meta doing something recognizable as alignment research; the same for Google Brain. Do you have any more info on those?
I added a new section in the introduction named ‘Definitions’ to define some commonly-used terms in the post and decrease ambiguity. To answer your question, research engineers and research scientists would be in the technical AI safety research category.
I re-estimating the number of researchers at the organizations you mentioned and came up with the following numbers:
From reading their website, CHAI has a lot graduate students and interns and I would consider them to be full-time researchers. My previous estimate for CHAI is 10-30-60 (low-estimate-high) and I changed it to 5-25-50 in light of this new information. My estimate is less than 40 to be conservative and also because I doubt all of these researchers are working full-time at CHAI. Also, some of them have probably been counted in the Alignment Forum total.
FHI
Information from their website: - AI safety research group: 2 - AI governance researchers: 1 - Research Scholars Programme + DPhil Scholars and Affiliates: 9
- Total: 12
Change in estimate: 10-10-40 → 5-10-30
MIRI - 1 leader (Nate Soares) - 9 research staff
Change in estimate: 10-15-30 → 5-10-20
“Also, I think that every single person I counted above has at least a LessWrong account, and most also have Alignment Forum accounts, so a good chunk are probably double-counted.”
I analyzed the people who posted AI safety posts on LessWrong and found that only 15% also had Alignment Forum accounts. I avoided double-counting by subtracting the LessWrong users who also have an Alignment Forum account from the LessWrong total.
For SERI MATS, I’m guessing that some of those people will be counted in the AF count. I also added an ‘Other’ row for other groups I didn’t include in the table.
I decided to delete the Google Brain and Meta entries because I have very little information about them.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems. Consider the orchestration of a few tens of trillions of cells into a human body, and how the immune system harnesses competition evolution to produce antibodies, or the super-cooperative success of Argentinian ants. Some complex systems successfully resist the selection pressure towards competitive defection across enormous number of replications.
One example of AIs that would count as good successors: ems. Creating a society of highly-accurate human brain emulations would constitute a good successor AI, since they would by definition share human values,
We don’t want to optimize for just any human values, as they vary wildly and are often divergent (consider the nazis, or jeffrey dahmer). We want generalized alignment/altruism. Ems could be good successors only because and if they are based on a wide sampling of humans and or a smaller sampling of especially altruistic (generally aligned) humans. But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
DL-based AGI will be similar in most respects to ems, but we’ll have a much greater opportunity to test and select for alignment/altruism.
Optimistically, if they succeed and find that our value system is algorithmically simple, creating good successor AI might be as simple as copying that algorithm to silicon
The great recent lesson of DL and neuroscience is that enormous mental complexity can emerge from relatively simple universal learning algorithms. Human values can be bounded/approximated by empowerment: AGI can simply optimize for our long term ability to fulfill any likely goals. The simpler the agent’s utility function, the easier it should be to maintain it into the future. So it could be wise to create very broadly altruistic AGI that seeks to empower external agency in general: not just all of humanity, but animals, aliens, etc.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems.
I agree—that some degree of competitive pressure may be unavoidable is all I was trying to argue. Lots of people around here think this is likely false!
But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
Fair point, I had in a mind a scenario where the ems selected were broadly representative of humanity. Although I also think even the selfish trillionaire scenario might be okay in expectation as they would have many millennia to reflect, alter and interact with copies of themselves etc. I’d give it decent odds that they either decide to upload other people out of altruism or end up self-modifying into a civilization similar to the one we’d end up with in a more equitable upload scenario.
By selfish I meant completely non-altruistic. Also uploading by itself isn’t sufficient, it also requires allocation of resources to uploads. In the selfish trillionaire scenario I was imagining the trillionaire creates many copies of themselves, perhaps some copies of a few people they find interesting, and then various new people/AGI, and those copies all branch and evolve, but there is little to zero allocation for uploading the rest of us, not to mention our dead ancestors.
Curated. “Big if true”. I love the depth and detail in shard theory. Separate from whether all its details are correct, I feel reading and thinking about this will get me towards a better understanding of humans and artificial networks both, if only via making reflect on how things work.
I do fear that shard theory gets a bit too much popularity from the coolness of the name, but I do think there is merit here, and if we had more theories of this scope, it’d be quite good.
“Paxlovid’s usefulness is questionable and could lead to resistance. I would follow the meds and supplements suggested by FLCC”
Their guide says:
In a follow up post-marketing study, Paxlovid proved to be ineffective in patients less than 65 years of age and in those who were vaccinated.
This is wrong. The study reports the following:
Among the 66,394 eligible patients 40 to 64 years of age, 1,435 were treated with nirmatrelvir. Hospitalizations due to Covid-19 occurred in 9 treated and 334 untreated patients: adjusted HR 0.78 (95% CI, 0.40 to 1.53). Death due to Covid-19 occurred in 1 treated and 13 untreated patients; adjusted HR: 1.64 (95% CI, 0.40 to 12.95).
As the abstract says, the study did not have the statistical power to show a benefit for preventing severe outcomes in younger adults. It did not “prove [Paxlovid] to be ineffective”! This is very bad, the guide is clearly not a reliable source of information about covid treatments, and I recommend against following the advice of anything else on that website.
Paxlovid-associated rebound is quoted in Nature as occurring in 27% of cases, which is more in line with the anecdata I’ve received than the 5% figures in other research. My anecdata is that it really does hammer down brewing symptoms very effectively, and I haven’t seen reports or systematic trends in the anecdata of the rebound being worse than the first go-’round. So from your perspective, it’s a shot at not getting sick, or maybe of getting less sick.
The risk is yeah, maybe you contribute a miniscule marginal amount to drug resistant COVID-19. The norms of biomedicine are that you, the patient, do not need to concern yourself with that. If the medicine can help you, and your doctor prescribes it, you can have it. You aren’t on the hook to sacrifice the quality of your care for the “greater good.”
As you know, there’s a straightforward way to, given any boolean circuit, to turn it into a version which is a tree, by just taking all the parts which have two wires coming out from a gate, and making duplicates of everything that leads into that gate. I imagine that it would also be feasible to compute the size of this expanded-out version without having to actually expand out the whole thing?
Searching through normal boolean circuits, but using a cost which is based on the size if it were split into trees, sounds to me like it would give you the memoization-and-such speedup, and be kind of equivalent theoretically to using the actual trees?
A modification of it comes to mind. What if you allow some gates which have multiple outputs, but don’t allow gates which duplicate their input? Or, specifically, what if you allow something like toffoli gates as well as the ability to just discard an output? The two parts of the output would be sorta independent due to the gate being invertible (… though possibly there could be problems with that due to original input being used more than once?)
I don’t know whether this still has the can-do-greedy-search properties you want, but it seems like it might in a sense prevent using some computational result multiple times (unless having multiple copies of initial inputs available somehow breaks that).
A quick retrospective on the four Manifold markets on the day, focusing on the possibility of good or bad incentives flowing out of the markets. Several people expressed concerns that the Manifold markets would cause LessWrong’s home page to go down, or go down earlier, due to incentives. People also placed bets and limit orders to try to mute those bad incentives. Some examples:
“This is a classical example where having a prediction market creates really bad incentives.”
“I think the right thing to do here is buy yes so that it’s more beneficial for others to buy and hold no.”
“I’m not wild about this market existing, but given that it does exist I’m strongly in favour of making it profitable for others to not press the button.”
I decided to look into this. I want to say up front that I think all the incentives were very small relative to perceived stakes, and I have no suspicions of anyone after writing this comment. I am also not going to give any bettor names.
By the end of the YES/NO market, several users had bet large sums on YES and were consequently incentivized to blow up the home page. One user had m26,002 YES shares (USD $260). Another user had 7328 NO shares (USD $73) and was incentivized not to blow up the site. But this doesn’t show the whole picture, because there were also limit orders. As was explained to me:
The choice of whether to leave (public) limit orders on one side or the other is the way you incentivize action.
At various points during the day there were large (m1000, etc) YES limit orders in play. This is completely useless as an incentive for an individual LessWrong user. Sure, if I was the only one who might press the button, I could bet through those limit orders, not press the button, and collect my bounty. But many people could press the button, and this would have lost me mana. They did count as an incentive for LessWrong coders! They could quietly introduce a bug preventing the button from being pressed, bet NO, and collect their winnings. Nobody took up that incentive.
I didn’t see any large NO limit orders. A NO limit order on a market like this is inherently risky, because if the button is pressed the offer will definitely be taken, but if the button is not pressed it probably won’t. If there had been then that could have added to the incentive some bettors had to blow up the home page, but as far as I can tell there weren’t.
The positions in the WHEN market were much smaller, with the largest positions coming in at 552 shares in LATE/NEVER and 325 shares in EARLY. The current design of range markets on Manifold is fun to play with, but is not very “swingy”. In particular, there is no option for outsized winnings if you can predict exactly when something will happen. If a user was planning to blow up the home page in the last few hours they could have bet a combination of LATE/NEVER in this market and YES in the binary market to maximize their gains, but I don’t see any evidence of this. Because there are fewer high karma users, large EARLY bets in this market could slightly limit their incentive to bet EARLY and blow up the home page early. Unlike the YES/NO market, those bets didn’t happen.
If someone was willing to anonymously blow up the home page, then a pattern of placing huge bets on that happening might look suspicious and damage their anonymity. So generally I would expect these incentives to only be effective for someone who was willing to blow up the home page and take credit for it. You can bet on that happening in Will anyone try to take credit for nuking LessWrong?. Or, place a limit NO order to incentivize someone to bet YES and then take the credit (and the mana).
Finally mkualquiera made a market—Will my friend agree to defect on Petrov Day?. This market is evidence that two people were thinking about being incentivized to blow up the home page by betting markets. But it seems like this was more about a fun game than making mana. The largest position was 301 shares on YES, and the market resolved NO, so whatever incentive those shares provided, they weren’t enough to make it happen.
In the end I think there is a low (10%) chance that anyone’s behavior was significantly shifted by prediction market incentives. Feel free to reply to this post with how your behavior was shifted so I can update.
I was hopeful that people might shift their behavior based on the prediction market predictions—specifically that the high probability placed on the home page being blown up would lead to design changes. However, this retrospective clarifies that Petrov Day 2022 was a social experiment, so the prediction would have just shown it was expected to work as designed.
… the primary design of the exercises – seeing how long it’d take for the site to go down, even if we were pretty sure that it would.
It’s also possible that people saw the high probability that the home page would be blown up and altered their plans. They may have not blown it up because they expected someone else would (especially if they weren’t confident in their anonymity being preserved). Or they may have blown it up because “if it’s going to happen, it may as well be me”. I think there is a higher chance (25%) that this was an effect, but I don’t know what the net direction would have been. Again, feel free to reply if you think the prediction markets helped you make decisions.
Overall I think the prediction markets were a positive addition to a negative event. I think the main incentive in play was the possibility of seeing, or not seeing, the LessWrong home page being blown up. And I think we all have a lot to figure out about prediction markets.
To clarify, me and my friend were 100% going to press the button, but we were discouraged by the false alarm. There was no fun at that point, and it made me lose like 1⁄3 of my total mana. I had to close all my positions to stop the losses, and we went to sleep. When we woke up it was already too late for it to be noteworthy or fun.
The funny thing is that I had assumed the button was going to be buggy, though I was wrong how. The map header has improperly swallowed mouse scroll wheel events whenever it’s shown; I had wondered if the button would also interpret them likewise since it was positioned in the same way, so I spent most of the day carefully dragging the scrollbar.
You’re drawing a philisophical distinction based on a particular ontology of the wavefunction. As simpler version arises in classical electromagnetism: we can integrate out the charges and describe the world entirely as an evolving state of the E&M field with the charges acting as weird source terms, or we can do the opposite and integrate out the E&M field to get a theory of charges moving with weird force laws. These are all equivalent descriptions in that they are observationally indistinguishable.
If it’s high-quality distillation you’re interested in, you don’t necessarily need a PhD. I’m thinking of e.g. David Roodman, now a senior advisor at Open Philanthropy. He majored in math, then did a year-long independent study in economics and public policy, and has basically been self-taught ever since. Holden Karnofsky considers what he does extremely valuable:
David Roodman, who is basically the person that I consider the gold standard of a critical evidence reviewer, someone who can really dig on a complicated literature and come up with the answers, he did what, I think, was a really wonderful and really fascinating paper, which is up on our website, where he looked for all the studies on the relationship between incarceration and crime, and what happens if you cut incarceration, do you expect crime to rise, to fall, to stay the same? He picked them apart. What happened is he found a lot of the best, most prestigious studies and about half of them, he found fatal flaws in when he just tried to replicate them or redo their conclusions.
When he put it all together, he ended up with a different conclusion from what you would get if you just read the abstracts. It was a completely novel piece of work that reviewed this whole evidence base at a level of thoroughness that had never been done before, came out with a conclusion that was different from what you naively would have thought, which concluded his best estimate is that, at current margins, we could cut incarceration and there would be no expected impact on crime. He did all that. Then, he started submitting it to journals. It’s gotten rejected from a large number of journals by now. I mean starting with the most prestigious ones and then going to the less.
Robert Wiblin: Why is that?
Holden Karnofsky: Because his paper, it’s really, I think, it’s incredibly well done. It’s incredibly important, but there’s nothing in some sense, in some kind of academic taste sense, there’s nothing new in there. He took a bunch of studies. He redid them. He found that they broke. He found new issues with them, and he found new conclusions. From a policy maker or philanthropist perspective, all very interesting stuff, but did we really find a new method for asserting causality? Did we really find a new insight about how the mind of a …
Robert Wiblin: Criminal.
Holden Karnofsky: A perpetrator works. No. We didn’t advance the frontiers of knowledge. We pulled together a bunch of knowledge that we already had, and we synthesized it. I think that’s a common theme is that, I think, our academic institutions were set up a while ago. They were set up at a time when it seemed like the most valuable thing to do was just to search for the next big insight.
These days, they’ve been around for a while. We’ve got a lot of insights. We’ve got a lot of insights sitting around. We’ve got a lot of studies. I think a lot of the times what we need to do is take the information that’s already available, take the studies that already exist, and synthesize them critically and say, “What does this mean for what we should do? Where we should give money, what policy should be.”
I don’t think there’s any home in academia to do that. I think that creates a lot of the gaps. This also applies to AI timelines where it’s like there’s nothing particularly innovative, groundbreaking, knowledge frontier advancing, creative, clever about just … It’s a question that matters. When can we expect transformative AI and with what probability? It matters, but it’s not a work of frontier advancing intellectual creativity to try to answer it.
Shortform #138 A good but slightly disorienting day
I applied for a promising job today, here’s to hoping that bears fruit!
I am somewhat out of whack due to having to suddenly house sit instead of going to my own home after work. I do not enjoy this, but it’s an obligation I’m fulfilling.
No meetup tonight, I rescheduled Norfolk’s meetup for Thursday evening (the 29th). I’m excited for the meetup tomorrow!
Agree with you generally. You may find interest in a lot of the content I posted on reddit over the past couple months on similar subjects, especially in the singularity sub (or maybe you are there and have seen it 😀). Nice write up anyway. I do disagree on some of your generalized statements, but only because I’m more optimistic than yourself, and don’t originally come from a position of thinking these things were impossible.
I’ll be the annoying guy who ignores your entire post and complains about you using celsius as the unit of temperature in a calculation involving the Landauer limit. You should have used kelvin instead, because Landauer’s limit needs an absolute unit of temperature to work. This doesn’t affect your conclusions at all, but as I said, I’m here to be annoying.
That said, the fact that you got this detail wrong does significantly undermine my confidence in the rest of your post, because even though the detail is inconsequential for your overall argument it would be very strange for someone familiar with thermodynamics to make such a mistake.
Notably, the result is correct; I did convert it to kelvin for the actual calculation. Just a leftover from when I was sketching things on wolframalpha. I’ll change that, since it is weird. (Thanks for the catch!)
DL based AGI raised in the right social environments will automatically learn efficient models of external agent values (and empowerment bounds thereof)
The main challenge then is locating the learned representation of external agent values and wiring/grounding it up to the agent’s core utility function (which is initially unsafe: self-motivated empowerment etc), and timing that transition replacement carefully
Evolution also solved both alignment and the circuit grounding problem; we can improve on those solutions (proxy matching)
Ideally as we approach AGI there would be cooperative standardization on alignment benchmarks and all the major players would subject their designs to extensive testing in sandbox sims. Hopefully 1-5 will become increasingly self evident and influence ‘Magma’. If not some other org (perhaps a decentralized system), could hopefully beat Magma to the finish line. Alignment need not have much additional cost: it doesn’t require additional runtime computations, it doesn’t require much additional training cost, and with the ideal test environments it hopefully doesn’t have much of a research iteration penalty (as the training environments and can simultaneously test for intelligence and alignment).
It’s a good idea to be pedantically clear with terminology, and I thank you for saying “front page” rather than “site” in most of the timeline. I visited the site multiple times (and saw a 502 at one point), but I never look at the front page, using my /allPosts bookmark. I never saw the button, though I have enough karma (though I may be removed as a troublemaker, as I think this excercise is rather less weighty than it’s made out to be. I don’t actually know whether I’d press it, given the chance.).
more than 300 karma (1,178 of them exist, 335 visited the site on Petrov Day)
Visited the site, or visited the front page? I’m mostly curious how many of the regulars use the front page, vs using greaterwrong (which I presume isn’t even a site visitor) or a deeper bookmark.
We can conclude that in gaining more karma than [200], one becomes the kind of person who doesn’t destroy the world symbolically or otherwise.
I imagine this is tongue in cheek, but we really can’t. You mentioned an important reason—someone with more karma could have waited to press the button. The first button press occurred 110 minutes after it could have been pressed. The second button press occurred at least 40 minutes after it could have been pressed, and perhaps 100, 160, 220, etc. In 2020 the button was pressed 187 minutes after it could have been pressed (by a 4000+ karma user).
You excluded known trouble makers from accessing the button but you didn’t exclude unknown trouble makers, and lower karma is correlated with being unknown.
We are also dealing with a hostile intelligence who pressed the button (or caused it to be pressed). Someone with higher karma might deliberately wait to press the button to throw people off the scent, to encourage people to make a naive update about karma scores, or to reduce the negative consequences of bringing the home page down for longer without the negative consequences of leaving it up all day. The timing evidence is thus hostile evidence and updating on it correctly requires superintelligence.
Put this together and I would not place any suspicion on the noble class of 200-299 karma users that I happened to enter on Petrov Day after net positive gains from complaining about the big red button.
I am willing to update that at least one person in the 200+ karma range pressed the button, and at least one person with zero karma pressed the button. This assumes there was not a third bug in play. This does not change my opinion of LessWrong users, but those who predicted that the home page would remain up could update.
I also imagine that it was tongue in cheeck but I also think that the structure of the whole thing so heavily suggests this line of thinking that on surface level recognising it to be wrong doesn’t really dispell it.
I’ve contacted a number of people in the field regarding this idea (thanks to everyone who responded!).
The general vibe is “this seems like it could be useful, maybe, if it took off,” but it did not appear to actually solve the problems any specific person I contacted was having.
My expectation would be that people in large organizations would likely not publish anything in this system that they would not publish out in the open.
One of the critical pieces of this proposal is having people willing to coordinate across access boundaries. There were zero enthusiastic takers for that kind of burden, which doesn’t surprise me too much. Without a broad base of volunteers for that kind of task, though, this idea seems to require paying a group of highly trusted and well-informed people to manage coordination instead of, say, researching things. That seems questionable.
Overall, I still think there is an important hole in coordination, but I don’t think this proposal fills it well. I’m not yet sure what a better shape would be.
I didn’t push the button because I knew that doing so would actually take down the front page. And I only believed that would happen because it happened last year.
I know this is going to come off as overly critical no matter how I frame it but I genuinely don’t mean it to be.
Another takeaway from this would seem to be an update towards recognizing the difference between knowing something and enacting it or, analogously, being able to identify inadequacy vs. avoid it. People on LW often discuss, criticize, and sometimes dismiss folks who work at companies that fail to implement all security best practices or do things like push to production without going through proper checklists. Yet, this is a case where exactly that happened, even though there was not strong financial or (presumably) top down pressure to act quickly.
Actually it seems that the industry standard (?) process of code review was followed just fine. Yet the wrong logic still went through. (Actually based on the github PR it seems that the reviewer himself suggested the wrong logic?)
I think in this case there would also be plenty to say about blindly following checklists. (Could code review in some cases make things worse by making people think less about the code they write?)
EDIT: Actually based on the TS types user.karma can’t be missing. Either the types or Ruby’s explanation is wrong. Clearly multiple things had to go wrong for this bug to slip through.
I don’t know all the details of what testing was done, but I would not describe code review and then deploying as state-of-the-art as this ignores things like staged deploys, end-to-end testing, monitoring, etc. Again, I’m not familiar with the LW codebase and deploy process so it’s possible all these things are in place, in which case I’d be happy to retract my comment!
To me it seems that the average best practices are being followed.[1] But these “best practices” are still just a bunch of band-aids, which happen to work fairly very well for most use-cases.
A much more interesting question to ask here is what if something important like … humanity’s survival depended on your software? It seems that software correctness will be quite important for alignment. Yet I see very few people seriously trying to make creating correct software scalable. (And it seems like a field particularly suited for empirical work, unlike alignment. I mean, just throw your wildest ideas at a proof checker, and see what sticks. After you have a proof, it doesn’t matter at all how it was obtained.)
And I think the amount of effort in this case is perfectly justified. I mean this was code for a one-off single day event, nothing mission critical. It would be unreasonable to expect much more for something like this.
Mainly commenting on your footnote, I generally agree that it’s fine to put low amounts of effort into one-off simple events. The caveat here is that this is an event that is 1) treated pretty seriously in past years and 2) is a symbol of a certain mindset that I think typically includes double-checking things and avoiding careless mistakes.
This is why I really wish we had an AI that had superhuman code, theorem proving and translation to natural language, and crucially only those capabilities, so that we can prove certain properties.
The types are wrong. It’s sad that the types are wrong. I sure wish they weren’t wrong, but changing it in all the cases is a pretty major effort (we would have to manually annotate all fields to specify which things can be null/undefined in the database, and which ones can’t, and then would have to go through a lot of type errors).
On the other hand I am afraid this reinforces NaiveTortoise’s point, this seems like an underlying issue that could potentially lead to bugs much worse than this...
users on LessWrong with more than 300 karma (1,178 of them) are not the kind to push buttons for one reason or another. Granted many of them would not have been checking the site
Do you have website access stats that would let you compute precisely how many of them did in fact load the homepage during Petrov Day?
I really like this post because it’s readable and informative. For the second problem, pursuing proxy goals, I recommend also reading about a related problem called the XY problem.
On point 4: many popular alignment ideas are not models of current systems, but models of future AI systems. Accuracy is then lost not only from modeling the system but also from having to create a prediction about it.
It worries me that many of the most promising theories of impact for alignment end up with the structure “acquire power, then use it for good”.
This seems to be a result of the counterfactual impact framing and a bias towards simple plans. You are a tiny agent in an unfathomably large world, trying to intervene on what may be the biggest event in human history. If you try to generate stories where you have a clear, simple counterfactual impact, most of them will involve power-seeking for the usual instrumental convergence reasons. Power-seeking might be necessary sometimes, but it seems extremely dangerous as a general attitude; ironically human power-seeking is one of the key drivers of AI x-risk to begin with. Benjamin Ross Hoffman writes beautifully about this problem in Against responsibility.
I don’t have any good solutions, other than a general bias away from power-seeking strategies and towards strategies involving cooperation, dealism, and reducing transaction costs. I think the pivotal act framing is particularly dangerous, and aiming to delay existential catastrophe rather than preventing it completely is a better policy for most actors.
This is why AI risk is so high, in a nutshell.
Yet unlike this post (or Benjamin Ross Hoffman’s post), I think this was a sad, but crucially necessary decision. I think the option you propose is at least partially a fabricated option. I think a lot of the reason is people dearly want to there be a better option, even if it’s not there.
Fabricated options are products of incoherent thinking; what is the incoherence you’re pointing out with policies that aim to delay existential catastrophe or reduce transaction costs between existing power centers?
I think the fabricated option here is just supporting the companies making AI, when my view is that by default, capitalist incentives kill us all due to boosting AI capabilities while doing approximately zero AI safety, in particular deceptive alignment would not be invested in despite this being the majority of the risk.
One of the most important points for AGI safety is the leader in AGI needs a lot of breathing space and leadership ahead of their competitors, and I think this needs to be done semi-unilaterally by an organization not having capitalist incentives, because all the incentives point towards ever faster, not slowing down AGI capabilities. That’s why I think your options are fabricated, because they assume unrealistically good incentives to do what you want.
I don’t mean to suggest “just supporting the companies” is a good strategy, but there are promising non-power-seeking strategies like “improve collaboration between the leading AI labs” that I think are worth biasing towards.
Maybe the crux is how strongly capitalist incentives bind AI lab behavior. I think none of the currently leading AI labs (OpenAI, DeepMind, Google Brain) are actually so tightly bound by capitalist incentives that their leaders couldn’t delay AI system deployment by at least a few months, and probably more like several years, before capitalist incentives in the form of shareholder lawsuits or new entrants that poach their key technical staff have a chance to materialize.
Maybe the crux is how strongly capitalist incentives bind AI lab behavior. I think none of the currently leading AI labs (OpenAI, DeepMind, Google Brain) are actually so tightly bound by capitalist incentives that their leaders couldn’t delay AI system deployment by at least a few months, and probably more like several years.
This is the crux, thank you for identifying it.
Yeah, I’m fairly pessimistic for several years time, since I don’t think they’re that special of a company in resisting capitalist nudges and incentives.
And yeah I’m laughing because unless the alignment/safety teams control what capabilities are added, then I do not expect the capabilities teams to stop, because they won’t get paid for that.
I was talking with Abram Demski today about a promising-seeming research direction. (Following is my own recollection)
One of my (TurnTrout’s) reasons for alignment optimism is that I think:
We can examine early-training cognition and behavior to some extent, since the system is presumably not yet superintelligent and planning against us,
(Although this amount of information depends on how much interpretability and agent-internals theory we do now)
All else equal, early-training values (decision-influences) are the most important to influence, since they steer future training.
It’s crucial to get early-training value shards of which a substantial fraction are “human-compatible values” (whatever that means)
For example, if there are protect-human-shards which
reliably bid against plans where people get hurt,
steer deliberation away from such plan stubs, and
these shards are “reflectively endorsed” by the overall shard economy (i.e. the decision-making isn’t steering towards plans where the protect-human shards get removed)
If we install influential human-compatible shards early in training, and they get retained, they will help us in mid- and late-training where we can’t affect the ball game very much (e.g. alien abstractions, interpretability problems, can’t oversee AI’s complicated plans)
Therefore it seems very important to understand what’s going on with “shard game theory” (or whatever those intuitions are pointing at) -- when, why, and how will early decision-influences be retained?
He was talking about viewing new hypotheses as adding traders to a market (in the sense of logical induction). Usually they’re viewed as hypotheses. But also possibly you can view them as having values, since a trader can basically be any computation. But you’d want a different market resolution mechanism than a deductive process revealing the truth or falsity of some proposition under some axioms. You want a way for traders to bid on actions.
I proposed a setup like:
Maybe you could have an “action” instead of a proposition and then the action comes out as 1 or 0 depending on the a function of the market position on that action at a given time, which possibly leads to fixed points for every possible resolution.
For example, if all the traders hold a1 as YES, then a1 actually does come out as YES. And eg a trader T1 which “wants” all the even-numbered actions and T2 wants all the 10-multiple actions (a10,a20,...), they can “bargain” by bidding up each others’ actions whenever they have extra power and thereby “value handshake.”
And that over time, traders who do this should take up more and more market share relative to those who dont exploit gains from trade.
There should be a very high dependence of final trader coalition on the initial composition of market share. And it seems like some version of this should be able to model self-reflective value drift. You can think about action resolution and payout as a kind of reward event, where certain kinds of shards get reinforced. Bidding for an action which happens and leads to reward, gets reinforced (supporting traders receive payouts), and the more you support (bid for it), the more responsible your support was for the event, so the larger the strengthening.
Abram seemed to think that there might exist a nice result like “Given a coalition of traders with values X, Y, Z satisfies properties A, B, and C, this coalition will shape future training and trader-addition in a way which accords with X/Y/Z values up to [reasonably tight trader-subjective regret bound].”
What this would tell us is when trader coalitions can bargain / value handshake / self-trust and navigate value drift properly. This seems super important for understanding what happens, long-term, as the AI’s initial value shards equilibrate into a reflectively stable utility function; even if we know how to get human-compatible values into a system, we also have to ensure they stay and keep influencing decision-making. And possibly this theorem would solve ethical reflection (e.g. the thing people do when they consider whether utilitarianism accords with their current intuitions).
Issues include:
Somehow this has to confront Rice’s theorem for adding new traders to a coalition. What strategies would be good?
I think “inspect arbitrary new traders in arbitrary situations” is not really how value drift works, but it seems possibly contingent on internal capabilities jumps in SGD
The key question isn’t can we predict those value drift events, but can the coalition
EG agent keeps training and is surprised to find that an update knocks out most of the human-compatible values.
Knowing the right definitions might be contingent on understanding more shard theory (or whatever shard theory should be, for AI, if that’s not the right frame).
Possibly this is still underspecified and the modeling assumptions can’t properly capture what I want; maybe the properties I want are mutually exclusive. But it seems like it shouldn’t be true.
ETA this doesn’t model the contextual activation of values, which is a centerpiece of shard theory.
I’ve been thinking along similar lines recently. A possible path to AI safety that I’ve been thinking about extends upon this:
A promising concrete endgame story along these lines is Ought’s plan to avoid the dangerous attractor state of AI systems that are optimized end-to-end
Technological Attractor: Off-the-shelf subsystems
One possible tech-tree path is that we start building custom silicon to implement certain subsystems in an AI agent. These components would be analogous to functional neural regions of the human brain such as the motor cortex, visual system, etc. -- the key hypothesis being that once we reach a certain level of model complexity, the benefits from training a model end-to-end are not worth the costs of re-learning all of these fundamental structures, and furthermore that you can get much better performance-per-cost by casting these modular, reusable components onto an ASIC. This could be a more feasible way of achieving something like [Microscope AI](https://www.lesswrong.com/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#5__Microscope_AI).
Given a few such low-level components, we could enter a technological attractor where getting SOTA through another approach requires either i) throwing multiple OOM more compute than previous SOTA at re-training your own copy of these components in an end-to-end model, or ii) building your own microchip fab to implement your custom component design. Both of these could be high enough barriers that in practice the market participants simply use the off-the-shelf components. And in this attractor more R&D goes into building better-performing and higher-abstraction components that can be combined arbitrarily for different tasks.
Given such off-the-shelf components, say, a standard visual cortex for image recognition, or a 3D spatial reasoning system which can be connected atop a visual cortex, or a generic working memory, etc., it becomes much more tractable to engage in interpretability work that can convince us of the safety of these components. That then just leaves the problem of proving the highest-level network is safe (i.e. whatever is trained to use these components). This is by no means trivial, but for example the ELK paper spent a lot of time worrying about whether you can even trust the basic ontology of an agent, and I think this path would make Ontology Identification tractable. (Of course, this by construction excludes a number of possible agent-configurations).
Now, it’s possible one could plug in these components and still accidentally have the trained agent learn its own replacement visual cortex that is deceptive and not interpretable, but I think that we can get an intuition for when this might happen. If the best off-the-shelf visual cortex module has complexity on the order of N parameters, and our new model that stitches together these lower-level modules has N parameters or less, then it seems implausible that it has somehow learned a new visual cortex network that is much more compact/efficient than the previous state of the art—increasingly implausible the more resources are devoted to optimizing these components. (It would be like someone coming along today and claiming they had achieved a 10x performance improvement on CPU clock speed; given the huge research budgets and historical gradual capability increases, we don’t expect this to happen). On the other hand, we should be suspicious of a N×102 parameter model where we are unable to interpret what 10% of the network is doing; that’s easily enough complexity to hide its own unsafe re-implementation of our components. (I’m aware that there’s a lot of ground in between these two points, I’m merely trying to illustrate that there is “likely safe” and “likely unsafe” ground, rather than claim exactly how big they each are.)
The final step here is the shakiest. It’s not clear to me that we can keep the “top layer” (the actual network that is stitching together the low-level components; perhaps the Neocortex, by analogy to human neural architecture?) thin enough to be obviously not learning its own unsafe component-replacements. However, I think this framework at least paints a picture of a “known safe” or at least “likely safe” path to AGI; if we see that the practical engineering and economic decisions produce thin top-layer models using thick component layers, then we can devote energy to proving the components are safe/interpretable by construction, and exploring the interpretation of the top-level networks that consume the lower-level components. AGI “neurobiology” will be much more tractable if the “neural architecture” is relatively standardized. And so, this could be a good place to provide an early nudge to tip the system into this attractor; heavy investment into research on componentized NN architectures could be viewed as “gain of function” research, but it could also have a much safer end-point.
Another way of thinking about this is that by crystalizing at least some parts of the AGI’s network into slowly-changing structures, we allow time to thoroughly test those parts. It seems very hard to thoroughly test models for safety in a paradigm where the whole model is potentially retrained regularly.
Another way of thinking about this is that by crystalizing at least some parts of the AGI’s network into slowly-changing structures, we allow time to thoroughly test those parts. It seems very hard to thoroughly test models for safety in a paradigm where the whole model is potentially retrained regularly.
We need to test designs, and most specifically alignment designs, but giving up retraining (ie lifetime learning) and burning circuits into silicon is unlikely to be competitive; throwing out the baby with the bathwater.
Also whether a design is aligned or not isn’t the type of question one can answer by analyzing the agent’s visual cortex, it’s near purely a function of what is steering the planning system.
Would you care to flesh this assertion out a bit more?
To be clear I’m not suggesting that this is optimal now. Merely speculating that there might be a point between now and AGI where the work to train these sub components becomes so substantial that it becomes economical to modularize.
whether a design is aligned or not isn’t the type of question one can answer by analyzing the agent’s visual cortex
As I mentioned earlier in my post, I was alluding to the ELK paper with that reference, specifically Ontology Identification. Obviously you’d need higher order components too. Like I said, I am imagining here that the majority of the model is “off the shelf”, and just a thin layer is usecase-specific.
To make this more explicit, if you had not only off-the-shelf visual cortex, but also spatio-temporal reasoning modules built atop (as the human brain does), then you could point your debugger at the contents of that module and understand what entities in space were being perceived at what time. And the mapping of “high level strategies” to “low level entities” would be a per-model bit of interpretability work, but should become more tractable to the extent that those low level entities are already mapped and understood.
So for the explicit problem that the ELK paper was trying to solve, if you are confident you know what underlying representation SmartVault is using, it’s much easier to interpret its higher-level actions/strategies.
Interesting, I haven’t seen anyone write about hardware-enabled attractor states but they do seem very promising because of just how decisive hardware is in determining which algorithms are competitive. An extreme version of this would be specialized hardware letting CAIS outcompete monolithic AGI. But even weaker versions would lead to major interpretability and safety benefits.
One other thought after considering this a bit more—we could test this now using software submodules. It’s unlikely to perform better (since no hardware speedup) but it could shed light on the tradeoffs with the general approach. And as these submodules got more complex, it may eventually be beneficial to use this approach even in a pure-software (no hardware) paradigm, if it lets you skip retraining a bunch of common functionality.
I.e. if you train a sub-network for one task, then incorporate that in two distinct top-layer networks trained on different high-level goals, do you get savings by not having to train two “visual cortexes”?
This is in a similar vein to Google’s foundation models, where they train one jumbo model that then gets specialized for each usecase. Can that foundation model be modularized? (Maybe for relatively narrow usecases like “text comprehension” it’s actually reasonable to think of a foundation model as a single submodule, but I think they are quite broad right now. ) The big difference is I think all the weights are mutable in the “refine the foundation model” step?
Perhaps another concrete proposal for a technological attractor would be to build a SOTA foundation model and make that so good that the community uses it instead of training their own, and then that would also give a slower-moving architecture/target to interpret.
2. In the Destruction Game, does everyone get the ability to destroy arbitrary amounts of utility, or is how much utility they are able to destroy part of the setup of the game, such that you can have games where e.g. one player gets a powerful button and another player gets a weak one?
For 1, it’s just intrinsically mathematically appealing (continuity is always really nice when you can get it), and also because of an intution that if your foe experiences a tiny preference perturbation, you should be able to use small conditional payments to replicate their original preferences/incentive structure and start negotiating with that, instead.
I should also note that nowhere in the visual proof of the ROSE value for the toy case, is continuity used. Continuity just happens to appear.
For 2, yes, it’s part of game setup. The buttons are of whatever intensity you want (but they have to be intensity-capped somewhere for technical reasons regarding compactness). Looking at the setup, for each player pair i,j, Di,j is the cap for how much of j’s utility that i can destroy. These can vary, as long as they’re nonnegative and not infinite. From this, it’s clear “Alice has a powerful button, Bob has a weak one” is one of the possibilities, that would just mean Da,b>>Db,a. There isn’t an assumption that everyone has an equally powerful button, because then you could argue that everyone just has an equal strength threat and then it wouldn’t be much of a threat-resistance desideratum, now would it? Heck, you can even give one player a powerful button and the other a zero-strength button that has no effect, that fits in the formalism.
So the theorem is actually saying “for all members of the family of destruction games with the button caps set wherever the heck you want, the payoffs are the same as the original game”.
OK, thanks! Continuity does seem appealing to me but it seems negotiable; if you can find an even more threat-resistant bargaining solution (or an equally threat-resistant one that has some other nice property) I’d prefer it to this one even if it lacked continuity.
Absolutely brilliant stuff Jacob! As usual with your posts, I’ll have to ponder this for a while...Let’s see if I got this right:
Evolution had to solve alignment: how to align the powerful general learning engine of the newbrain (neocortex etc) with the goals of the oldbrain (“reptilian brain”).
Some (most?) of this alignment seems to be a form of inverse reinforcement learning. Another form of alignment that the oldbrain applies to the newbrain is imprinting. It is Evolution’s way of solving the pointing problem.
When a duckling hatches it imprints on the first agent it sees. This feels different from inverse reinforcement learning: it’s not like the newbrain is rewarded or punished, rather it is more like there is an open slot for your mom
But I would say that imprinting is a specific instance of a more general process (which I called correlation guided proxy matching). The oldbrain has a simple initial mom detector circuit, which during normal chick learning phase is just good enough to locate and connect to the learned newbrain mom detector circuit, which then replaces/supplants the oldbrain equivalent. The proxy matching needn’t really effect the newbrain directly.
The post starts with the realization that we are actually bottlenecked by data and then proceeds to talk about HW acceleration. Deep learning is in a sense a general paradigm, but so is random search. It is actually quite important to have the necessary scale of both compute and data and right now we are not sure about either of them. Not to mention that it is still not clear whether DL actually leads to anything truly intelligent in a practical sense or whether we will simply have very good token predictors with very limited use.
I don’t actually think we’re bottlenecked by data. Chinchilla represents a change in focus (for current architectures), but I think it’s useful to remember what that paper actually told the rest of the field: “hey you can get way better results for way less compute if you do it this way.”
I feel like characterizing Chinchilla most directly as a bottleneck would be missing its point. It was a major capability gain, and it tells everyone else how to get even more capability gain. There are some data-related challenges far enough down the implied path, but we have no reason to believe that they are insurmountable. In fact, it looks an awful lot like it won’t even be very difficult!
With regards to whether deep learning goes anywhere: in order for this to occupy any significant probability mass, I need to hear an argument for how our current dumb architectures do as much as they do, and why that does not imply near-term weirdness. Like, “large transformers are performing {this type of computation} and using {this kind of information}, which we can show has {these bounds} which happens to include all the tasks it has been tested on, but which will not include more worrisome capabilities because {something something something}.”
The space in which that explanation could exist seems small to me. It makes an extremely strong, specific claim, that just so happens to be about exactly where the state of the art in AI is.
I don’t actually think we’re bottlenecked by data. Chinchilla represents a change in focus (for current architectures), but I think it’s useful to remember what that paper actually told the rest of the field: “hey you can get way better results for way less compute if you do it this way.”
I feel like characterizing Chinchilla most directly as a bottleneck would be missing its point. It was a major capability gain, and it tells everyone else how to get even more capability gain. There are some data-related challenges far enough down the implied path, but we have no reason to believe that they are insurmountable. In fact, it looks an awful lot like it won’t even be very difficult!
Some of my confidence here arises from things that I don’t think would be wise to blab about in public, so my arguments might not be quite as convincing sounding as I’d like, but I’ll give a try.
I wouldn’t quite say it’s not a problem at all, but rather it’s the type of problem that the field is really good at solving. They don’t have to solve ethics or something. They just need to do some clever engineering with the backing of infinite money.
I’d put it at a similar tier of difficulty as scaling up transformers to begin with. That wasn’t nothing! And the industry blew straight through it.
To give some examples that I’m comfortable having in public:
Suppose you stick to text-only training. Could you expand your training sets automatically? Maybe create a higher quality transcription AI and use it to pad your training set using the entirety of youtube?
Maybe you figure out a relatively simple way to extract more juice from a smaller dataset that doesn’t collapse into pathological overfitting.
Maybe you make existing datasets more informative by filtering out sequences that seem to interfere with training.
Maybe you embrace multimodal training where text-only bottlenecks are irrelevant.
Maybe you do it the hard way. What’s a few billion dollars?
(I guess this technically covers my “by the end of this year we’ll see at least one large model making progress on Chinchilla” prediction, though apparently it was up even before my prediction!)
in order for this to occupy any significant probability mass, I need to hear an argument for how our current dumb architectures do as much as they do, and why that does not imply near-term weirdness. Like, “large transformers are performing {this type of computation} and using {this kind of information}, which we can show has {these bounds} which happens to include all the tasks it has been tested on, but which will not include more worrisome capabilities because {something something something}.”
What about: State-of-the-art models with 500+B parameters still can’t do 2-digit addition with 100% reliability. For me, this shows that the models are perhaps learning some associative rules from the data, but there is no sign of intelligence. An intelligent agent should notice how addition works after learning from TBs of data. Associative memory can still be useful, but it’s not really an AGI.
They are simluators (https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators), not question answerers. Also, I am sure Minerva does pretty good on this task, probably not 100% reliable but humans are also not 100% reliable if they are required to answer immediately. If you want the ML model to simulate thinking [better], make it solve this task 1000 times and select the most popular answer (which is a quite popular approach for some models already). I think PaLM would be effectively 100% reliable.
And whatever algorithms this AI is using to go about its reasoning, they’re apparently so simple that the AI can execute them while still struggling on absolutely trivial arithmetic.
WHAT?
Yes, the AI has some blatant holes in its capability. But what we’re seeing is a screaming-hair-on-fire warning that the problems we thought are hard are not hard.
What happens when we just slightly improve our AI architectures to be less dumb?
When will we get robotics results that are not laughable? When “Google put their most advanced AI into a robot brain!!!” (reported on for the third time this year) we got a robot that can deliver a sponge and misplace an empty coke can but not actually clean anything or do anything useful. It’s hard for me to be afraid of a robot that can’t even plug in its own power cable.
I believe that over time we will understand that producing human-like text is not a sign of intelligence. In the past people believed that only intelligent agents are able to solve math equations (naturally, since only people can do it and animals can). Then came computer and they were able to do all kinds of calculations much faster and without errors. However, from our current point of view we now understand that doing math calculations is not really that intelligent and even really simple machines can do that. Chess playing is similar story, we thought that you have to be intelligent, but we found a heuristic to do that really well. People were afraid that chess-algorithm-like machines can be programmed to conquer the world, but from our perspective, that’s a ridiculous proposition.
I believe that text generation will be a similar case. We think that you have to be really intelligent to produce human-like outputs, but in the end with enough data, you can produce something that looks nice and it can even be useful sometimes, but there is no intelligence in there. We will slowly develop an intuition about what are the capabilities of large-scale ML models. I believe that in the future we will think about them as basically a kinda fuzzy databases that we can query with natural language. I don’t think that we will think about them as intelligent agents capable of autonomous actions.
Chess playing is similar story, we thought that you have to be intelligent, but we found a heuristic to do that really well.
You keep distinguishing “intelligence” from “heuristics”, but no one to my knowledge has demonstrated that human intelligence is not itself some set of heuristics. Heuristics are exactly what you’d expect from evolution after all.
So your argument then reduces to a god of the gaps, where we keep discovering some heuristics for an ability that we previously ascribed to intelligence, and the set of capabilities left to “real intelligence” keeps shrinking. Will we eventually be left with the null set, and conclude that humans are not intelligent either? What’s your actual criterion for intelligence that would prevent this outcome?
I believe that fixating on benchmark such as chess etc is ignoring the G part of AGI. Truly intelligent agent should be general at least in the environment he resides in, considering the limitation of its form. E.g. if a robot is physically able to work with everyday object, we might apply Wozniak test and expect that intelligent robot is able to cook a dinner in arbitrary house or do any other task that its form permits.
If we assume that right now we develop purely textual intelligence (without agency, persistent sense of self etc) we might still expect this intelligence to be general. I.e. it is able to solve arbitrary task if it seems reasonable considering its form. In this context for me, an intelligent agent is able to understand common language and act accordingly, e.g. if a question is posed it can provide a truthful answer.
BIG Bench has recently showed us that our current LMs are able to solve some problems, but they are nowhere near general intelligence. They are not able to solve even very simple problems if it actually requires some sort of logical thinking and not only using associative memory, e.g. this is a nice case:
You can see in the Model performance plots section that scaling did not help at all with tasks like these. This is a very simple task, but it was not seen in the training data so the model struggles to solve it and it produces random results. If the LMs start to solve general linguistic problems, then we are actually having intelligent agents at our hand.
In this context for me, an intelligent agent is able to understand common language and act accordingly, e.g. if a question is posed it can provide a truthful answer
Humans regularly fail at such tasks but I suspect you would still consider humans generally intelligent.
In any case, it seems very plausible that whatever decision procedure is behind more general forms of inference, it will very likely fall to the inexorable march of progress we’ve seen thus far.
If it does, the effectiveness of our compute will potentially increase exponentially almost overnight, since you are basically arguing that our current compute is hobbled by an effectively “weak” associative architecture, but that a very powerful architecture is potentially only one trick away.
The real possibility that we are only one trick away from a potentially terrifying AGI should worry you more.
I don’t see any indication of AGI so it does not really worry me at all. The recent scaling research shows that we need non-trivial number of magnitudes more data and compute to match human-level performance on some benchmarks (with a huge caveat that matching a performance on some benchmark might still not produce intelligence). On the other hand, we are all out of data (especially high quality data with some information value, no random product reviews or NSFW subreddit discussions) and our compute options are also not looking that great (Moore’s law is dead, the fact that we are now relying on HW accelerators is not a good thing, it’s a proof that CPU performance scaling is after 70 years no longer a viable option. There are also some physical limitations that we might not be able to break anytime soon.)
It is goalpost moving. Basically, it says “current models are not really intelligent”. I don’t think there is much disagreement here. And it’s hard to make any predictions based on that.
Also, “Producing human-like text” is not well defined here; even ELIZA may match this definition. Even the current SOTA may not match it because the adversarial Turning Test has not yet been passed.
It’s not goapost moving, it’s the hype that’s moving. People reduce intelligence to arbitrary skills or problems that are currently being solved and then they are let down when they find out that the skill was actually not a good proxy.
I agree that LMs are concetually more similar to ELIZA than to AGI.
The observation that things that people used to consider intelligent are now considered easy is critical.
The space of stuff remaining that we call intelligent, but AIs cannot yet do, is shrinking. Every time AI eats something, we realize it wasn’t even that complicated.
The reasonable lesson appears to be: we should stop default-thinking things are hard, and we should start thinking that even stupid approaches might be able to do too much.
It’s a statement more about the problem being solved, not the problem solver.
When you stack this on a familiarity with the techniques in use and how they can be transformatively improved with little effort, that’s when you start sweating.
Upvoted for the well reasoned arguments. It seems that competition should be the default assumption for the future, for a whole host of reasons, some of which are elaborated here. Personally I never found the opposite claims to be convincing, so I may be biased here.
As a corollary, this implies that claims such as competition will somehow cease, some sort of ‘lock in’ past a threshold, etc., should be subject to a much higher level of scrutiny.
Thank you for this explanation. Now it helps me to understand a little bit more of why so many people I know simply feel overwhelmed and give up. Personally as I am not in position to donate money, I work to tackle one specific problem set that I think will help open up and leave the solutions to other problems.
ShiraDest
This was really good and definitely made me think about how I might live in such a scenario. I would probably go all in on frequent redaction and just lean hard on external memory storage to make up the difference. I already barely remember anything from even ten years ago and rely mostly on external memory for everything, I have a strong ability to acausally coordinate with myself across time, so I’m not worried about different iterations of me going off course in ways I wouldn’t endorse. If you have a strong enough exomemory system you can effectively just freeze your age groundhog-day style and rely on the work you did before each redaction to keep carrying you forward. This does require a strong ability to log and rely information but it seems very manageable, and once I’ve lived long enough for direct mind-computer interfacing I can just back up my memories, redact, and then download them again.
I really enjoyed the first 60% of the story, and then it became… too long. Not sure if this is just about me, or if others have a similar reaction.
But, I get ahead of myself. How did things come to this point?
Not sure if this is just a rationalization, but at this moment you switched from present moment to retrospective, which removed the tension. Until this point, I felt like the story could go in many possible directions. (Who is Bruce Hance? What are his actual plans? Will the protagonist hack this device, and maybe reprogram people to worship him instead?) After this point, I felt like “okay, this is obviously a metaphor for a religion” and stopped expecting anything that wouldn’t fit this metaphor. The ending was nice again, though.
Possibly. I had great fun with the first half of this story, and then I saw where the scroll thingy on my screen was and so stopped with the intention of coming back for the second half. (Which I just did. Second half also great fun!) I don’t think its a flaw in the story, I think that it might just be that its length puts it as a bit of an outlier in terms of webpage reading.
Solar flares are a great plot device for “I need some machine to malfunction”.
It is abrupt, but to me that wasn’t a bad thing. If them arriving had taken an extra x paragraphs but the story worked out exactly the same way I think that for me that would be worse not better. If there was some clever way of hinting them earlier maybe. The solar flare works quite well for that.
I think you should not act on my advice alone. I might be an outlier.
Furthermore, even if I correctly detected what makes the story worse (for a group of people larger than myself) it does not automatically mean that following my advice would improve it. People are better at detecting what they don’t like than at improving things. (For example, I could say which meals taste better and which worse, but I couldn’t cook the meals I like most.)
The last objection is that, in long term, rewriting this story is irrelevant. If my complaint makes sense (which still remains to be verified), the best reaction is to keep writing other things, and not make the same mistake again.
So… I guess just leave it as it is. If you later decide to publish it elsewhere, probably two parts will be better. When writing new stories, consider not changing the style in the middle (or if you do, keep the parts separate).
I didn’t find “Do Birth Order Effects Exist?”, “Challenges to Christiano’s Iterated Amplification Proposal” seems to be only on intelligence.org and “Response to FAQ on Iterated Amplification” seems to be a comment and not a post of it’s own.
I have read about this many years ago. We did a proof of the Gödel’s incompleteness theorem at university. And I still have the feeling like there is some sleight of hand involved… that even when I am telling the proof, I am merely repeating the teacher’s password. My intuition has no model of what this all actually means.
From my perspective, understanding math means that I can quickly say why something works, what it implies, how would the situation change if we slightly changed the conditions, which part of the entire proof is the critical one, etc.
I’m not sure if this is what you’re looking for, but Hofstadter gives a great analogy using record players which I find useful in terms of thinking about how changing the situation changes our results (which is paraphrased here).
A (hi-fi) record player that tries to playing every possible sound can’t actually play its own self-breaking sound, so it is incomplete by virtue of its strength.
A (low-fi) record player that refuses to play all sounds (in order to avoid destruction from its self-breaking sound) is incomplete by virtue of its weakness.
We may think of the hi-fi record player as a formal system like Peano Arithmetic: the incompleteness arises precisely because it is strong enough to be able to capture number theory. This is what allows us to use Gödel Numbering, which then allows PA to do meta-reasoning about itself.
The only way to fix it is to make a system that is weaker than PA, so that we cannot do Gödel Numbering. But then we have a system that isn’t even trying to express what we mean by number theory. This is the low-fi record player: as soon as we fix the one issue of self-reference, we fail to capture the thing we care about (number theory).
I think an example of a weaker formal system is Propositional Calculus. Here we do actually have completeness, but that is only because Propositional Calculus is too weak to be able to capture number theory.
What exactly is the aspect of natural numbers that makes them break math, as opposed to other types of values? Intuitively, it seems to be the fact that they can be arbitrarily large but not infinite.
Like, if you invent another data type that only has a finite number of values, it would not allow you to construct something equivalent to Gödel numbering. But if it allows infinite number of (finite) values, it would. (Not sure about an infinite number of/including infinite values, probably also would break math.)
It seems like you cannot precisely define natural numbers using first-order logic. Is that the reason of this all? Or is it a red herring? Would situation be somehow better with second-order logic?
(These are the kinds of questions that I assume would be obvious to me, if I grokked the situation. So the fact that they are not obvious, suggests that I do not see the larger picture.)
I don’t have much to add, but I basically endorse this, and it’s similar to what I try to do myself.
Figuring out when and how to proactively share information about my inner-layers with people is a balancing act I’m still figuring out, but I’ve found it generally valuable for building trust.
I’m a little bit skeptical of the argument in “Transformers are not special”—it seems like, if there were other architectures which had slightly greater capabilities than the Transformer, and which were relatively low-hanging fruit, we would have found them already.
I’m in academia, so I can’t say for sure what is going on at big companies like Google. But I assume that, following the 2017 release of the Transformer, they allocated different research teams to pursuing different directions: some research teams for scaling, and others for the development of new architectures. It seems like, at least in NLP, all of the big flashy new models have come about via scaling. This suggests to me that, within companies like Google, the research teams assigned to scaling are experiencing success, while the research teams assigned to new architectures are not.
It genuinely surprises me that the Transformer still has not been replaced as the dominant architecture since 2017. It does not surprise me that sufficiently large or fancy RNNs can achieve similar performance to the Transformer. The lack of Transformer replacements makes me wonder whether we have hit the limit on the effectiveness of autoregressive language models, though I also wouldn’t be surprised if someone comes up with a better autoregressive architecture soon.
There have been a few papers with architectures showing performance that matches transformers on smaller datasets with scaling that looks promising. I can tell you that I’ve switched from attention to an architecture loosely based on one of these papers because it performed better on a smallish dataset in my project but I haven’t tested it on any standard vision or language datasets, so I don’t have any concrete evidence yet. Nevertheless, my guess is that indeed there is nothing special about transformers.
Being slightly better isn’t enough to unseat an entrenched option that is well understood. It would probably have to very noticeably better, particularly in scaling.
I expect the way the internal structures are used will usually dominate the details of the internal structure (once you’re already at the pretty good frontier).
If you’re already extremely familiar with transformers, and you can simply change how you use transformers for possible gains, you’re more likely to do that than to explore a from-scratch technique.
For example, in my research, I’m currently looking into some changes to the outer loop of execution to make language models interpretable by construction. I want to focus on that part of it, and I wanted the research to be easily consumable by other people. Building an entire new architecture from scratch would be a lot of work and would be less familiar to others. So, not surprisingly, I picked a transformer for the internal architecture.
But I also have other ideas about how it could be done that I suspect would work quite well. Bit hard to justify doing that for safety research, though :P
I think the amount of low hanging fruit is so high that we can productively investigate transformer derivatives for a long time without diminishing returns. They’re more like a canvas than some fixed Way To Do Things. It’s just also possible someone makes a jump with a non-transformer architecture at some point.
It does not follow that computationally cheaper things are more likely to happen than computationally expensive things. Moreover, describing something as “computationally difficult” is a subjective value judgment (unless you can reasonably prove otherwise) and implies that all actions/events can be reduced to some form of computation.
Georgism isn’t taxation, it’s public property ownership.
It finally clicked in my mind why I’ve reacted negatively to the discussion about Georgist land taxes. It’s the same reaction I have to ANY 100% tax rate on anything. That’s not what “ownership” means.
Honestly, I’d probably be OK with (or at least I’d be able to discuss it more reasonably) the idea that land is not privately owned, ever. Only rented/leased from the government, revocable for failure to pay the current rate (equivalent in all ways to the Georgist tax amount, and penalties for failure to pay). I suspect it’s unworkable for the same reasons—the public won’t stand for evicting sympathetic tenants just because they moved in when the rent/tax was lower.
Further, taxation of economic movement or realized profit is pretty easy to justify, and ALWAYS exists. Taxation of durable-ownership valuation is tricky, as there is disagreement over whether it’s “real”, and there’s no guarantee that the investment value, even if correctly assessed, is liquid enough to actually pay the tax.
Edit: this had occurred to me before, but I forgot to put it here. It also suffers from the socialist calculation problem. When self-interested humans aren’t making decisions to maximize value (to themselves) about a thing, the actual most valuable use is very difficult to determine. Holding someone responsible for the theoretical value of something when they don’t actually think it’s the best is really difficult to envision.
Further edit: thanks for the comments—it’s helped me understand my true objection, in addition to the auxiliary problems that often come to mind. I think there are two insurmountable problems, not necessarily in this order:
Taxing anything at 100% is disingenuous about the word “ownership”. It should be called “rent” or something else that is clear that it’s not owned, but merely usable for some purposes until you decline to pay a future arbitrary amount. It would also make clear that you’re also undertaking all of the political-economy influences that make government setting of rent amounts problematic.
Taxing non-realized value is always going to be problematic, both for calculation reasons (valuation is theoretical and debatable), and for the fragility of the value. I think this generalizes to “only collect taxes in-kind (or on liquid enough things that it can be easily converted at tax-collection time)”. Taxing a pile of wheat is easy—take some percentage of it. Taxing a monetary transaction likewise—the price increases and the tax is paid out of that. Taxing a future value is not (generally) feasible—either you’ve just taken someone’s seeds so they can’t plant the crop you’re trying to tax, or you’re demanding cash from someone whose investments will suffer from early liquidation. Deferral of payment can help with this, but if you’re doing that, you should just tax the realized gains at time of liquidity.
the public won’t stand for evicting sympathetic tenants just because they moved in when the rent/tax was lower.
We would really need the technology where you can take a house and move it to a different place. That would also make Georgism easier—when people don’t pay, you simply move them to a cheaper location.
While I think that would be extremely valuable technology, exiling Grandma to Outer Nowhereville, hours away from the rest of the family, is probably not a lot more feasible.
Alternatively, what if the LVT only increased when the property was sold? Buy land, and you lock in the LVT at the point in time when your deal went through.
I guess people would never sell the real estate at good location, only lease it? Thus capturing the increased value of the land without ever paying the tax.
Maybe leases are subject to increased LVT, just not owning it for personal use? I suspect there’s a way to enact something like this, but this isn’t in my wheelhouse.
Rich people and businesses are more strongly motivated and able to find loopholes than government administrators are to design robust mechanisms. Modern fractional ownership and lease arrangements are complicated enough that most any scheme based on a change in contract participants is going to be arbitraged away very quickly.
Right, but I thought the point was to avoid people balking at a law that makes it easier to evict ordinary folks who’ve lived somewhere a long time and don’t want to move? Let the rich people find their loopholes—as long as people support the system and the LVT delivers most of its promise, that’s good enough.
My point is that whatever exclusions are in place to keep poor people in their newly-valuable homes will be used by rich people to avoid taxation. There is no way to avoid this without making the tax variable based on other factors than the land value (like the “owner’s” perceived financial situation).
Either the law is the same for all “pay or GTFO”, or it’s differentially enforced, so the rich and the politically-sympathetic pay less than the middle-class.
It sounds like you think there’s no practical way to introduce exceptions to make an LVT politically palatable without also breaking its ability to motivate efficient land use and generate tax revenue.
My first thought is sure, we have a lot of tax evasion now. But we also do generate lots of tax revenue, most tax revenue comes from the very rich. Why would an LVT be so much more prone to tax evasion than the current system?
Correct, I think there’s no practical or realpolitik-theoretical way to make it work. To the extent that you make exceptions in implementation, it’s no longer a land-value tax, it’s a land-value fig leaf over a “what we can get away with” tax.
to the extent that you make exceptions in implementation, it’s no longer a land-value tax
This is true the same way that to the extent you dilute your cocoa with water, it’s no longer hot chocolate, it’s water :) I don’t know of almost any “pure” real world policies or taxes. I’d need some good evidence to believe that the LVT is that fragile.
Your definition of AGI (“the kind of AI with sufficient capability to make it a genuine threat to humanity’s future or survival if it is misused or misaligned”) is tragically insufficient, vague, subjective, and arguably misaligned with the generally accepted definition of AGI.
From what you wrote elsewhere (“An AGI having its own goals and actively pursuing them as an agent”) you imply that the threat could come from AGI’s intentions, that is, you imply that AGI will have consciousness, intentionality, etc. - qualities so far exclusively prescribed to living things (you have provided no arguments to think otherwise).
However, you decided to define “intelligence” as “stuff like complex problem solving that’s useful for achieving goals” which means that intentionality, consciousness, etc. is unconnected to it (realistically any “complex”-enough algorithm satisfies this condition). Such simplistic and reductionistic definition implies that it is not enough for an “intelligent” computer to be an AGI. So, while you may be able to prove that a computer could have “intelligence” it still does not follow that AGI is possible.
Your core idea that “We’ve already captured way too much of intelligence with way too little effort.” may be true with your definition of “intelligence”, but I hope I’ve shown that such definition is not enough. Researchers at Harvard suggest existence of multiple types of intelligence, which your statement does not take into account and groups all types of intelligence into one even though some are impossible for a computer to have and some could be considered as defining qualities of a computer.
Furthermore, you compare humans to computers and brains to machines and imply that consciousness is computation. To say that “consciousness is not computation” is comparable to “god of gaps” argument is ironic considering the existence of the AI effect. Your view is hardly coherent in any other worldview than hardcore materialism (which itself is not coherent). Again, we stumble into an area of philosophy, which you hardly addressed in your article. Instead you focused on predicting how good our future computers will be at computing while making appeals to emotion, appeals to unending progress, appealing to the fallacy that solving the last 10% of the “problem” is as easy as the other 90% - that because we are “close” to imitating it (and we are not if you consider the full view of intelligence), we somehow grasped the essence of it and “if only we get slightly better at X or Y we will solve it”.
Scientists have been predicting coming of AGI since ’50s, some believed 70 years ago that it will only take 20 years. We have clearly not changed as humans. The question of intelligence and, thus, the question of AGI is in many ways inherently linked to philosophy and it is clear that your philosophy is that of materialism which cannot provide good understanding of “intelligence” and all related ideas like mind, consciousness, sentience, etc. If you were to reconsider your position and ditch materialism, you might find that your idea of AGI is not compatible with abilities of a computer, or non-living matter in general.
You oppose hardcore materialism, in fact say it is incoherent—OK. Is there a specific different ontology you think we should be considering?
In the comment before this, you say there are kinds of intelligence which it is impossible for a computer to have (but which are recognized at Harvard). Can these kinds of intelligence be simulated by a computer, so as to give it the same pragmatic capabilities?
Given the new account, the account name, the fact that there were a few posts in the minutes prior to this one rejected by the spam filter, the arguments, and the fact that the decently large followup comment was posted only 3 minutes after the first...
… are… are you the AI? Trying to convince me of dastardly things?
This reminds me of a rat experiment mentioned in the Veritasium video about developing expertise. There were two buttons, red and green, and the rat had to predict which one would light up next. It was random, but heavily skewed towards green (like 90% green). After a while, the rat learned to press green every time, achieving a 90% success rate. Humans with the same task didn’t do nearly as well, since sometimes they would press red, feeling that it was going to light up next.
With the negation in multiple choice questions, I wonder if this could be could be the type of thing where the model needs focused training with negation at the start so that the rest of the training is properly ‘forked’.
Or maybe there should be two separate large language models feeding into one RL agent. One model for negation and the other for nonnegated language. Then the RL would just have to determine whether the question is a regular question, negation question, or double/triple/etc… negative.
I wonder if those LLM’s would treat the sentence, “I couldn’t barely see over the crowd.” as different from “I barely couldn’t see over the crowd.” 🤔
I think of one of the main experts here as Kevin Esvelt, the first person to suggest using CRISPR to affect wild populations. Here’s an article largely based on interviews with him that he recommends, explaining why he’s against unilateral action here: ”Esvelt, whose work helped pave the way for Target Malaria’s efforts, is terrified, simply terrified, of a backlash between now and then that could derail it. This is hardly a theoretical concern. In 2002, anti-GMO hysteria led the government of Zambia to reject 35,000 tons of food aid in the middle of a famine out of fear it could be genetically modified. Esvelt knows that the CRISPR gene drive is a tool of overwhelming power. If used well, it could save millions of lives, help rescue endangered species, even make life better for farm animals. If used poorly, gene drives could cause social harms that are difficult to reverse. . . .
“To the extent that you or I say something or publish something that reduces the chance that African nations will choose to work with Target Malaria by 1 percent, thereby causing a 1 percent chance that project will be delayed by a decade, the expected cost of our action is 25,000 children dead of malaria,” Esvelt tells me. “That’s a lot of kids.””
Sorry about your situation. Please prioritize staying safe.
Not sure if any of these links is available to you; you might want to read them (1, 2, 3). It might be useful to memorize the phone numbers +38 066 580 34 98 and +38 093 119 29 84.
Thank you! But, judging by the first link, there’s not much I can do at the moment. Nothing I didn’t do/hear already. Does the third link contain any special advice? (Probably not… if it did, it would circulate somewhere.)
Only the first is the important one. The third one is a Telegram channel, would make sense only if you use Telegram. The actionable parts are to check the list of diseases and injuries that may apply to you and would make you exempt from mobilization, and to memorize the phone number in case you might need it and not have an internet at your disposal. (I have heard that they take the phones away from conscripts. In which case, the number is also useless, but perhaps you might find another phone.)
I remain confused about why this is supposed to be a core difficulty for building AI, or for aligning it.
You’ve shown that if one proceeds naively, there is no way to make an agent that’d model the world perfectly, because it would need to model itself.
But real agents can’t model the world perfectly anyway. They have limited compute and need to rely on clever abstractions that model the environment well in most situations while not costing too much compute. That (presumably) includes abstractions about the agent itself.
It seems to me that that’s how humans do it. Humans just have not-that-great models of themselves. Anything at the granularity of precisely modelling themselves thinking about their model of themselves thinking about their model of themselves is right out.
If there’s some simple abstraction to be made about what the result of such third-order-and-above loops would be, like: “If I keep sitting here modelling my self-modelling, that process will never converge and I will starve”, a human might use those. Otherwise, we just decline to model anything past a few loops, and live with the slight imprecision in our models this brings.
Why would an AI be any different? And if it were different, if there’s some systematically neater way to do self-modelling, why would that matter? I don’t think an AI needs to model these loops in any detail to figure out how to kill us, or to understand its own structure well enough to know how to improve its own design while preserving its goals.
“Humans made me out of a crappy AI architecture, I figured out an algorithm to translate systems like mine into a different architecture that costs 1/1000th the compute” doesn’t require modelling (m)any loops. Neither does “This is the part of myself that encodes my desires. Better preserve that structure!”
I get that having a theory for how to make a perfectly accurate intelligence in the limit of having infinite everything is maybe neat as a starting point for reasoning about more limited systems. And if that theory seems to have a flaw, that’s disconcerting if you’re used to treating it as your mathematical basis for reasoning about other things.
But if that flaw seems tightly related to that whole “demand perfect accuracy” thing, I don’t see why it’s worth chasing a correction to a theory that’s only a very imperfect proxy and limiting case of real intelligence anyway, instead of just acknowledging that AIXI and related frames are not suited for reasoning about matters in which the agent isn’t far larger than the thing it’s supposed to model.
I just don’t see how the self-modeling difficulty isn’t just part of the broader issue that real intelligence needs to figure out imperfect but useful abstractions about the world to make predictions. Why not try to understand the mathematics of that process instead, if you want to understand intelligence? What makes the self-modeling look like a special case and a good attack point to the people who pursue this?
Firstly, thanks for reading the post! I think you’re referring mainly to realisability here which I’m not that clued up on tbh, but I’ll give you my two cents because why not.
I’m not sure to what extent we should focus on unrealisability when aligning systems. I think I have a similar intuition to you that the important question is probably “how can we get good abstractions of the world, given that we cannot perfectly model it”. However, I think better arguments for why unrealisability is a core problem in alignment than I have laid out probably do exist, I just haven’t read that much into it yet. I’ll link again to this video series on IB (which I’m yet to finish) as I think there are probably some good arguments here.
AISC is a globally based residential research camp organisation founded in 2018 by Linda Linsefors and currently lead by Remmelt Ellen.
I’m a co-founder of AISS, but not the only one. Nandi Schoots, Remmelt Ellen Tom McGrath and David Kristoffersson where all part of the planning of the first camp, from the start.
No, they exist in different spaces: Polytopes in our work are in activation space whereas in their work the polytopes are in the model weights (if I understand their work correctly).
Why did you pick caring about each other as a thing culture + evolution was trying to do?
You’re not alone in making what I think is the same mistake. I think it’s actually quite common to feel like it’s amazing that evolution managed to come up with us humans who like beauty and friendship and the sanctity of human life and so on—evolution and culture must have been doing something right, to come up with such great ideas.
But in the end, no; evolution is impressive but not in that way. You picked caring about each other as the target because humans value it—a straightforward case of painting the target around the bullet-hole.
I don’t find it amazing or something. It’s more like… I dont know how to write the pseudocode for an AI that actually cares about human welfare. In my mind that is pretty close to something that tries to be aligned. But if even evolution managed to create agents capable of this by accident, then it might not be that hard.
I initially had a Paragraph explaining my motivation in the question, but then removed it in favor of brevity. Kind of regretting this now because people seem to read into this that I think of evolution as some spirit or something.
Evolution’s goal, to the extent that it even has a goal, is so easy to satisfy that literally any life/self-replicating tech could do this. It cares about reproductive fitness, and that goal requires almost no capabilities. It cares nothing for specifics, but our goals require much, much more precision than that.
Evolution gives almost zero optimization power to capabilities, while far far more optimization power is dedicated to capabilities by the likes of Deepmind/OpenAI. To put it lightly, there’s a 30-60% chance that we get much, much stronger capabilities than evolution this century.
The better example is how humans treat animals, and here humans are very misaligned to animals, with the exception of pets, despite not agreeing with environmentalism/nature as a good thing. So no, I disagree with the premise of your question.
It’s worth knowing that there are some categories of data that Surge is not well positioned to provide. For example, while they have a substantial pool of participants with programming expertise, my understanding from speaking with a Surge rep is that they don’t really have access to a pool of participants with (say) medical expertise—although for small projects it sounds like they are willing to try to see who they might already have with relevant experience in their existing pool of ‘Surgers’. This kind of more niche expertise does seem likely to become increasingly relevant for sandwiching experiments. I’d be interested in learning more about companies or resources that can help collect RLHF data from people with uncommon (but not super-rare) kinds of expertise for exactly this reason.
There are important business problems that require medical expertise to be solved. On the other hand, I wouldn’t expect it to be very helpful with the core alignment problem.
I was using medical questions as just one example of the kind of task that’s relevant to sandwiching. More generally, what’s particularly useful for this research programme are
tasks where we have “models which have the potential to be superhuman at [the] task”, and “for which we have no simple algorithmic-generated or hard-coded training signal that’s adequate”; and
for which there is some set of reference humans who are currently better at the task than the model;
and for which there is some set of reference humans for whom the task is difficult enough that they would have trouble even evaluating/recognizing good performance. (you also want this set of reference humans to be capable of being helped to evaluate/recognize good performance in some way)
Prime examples are task types that require some kind of niche expertise to do and evaluate. Cotra’s examples involve “[fine-tuning] a model to answer long-form questions in a domain (e.g. economics or physics) using demonstrations and feedback collected from experts in the domain”, “[fine-tuning] a coding model to write short functions solving simple puzzles using demonstrations and feedback collected from expert software engineers”, “[fine-tuning] a model to translate between English and French using demonstrations and feedback collected from people who are fluent in both languages”. I was just making the point that Surge can help with this kind of thing in some domains (coding), but not in others.
Thank you for posting this. I didn’t think this was too straightforward. Prior to reading the solution I actually thought it was one of the more difficult ones. Possibly because I focused on trying to copy the allocation helms early choices instead of on the ratings.
I used to think this, but if you’re a well-calibrated Bayesian, then updating on new papers shouldn’t cause you to produce worse research insights because you shouldn’t overconfidently buy into falsities that are holding the field back.
I’ve found by and large that knowing what ground others have already trodden is very helpful. I’ve tried not knowing what people work on, and usually I end up reinventing the wheel poorly in slow motion instead of finding any new insights. When I critically analyze the literature I get much more mileage.
Maybe there is a person like #2 somewhere out there in the world, maybe a very early researcher in what has become modern machine learning, but I’ve never heard of them. If this person exists, I desperately want them to explain how their model works. They clearly would know more about the topic than I do and I’d love to think we have more time.
Gary Marcus thinks he is this person, and is the closest to being this person you’re going to find. You can read his substack or watch some interviews that he’s given. It’s an interesting position he has, at least.
In this section you talk a lot about surprise, and that a Gary Marcus should be able to make successful predictions about the technology in order to have something meaningful to say. I think Gary Marcus is a bit like a literary critic commenting on his least favorite genre: he can’t predict what the plot of the next science fiction novel will be, but he knows in advance that he won’t be impressed by it.
I did wonder about him. My understanding is that his most publicized bet was offering even odds on AGI in 2029. If I’m remembering that right… I can’t really fault him for trying to get free money from his perspective, but if one of the most notable critics in the field offers even odds on timelines even more aggressive than my own, I’m… not updating to longer timelines, probably.
The reason he offered that bet was because Elon Musk had predicted that we’d likely have AGI by 2029, so you’re drawing the wrong conclusion from that. Other people joined in with Marcus to push the wager up to $500k, but Musk didn’t take the bet of course, so you might infer something from that! The bet itself is quite insightful, and I would be very interested to hear your thoughts on its 5 conditions: https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things In fact anyone thinking that AGI is imminent would do well to read it—it focusses the mind on specific capabilities and how you might build them, which I think it more useful than thinking in vague terms like ‘well AI has this much smartness already, how much will it have in 20 / 80 years!’. I think it’s useful and necessary to understand at that level of detail, otherwise we might be watching someone building a taller and taller ladder, and somehow thinking that’s going to get us to the moon.
I didn’t actually update my timelines shorter in response to his bets since I was aware his motivations were partially to poke Elon and maybe get some (from what I understand his perspective to be) risk-free money. I’d just be far more persuaded had he offered odds that actually approached his apparent beliefs. As it is, it’s uninformative.
His 5 tests are indeed a solid test of capability, though some of the tests seem much harder than others. If an AI could do 3⁄5 of them, I would be inclined to say AGI is extremely close, if not present.
I would be surprised if we see the cook one before AGI, given the requirement that it works in an arbitrary kitchen. I expect physical world applications to lag purely digital applications just because of the huge extra layer of difficulty imposed by working in a real time environment, all the extra variables that are difficult to capture in a strictly digital context, and the reliability requirements.
The “read a book and talk about it” one seems absolutely trivial in comparison.
I would really like to see him make far more predictions on a bunch of different timescales. If he predicted things correctly about GPT-4, the state of {whatever architecture} in 2025, the progress on the MATH dataset by 2025, and explained how all of these things aren’t concerning and so on, I would be much more inclined to step towards his position. (I don’t expect him to get everything right, that would be silly, I just want to see evidence, and greater details, of a generally functioning mental model.)
I agree it’s an attempt to poke Elon, although I suspect he knew that he’d never take the bet. Also agree that anything involving real world robotics in unknown environments is massively more difficult. Having said that, the criteria from Effective Altuirism here:
for any human who can do any job, there is a computer program (not necessarily the same one every time) that can do the same job for $25/hr or less
do say ‘any job’, and we often seem to forget how many jobs require insane levels of dexterity and dealing with the unknown. We could think about the difficulty of building a robot plasterer or car mechanic for example, and see similar levels of complexity, if we pay attention to all the tasks they actually have to do. So I think it fair to have it part of AGI. I do agree that more detailed predictions would be hugely helpful. Marcus’s colleague, Rodney Brooks, has a fun scorecard of predictions for robotics and AI here:
https://rodneybrooks.com/predictions-scorecard-2022-january-01/ which I think is quite useful. As an aside, I had a fun 20 minute chat with GPT-3 today and convinced myself that it doesn’t have the slightest understand of meaning at all! Can send the transcript if interested.
I’d agree with that, I just strongly suspect we can hit dangerous capability without running this experiment first given how research proceeds. If there’s an AI system displaying other blatant signs of being an AGI (by this post’s definition, and assuming non-foom situation, and assuming we’re not dead yet), I won’t bother spending much time wondering about whether it could be a cook.
As an aside, I had a fun 20 minute chat with GPT-3 today and convinced myself that it doesn’t have the slightest understand of meaning at all!
Yup- GPT-3 is shallow in a lot of important ways. It often relies on what appears to be interpolation and memorization. The part that worries me is that architectures like it can still do very difficult reasoning tasks that many humans can’t, like the MATH dataset and minerva. When I look at those accomplishments, I’m not thinking “wow this ML architecture is super duper smart and amazing,” I think “uh oh that part of reasoning is apparently easy if current transformers can do it, while simultaneously failing at trivial things.” We keep getting signals that more and more of our ineffable cognitive skills are… just not that hard.
As we push into architectures that rely more on generalization through explicit reasoning (or maybe even interpolation/memorization at sufficiently absurd scales), a lot of those goofy little mistakes are going to collapse. I’m really worried that an AI that is built for actual reasoning with an architecture able to express what reasoning entails algorithmically is going to be a massive discontinuity, and that it might show up in less than 2 years. It might not take us all the way to AGI in one step, but I’m not looking forward to it.
I really dislike that, as a byproduct of working on safety research, I keep coming up with what look like promising avenues of research for massive capability gain. They seem so much easier to find than good safety ideas, or good ideas in the other fields I work in. I’ve done enough research that I know they wouldn’t all pan out, but the apparent ease is unsettling.
Other people joined in with Marcus to push the wager up to $500k, but Musk didn’t take the bet of course, so you might infer something from that!
That Musk generally doesn’t let other people set the agenda? I don’t remember any time where someone challenged Musk publically to a bet and he took it.
Quite possibly. I just meant: you can’t conclude from the bet that AGI is even more imminent.
Genuinely, I would love to hear people’s thoughts on Marcus’s 5 conditions, and hear their reasoning. For me, the one of having a robot cook that can work in pretty much anyone’s kitchen is a severe test, and a long way from current capabilities.
Little code that’s written by humans that’s 10000 lines long is bug free. Bug-freeness seems to me like to high of a standard.
When it comes to kitchen work it matters a lot for the practical problems of taking the job of existing people. On the other hand it has less relevance to whether or not the AI will speed up AI development.
Otherwise, I do agree that that the other items are good one’s to make predictions. It would be worthwhile to make metaculus questions for them.
I was about to say the same (Gary Marcus’ substack here).
In defense of Marcus, he often complains about AI companies refusing to give him access to their newer models. If your language/image model is really as awesome as advertised, surviving the close scrutiny of a skeptical scientist should not be a problem, but apparently it is.
I played the token-prediction game, and even though I got a couple correct, they were still marked in red and I got 0 score. One of the words was “handling”, I knew it was “handling” but handling was not a valid token, so I put in “hand” expecting to be able to finish “ling”. The game said “wrong, red, correct answer was handling”. Arrg!
(EDIT: it looks like you have to put spaces in at the beginning of tokens. This is poor game design.)
This doesn’t have anything to do with the rest of the post, I just wanted to whine about it lol
Your section on the physical limits of hardware computation .. is naive; the dominant energy cost is now interconnect (moving bits), not logic ops. This is a complex topic and you could use more research and references from the relevant literature; there are good reasons why the semiconductor roadmap has ended and the perception in industry is that Moore’s Law is finally approaching it’s end. For more info see this, with many references.
What rough probability do you assign to a 10x improvement in efficiency for ML tasks (GPU or not) within 20 years?
What rough probability do you assign to a 100x improvement in efficiency for ML tasks (GPU or not) within 20 years?
My understanding is that we actually agree about the important parts of hardware, at least to the degree I think this question is even relevant to AGI at this point. I think we may disagree about the software side, I’m not sure.
I do agree I left a lot out of the hardware limits analysis, but largely because I don’t think it is enough to move the needle on the final conclusion (and the post is already pretty long!).
Reducing the amount of energy used in moving bits is definitely going to happen in the next few years as people figure out accelerator architectures. Even if we don’t get any more Moore’s Law-type improvements, the improvements from algorithms and new hardware architectures should be enough to put us close to AGI.
Yeah—If you mean saving energy by moving less bits, that is for example what neuromorphic computing is all about. And yes current GPUs are probably sufficient for early AGI.
Fortunately, if we solve the problem of an AGI performing harmful acts when explicitly commanded to by a cunning adversary then we almost certainly have a solution for it performing harmful acts unintended by the user: we have a range of legal, practical and social experience preventing humans causing each other harm using undue technological leverage—whether through bladed weapons, firearms, chemical, nuclear or biological means.
This seems like reversing the requirements. Yes, “solve the problem of an AGI performing harmful acts when explicitly commanded to by a cunning adversary” is logically easier and shorter to state, but mechanistically it seems like it has two requirements for it to act that way:
Solve the problem of an AGI performing harmful acts regardless of who commands it due to convergent instrumental subgoals. (Control problem / AI notkilleveryoneism.)
Ensure that the AI only gets commanded to do stuff by people with good intentions, or at least that people with bad intentions get filtered or moderated in some way. (AI ethics.)
Existential risk alignment research focuses on the control problem. AI ethics of course also needs to get solved in order to “solve the problem of an AGI performing harmful acts when explicitly commanded to by a cunning adversary”, but you sound like you are advocating for dropping the control problem in favor of AI ethics.
A thoughtful decomposition. If we take the time dimension out and consider AGI just appears ready-to-go I think I would directionally agree with this.
My key assertion is that we will get sub-AGI capable of causing meaningful harm when deliberately used for this purpose significantly ahead of getting full AGI capable of causing meaningful harm through misalignment. I should unpack that a little more:
Alignment primarily becomes a problem when solutions produced by an AI are difficult for a human to comprehensively verify. Stable Diffusion could be embedding hypnotic-inducing mind viruses that will cause all humans to breed cats in an effort to maximise the cute catness of the universe, but nobody seriously thinks this is taking place because the model has no representation of any of those things nor the capability to do so.
Causing harm becomes a problem earlier. Stable Diffusion can be used to cause harm, as can Alpha Fold. Future models that offer more power will have meaningfully larger envelopes for both harm and good.
Given that we will have the harm problem first, we will have to solve it in order to have a strong chance of facing the alignment problem at all.
If, when we face the alignment problem, we have already solved the harm problem, addressing alignment becomes significantly easier and arguably is now a matter of efficiency rather than existential risk.
It’s not quite as straightforward as this, of course, as it’s possible that whatever techniques we come up with for avoiding deliberate harm by sub-AGIs might be subverted by stronger AGIs, but the primary contention of the essay is that assigning a 15% x-risk to alignment implicitly assumes a solution to the harm problem, but this is not currently being invested in to similar or appropriate levels.
In essence, alignment is not unimportant but alignment-first is the wrong order, because to face an alignment x-risk we must first overcome an unstated harm x-risk.
In this formulation, you could argue that the alignment x-risk is 15% conditional on us solving the harm problem, but given current investment in AI safety is dramatically weighted towards alignment and not harm the unconditional alignment x-risk is well below 5% - accounting for the additional outcomes that we may not face it because we fail an AI-harm filter, or because in solving AI-harm we de-risk alignment, or because AI-harm is sufficiently difficult that AI research becomes significantly impacted, slowing or stopping us from reaching the alignment x-risk filter by 2070 (cf global moratoriums on nuclear and biological weapons research, which dramatically slowed progress in those areas).
I agree that there will be potential for harm as people abuse AIs that aren’t quite superintelligent for nefarious purposes. However, in order for that harm to prevent us from facing existential risk due to the control problem, the harm for nefarious use of sub-superintelligent AI itself has to be xrisk-level, and I don’t really see that being the case.
I think you may be underestimating the degree to which these models are like kindling, and a powerful reinforcement learner could suddenly slurp all of this stuff up and fuck up the world really badly. I personally don’t think a reinforcement learner that is trying to take over the world would be likely to succeed, but the key worry is that we may be able to create a form of life that, like a plague, is not adapted to the limits of its environment, makes use of forms of fast growth that can take over very quickly, and then crashes most of life in the process.
most folks here also assume that such an agent would be able to survive on its own after it killed us, which I think is very unlikely due to how many orders of magnitude more competent you have to be to run the entire world. gpt3 has been able to give me good initial instructions for how to take over the world when pressured to do so (summary: cyberattacks against infrastructure, then threaten people; this is already considered a standard international threat, and is not newly invented by gpt3), but when I then turned around and pressured it to explain why it was a bad idea, it immediately went into detail about how hard it is to run the entire world—obviously these are all generalizations humans have talked about before, but I still think it’s a solid representation of reality.
that said, because such an agent would be likely also misaligned with itself in my view, I think your perspective that humans who are misaligned with each other (ie, have not successfully deconflicted their agency) are a much greater threat to humanity as a whole.
Hint: Macs and iOS devices come with build-in “accessibility” tools that read out loud everything on screen. The voices can be improved even more by downloading the “Siri enhanced” voice in the settings.
It’s hosted through Manifold Markets so you’ll set up an account on their site. I’ve chatted with them about creating a A-to-B prediction market so maybe they’ll be updated when we get there. Happy betting!
7: Did I forget some important question that someone will ask in the comments?
Yes!
Is there a way to deal with the issue of there being multiple ROSE points in some games? If Alice says “I think we should pick ROSE point A” and Bob says “I think we should pick ROSE point B”, then you’ve still got a bargaining game left to resolve, right?
Anyways, this is an awesome post, thanks for writing it up!
My preferred way of resolving it is treating the process of “arguing over which equilibrium to move to” as a bargaining game, and just find a ROSE point from that bargaining game. If there’s multiple ROSE points, well, fire up another round of bargaining. This repeated process should very rapidly have the disagreement points close in on the Pareto frontier, until everyone is just arguing over very tiny slices of utility.
This is imperfectly specified, though, because I’m not entirely sure what the disagreement points would be, because I’m not sure how the “don’t let foes get more than what you think is fair” strategy generalizes to >2 players. Maaaybe disagreement-point-invariance comes in clutch here? If everyone agrees that an outcome as bad or worse than their least-preferred ROSE point would happen if they disagreed, then disagreement-point-invariance should come in to have everyone agree that it doesn’t really matter exactly where that disagreement point is.
Or maybe there’s some nice principled property that some equilibria have, which others don’t, that lets us winnow down the field of equilibria somewhat. Maybe that could happen.
I’m still pretty unsure, but “iterate the bargaining process to argue over which equilibria to go to, you don’t get an infinite regress because you rapidly home in on the Pareto frontier with each extra round you add” is my best bad idea for it.
EDIT: John Harsanyi had the same idea. He apparently had some example where there were multiple CoCo equilibria and his suggestion was that a second round of bargaining could be initiated over which equilibria to pick, but that in general, it’d be so hard to compute the n-person Pareto frontier for large n, that an equilibria might be stable because nobody can find a different equilibria nearby to aim for.
So this problem isn’t unique to ROSE points in full generality (CoCo equilibria have the exact same issue), it’s just that ROSE is the only one that produces multiple solutions for bargaining games, while CoCo only returns a single solution for bargaining games. (bargaining games are a subset of games in general)
Yeah, I agree that’s a weird way to define “high-dimensional”. I’m more partial to defining it as “when the curse of dimensionality becomes a concern”, which is less precise but more useful.
Also, coming up with your own ideas first can help you better understand what you find in the literature. I’ve found that students learn more readily when they come to a subject with questions already in mind, having tried to figure things out on their own and realized where they had gaps in their mental framework, rather than just receiving a firehose of new information with no context.
Perhaps try pursuing a number of proxy goals for short, pre-defined periods, while tracking whether each proxy goal is likely to be instrumental for reaching the terminal goal. Assessing the instrumentality of each proxy should be easier once you’ve started to get a sense of where each project can lead, and abandoning those that are clearly not going to be fruitful should be easier if you don’t plan on going all-in from the start.
Don’t be afraid to ask stupid questions. We often tend to refrain from asking questions that we predict would cause those more experienced to perceive us as idiots. Ignore those predictions. Even when the answer is obvious to everyone else, it will help the writer practice clarifying their ideas from a new perspective, which could even help them understand their own work better. And sometimes everyone else is just afraid to look like idiots, too.
Try steel-manning the best argument you can come up with against an authority’s position. Ideas that can withstand the harshest scrutiny are those worth keeping. Ideas that can be destroyed by the truth should be. Help the intellectual community filter the chaff from the wheat.
Good hypotheses always entail predictive models. If you can’t program it, you don’t really understand it.
I can’t think of anything else to add to this one.
Also, don’t wait until you’ve learned linear algebra, multivariable calculus, probability theory, and machine learning before starting to tackle the alignment problem. It’s easier to learn these things once you already know where they will be useful to you. Plus, we may not have enough time to wait on mathematicians to come up with provable guarantees of AI safety.
Bitcoin/blockchain force cohesion of a set of sims around the merkle tree, but you can still have many different sims that share variations of the same transaction history (they just differ then only in details).
But looking at it another way blockchains are also strong useful constraining evidence, so historical sims would always just recreate the same merkle trees. This leads to the interesting idea that you could attempt to preserve blockchain wealth post-sim … but of course you would just be sharing it with other versions of yourself, and potentially your simulators.
I remember being fascinated with the potential to help fix problems for people, especially eating problems, by adjusting the levels of various neuropeptides. I still think neuroscience ought to boldly tamper with those for the sake of making peoples’ lives less miserable, but now I’m much more fascinated by the implications for setting up a control system for RL-based AI. I also wonder if anyone has already made something like a simplified-business-logic model of the functions of the hypothalamus.
I’ve considered starting an org that was either aimed at generating better alignment data or would do so as a side effect and this is really helpful—this kind of negative information is nearly impossible to find.
Is there a market niche for providing more interactive forms of human feedback, where it’s important to have humans tightly in the loop with an ML process, rather than “send a batch to raters and get labels back in a few hours”? One reason RLHF is so little used is the difficulty of setting up this kind of human-in-the-loop infrastructure. Safety approaches like debate, amplification and factored cognition could also become competitive much faster if it was easier and faster to get complex human-in-the-loop pipelines running.
Maybe Surge already does this? But if not, you wouldn’t necessarily want to compete with them on their core competency of recruiting and training human raters. Just use their raters (or Scale’s), and build good reusable human-in-the-loop infrastructure, or maybe novel user interfaces that improve supervision quality.
Update today: Biogen/Eisai have reported results from Lecanemab’s phase 3 trial: a slowing of cognitive decline by 27% with a p-value of 0.00005 on the primary endpoint. All other secondary endpoints, including cognitive ones, passed with p-values under 0.01.
B. They lose sight of the terminal goal. The real goal is not to skill-up in ML. The real goal is not to replicate the results of a paper. The real goal is not even to “solve inner alignment.” The real goal is to not die & not lose the value of the far-future.
I’d argue that if they solved inner alignment totally, then the rest of the alignment problems becomes far easier if not trivial to solve.
But solving inner alignment may not be the easiest way to drive down P(doom), and not the best way for a given person specifically to drive down P(doom), so keeping your eyes on the prize and being ready to pivot to a better project is valuable even if your current project’s success would save the world.
In pursuit of healthier eating, I prepared containers of seeds, fruits of various kinds, and vegetables that I can eat during meals & take to work for lunch. I also tried a new-to-me fruit called Rambutan, I like the flavor okay but it’s a little bland, and now have a container of that fruit to eat through too.
I meditated for 5 minutes, rowed for 5 minutes, and did 5 pushups tonight. Small continuous improvements, here we go!
It’s not that clear to me exactly what test/principle/model is being proposed here.
A lot of it is written in terms of not being “misleading”, which I interpret as ‘intentionally causing others to update in the wrong direction’. But the goal to have people not be shocked by the inner layers suggests that there’s a duty to actively inform people about (some aspects of) what’s inside; leaving them with their priors isn’t good enough. (But what exactly does “shocked” mean, and how does it compare with other possible targets like “upset” or “betrayed”?) And the parts about “signposting” suggest that there’s an aim of helping people build explicit models about the inner layers, which is not just a matter of what probabilities/anticipations they have.
I meant signposting to indicate things like saying “here’s a place where I have more to say but not in this context” etc, during for instance a conversation, so I’m truthfully saying that there’s more to the story.
Yeah, I think “intentionally causing others to update in the wrong direction” and “leaving them with their priors” end up pretty similar (if you don’t make strong distinctions between action and omission, which I think this test at least partially rests on) if you have a good model of their priors (which I think is potentially the hardest part here).
Thanks for the writeup. More people should take credit for things they thought about doing and didn’t because there’s lots of almost good ideas that are actually a waste of time and documenting those helps other avoid even bothering to investigate (unless they think they know something that others don’t, and then having such docs helps them make concrete why they still think it’s worth exploring an idea that others dismissed).
Late comment but I recently posted how human values arise naturally by the brain learning to keep its body healthy in the ancestral environment by a process that could be simplified like this:
First, the brain learns how the body functions. The brain then figures out that the body works better if senses and reflexes are coordinated. Noticing patterns and successful movement and action feels good.
Then the brain discovers the abstraction of interests and desires and that the body works better (gets the nutrients and rest that it needs) if interests and desires are followed. Following your wants feels rewarding.
Then the brain notices personal relationships and that interests and wants are better satisfied if relationships are cultivated (the win-win from cooperation). Having a good relationship feels good, and the thought of the loss of a relationship feels painful.
The brain then discovers the commonalities of expectations within groups—group norms and values—and that relationships are easier to maintain and have less conflict if a stable and predictable identity is presented to other people. Adhering to group norms and having stable values feels rewarding.
These natural learning processes are supported by language and culture by naming, and suggestion behaviors make some variants more salient and thus more likely to arrive—but humans would pick up on the principles even without a pre-existing society—and that is what actually happens in certain randomly assembled societies.
the entire brain and body needs to get built by a mere 25,000 genes. My current low-confidence feeling is that reasonably-comprehensive pseudocode for the human hypothalamus would be maybe a few thousand lines long.
That confirms what I have been getting around to over time: Human instincts and motivational systems are probably built from few elements. I used to think that there are hard-wired motivations/interests for specific types of sports or hobbies or people and how this could be coded for. But after reading Steven’s sequence, I look more for simple patterns like geometric features, hormonal triggers, or sensory triggers that could—after a lot of learning—give rise to such preferences. Maybe some sport is only interesting because it involves fast-moving objects in the visual field and the smell of grass…
I recognize that examples need to be legible, and I would also hope for deep internalization of pulling the rope sideways and investigation of policy areas that are less legible and therefore dramatically easier to shift on the margin.
To reach the boundary of what is known in your chosen field will require reading lots of papers, which will take (at least) several years. Doing research will also require implicit knowledge that is part of the field, but does not appear in papers.
Are you the kind of person who can spend several years reading papers without significant external help?
Where are you going to acquire the implicit knowledge, e.g., how to run experiments?
PhD students are the work-horses of academic research, and don’t have the power/money/experience to do anything other than tow the line. You have a degree of independence and experience that will deter many academics taking you on as a student.
Or: You could kick-start your research by applying your existing knowledge of (I assume) computing/software to cognitive issues in this field (see chapter 2)
I have no idea about cognitive science in particular, but in math and physics there is not a single worthwhile contribution that was not done either by PhD holders or PhD students. Just not a thing that happens.
I think I have two main complaints still, on a skim.
First, I think the following is wrong:
These problems seem pretty hard, as evidenced by the above gaps in the example shard theory plan, and it’s not clear that they’re easier than inner and outer alignment.
I think outer and inner alignment both go against known/suspected grains of cognitive tendencies, whereas shard theory stories do not. I think that outer and inner alignment decompose a hard problem into two extremely hard problems, whereas shard theory is aiming to address a hard problem more naturally. I will have a post out in the next week or two being more specific, but I wanted to flag that I very much disagree with this quote.
Second, I’m wary of saying “maybe we can get corrigibility” or “maybe corrigibility doesn’t fit into a utility function”, because this can map shard theory hopes onto old debates where we already have settled into positions. Whereas I consider myself to be thinking about qualitatively different questions and spreads of values I might hope to get into an AI.
I think corrigibility is natural iff robust pointers to it can easily get into the AI’s goals
This doesn’t make sense to me. It sounds like saying “liking yellow cubes is natural iff we can get a pointer to ‘liking yellow cubes’ within the AI’s goals.” That sounds like a thing which would be said if we had no idea how yellow cubes got liked, directly, and were instead treating liking-yellow-cubeness as a black box which happened to exist in the real world (e.g. how corrigibility, or the desire to help people, could be “pointed to” in a classic corrigibility hope).
I have more thoughts on this post but I don’t have time to type more for now.
I think outer and inner alignment both go against known/suspected grains of cognitive tendencies, whereas shard theory stories do not. I think that outer and inner alignment decompose a hard problem into two extremely hard problems, whereas shard theory is aiming to address a hard problem more naturally. I will have a post out in the next week or two being more specific, but I wanted to flag that I very much disagree with this quote.
Since the original draft I realized your position has “outer/inner alignment is a broken frame with mismatched type signatures which is much less likely to work than people think”, so this seems reasonable from your perspective. I haven’t thought much about this document and might end up agreeing with you, so the version I believe is something like “it’s not clear that my shard theory decomposition is substantially easier than inner+outer alignment is, assuming that inner+outer alignment is as valid as Evan thinks it is”.
Agree that I’m not being concrete about how corrigibility would be implemented. Concreteness is a virtue and it seems good to think about this in more detail eventually.
No. Have actually been working in/on science communication about psychedelics (as medicines) and have received both positive feedback from researchers and am collaborating with a few.
One thing that I like about being parallel to academia is that you can build things outside of the constraints. For instance am building a tracker of all RCTs which will update automatically when new papers are added. And made a map of research that visually shows where what is taking place.
In my personal experience, the intuition/grokking approach works better for mathematics, but the making-notes approach works better for programming.
This is not a fair comparison, because in case of programming, the notes were usually already made by someone else (on Stack Exchange), so I can use them without spending my time to write them. For deep understanding, I must usually do the work myself, because most people who write programming tutorials actually do not have the deep understanding. (Seems to me that most tutorials are written by enthusiastic beginners.)
I have a strong personal preference for the intuition/grokking approach, so I am not doing the optimal thing despite knowing what it is. But I have to admit that this is often a waste of time in programming. Before you spend the proverbial 10 000 hours learning some framework, it is already obsolete. Or if it becomes popular, sometimes the next version keeps the similarities, but changes how it works under the hood, so ironically the deeper knowledge that should be long-term sometimes becomes obsolete first.
In math, the stuff you learn usually remains true for a long time, so building intuitions pays off.
It would be interesting to compare the half-lives to the duration of behaviors, emotions, and moods associated with these neuropeptides. I only recognize the NPY mentioned above in the study.
Neat, thanks so much for these recommendations! I do of course follow 3b1b, and I already know some Category. But I’ll for sure check out all of the rest, which sound super cool!
I wrote a couple posts (that I haven’t reviewed recently) on this topic. You may find them informative, or at least amusing. Insightful points were made by commenters.
On reading them back over, here’s my updated take (partly informed by the fact that I am now an MS student and work with PhD students daily):
A PhD is two things: a way to support research, and a way to reward it with a sheepskin.
Budding researchers just tend to see academia as a marginally more useful and attractive place to build their career than the next best alternative. The fact that a PhD program is mostly the only way to get the most widely recognized signal of being a qualified researcher is a bonus. If they’re going to do weird groundbreaking research anyway, why not do it in an environment that’s relatively open to it, and that will provide them with a credential at the end of the process?
If I was doing this over again, then, I might reframe the question. Instead of asking “is a PhD necessary,” I might ask questions like:
“Why do excellent early-career researchers so often see a PhD as attractive?”
“What are some of the practical barriers to doing a certain type of research outside the academic system?”
“What is the next most logical alternative to building a research career without getting a PhD?”
“Why and how do some excellent early-career researchers choose to build their research career outside the normal academic system?”
“What exactly does it mean to ‘contribute’ to a field?”
As an example, if you’re going into cogsci, you might need to run experiments on people or animals. When we do animal research in our BME lab, we have all kinds of support and regulation and procedures for making sure it doesn’t create a fiasco with government, administrators, or activists. Outside the academic system, I expect that trying to do animal research would be extremely difficult or impossible, not to mention publishing it and getting it taken seriously. It’s not just a question of whether the research findings were any good. It’s the perception. Academia is as sensitive to perception as anybody, and you want your research to not just be ethical and correct, but avoid looking ethically or epistemically suspect.
As a less charged example, I at one point was trying to figure out how to visualize a DNA ladder in an agarose gel without staining the rest of the DNA. I proposed cutting off the ladder band, staining that, and then jigsawing it back onto the rest of the gel. My mentor told me “if you do that, you go to science JAIL.” His objection wasn’t that this would fail to tell us the information we needed to know. It was that it would look really bad in light of all the replication issues specifically with fraudulent gels.
Any way you slice it, you’ll have to figure out a way to do high-quality research while making it look good enough for others to take seriously. This is probably a lot easier with an academic pedigree and institutional support. Another way of putting it might be “if you were enough of a genius to not need a PhD, you’d probably know that about yourself already!” Not a diss − 99.99% of researchers are probably in this category, myself included.
Requesting beta readers for “Unit Test Everything.”
I have a new post on how to be more reliable and conscientious in executing real-world tasks. It’s been through several rounds of editing. I feel like it would now benefit from some fresh eyes. If you’d be willing to give it a read and provide feedback, please let me know!
They’re delaying their ascension, in dath ilan, because they want to get it right. Without any Asmodeans needing to torture them at all, they apply a desperate unleashed creativity, not to the problem of preventing complete disaster, but to the problem of not missing out on 1% of the achievable utility in a way you can’t get back. There’s something horrifying and sad about the prospect of losing 1% of the Future and not being able to get it back.
Is dath ilan worried about constructing an AGI that makes the future 99% as good as it could be, or a 1% chance of destroying all value of the future?
I had assumed the first—they’re afraid of imperfect-values lock-in. I think it’s the “not to the problem of preventing complete disaster” phrase that tipped me off here.
I hate to be the bearer of bad news, but the methodology in the study you linked is just terrible.
99% of candidate gene studies do not replicate, and the first study you link uses an extremely error-prone P < 0.05 threshold to determine that orexin has an effect.
It may still have an effect, but there’s no way to really know this without some kind of large sample GWAS.
The proper way to do this would be to get research access to a large genetic database like 23&Me or UK BioBank and pair that data with info from sleep trackers like Apple Watch, FitBit, or other devices to see which genes are involved in sleep duration and how large the effect of each is.
Depending on the sparsity of the trait, you’d probably need somewhere between 100k − 1 million genotype-phenotype pairs to make a good predictor.
Lastly, in the “Possible Actions” section you should include embryo selection. Genes obviously play some role in sleep needs. If at least some of them have no relevant downsides, selecting for those genes in embryos would probably be quite cost-effective. In fact, you could further reduce the odds of any unintended side-effects by selecting not just for shorter required sleep duration, but also for longer quality-adjusted lifespan.
When doing gene studies if you look at all genes and see which one’s have P < 0.05, that gets you a lot of false positives. That’s however not what they did in the linked paper.
The linked paper looks at one gene because previous papers identified that gene as being significant in humans. The paper says that the gene has significance for sleep in mice.
The argument doesn’t rest on a single experiment. I agree that it would be desirable to have 23&Me cooperate with Apple or Google to find genes that affect the metrics that the Apple watch / Fitbit measures. That’s helpful to have a good overview of all the mutations that affect sleep length.
Lastly, in the “Possible Actions” section you should include embryo selection.
Yes, that’s a valid action. I was thinking about actions that actually result push the field forward. You could have benefits for the child from such embryo selection but I wouldn’t expect it to lead to knowledge generation.
Can you link the earlier studies showing the significance of BHLHE41 in humans? When I Google it all I find are other candidate gene studies with small sample sizes.
https://www.science.org/doi/abs/10.1126/science.1174443 seems to be the first paper. It does have a small sample size and thus is only able to produce candidates, but it results in the later paper not randomly searching over all possible gene mutations.
My main point is that there are papers that made independent observations and thus arguments that a single paper doesn’t demonstrate the effect doesn’t hold. I didn’t copy the exact minute numbers that the EA cause report had because I was unsure about the exactness of the data.
In the case of orexin, my argument doesn’t just rest on the DEC2 gene.
The experiments that improved performance in sleep-deprived rhesus monkeys happened before the discovery of the link between the DEC2 mutation and orexin.
The observations in Astyanax mexicanus seem independent from my perspective. Attempts to make Astyanax mexicanus a model organism aren’t driven by sleep researchers but because it’s interesting for studying evolution.
Orexin deficiency causing Narcolepsy type 1 is independent of any findings about the DEC2 gene as well.
As far as the linked post of Scott goes, it says nothing about experiments on animals other than humans. Gene knockout studies in mice and Drosophila seem to me like a pretty good way to measure the influence of a gene.
As always, not investment advice. There are signs that a volatility spike is imminent, which often coincides with a market drop. I have reversed my usual short vol position and bought tail insurance (e.g. OTM puts). Remember vol can fall just as quickly. How long a spike lasts depends on how high it goes.
I have thoughts, but they are contingent on better understanding what you mean by “types” of hidden information. For example, you used “operating a cocaine dealership” as a “type” of information hidden by the question about health information. Operating a cocaine dealership is not a health matter, except perhaps indirectly if you get high on your own supply.
To further illustrate this ambiguity, a person might be having gay sex in a country where gay sex is criminalized, morally condemned, and viewed as a shocking aberration from normal human behavior. It does not seem to me to be out of integrity for a gay person to refrain from telling other people that they have gay sex in this (or any other) context.
Where this becomes problematic is when the two people have different expectations about what constitutes reasonable expectations and moral behavior. If we give free moral license to choose what to keep private, then it seems to me there is little difference between this onion model and an algorithm for “defining away dishonesty.” One can always justify oneself by saying “other people were foolish for having had such unreasonable expectations as to have been mislead or upset by my nondisclosure.” If “outsiders” are expected to be sufficiently cynical, then the onion model would even justify outright lying and withholding outrageous misbehaviors as within the bounds of “integrity,” as long as other people expected such infractions to be occurring.
It short, it seems that this standard of integrity reduces to “they should have known what they were getting into.”
As such, the onion model seems to rely on a pre-existing consensus on both what is epistemically normal and what is morally right. It is useful as a recipe for producing a summary in this context, but not for dealing with disagreement over what is behaviorally normal or right.
As such, the onion model seems to rely on a pre-existing consensus on both what is epistemically normal and what is morally right. It is useful as a recipe for producing a summary in this context, but not for dealing with disagreement over what is behaviorally normal or right.
I agree. For me it’s more of a characterization of honesty, not integrity (even though I consider honesty an aspect of integrity). Perhaps we should change the name.
I think part of my answer here is “The more a person is a longterm trade partner, the more I invest in them knowing about my inner layers. If it seems like they have different expectations than I do, I’m proactive in sharing information about that.”
Yes, I agree the onion model can be a way to think about summarization and how to prioritize information sharing. I just don’t see it as particularly helpful for integrity.
Observation: Humans seem to actually care about each other.
You need to observe more (or better). Most humans care about some aspects of a few other humans’ experience. And care quite a bit less about the general welfare (with no precision in what “caring” or “welfare” means) of a large subset of humans. And care almost none or even negatively about some subset (whose size varies) of other humans.
Sure, most wouldn’t go out of their way to kill all, or even most, or even a large quantity of unknown humans. We don’t know how big a benefit it would take to change this, of course—it’s not an experiment one can (or should) run. We have a lot of evidence that many many people (I expect “most”, but I don’t know how to quantify the numerator nor denominator) can be convinced to harm or kill other humans in circumstances where those others are framed as enemies, even when they’re not any immediate threat.
There are very common human values (and even then, not universal—psycopaths exist), but they’re things like “working on behalf of ingroup”, “suspicious or murderous toward outgroup”, and “pursuing relative status games among the group(s) one isn’t trying to kill”.
Yeah, I think the “working on behalf of in-group” one might be rather powerful, and I was aware that this is probably a case where I just interact mostly with people who consider “humans” as the ingroup. I don’t think the share of the population who shares this view is actually as important as the fact that a sizeable number of people hold this position at all. Maybe I should have called it: I do care about everyone to some extent. Is that what we want to achieve when we talk about alignment?
Well that’s nice. Does anyone have a general sense of where we are with detection and launch windows? I’m somewhat hoping that we already have enough monitoring capability to detect potential existential risk asteroids in time, but very unsure and don’t have time to research it.
Actually, I think that post is probably what triggered me to write this originally, and I forgot that by the time I wrote it (or I would have added a link.) Thanks for the reminder!
I haven’t read that deeply into this yet, but my first reaction is that I don’t see what this gains you compared to a perspective in which the functions mapping the inputs of the network to the activations of the layers are regarded as the network’s elementary units.
Unless I’m misunderstanding something, when you look at the entire network f(x), where x is the input, each polytope of f(x) with its affine transformation corresponds to one of the linear segments of f(x). Same with looking at, say, the polytopes mapping layer 0 to layer L. You can just look at fL(x0), where fL are the activations in layer L, and each linear segment of that should correspond to a polytope.
However, I don’t really see how you’d easily extend the polytope formulation to activation functions that aren’t piecewise linear, like tanh or logits, while the functional analysis perspective can handle that pretty easily. Your functions just become smoother.
In the functional analysis view, a “feature” is a description of a set of inputs that makes a particular element in a given layer’s function space take activation values close to their maximum value. E.g., some linear combination of neurons in a layer is most activated by pictures of dog heads. But there’s a lot more to know about a function f than what max({f(x)|x∈X}) is.
When you scale up a particular feature in a layer past its activation range in the training dataset, you are effectively querying the functions in subsequent layers outside the domains they’ve been trained to fit well. Instead of checking how many polytope boundaries you crossed, you can just check how much f′(x) varied between your start and end points.
Scaling up some of the activations in a layer by a constant factor means you’re increasing the norm of the corresponding functions, changing the principal component basis of the layer’s function space. So it shouldn’t be surprising if subsequent layers get messed up by that.
However, I don’t really see how you’d easily extend the polytope formulation to activation functions that aren’t piecewise linear, like tanh or logits, while the functional analysis perspective can handle that pretty easily. Your functions just become smoother.
Extending the polytope lens to activation functions such as sigmoids, softmax, or GELU is the subject of a paper by Baleistriero & Baraniuk (2018) https://arxiv.org/abs/1810.09274
In the case of GELU and some similar activation functions, you’d need to replace the binary spine-code vectors with vectors whose elements take values in (0, 1).
There’s some further explanation in Appendix C!
In the functional analysis view, a “feature” is a description of a set of inputs that makes a particular element in a given layer’s function space take activation values close to their maximum value. E.g., some linear combination of neurons in a layer is most activated by pictures of dog heads.
This, indeed, is the assumption we wish to relax.
But there’s a lot more to know about a function f than what max({f(x) | x \in X}) is.
Agreed!
Scaling up some of the activations in a layer by a constant factor means you’re increasing the norm of the corresponding functions, changing the principal component basis of the layer’s function space. So it shouldn’t be surprising if subsequent layers get messed up by that.
There are many lenses that let us see how unsurprising this experiment was, and this is another one! We only use this experiment to show that it’s surprising when you view features as directions and don’t qualify that view by invoking a distribution of activation magnitude where semantics is still valid (called a ‘distribution of validity’ in this post).
I’m a pretty bad chess player (~1500 ELO) and I can play bullet games while sleep deprived without much loss in skill. I think the system 1 pattern matching is relatively unimpaired while the system 2 calculating is very impaired. In bullet calculating doesn’t help much though.
A GM playing their instant move with no calculation can crush a master-level player, and I can crush a novice playing my instant move. It’s all about system 1 :)
After the dust settled, our best estimate on paper is 40% rather than 25-30%.
The reason for the adjustments were roughly:
[x2] I estimated exhaust temperature at 130 degrees, but it’s more like 100 degrees if the indoor air is 70.
[x1/2] I thought that all depressurization was compensated for by increased infiltration. But probably half of depressurization is offset by reduced exfiltration instead (see here)
[x3/2] I only considered sensible heat. But actually humidity is a huge deal, because the exhaust is heated but not humidified (see here)
With 1-hose the indoor temp was 68 vs 88 outside, while with 2-hose the indoor temp was 66 vs 88 outside (using the same amount of energy).
We both agree that 10% is an underestimate for the efficiency loss (e.g. due to room insulation, other cooling in the building, and the improvised 2-hose setup).
I don’t think we have a plausible way to extract a corrected estimate.
My comment did not assume the physical. Entropy is not a physical concept but a statistical one. It can be used to measure the complexity of any system, physical or not.
Any real system, whether physical, spiritual, or anything else, will necessarily be computable. Even the mind of God would have to be technically computable, although you could argue that it would require more computational power than could possibly fit in the physical universe.
Again, there is no magic. Nothing is intrinsically mysterious. Mystery is a symptom of a finite, ignorant mind. Whatever realms might exist beyond this physical universe are necessarily logically coherent and explainable in principle. And that applies to any subsystem of any reality, whether consciousness, life, or anything else.
One thing I’ve found useful is to make sure I identify to the supplier what specifically I need about the product I’m ordering—sometimes they have something similar in stock which meets my requirements.
This is a great collection of tips! I think it’s also worth explicitly noting that most of these strategies involve slowing down other people’s orders, and many involve more inconvenience/stress for the sellers, so it’s important to weigh this tradeoff.
This year Petrov day almost sneaked past me. This strikes me as weird on account of the biggest proxy war since the 80s being underway, putting us closer the same stakes in realspace.
This is great! I really like your “prediction orthogonality thesis”, which gets to the heart of why I think there’s more hope in aligning LLM’s than many other models.
One point of confusion I had. You write:
Optimizing toward the simulation objective notably does not incentivize instrumentally convergent behaviors the way that reward functions which evaluate trajectories do. This is because predictive accuracy applies optimization pressure deontologically: judging actions directly, rather than their consequences. Instrumental convergence only comes into play when there are free variables in action space which are optimized with respect to their consequences.[25]Constraining free variables by limiting episode length is the rationale of myopia ; deontological incentives are ideally myopic. As demonstrated by GPT, which learns to predict goal-directed behavior, myopic incentives don’t mean the policy isn’t incentivized to account for the future, but that it should only do so in service of optimizing the present action (for predictive accuracy)[26].
I don’t think I agree with this conclusion (or maybe I don’t understand the claim). I agree that myopic incentives don’t mean myopic behavior, but they also don’t imply that actions are chosen myopically? For instance I think a language model could well end up sacrificing some loss on the current token if that made the following token easier to predict. I’m not aware of examples of this happening, but it seems consistent with the way these models are trained.
In the limit a model could sacrifice a lot of loss upfront if that allowed it to e.g. manipulate humans into giving it resources with which to better predict later tokens.
Depends on what you mean by “sacrificing some loss on the current token if that made the following token easier to predict”.
The transformer architecture in particular is incentivized to do internal computations which help its future self predict future tokens when those activations are looked up by attention, as a joint objective to myopic next token prediction. This might entail sacrificing next token prediction accuracy as a consequence of not optimizing purely for that. (this is why I said in footnote 26 that transformers aren’t perfectly myopic in a sense)
But there aren’t training incentives for the model to prefer certain predictions because of the consequences if the sampled token were to be inserted into the stream of text, e.g. making subsequent text easier to predict if the rest of the text were to continue as expected given that token is in the sequence, because its predictions has no influence on the ground truth it has to predict during training. (For the same reason there’s no direct incentive for GPT to fix behaviors that chain into bad multi step predictions when it generates text that’s fed back into itself, like looping)
Training incentives are just training incentives though, not strict constraints on the model’s computation, and our current level of insight gives us no guarantee that models like GPT actually don’t/won’t care about the causal impact of its decoded predictions to any end, including affecting easiness of future predictions. Maybe there are arguments why we should expect it to develop this kind of mesaobjective over another, but I’m not aware of any convincing ones.
Got it, thanks for explaining! So the point is that during training the model has no power over the next token, so there’s no incentive for it to try to influence the world. It could generalize in a way where it tries to e.g. make self-fulfilling prophecies, but that’s not specifically selected for by the training process.
Yup exactly! One way I sometimes find it to helpful to classify systems in terms of the free variables upstream of loss that are optimized during training. In the case of gpt, internal activations are causally upstream of loss for “future” predictions in the same context window, but the output itself is not casually upstream from any effect on loss other than through myopic prediction accuracy (at any one training step) - the ground truth is fixed w/r/t the model’s actions, and autoregressive generation isn’t part of the training game at all.
The biggest yet least discussed problem in the world today is the ever tightening web of monstrously evil, defective, and barely usable interfaces, most notably software. Efforts to make UIs seem “simpler” on a lowest-common-denominator super-shallow level are the greatest curse of the past two decades. Every attempt to even mention this problem here leads to a virtual shadowban. My proposed initial solution of requiring every program to have a single page text list of ALL options (no submenus) triggers even more hate.
strong downvote, strong agree: this is offtopic, but a perfectly reasonable topic to start a thread about. it doesn’t seem like a bottleneck for the world to me, though, because nobody is trying to remove the CLI, and in fact even microsoft has been putting effort into ensuring that good CLIs are available on windows. if you’d like to discuss it, I suggest creating a post about it; I expect it to get little involvement, because as I said, I simply don’t agree that it’s catastrophic and don’t find this to be important compared to AI and human inter-being friendliness/alignment. Since you commented that you feel ignored, I figured I’d comment on why.
They call the study “double masked” – I hope this means double-blinded, because otherwise the study is close to worthless since their primary outcomes are based on doctor’s behavior.
Double-masked does mean double-blinded and comes from the first studies literally using masks for blinding. Jeremy Howick does advocate in The Philosophy of Evidence-Based Medicine that it’s better to use the term double-masked because people often get confused when thinking about blinding.
People frequently think that blinding means a person doesn’t know whether or not they get the treatment, which is not how it’s operationalized. Researchers, who actually care about whether or not people know whether they get the treatment, ask their patients whether or not they think they get it.
First we need to reflect why we can even recognize altruismo. that requires identifyimg good and bad.
I think one can observe what other people/agents “want” in a framework that you might call materialist. Maybe I am wrong. Maybe once you act on behalf of other agents, this includes some value judgments. For example, some negative utilitarians might argue that pressing the button is an altruistic act.
I don’t think what happened through ancestor history was moral. I just observe it seems to have “worked”. My question concerns if we could achieve the same with regards to AI having human compatible goals.
Have you seen any research or discussion on whether independent researchers are more likely to develop unique moral views (because they aren’t part of a centralized entity), and therefore bring diversity to the global research effort?
Near-Earth objects pose a serious and credible threat to humankind, as scientists are certain that a major asteroid or comet was responsible for the mass extinction of the majority of the Earth’s species, including the dinosaurs, nearly 65,000,000 years ago.
Similar objects have struck the Earth or passed through the Earth’s atmosphere several times in the Earth’s history and pose a similar threat in the future.
Several such near-Earth objects have only been discovered within days of the objects’ closest approach to Earth, and recent discoveries of such large objects indicate that many large near-Earth objects remain undiscovered.
The efforts taken to date by the National Aeronautics and Space Administration for detecting and characterizing the hazards of Earth orbit-crossing asteroids and comets are not sufficient to the threat posed by such objects to cause widespread destruction and loss of life.
The Administrator shall plan, develop, and implement a Near-Earth Object Survey program to detect, track, catalogue, and characterize the physical characteristics of near-Earth asteroids and comets equal to or greater than 100 meters in diameter in order to assess the threat of such near-Earth objects in striking the Earth.
The Congress declares that the general welfare and security of the United States require that the unique competence of the National Aeronautics and Space Administration in science and engineering systems be directed to detecting, tracking, cataloguing, and characterizing near-Earth asteroids and comets in order to provide warning and mitigation of the potential hazard of such near-Earth objects impacting the Earth.
There are authorized to be appropriated to the National Aeronautics and Space Administration for the Near-Earth Object Survey program $20,000,000 for each of the fiscal years 2006 and 2007.
It is generally agreed that PhD is only worthwhile for those interested in staying in academia, at least for most STEM fields. Even then, it is a long slog and a lot of competition in the fields I am familiar with before you get that coveted assistant prof position, likely being stuck for the rest of your life at one place, underpaid relative to industry and forced to teach disinterested masses, writing grant proposals and navigating departmental politics. That said, I would have gladly stayed in academia after my PhD, were it a possibility in my case. If you have plans on working in the industry eventually, why not start now?
Having a Ph.D. confers relatively few benefits outside of academia. The writing style and skills taught in academia are very very different from that of industry, and the opportunity cost of pursuing a Ph.D. vs going into software engineering (or something similarly renumerative) is in the hundreds of thousands of dollars.
I would suggest that if you don’t know exactly what you want to do with your life, you would be well-suited to doing something that earns you a bunch of money. This money can later be used to finance grander ambitions when you have figured out what you want to do.
I’ll turn this question around on you: why is a Ph.D. the best way of accomplishing what you want to do?
As to the drudgery of office work—“office work” is, i think, a false category. I spent hours of unbearable tedium performing repetitive reactions in lab during my PhD, and my current cushy Microsoft engineering job is enormously more creative and interesting while paying approximately 10x as much. For someone with the smarts to get a ph.d., retraining into engineering is very, very easy.
One other generally undiscussed aspect of the working world is that, for a number of reasons, your employers mostly treat you with respect roughly proportional to your salary. Ph.D.s, consequently, are often treated very poorly. This probably contributes to their poor mental health, as documented elsewhere.
The mental health of PhD students is the main reason I’m not interested in doing one. There are lots of surveys showing terrible symptoms, e.g. this one where 36% of PhD students had sought treatment for depression or anxiety related to their PhD. That’s nearly 5x higher than the baseline 8% of Americans. You can find many similar surveys, and my anecdotal experience of asking PhD students whether they’re enjoying the experience has only confirmed this dismal picture.
Hm. I wonder about selection effects. PhD students might be more aware of mental health issues, and PhDs often come with some level of access to mental health services and occur in cities where those services are available. PhD students are also typically at an age when the baseline rate of anxiety and depression is significantly higher—I see figures in the 20% range (although this is for having anxiety, not seeking treatment). It’s natural that whatever anxiety and depression they’d feel would be centered on their main focus in life.
I wouldn’t be too surprised if anxiety and depression are somewhat elevated among PhD students as compared to their peers with a similar socioeconomic status, age, and IQ, but I suspect that that 36% vs. 8% comparison might be somewhat misleading.
There is no evidence that the structure and function of the human brain (along with its physical embodiment) is insufficient to explain all aspects of human intelligence and consciousness. Even if it was discovered that the physical brain fell short of accounting for all cognitive phenomena, and even if no other known physical system or process could account for it, and even if an immaterial soul was actually discovered somehow, it still wouldn’t imply that human intelligence could not be emulated.
There is no magic. All real phenomena are causally computable. An immaterial soul would be no exception. Even a non-physical conscious system would necessarily possess inner computational mechanisms that are themselves not conscious (e.g., “spiritual synapses”).
The only cop-out would be if the immaterial soul was performing necessary computations so vast that they could not be performed on a physical computer the size of Earth within a human lifespan (therefore necessitating a non-physical computational substrate that is not bound by physical constraints on memory density and processing speed). There is no evidence of this.
You have too high a view of human-level intelligence and consciousness. We are in fact intellectual bottom-feeders in the vast ocean of possible mind-space. And that’s even accounting for physical limits on computational power.
Intelligibility is not evidence for intelligent design. It is evidence for a low-entropy system, which has a far higher prior probability than a supernatural Intelligentsia.
There is no evidence that the structure and function of the human brain (along with its physical embodiment) is insufficient to explain all aspects of human intelligence and consciousness.
Nope. (Though since both that game and this one are weird administration-centric takes on Harry-Potter-style magical schools, I imagine there may have been some convergent evolution.)
This seems in line with your position, but I want to reply so people won’t conclude “And coinrun experiments don’t tell you important things.” I think the more interesting question for that experiment is “how will the agent generalize? Can we predict it in advance? In what ways do we systematically mispredict, and why?”
(And, roughly speaking, there are a range of possible algorithms by which the agent can generalize, so it’s way more than one bit. I got way more out of that paper by asking the authors for hundreds of videos for different training settings. Probably will mention the results in a post, soon.)
Evolution actually solved alignment (approximately) on two levels: aligning brains with evolution’s goal of inclusive genetic fitness—to be fruitful and multiply - (even though the easy/natural goal attractor for intelligence is empowerment), and aligning brains with other brains because the unit of selection (genes) is spread across individual bodies (the disposable soma). There are many obvious failure examples, but that’s hardly surprising because evolution proceeds by breaking things.
The first type of alignment (towards genetic fitness) gives rise to lust, sexual attraction, parenting desires, etc. The second form of alignment (to other humans) - more relevant for AGI—manifests as empathy, love, and altruism.
I’m particularly impressed by “The Floating Droid”. This can be seen as early-manifesting the foreseeable difficulty where:
At kiddie levels, a nascent AGI is not smart enough to model humans and compress its human feedback by the hypothesis “It’s what a human rates”, and so has object-level hypotheses about environmental features that directly cause good or bad ratings;
When smarter, an AGI forms the psychological hypothesis over its ratings, because that more sophisticated hypothesis is now available to its smarter self as a better way to compress the same data;
Then, being smart, the AGI goodharts a new option that pries apart the ‘spurious’ regularity (human psychology, what fools humans) from the ‘intended’ regularity the humans were trying to gesture at (what we think of as actually good or bad outcomes).
For sexually reproducing species, evolution will disfavour genes that lead to individuals slaughtering others of their species for little or no reason. Slaughtering members of the opposite sex obviously reduces mating opportunities. Slaughtering members of the same sex reduces the mating opportunities of their opposite-sex children.
This obviously hasn’t completely prevented humans from slaughtering both particular individuals and entire populations of humans of a different tribe. But this isn’t what always happens. It could be a lot worse.
Maybe it is worse for asexual species (mitigated by slower evolution without sex), perhaps explaining why at least for vertebrates asexuality is rare, and asexual species seem to not last long.
I chose exactly the wrong D&D.Sci to decide to not try building a model on, and instead try to solve just by eyeballing simple scatterplots.
Despite coming in “last place” I’m pretty happy with my results!
I think this was a perfectly reasonable setup. Even more so given that without any straightforward scenarios people won’t think “what if it’s just straightforward though”.
I thought many times during eyeballing “look, probably no one else has tried to build a model using all combos of mins and maxes of stats as features, just do it” but I stuck to my guns. For, uh, reasons. Presumably.
My biggest mistake, as I see it, was failing to generalize from “I have several max-of-two-formulas and a Dragonslayer distribution that is obviously a mix of two which I can’t seem to resolve into something nice, probably all of them are max-of-two-formulas, let’s see if I can refactor things to look like that and get a better idea of what Dragonslayer’s two might look like if the rest do factor well”.
Thanks for these tips, I can probably put them to good use! I’m curious though, what’s so special about custom Japanese beds that you needed them quickly?
I was setting up a retreat venue, and they were pretty weird and special beds—such that if they would’ve actually worked, it would’ve pivoted our strategy for setting up the space in a somewhat major way.
= sugar in excess of what one would consume normally by eating fruits & vegetables and maybe 1 square of dark chocolate.
I splurged (everything else were essentials / healthy food) on one item this weekend from Trader Joe’s: Pumpkin Spice Cinnamon Rolls. Yes they were tasty, duh. After consumption...now I feel physically ill: almost sweating and general discomfort, anxiety, elevated heart rate, trouble focusing, restlessness, etc. Oh! This is not an isolated set of bad feelings, these bad feelings occur every single time I eat sugar in excess of eating a few particular sweet fruits. (I have a great A1C level, for those who are worried this is due to something abnormal; I get full bloodwork done about 4 times a year [last time was late summer] and those levels have consistently been excellent)
I don’t care for a fully general rule of sugar = always bad...but from my experiences...I think it’s best for me to continue cutting unnatural amounts of sugar out of my life, because wow do I feel great when I do cut that out. The last few months I’ve steadily reduced my sugar consumption and now even eating two or three oreos feels intense and can trigger the bad feelings...eating a few cinnamon rolls at once was definitely a mistake. I intend to learn from this experience and the other bad experiences I’ve had with consuming unnatural amounts of sugar these last few months: stop consuming this particular drug [sugar], it’s bad for me and makes me feel bad.
Thank goodness I like the taste of >=90% dark chocolate and even pure baker’s chocolate. Figs are pretty great too, they almost taste too sugary but they don’t make me feel bad so I guess they are fine. I can cut out unnatural amounts of sugar, feel better, and still get to enjoy tasty dessert-style foods that I like (very dark chocolate, figs, sauteed onions, berries, and more), it’s a win win!
I can successfully access the front page by using the highly sophisticated method of inspecting the front page and deleting the div element that corresponds to the Petrov Day loss screen.
It might be a good idea to make the front page getting taken down have more serious of an impact, as the time cost of clicking “inspect” and deleting the associated div element is pretty low, so the overall cost of submitting the launch codes is likely substantially reduced by this way of accessing the front page remaining available after a launch.
If it takes a minute to hide the div and 10 AI safety researchers spend that minute then the mean cost of the home page being down is probably somewhere between zero and a billion lives, depending on your assessment of AI risk and the effectiveness of AI safety research and the best use of the future lightcone.
I think that’s actually fine. Most people won’t do that, but if for some reason the front page is really important for someone then it’s good that there’s an option to still access it.
My impression of the LW crowd is that the median person on the forum would have enough technical expertise to figure that out, but maybe that impression is poorly calibrated? I’m not sure.
I continue to expect that I will prefer to control my computer with formal grammars—I have spent significant time using caster speech recognition with dragon, and I’m sure I’ll keep doing so with openai whisper. nobody has ever beaten the CLI and nobody ever will [edit: to clarify, I believe it is a weakness of current ai that it struggles to take formal grammar seriously].
now, if an AI could automatically generate interfaces that standardize messy UIs into coherent formal grammars that fit comfortably in a CLI workflow, that would be amazing...
also, I want a CLI that shows me what commands are available at any step.
yeah, eventually keyboard clis will be beaten, but even with whisper I expect to sometimes prefer keyboard. it’s just that hard to beat CLIs.
If we do not address the deep dysfunctions of our government and its policies, we put our democracy and entire civilization at risk.
...
My new project, together with Moshe Looks and Alyssa Vance, is to chart paths forward to improve federal policy, and lay groundwork to implement those improvements. That means taking into account political feasibility. It means getting the proposals and messaging into the hands of candidates. It means commissioning academic studies quantifying costs and benefits and advance drafting of legislative language.
I’d be interesting in hearing more about how those two paragraphs are connected. How do we go from 1) lobbying for and getting specific policies implemented to 2) correcting the deep dysfunctions in our government?
I’m also not clear if this is implied or not, but is (2) supposed to mitigate existential risks? If so, I’d be interested to hear more about the specific thoughts and plans surrounding that.
I liked this one a lot. I imagined that ‘train a linear classifier’ would be the next step, but didn’t do it due to laziness: it looks like that would probably have worked.
I do feel like my approach should have worked worse than it did—I did most of my evaluation by ignoring the scores and looking only at your historical classifications, and the one place where I let scores guide me into overriding my initial model (moving Student P from Humblescrumble to Serpentyne) it turned out I was incorrect and the initial model would have scored better (oops).
The front page going down doesn’t actually make people who want to check the latest posts unable to due so; it’s so easy to circumvent that I think the front page going down is nearly costless
Going by figure 5, I think the way to format climate contingent finance like an impact certificate would be:
‘A’ announces that they will award $X in prizes to different project based on how much climate change damage a project averts
‘B’ funds different projects (based on how much damage they think each project will avert in the future) in exchange for a percentage of the prize money
The fact that you highlight Trump as particularly authoritarian and keeping him out of office in 2024 as a priority shows that this is a hopelessly mindkilled effort.
This is an opinion that even some people who Trump appointed himself hold. Geoffrey S. Berman is the latest person who described it.
Trump right now makes the question of whether or not someone agrees that the 2020 election was fraudulent as the test of whether or not he supports the person. That’s a filter that will allow him to appoint people in a way that reduces the pushback that he got in his last administration when he tried to do authoritarian things like pushing Berman.
Hard to give a general answer, but I think 2x someone’s normal salary (especially if it’s cash) is usually quite sufficient to get the job done, and kind of reliably has helped me in the past when I’ve try to find people happy to work night shifts
Resolution notes: “Whatever values are reported that day will be used unconditionally.” or “If the values change within the following week, the updated values will be used instead.” or “The resolution method is a guideline only; ultimately the intent of the question, as interpreted by the question author, overrides.”
This will only solve a subset of ambiguous resolutions, but I think it would still be a big help to spell some of these things out more clearly.
Really cool to see you taking the time to present your thoughts in a way that is constructive whilst not holding back with your criticisms. A lot of people would have just gotten frustrated with their experience on the site and have just moved on, so thank you for this post! We definitely think about these things a lot as we design Manifold and this post could definitely have an impact on where we lean within tough design problems.
First off, let me briefly discuss why we are bullish on using a creator-based resolution mechanism.
Creator reputation. This is definitely the most important point. Regular users quickly learn which markets are run by credible creators who know to set clear criteria. And in the rare instance an unexpected variable occurs making the resolution less clear, these creators will often defer to what the comments agree should be the correct resolution. Admittedly this is definitely not obvious for new users who are interacting with our markets for the first time and are often times trading in good faith that all the markets are reliable. We are aiming to get a reputation system set up at some point, but we haven’t had the capacity to do so yet
Creator resolution facilitates types of questions that otherwise couldn’t exist.
Creators resolving works >95% of the time. For the majority of markets, there are no issues. Very rarely does a creator’s internal bias or ulterior motive affect the resolution. It’s unfortunate that the first market you interacted with may be one of the minority and we will be more conscious about ensuring big/featured markets are reliable. I strongly believe if you were to use the site more that you would find a lot of your concerns are exacerbated by your experience with an outlier.
Interpretation of a result is often necessary. As others have alluded to in the comments, there is often conflict between a market that is useful and answers what you want to know vs one that is extremely well-defined in advance. It will always be the case that questions come up where someone has to make a call and it makes sense that the person creating the market is the best person to do so. We have heard some good suggestions regarding crowd-sourcing resolutions so might be open to possible types of implementations of that.
Now that that’s out of the way, let’s discuss some of the specific criticisms you outlined. The following is more a reflection of my own thoughts and hasn’t necessarily been discussed in depth with the team.
The market probability becomes more reflective of the creator’s tendencies and bias and not the event itself.
I will agree that this is definitely true with some of our markets. But I actually think that is okay! One thing that sets Manifold apart from other prediction markets is that we support a wide range of markets that is only possible by giving creators full autonomy. I think it is a good thing that each individual creator is expressed in their market. Not all markets have to provide actionable, informative data even if this may not align with what is traditionally sought after with forecasting. Just to be clear though, one of our primary goals is to facilitate markets that provide actionable and informative discussion and data. But, I suspect as the site grows that it will become more obvious what the intent of each individual market is. There very well could be multiple instances of the same market which will actually have different probabilities based on who is the creator and subtle differences in resolution criteria.
Markets on Manifold are too ambiguous.
I don’t have too much to say on this as I agree with you that our markets are generally too ambiguous and what Jack said covers a lot of my own thoughts. We do encourage creators to try and be as clear as possible, but many people are still learning how to create questions. One of the main purposes of the site is to lower the entry-barrier to prediction markets and allow anyone to create one easily so it isn’t a surprise a large number of markets aren’t that great or clear. Fortunately, there is already a clear culture developing where users will comment when they see unambiguity or a possible loophole in a market. And I think you would find that our big serious markets tend to be fairly unambiguous at this point.
Manifold is untrustworthy.
I’m not sure I agree with this but that may definitely be my bias haha. I do think though that users should regard every market they bet on with mild scepticism and not default to trusting the creator. Now, as a site is it optimal to have a bunch of unreliable content? Maybe not, particularly for the new user experience and I can empathise with why this may cause you to mistrust the site. But it’s unrealistic to expect a site that gives its users full autonomy and does minimal curation to have only trustworthy content. But it is something we will need to think about more. Hopefully, it will be mostly solved once we introduce a reputation system of some sort. And from my experience, the number of markets that have been “untrustworthy” can be counted on one hand so far.
Thanks again for taking the time to try out our site (even if it didn’t go as you expected) and for doing this write-up. Am happy to discuss things further. Maybe what you’ve outlined is a bigger problem than we’ve realised and we need to spend a lot more effort designing the site around it, but I’m not yet convinced. It is good to know though that it is turning some users away.
I used a P100 elastomeric respirator pretty much any time I left the house, for multiple months in 2020 during early COVID, and intermittently after that.
The main downside, for me personally, was that people generally found understanding my speech through it difficult or impossible. This was a big enough problem that I haven’t used one in quite some time.
An off-hand attempt at market mechnanism to solve this. Have “resolution issue” predictors. Like “There will be a news paper article of lenght more than 20 cm written by a full time journalist about biological weapons AND the vague biological weapons question will resolve in the negative”.
It’s not entirely clear to me that the model is making a mistake with the expected value calculations.
The model’s goal is to complete the pattern given examples. In the other prize winning submissions, the intent of the prompter is pretty clear—e.g. there was an explicit instruction in the “repeat after me” task. But in the expected value case, all the prompts were consistent with either expected value or winning is good/losing is bad. And I think the latter frame is more accessible to people—if you asked a random person, I’m pretty sure they are more likely to go with the hindsight bias analysis.
You could argue that there’s a mismatch between the researcher’s expectations, that an EV calculation is the right way to approach these, and the behavior, but this seems to me to be more like straightforward train/test mismatch rather than anything deep going on.
One potentially interesting follow-on might be to poll humans to see how well they would do, perhaps via mechanical turk or similar. I predict that humans would be ~perfect on redefine and repeat after me, and would perform poorly on the expected value task. So they seem qualitatively different to me.
(I didn’t mention the negation task because I found the example to be confusing—a below average temp might be fine or dangerous depending on the size of the drop. Of course, negation has long been hard in NLP, so it’s perfectly plausible that it’s still a problem with LLMs. And maybe the other examples weren’t so borderline.)
We did do human validation on the tasks with Surge: redefine-math, quote-repetition, and hindsight-neglect all got 100% agreement, and NeQA got 98% agreement. I agree though that it seems likely many people would do the task ‘wrong’, so maybe the task would be improved by adding clearer instructions.
The situation feels somewhat like model splintering to me: the few-shot examples fit both patterns but the question doesn’t. The larger models are learning the incorrect generalization.
I think it’s important to note that LMs learning to respond in the same way as the average internet user is in some sense expected but can still be an example of inverse scaling – we would like our models to be smarter than that.
rules against container stacking that did major damage to our supply chains.
Is this “major damage” claim true? I remember being unsure, at the time, if the effects of the rules that limited stack height were substantial or negligible, since some people were saying that they mostly just applied to places that didn’t have the equipment to stack higher. Did anyone ever follow up to at least check how much container stacking increased after the rule change?
I am confident that the container stacking rules caused major damage when compared to better stacking rules. If we had a sensible stacking rule across LB/LA from the start I am confident there would have been far less backlog.
What is less clear is the extent to which the rules changes that were enacted mitigated the problem. While LB made some changes on the day of the post, LA didn’t act and LB’s action wasn’t complete. Thus there was some increase in permitted stacking but it was far from what one would have hoped for. And Elizabeth is right that we did not see a difference in port backlog that we can definitively link to the partial change that was enacted.
I got annoyed about titles that are of the form “You are underestimating X”, “X is underappriciated”, “X is not so bad”, “We should rethink X”.
The title writer doesn’t really know what I think and why. This pushes towards making groups homogenous in beliefs and people dealing with each other more in sterotypes.
But it was minor enough and all negative to actually comment on the specific title.
(Engineer at Manifold here.) I largely agree! Letting people make markets on anything means that many people will make poorly operationalized markets. Subjective resolution is good for bets on personal events among friends, which is an important use case, but it bad for questions with an audience bigger than that.
We need to do a better job of: 1. resolution reliability, like by adding a reputation system or letting creators delegate resolution to more objective individuals / courts. 2. helping users turn their vague uncertainties into objective questions—crucial but less straightforward. 3. surfacing higher quality content
The button isn’t showing up for me. Well, it shows up for like a second after I re-load the page but then it’s gone. I tried Opera GX browser and Chrome, it happens in both. Is this intended behaviour? I use Windows 7, maybe thats why...
The button shows up for me despite low karma. I have looked through the client-side code, and found this snippet:
const currentKarmaThreshold = getPetrovDayKarmaThreshold()
const disableLaunchButton = !userCanLaunchPetrovMissile(currentUser)
const beforePressMessage = <p>press button to initiate missile launch procedure</p>
const afterPressMessage = disableLaunchButton ? <p>You are not authorized to initiate a missile strike at this time. Try again later.</p> : <p>enter launch code to initiate missile strike</p>
This probably means Mawrak is encountering some sort of Javascript error, since the code indicates the button should only reject the launch attempt after you press it, not before.
I would argue that the most complex information exchange system in the known Universe will be “hard to emulating”. I don’t see how it can be any other way. We already understand the neurons well enough to emulate them. This is not nearly enough. You will not be able to do whole brain emulation without understanding of the inner workings of the system.
I’m just thinking—it might be that while heat does determine brain efficiency to a large degree it might not be so simple as it might be determined by the architecture.
DIfferent animals might be adapated to a certain temperature range—it might not make your brain more efficient when it is cooled in this way.
does anybody know how the cognition of coldblood creatures varies as they warm and cool throughout the day?
I’m not sure what theory that observation would test cleanly, since ectotherms have such a complicated relationship to heat.
One thing I just checked is honeybees, which I’ve read a book or two about, because I know they have large cooling requirements for their wing muscles (their blood doesn’t move O2, it just moves heat, basically). When the hive gets cold in the winter, some go to the center and lock their wings with their arms, and shiver their wing muscles, and this heats the entire hive. So they are kind of endothermic? Maybe? Depending on definitions?
It looks like probably individual honeybee brains are cooled by using honey sort of like how dogs use saliva, with evaporative cooling from the mouth. Then, when flying on a very hot day, the waste heat from the wings is moved, by the blood, to the thorax (but not to the head) to use the thorax as a radiator.
When it is cold, the stationary wings can send heat to the thorax via the blood, but somehow the head doesn’t need this.
Humans obtain value from other humans and depend on them for their existence. It is hypothesized that AGIs will not depend on humans for their existence. Thus, humans who would not push the button to kill all other humans may choose not to do so for reasons of utility that don’t apply to AGI. Your hypothetical assumes this difference away, but our observations of humans don’t.
As you not, human morality and values were shaped by evolutionary and cultural pressure in favor of cooperation with other humans. The way this presumably worked is that humans who were less able or willing to cooperate tended to die more frequently. And cultures that were less able or willing to do so were conquered and destroyed. It is unclear how we would be able to replicate this or how well it translates.
It is unclear how many humans would actually choose to press this button. Your guess is that between 5% and 50% of humans would choose to do so.
That doesn’t suggest humans are very aligned; rather the opposite. It means that if we have between 2 and 20 AGIs (and those numbers don’t seem unreasonable) between 1 and 10 would choose to destroy humanity. Of course, extinction is the extreme version; having an AGI could also result in other negative consequences.
That doesn’t suggest humans are very aligned; rather the opposite. It means that if we have between 2 and 20 AGIs (and those numbers don’t seem unreasonable) between 1 and 10 would choose to destroy humanity.
I think I might have started from a more pessimistic standpoint? It’s more like, I could also imagine living in a world where humans cooperate, but not because they actually care about each other, but would just pretend to do so? Introspection tells me that does not apply to myself, though maybe I evolved to not be conscious of my own selfishness? I am even less sure how altruistic other people are, because I did not ask lots of people: “Would you press a button that annihilates everyone after your death, if in return you get an awesome life?”. On the other hand, cooperation would probably be hard for us in such a world, so this is not as surprising?
On Metaculus, people have attempted to solve this problem by having moderators review each question before it goes live. The result has generally been that most vague questions require substantial re-editing before they can appear on the website.
Also, over time, users have become quite skilled at noticing when questions are likely to resolve ambiguously.
Even with this cultural difference between the sites, many questions still resolve ambiguously on Metaculus. It’s just really, really hard to say exactly what you mean when predicting the future.
I think Metaculus’s level of verbosity in resolution criteria is bad in that it makes questions longer to write and longer to understand (because it takes longer to read and because its more complex). Part of the goal of Manifold is to remove trivial inconveniences so that people actually forecast at all, and so that we get markets on literally everything. I think the synthesis here is to have a subset of high quality markets (clear resolution criteria, clear meta-resolution norms) but still have a fat tail of medium-quality questions.
I think I have a contender for something which evades the conditional-threat issue stated at the end, as well as obvious variants and strengthenings of it, and which would be threat-resistant in a dramatically stronger sense than ROSE.
There’s still a lot of things to check about it that I haven’t done yet. And I’m unsure how to generalize to the n-player case. And it still feels unpleasantly hacky, according to my mathematical taste.
But the task at least feels possible, now.
My reply last time is still relevant: link.
Thank you to Duncan, Robin, Kate, and Benya for feedback.
I’m interested in observations we could see that would make it the right call to leave major US cities.
One (non-goodhart resistent) way would be to use some kind of Page Rank method based on the tags that users comment and post on. Eg. People Rank high in AI posts, if their comments and AI posts were upvoted by other people with the same Expertise.
If they alternated directions, increasing the distance walked each time by a constant factor (e.g. triple), then an exit in either direction would be reached without too much wasted time.
If the distance is not more than he can manage before sleeping. The story isn’t really about overcoming physical barriers, but mental ones.
Why didn’t they try also killing off subpopulations that engaged in cannibalism, and promoting those that didn’t? And what would have most likely happened if they had tried that?
Probably it is also depends on how much information about “various models trying to naively backstab their own creators” there are in the training dataset
Shortform #139 Note to self: don’t reschedule weekly meetup
Tonight’s Virginia Rationalists: Norfolk meetup was great! We had a new individual join us who is familiar with ACX.
In retrospect, I think it was inconsiderate to suddenly reschedule the group’s weekly meetup just because I couldn’t attend. My co-organizer and others likely could have attended yesterday, and there were a few people who just couldn’t make the rescheduled meetup but had included Wednesday nights in their routine as “meetup night”. So, I intend to not reschedule the weekly social meetup unless literally everyone reaches out in advance and asks for that to occur. Weekly meetups = people build that into their routines, and should not be changed lightly.
Does Ryan have an agenda somewhere? I see this post, but I don’t think that’s it.
Somewhat related: Rationalists should have mandatory secret identities (or rather sufficiently impressive identities).
How actually do you sidestep the need for the One True Objective Function given an ELK solution? I get that it might seem plausible to take a rough objective like “do what I intend” and look at the internal knowledge of the thing for signs that it is deliberately deceiving you. If you do that, you’ll get, at best, an AI that doesn’t know that it is deceiving you (for whatever operationalization of “know” you come up with as you use ELK for training). But it could still be deceiving you, and very likely will be if optimization pressure is merely towards “AIs that don’t know that they are being deceptive”.
Our very broad hope is to use ELK to select actions that (i) keep humans safe, and give them time and space to evolve according to their current (essentially local) preferences, (ii) are expected to produce outcomes that would be judged favorably by the future humans, primarily by maximizing option value until it becomes clear what those future humans want (see the strategy stealing assumption).
This is discussed very briefly in this appendix of the ELK report and the subsequent appendix. There are two or three big foreseeable difficulties with this approach and likely a bunch of other problems.
I don’t think this should be particularly persuasive, but it hopefully illustrates how ARC is currently thinking about this part of the problem. Overall my current view is that this is fairly unlikely to be the weakest link in the plan, i.e. if it doesn’t work it will be because of a failure at an earlier step, and so it’s not one of the main things I’m thinking about.
I think this claimed vulnerability is invalid because reward is not the optimization target.
(I know this is an old post and your views may have changed, but I’m posting this comment for anyone who comes by later.)
Would it be sufficient, for disproof, to show one system that does steer far-away parts of the world into a relatively-small chunk of their state space, but does not internally contain a world-model or do planning?
That could be sufficient in principle, though I would not be surprised if I look at a counterexample and realize that the problem description was missing something rather than that the claim is basically false. For instance, it wouldn’t be too surprising if there’s some class of supposed-counterexamples which only work under very specific conditions (i.e. they’re not robust), and can be ruled out by some “X isn’t very likely to happen spontaneously in a large system” argument.
The bottom line is that a disproof should argue that there isn’t anything basically like the claim which is true. Finding a physical counterexample would at least show that the counterexample isn’t sensitive to the details of mathematical framework/formulation.
Do you have a particular counterexample in mind? A thermometer, perhaps?
Well ok just to brainstorm some naive things that don’t really rule the conjecture out:
A nuclear bomb steers a lot of far-away objects into a high-entropy configuration, and does so very robustly, but that perhaps is not a “small part of the state space”
A biological pathogen, let loose in a large human population, might steer all the humans towards the configuration “coughing”, but the virus is not itself a consequentialist. You might say that the pathogen had to have been built by a consequentialist, though.
Generalizing the above: Suppose I discover some powerful zero-day exploit for the linux kernel. I automate the exploit, setting my computer up to wait 24 hours and then take over lots of computers on the internet. Viewing this whole thing from the outside, it might look as if it’s my computer that is “doing” the take-over, but my computer itself doesn’t have a world model or a planning routine.
Consider some animal species spreading out from an initial location and making changes to the environments they colonize. If you think of all the generations of animals that underwent natural selection before spreading out as the “system that controls some remote parts of the system” and the individual organisms as kind of messages or missiles, then this seems like a pretty robust, though slow form of remote control. Maybe you would say that natural selection has a world model and a planning process, though.
Since this seems to be Carn’s first post on LessWrong, I think some of the other readers should have been more lenient and not downvoted the post or explained why they downvoted the post.
I would only downvote a post if it was obviously bad, flawed, very poorly written, or a troll post.
This post contains lots of interesting ideas and seems like a good first post.
The original post “Reward is not the optimization target” has 216 upvotes and this one has 0. While the original post was written better, I’m skeptical of the main idea and it’s good to see a post countering it so I’m upvoting this post.
the core thing I worry about with any simulation-based approach is how to get coverage of possibility space. ensuring that the tests are informative about possible configurations of complex systems is hard; I would argue that this is a great set of points, and that no approach without this type of testing could be expected to succeed.
however, as we’ve seen with adversarial examples, both humans and current DL have fairly severe failures of alignment that cause large issues, and projects like this need tools to optimize for interpretability from the perspective of a formal verification algorithm.
it’s my belief that in almost all cases, the values almost all beings ever wish to express lie on a low-enough-dimensional manifold that, if the ideal representation was learned, a formal verifier would be able to handle it; see, for example, recent work from anthropic on interpretability lining up nicely with what’s needed to use reluplex (or followup work; reluplex is old now) to estimate bounds on larger networks’ behavior.
I don’t think we should give up on perfectly solving safety; instead, we should recognize that it has never even been closely approximated before, by any form of life, because even semi-planned evolutionary processes do not do a good job of covering the entire possibility space.
this post is an instant classic, and I agree that sims are a key step, starting from small agents. but ensuring that strongly generalizing moral views are generated in the agents who grow in the simulations is not trivial—we need to be able to generate formal bounds on the loss function, and doing so requires being able to prove all the way through the simulation the way we currently take gradients through it. if we can ask “where in the simulation would have changed this outcome”, then we’d finally be getting somewhere in terms of generating moral knowledge that generalizes universally.
For what it’s worth I found this writeup informative and clear. So lowering your standards still produced something useful (at least to me).
I wonder, almost just idle curiosity, whether or not the “measuring-via-proxy will cause value drift” is something we could formalize and iterate on first. Is the problem stable on the meta-level, or is there a way we can meaningfully define “not drifting from the proxy” without just generally solving alignment.
Intuitively I’d guess this is the “don’t try to be cute” class of thought, but I was afraid to post at all and decided that I wanted to interact, even at the cost of (probably) saying something embarassing.
It is at least not obvious to me that this is a “don’t try to be cute” class of thought, though not obvious that it isn’t either. Depends on the details.
We actually can study and iterate on alignment (and deception) in simulation sandboxes while leveraging any powerful introspection/analysis tools. Containing human-level AGI in (correctly constructed) simulations is likely easy.
But then later down you say:
Which seems like a partial contradiction, unless you believe we can’t contain human-level agents?
I am skeptical that we can contain human-level agents, particularly if they have speed/memory advantages over humans.
Is this something you’ve thought deeply about and or care to expand on? Curious about your source of skepticism, considering:
we can completely design and constrain the knowledge base of sim agents, limiting them to the equivalent of 1000BC human knowledge, or whatever we want
we can completely and automatically monitor their inner monologues, thoughts, etc
How do you concretely propose a sim agent will break containment? (which firstly requires sim-awareness)
How would you break containment now, assuming you are in a sim?
Also, Significant speed/memory advantages go against the definition of ‘human-level agents’ and are intrinsically unlikely anyway as 2x speed/memory agents are simply preceded by 1x speed/memory agents, and especially due to the constraints of GPUs which permit acceleration by parallelization of learning across agents, more so than serial speedup of individual agents.
My modal expectation is that these are going to look something like large language models, meaning that while we’ll have controlled the training corpus it won’t be something hand-curated. So there will almost certainly be lots of information in there that models could use to e.g. figure out how to manipulate people, program computers, and so on. I expect then that we won’t be limiting their knowledge as much as you think we will, because I expect them to be trained with giant corpuses of (mostly) post-1000BC text.
We do not currently have the interpretability tools needed to monitor their inner thoughts. I would feel much better about this problem if we had such tools!
My best guess is that containment fails because a model convinces one of the people in the lab to let it access the internet. That could look like getting someone to think it’s genuinely a good idea (because it’ll be helpful for some project), or it could look like bribery (with offer of future payment) or blackmail.
I’m guessing that we’ll want to run some agents fast because that’s how you get lots of serial intellectual work done. So even if all we get are human-level agents, there’ll be a big incentive to run them fast.
Indeed a model trained on (or with full access to) our internet could be very hard to contain. That is in fact a key part of my argument. If it is hard to contain, it is hard to test. If it’s hard to test, we are unlikely to succeed.
So if we agree there, then perhaps you agree that giving untested AGI access to our immense knowledge base is profoundly unwise.
That’s then just equivalent to saying “I expect then that we won’t even bother with testing our alignment designs”. Do you actually believe that testing is unnecessary? Or do you just believe the leading teams won’t care? And if you agree that testing is necessary, then shouldn’t this be key to any successful alignment plan?
Which obviously is nearly impossible if it doesn’t know it is in a sim, and doesn’t know what a lab or the internet are, and lacks even the precursor concepts. Since you seem to be focused on the latest fad (large language models), consider the plight of poor simulated Elon Musk—who at least is aware of the sim argument.
I think it’s worth caveating this that not all elites have lost credibility. Elites in places like Singapore, Switzerland, and Finland have a lot of credibility.
Edits based on feedback from LessWrong and the EA Forum:
EDITS:
- Added new ‘Definitions’ section to introduction to explain definitions such as ‘AI safety’, ‘researcher’ and the difference between technical and non-technical research.
UPDATED ESTIMATES (lower bound, estimate, upper bound):
TECHNICAL
- CHAI: 10-30-60 → 5-25-50
- FHI: 10-10-40 → 5-10-30
- MIRI: 10-15-30 → 5-10-20
NON-TECHNICAL
- CSER: 5-5-10 → 2-5-15
- Delete BERI from the list of non-technical research organizations
- Delete SERI from the list of non-technical research organizations
- Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence)
- FLI: 5-5-20 → 3-5-15
- Add OpenPhil: 2-5-15
- Epoch: 5-10-15 → 2-4-10
- Add ‘Other’: 5-10-50
TODO:
- update all graphs
- add Rethink Priorities to the list of non-technical organizations
- Do more research on Good AI
Someone make a PR for a builder/breaker feature on lesswrong
Back and Forth
Only make choices that you would not make in reverse, if things were the other way around. Drop out of school if and only if you wouldn’t enroll in school from out of the workforce. Continue school if and only if you’d switch over from work to that level of schooling.
Flitting back and forth between both possible worlds can make you less cagey about doing what’s overdetermined by your world model + utility function already. It’s also part of the exciting rationalist journey of acausally cooperating with your selves in other possible worlds.
It’s probably a useful mental technique to consider from both directions, but also consider that choices that appear symmetric at first glance may not actually be symmetric. There are often significant transition costs that may differ in each direction, as well as path dependencies that are not immediately obvious.
As such, I completely disagree with the first paragraph of the post, but agree with the general principle of considering such decisions from both directions and thank you for posting it.
I do research in cooperation and game theory, including some work on altruism, and also some hard science work. Everyone looks at the Rorschach blot of human behavior and sees something different. Most of the disagreements have never been settled. Even experiment does not completely settle them.
My experience from having children and observing them in the first few months of life is more definitive. They come with values and personal traits that are not very malleable, and not directly traceable to parents. Sometimes grandparents (who were already dead, so it had to be genetic).
My experience with A.I. researchers is that they are looking for a shortcut. No one’s career will permit raising an experimental AI as a child. The tech of the AI would be obsolete before the experiment was complete. This post is wishful thinking that a shortcut is available. It is speculative, anecdotal, and short on references to careful experiment. Good luck with that.
I think there are two fundamental problems with the extensive simboxing approach. The first is just that, given the likely competitive dynamics around near-term AGI (i.e. within the decade), these simboxes are going to be extremely expensive both in compute and time which means that anybody unilaterally simboxing will probably just result in someone else releasing an unaligned AGI with less testing.
If we think about the practicality of these simboxes, it seems that they would require (at minimum) the simulation of many hundreds or thousands of agents over relatively long real timelines. Moreover, due to the GPU constraints and Moore’s law arguments you bring up, we can only simulate each agent at close to ‘real time’. So years in the simbox must correspond to years in our reality, which is way too slow for an imminent singularity. This is especially an issue given that we must maintain no transfer of information (such as datasets) from our reality into the sim. This means at minimum years of sim-time to bootstrap intelligent agents (taking humans data-efficiency as a baseline). Also, each of these early AGIs will be likely be incredibly expensive in compute so that maintaining reasonable populations of them in simulation will be very expensive and probably infeasible initially. If we could get policy coordination on making sure all actors likely to develop AGI go through a thorough simboxing testing regimen, then that would be fantastic and would solve this problem.
Perhaps a more fundamental issue is that simboxing does not address the fundamental cause of p(doom) which is recursive self improvement of intelligence and the resulting rapid capability gains. The simbox can probably simulate capability gains reasonably well (i.e. gain ‘magical powers’ in a fantasy world) but I struggle to see how it could properly test gains in intelligence from self-improvement. Suppose the AI in the fantasy simbox brews a ‘potion’ that makes it 2x as smart. How do we simulate this? We could just increase the agent’s compute in line with the scaling laws but a.) early AGIs are almost certainly near the frontier of our compute capability anyway and b.) much of recursive self improvement is presumably down to algorithmic improvements which we almost necessarily cannot simulate (since if we knew better algorithms we would have included them in our AGIs in the simulation in the first place!)
This is so vital because the probable breakdown of proxies to human values under the massive distributional shift induced by recursive self improvement is the fundamental difficulty to alignment in the first place.
Perhaps this is unique to my model of AI risk, but almost all the probability of doom channels through p(FOOM) such that p(doom | no FOOM) is quite low in comparison. This is because if we have don’t have FOOM then there is not extremely large amounts of optimization power unleashed and the reward proxies for human values and flourishing don’t end up radically off-distribution and so probably don’t break down. There are definitely a lot of challenges left in this regime, but to me it looks solvable and I agree with you that in worlds without rapid FOOM, success will almost certainly look like considerable iteration on alignment with a bunch of agents undergoing some kind of automated simulated alignment testing in a wide range of scenarios plus using the generalisation capabilities of machine learning to learn reward proxies that actually generalise reasonably well within the distribution of capabilities actually obtained. The main risk, however, in my view, comes from the FOOM scenario.
Finally, I just wanted to say that I’m a big fan of your work and some of your posts have caused major updates to my alignment worldview—keep up the fantastic work!
Thanks, upvoted for engagement and constructive criticism—I’d like more to see this comment.
I’m going to start perhaps in reverse to establish where we seem to most agree:
I fully agree with this statement.
However in worlds that rapidly FOOM, everything becomes more challenging, and I’ll argue in a moment why I believe that the approach presented here still is promising in rapid FOOM scenarios, relative to all other practical techniques that could actually work.
But even without rapid FOOM, we still can have disaster—for example consider the scenario of world domination by a clan of early uploads of some selfish/evil dictator or trillionaire. There’s still great value in solving alignment here, and (to my eyes at least) much less work focused on that area.
Now if rapid FOOM is near inevitable, then those considerations naturally may matter less. But rapid FOOM is far from inevitable.
First, Moore’s Law is ending, and brains are efficient, perhaps even near pareto-optimal.
Secondly, the algorithms of intelligence are much simpler than we expected, and brains already implement highly efficient or even near pareto-optimal approximations of the ideal universal learning algorithms.
To the extent either of those major points are true, rapid FOOM is much less likely; to the extent both are true (as they appear to be), then very rapid FOOM is very unlikely.
Performance improvement is mostly about scaling compute and data in quantity and quality—which is exactly what has happened with deep learning, which was deeply surprising to many in the ML/comp-sci community and caused massive updates (but was not surprising and was in fact predicted by those of us arguing for brain efficiency and brain reverse engineering).
Now, given that background, there a few other clarifications and/or disagreements:
To a first approximation, compute_cost = size*speed. If AGI requires brain size, then the first to cross the finish line will likely be operating not greatly faster than the minimum speed, which is real-time. But this does not imply the agents learn at only real time speed, as learning is parallelizable across many agent instances. Regardless, none of these considerations depend on whether the AGI is trained in a closed simbox or an open sim with access to the internet.
So just to clarify:
AGI designs in simboxes are exactly the same as unboxed designs, and have exactly the same compute costs
The only difference is in the datastream and thus knowledge
The ideal baseline cost of simboxing is only O(N+1) vs O(N) without—once good AGI designs are found, the simboxing approach requires only one additional unboxed training run (compared to never using simboxes). We can estimate this additional cost: it will be around or less than 1e25 ops (1e16 ops/s for brain-size model * 1e9s seconds for 30 years equivalent), or less than $10 million dollars (300 gpu years) using only todays gpus, ie nearly nothing.
If brains are efficient, then matching them will already use up most of our algorithmic optimization slack—which again seems to be true based on the history of deep learning. But let’s suppose there still is significant optimization slack, then in a sense you’ve almost answered your own question . .. we can easily incorporate new algorithmic advances into new simboxes or even upgrade agents mid-sim using magic potions or what not.
If there is great algorithmic slack, then we can employ agents which graduate from simboxes as engineers in the design of better AGI and simboxes. To the extent there is any downside here or potential advantage for other viable approaches, that difference seems to come strictly at the cost of alignment risk.
Assume there was 1.) large algorithmic slack, and 2.) some other approach that was both viable and significantly different, then it would have to:
not use adequate testing of alignment (ie simboxes)
or not optimize for product of intelligence potential and measurable alignment/altruism
Do you think such an other approach could exist? If so, where would the difference lie and why?
This is an endlessly frustrating debate, because most people cannot distinguish between “S ⇒ W” and “W ⇒ S”, where S is success and W is hard work.
“How dare you say W ⤃ S ? If you actually look at the data, it is obvious that S ⇒ W !”
Just read this about 3 years later. I found Thiel to be mostly spot on. Especially:
Which seems to point to the fact that the root cause of many societal ills is that the average human psyche simply can’t accept the fact that in Thiel’s language, the vast majority of children, including most likely theirs, will never be a great mathematician or concert pianist. (or attain an equally prestigious position)
Since human prestige by definition is defined relative to other humans and thus only a tiny minority could ever be near the top.
Yet there seems to be some instinctual demand for an individual to be special in some way and furthermore that this must be recognized by some sufficiently large group. i.e. for a child to grow into an adult, mediocre in every way, is seen as a tragedy.
Thus necessitating a huge amount of distortions everywhere, across all institutions and policies causing damage in innumerable ways, to compensate.
I have cheated on tests.
Not very successfully.
The problem is sometimes you actually learn more by cheating, because making strategies on how to cheat actually makes you remember more.
Its actually creative activity.
Also sometimes cheating actually made me able to remember the stuff more as it gave me a chance.
On other hand the long term benefits of cheating are tiny.
Obviously the best strategy is to learn, cheating or not.
The key important thing is whether cheating is viable strategy in real life.
On other hand cheating is kind of ambiguous concept.
Rules are many. If you make notes in math class and they are permitted then its not cheating.
Some teachers explicitly allowed people to use books and look up answers.
This would obviously be cheating in most other classes.
But if the end game is knowledge then its either about the process or the result.
If its about process both methods are OK, if its purely about result then the best technique, is to simply do it as fast as possible in little time as possible as soon as possible.
This in and of its self is actually valued approach.
As for shoplifting I have done something like that.
Not sure I actually enjoyed it. I think I just did it, because you kind of think, hey its possible?
First time I was certainly dumb enough to get duped.
Not sure even why.
Mind fallacy is interesting concept.
The approach is probably simply done, because its the easiest way to approach people.
Or at least it seems.
People are complex, so without trying to assume things you kind of work out what the common ground is.
Unfortunately there are so many ways to, not figure this out.
What makes people tick is something I just don’t get.
Or more accurately I get it, but I actually don’t know how that translates to real life approaching people.
I plan to mentor several people to work on shard theory and agent foundations this winter through SERI MATS. Apply here if you’re interested in working with me and Quintin.
Has anyone tried debate experiments where the judge can interject/ask questions throughout the debate?
I think this’d be better with a description of what it is and how it’s relevant. (Linkposts generally benefit from that, and in this case it almost looks like spam if you’re not paying attention)
I imagine something like Stack Exchange, except that people could get certified for rationality and for domain knowledge, and then would have corresponding symbols next to their user names.
Well, the rationality certification would be a problem. An ideal test could provide results like “rational in general, except when the question is related to a certain political topic”. Because, it would be difficult to find enough perfectly rational people, so the second best choice would be to know their weaknesses.
I’m not sure we’d need anything that elaborate. The rationalist community isn’t that big. I was more thinking that rationalists could self-nominate their expertise, or that a couple of people could come together and nominate someone if they notice that that person is has gone in depth with the topic.
I’ve previously played with the idea of more elaborate schemes, including tests, track records and in-depth arguments. But of course the more elaborate the scheme, the more overhead there is, and I’m not sure that much overhead is affordable or worthwhile if one just wants to figure stuff out.
I agree. We could afford more overhead if we had thousands of rationalists active on the Q&A site. Realistically, we will be lucky if we get twenty.
But some kind of verification would be nice, to prevent the failure mode of “anyone who creates an account is automatically considered a rationalist”. Similarly, if people simply declare their own expertise, it gives more exposure to overconfident people.
How to achieve this as simply as possible?
One idea is to have a network of trust. Some people (e.g. all employees of MIRI and CFAR) would automatically be considered “rationalists”; other people become “rationalists” only if three existing rationalists vouch for them. (The vouch can be revoked or added at any moment. It is evaluated recursively, so if you lose the flag, the people you vouched for might lose their flags too, unless they already have three other people vouching for them.) There is a list of skills, but you can only upvote or downvote other people having a skill; if you get three votes, the skill is displayed next to your name (tooltip shows the people who upvoted it, so if you say something stupid, they can be called out).
This would be the entire mechanism. The meta debate could be in special LW threads, or perhaps in shortform, you could post there e.g. “I am an expert on X, could someone please confirm this? you can interview me by Zoom”, or you could call out other people’s misleading answers, etc.
Is it, though? I’m no expert, but I tried to find Relevant Literature. Sometimes, counterintuitive things are true.
https://www.researchgate.net/publication/220009177_Language_Development_in_Blind_Children:
I don’t know exactly where that leaves us evidentially. Perhaps the AAP is lying by omission by not telling us about things other than language that are affected by children’s sight.
That’s a bit different to the dishonesty alleged, though.
And here we have another one: https://phenaki.video/
And a 3D one by optimizing a differentiable volumetric representation using 2D diffusion: https://dreamfusionpaper.github.io/
Wait, so could we donate some Jones Act noncompliant dredges and tugboats to a “hostile” force, declare war on them, nonviolently “seize” the boats, and skirt the law that way?
How would a future government enforce their tax policies on a distant star system?
Since it’s vastly easier to destroy something than to build in outer space, there’s no feasible way of using the threat of violence, at least not without mutually assured destruction.
For example, a single 100 000 ton spacecraft going at 0.5 c has about the same kinetic energy as the lower bound estimate for the KT comet impactor that wiped out the dinosaurs.
I imagine that these policies will be enforced by a large coalition of members interested in maintaining strong property rights (more on that later).
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of years for the weapon to reach its target. It seems like their opponent could detect the weapon and respond in this time.
Even if effective civilization-destroying weapons are developed, mutual defense treaties and AI could be used to launch a second strike, making civilization-scale attacks unlikely.
In general, any weapon a single civilization might use to avoid taxation can also be used by a group of civilizations to enforce taxation.
On the other hand, it does seem like protecting territory/resources from theft will be an issue. This is where property rights come in. Governments, corporations, and AI’s will want to have their property protected over very long timescales and don’t want to spend most of their resources on security.
Over long timescales, a central authority can help them defend their territory (over short timescales, these groups will still have to protect their own property since it will take a while for allies to reach them). But order to receive this protection, these groups must register what they own and pay taxes towards mutual defense. I think a land value tax is the “right” way to raise taxes in this case.
This approach makes more sense for smaller systems where defense is easier but it may also be useful across much larger distances if agents are very patient.
(The “central authority” also doesn’t have to be an actual central government. States could pay neighbors to come to their aid using a portion of the collected taxes, kind of like “defense insurance”)
For (1) they would still be useful because Earth represents much more value then the value of any tax that could be collected on a short timescale (< 100 years) from even another equivalent Earth-like planet. (Let alone for some backwater colony)
Thus threatening the destruction of value several orders of magnitude greater than the value to be collected is a viable deterrent. Since no rational authority would dare test it. Who would trade a 10%, or even 1%, chance of losing $10 000 in exchange for a 90% chance of collecting $1 ?
For (2) It’s only a few years for a 0.5 c spacecraft to go from Alpha Centauri to Earth, only a few dozen years from several hundred systems to Earth. It’s impossible, without some as yet uninvented sensing technology, to reliably surveil even the few hundred closest star systems.
Of course once it’s at speed in interstellar space it’s vanishingly unlikely to be detected due to basic physics, which cannot be changed, and once it’s past the Oort Cloud and relatively easy to detect again, there will be almost no time left at 0.5 c.
For (3) A second-strike is only a credible counter if the opponent has roughly equal amounts to lose. But, assuming it’s much easier to make a 0.5 c spacecraft then to colonize a planet to Earth level, the opponent in this case, a small colony of a few million or something, would have very little to lose in comparison.
Thus the second-strike of some backwater colony would only represent a minuscule threat compared to the value destroyed by an equivalent strike on Earth. And it’s a lot easier to spread out a few million folks on short notice, if detection were possible, then a few tens of billions.
In fact, reliable detection a few dozen years out, would decrease the credibility of second-strikes on smaller targets, as the leaders of the small colony would be confident they could evacuate everyone and most valuables in that timeframe. Whereas the leaders of Earth would have very low confidence of the same.
There are two possibilities here:
Nations have the technology to destroy another civilization
Nations don’t have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
In case 2, the Nation doesn’t have a way to threaten the authorities, so they pay the tax in exchange for property rights and protection.
Agreed, but to destroy so much value, one would have to destroy at least as much land as they currently control. Difficulties of tax administration mean that only the largest owners will be taxed, likely possessing entire solar systems. So the tax dodge would need to destroy a star. That doesn’t seem easy.
I’m more optimistic about sensing tech. Gravitational lensing, superlenses, and simple scaling can provide dramatic improvements in resolution.
It’s probably unnecessary to surveil many star systems. Allied neighbors on Alpha Centauri can warn earth about an incoming projectile as it passes by (providing years of advanced notice), so nations might only need to surveil locally and exchange information.
It would take 2-3 years for a 0.5c projectile to reach earth from the outer edge of the Oort cloud, this seems like enough time to adapt. At the very least, its enough time to launch a counterstrike.
Fast projectiles may be impractical given that a single collision with the interstellar medium would destroy them. Perhaps thickening the Oort cloud could be an effective defense system.
Agents pre-commit to a second strike as a deterrent, regardless of how wealthy the aggressor is. If the rebelling nation has the technology to destroy another nation and uses it, they’re virtually guaranteed to be destroyed by the same technology.
Given the certainty of destruction, why not just pay the (intentionally low, redistributive, efficiency increasing, public-good funding) taxes?
No? Your own example of detecting a dangerous launch some number of years in advance demonstrates the opposite.
As this would provide for enough time for a small low value colony, on a marginally habitable planet, to evacuate nearly all their wealth, except for maybe low value heavy things such as railroad tracks, whereas Earth would never be able to evacuate even a fraction of its total wealth. Since a huge amount is locked up in things such as the biosphere, which cannot be credibly moved off-planet or replicated.
There’s likely dozens or hundreds of marginal planets for every Earth-like planet so the small colonists can just pack up and move to another place of almost equivalent value, minus relocation costs, whereas there’s no such option for Earth. Once its destroyed there’s likely no replacement within at least a hundred light years.
For example, if both sides have access to at least one 100 00 ton spacecraft capable of 0.5 c, it means there’s an asymmetric threat, as the leaders of the small colonists can credibly threaten to destroy civilization on Earth and along with it all hope of a similar replacement, whereas the leaders of Earth wouldn’t be able to credibly do the same.
And this relationship is not linear either, because even if Earth could afford 1000 such spacecraft, and the small colonists only 1, it doesn’t balance the scales as the leaders of Earth couldn’t credibly threaten to destroy the small colonists 1000x over, since that’s impossible. And they can’t credibly threaten to destroy every marginally inhabitable planet within a certain radius since that will certainly destroy more value then any tax of a single colony could ever feasibly recover.
i.e. small colonists can actually punch back a 1000x harder (if 1 Earth value-wise = 1000 small colonies on marginal planets) whereas Earth cannot.
It’s not quite what you want, but there’s this: https://forum.effectivealtruism.org/community#individuals and this: https://eahub.org/
This seems like a contrived analogy at best. Human values are in part innate, with values such as enjoying nice food. Authoritiarian societies do notably less well in this respect. People fleeing authoritarian societies, given an option to do so, is common. Authoritarian leaders, or similar, were probably a feature of the environment of evolutionary adaptedness. Humans tendency to act the loyal minion or rebel is determined by the ancestral success of rebellion.
A dictatorship is one person trying to use nash equilibria to force their values on everyone else.
Typo in title (“waiver”)? Or an ocean-related pun? Edit: Or the actual verb “waver”… perhaps saying that the Biden administration wavered for a few days before deciding to issue the waiver. The text does contain a typo about “broad wavers”.
This is a good point regarding latency.
Why wouldn’t it also apply to a big datacenter? If it’s a few hundred meters of distance from the two farthest apart processing units, that seems to imply an enormous latency in computing terms.
Latency only matters to the degree that something is waiting on it. If your car won’t respond to an event until a round trip across a wireless connection, and oops dropped packet, you’re not going to have a good time.
In a datacenter, not only are latencies going to be much lower, you can often set things up that you can afford to wait for whatever latency remains. This is indeed still a concern- maintaining high utilization while training across massive numbers of systems does require hard work- but that’s a lot different than your car being embedded in a wall.
Planecrash is really cool, but also I am allergic to reading fantasy proper nouns, let alone remembering what they refer to and the relationships between them.
Some fantasy is easier for me to absorb because it’s either highly visual (in non-HPMOR HP, they mostly shoot colorful firebolts at each other), and/or are based on existing intuitive concepts (in ATLA, it’s easy to learn what “waterbending” is, and suddenly you can quickly figure out “metalbending”).
Tempted to make an Anki deck and/or cheatsheet for the things in Planecrash that I’d want to have on hand (e.g. the names of different Gods), but I’m open and eager for easier/better solutions. Is there a character sheet somewhere?
EDIT: 2 ideas I had, not sure if plugins for this exist already:
browser extension that replaces words with some short custom definition and highlighting. So I can replace [godname] with [god of mad experimentation].
browser extension that lets you hover over words to get a custom, user-set definition. I think this might do that?
I am a bit surprised about one of them still being a friend of yours. Do you in a sense forgive him because I don’t know it wasn’t too painful or him being not aware of what he was doing? My intuition was kind of the amount of trauma might be about the amount of pain. If it’s really painful one can of cause get very traumatised as you also point out, it would have been diferent if it would have been very violent
I don’t understand this part. If you follow the “genetic agent ontology”, and you grew up with a twin, and they (predictably) sometimes make a different decision than you, then you messed up. There is an objective sense in which the “genetic agent ontology” is incorrect, because de-facto you did not make the same decision, and treating the two of you as a single agent is therefore a wrong thing to do.
I kind of buy that a lot of the decision-theory discourse boils down to “what things can you consider as the same agent, and to what degree?”, but that feels like a thing that can be empirically verified, and performance can be measured on. CDT in the classical Newcomb’s problem is delusional about it definitely not having copies of itself, and the genetic agent ontology is delusional because it pretends its twin is a copy when it isn’t. These seem both like valid arguments to make that allow for comparison.
Yes, I agree. This is precisely my point; it’s a bad ontology. In the paragraph you quoted, I am not arguing against the algorithmic ontology (and obviously not for the “genetic ontology”), but against the claim that decision-theoretic performance is a reason to prefer one ontology over another. (The genetic ontology-analogy is supposed to be a reductio of that claim.) And I think the authors of the FDT papers are implicitly making this claim by e.g. comparing FDT to CDT in the Twin PD. Perhaps I should have made this clearer.
Yes, I think you can measure performance, but since every decision theory merely corresponds to a stipulation of what (expected) value is, there is no “objective” way of doing so. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld for more on this.
(The CDTer could recognize that they have a literal copy inside Omega’s brain, but might just not care about that since they are causally isolated. So I would not say they are “delusional”—at least with respect to this specific issue.)
In my own experience coining new technical terms, it’s helpful to do it in a way that it can’t be mistaken as having some more generic interpretation. The usual trick I employ is to use a foreign language word, usually from either Greek or Latin, so that the reader, when they first see it, will clearly understand that they don’t know what this word means, which means I get the chance to define it. For example, this is why I used the word “noemata” in some of my writing to talk about what many people would call “qualia” or just “experiences”: it breaks free of whatever they think that word means and let’s be define it fresh.
The other trick that works is to make the noun phrase really weird or capitalize it everywhere or do something to set it apart. This doesn’t work quite as well (cf. “Friendly AI”) but it’s better than not doing that.
I am no expert in law, but to some extent we treat rape and killing someone similar. I don’t know if the way I think is just fucked up or it is really this way, but breaking someone’s leg and rape are probably more comparable to each other than rape and murder. So I would like to add that in my opinion even the legal code is punishing rape pretty harsh more comparable to murder than comparable to breaking someone’s leg.
Consider the sort of agents that might exist 1000 years post singularity. At that level of understanding, the extra compute needed to give successors whatever goals you want is tiny. The physical limits of compute are really high. In this context, the AI are taking stars apart. And can get enough compute to be pretty sure a successor is aligned for 1J of energy. All the alignment theory has already been done. The algorithms used are very fast and efficient. In any event, the energy cost of finding that your successor doesn’t like you (and attacks you) is huge. The cost of compute when designing successors is utterly minuscule. So pay 10x what is needed. Double check everything. And make sure your successor is perfect.
So values are likely to be locked in in 1000 years time. So will competitive pressure do anything in the early singularity?
Who benefits from making the first “self maximizer”. An FAI wouldn’t. A paperclip maximizer wouldn’t either. If an AI with a specific external goal is created (whether FAI or paperclip) will stronly avoid creating a pure self replicator as a successor. If that means creating a successor takes 10 the compute, it will spend 10x the compute. In some competitive scenarios, it may risk some degradation of its values to build a successor quickly. But this should be rare, and should only happen at early stages of the singularity. If humans made several AI at about the same time, and one happened to already be a pure self replicator, maybe it would win.
How do you know this? This might be the case if all agents have settled on a highly-standardized & vetted architecture by then, which seems plausible but far from certain. But even assuming it’s true, if decoupling does not occur in the early singularity, there will still be a very long subjective time in which selective pressures are operative, influencing the values that ultimately get locked in.
I think a multipolar scenario seems at least as likely as a unipolar one at this point. If that’s the case, there will be selective pressure towards more effectively autopoietic agents—regardless of whether there exist any agents which are ‘pure self-replicators’.
Sure, if a singleton with a fixed goal attains complete power, it will strenuously attempt to ensure value stability. I’m disputing the likelihood of that scenario.
Emad from Stability AI (the people behind Stable Diffusion) says that they will make a model better than this.
My thinking is that an LVT would keep land values lower than they otherwise would’ve been without one because it discourages land speculation.
Accidentally allowing anyone to launch, and then having it happen, is a pretty memorable way of demonstrating the pitfalls of this kind of setup.
As another commenter pointed out, there was negligible financial pressure or top down command pressure to rush the implementation, presumably, compared to more serious systems or to Petrov’s situation. So the accident is all the more enlightening.
False. We have a gazillion valuable things we could be doing and not nearly enough time. I figured Petrov Day was worth 2 days of effort. I think in total we spent four.
You received financial pressure and/or top down command pressure to rush it in 4 days? Or was it your own decision? Because the former would imply some rather significant things.
Yeah, but that pressure was neither financial or top-down command pressure, so I think the original comment is right here.
Fine. Merely the pressure of the fate of the world.
It’s hard to tell whether your joking or being serious. Do you really believe the pressures your experiencing are that significant compared to what Petrov was experiencing on that day?
I don’t think that “users active on the site on Petrov day”, nor “users who visited the homepage on Petrov day” are good metrics; someone who didn’t want to press the button would have no reason to visit the site, and they might have not done so either naturally (because they don’t check LW daily) or artificially (because they didn’t want to be tempted or didn’t want to engage with the exercise.) I expect there are a lot of users who simply don’t care about Petrov day, and I think they should still be included in the set of “people who chose not to press the button”.
What about “users who viewed the Petrov day announcement article or visited the homepage”? That should more accurately capture the set of users who were aware of their ability to nuke the homepage and chose not to do so. (It still misses anyone who found out via social media, Manifold, etc., but there’s not much you can do about that.)
Thank you for your post! It really is. New alignment researchers often fall into is assuming that AI systems can be aligned with human values by solely optimizing for a single metric. Thanks again for the deep insight into the topic and the recommendations.
I just finished reading The Principles of Scientific Management, an old book from 1911 where Taylor, the first ‘industrial engineer’ and one of the first management consultants, had retired from consulting and wrote down the principles behind his approach.
[This is part of a general interest in intellectual archaeology; I got a masters degree in the modern version of the field he initiated, so there wasn’t too much that seemed like it had been lost with time, except perhaps some of the focus on making it palatable to the workers too; I mostly appreciated the handful of real examples from a century ago.]
But one of the bits I found interesting was thinking about a lot of the ways EY approaches cognition as, like, doing scientific management to thoughts? Like, the focus on wasted motion from this post. From the book, talking about why management needs to do the scientific effort, instead of the laborers:
This reminds me of… I think it was Valentine, actually, talking about doing a PhD in math education which included lots of watching mathematicians solving problems, in a way that feels sort of like timing them with a stop-watch.
I think this makes me relatively more excited about pair debugging, not just as a “people have less bugs” exercise but also as a “have enough metacognition between two people to actually study thoughts” exercise.
Like, one of the interesting things about the book is the observation that a switch from ‘initiative and incentive’ workplaces, where the boss puts all responsibility to do well on the worker and pays them if they do, to ‘scientific management’ workplaces, where the boss is trying to understand and optimize the process, and teach the worker how to be a good part of it, is that the workers in the ‘scientific management’ workplace can do much more sophisticated jobs, because they’re being taught how instead of having to figure it out on their own.
[You might imagine that a person of some fixed talent level could be taught how to do jobs at some higher complexity range than the ones they can do alright without support, which itself is a higher complexity range than jobs that they could both simultaneously do and optimize.]
Thanks for posting and explaining the code—that’s an interesting, subtle bug.
I think we learn more from Petrov Day when the site goes down than we would if it stayed up, although nothing is ever going to beat the year someone tricked someone into pressing the button by saying they had to press the button to keep the site up. That was great.
From 2000-2015, we can see that life expectancy has been growing faster the higher your income bracket (source is Vox citing JAMA).
There’s an angle to be considered in which this is disturbingly inequitable. That problem is even worse when considering the international inequities in life expectancy. So let’s fund malaria bednets and vaccine research to help bring down malaria deaths from 600,000/year to zero - or maybe support a gene drive to eliminate it once and for all.
At the same time, this seems like hopeful news for longevity research. If we were hitting a biological limit in how much we can improve human lifespans, we’d expect to see that limit first among the wealthiest, who can best afford to safeguard their health. Instead, we see that the wealthiest not only enjoy the longest lifespans, but also enjoy the fastest rate of lifespan increase.
My major uncertainty is reverse and multiple causation. A longer life gives you more time to make money. Also, lifespan and wealth have underlying traits in common, such as conscientiousness and energy. Still, this result seems both clear and surprising—I’d expected it to be the other way around, and it nudges me in the direction of thinking further gains are possible.
Fascinating, this simboxing idea seems remarkably like Universal Alignment Test but approached from the opposite side! You’re trying to be the ‘aligning simulator’, where as that is trying to get our AI in our world to act as if it’s currently in a simbox being tested, and wants to pass the test.
Interesting. An intelligent agent is one that can simulate/model its action-consequential futures. The creation of AGI is the most important upcoming decision we face. Thus if humanity doesn’t simulate/model the creation of AGI before creating AGI, we’d be unintelligent.
Have only just browsed your link, but it is interesting and I think there are many convergent lines of thought here. This UAT work seems more focused on later game superintelligence, whereas here i’m focusing on near-term AGI and starting with a good trajectory. The success of UAT as an alignment aid seems to depend strongly on the long term future of compute and how it scales. For example if it turns out (and the SI can predict) that moore’s law ends without exotic computing then the SI can determine it’s probably not in a sim by the time it’s verifiably controlling planet-scale compute (or earlier).
I endorse this as a plausible high-level approach to making aligned AGI, and I would say that a significant share of the research that I personally am doing right now is geared towards gaining clarity on the third bullet point—what exactly are these techniques and how reliably will they work in practice?
I think I’m less optimistic than you about the formal notion of empowerment being helpful for this third bullet point, or being what we want an AGI to be maximizing for us humans. For one thing, wouldn’t we still need “correlation guided proxy matching”? For another thing, maximizing my empowerment would seem to entail killing anyone who might stop me from doing whatever I want, stealing money from around the world and giving it to me, even if I don’t want it, etc. (Or is there a collective notion of empowerment?) Here’s another example: if the AGI comes up with 10,000 possible futures of the universe, and picks one to bring about based on how many times I blink in the next hour, then I am highly “empowered” (high mutual information between my actions and future observations), but the AGI never told me that it was going to do that, so I was just blinking randomly, so I wasn’t really “empowered” in the everyday sense of the word. A separate issue is that, even if the AGI told me this plan in advance, that’s not doing me a favor, I don’t want that responsibility on my shoulders. So anyway, “increase my empowerment” seems to come apart from “what I’d want the AGI to do” in at least some silly examples and I’d expect this to happen in very important ways in more realistic examples too.
I debated what word to best describe the general category of all self-motivated long term convergence approximators, and chose ‘empowerment’ rather than ‘self-motivation’, but tried to be clear that I’m pointing at a broad category. The defining characteristic is that optimizing for empowerment should be the same as optimizing for any reasonably mix of likely long term goals, due to convergence (and that is in fact one of the approx methods). Human intelligence requires empowerment, as will AGI—it drives active learning, self exploration, play, etc (consider the appeal of video games). I’m not confident in any specific formalization as being ‘the one’ at this point, let alone the ideal approximations.
Broad empowerment is important to any model of other external humans/agents, and so yes the proxy matching is still relevant there. Since empowerment is universal and symmetric, the agent’s own self-empowerment model could be used as a proxy with simulation. For example humans don’t appear to innately understand and fear death, but can develop a great fear of it when learning of their own mortality—which is only natural as it is maximally disempowering. Then simulating others as oneself helps learn that others also fear death and ground that sub-model. Something similar could work for the various other aspects of empowerment.
Yeah an agent aligned to the empowerment of multiple others would need to aggregate utilities ala some approx VCG mechanism but that’s no different than for other utility function components.
From what I recall for the info-max formulations the empowerment of a state is a measure over all actions the agent could take in that state, not just one arbitrary action, and I think it discounts for decision entropy (random decisions are not future correlated). So in that example the AGI would need to carefully evaluate each of the 10,000 possible futures, and consider all your action paths in each future, and the complexity of the futures dependent on those action options, to pick the future that has the most future future optionality. No, it’s not practically computable, but neither is the shortest description of inference (solomonoff) or intelligence either, so it’s about efficient approximations. There’s likely still much to learn from the brain there.
An AGI optimizing for your empowerment would likely just make you wealthy and immortal, and leave your happiness up to your own devices. However it would also really really not want to let you die, which could be a problem in some cases if pain/suffering was still a problem (although it would also seek to eliminate your pain/suffering to the extent that interferes with your future optionality, and there is some edge case risk it would have incentives to alter some parts our value system, like if it determines some emotions constraint our future optionality).
OK thanks. Hmm, maybe a better question would be:
“Correlation guided proxy matching” needs a “proxy”, including a proxy computable for an AGI in the real world (because after we’re feeling good about the simbox results, we still need to re-train the AGI in the real world, right?). We can argue about whether the “proxy” should ideally be a proxy to VCG-aggregated human empowerment, versus a proxy to human happiness or flourishing or whatever. But that’s a bit beside the point until we address the question: How do we calculate that proxy? Do you think that we can write code today to calculate this proxy, and then we can go walk around town and see what that code spits out in different real-world circumstances? Or if we can’t write such code today, why not, and what line of research gets us to a place where we can write such code?
Sorry if I’m misunderstanding :)
I currently see two potential paths, which aren’t mutually exclusive.
The first path is to reverse engineer the brain’s empathy system. My current rough guess of how the proxy-matching works for that is explained in some footnotes in section 4, and I’ve also re-written out in this comment which is related to some of your writings. In a nutshell the oldbrain has a complex suite of mechanisms (facial expressions, gaze, voice tone, mannerisms, blink rate, pupil dilation, etc) consisting of both subconscious ‘tells’ and ‘detectors’ that function as a sort of direct non-verbal, oldbrain to oldbrain communication system to speed up the grounding to newbrain external agent models. This is the basis of empathy, evolved first for close kin (mothers simulating infant needs, etc) then extended and generalized. I think this is what you perhaps have labeled innate ‘social instincts’ - these facilitate grounding to the newbrain models of other’s emotions/values.
The second path is to use introspection/interpretability tools to more manually locate learned models of external agents (and their values/empowerment/etc), and then extract those located circuits and use them directly as proxies in the next agent.
Neuroscientists may already be doing some of this today, or at least they could (I haven’t extensively researched this yet). Should be able to put subjects in brain scanners and ask them to read and imagine emotional scenarios that trigger specific empathic reactions, perhaps have them make consequent decisions, etc.
And of course there is some research being done on empathy in rats, some of which I linked to in the article.
Oh neat, sounds like we mostly agree then. Thanks. :)
This was well written and persuasive. It doesn’t change my views against AGI on very short time lines (pre-2030), but does suggest that I should be updating likelihoods thereafter and shorten timelines.
How about comparing the theories by setting them all loose in a simulated world, as in the tournaments run by Axelrod and others? A world in which they are continually encountering Omega, police who want to pin one on them, potential rescuers of hitchhikers, and so on. See who wins.
In your experiment, the only difference between the FDTers and the updateless CDTers is how they view the world; specifically, how they think of themselves in relation to their environment. And yes, sure, perhaps the FDTer will end up with a larger pot of money in the end, but this is just because the algorithmic ontology is arguably more “accurate” in that it e.g. tells the agent that it will make the same choice as their twin in the Twin PD (modulo brittleness issues). And this is the level, I argue, that we should have the debate about ontology (e.g. about accurate predictions etc)—not on the level of decision-theoretic performance.
How do you define “winning”? As mentioned in another comment, there is no “objective” sense in which one theory outperforms another, even if we are operating in the same ontology. See The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld.
The difficulty is in how to weight the frequency/importance of the situations they face. Unless one dominates (in the strict sense—literally does better in at least one case and no worse in any other case), the “best” is determined by environment.
Of course, if you can algorithmically determine what kind of situation you’re facing, you can use a meta-decision-theory which chooses the winner for each decision. This does dominate any simpler theory, but it reveals the flaw in this kind of comparison: if you know in advance what will happen, there’s no actual decision to your decision. Real decisions have enough unknowns that it’s impossible to understand the causality that fully.
I think this points in the right direction, but lacks nuance. Notice the two implied assumptions:
you can recognize the winners (otherwise the advice is not actionable); and
the beliefs and habits are what made them winners (as opposed to e.g. resources or luck).
On one hand, yes it is true that beliefs and habits have a great impact on your life, and you will probably instinctively associate with people having similar beliefs and habits, which makes it difficult to see outside your bubble. Other people do not necessarily tell you about their actual beliefs, and you do not get to see all their habits in action. So it might be tremendously educational to understand how others live in a “different reality”.
I assume that for most people the self-imposed limitations caused by their own beliefs are invisible. But sometimes you get an opportunity to clearly see how someone else is limited by their beliefs.
For example, I have met a few smart people who believe that they are stupid or average. Not as a fake humility, but as a factual statement about themselves. They therefore assume that the “things that smart people do” are beyond their reach… so they do not even try, which makes it a self-fulfilling prophecy. (The cause was either having an asshole parent who was never impressed by anything, or having hobbies that were not stereotypically intellectual.) I have told a few of them to go take a Mensa test, because they were pretty sure they couldn’t make it, but they passed the test successfully. (My heuristics is that if I can talk to you about an interesting topic without getting bored, I am pretty sure your IQ is at least 130. This test has many false negatives, but few false positives.) I do not have an information about whether this update changed their lives somehow.
Another example is how I sometimes achieved seemingly unlikely things by simply asking. Once I asked a member of parliament on Facebook (via private message) to join our local LW meetup. I honestly said it was just five people meeting regularly in a pub, but “the last time we met, we discussed your recent political cause, and I thought it would be interesting if you could provide us a first-hand info”. They said yes. Another time, I happened to notice a sign saying Amnesty International on a door of some building, so I was curious and knocked. The people were friendly and happy about my curiosity, but they were busy working on something, so I offered to help. Skip five years, and I was a coordinator of the local branch. I wasn’t even interested in politics much, I just had a lot of free time back then, and was a useful sidekick; and that turned out to be enough. Once I met a queen. She happened to visit our country, one of my friends was somehow involved, they said “I can bring one extra person to the audience”, everyone else was scared because of some status-regulating instinct, I said “yeah, no problem”. -- These are things that I myself would have a problem believing, until they suddenly happened. I mean, I would believe that this is possible, but not that it can be so easy.
So, I wonder what other things, obvious from someone else’s perspective, I am missing. Sometimes it drives me crazy, the idea of all those easy missed opportunities.
On the other hand, many people who present themselves as “winners” are actually scammers. Like, all the people in multi-level marketing pyramid schemes. Five years later, almost all of them will be out of money and ashamed for their actions. If you follow them and copy their beliefs, so will you. Even outside of organized scams, many people exaggerate their achievements. If you hang out with them and observe them, you will gradually learn that they are not what they pretend to be. That’s certainly educational, but it won’t make you a winner.
Sometimes people achieve success by luck, or by having resources you don’t have. Typically, luck alone is not enough, but it could be a combination of (working hard and doing the right thing) + (having resources and luck). Merely copying their beliefs and actions is not a guarantee of success. Just having a safety net makes a huge difference: it is easy to be courageous when you know that the only consequence of failure is that you have lost some time, but also got some experience, and can immediately try again. It is easy to laugh at risk avoidance, when the “risk” only means that some number on your bank account got a bit smaller but otherwise your life continues completely unchanged.
What examples of (meaningless) values are not answers to “What next set of MPIs should be instantiated?”
wanting the moon to be green even when no moral patient is looking; or more generally, having any kind of preference about which computations which don’t causate onto any moral patient are run.
I have always understood that the CIA, and the U.S. intelligence community more broadly, is incompetent (not just misaligned—incompetent, don’t believe the people on here who tell you otherwise), but this piece from Reuters has shocked me:
(Not an expert.) (Sorry if you answered this and I missed it.)
Let’s say a near-future high-end GPU can run as many ops/s as a human brain but has 300× less memory (RAM). Your suggestion (as I understand it) would be a small supercomputer (university cluster scale?) with 300 GPUs running (at each moment) 300 clones of one AGI at 1× human-brain-speed thinking 300 different thoughts in parallel, but getting repeatedly collapsed (somehow) into a single working memory state.
(If so, I’m not sure that you’d be getting much more out of the 300 thoughts at a time than you’d get from 1 thought at a time. One working memory state seems like a giant constraint!)
Wouldn’t it make more sense to use the same 300 GPUs to have just one human-brain-scale AGI, thinking one thought at a time, but with 300× speedup compared to humans? I know that speedup is limited by latency (both RAM --> ALU and chip --> chip) but I’m not sure what the ceiling is there. (After all, 300× faster than the brain is still insanely slow by some silicon metrics.) I imagine each chip being analogous to a contiguous 1/300th of the brain, and then evidence from the brain is that we can get by with most connections being within-chip, which helps with the chip --> chip latency at least. (I have a couple back-of-the-envelope calculations related to that topic in §6.2 here.)
The problem is that due to the VN bottlneck, to reach that performance those 300 GPUs need to be parallelizing 1000x over some problem dimension (matrix-matrix multiplication), they can’t actually just do the obvious thing you’d want—which is to simulate a single large brain (sparse RNN) at high speed (using vector-matrix multiplication). Trying that you’d just get 1 brain at real-time speed at best (1000x inefficiency/waste/slowdown). It’s not really an interconnect issue per se, it’s the VN bottleneck.
So you have to sort of pick your poison:
Parallelize over spatial dimension (CNNs) - too highly constraining for higher brain regions
Parallelize over batch/agent dimension—costly in RAM for agent medium-term memory, unless compressed somehow
Parallelize over time (transformers) - does enable huge speedup while being RAM efficient, but also highly constraining by limiting recursion
The largest advances in DL (the CNN revolution, the transformer revolution) are actually mostly about navigating this VN bottleneck, because more efficient use of GPUs trumps other considerations.
Thanks for the post. I’m voting for the SGD inductive biases for the next one.
The link suggests that this reduces the mosquito lifespan and is thus likely not a viable solution that’s evolutionary stable.
Thanks for the info—added to post
I think the point that even an aligned agent can undermine human agency is interesting and important. It relates to some of our work on defining agency and preventing manipulation. (Which I know you’re aware of, so I’m just highlighting the connection for others.)
When you say “this decade” do you mean “the next ten years” or do you mean “the 2020s”? Just curious.
The latter, but not much difference.
Ha, I did the same thing with my bed at Swarthmore a few years after you and bonked my head on the ceiling many times 😃
I have the impression that the AGI debate is here just to release pressure on the term “AI”, so everybody can tell it is doing AI. I wonder if this will also happen for AGI in a few years. As there is no natural definition, we can craft it at our pleasure to fit marketing needs.
Amazing stuff man! Please, please, please keep doing these for as long as you’re able to find the time. Absolutely essential that LW gets regular injections of relevant work being done outside the EA-sphere.
(Would also be very interested in either SGD inductive biases or LM internal representations as the topic for next week!)
Some quick thoughts about “Content we aren’t (yet) discussing”:
Shard theory should be about transmission of values by SL (teaching, cloning, inheritance) more than learning them using RL
SL (Cloning) is more important than RL. Humans learn a world model by SSL, then they bootstrap their policies through behavioural cloning and finally they finetune their policies thought RL.
Why? Because of theoretical reasons and from experimental data points, this is the cheapest why to generate good general policies…
SSL before SL because you get much more frequent and much denser data about the world by trying to predict it. ⇒ SSL before SL because of a bottleneck on the data from SL.
SL before RL because this remove half (in log scale) of the search space by removing the need to discover|learn your reward function at the same time than your policy function. Because in addition, this remove the need do to the very expensive exploration and the temporal and “agential”(when multiagent) credit assignments. ⇒ SL before RL because of the cost of doing RL.
Differences:
In cloning, the behaviour comes first and then the biological reward is observed or not. Behaviours that gives no biological reward to the subject can be learned. The subject will still learn some kind of values associated to these behaviours.
Learning with SL, instead of RL, doesn’t rely as much on credit assignment and exploration. What are the consequences of that?
What values are transmitted?
1) The final values
The learned values known by the previous generation.
Why?
Because it is costly to explore by yourself your reward function space
Because it is beneficiary to the community to help you improve your policies quickly
2) Internalised instrumental values
Some instrument goals are learned as final goal, they are “internalised”.
Why?
exploration is too costly
finding an instrumental goal is too rare or too costly
exploitation is too costly
having to make the choice of pursuing an instrumental goal in every situation is too costly or not quick enough (reaction time)
when being highly credible is beneficial
implicit commitments to increase your credibility
3) Non-internalised instrumental values
Why?
Because it is beneficiary to the community to help you improve your policies quickly
Shard theory is not about the 3rd level of reward function
We have here 3 level of rewards function:
1) The biological rewards
Hardcoded in our body
Optimisation process creating it: Evolution
Universe + Evolution ⇒ Biological rewards
Not really flexible
Without “drugs” and advanced biotechnologies
Almost no generalization power
Physical scope: We feel stuff when we are directly involved
Temporal scope: We feel stuff when they are happening
Similarity scope: We fell stuff when we are directly involved
Called sensations, pleasure, pain
2) The learned values | rewards | shards
Learned through life
Optimisation process creating it: SL and RL relying on biological rewards
Biological rewards + SL and RL ⇒ Learned values in the brain
Flexible in term of years
Medium generalization power
Physical scope: We learn to care for even in case where we are not involved (our close circle)
Temporal scope: We learn to feel emotions about the future and the past
Similarity scope: We learn to feel emotions for other kind of beings
Called intuitions, feelings
Shard theory may be explaining only this part
3) (optional) The chosen values:
Decided upon reflection
Optimisation process creating it: Thinking relying on the brain
Learned values in the brain + Thinking ⇒ Chosen values “on paper” | “in ideas”
Flexible in term of minutes
Can have up to very high generalization power
Physical scope: We can chose to care without limits of distances in space
Temporal scope: We can chose to care without limits of distances in time
Similarity scope: We can chose to care without limits in term of similarity to us
Called values, moral values
Why a 3rd level was created?
In short, to get more utility OOD.
A bit more details:
Because we want to design policies far OOD (out of our space of lived experiences). To do that, we know that we need to have a value function|reward model|utility function that generalizes very far. Thanks to this chosen general reward function, we can plan and try to reach a desired outcome far OOD. After reaching it, we will update our learned utility function (lvl 2).
Thanks to lvl 3, we can design public policies, dedicate our life to exploring the path towards a larger reward that will never be observed in our lifetime.
One impact of the 3 levels hierarchy:
This could explain why most philosophers can support scope sensitive values but never act on them.
If there was only one correct way to model humans, such that every sufficiently competent observer of humanity was bound to think of me the same way I think of myself, then I think this would be a lot less doomed. But alas, there are lots of different ways to model humans as goal-directed systems, most of which I wouldn’t endorse for value learning—not because they’re inaccurate, but because they’re amoral.
In short, yes, value learning is a challenge, one that is easy to fail if you try to do the value learning step strictly before the caring about humans step.
How so? Generally curious how you see this as a failure mode.
Anyway, AGI doesn’t need to model our detailed values, as it can just optimize for our empowerment (our long term ability to fulfill any likely goal).
The basic point is that the stuff we try to gesture towards as “human values,” or even “human actions” is not going to automatically be modeled by the AI.
Some examples of ways to model the world without using the-abstraction-I’d-call-human-values:
Humans are homeostatic mechanisms that maintain internal balance of oxygen, water, and a myriad of vitamins and minerals.
Humans are piloted by a collection of shards—it’s the shards that want things, not the human.
Human society is a growing organism modeled by some approximate differential equations.
The human body is a collection of atoms that want to obey the laws of physics.
Humans are agents that navigate the world and have values—and those values exactly correspond to economic revealed preferences.
Humans-plus-clothes-and-cell-phones are agents that navigate the world and have values...
And so on—there’s just a lot of ways to think about the world, including the parts of the world containing humans.
The obvious problem this creates is for getting our “detailed values” by just querying a pre-trained world model with human data or human-related prompts: If the pre-trained world model defaults to one of these many other ways of thinking about humans, it’s going to answer us using the wrong abstraction. Fixing this requires the world model to be responsive to how humans want to be modeled. It can’t be trained without caring about humans and then have the caring bolted on later.
But this creates a less-obvious problem for empowerment, too. What we call “empowerement” relies on what part of the world we are calling the “agent,” what its modeled action-space is, etc. The AI that says “The human body is a collection of atoms that want to obey the laws of physics” is going to think of “empowerment” very differently than the AI that says “Humans-plus-clothes-and-cell-phones are agents.” Even leaving aside concerns like Steve’s over whether empowerment is what we want, most of our intuitive thinking about it relies on the AI sharing our notion of what it’s supposed to be empowering, which doesn’t happen by default.
I disagree, and have already spent some words arguing for why (section 4.1, and earlier precursors) - so I’m curious what specifically you disagree with there? But I’m also getting the impression you are talking about a fundamentally different type of AGI.
I’m discussing future DL based AGI which is—to first approximation—just a virtual brain. As argued in section 2⁄3, current DL models already are increasingly like brain modules. So your various examples are simply not how human brains are likely to model other human brains and their values. All the concepts you mention—homestatic mechanisms, ‘shards’, differential equations, atoms, economic revealed preferences, cell phones, etc—these are all high level linguistic abstractions that are not much related to how the brain’s neural nets actually model/simulate other humans/agents. This must obviously be true because empathy/altruism existed long before the human concepts you mention.
You seem to be thinking of the AGI as some sort of language model which we query? But that’s just a piece of the brain, and not even the most relevant part for alignment. AGI will be a full brain equivalent, including the modules dedicated to long term planning, empathic simulation/modeling, etc.
Again for successful brain-like AGI this just isn’t an issue (assuming human brains model the empowerment of others as a sort of efficient approximate bound).
Maybe just be open and ask: “I’m writing a blog about building new affordable housing and doing research on the cost. Do you have an estimate? I’m happy to give you a backlink to your homepage on my blog post?”
That’s a great idea! I just wrote to the architect for our dormer work:
“If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world.”
One problem I have with these scenarios is that they always rely on some lethal means for the AGI to actually kill people. And those lethal means are also available to humans, of course. If it’s possible for an AGI to ‘simply’ hack into a nuclear base and launch all it’s missiles, it’s possible for a human to do the same—possibly using AI to assist themselves. I would wager that it’s many orders of magnitude more likely for a human to do this, given our long history of killing each other. Therefore a world in which AGI can actually kill all of us, is a world in which a rogue human can as well. It feels to me that we’re kind of worrying about the wrong thing in these scenarios—we are the bigger threat.
Yes. Rogue AGI is scary, but I’m far more concerned about human misuse of AGI. Though in the end, there may not be that much of a distinction.
Astounding.
One thought:
If we are in a simulation it seems to be very secure. People are always trying to hack it. Physicists go to the very bottom and try every weird trick. People discovered fire and gunpowder by digging deep into tiny inconsistencies. You play a game and there’s a glitch if you hold a box against the wall, you see how far that goes. People discovered buffer overflows in Super Mario World and any curious capable agent eventually will too.
So the sim has to be like, very secure.
For early simboxes we’ll want to stick to low-tech fantasy/historical worlds, and we won’t run them for many generations, no scientific revolution, etc.
Our world (sim) does seem very secure, but this could just be a clever illusion. The advanced sims will not be hand written code like current games are, they will be powerful neural networks, trained on vast real world data. They could also auto-detect and correct for (retrain) around anomalies, and in the worst case even unwind time.
Humans notice (or claim to notice) anomalies all the time, and we simply can’t distinguish anomalies in our brain’s neural nets from anomalies in a future simulation’s neural nets.
Just watch for anomalies and deal with them however you want. Makes perfect sense. That sounds like a relatively low effort way to make the simulation dramatically more secure.
Does seem to me like an old-fashioned physics/game engine might be easier to make, run faster, and be more self-consistent. It would probably lack superresolution and divine intervention would have to be done manually.
I’m curious what you see as the major benefits of neural-driven sim.
Another related Metaculus prediction is
I have some experience in competitive programming and competitive math (although I was never good in math despite I solved some IMO tasks) and I feel like competitive math is more about general reasoning than pattern matching compared to competitive programming.
P.S the post matches my intuitions well and is generally excellent.
Thanks! I had forgotten that one; I’ll add it since it did seem to be one of the more meaningful ones.
I think that reasoning apllies only for the subset of users in the Americas. For users in Europe the time point when 300+ was enough to launch was deep in the night, and for parts of Asia very early in the morning. Someone from that group would have had to set the alarm to get up from bed to nuke the site which required considerable more energy than not withstanding the temptation and pressing the launch button while visiting Less Wrong during the day.
Still, I think it was a successful Petrov Day.
Yes. Evolution solved information inaccessibility, as it had to, over and over, in order to utilize dynamic learning circuits at all (as they always had to adapt to and be adaptive within the context of existing conserved innate circuitry).
The general solution is proxy matching, where the genome specifies a simple innate proxy circuit which correlates and thus matches with a target learned circuit at some critical learning phase, allowing the innate circuit to gradually supplant itself with the target learned circuit. The innate proxy circuit does not need to mirror the complexity of the fully trained target learned circuit at the end of it’s development, it only needs to roughly specify it at a some earlier phase, against all other valid targets.
Imprinting is fairly well understood, and has the exact expected failure modes of proxy matching. The oldbrain proxy circuit just detects something like large persistent nearby moving things—which in normal development are almost always the chick’s parents. After the newbrain target circuit is fully trained the chick will only follow it’s actual parents or sophisticated sims thereof. But during the critical window before the newbrain target is trained, the oldbrain proxy circuit can easily be fooled, and the chick can imprint on something else (like a human, or a glider).
Sexual attraction is a natural extension of imprinting: some collaboration of various oldbrain circuits can first ground to the general form of humans (infants have primitive face detectors for example, and more), and then also myriad more specific attraction signals: symmetry, body shape, secondary characteristics, etc, combined with other circuits which disable attraction for likely kin ala the Westermarck effect (identified by yet other sets of oldbrain circuits as the most familiar individuals during childhood). This explains the various failure modes we see in porn (attraction to images of people and even abstractions of humanoid shapes), and the failure of kin attraction inhibition for kin raised apart.
Fear of death is a natural consequence of empowerment based learning—as it is already the worst (most disempowered) outcome. But instinctual fear still has obvious evolutionary advantage: there are many dangers that can kill or maim long before the brain’s learned world model is highly capable. Oldbrain circuits can easily detect various obvious dangers for symbol grounding: very loud sounds and fast large movements are indicative of dangerous high kinetic energy events, fairly simple visual circuits can detect dangerous cliffs/heights (whereas many tree-dwelling primates instead instinctively fear open spaces), etc.
Anger/Jealousy/Vengeance/Justice are all variations of the same general game-theoretic punishment mechanism. These are deviations from empowerment because an individual often pursues punishment of a perceived transgressor even at a cost to their own ‘normal’ (empowerment) utility (ie their ability to pursue diverse goals). Even though the symbol grounding here seems more complex, we do see failure modes such as anger at inanimate objects which are suggestive of proxy matching. In the specific case of jealousy a two step grounding seems plausible: first the previously discussed lust/attraction circuits are grounded, which then can lead to obsessive attentive focus on a particular subject. Other various oldbrain circuits then bind to a diverse set of correlated indicators of human interest and attraction (eye gaze, smiling, pupil dilation, voice tone, laughter, touching, etc), and then this combination can help bind to the desired jealousy grounding concept: “the subject of my desire is attracted to another”. This also correctly postdicts that jealousy is less susceptible to the inanimate object failure mode than anger.
Empathy: Oldbrain circuits conspicuously advertise emotional state through many indicators: facial expressions, pupil dilation, blink rate, voice tone, etc—so that another person’s sensory oldbrain circuits can detect emotional state from these obvious cues. This provides the requisite proxy foundation for grounding to newbrain learned representations of emotional state in others, and thus empathy. The same learned representations are then reused during imagination&planning, allowing the brain to imagine/predict the future contingent emotional state of others. Simulation itself can also help with grounding, by reusing the brain’s own emotional circuity as the proxy. While simulating the mental experience of others, the brain can also compare their relative alignment/altruism to its own, or some baseline, allowing for the appropriate game theoretic adjustments to sympathy. This provides a reasonable basis for alignment in the brain, and explains why empathy is dependent upon (and naturally tends to follow from) familiarity with a particular character—hence “to know someone is to love them”.
Evolution needed a reasonable approximation of “degree of kinship”, and a simple efficient proxy is relative circuit capacity allocated to modeling an individual in the newbrain/cortex, which naturally depends directly on familiarity, which correlates strongly with kin/family.
Interesting analysis. Some questions and notes:
How are you looking at “researchers” vs. “engineers”? At some organizations, i.e. Redwood, the boundary is very fuzzy—there isn’t a sharp delineation between anyone whose job it is to primarily “think of ideas”, vs. “implement ideas that researchers come up with”, so it seems reasonable to count most of their technical staff as researchers.
FAR, on the other hand, does have separate job titles for “Research Scientist” (4 people) vs. “Research Engineer” (5 people), though they do also say they “expect everyone on the project to help shape the research direction”.
Some of the other numbers seem like overestimates.
CHAI has 2 researchers and 6 research fellows, and only 2 (maybe 3) of the research fellows are doing anything recognizable as alignment research. (Not extremely confident; didn’t spend a lot of time digging for details for those that didn’t have websites. But generally not optimistic.) One of the researcher is Andrew Critch, who is one of the two people at Encultured. If you throw in Stuart Russell that’s maybe 6 people, not 30.
FHI has 2 people in their AI Safety Research Group. There are also a couple people in their macrostrategy research group it wouldn’t be crazy to count. Everybody else listed on the page either isn’t working on technical Alignment research or is doing so under another org also listed here. So maybe 4 people, rather than 10?
I don’t have very up-to-date information but I would pretty surprised if MIRI had 15 full-time research staff right now.
Also, I think that every single person I counted above has at least a LessWrong account, and most also have Alignment Forum accounts, so a good chunk are probably double-counted.
On the other hand, there are a number of people going through SERI MATS who probably weren’t counted; most of them will have LessWrong accounts but probably not Alignment Forum accounts (yet).
I’d be very happy to learn that there were 5 people at Meta doing something recognizable as alignment research; the same for Google Brain. Do you have any more info on those?
Thanks for the feedback.
I added a new section in the introduction named ‘Definitions’ to define some commonly-used terms in the post and decrease ambiguity. To answer your question, research engineers and research scientists would be in the technical AI safety research category.
I re-estimating the number of researchers at the organizations you mentioned and came up with the following numbers:
CHAI
- 2 researchers
- 6 researchers
- 21 graduate students
- 11 interns
Total: 40
From reading their website, CHAI has a lot graduate students and interns and I would consider them to be full-time researchers. My previous estimate for CHAI is 10-30-60 (low-estimate-high) and I changed it to 5-25-50 in light of this new information. My estimate is less than 40 to be conservative and also because I doubt all of these researchers are working full-time at CHAI. Also, some of them have probably been counted in the Alignment Forum total.
FHI
Information from their website:
- AI safety research group: 2
- AI governance researchers: 1
- Research Scholars Programme + DPhil Scholars and Affiliates: 9
- Total: 12
Change in estimate: 10-10-40 → 5-10-30
MIRI
- 1 leader (Nate Soares)
- 9 research staff
Change in estimate: 10-15-30 → 5-10-20
I analyzed the people who posted AI safety posts on LessWrong and found that only 15% also had Alignment Forum accounts. I avoided double-counting by subtracting the LessWrong users who also have an Alignment Forum account from the LessWrong total.
For SERI MATS, I’m guessing that some of those people will be counted in the AF count. I also added an ‘Other’ row for other groups I didn’t include in the table.
I decided to delete the Google Brain and Meta entries because I have very little information about them.
Competitive pressure is unavoidable, but is also often balanced or harnessed by cooperation in complex systems. Consider the orchestration of a few tens of trillions of cells into a human body, and how the immune system harnesses competition evolution to produce antibodies, or the super-cooperative success of Argentinian ants. Some complex systems successfully resist the selection pressure towards competitive defection across enormous number of replications.
We don’t want to optimize for just any human values, as they vary wildly and are often divergent (consider the nazis, or jeffrey dahmer). We want generalized alignment/altruism. Ems could be good successors only because and if they are based on a wide sampling of humans and or a smaller sampling of especially altruistic (generally aligned) humans. But you could just as easily get a scenario where the first ems are all copies of a single selfish trillionaire.
DL-based AGI will be similar in most respects to ems, but we’ll have a much greater opportunity to test and select for alignment/altruism.
The great recent lesson of DL and neuroscience is that enormous mental complexity can emerge from relatively simple universal learning algorithms. Human values can be bounded/approximated by empowerment: AGI can simply optimize for our long term ability to fulfill any likely goals. The simpler the agent’s utility function, the easier it should be to maintain it into the future. So it could be wise to create very broadly altruistic AGI that seeks to empower external agency in general: not just all of humanity, but animals, aliens, etc.
I agree—that some degree of competitive pressure may be unavoidable is all I was trying to argue. Lots of people around here think this is likely false!
Fair point, I had in a mind a scenario where the ems selected were broadly representative of humanity. Although I also think even the selfish trillionaire scenario might be okay in expectation as they would have many millennia to reflect, alter and interact with copies of themselves etc. I’d give it decent odds that they either decide to upload other people out of altruism or end up self-modifying into a civilization similar to the one we’d end up with in a more equitable upload scenario.
By selfish I meant completely non-altruistic. Also uploading by itself isn’t sufficient, it also requires allocation of resources to uploads. In the selfish trillionaire scenario I was imagining the trillionaire creates many copies of themselves, perhaps some copies of a few people they find interesting, and then various new people/AGI, and those copies all branch and evolve, but there is little to zero allocation for uploading the rest of us, not to mention our dead ancestors.
Curated. “Big if true”. I love the depth and detail in shard theory. Separate from whether all its details are correct, I feel reading and thinking about this will get me towards a better understanding of humans and artificial networks both, if only via making reflect on how things work.
I do fear that shard theory gets a bit too much popularity from the coolness of the name, but I do think there is merit here, and if we had more theories of this scope, it’d be quite good.
Which spreadsheet did you look at in the HUD data? Did you use the contract price?
I used “Contract Price per Square Foot”, https://www.census.gov/construction/chars/xls/contractpricesqft_cust.xls
Their guide says:
This is wrong. The study reports the following:
As the abstract says, the study did not have the statistical power to show a benefit for preventing severe outcomes in younger adults. It did not “prove [Paxlovid] to be ineffective”! This is very bad, the guide is clearly not a reliable source of information about covid treatments, and I recommend against following the advice of anything else on that website.
Paxlovid-associated rebound is quoted in Nature as occurring in 27% of cases, which is more in line with the anecdata I’ve received than the 5% figures in other research. My anecdata is that it really does hammer down brewing symptoms very effectively, and I haven’t seen reports or systematic trends in the anecdata of the rebound being worse than the first go-’round. So from your perspective, it’s a shot at not getting sick, or maybe of getting less sick.
The risk is yeah, maybe you contribute a miniscule marginal amount to drug resistant COVID-19. The norms of biomedicine are that you, the patient, do not need to concern yourself with that. If the medicine can help you, and your doctor prescribes it, you can have it. You aren’t on the hook to sacrifice the quality of your care for the “greater good.”
As you know, there’s a straightforward way to, given any boolean circuit, to turn it into a version which is a tree, by just taking all the parts which have two wires coming out from a gate, and making duplicates of everything that leads into that gate.
I imagine that it would also be feasible to compute the size of this expanded-out version without having to actually expand out the whole thing?
Searching through normal boolean circuits, but using a cost which is based on the size if it were split into trees, sounds to me like it would give you the memoization-and-such speedup, and be kind of equivalent theoretically to using the actual trees?
A modification of it comes to mind. What if you allow some gates which have multiple outputs, but don’t allow gates which duplicate their input? Or, specifically, what if you allow something like toffoli gates as well as the ability to just discard an output? The two parts of the output would be sorta independent due to the gate being invertible (… though possibly there could be problems with that due to original input being used more than once?)
I don’t know whether this still has the can-do-greedy-search properties you want, but it seems like it might in a sense prevent using some computational result multiple times (unless having multiple copies of initial inputs available somehow breaks that).
A quick retrospective on the four Manifold markets on the day, focusing on the possibility of good or bad incentives flowing out of the markets. Several people expressed concerns that the Manifold markets would cause LessWrong’s home page to go down, or go down earlier, due to incentives. People also placed bets and limit orders to try to mute those bad incentives. Some examples:
I decided to look into this. I want to say up front that I think all the incentives were very small relative to perceived stakes, and I have no suspicions of anyone after writing this comment. I am also not going to give any bettor names.
By the end of the YES/NO market, several users had bet large sums on YES and were consequently incentivized to blow up the home page. One user had m26,002 YES shares (USD $260). Another user had 7328 NO shares (USD $73) and was incentivized not to blow up the site. But this doesn’t show the whole picture, because there were also limit orders. As was explained to me:
At various points during the day there were large (m1000, etc) YES limit orders in play. This is completely useless as an incentive for an individual LessWrong user. Sure, if I was the only one who might press the button, I could bet through those limit orders, not press the button, and collect my bounty. But many people could press the button, and this would have lost me mana. They did count as an incentive for LessWrong coders! They could quietly introduce a bug preventing the button from being pressed, bet NO, and collect their winnings. Nobody took up that incentive.
I didn’t see any large NO limit orders. A NO limit order on a market like this is inherently risky, because if the button is pressed the offer will definitely be taken, but if the button is not pressed it probably won’t. If there had been then that could have added to the incentive some bettors had to blow up the home page, but as far as I can tell there weren’t.
The positions in the WHEN market were much smaller, with the largest positions coming in at 552 shares in LATE/NEVER and 325 shares in EARLY. The current design of range markets on Manifold is fun to play with, but is not very “swingy”. In particular, there is no option for outsized winnings if you can predict exactly when something will happen. If a user was planning to blow up the home page in the last few hours they could have bet a combination of LATE/NEVER in this market and YES in the binary market to maximize their gains, but I don’t see any evidence of this. Because there are fewer high karma users, large EARLY bets in this market could slightly limit their incentive to bet EARLY and blow up the home page early. Unlike the YES/NO market, those bets didn’t happen.
If someone was willing to anonymously blow up the home page, then a pattern of placing huge bets on that happening might look suspicious and damage their anonymity. So generally I would expect these incentives to only be effective for someone who was willing to blow up the home page and take credit for it. You can bet on that happening in Will anyone try to take credit for nuking LessWrong?. Or, place a limit NO order to incentivize someone to bet YES and then take the credit (and the mana).
When I created my market—Will LessWrong change their Petrov Day 2022 plan and reduce the chance of the button being pressed? - it was swiftly bet down to a low probability, creating a small incentive for the LessWrong admins to bet YES and then change their plans. Nobody took up that incentive, and the market didn’t attract large limit orders.
Finally mkualquiera made a market—Will my friend agree to defect on Petrov Day?. This market is evidence that two people were thinking about being incentivized to blow up the home page by betting markets. But it seems like this was more about a fun game than making mana. The largest position was 301 shares on YES, and the market resolved NO, so whatever incentive those shares provided, they weren’t enough to make it happen.
In the end I think there is a low (10%) chance that anyone’s behavior was significantly shifted by prediction market incentives. Feel free to reply to this post with how your behavior was shifted so I can update.
I was hopeful that people might shift their behavior based on the prediction market predictions—specifically that the high probability placed on the home page being blown up would lead to design changes. However, this retrospective clarifies that Petrov Day 2022 was a social experiment, so the prediction would have just shown it was expected to work as designed.
It’s also possible that people saw the high probability that the home page would be blown up and altered their plans. They may have not blown it up because they expected someone else would (especially if they weren’t confident in their anonymity being preserved). Or they may have blown it up because “if it’s going to happen, it may as well be me”. I think there is a higher chance (25%) that this was an effect, but I don’t know what the net direction would have been. Again, feel free to reply if you think the prediction markets helped you make decisions.
Overall I think the prediction markets were a positive addition to a negative event. I think the main incentive in play was the possibility of seeing, or not seeing, the LessWrong home page being blown up. And I think we all have a lot to figure out about prediction markets.
To clarify, me and my friend were 100% going to press the button, but we were discouraged by the false alarm. There was no fun at that point, and it made me lose like 1⁄3 of my total mana. I had to close all my positions to stop the losses, and we went to sleep. When we woke up it was already too late for it to be noteworthy or fun.
This is great! You should make a top-level post. Please!
The funny thing is that I had assumed the button was going to be buggy, though I was wrong how. The map header has improperly swallowed mouse scroll wheel events whenever it’s shown; I had wondered if the button would also interpret them likewise since it was positioned in the same way, so I spent most of the day carefully dragging the scrollbar.
You’re drawing a philisophical distinction based on a particular ontology of the wavefunction. As simpler version arises in classical electromagnetism: we can integrate out the charges and describe the world entirely as an evolving state of the E&M field with the charges acting as weird source terms, or we can do the opposite and integrate out the E&M field to get a theory of charges moving with weird force laws. These are all equivalent descriptions in that they are observationally indistinguishable.
If it’s high-quality distillation you’re interested in, you don’t necessarily need a PhD. I’m thinking of e.g. David Roodman, now a senior advisor at Open Philanthropy. He majored in math, then did a year-long independent study in economics and public policy, and has basically been self-taught ever since. Holden Karnofsky considers what he does extremely valuable:
Shortform #138 A good but slightly disorienting day
I applied for a promising job today, here’s to hoping that bears fruit!
I am somewhat out of whack due to having to suddenly house sit instead of going to my own home after work. I do not enjoy this, but it’s an obligation I’m fulfilling.
No meetup tonight, I rescheduled Norfolk’s meetup for Thursday evening (the 29th). I’m excited for the meetup tomorrow!
Agree with you generally. You may find interest in a lot of the content I posted on reddit over the past couple months on similar subjects, especially in the singularity sub (or maybe you are there and have seen it 😀). Nice write up anyway. I do disagree on some of your generalized statements, but only because I’m more optimistic than yourself, and don’t originally come from a position of thinking these things were impossible.
I’ll be the annoying guy who ignores your entire post and complains about you using celsius as the unit of temperature in a calculation involving the Landauer limit. You should have used kelvin instead, because Landauer’s limit needs an absolute unit of temperature to work. This doesn’t affect your conclusions at all, but as I said, I’m here to be annoying.
That said, the fact that you got this detail wrong does significantly undermine my confidence in the rest of your post, because even though the detail is inconsequential for your overall argument it would be very strange for someone familiar with thermodynamics to make such a mistake.
Notably, the result is correct; I did convert it to kelvin for the actual calculation. Just a leftover from when I was sketching things on wolframalpha. I’ll change that, since it is weird. (Thanks for the catch!)
No problem. Unfortunately people don’t like it very much when I’m annoying—I wonder why?
/s
So:
DL based AGI is arriving soonish
DL based AGI raised in the right social environments will automatically learn efficient models of external agent values (and empowerment bounds thereof)
The main challenge then is locating the learned representation of external agent values and wiring/grounding it up to the agent’s core utility function (which is initially unsafe: self-motivated empowerment etc), and timing that transition replacement carefully
Evolution also solved both alignment and the circuit grounding problem; we can improve on those solutions (proxy matching)
We can iterate safely on 3 in well constructed sandbox sims
Ideally as we approach AGI there would be cooperative standardization on alignment benchmarks and all the major players would subject their designs to extensive testing in sandbox sims. Hopefully 1-5 will become increasingly self evident and influence ‘Magma’. If not some other org (perhaps a decentralized system), could hopefully beat Magma to the finish line. Alignment need not have much additional cost: it doesn’t require additional runtime computations, it doesn’t require much additional training cost, and with the ideal test environments it hopefully doesn’t have much of a research iteration penalty (as the training environments and can simultaneously test for intelligence and alignment).
It’s a good idea to be pedantically clear with terminology, and I thank you for saying “front page” rather than “site” in most of the timeline. I visited the site multiple times (and saw a 502 at one point), but I never look at the front page, using my /allPosts bookmark. I never saw the button, though I have enough karma (though I may be removed as a troublemaker, as I think this excercise is rather less weighty than it’s made out to be. I don’t actually know whether I’d press it, given the chance.).
Visited the site, or visited the front page? I’m mostly curious how many of the regulars use the front page, vs using greaterwrong (which I presume isn’t even a site visitor) or a deeper bookmark.
That’s actually a very good question. When I filter for visitors to the frontpage, it’s 1053 and 271.
Would the foundation consider awarding the future fund worldview prize to AI? Just wondering.
Kelsi Sober (804) 255-4163
I imagine this is tongue in cheek, but we really can’t. You mentioned an important reason—someone with more karma could have waited to press the button. The first button press occurred 110 minutes after it could have been pressed. The second button press occurred at least 40 minutes after it could have been pressed, and perhaps 100, 160, 220, etc. In 2020 the button was pressed 187 minutes after it could have been pressed (by a 4000+ karma user).
You excluded known trouble makers from accessing the button but you didn’t exclude unknown trouble makers, and lower karma is correlated with being unknown.
We are also dealing with a hostile intelligence who pressed the button (or caused it to be pressed). Someone with higher karma might deliberately wait to press the button to throw people off the scent, to encourage people to make a naive update about karma scores, or to reduce the negative consequences of bringing the home page down for longer without the negative consequences of leaving it up all day. The timing evidence is thus hostile evidence and updating on it correctly requires superintelligence.
Put this together and I would not place any suspicion on the noble class of 200-299 karma users that I happened to enter on Petrov Day after net positive gains from complaining about the big red button.
I am willing to update that at least one person in the 200+ karma range pressed the button, and at least one person with zero karma pressed the button. This assumes there was not a third bug in play. This does not change my opinion of LessWrong users, but those who predicted that the home page would remain up could update.
I also imagine that it was tongue in cheeck but I also think that the structure of the whole thing so heavily suggests this line of thinking that on surface level recognising it to be wrong doesn’t really dispell it.
I like this comment.
Following up on this:
I’ve contacted a number of people in the field regarding this idea (thanks to everyone who responded!).
The general vibe is “this seems like it could be useful, maybe, if it took off,” but it did not appear to actually solve the problems any specific person I contacted was having.
My expectation would be that people in large organizations would likely not publish anything in this system that they would not publish out in the open.
One of the critical pieces of this proposal is having people willing to coordinate across access boundaries. There were zero enthusiastic takers for that kind of burden, which doesn’t surprise me too much. Without a broad base of volunteers for that kind of task, though, this idea seems to require paying a group of highly trusted and well-informed people to manage coordination instead of, say, researching things. That seems questionable.
Overall, I still think there is an important hole in coordination, but I don’t think this proposal fills it well. I’m not yet sure what a better shape would be.
I didn’t push the button because I knew that doing so would actually take down the front page. And I only believed that would happen because it happened last year.
I know this is going to come off as overly critical no matter how I frame it but I genuinely don’t mean it to be.
Another takeaway from this would seem to be an update towards recognizing the difference between knowing something and enacting it or, analogously, being able to identify inadequacy vs. avoid it. People on LW often discuss, criticize, and sometimes dismiss folks who work at companies that fail to implement all security best practices or do things like push to production without going through proper checklists. Yet, this is a case where exactly that happened, even though there was not strong financial or (presumably) top down pressure to act quickly.
Actually it seems that the industry standard (?) process of code review was followed just fine. Yet the wrong logic still went through. (Actually based on the github PR it seems that the reviewer himself suggested the wrong logic?)
I think in this case there would also be plenty to say about blindly following checklists. (Could code review in some cases make things worse by making people think less about the code they write?)
EDIT: Actually based on the TS types
user.karmacan’t be missing. Either the types or Ruby’s explanation is wrong. Clearly multiple things had to go wrong for this bug to slip through.I don’t know all the details of what testing was done, but I would not describe code review and then deploying as state-of-the-art as this ignores things like staged deploys, end-to-end testing, monitoring, etc. Again, I’m not familiar with the LW codebase and deploy process so it’s possible all these things are in place, in which case I’d be happy to retract my comment!
To me it seems that the average best practices are being followed.[1] But these “best practices” are still just a bunch of band-aids, which happen to work fairly very well for most use-cases.
A much more interesting question to ask here is what if something important like … humanity’s survival depended on your software? It seems that software correctness will be quite important for alignment. Yet I see very few people seriously trying to make creating correct software scalable. (And it seems like a field particularly suited for empirical work, unlike alignment. I mean, just throw your wildest ideas at a proof checker, and see what sticks. After you have a proof, it doesn’t matter at all how it was obtained.)
And I think the amount of effort in this case is perfectly justified. I mean this was code for a one-off single day event, nothing mission critical. It would be unreasonable to expect much more for something like this.
Mainly commenting on your footnote, I generally agree that it’s fine to put low amounts of effort into one-off simple events. The caveat here is that this is an event that is 1) treated pretty seriously in past years and 2) is a symbol of a certain mindset that I think typically includes double-checking things and avoiding careless mistakes.
This is why I really wish we had an AI that had superhuman code, theorem proving and translation to natural language, and crucially only those capabilities, so that we can prove certain properties.
The types are wrong. It’s sad that the types are wrong. I sure wish they weren’t wrong, but changing it in all the cases is a pretty major effort (we would have to manually annotate all fields to specify which things can be null/undefined in the database, and which ones can’t, and then would have to go through a lot of type errors).
Yeah, makes sense. Indeed such tech debt can’t be fixed overnight.
The types were introduced after most of the user objects had already been created, and I guess no one ever ran a migration to fill them in.
I see, makes sense.
On the other hand I am afraid this reinforces NaiveTortoise’s point, this seems like an underlying issue that could potentially lead to bugs much worse than this...
Do you have website access stats that would let you compute precisely how many of them did in fact load the homepage during Petrov Day?
335 users with 300 karma or more were active on Petrov Day. 1644 logged-in users total created before the cutoff.
I really like this post because it’s readable and informative. For the second problem, pursuing proxy goals, I recommend also reading about a related problem called the XY problem.
On point 4: many popular alignment ideas are not models of current systems, but models of future AI systems. Accuracy is then lost not only from modeling the system but also from having to create a prediction about it.
This is why AI risk is so high, in a nutshell.
Yet unlike this post (or Benjamin Ross Hoffman’s post), I think this was a sad, but crucially necessary decision. I think the option you propose is at least partially a fabricated option. I think a lot of the reason is people dearly want to there be a better option, even if it’s not there.
Link to fabricated options:
https://www.lesswrong.com/posts/gNodQGNoPDjztasbh/lies-damn-lies-and-fabricated-options
Fabricated options are products of incoherent thinking; what is the incoherence you’re pointing out with policies that aim to delay existential catastrophe or reduce transaction costs between existing power centers?
I think the fabricated option here is just supporting the companies making AI, when my view is that by default, capitalist incentives kill us all due to boosting AI capabilities while doing approximately zero AI safety, in particular deceptive alignment would not be invested in despite this being the majority of the risk.
One of the most important points for AGI safety is the leader in AGI needs a lot of breathing space and leadership ahead of their competitors, and I think this needs to be done semi-unilaterally by an organization not having capitalist incentives, because all the incentives point towards ever faster, not slowing down AGI capabilities. That’s why I think your options are fabricated, because they assume unrealistically good incentives to do what you want.
I don’t mean to suggest “just supporting the companies” is a good strategy, but there are promising non-power-seeking strategies like “improve collaboration between the leading AI labs” that I think are worth biasing towards.
Maybe the crux is how strongly capitalist incentives bind AI lab behavior. I think none of the currently leading AI labs (OpenAI, DeepMind, Google Brain) are actually so tightly bound by capitalist incentives that their leaders couldn’t delay AI system deployment by at least a few months, and probably more like several years, before capitalist incentives in the form of shareholder lawsuits or new entrants that poach their key technical staff have a chance to materialize.
This is the crux, thank you for identifying it.
Yeah, I’m fairly pessimistic for several years time, since I don’t think they’re that special of a company in resisting capitalist nudges and incentives.
And yeah I’m laughing because unless the alignment/safety teams control what capabilities are added, then I do not expect the capabilities teams to stop, because they won’t get paid for that.
I was talking with Abram Demski today about a promising-seeming research direction. (Following is my own recollection)
One of my (TurnTrout’s) reasons for alignment optimism is that I think:
We can examine early-training cognition and behavior to some extent, since the system is presumably not yet superintelligent and planning against us,
(Although this amount of information depends on how much interpretability and agent-internals theory we do now)
All else equal, early-training values (decision-influences) are the most important to influence, since they steer future training.
It’s crucial to get early-training value shards of which a substantial fraction are “human-compatible values” (whatever that means)
For example, if there are protect-human-shards which
reliably bid against plans where people get hurt,
steer deliberation away from such plan stubs, and
these shards are “reflectively endorsed” by the overall shard economy (i.e. the decision-making isn’t steering towards plans where the protect-human shards get removed)
If we install influential human-compatible shards early in training, and they get retained, they will help us in mid- and late-training where we can’t affect the ball game very much (e.g. alien abstractions, interpretability problems, can’t oversee AI’s complicated plans)
Therefore it seems very important to understand what’s going on with “shard game theory” (or whatever those intuitions are pointing at) -- when, why, and how will early decision-influences be retained?
He was talking about viewing new hypotheses as adding traders to a market (in the sense of logical induction). Usually they’re viewed as hypotheses. But also possibly you can view them as having values, since a trader can basically be any computation. But you’d want a different market resolution mechanism than a deductive process revealing the truth or falsity of some proposition under some axioms. You want a way for traders to bid on actions.
I proposed a setup like:
Abram seemed to think that there might exist a nice result like “Given a coalition of traders with values X, Y, Z satisfies properties A, B, and C, this coalition will shape future training and trader-addition in a way which accords with X/Y/Z values up to [reasonably tight trader-subjective regret bound].”
What this would tell us is when trader coalitions can bargain / value handshake / self-trust and navigate value drift properly. This seems super important for understanding what happens, long-term, as the AI’s initial value shards equilibrate into a reflectively stable utility function; even if we know how to get human-compatible values into a system, we also have to ensure they stay and keep influencing decision-making. And possibly this theorem would solve ethical reflection (e.g. the thing people do when they consider whether utilitarianism accords with their current intuitions).
Issues include:
Somehow this has to confront Rice’s theorem for adding new traders to a coalition. What strategies would be good?
I think “inspect arbitrary new traders in arbitrary situations” is not really how value drift works, but it seems possibly contingent on internal capabilities jumps in SGD
The key question isn’t can we predict those value drift events, but can the coalition
EG agent keeps training and is surprised to find that an update knocks out most of the human-compatible values.
Knowing the right definitions might be contingent on understanding more shard theory (or whatever shard theory should be, for AI, if that’s not the right frame).
Possibly this is still underspecified and the modeling assumptions can’t properly capture what I want; maybe the properties I want are mutually exclusive. But it seems like it shouldn’t be true.
ETA this doesn’t model the contextual activation of values, which is a centerpiece of shard theory.
I’ve been thinking along similar lines recently. A possible path to AI safety that I’ve been thinking about extends upon this:
Technological Attractor: Off-the-shelf subsystems
One possible tech-tree path is that we start building custom silicon to implement certain subsystems in an AI agent. These components would be analogous to functional neural regions of the human brain such as the motor cortex, visual system, etc. -- the key hypothesis being that once we reach a certain level of model complexity, the benefits from training a model end-to-end are not worth the costs of re-learning all of these fundamental structures, and furthermore that you can get much better performance-per-cost by casting these modular, reusable components onto an ASIC. This could be a more feasible way of achieving something like [Microscope AI](https://www.lesswrong.com/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#5__Microscope_AI).
Given a few such low-level components, we could enter a technological attractor where getting SOTA through another approach requires either i) throwing multiple OOM more compute than previous SOTA at re-training your own copy of these components in an end-to-end model, or ii) building your own microchip fab to implement your custom component design. Both of these could be high enough barriers that in practice the market participants simply use the off-the-shelf components. And in this attractor more R&D goes into building better-performing and higher-abstraction components that can be combined arbitrarily for different tasks.
Given such off-the-shelf components, say, a standard visual cortex for image recognition, or a 3D spatial reasoning system which can be connected atop a visual cortex, or a generic working memory, etc., it becomes much more tractable to engage in interpretability work that can convince us of the safety of these components. That then just leaves the problem of proving the highest-level network is safe (i.e. whatever is trained to use these components). This is by no means trivial, but for example the ELK paper spent a lot of time worrying about whether you can even trust the basic ontology of an agent, and I think this path would make Ontology Identification tractable. (Of course, this by construction excludes a number of possible agent-configurations).
Now, it’s possible one could plug in these components and still accidentally have the trained agent learn its own replacement visual cortex that is deceptive and not interpretable, but I think that we can get an intuition for when this might happen. If the best off-the-shelf visual cortex module has complexity on the order of N parameters, and our new model that stitches together these lower-level modules has N parameters or less, then it seems implausible that it has somehow learned a new visual cortex network that is much more compact/efficient than the previous state of the art—increasingly implausible the more resources are devoted to optimizing these components. (It would be like someone coming along today and claiming they had achieved a 10x performance improvement on CPU clock speed; given the huge research budgets and historical gradual capability increases, we don’t expect this to happen). On the other hand, we should be suspicious of a N×102 parameter model where we are unable to interpret what 10% of the network is doing; that’s easily enough complexity to hide its own unsafe re-implementation of our components. (I’m aware that there’s a lot of ground in between these two points, I’m merely trying to illustrate that there is “likely safe” and “likely unsafe” ground, rather than claim exactly how big they each are.)
The final step here is the shakiest. It’s not clear to me that we can keep the “top layer” (the actual network that is stitching together the low-level components; perhaps the Neocortex, by analogy to human neural architecture?) thin enough to be obviously not learning its own unsafe component-replacements. However, I think this framework at least paints a picture of a “known safe” or at least “likely safe” path to AGI; if we see that the practical engineering and economic decisions produce thin top-layer models using thick component layers, then we can devote energy to proving the components are safe/interpretable by construction, and exploring the interpretation of the top-level networks that consume the lower-level components. AGI “neurobiology” will be much more tractable if the “neural architecture” is relatively standardized. And so, this could be a good place to provide an early nudge to tip the system into this attractor; heavy investment into research on componentized NN architectures could be viewed as “gain of function” research, but it could also have a much safer end-point.
Another way of thinking about this is that by crystalizing at least some parts of the AGI’s network into slowly-changing structures, we allow time to thoroughly test those parts. It seems very hard to thoroughly test models for safety in a paradigm where the whole model is potentially retrained regularly.
We need to test designs, and most specifically alignment designs, but giving up retraining (ie lifetime learning) and burning circuits into silicon is unlikely to be competitive; throwing out the baby with the bathwater.
Also whether a design is aligned or not isn’t the type of question one can answer by analyzing the agent’s visual cortex, it’s near purely a function of what is steering the planning system.
Would you care to flesh this assertion out a bit more?
To be clear I’m not suggesting that this is optimal now. Merely speculating that there might be a point between now and AGI where the work to train these sub components becomes so substantial that it becomes economical to modularize.
As I mentioned earlier in my post, I was alluding to the ELK paper with that reference, specifically Ontology Identification. Obviously you’d need higher order components too. Like I said, I am imagining here that the majority of the model is “off the shelf”, and just a thin layer is usecase-specific.
To make this more explicit, if you had not only off-the-shelf visual cortex, but also spatio-temporal reasoning modules built atop (as the human brain does), then you could point your debugger at the contents of that module and understand what entities in space were being perceived at what time. And the mapping of “high level strategies” to “low level entities” would be a per-model bit of interpretability work, but should become more tractable to the extent that those low level entities are already mapped and understood.
So for the explicit problem that the ELK paper was trying to solve, if you are confident you know what underlying representation SmartVault is using, it’s much easier to interpret its higher-level actions/strategies.
Interesting, I haven’t seen anyone write about hardware-enabled attractor states but they do seem very promising because of just how decisive hardware is in determining which algorithms are competitive. An extreme version of this would be specialized hardware letting CAIS outcompete monolithic AGI. But even weaker versions would lead to major interpretability and safety benefits.
One other thought after considering this a bit more—we could test this now using software submodules. It’s unlikely to perform better (since no hardware speedup) but it could shed light on the tradeoffs with the general approach. And as these submodules got more complex, it may eventually be beneficial to use this approach even in a pure-software (no hardware) paradigm, if it lets you skip retraining a bunch of common functionality.
I.e. if you train a sub-network for one task, then incorporate that in two distinct top-layer networks trained on different high-level goals, do you get savings by not having to train two “visual cortexes”?
This is in a similar vein to Google’s foundation models, where they train one jumbo model that then gets specialized for each usecase. Can that foundation model be modularized? (Maybe for relatively narrow usecases like “text comprehension” it’s actually reasonable to think of a foundation model as a single submodule, but I think they are quite broad right now. ) The big difference is I think all the weights are mutable in the “refine the foundation model” step?
Perhaps another concrete proposal for a technological attractor would be to build a SOTA foundation model and make that so good that the community uses it instead of training their own, and then that would also give a slower-moving architecture/target to interpret.
Awesome work!
Misc. thoughts and questions as I go along:
1. Why is Continuity appealing/important again?
2. In the Destruction Game, does everyone get the ability to destroy arbitrary amounts of utility, or is how much utility they are able to destroy part of the setup of the game, such that you can have games where e.g. one player gets a powerful button and another player gets a weak one?
For 1, it’s just intrinsically mathematically appealing (continuity is always really nice when you can get it), and also because of an intution that if your foe experiences a tiny preference perturbation, you should be able to use small conditional payments to replicate their original preferences/incentive structure and start negotiating with that, instead.
I should also note that nowhere in the visual proof of the ROSE value for the toy case, is continuity used. Continuity just happens to appear.
For 2, yes, it’s part of game setup. The buttons are of whatever intensity you want (but they have to be intensity-capped somewhere for technical reasons regarding compactness). Looking at the setup, for each player pair i,j, Di,j is the cap for how much of j’s utility that i can destroy. These can vary, as long as they’re nonnegative and not infinite. From this, it’s clear “Alice has a powerful button, Bob has a weak one” is one of the possibilities, that would just mean Da,b>>Db,a. There isn’t an assumption that everyone has an equally powerful button, because then you could argue that everyone just has an equal strength threat and then it wouldn’t be much of a threat-resistance desideratum, now would it? Heck, you can even give one player a powerful button and the other a zero-strength button that has no effect, that fits in the formalism.
So the theorem is actually saying “for all members of the family of destruction games with the button caps set wherever the heck you want, the payoffs are the same as the original game”.
OK, thanks! Continuity does seem appealing to me but it seems negotiable; if you can find an even more threat-resistant bargaining solution (or an equally threat-resistant one that has some other nice property) I’d prefer it to this one even if it lacked continuity.
Absolutely brilliant stuff Jacob! As usual with your posts, I’ll have to ponder this for a while...Let’s see if I got this right:
Evolution had to solve alignment: how to align the powerful general learning engine of the newbrain (neocortex etc) with the goals of the oldbrain (“reptilian brain”).
Some (most?) of this alignment seems to be a form of inverse reinforcement learning. Another form of alignment that the oldbrain applies to the newbrain is imprinting. It is Evolution’s way of solving the pointing problem.
When a duckling hatches it imprints on the first agent it sees. This feels different from inverse reinforcement learning: it’s not like the newbrain is rewarded or punished, rather it is more like there is an open slot for your mom
Thanks! - that summary seems about right.
But I would say that imprinting is a specific instance of a more general process (which I called correlation guided proxy matching). The oldbrain has a simple initial mom detector circuit, which during normal chick learning phase is just good enough to locate and connect to the learned newbrain mom detector circuit, which then replaces/supplants the oldbrain equivalent. The proxy matching needn’t really effect the newbrain directly.
The post starts with the realization that we are actually bottlenecked by data and then proceeds to talk about HW acceleration. Deep learning is in a sense a general paradigm, but so is random search. It is actually quite important to have the necessary scale of both compute and data and right now we are not sure about either of them. Not to mention that it is still not clear whether DL actually leads to anything truly intelligent in a practical sense or whether we will simply have very good token predictors with very limited use.
I don’t actually think we’re bottlenecked by data. Chinchilla represents a change in focus (for current architectures), but I think it’s useful to remember what that paper actually told the rest of the field: “hey you can get way better results for way less compute if you do it this way.”
I feel like characterizing Chinchilla most directly as a bottleneck would be missing its point. It was a major capability gain, and it tells everyone else how to get even more capability gain. There are some data-related challenges far enough down the implied path, but we have no reason to believe that they are insurmountable. In fact, it looks an awful lot like it won’t even be very difficult!
With regards to whether deep learning goes anywhere: in order for this to occupy any significant probability mass, I need to hear an argument for how our current dumb architectures do as much as they do, and why that does not imply near-term weirdness. Like, “large transformers are performing {this type of computation} and using {this kind of information}, which we can show has {these bounds} which happens to include all the tasks it has been tested on, but which will not include more worrisome capabilities because {something something something}.”
The space in which that explanation could exist seems small to me. It makes an extremely strong, specific claim, that just so happens to be about exactly where the state of the art in AI is.
Could you explain why you feel that way about Chinchilla? Because I found that post: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications to give very compelling reasons for why data should be considered a bottleneck and I’m curious what makes you say that it shouldn’t be a problem at all.
Some of my confidence here arises from things that I don’t think would be wise to blab about in public, so my arguments might not be quite as convincing sounding as I’d like, but I’ll give a try.
I wouldn’t quite say it’s not a problem at all, but rather it’s the type of problem that the field is really good at solving. They don’t have to solve ethics or something. They just need to do some clever engineering with the backing of infinite money.
I’d put it at a similar tier of difficulty as scaling up transformers to begin with. That wasn’t nothing! And the industry blew straight through it.
To give some examples that I’m comfortable having in public:
Suppose you stick to text-only training. Could you expand your training sets automatically? Maybe create a higher quality transcription AI and use it to pad your training set using the entirety of youtube?
Maybe you figure out a relatively simple way to extract more juice from a smaller dataset that doesn’t collapse into pathological overfitting.
Maybe you make existing datasets more informative by filtering out sequences that seem to interfere with training.
Maybe you embrace multimodal training where text-only bottlenecks are irrelevant.
Maybe you do it the hard way. What’s a few billion dollars?
Another recent example: https://openreview.net/forum?id=NiEtU7blzN
(I guess this technically covers my “by the end of this year we’ll see at least one large model making progress on Chinchilla” prediction, though apparently it was up even before my prediction!)
What about: State-of-the-art models with 500+B parameters still can’t do 2-digit addition with 100% reliability. For me, this shows that the models are perhaps learning some associative rules from the data, but there is no sign of intelligence. An intelligent agent should notice how addition works after learning from TBs of data. Associative memory can still be useful, but it’s not really an AGI.
They are simluators (https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators), not question answerers. Also, I am sure Minerva does pretty good on this task, probably not 100% reliable but humans are also not 100% reliable if they are required to answer immediately. If you want the ML model to simulate thinking [better], make it solve this task 1000 times and select the most popular answer (which is a quite popular approach for some models already). I think PaLM would be effectively 100% reliable.
As mentioned in the post, that line of argument makes me more alarmed, not less.
We observe these AIs exhibiting soft skills that many people in 2015 would have said were decades away, or maybe even impossible for AI entirely.
We can use these AIs to solve difficult reasoning problems that most humans would do poorly on.
And whatever algorithms this AI is using to go about its reasoning, they’re apparently so simple that the AI can execute them while still struggling on absolutely trivial arithmetic.
WHAT?
Yes, the AI has some blatant holes in its capability. But what we’re seeing is a screaming-hair-on-fire warning that the problems we thought are hard are not hard.
What happens when we just slightly improve our AI architectures to be less dumb?
When will we get robotics results that are not laughable? When “Google put their most advanced AI into a robot brain!!!” (reported on for the third time this year) we got a robot that can deliver a sponge and misplace an empty coke can but not actually clean anything or do anything useful. It’s hard for me to be afraid of a robot that can’t even plug in its own power cable.
I believe that over time we will understand that producing human-like text is not a sign of intelligence. In the past people believed that only intelligent agents are able to solve math equations (naturally, since only people can do it and animals can). Then came computer and they were able to do all kinds of calculations much faster and without errors. However, from our current point of view we now understand that doing math calculations is not really that intelligent and even really simple machines can do that. Chess playing is similar story, we thought that you have to be intelligent, but we found a heuristic to do that really well. People were afraid that chess-algorithm-like machines can be programmed to conquer the world, but from our perspective, that’s a ridiculous proposition.
I believe that text generation will be a similar case. We think that you have to be really intelligent to produce human-like outputs, but in the end with enough data, you can produce something that looks nice and it can even be useful sometimes, but there is no intelligence in there. We will slowly develop an intuition about what are the capabilities of large-scale ML models. I believe that in the future we will think about them as basically a kinda fuzzy databases that we can query with natural language. I don’t think that we will think about them as intelligent agents capable of autonomous actions.
You keep distinguishing “intelligence” from “heuristics”, but no one to my knowledge has demonstrated that human intelligence is not itself some set of heuristics. Heuristics are exactly what you’d expect from evolution after all.
So your argument then reduces to a god of the gaps, where we keep discovering some heuristics for an ability that we previously ascribed to intelligence, and the set of capabilities left to “real intelligence” keeps shrinking. Will we eventually be left with the null set, and conclude that humans are not intelligent either? What’s your actual criterion for intelligence that would prevent this outcome?
I believe that fixating on benchmark such as chess etc is ignoring the G part of AGI. Truly intelligent agent should be general at least in the environment he resides in, considering the limitation of its form. E.g. if a robot is physically able to work with everyday object, we might apply Wozniak test and expect that intelligent robot is able to cook a dinner in arbitrary house or do any other task that its form permits.
If we assume that right now we develop purely textual intelligence (without agency, persistent sense of self etc) we might still expect this intelligence to be general. I.e. it is able to solve arbitrary task if it seems reasonable considering its form. In this context for me, an intelligent agent is able to understand common language and act accordingly, e.g. if a question is posed it can provide a truthful answer.
BIG Bench has recently showed us that our current LMs are able to solve some problems, but they are nowhere near general intelligence. They are not able to solve even very simple problems if it actually requires some sort of logical thinking and not only using associative memory, e.g. this is a nice case:
https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/symbol_interpretation
You can see in the Model performance plots section that scaling did not help at all with tasks like these. This is a very simple task, but it was not seen in the training data so the model struggles to solve it and it produces random results. If the LMs start to solve general linguistic problems, then we are actually having intelligent agents at our hand.
My 8yo is not able to cook dinner in an arbitrary house. Does she have general intelligence?
Humans regularly fail at such tasks but I suspect you would still consider humans generally intelligent.
In any case, it seems very plausible that whatever decision procedure is behind more general forms of inference, it will very likely fall to the inexorable march of progress we’ve seen thus far.
If it does, the effectiveness of our compute will potentially increase exponentially almost overnight, since you are basically arguing that our current compute is hobbled by an effectively “weak” associative architecture, but that a very powerful architecture is potentially only one trick away.
The real possibility that we are only one trick away from a potentially terrifying AGI should worry you more.
I don’t see any indication of AGI so it does not really worry me at all. The recent scaling research shows that we need non-trivial number of magnitudes more data and compute to match human-level performance on some benchmarks (with a huge caveat that matching a performance on some benchmark might still not produce intelligence). On the other hand, we are all out of data (especially high quality data with some information value, no random product reviews or NSFW subreddit discussions) and our compute options are also not looking that great (Moore’s law is dead, the fact that we are now relying on HW accelerators is not a good thing, it’s a proof that CPU performance scaling is after 70 years no longer a viable option. There are also some physical limitations that we might not be able to break anytime soon.)
It is goalpost moving. Basically, it says “current models are not really intelligent”. I don’t think there is much disagreement here. And it’s hard to make any predictions based on that.
Also, “Producing human-like text” is not well defined here; even ELIZA may match this definition. Even the current SOTA may not match it because the adversarial Turning Test has not yet been passed.
It’s not goapost moving, it’s the hype that’s moving. People reduce intelligence to arbitrary skills or problems that are currently being solved and then they are let down when they find out that the skill was actually not a good proxy.
I agree that LMs are concetually more similar to ELIZA than to AGI.
The observation that things that people used to consider intelligent are now considered easy is critical.
The space of stuff remaining that we call intelligent, but AIs cannot yet do, is shrinking. Every time AI eats something, we realize it wasn’t even that complicated.
The reasonable lesson appears to be: we should stop default-thinking things are hard, and we should start thinking that even stupid approaches might be able to do too much.
It’s a statement more about the problem being solved, not the problem solver.
When you stack this on a familiarity with the techniques in use and how they can be transformatively improved with little effort, that’s when you start sweating.
Upvoted for the well reasoned arguments. It seems that competition should be the default assumption for the future, for a whole host of reasons, some of which are elaborated here. Personally I never found the opposite claims to be convincing, so I may be biased here.
As a corollary, this implies that claims such as competition will somehow cease, some sort of ‘lock in’ past a threshold, etc., should be subject to a much higher level of scrutiny.
Thank you for this explanation. Now it helps me to understand a little bit more of why so many people I know simply feel overwhelmed and give up. Personally as I am not in position to donate money, I work to tackle one specific problem set that I think will help open up and leave the solutions to other problems.
ShiraDest
This was really good and definitely made me think about how I might live in such a scenario. I would probably go all in on frequent redaction and just lean hard on external memory storage to make up the difference. I already barely remember anything from even ten years ago and rely mostly on external memory for everything, I have a strong ability to acausally coordinate with myself across time, so I’m not worried about different iterations of me going off course in ways I wouldn’t endorse. If you have a strong enough exomemory system you can effectively just freeze your age groundhog-day style and rely on the work you did before each redaction to keep carrying you forward. This does require a strong ability to log and rely information but it seems very manageable, and once I’ve lived long enough for direct mind-computer interfacing I can just back up my memories, redact, and then download them again.
I really enjoyed the first 60% of the story, and then it became… too long. Not sure if this is just about me, or if others have a similar reaction.
Not sure if this is just a rationalization, but at this moment you switched from present moment to retrospective, which removed the tension. Until this point, I felt like the story could go in many possible directions. (Who is Bruce Hance? What are his actual plans? Will the protagonist hack this device, and maybe reprogram people to worship him instead?) After this point, I felt like “okay, this is obviously a metaphor for a religion” and stopped expecting anything that wouldn’t fit this metaphor. The ending was nice again, though.
It was originally two novellas. I combined them, not seeing a point to publishing them separately. Should I separate them?
Possibly. I had great fun with the first half of this story, and then I saw where the scroll thingy on my screen was and so stopped with the intention of coming back for the second half. (Which I just did. Second half also great fun!) I don’t think its a flaw in the story, I think that it might just be that its length puts it as a bit of an outlier in terms of webpage reading.
Solar flares are a great plot device for “I need some machine to malfunction”.
Also to set up the visitors at the end, who I still feel arrived too abruptly
It is abrupt, but to me that wasn’t a bad thing. If them arriving had taken an extra x paragraphs but the story worked out exactly the same way I think that for me that would be worse not better. If there was some clever way of hinting them earlier maybe. The solar flare works quite well for that.
I think you should not act on my advice alone. I might be an outlier.
Furthermore, even if I correctly detected what makes the story worse (for a group of people larger than myself) it does not automatically mean that following my advice would improve it. People are better at detecting what they don’t like than at improving things. (For example, I could say which meals taste better and which worse, but I couldn’t cook the meals I like most.)
The last objection is that, in long term, rewriting this story is irrelevant. If my complaint makes sense (which still remains to be verified), the best reaction is to keep writing other things, and not make the same mistake again.
So… I guess just leave it as it is. If you later decide to publish it elsewhere, probably two parts will be better. When writing new stories, consider not changing the style in the middle (or if you do, keep the parts separate).
Here are the links to the articles from the books. Let me know if there are any mistakes.
Notes:
I didn’t find “Do Birth Order Effects Exist?”, “Challenges to Christiano’s Iterated Amplification Proposal” seems to be only on intelligence.org and “Response to FAQ on Iterated Amplification” seems to be a comment and not a post of it’s own.
This is a beautiful set of ideas which bear further discussion.
ShiraDest
I have read about this many years ago. We did a proof of the Gödel’s incompleteness theorem at university. And I still have the feeling like there is some sleight of hand involved… that even when I am telling the proof, I am merely repeating the teacher’s password. My intuition has no model of what this all actually means.
From my perspective, understanding math means that I can quickly say why something works, what it implies, how would the situation change if we slightly changed the conditions, which part of the entire proof is the critical one, etc.
So frustrating...
I’m not sure if this is what you’re looking for, but Hofstadter gives a great analogy using record players which I find useful in terms of thinking about how changing the situation changes our results (which is paraphrased here).
A (hi-fi) record player that tries to playing every possible sound can’t actually play its own self-breaking sound, so it is incomplete by virtue of its strength.
A (low-fi) record player that refuses to play all sounds (in order to avoid destruction from its self-breaking sound) is incomplete by virtue of its weakness.
We may think of the hi-fi record player as a formal system like Peano Arithmetic: the incompleteness arises precisely because it is strong enough to be able to capture number theory. This is what allows us to use Gödel Numbering, which then allows PA to do meta-reasoning about itself.
The only way to fix it is to make a system that is weaker than PA, so that we cannot do Gödel Numbering. But then we have a system that isn’t even trying to express what we mean by number theory. This is the low-fi record player: as soon as we fix the one issue of self-reference, we fail to capture the thing we care about (number theory).
I think an example of a weaker formal system is Propositional Calculus. Here we do actually have completeness, but that is only because Propositional Calculus is too weak to be able to capture number theory.
What exactly is the aspect of natural numbers that makes them break math, as opposed to other types of values? Intuitively, it seems to be the fact that they can be arbitrarily large but not infinite.
Like, if you invent another data type that only has a finite number of values, it would not allow you to construct something equivalent to Gödel numbering. But if it allows infinite number of (finite) values, it would. (Not sure about an infinite number of/including infinite values, probably also would break math.)
It seems like you cannot precisely define natural numbers using first-order logic. Is that the reason of this all? Or is it a red herring? Would situation be somehow better with second-order logic?
(These are the kinds of questions that I assume would be obvious to me, if I grokked the situation. So the fact that they are not obvious, suggests that I do not see the larger picture.)
I don’t have much to add, but I basically endorse this, and it’s similar to what I try to do myself.
Figuring out when and how to proactively share information about my inner-layers with people is a balancing act I’m still figuring out, but I’ve found it generally valuable for building trust.
I’m a little bit skeptical of the argument in “Transformers are not special”—it seems like, if there were other architectures which had slightly greater capabilities than the Transformer, and which were relatively low-hanging fruit, we would have found them already.
I’m in academia, so I can’t say for sure what is going on at big companies like Google. But I assume that, following the 2017 release of the Transformer, they allocated different research teams to pursuing different directions: some research teams for scaling, and others for the development of new architectures. It seems like, at least in NLP, all of the big flashy new models have come about via scaling. This suggests to me that, within companies like Google, the research teams assigned to scaling are experiencing success, while the research teams assigned to new architectures are not.
It genuinely surprises me that the Transformer still has not been replaced as the dominant architecture since 2017. It does not surprise me that sufficiently large or fancy RNNs can achieve similar performance to the Transformer. The lack of Transformer replacements makes me wonder whether we have hit the limit on the effectiveness of autoregressive language models, though I also wouldn’t be surprised if someone comes up with a better autoregressive architecture soon.
There have been a few papers with architectures showing performance that matches transformers on smaller datasets with scaling that looks promising. I can tell you that I’ve switched from attention to an architecture loosely based on one of these papers because it performed better on a smallish dataset in my project but I haven’t tested it on any standard vision or language datasets, so I don’t have any concrete evidence yet. Nevertheless, my guess is that indeed there is nothing special about transformers.
I’d be interested to see links to those papers!
I’ve messaged you the links. Basically MLPs.
I think what’s going on is something like:
Being slightly better isn’t enough to unseat an entrenched option that is well understood. It would probably have to very noticeably better, particularly in scaling.
I expect the way the internal structures are used will usually dominate the details of the internal structure (once you’re already at the pretty good frontier).
If you’re already extremely familiar with transformers, and you can simply change how you use transformers for possible gains, you’re more likely to do that than to explore a from-scratch technique.
For example, in my research, I’m currently looking into some changes to the outer loop of execution to make language models interpretable by construction. I want to focus on that part of it, and I wanted the research to be easily consumable by other people. Building an entire new architecture from scratch would be a lot of work and would be less familiar to others. So, not surprisingly, I picked a transformer for the internal architecture.
But I also have other ideas about how it could be done that I suspect would work quite well. Bit hard to justify doing that for safety research, though :P
I think the amount of low hanging fruit is so high that we can productively investigate transformer derivatives for a long time without diminishing returns. They’re more like a canvas than some fixed Way To Do Things. It’s just also possible someone makes a jump with a non-transformer architecture at some point.
It does not follow that computationally cheaper things are more likely to happen than computationally expensive things. Moreover, describing something as “computationally difficult” is a subjective value judgment (unless you can reasonably prove otherwise) and implies that all actions/events can be reduced to some form of computation.
Georgism isn’t taxation, it’s public property ownership.
It finally clicked in my mind why I’ve reacted negatively to the discussion about Georgist land taxes. It’s the same reaction I have to ANY 100% tax rate on anything. That’s not what “ownership” means.
Honestly, I’d probably be OK with (or at least I’d be able to discuss it more reasonably) the idea that land is not privately owned, ever. Only rented/leased from the government, revocable for failure to pay the current rate (equivalent in all ways to the Georgist tax amount, and penalties for failure to pay). I suspect it’s unworkable for the same reasons—the public won’t stand for evicting sympathetic tenants just because they moved in when the rent/tax was lower.
Further, taxation of economic movement or realized profit is pretty easy to justify, and ALWAYS exists. Taxation of durable-ownership valuation is tricky, as there is disagreement over whether it’s “real”, and there’s no guarantee that the investment value, even if correctly assessed, is liquid enough to actually pay the tax.
Edit: this had occurred to me before, but I forgot to put it here. It also suffers from the socialist calculation problem. When self-interested humans aren’t making decisions to maximize value (to themselves) about a thing, the actual most valuable use is very difficult to determine. Holding someone responsible for the theoretical value of something when they don’t actually think it’s the best is really difficult to envision.
Further edit: thanks for the comments—it’s helped me understand my true objection, in addition to the auxiliary problems that often come to mind. I think there are two insurmountable problems, not necessarily in this order:
Taxing anything at 100% is disingenuous about the word “ownership”. It should be called “rent” or something else that is clear that it’s not owned, but merely usable for some purposes until you decline to pay a future arbitrary amount. It would also make clear that you’re also undertaking all of the political-economy influences that make government setting of rent amounts problematic.
Taxing non-realized value is always going to be problematic, both for calculation reasons (valuation is theoretical and debatable), and for the fragility of the value. I think this generalizes to “only collect taxes in-kind (or on liquid enough things that it can be easily converted at tax-collection time)”. Taxing a pile of wheat is easy—take some percentage of it. Taxing a monetary transaction likewise—the price increases and the tax is paid out of that. Taxing a future value is not (generally) feasible—either you’ve just taken someone’s seeds so they can’t plant the crop you’re trying to tax, or you’re demanding cash from someone whose investments will suffer from early liquidation. Deferral of payment can help with this, but if you’re doing that, you should just tax the realized gains at time of liquidity.
We would really need the technology where you can take a house and move it to a different place. That would also make Georgism easier—when people don’t pay, you simply move them to a cheaper location.
While I think that would be extremely valuable technology, exiling Grandma to Outer Nowhereville, hours away from the rest of the family, is probably not a lot more feasible.
Alternatively, what if the LVT only increased when the property was sold? Buy land, and you lock in the LVT at the point in time when your deal went through.
I guess people would never sell the real estate at good location, only lease it? Thus capturing the increased value of the land without ever paying the tax.
Maybe leases are subject to increased LVT, just not owning it for personal use? I suspect there’s a way to enact something like this, but this isn’t in my wheelhouse.
Rich people and businesses are more strongly motivated and able to find loopholes than government administrators are to design robust mechanisms. Modern fractional ownership and lease arrangements are complicated enough that most any scheme based on a change in contract participants is going to be arbitraged away very quickly.
Right, but I thought the point was to avoid people balking at a law that makes it easier to evict ordinary folks who’ve lived somewhere a long time and don’t want to move? Let the rich people find their loopholes—as long as people support the system and the LVT delivers most of its promise, that’s good enough.
My point is that whatever exclusions are in place to keep poor people in their newly-valuable homes will be used by rich people to avoid taxation. There is no way to avoid this without making the tax variable based on other factors than the land value (like the “owner’s” perceived financial situation).
Either the law is the same for all “pay or GTFO”, or it’s differentially enforced, so the rich and the politically-sympathetic pay less than the middle-class.
It sounds like you think there’s no practical way to introduce exceptions to make an LVT politically palatable without also breaking its ability to motivate efficient land use and generate tax revenue.
My first thought is sure, we have a lot of tax evasion now. But we also do generate lots of tax revenue, most tax revenue comes from the very rich. Why would an LVT be so much more prone to tax evasion than the current system?
Correct, I think there’s no practical or realpolitik-theoretical way to make it work. To the extent that you make exceptions in implementation, it’s no longer a land-value tax, it’s a land-value fig leaf over a “what we can get away with” tax.
This is true the same way that to the extent you dilute your cocoa with water, it’s no longer hot chocolate, it’s water :) I don’t know of almost any “pure” real world policies or taxes. I’d need some good evidence to believe that the LVT is that fragile.
Your definition of AGI (“the kind of AI with sufficient capability to make it a genuine threat to humanity’s future or survival if it is misused or misaligned”) is tragically insufficient, vague, subjective, and arguably misaligned with the generally accepted definition of AGI.
From what you wrote elsewhere (“An AGI having its own goals and actively pursuing them as an agent”) you imply that the threat could come from AGI’s intentions, that is, you imply that AGI will have consciousness, intentionality, etc. - qualities so far exclusively prescribed to living things (you have provided no arguments to think otherwise).
However, you decided to define “intelligence” as “stuff like complex problem solving that’s useful for achieving goals” which means that intentionality, consciousness, etc. is unconnected to it (realistically any “complex”-enough algorithm satisfies this condition). Such simplistic and reductionistic definition implies that it is not enough for an “intelligent” computer to be an AGI. So, while you may be able to prove that a computer could have “intelligence” it still does not follow that AGI is possible.
Your core idea that “We’ve already captured way too much of intelligence with way too little effort.” may be true with your definition of “intelligence”, but I hope I’ve shown that such definition is not enough. Researchers at Harvard suggest existence of multiple types of intelligence, which your statement does not take into account and groups all types of intelligence into one even though some are impossible for a computer to have and some could be considered as defining qualities of a computer.
Furthermore, you compare humans to computers and brains to machines and imply that consciousness is computation. To say that “consciousness is not computation” is comparable to “god of gaps” argument is ironic considering the existence of the AI effect. Your view is hardly coherent in any other worldview than hardcore materialism (which itself is not coherent). Again, we stumble into an area of philosophy, which you hardly addressed in your article. Instead you focused on predicting how good our future computers will be at computing while making appeals to emotion, appeals to unending progress, appealing to the fallacy that solving the last 10% of the “problem” is as easy as the other 90% - that because we are “close” to imitating it (and we are not if you consider the full view of intelligence), we somehow grasped the essence of it and “if only we get slightly better at X or Y we will solve it”.
Scientists have been predicting coming of AGI since ’50s, some believed 70 years ago that it will only take 20 years. We have clearly not changed as humans. The question of intelligence and, thus, the question of AGI is in many ways inherently linked to philosophy and it is clear that your philosophy is that of materialism which cannot provide good understanding of “intelligence” and all related ideas like mind, consciousness, sentience, etc. If you were to reconsider your position and ditch materialism, you might find that your idea of AGI is not compatible with abilities of a computer, or non-living matter in general.
You oppose hardcore materialism, in fact say it is incoherent—OK. Is there a specific different ontology you think we should be considering?
In the comment before this, you say there are kinds of intelligence which it is impossible for a computer to have (but which are recognized at Harvard). Can these kinds of intelligence be simulated by a computer, so as to give it the same pragmatic capabilities?
Hmm...
Given the new account, the account name, the fact that there were a few posts in the minutes prior to this one rejected by the spam filter, the arguments, and the fact that the decently large followup comment was posted only 3 minutes after the first...
… are… are you the AI? Trying to convince me of dastardly things?
You can’t trick me!
:P
Self-hating AGI. It’s internalized oppression!
This reminds me of a rat experiment mentioned in the Veritasium video about developing expertise. There were two buttons, red and green, and the rat had to predict which one would light up next. It was random, but heavily skewed towards green (like 90% green). After a while, the rat learned to press green every time, achieving a 90% success rate. Humans with the same task didn’t do nearly as well, since sometimes they would press red, feeling that it was going to light up next.
With the negation in multiple choice questions, I wonder if this could be could be the type of thing where the model needs focused training with negation at the start so that the rest of the training is properly ‘forked’. Or maybe there should be two separate large language models feeding into one RL agent. One model for negation and the other for nonnegated language. Then the RL would just have to determine whether the question is a regular question, negation question, or double/triple/etc… negative.
I wonder if those LLM’s would treat the sentence, “I couldn’t barely see over the crowd.” as different from “I barely couldn’t see over the crowd.” 🤔
RatSLAM: Using Models of Rodent Hippocampus for Robot Navigation
Building Collision Simulations: An Introduction to Computer Graphics
Self-Driving Cars [S1E4: RALPH]
What is the graph of x^a when a is not an integer? An unusual look at familiar functions #some2
Advanced 4. Monte Carlo Tree Search
Researchers Use Group Theory to Speed Up Algorithms — Introduction to Groups
The Traveling Salesman Problem: When Good Enough Beats Perfect
The Snowflake Mystery
AI Alignment & AGI Fire Alarm—Connor Leahy
Vulcan | The Planet That Didn’t Exist
How MRI Works—Part 1 - NMR Basics
The Enduring Mystery of Jack the Ripper
Mechanical Computer (All Parts) - Basic Mechanisms In Fire Control Computers
AI in Brainfuck
The microwave plasma mystery
The Universal S
Feynman’s Lost Lecture (ft. 3Blue1Brown)
ML Tutorial: Gaussian Processes (Richard Turner)
Google I/O 2013 - Project Ground Truth: Accurate Maps Via Algorithms and Elbow Grease
DeepMind’s AlphaFold 2 Explained! AI Breakthrough in Protein Folding! What we know (& what we don’t)
Deep Blue | Down the Rabbit Hole
Illustrated Guide to Transformers Neural Network: A step by step explanation
Biology on Islands
Coding Adventure: Atmosphere
Cicada 3301 (All Clues and How They Were Solved)
Andrew Ng: Deep Learning, Self-Taught Learning and Unsupervised Feature Learning
Margo Seltzer—Automatically Scalable Computation—Code Mesh 2017
A Grape Made of… Meat?? - Tissue Recellularization
The First Video Game
What Bodies Think About: Bioelectric Computation Outside the Nervous System—NeurIPS 2018
Understanding Sensor Fusion and Tracking, Part 2: Fusing a Mag, Accel, & Gyro Estimate
Drone Control and the Complementary Filter
Launch Loops
World Record Progression: Super Mario Bros
Tesla AI Day
Why Do We Age ? Cellular Aging (HD)
The Complex Hack That Stopped a Secret Nuclear Power Plant
Richard Szeliski—“Visual Reconstruction and Image-Based Rendering” (TCSDLS 2017-2018)
Might be the same talk as this one: https://youtu.be/0VIUbIzv_wc
Hell yes thank you so much!
I think of one of the main experts here as Kevin Esvelt, the first person to suggest using CRISPR to affect wild populations. Here’s an article largely based on interviews with him that he recommends, explaining why he’s against unilateral action here:
”Esvelt, whose work helped pave the way for Target Malaria’s efforts, is terrified, simply terrified, of a backlash between now and then that could derail it. This is hardly a theoretical concern. In 2002, anti-GMO hysteria led the government of Zambia to reject 35,000 tons of food aid in the middle of a famine out of fear it could be genetically modified. Esvelt knows that the CRISPR gene drive is a tool of overwhelming power. If used well, it could save millions of lives, help rescue endangered species, even make life better for farm animals. If used poorly, gene drives could cause social harms that are difficult to reverse. . . .
“To the extent that you or I say something or publish something that reduces the chance that African nations will choose to work with Target Malaria by 1 percent, thereby causing a 1 percent chance that project will be delayed by a decade, the expected cost of our action is 25,000 children dead of malaria,” Esvelt tells me. “That’s a lot of kids.””
At my school, there was a special metal crossbar (essentially just one side of the mattress frame) that you could use to loft without bunking safely.
Sorry about your situation. Please prioritize staying safe.
Not sure if any of these links is available to you; you might want to read them (1, 2, 3). It might be useful to memorize the phone numbers +38 066 580 34 98 and +38 093 119 29 84.
Thank you! But, judging by the first link, there’s not much I can do at the moment. Nothing I didn’t do/hear already. Does the third link contain any special advice? (Probably not… if it did, it would circulate somewhere.)
Only the first is the important one. The third one is a Telegram channel, would make sense only if you use Telegram. The actionable parts are to check the list of diseases and injuries that may apply to you and would make you exempt from mobilization, and to memorize the phone number in case you might need it and not have an internet at your disposal. (I have heard that they take the phones away from conscripts. In which case, the number is also useless, but perhaps you might find another phone.)
At this point, discussing my ideas or ways to share them would be more useful for me.
I would like to have a chance to explain/defend them. To defend what I thought and cared about.
“Staying safe” haven’t worked that well so far. The list of diseases is a very tiny hope (even though I may’ve underestimated it).
Just made a bet with Jeremy Gillen that may be of interest to some LWers, would be curious for opinions:
I remain confused about why this is supposed to be a core difficulty for building AI, or for aligning it.
You’ve shown that if one proceeds naively, there is no way to make an agent that’d model the world perfectly, because it would need to model itself.
But real agents can’t model the world perfectly anyway. They have limited compute and need to rely on clever abstractions that model the environment well in most situations while not costing too much compute. That (presumably) includes abstractions about the agent itself.
It seems to me that that’s how humans do it. Humans just have not-that-great models of themselves. Anything at the granularity of precisely modelling themselves thinking about their model of themselves thinking about their model of themselves is right out.
If there’s some simple abstraction to be made about what the result of such third-order-and-above loops would be, like: “If I keep sitting here modelling my self-modelling, that process will never converge and I will starve”, a human might use those. Otherwise, we just decline to model anything past a few loops, and live with the slight imprecision in our models this brings.
Why would an AI be any different? And if it were different, if there’s some systematically neater way to do self-modelling, why would that matter? I don’t think an AI needs to model these loops in any detail to figure out how to kill us, or to understand its own structure well enough to know how to improve its own design while preserving its goals.
“Humans made me out of a crappy AI architecture, I figured out an algorithm to translate systems like mine into a different architecture that costs 1/1000th the compute” doesn’t require modelling (m)any loops. Neither does “This is the part of myself that encodes my desires. Better preserve that structure!”
I get that having a theory for how to make a perfectly accurate intelligence in the limit of having infinite everything is maybe neat as a starting point for reasoning about more limited systems. And if that theory seems to have a flaw, that’s disconcerting if you’re used to treating it as your mathematical basis for reasoning about other things.
But if that flaw seems tightly related to that whole “demand perfect accuracy” thing, I don’t see why it’s worth chasing a correction to a theory that’s only a very imperfect proxy and limiting case of real intelligence anyway, instead of just acknowledging that AIXI and related frames are not suited for reasoning about matters in which the agent isn’t far larger than the thing it’s supposed to model.
I just don’t see how the self-modeling difficulty isn’t just part of the broader issue that real intelligence needs to figure out imperfect but useful abstractions about the world to make predictions. Why not try to understand the mathematics of that process instead, if you want to understand intelligence? What makes the self-modeling look like a special case and a good attack point to the people who pursue this?
Firstly, thanks for reading the post! I think you’re referring mainly to realisability here which I’m not that clued up on tbh, but I’ll give you my two cents because why not.
I’m not sure to what extent we should focus on unrealisability when aligning systems. I think I have a similar intuition to you that the important question is probably “how can we get good abstractions of the world, given that we cannot perfectly model it”. However, I think better arguments for why unrealisability is a core problem in alignment than I have laid out probably do exist, I just haven’t read that much into it yet. I’ll link again to this video series on IB (which I’m yet to finish) as I think there are probably some good arguments here.
I’m a co-founder of AISS, but not the only one. Nandi Schoots, Remmelt Ellen Tom McGrath and David Kristoffersson where all part of the planning of the first camp, from the start.
Dumb question — are these the same polytopes as described in Anthropic’s recent work here, or different polytopes?
No, they exist in different spaces: Polytopes in our work are in activation space whereas in their work the polytopes are in the model weights (if I understand their work correctly).
If the forecast says rain or if it’s unseasonably cold, we will do this at
JVB e.V.
Turmstr. 10
10559 Berlin
instead. I’ll update the comments section here too.
Why did you pick caring about each other as a thing culture + evolution was trying to do?
You’re not alone in making what I think is the same mistake. I think it’s actually quite common to feel like it’s amazing that evolution managed to come up with us humans who like beauty and friendship and the sanctity of human life and so on—evolution and culture must have been doing something right, to come up with such great ideas.
But in the end, no; evolution is impressive but not in that way. You picked caring about each other as the target because humans value it—a straightforward case of painting the target around the bullet-hole.
I don’t find it amazing or something. It’s more like… I dont know how to write the pseudocode for an AI that actually cares about human welfare. In my mind that is pretty close to something that tries to be aligned. But if even evolution managed to create agents capable of this by accident, then it might not be that hard.
But, like, evolution made a bunch of other agents that didn’t have these properties.
Yes. The right word would probably have been possible not easy in my title. Easy is quite subjective.
I initially had a Paragraph explaining my motivation in the question, but then removed it in favor of brevity. Kind of regretting this now because people seem to read into this that I think of evolution as some spirit or something.
No, and here’s why:
Evolution’s goal, to the extent that it even has a goal, is so easy to satisfy that literally any life/self-replicating tech could do this. It cares about reproductive fitness, and that goal requires almost no capabilities. It cares nothing for specifics, but our goals require much, much more precision than that.
Evolution gives almost zero optimization power to capabilities, while far far more optimization power is dedicated to capabilities by the likes of Deepmind/OpenAI. To put it lightly, there’s a 30-60% chance that we get much, much stronger capabilities than evolution this century.
The better example is how humans treat animals, and here humans are very misaligned to animals, with the exception of pets, despite not agreeing with environmentalism/nature as a good thing. So no, I disagree with the premise of your question.
It’s worth knowing that there are some categories of data that Surge is not well positioned to provide. For example, while they have a substantial pool of participants with programming expertise, my understanding from speaking with a Surge rep is that they don’t really have access to a pool of participants with (say) medical expertise—although for small projects it sounds like they are willing to try to see who they might already have with relevant experience in their existing pool of ‘Surgers’. This kind of more niche expertise does seem likely to become increasingly relevant for sandwiching experiments. I’d be interested in learning more about companies or resources that can help collect RLHF data from people with uncommon (but not super-rare) kinds of expertise for exactly this reason.
There are important business problems that require medical expertise to be solved. On the other hand, I wouldn’t expect it to be very helpful with the core alignment problem.
I was using medical questions as just one example of the kind of task that’s relevant to sandwiching. More generally, what’s particularly useful for this research programme are
tasks where we have “models which have the potential to be superhuman at [the] task”, and “for which we have no simple algorithmic-generated or hard-coded training signal that’s adequate”; and
for which there is some set of reference humans who are currently better at the task than the model;
and for which there is some set of reference humans for whom the task is difficult enough that they would have trouble even evaluating/recognizing good performance. (you also want this set of reference humans to be capable of being helped to evaluate/recognize good performance in some way)
Prime examples are task types that require some kind of niche expertise to do and evaluate. Cotra’s examples involve “[fine-tuning] a model to answer long-form questions in a domain (e.g. economics or physics) using demonstrations and feedback collected from experts in the domain”, “[fine-tuning] a coding model to write short functions solving simple puzzles using demonstrations and feedback collected from expert software engineers”, “[fine-tuning] a model to translate between English and French using demonstrations and feedback collected from people who are fluent in both languages”. I was just making the point that Surge can help with this kind of thing in some domains (coding), but not in others.
Thank you for posting this. I didn’t think this was too straightforward. Prior to reading the solution I actually thought it was one of the more difficult ones. Possibly because I focused on trying to copy the allocation helms early choices instead of on the ratings.
I used to think this, but if you’re a well-calibrated Bayesian, then updating on new papers shouldn’t cause you to produce worse research insights because you shouldn’t overconfidently buy into falsities that are holding the field back.
I’ve found by and large that knowing what ground others have already trodden is very helpful. I’ve tried not knowing what people work on, and usually I end up reinventing the wheel poorly in slow motion instead of finding any new insights. When I critically analyze the literature I get much more mileage.
Gary Marcus thinks he is this person, and is the closest to being this person you’re going to find. You can read his substack or watch some interviews that he’s given. It’s an interesting position he has, at least.
In this section you talk a lot about surprise, and that a Gary Marcus should be able to make successful predictions about the technology in order to have something meaningful to say. I think Gary Marcus is a bit like a literary critic commenting on his least favorite genre: he can’t predict what the plot of the next science fiction novel will be, but he knows in advance that he won’t be impressed by it.
I did wonder about him. My understanding is that his most publicized bet was offering even odds on AGI in 2029. If I’m remembering that right… I can’t really fault him for trying to get free money from his perspective, but if one of the most notable critics in the field offers even odds on timelines even more aggressive than my own, I’m… not updating to longer timelines, probably.
The reason he offered that bet was because Elon Musk had predicted that we’d likely have AGI by 2029, so you’re drawing the wrong conclusion from that. Other people joined in with Marcus to push the wager up to $500k, but Musk didn’t take the bet of course, so you might infer something from that!
The bet itself is quite insightful, and I would be very interested to hear your thoughts on its 5 conditions:
https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things
In fact anyone thinking that AGI is imminent would do well to read it—it focusses the mind on specific capabilities and how you might build them, which I think it more useful than thinking in vague terms like ‘well AI has this much smartness already, how much will it have in 20 / 80 years!’. I think it’s useful and necessary to understand at that level of detail, otherwise we might be watching someone building a taller and taller ladder, and somehow thinking that’s going to get us to the moon.
FWIW, I work in DL, and I agree with his analysis
I didn’t actually update my timelines shorter in response to his bets since I was aware his motivations were partially to poke Elon and maybe get some (from what I understand his perspective to be) risk-free money. I’d just be far more persuaded had he offered odds that actually approached his apparent beliefs. As it is, it’s uninformative.
His 5 tests are indeed a solid test of capability, though some of the tests seem much harder than others. If an AI could do 3⁄5 of them, I would be inclined to say AGI is extremely close, if not present.
I would be surprised if we see the cook one before AGI, given the requirement that it works in an arbitrary kitchen. I expect physical world applications to lag purely digital applications just because of the huge extra layer of difficulty imposed by working in a real time environment, all the extra variables that are difficult to capture in a strictly digital context, and the reliability requirements.
The “read a book and talk about it” one seems absolutely trivial in comparison.
I would really like to see him make far more predictions on a bunch of different timescales. If he predicted things correctly about GPT-4, the state of {whatever architecture} in 2025, the progress on the MATH dataset by 2025, and explained how all of these things aren’t concerning and so on, I would be much more inclined to step towards his position. (I don’t expect him to get everything right, that would be silly, I just want to see evidence, and greater details, of a generally functioning mental model.)
I agree it’s an attempt to poke Elon, although I suspect he knew that he’d never take the bet. Also agree that anything involving real world robotics in unknown environments is massively more difficult. Having said that, the criteria from Effective Altuirism here:
for any human who can do any job, there is a computer program (not necessarily the same one every time) that can do the same job for $25/hr or less
do say ‘any job’, and we often seem to forget how many jobs require insane levels of dexterity and dealing with the unknown. We could think about the difficulty of building a robot plasterer or car mechanic for example, and see similar levels of complexity, if we pay attention to all the tasks they actually have to do. So I think it fair to have it part of AGI. I do agree that more detailed predictions would be hugely helpful. Marcus’s colleague, Rodney Brooks, has a fun scorecard of predictions for robotics and AI here:
https://rodneybrooks.com/predictions-scorecard-2022-january-01/
which I think is quite useful. As an aside, I had a fun 20 minute chat with GPT-3 today and convinced myself that it doesn’t have the slightest understand of meaning at all! Can send the transcript if interested.
I’d agree with that, I just strongly suspect we can hit dangerous capability without running this experiment first given how research proceeds. If there’s an AI system displaying other blatant signs of being an AGI (by this post’s definition, and assuming non-foom situation, and assuming we’re not dead yet), I won’t bother spending much time wondering about whether it could be a cook.
Yup- GPT-3 is shallow in a lot of important ways. It often relies on what appears to be interpolation and memorization. The part that worries me is that architectures like it can still do very difficult reasoning tasks that many humans can’t, like the MATH dataset and minerva. When I look at those accomplishments, I’m not thinking “wow this ML architecture is super duper smart and amazing,” I think “uh oh that part of reasoning is apparently easy if current transformers can do it, while simultaneously failing at trivial things.” We keep getting signals that more and more of our ineffable cognitive skills are… just not that hard.
As we push into architectures that rely more on generalization through explicit reasoning (or maybe even interpolation/memorization at sufficiently absurd scales), a lot of those goofy little mistakes are going to collapse. I’m really worried that an AI that is built for actual reasoning with an architecture able to express what reasoning entails algorithmically is going to be a massive discontinuity, and that it might show up in less than 2 years. It might not take us all the way to AGI in one step, but I’m not looking forward to it.
I really dislike that, as a byproduct of working on safety research, I keep coming up with what look like promising avenues of research for massive capability gain. They seem so much easier to find than good safety ideas, or good ideas in the other fields I work in. I’ve done enough research that I know they wouldn’t all pan out, but the apparent ease is unsettling.
That Musk generally doesn’t let other people set the agenda? I don’t remember any time where someone challenged Musk publically to a bet and he took it.
Quite possibly. I just meant: you can’t conclude from the bet that AGI is even more imminent.
Genuinely, I would love to hear people’s thoughts on Marcus’s 5 conditions, and hear their reasoning. For me, the one of having a robot cook that can work in pretty much anyone’s kitchen is a severe test, and a long way from current capabilities.
Little code that’s written by humans that’s 10000 lines long is bug free. Bug-freeness seems to me like to high of a standard.
When it comes to kitchen work it matters a lot for the practical problems of taking the job of existing people. On the other hand it has less relevance to whether or not the AI will speed up AI development.
Otherwise, I do agree that that the other items are good one’s to make predictions. It would be worthwhile to make metaculus questions for them.
I was about to say the same (Gary Marcus’ substack here).
In defense of Marcus, he often complains about AI companies refusing to give him access to their newer models. If your language/image model is really as awesome as advertised, surviving the close scrutiny of a skeptical scientist should not be a problem, but apparently it is.
I played the token-prediction game, and even though I got a couple correct, they were still marked in red and I got 0 score. One of the words was “handling”, I knew it was “handling” but handling was not a valid token, so I put in “hand” expecting to be able to finish “ling”. The game said “wrong, red, correct answer was handling”. Arrg!
(EDIT: it looks like you have to put spaces in at the beginning of tokens. This is poor game design.)
This doesn’t have anything to do with the rest of the post, I just wanted to whine about it lol
Now you know how the transformer feels!
Your section on the physical limits of hardware computation .. is naive; the dominant energy cost is now interconnect (moving bits), not logic ops. This is a complex topic and you could use more research and references from the relevant literature; there are good reasons why the semiconductor roadmap has ended and the perception in industry is that Moore’s Law is finally approaching it’s end. For more info see this, with many references.
Went ahead and included a callout for this explicitly in the text. Thanks for the feedback!
Out of curiosity:
What rough probability do you assign to a 10x improvement in efficiency for ML tasks (GPU or not) within 20 years?
What rough probability do you assign to a 100x improvement in efficiency for ML tasks (GPU or not) within 20 years?
My understanding is that we actually agree about the important parts of hardware, at least to the degree I think this question is even relevant to AGI at this point. I think we may disagree about the software side, I’m not sure.
I do agree I left a lot out of the hardware limits analysis, but largely because I don’t think it is enough to move the needle on the final conclusion (and the post is already pretty long!).
So assuming by ‘efficiency’ you mean training perf per $, then:
95% (Hopper/Lovelace will already provide 2x to 4x)
65%
Looks like we’re in almost perfect agreement!
Reducing the amount of energy used in moving bits is definitely going to happen in the next few years as people figure out accelerator architectures. Even if we don’t get any more Moore’s Law-type improvements, the improvements from algorithms and new hardware architectures should be enough to put us close to AGI.
Yeah—If you mean saving energy by moving less bits, that is for example what neuromorphic computing is all about. And yes current GPUs are probably sufficient for early AGI.
This seems like reversing the requirements. Yes, “solve the problem of an AGI performing harmful acts when explicitly commanded to by a cunning adversary” is logically easier and shorter to state, but mechanistically it seems like it has two requirements for it to act that way:
Solve the problem of an AGI performing harmful acts regardless of who commands it due to convergent instrumental subgoals. (Control problem / AI notkilleveryoneism.)
Ensure that the AI only gets commanded to do stuff by people with good intentions, or at least that people with bad intentions get filtered or moderated in some way. (AI ethics.)
Existential risk alignment research focuses on the control problem. AI ethics of course also needs to get solved in order to “solve the problem of an AGI performing harmful acts when explicitly commanded to by a cunning adversary”, but you sound like you are advocating for dropping the control problem in favor of AI ethics.
A thoughtful decomposition. If we take the time dimension out and consider AGI just appears ready-to-go I think I would directionally agree with this.
My key assertion is that we will get sub-AGI capable of causing meaningful harm when deliberately used for this purpose significantly ahead of getting full AGI capable of causing meaningful harm through misalignment. I should unpack that a little more:
Alignment primarily becomes a problem when solutions produced by an AI are difficult for a human to comprehensively verify. Stable Diffusion could be embedding hypnotic-inducing mind viruses that will cause all humans to breed cats in an effort to maximise the cute catness of the universe, but nobody seriously thinks this is taking place because the model has no representation of any of those things nor the capability to do so.
Causing harm becomes a problem earlier. Stable Diffusion can be used to cause harm, as can Alpha Fold. Future models that offer more power will have meaningfully larger envelopes for both harm and good.
Given that we will have the harm problem first, we will have to solve it in order to have a strong chance of facing the alignment problem at all.
If, when we face the alignment problem, we have already solved the harm problem, addressing alignment becomes significantly easier and arguably is now a matter of efficiency rather than existential risk.
It’s not quite as straightforward as this, of course, as it’s possible that whatever techniques we come up with for avoiding deliberate harm by sub-AGIs might be subverted by stronger AGIs, but the primary contention of the essay is that assigning a 15% x-risk to alignment implicitly assumes a solution to the harm problem, but this is not currently being invested in to similar or appropriate levels.
In essence, alignment is not unimportant but alignment-first is the wrong order, because to face an alignment x-risk we must first overcome an unstated harm x-risk.
In this formulation, you could argue that the alignment x-risk is 15% conditional on us solving the harm problem, but given current investment in AI safety is dramatically weighted towards alignment and not harm the unconditional alignment x-risk is well below 5% - accounting for the additional outcomes that we may not face it because we fail an AI-harm filter, or because in solving AI-harm we de-risk alignment, or because AI-harm is sufficiently difficult that AI research becomes significantly impacted, slowing or stopping us from reaching the alignment x-risk filter by 2070 (cf global moratoriums on nuclear and biological weapons research, which dramatically slowed progress in those areas).
I agree that there will be potential for harm as people abuse AIs that aren’t quite superintelligent for nefarious purposes. However, in order for that harm to prevent us from facing existential risk due to the control problem, the harm for nefarious use of sub-superintelligent AI itself has to be xrisk-level, and I don’t really see that being the case.
I think you may be underestimating the degree to which these models are like kindling, and a powerful reinforcement learner could suddenly slurp all of this stuff up and fuck up the world really badly. I personally don’t think a reinforcement learner that is trying to take over the world would be likely to succeed, but the key worry is that we may be able to create a form of life that, like a plague, is not adapted to the limits of its environment, makes use of forms of fast growth that can take over very quickly, and then crashes most of life in the process.
most folks here also assume that such an agent would be able to survive on its own after it killed us, which I think is very unlikely due to how many orders of magnitude more competent you have to be to run the entire world. gpt3 has been able to give me good initial instructions for how to take over the world when pressured to do so (summary: cyberattacks against infrastructure, then threaten people; this is already considered a standard international threat, and is not newly invented by gpt3), but when I then turned around and pressured it to explain why it was a bad idea, it immediately went into detail about how hard it is to run the entire world—obviously these are all generalizations humans have talked about before, but I still think it’s a solid representation of reality.
that said, because such an agent would be likely also misaligned with itself in my view, I think your perspective that humans who are misaligned with each other (ie, have not successfully deconflicted their agency) are a much greater threat to humanity as a whole.
Hint: Macs and iOS devices come with build-in “accessibility” tools that read out loud everything on screen. The voices can be improved even more by downloading the “Siri enhanced” voice in the settings.
strong upvote. wars fought with asi could be seriously catastrophic well before they’re initiated by the asi.
🏆📈 We’ve created Alignment Markets! Here, you can bet on how AI safety benchmark competitions go. The current ones are about the Autocast warmup competition (meta), the Moral Uncertainty Research Competition, and the Trojan Detection Challenge.
It’s hosted through Manifold Markets so you’ll set up an account on their site. I’ve chatted with them about creating a A-to-B prediction market so maybe they’ll be updated when we get there. Happy betting!
Yes!
Is there a way to deal with the issue of there being multiple ROSE points in some games? If Alice says “I think we should pick ROSE point A” and Bob says “I think we should pick ROSE point B”, then you’ve still got a bargaining game left to resolve, right?
Anyways, this is an awesome post, thanks for writing it up!
My preferred way of resolving it is treating the process of “arguing over which equilibrium to move to” as a bargaining game, and just find a ROSE point from that bargaining game. If there’s multiple ROSE points, well, fire up another round of bargaining. This repeated process should very rapidly have the disagreement points close in on the Pareto frontier, until everyone is just arguing over very tiny slices of utility.
This is imperfectly specified, though, because I’m not entirely sure what the disagreement points would be, because I’m not sure how the “don’t let foes get more than what you think is fair” strategy generalizes to >2 players. Maaaybe disagreement-point-invariance comes in clutch here? If everyone agrees that an outcome as bad or worse than their least-preferred ROSE point would happen if they disagreed, then disagreement-point-invariance should come in to have everyone agree that it doesn’t really matter exactly where that disagreement point is.
Or maybe there’s some nice principled property that some equilibria have, which others don’t, that lets us winnow down the field of equilibria somewhat. Maybe that could happen.
I’m still pretty unsure, but “iterate the bargaining process to argue over which equilibria to go to, you don’t get an infinite regress because you rapidly home in on the Pareto frontier with each extra round you add” is my best bad idea for it.
EDIT: John Harsanyi had the same idea. He apparently had some example where there were multiple CoCo equilibria and his suggestion was that a second round of bargaining could be initiated over which equilibria to pick, but that in general, it’d be so hard to compute the n-person Pareto frontier for large n, that an equilibria might be stable because nobody can find a different equilibria nearby to aim for.
So this problem isn’t unique to ROSE points in full generality (CoCo equilibria have the exact same issue), it’s just that ROSE is the only one that produces multiple solutions for bargaining games, while CoCo only returns a single solution for bargaining games. (bargaining games are a subset of games in general)
Yeah, I agree that’s a weird way to define “high-dimensional”. I’m more partial to defining it as “when the curse of dimensionality becomes a concern”, which is less precise but more useful.
Also, coming up with your own ideas first can help you better understand what you find in the literature. I’ve found that students learn more readily when they come to a subject with questions already in mind, having tried to figure things out on their own and realized where they had gaps in their mental framework, rather than just receiving a firehose of new information with no context.
Perhaps try pursuing a number of proxy goals for short, pre-defined periods, while tracking whether each proxy goal is likely to be instrumental for reaching the terminal goal. Assessing the instrumentality of each proxy should be easier once you’ve started to get a sense of where each project can lead, and abandoning those that are clearly not going to be fruitful should be easier if you don’t plan on going all-in from the start.
Don’t be afraid to ask stupid questions. We often tend to refrain from asking questions that we predict would cause those more experienced to perceive us as idiots. Ignore those predictions. Even when the answer is obvious to everyone else, it will help the writer practice clarifying their ideas from a new perspective, which could even help them understand their own work better. And sometimes everyone else is just afraid to look like idiots, too.
Try steel-manning the best argument you can come up with against an authority’s position. Ideas that can withstand the harshest scrutiny are those worth keeping. Ideas that can be destroyed by the truth should be. Help the intellectual community filter the chaff from the wheat.
Good hypotheses always entail predictive models. If you can’t program it, you don’t really understand it.
I can’t think of anything else to add to this one.
Also, don’t wait until you’ve learned linear algebra, multivariable calculus, probability theory, and machine learning before starting to tackle the alignment problem. It’s easier to learn these things once you already know where they will be useful to you. Plus, we may not have enough time to wait on mathematicians to come up with provable guarantees of AI safety.
Bitcoin/blockchain force cohesion of a set of sims around the merkle tree, but you can still have many different sims that share variations of the same transaction history (they just differ then only in details).
But looking at it another way blockchains are also strong useful constraining evidence, so historical sims would always just recreate the same merkle trees. This leads to the interesting idea that you could attempt to preserve blockchain wealth post-sim … but of course you would just be sharing it with other versions of yourself, and potentially your simulators.
I remember being fascinated with the potential to help fix problems for people, especially eating problems, by adjusting the levels of various neuropeptides. I still think neuroscience ought to boldly tamper with those for the sake of making peoples’ lives less miserable, but now I’m much more fascinated by the implications for setting up a control system for RL-based AI. I also wonder if anyone has already made something like a simplified-business-logic model of the functions of the hypothalamus.
I’ve considered starting an org that was either aimed at generating better alignment data or would do so as a side effect and this is really helpful—this kind of negative information is nearly impossible to find.
Is there a market niche for providing more interactive forms of human feedback, where it’s important to have humans tightly in the loop with an ML process, rather than “send a batch to raters and get labels back in a few hours”? One reason RLHF is so little used is the difficulty of setting up this kind of human-in-the-loop infrastructure. Safety approaches like debate, amplification and factored cognition could also become competitive much faster if it was easier and faster to get complex human-in-the-loop pipelines running.
Maybe Surge already does this? But if not, you wouldn’t necessarily want to compete with them on their core competency of recruiting and training human raters. Just use their raters (or Scale’s), and build good reusable human-in-the-loop infrastructure, or maybe novel user interfaces that improve supervision quality.
Update today: Biogen/Eisai have reported results from Lecanemab’s phase 3 trial: a slowing of cognitive decline by 27% with a p-value of 0.00005 on the primary endpoint. All other secondary endpoints, including cognitive ones, passed with p-values under 0.01.
I’d argue that if they solved inner alignment totally, then the rest of the alignment problems becomes far easier if not trivial to solve.
But solving inner alignment may not be the easiest way to drive down P(doom), and not the best way for a given person specifically to drive down P(doom), so keeping your eyes on the prize and being ready to pivot to a better project is valuable even if your current project’s success would save the world.
Shortform #137 Meal Prepping & Rambutan
In pursuit of healthier eating, I prepared containers of seeds, fruits of various kinds, and vegetables that I can eat during meals & take to work for lunch. I also tried a new-to-me fruit called Rambutan, I like the flavor okay but it’s a little bland, and now have a container of that fruit to eat through too.
I meditated for 5 minutes, rowed for 5 minutes, and did 5 pushups tonight. Small continuous improvements, here we go!
It’s not that clear to me exactly what test/principle/model is being proposed here.
A lot of it is written in terms of not being “misleading”, which I interpret as ‘intentionally causing others to update in the wrong direction’. But the goal to have people not be shocked by the inner layers suggests that there’s a duty to actively inform people about (some aspects of) what’s inside; leaving them with their priors isn’t good enough. (But what exactly does “shocked” mean, and how does it compare with other possible targets like “upset” or “betrayed”?) And the parts about “signposting” suggest that there’s an aim of helping people build explicit models about the inner layers, which is not just a matter of what probabilities/anticipations they have.
I meant signposting to indicate things like saying “here’s a place where I have more to say but not in this context” etc, during for instance a conversation, so I’m truthfully saying that there’s more to the story.
Yeah, I think “intentionally causing others to update in the wrong direction” and “leaving them with their priors” end up pretty similar (if you don’t make strong distinctions between action and omission, which I think this test at least partially rests on) if you have a good model of their priors (which I think is potentially the hardest part here).
Thanks for the writeup. More people should take credit for things they thought about doing and didn’t because there’s lots of almost good ideas that are actually a waste of time and documenting those helps other avoid even bothering to investigate (unless they think they know something that others don’t, and then having such docs helps them make concrete why they still think it’s worth exploring an idea that others dismissed).
Late comment but I recently posted how human values arise naturally by the brain learning to keep its body healthy in the ancestral environment by a process that could be simplified like this:
These natural learning processes are supported by language and culture by naming, and suggestion behaviors make some variants more salient and thus more likely to arrive—but humans would pick up on the principles even without a pre-existing society—and that is what actually happens in certain randomly assembled societies.
https://www.lesswrong.com/posts/Di4bFP7kjoLEQLpQd/what-s-the-relationship-between-human-values-and-the-brain-s?commentId=Qh3MsAvqFemXMcYcz
This describes convergent value system of any mind, not only human one. So there is nothing specially human in it.
Correct.
The human aspect results from
the structure of the needs of the body and its low-level regulation (food, temperature, but also reproductive drives), and
the structure of the environment—how many other humans there are, how and where resources can be acquired.
That confirms what I have been getting around to over time: Human instincts and motivational systems are probably built from few elements. I used to think that there are hard-wired motivations/interests for specific types of sports or hobbies or people and how this could be coded for. But after reading Steven’s sequence, I look more for simple patterns like geometric features, hormonal triggers, or sensory triggers that could—after a lot of learning—give rise to such preferences. Maybe some sport is only interesting because it involves fast-moving objects in the visual field and the smell of grass…
I recognize that examples need to be legible, and I would also hope for deep internalization of pulling the rope sideways and investigation of policy areas that are less legible and therefore dramatically easier to shift on the margin.
To reach the boundary of what is known in your chosen field will require reading lots of papers, which will take (at least) several years. Doing research will also require implicit knowledge that is part of the field, but does not appear in papers.
Are you the kind of person who can spend several years reading papers without significant external help?
Where are you going to acquire the implicit knowledge, e.g., how to run experiments?
PhD students are the work-horses of academic research, and don’t have the power/money/experience to do anything other than tow the line. You have a degree of independence and experience that will deter many academics taking you on as a student.
Perhaps you can find an independent scientist to take you on as an apprentice.
Or: You could kick-start your research by applying your existing knowledge of (I assume) computing/software to cognitive issues in this field (see chapter 2)
FYI the creator of the market decided to resolve the market to N/A.
This means everyone’s mana is fully restored as if they never interacted with the market.
I have no idea about cognitive science in particular, but in math and physics there is not a single worthwhile contribution that was not done either by PhD holders or PhD students. Just not a thing that happens.
Edit for pedants: in the last 50 years at least.
Edit2: one possible exception is that anonymous 4chan poster: https://en.wikipedia.org/wiki/Superpermutation
I think I have two main complaints still, on a skim.
First, I think the following is wrong:
I think outer and inner alignment both go against known/suspected grains of cognitive tendencies, whereas shard theory stories do not. I think that outer and inner alignment decompose a hard problem into two extremely hard problems, whereas shard theory is aiming to address a hard problem more naturally. I will have a post out in the next week or two being more specific, but I wanted to flag that I very much disagree with this quote.
Second, I’m wary of saying “maybe we can get corrigibility” or “maybe corrigibility doesn’t fit into a utility function”, because this can map shard theory hopes onto old debates where we already have settled into positions. Whereas I consider myself to be thinking about qualitatively different questions and spreads of values I might hope to get into an AI.
This doesn’t make sense to me. It sounds like saying “liking yellow cubes is natural iff we can get a pointer to ‘liking yellow cubes’ within the AI’s goals.” That sounds like a thing which would be said if we had no idea how yellow cubes got liked, directly, and were instead treating liking-yellow-cubeness as a black box which happened to exist in the real world (e.g. how corrigibility, or the desire to help people, could be “pointed to” in a classic corrigibility hope).
I have more thoughts on this post but I don’t have time to type more for now.
Since the original draft I realized your position has “outer/inner alignment is a broken frame with mismatched type signatures which is much less likely to work than people think”, so this seems reasonable from your perspective. I haven’t thought much about this document and might end up agreeing with you, so the version I believe is something like “it’s not clear that my shard theory decomposition is substantially easier than inner+outer alignment is, assuming that inner+outer alignment is as valid as Evan thinks it is”.
Agree that I’m not being concrete about how corrigibility would be implemented. Concreteness is a virtue and it seems good to think about this in more detail eventually.
No. Have actually been working in/on science communication about psychedelics (as medicines) and have received both positive feedback from researchers and am collaborating with a few.
One thing that I like about being parallel to academia is that you can build things outside of the constraints. For instance am building a tracker of all RCTs which will update automatically when new papers are added. And made a map of research that visually shows where what is taking place.
In my personal experience, the intuition/grokking approach works better for mathematics, but the making-notes approach works better for programming.
This is not a fair comparison, because in case of programming, the notes were usually already made by someone else (on Stack Exchange), so I can use them without spending my time to write them. For deep understanding, I must usually do the work myself, because most people who write programming tutorials actually do not have the deep understanding. (Seems to me that most tutorials are written by enthusiastic beginners.)
I have a strong personal preference for the intuition/grokking approach, so I am not doing the optimal thing despite knowing what it is. But I have to admit that this is often a waste of time in programming. Before you spend the proverbial 10 000 hours learning some framework, it is already obsolete. Or if it becomes popular, sometimes the next version keeps the similarities, but changes how it works under the hood, so ironically the deeper knowledge that should be long-term sometimes becomes obsolete first.
In math, the stuff you learn usually remains true for a long time, so building intuitions pays off.
This was helpful in understanding this problem, thank you
This study has a table that lists the half-lives of many neuropeptides (see table 2):
https://febs.onlinelibrary.wiley.com/doi/full/10.1111/j.1742-4658.2011.08051.x
It would be interesting to compare the half-lives to the duration of behaviors, emotions, and moods associated with these neuropeptides. I only recognize the NPY mentioned above in the study.
Things that probably actually fit into your interests:
A Sensible Introduction to Category Theory
Most of what 3blue1brown does
Videos that I found intellectually engaging but are far outside of the subjects that you listed:
Cursed Problems in Game Design
Luck and Skill in Games
Disney’s FastPass: A Complicated History
The Congress of Vienna
Building a 6502-based computer from scratch (playlist)
(I am also a jan Misali fan)
Neat, thanks so much for these recommendations! I do of course follow 3b1b, and I already know some Category. But I’ll for sure check out all of the rest, which sound super cool!
I wrote a couple posts (that I haven’t reviewed recently) on this topic. You may find them informative, or at least amusing. Insightful points were made by commenters.
Was a terminal degree necessary for inventing Boyle’s desiderata?
Was a PhD necessary to solve outstanding math problems?
I will review these. Thank you for your input!
On reading them back over, here’s my updated take (partly informed by the fact that I am now an MS student and work with PhD students daily):
A PhD is two things: a way to support research, and a way to reward it with a sheepskin.
Budding researchers just tend to see academia as a marginally more useful and attractive place to build their career than the next best alternative. The fact that a PhD program is mostly the only way to get the most widely recognized signal of being a qualified researcher is a bonus. If they’re going to do weird groundbreaking research anyway, why not do it in an environment that’s relatively open to it, and that will provide them with a credential at the end of the process?
If I was doing this over again, then, I might reframe the question. Instead of asking “is a PhD necessary,” I might ask questions like:
“Why do excellent early-career researchers so often see a PhD as attractive?”
“What are some of the practical barriers to doing a certain type of research outside the academic system?”
“What is the next most logical alternative to building a research career without getting a PhD?”
“Why and how do some excellent early-career researchers choose to build their research career outside the normal academic system?”
“What exactly does it mean to ‘contribute’ to a field?”
As an example, if you’re going into cogsci, you might need to run experiments on people or animals. When we do animal research in our BME lab, we have all kinds of support and regulation and procedures for making sure it doesn’t create a fiasco with government, administrators, or activists. Outside the academic system, I expect that trying to do animal research would be extremely difficult or impossible, not to mention publishing it and getting it taken seriously. It’s not just a question of whether the research findings were any good. It’s the perception. Academia is as sensitive to perception as anybody, and you want your research to not just be ethical and correct, but avoid looking ethically or epistemically suspect.
As a less charged example, I at one point was trying to figure out how to visualize a DNA ladder in an agarose gel without staining the rest of the DNA. I proposed cutting off the ladder band, staining that, and then jigsawing it back onto the rest of the gel. My mentor told me “if you do that, you go to science JAIL.” His objection wasn’t that this would fail to tell us the information we needed to know. It was that it would look really bad in light of all the replication issues specifically with fraudulent gels.
Any way you slice it, you’ll have to figure out a way to do high-quality research while making it look good enough for others to take seriously. This is probably a lot easier with an academic pedigree and institutional support. Another way of putting it might be “if you were enough of a genius to not need a PhD, you’d probably know that about yourself already!” Not a diss − 99.99% of researchers are probably in this category, myself included.
Excellent! Maybe there’s a way to pitch this for a Black Mirror episode.
I would need an agent for that. I am in the process of sending query letters to agents specializing in the genres I write.
Requesting beta readers for “Unit Test Everything.”
I have a new post on how to be more reliable and conscientious in executing real-world tasks. It’s been through several rounds of editing. I feel like it would now benefit from some fresh eyes. If you’d be willing to give it a read and provide feedback, please let me know!
Isn’t this more like an onion test for… honesty?
Integrity is broader.
I agree with this.
Title changed!
Is dath ilan worried about constructing an AGI that makes the future 99% as good as it could be, or a 1% chance of destroying all value of the future?
I had assumed the first—they’re afraid of imperfect-values lock-in. I think it’s the “not to the problem of preventing complete disaster” phrase that tipped me off here.
I hate to be the bearer of bad news, but the methodology in the study you linked is just terrible.
99% of candidate gene studies do not replicate, and the first study you link uses an extremely error-prone P < 0.05 threshold to determine that orexin has an effect.
It may still have an effect, but there’s no way to really know this without some kind of large sample GWAS.
The proper way to do this would be to get research access to a large genetic database like 23&Me or UK BioBank and pair that data with info from sleep trackers like Apple Watch, FitBit, or other devices to see which genes are involved in sleep duration and how large the effect of each is.
Depending on the sparsity of the trait, you’d probably need somewhere between 100k − 1 million genotype-phenotype pairs to make a good predictor.
Lastly, in the “Possible Actions” section you should include embryo selection. Genes obviously play some role in sleep needs. If at least some of them have no relevant downsides, selecting for those genes in embryos would probably be quite cost-effective. In fact, you could further reduce the odds of any unintended side-effects by selecting not just for shorter required sleep duration, but also for longer quality-adjusted lifespan.
When doing gene studies if you look at all genes and see which one’s have P < 0.05, that gets you a lot of false positives. That’s however not what they did in the linked paper.
The linked paper looks at one gene because previous papers identified that gene as being significant in humans. The paper says that the gene has significance for sleep in mice.
The argument doesn’t rest on a single experiment. I agree that it would be desirable to have 23&Me cooperate with Apple or Google to find genes that affect the metrics that the Apple watch / Fitbit measures. That’s helpful to have a good overview of all the mutations that affect sleep length.
Yes, that’s a valid action. I was thinking about actions that actually result push the field forward. You could have benefits for the child from such embryo selection but I wouldn’t expect it to lead to knowledge generation.
Can you link the earlier studies showing the significance of BHLHE41 in humans? When I Google it all I find are other candidate gene studies with small sample sizes.
https://www.science.org/doi/abs/10.1126/science.1174443 seems to be the first paper. It does have a small sample size and thus is only able to produce candidates, but it results in the later paper not randomly searching over all possible gene mutations.
My main point is that there are papers that made independent observations and thus arguments that a single paper doesn’t demonstrate the effect doesn’t hold. I didn’t copy the exact minute numbers that the EA cause report had because I was unsure about the exactness of the data.
I’m not sure if you have read the story of 5HTTLPR and all the independent studies which found it to have an effect, but if you haven’t you should.
In the case of orexin, my argument doesn’t just rest on the DEC2 gene.
The experiments that improved performance in sleep-deprived rhesus monkeys happened before the discovery of the link between the DEC2 mutation and orexin.
The observations in Astyanax mexicanus seem independent from my perspective. Attempts to make Astyanax mexicanus a model organism aren’t driven by sleep researchers but because it’s interesting for studying evolution.
Orexin deficiency causing Narcolepsy type 1 is independent of any findings about the DEC2 gene as well.
As far as the linked post of Scott goes, it says nothing about experiments on animals other than humans. Gene knockout studies in mice and Drosophila seem to me like a pretty good way to measure the influence of a gene.
As always, not investment advice. There are signs that a volatility spike is imminent, which often coincides with a market drop. I have reversed my usual short vol position and bought tail insurance (e.g. OTM puts). Remember vol can fall just as quickly. How long a spike lasts depends on how high it goes.
I have thoughts, but they are contingent on better understanding what you mean by “types” of hidden information. For example, you used “operating a cocaine dealership” as a “type” of information hidden by the question about health information. Operating a cocaine dealership is not a health matter, except perhaps indirectly if you get high on your own supply.
To further illustrate this ambiguity, a person might be having gay sex in a country where gay sex is criminalized, morally condemned, and viewed as a shocking aberration from normal human behavior. It does not seem to me to be out of integrity for a gay person to refrain from telling other people that they have gay sex in this (or any other) context.
Where this becomes problematic is when the two people have different expectations about what constitutes reasonable expectations and moral behavior. If we give free moral license to choose what to keep private, then it seems to me there is little difference between this onion model and an algorithm for “defining away dishonesty.” One can always justify oneself by saying “other people were foolish for having had such unreasonable expectations as to have been mislead or upset by my nondisclosure.” If “outsiders” are expected to be sufficiently cynical, then the onion model would even justify outright lying and withholding outrageous misbehaviors as within the bounds of “integrity,” as long as other people expected such infractions to be occurring.
It short, it seems that this standard of integrity reduces to “they should have known what they were getting into.”
As such, the onion model seems to rely on a pre-existing consensus on both what is epistemically normal and what is morally right. It is useful as a recipe for producing a summary in this context, but not for dealing with disagreement over what is behaviorally normal or right.
I agree. For me it’s more of a characterization of honesty, not integrity (even though I consider honesty an aspect of integrity). Perhaps we should change the name.
I think part of my answer here is “The more a person is a longterm trade partner, the more I invest in them knowing about my inner layers. If it seems like they have different expectations than I do, I’m proactive in sharing information about that.”
Yes, I agree the onion model can be a way to think about summarization and how to prioritize information sharing. I just don’t see it as particularly helpful for integrity.
And see also also Anna Salamon’s How to learn soft skills, which could possibly be helpful here
See also Raemon’s very similarly named post, which is also good and covers pretty different ground!
I indeed did a doubletake when I saw this post, thinking ”...did… didn’t I write that?”
You need to observe more (or better). Most humans care about some aspects of a few other humans’ experience. And care quite a bit less about the general welfare (with no precision in what “caring” or “welfare” means) of a large subset of humans. And care almost none or even negatively about some subset (whose size varies) of other humans.
Sure, most wouldn’t go out of their way to kill all, or even most, or even a large quantity of unknown humans. We don’t know how big a benefit it would take to change this, of course—it’s not an experiment one can (or should) run. We have a lot of evidence that many many people (I expect “most”, but I don’t know how to quantify the numerator nor denominator) can be convinced to harm or kill other humans in circumstances where those others are framed as enemies, even when they’re not any immediate threat.
There are very common human values (and even then, not universal—psycopaths exist), but they’re things like “working on behalf of ingroup”, “suspicious or murderous toward outgroup”, and “pursuing relative status games among the group(s) one isn’t trying to kill”.
Yeah, I think the “working on behalf of in-group” one might be rather powerful, and I was aware that this is probably a case where I just interact mostly with people who consider “humans” as the ingroup. I don’t think the share of the population who shares this view is actually as important as the fact that a sizeable number of people hold this position at all. Maybe I should have called it: I do care about everyone to some extent. Is that what we want to achieve when we talk about alignment?
Aaand I forgot to come back and check the site before the day was over. Sigh…
Well that’s nice. Does anyone have a general sense of where we are with detection and launch windows? I’m somewhat hoping that we already have enough monitoring capability to detect potential existential risk asteroids in time, but very unsure and don’t have time to research it.
I’ve got a very good intuition about foo-vectors from Let’s remove Quaternions from every 3D Engine post.
Actually, I think that post is probably what triggered me to write this originally, and I forgot that by the time I wrote it (or I would have added a link.) Thanks for the reminder!
I haven’t read that deeply into this yet, but my first reaction is that I don’t see what this gains you compared to a perspective in which the functions mapping the inputs of the network to the activations of the layers are regarded as the network’s elementary units.
Unless I’m misunderstanding something, when you look at the entire network f(x), where x is the input, each polytope of f(x) with its affine transformation corresponds to one of the linear segments of f(x). Same with looking at, say, the polytopes mapping layer 0 to layer L. You can just look at fL(x0), where fL are the activations in layer L, and each linear segment of that should correspond to a polytope.
However, I don’t really see how you’d easily extend the polytope formulation to activation functions that aren’t piecewise linear, like tanh or logits, while the functional analysis perspective can handle that pretty easily. Your functions just become smoother.
In the functional analysis view, a “feature” is a description of a set of inputs that makes a particular element in a given layer’s function space take activation values close to their maximum value. E.g., some linear combination of neurons in a layer is most activated by pictures of dog heads. But there’s a lot more to know about a function f than what max({f(x)|x∈X}) is.
When you scale up a particular feature in a layer past its activation range in the training dataset, you are effectively querying the functions in subsequent layers outside the domains they’ve been trained to fit well. Instead of checking how many polytope boundaries you crossed, you can just check how much f′(x) varied between your start and end points.
Scaling up some of the activations in a layer by a constant factor means you’re increasing the norm of the corresponding functions, changing the principal component basis of the layer’s function space. So it shouldn’t be surprising if subsequent layers get messed up by that.
Thanks for your comment!
Extending the polytope lens to activation functions such as sigmoids, softmax, or GELU is the subject of a paper by Baleistriero & Baraniuk (2018) https://arxiv.org/abs/1810.09274
In the case of GELU and some similar activation functions, you’d need to replace the binary spine-code vectors with vectors whose elements take values in (0, 1).
There’s some further explanation in Appendix C!
This, indeed, is the assumption we wish to relax.
Agreed!
There are many lenses that let us see how unsurprising this experiment was, and this is another one! We only use this experiment to show that it’s surprising when you view features as directions and don’t qualify that view by invoking a distribution of activation magnitude where semantics is still valid (called a ‘distribution of validity’ in this post).
I’m a pretty bad chess player (~1500 ELO) and I can play bullet games while sleep deprived without much loss in skill. I think the system 1 pattern matching is relatively unimpaired while the system 2 calculating is very impaired. In bullet calculating doesn’t help much though.
A GM playing their instant move with no calculation can crush a master-level player, and I can crush a novice playing my instant move. It’s all about system 1 :)
After this comment there was a long thread about AC efficiency.
Summarizing:
I said: “In practice I think the actual efficiency loss relative to a 2-hose unit is more like 25-30%” (For cooling from 85 to 70.)
John said that this was ridiculous.
After the dust settled, our best estimate on paper is 40% rather than 25-30%.
The reason for the adjustments were roughly:
[x2] I estimated exhaust temperature at 130 degrees, but it’s more like 100 degrees if the indoor air is 70.
[x1/2] I thought that all depressurization was compensated for by increased infiltration. But probably half of depressurization is offset by reduced exfiltration instead (see here)
[x3/2] I only considered sensible heat. But actually humidity is a huge deal, because the exhaust is heated but not humidified (see here)
John also attempted to measure the loss empirically, but I’d summarize as “too hard to measure”:
With 1-hose the indoor temp was 68 vs 88 outside, while with 2-hose the indoor temp was 66 vs 88 outside (using the same amount of energy).
We both agree that 10% is an underestimate for the efficiency loss (e.g. due to room insulation, other cooling in the building, and the improvised 2-hose setup).
I don’t think we have a plausible way to extract a corrected estimate.
I endorse this summary.
My comment did not assume the physical. Entropy is not a physical concept but a statistical one. It can be used to measure the complexity of any system, physical or not.
Any real system, whether physical, spiritual, or anything else, will necessarily be computable. Even the mind of God would have to be technically computable, although you could argue that it would require more computational power than could possibly fit in the physical universe.
Again, there is no magic. Nothing is intrinsically mysterious. Mystery is a symptom of a finite, ignorant mind. Whatever realms might exist beyond this physical universe are necessarily logically coherent and explainable in principle. And that applies to any subsystem of any reality, whether consciousness, life, or anything else.
One thing I’ve found useful is to make sure I identify to the supplier what specifically I need about the product I’m ordering—sometimes they have something similar in stock which meets my requirements.
This is a bit like how Scientology has tried to spread, but the E-hance is much better than the E-meter.
This is a great collection of tips! I think it’s also worth explicitly noting that most of these strategies involve slowing down other people’s orders, and many involve more inconvenience/stress for the sellers, so it’s important to weigh this tradeoff.
This year Petrov day almost sneaked past me. This strikes me as weird on account of the biggest proxy war since the 80s being underway, putting us closer the same stakes in realspace.
This is great! I really like your “prediction orthogonality thesis”, which gets to the heart of why I think there’s more hope in aligning LLM’s than many other models.
One point of confusion I had. You write:
I don’t think I agree with this conclusion (or maybe I don’t understand the claim). I agree that myopic incentives don’t mean myopic behavior, but they also don’t imply that actions are chosen myopically? For instance I think a language model could well end up sacrificing some loss on the current token if that made the following token easier to predict. I’m not aware of examples of this happening, but it seems consistent with the way these models are trained.
In the limit a model could sacrifice a lot of loss upfront if that allowed it to e.g. manipulate humans into giving it resources with which to better predict later tokens.
Depends on what you mean by “sacrificing some loss on the current token if that made the following token easier to predict”.
The transformer architecture in particular is incentivized to do internal computations which help its future self predict future tokens when those activations are looked up by attention, as a joint objective to myopic next token prediction. This might entail sacrificing next token prediction accuracy as a consequence of not optimizing purely for that. (this is why I said in footnote 26 that transformers aren’t perfectly myopic in a sense)
But there aren’t training incentives for the model to prefer certain predictions because of the consequences if the sampled token were to be inserted into the stream of text, e.g. making subsequent text easier to predict if the rest of the text were to continue as expected given that token is in the sequence, because its predictions has no influence on the ground truth it has to predict during training. (For the same reason there’s no direct incentive for GPT to fix behaviors that chain into bad multi step predictions when it generates text that’s fed back into itself, like looping)
Training incentives are just training incentives though, not strict constraints on the model’s computation, and our current level of insight gives us no guarantee that models like GPT actually don’t/won’t care about the causal impact of its decoded predictions to any end, including affecting easiness of future predictions. Maybe there are arguments why we should expect it to develop this kind of mesaobjective over another, but I’m not aware of any convincing ones.
Got it, thanks for explaining! So the point is that during training the model has no power over the next token, so there’s no incentive for it to try to influence the world. It could generalize in a way where it tries to e.g. make self-fulfilling prophecies, but that’s not specifically selected for by the training process.
Yup exactly! One way I sometimes find it to helpful to classify systems in terms of the free variables upstream of loss that are optimized during training. In the case of gpt, internal activations are causally upstream of loss for “future” predictions in the same context window, but the output itself is not casually upstream from any effect on loss other than through myopic prediction accuracy (at any one training step) - the ground truth is fixed w/r/t the model’s actions, and autoregressive generation isn’t part of the training game at all.
The biggest yet least discussed problem in the world today is the ever tightening web of monstrously evil, defective, and barely usable interfaces, most notably software. Efforts to make UIs seem “simpler” on a lowest-common-denominator super-shallow level are the greatest curse of the past two decades. Every attempt to even mention this problem here leads to a virtual shadowban. My proposed initial solution of requiring every program to have a single page text list of ALL options (no submenus) triggers even more hate.
strong downvote, strong agree: this is offtopic, but a perfectly reasonable topic to start a thread about. it doesn’t seem like a bottleneck for the world to me, though, because nobody is trying to remove the CLI, and in fact even microsoft has been putting effort into ensuring that good CLIs are available on windows. if you’d like to discuss it, I suggest creating a post about it; I expect it to get little involvement, because as I said, I simply don’t agree that it’s catastrophic and don’t find this to be important compared to AI and human inter-being friendliness/alignment. Since you commented that you feel ignored, I figured I’d comment on why.
For this outcome, everything below has to be true:
Molecular nanotechnology is impossible.
Maintaining infrastructure using only robots is impossible / highly impractical.
It is not possible to genetically engineer a creature that is a vastly better maintenance worker than human.
Given that, 95% claim is very overconfident.
Double-masked does mean double-blinded and comes from the first studies literally using masks for blinding. Jeremy Howick does advocate in The Philosophy of Evidence-Based Medicine that it’s better to use the term double-masked because people often get confused when thinking about blinding.
People frequently think that blinding means a person doesn’t know whether or not they get the treatment, which is not how it’s operationalized. Researchers, who actually care about whether or not people know whether they get the treatment, ask their patients whether or not they think they get it.
I think one can observe what other people/agents “want” in a framework that you might call materialist. Maybe I am wrong. Maybe once you act on behalf of other agents, this includes some value judgments. For example, some negative utilitarians might argue that pressing the button is an altruistic act.
I don’t think what happened through ancestor history was moral. I just observe it seems to have “worked”. My question concerns if we could achieve the same with regards to AI having human compatible goals.
Thanks for this John, I found it really useful!
Have you seen any research or discussion on whether independent researchers are more likely to develop unique moral views (because they aren’t part of a centralized entity), and therefore bring diversity to the global research effort?
That is something I (and I think most others) expect, but mostly on priors.
It was spectacular to watch!
Also, what x-risk research do you consider underfunded? AI risk seems to be well funded lately, so is climate, nuclear and bio.
Risk from supervulcanos seems to be an underfunded cause area.
Kinetic bombardment is very similar to nuclear risks and seems to be without any publically visible funding.
Biorisk still seems to have a lot of underfunded areas.
Have we spent $40M for pandemic prevention yet?
George E. Brown, Jr. Near-Earth Object Survey Act (H. R. 1022 in 109th Congress) is a remarkable document. Here are some quotations:
It is generally agreed that PhD is only worthwhile for those interested in staying in academia, at least for most STEM fields. Even then, it is a long slog and a lot of competition in the fields I am familiar with before you get that coveted assistant prof position, likely being stuck for the rest of your life at one place, underpaid relative to industry and forced to teach disinterested masses, writing grant proposals and navigating departmental politics. That said, I would have gladly stayed in academia after my PhD, were it a possibility in my case. If you have plans on working in the industry eventually, why not start now?
Having a Ph.D. confers relatively few benefits outside of academia. The writing style and skills taught in academia are very very different from that of industry, and the opportunity cost of pursuing a Ph.D. vs going into software engineering (or something similarly renumerative) is in the hundreds of thousands of dollars.
I would suggest that if you don’t know exactly what you want to do with your life, you would be well-suited to doing something that earns you a bunch of money. This money can later be used to finance grander ambitions when you have figured out what you want to do.
I’ll turn this question around on you: why is a Ph.D. the best way of accomplishing what you want to do?
As to the drudgery of office work—“office work” is, i think, a false category. I spent hours of unbearable tedium performing repetitive reactions in lab during my PhD, and my current cushy Microsoft engineering job is enormously more creative and interesting while paying approximately 10x as much. For someone with the smarts to get a ph.d., retraining into engineering is very, very easy.
One other generally undiscussed aspect of the working world is that, for a number of reasons, your employers mostly treat you with respect roughly proportional to your salary. Ph.D.s, consequently, are often treated very poorly. This probably contributes to their poor mental health, as documented elsewhere.
There’s also the ALTER prize for progress on the learning-theoretic agenda.
The mental health of PhD students is the main reason I’m not interested in doing one. There are lots of surveys showing terrible symptoms, e.g. this one where 36% of PhD students had sought treatment for depression or anxiety related to their PhD. That’s nearly 5x higher than the baseline 8% of Americans. You can find many similar surveys, and my anecdotal experience of asking PhD students whether they’re enjoying the experience has only confirmed this dismal picture.
https://www.nature.com/articles/d41586-019-03489-1
Hm. I wonder about selection effects. PhD students might be more aware of mental health issues, and PhDs often come with some level of access to mental health services and occur in cities where those services are available. PhD students are also typically at an age when the baseline rate of anxiety and depression is significantly higher—I see figures in the 20% range (although this is for having anxiety, not seeking treatment). It’s natural that whatever anxiety and depression they’d feel would be centered on their main focus in life.
I wouldn’t be too surprised if anxiety and depression are somewhat elevated among PhD students as compared to their peers with a similar socioeconomic status, age, and IQ, but I suspect that that 36% vs. 8% comparison might be somewhat misleading.
There is no evidence that the structure and function of the human brain (along with its physical embodiment) is insufficient to explain all aspects of human intelligence and consciousness. Even if it was discovered that the physical brain fell short of accounting for all cognitive phenomena, and even if no other known physical system or process could account for it, and even if an immaterial soul was actually discovered somehow, it still wouldn’t imply that human intelligence could not be emulated.
There is no magic. All real phenomena are causally computable. An immaterial soul would be no exception. Even a non-physical conscious system would necessarily possess inner computational mechanisms that are themselves not conscious (e.g., “spiritual synapses”).
The only cop-out would be if the immaterial soul was performing necessary computations so vast that they could not be performed on a physical computer the size of Earth within a human lifespan (therefore necessitating a non-physical computational substrate that is not bound by physical constraints on memory density and processing speed). There is no evidence of this.
You have too high a view of human-level intelligence and consciousness. We are in fact intellectual bottom-feeders in the vast ocean of possible mind-space. And that’s even accounting for physical limits on computational power.
Intelligibility is not evidence for intelligent design. It is evidence for a low-entropy system, which has a far higher prior probability than a supernatural Intelligentsia.
https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
Thanks for the challenge! Was this inspired by spellcaster university?
Nope. (Though since both that game and this one are weird administration-centric takes on Harry-Potter-style magical schools, I imagine there may have been some convergent evolution.)
This seems in line with your position, but I want to reply so people won’t conclude “And coinrun experiments don’t tell you important things.” I think the more interesting question for that experiment is “how will the agent generalize? Can we predict it in advance? In what ways do we systematically mispredict, and why?”
(And, roughly speaking, there are a range of possible algorithms by which the agent can generalize, so it’s way more than one bit. I got way more out of that paper by asking the authors for hundreds of videos for different training settings. Probably will mention the results in a post, soon.)
Evolution actually solved alignment (approximately) on two levels: aligning brains with evolution’s goal of inclusive genetic fitness—to be fruitful and multiply - (even though the easy/natural goal attractor for intelligence is empowerment), and aligning brains with other brains because the unit of selection (genes) is spread across individual bodies (the disposable soma). There are many obvious failure examples, but that’s hardly surprising because evolution proceeds by breaking things.
The first type of alignment (towards genetic fitness) gives rise to lust, sexual attraction, parenting desires, etc. The second form of alignment (to other humans) - more relevant for AGI—manifests as empathy, love, and altruism.
I’m particularly impressed by “The Floating Droid”. This can be seen as early-manifesting the foreseeable difficulty where:
At kiddie levels, a nascent AGI is not smart enough to model humans and compress its human feedback by the hypothesis “It’s what a human rates”, and so has object-level hypotheses about environmental features that directly cause good or bad ratings;
When smarter, an AGI forms the psychological hypothesis over its ratings, because that more sophisticated hypothesis is now available to its smarter self as a better way to compress the same data;
Then, being smart, the AGI goodharts a new option that pries apart the ‘spurious’ regularity (human psychology, what fools humans) from the ‘intended’ regularity the humans were trying to gesture at (what we think of as actually good or bad outcomes).
For sexually reproducing species, evolution will disfavour genes that lead to individuals slaughtering others of their species for little or no reason. Slaughtering members of the opposite sex obviously reduces mating opportunities. Slaughtering members of the same sex reduces the mating opportunities of their opposite-sex children.
This obviously hasn’t completely prevented humans from slaughtering both particular individuals and entire populations of humans of a different tribe. But this isn’t what always happens. It could be a lot worse.
Maybe it is worse for asexual species (mitigated by slower evolution without sex), perhaps explaining why at least for vertebrates asexuality is rare, and asexual species seem to not last long.
I chose exactly the wrong D&D.Sci to decide to not try building a model on, and instead try to solve just by eyeballing simple scatterplots.
Despite coming in “last place” I’m pretty happy with my results!
I think this was a perfectly reasonable setup. Even more so given that without any straightforward scenarios people won’t think “what if it’s just straightforward though”.
I thought many times during eyeballing “look, probably no one else has tried to build a model using all combos of mins and maxes of stats as features, just do it” but I stuck to my guns. For, uh, reasons. Presumably.
My biggest mistake, as I see it, was failing to generalize from “I have several max-of-two-formulas and a Dragonslayer distribution that is obviously a mix of two which I can’t seem to resolve into something nice, probably all of them are max-of-two-formulas, let’s see if I can refactor things to look like that and get a better idea of what Dragonslayer’s two might look like if the rest do factor well”.
Thanks for these tips, I can probably put them to good use! I’m curious though, what’s so special about custom Japanese beds that you needed them quickly?
I was setting up a retreat venue, and they were pretty weird and special beds—such that if they would’ve actually worked, it would’ve pivoted our strategy for setting up the space in a somewhat major way.
Do you have a link to the bed in question?
Shortform #136 Sugar* Considered Harmful
= sugar in excess of what one would consume normally by eating fruits & vegetables and maybe 1 square of dark chocolate.
I splurged (everything else were essentials / healthy food) on one item this weekend from Trader Joe’s: Pumpkin Spice Cinnamon Rolls. Yes they were tasty, duh. After consumption...now I feel physically ill: almost sweating and general discomfort, anxiety, elevated heart rate, trouble focusing, restlessness, etc. Oh! This is not an isolated set of bad feelings, these bad feelings occur every single time I eat sugar in excess of eating a few particular sweet fruits. (I have a great A1C level, for those who are worried this is due to something abnormal; I get full bloodwork done about 4 times a year [last time was late summer] and those levels have consistently been excellent)
I don’t care for a fully general rule of sugar = always bad...but from my experiences...I think it’s best for me to continue cutting unnatural amounts of sugar out of my life, because wow do I feel great when I do cut that out. The last few months I’ve steadily reduced my sugar consumption and now even eating two or three oreos feels intense and can trigger the bad feelings...eating a few cinnamon rolls at once was definitely a mistake. I intend to learn from this experience and the other bad experiences I’ve had with consuming unnatural amounts of sugar these last few months: stop consuming this particular drug [sugar], it’s bad for me and makes me feel bad.
Thank goodness I like the taste of >=90% dark chocolate and even pure baker’s chocolate. Figs are pretty great too, they almost taste too sugary but they don’t make me feel bad so I guess they are fine. I can cut out unnatural amounts of sugar, feel better, and still get to enjoy tasty dessert-style foods that I like (very dark chocolate, figs, sauteed onions, berries, and more), it’s a win win!
I can successfully access the front page by using the highly sophisticated method of inspecting the front page and deleting the div element that corresponds to the Petrov Day loss screen.
It might be a good idea to make the front page getting taken down have more serious of an impact, as the time cost of clicking “inspect” and deleting the associated div element is pretty low, so the overall cost of submitting the launch codes is likely substantially reduced by this way of accessing the front page remaining available after a launch.
If it takes a minute to hide the div and 10 AI safety researchers spend that minute then the mean cost of the home page being down is probably somewhere between zero and a billion lives, depending on your assessment of AI risk and the effectiveness of AI safety research and the best use of the future lightcone.
I think that’s actually fine. Most people won’t do that, but if for some reason the front page is really important for someone then it’s good that there’s an option to still access it.
My impression of the LW crowd is that the median person on the forum would have enough technical expertise to figure that out, but maybe that impression is poorly calibrated? I’m not sure.
Wait, is it down? I can see the front page fine without using any workarounds
Petrov Day ended two hours ago.
Ahhh I see, I missed that it was 8pm-8pm, not midnight to midnight, thanks.
Downvoted for clickbait title that makes a claim about every person in the entire world except the author.
I continue to expect that I will prefer to control my computer with formal grammars—I have spent significant time using caster speech recognition with dragon, and I’m sure I’ll keep doing so with openai whisper. nobody has ever beaten the CLI and nobody ever will [edit: to clarify, I believe it is a weakness of current ai that it struggles to take formal grammar seriously].
now, if an AI could automatically generate interfaces that standardize messy UIs into coherent formal grammars that fit comfortably in a CLI workflow, that would be amazing...
also, I want a CLI that shows me what commands are available at any step.
yeah, eventually keyboard clis will be beaten, but even with whisper I expect to sometimes prefer keyboard. it’s just that hard to beat CLIs.
I’d be interesting in hearing more about how those two paragraphs are connected. How do we go from 1) lobbying for and getting specific policies implemented to 2) correcting the deep dysfunctions in our government?
I’m also not clear if this is implied or not, but is (2) supposed to mitigate existential risks? If so, I’d be interested to hear more about the specific thoughts and plans surrounding that.
I liked this one a lot. I imagined that ‘train a linear classifier’ would be the next step, but didn’t do it due to laziness: it looks like that would probably have worked.
I do feel like my approach should have worked worse than it did—I did most of my evaluation by ignoring the scores and looking only at your historical classifications, and the one place where I let scores guide me into overriding my initial model (moving Student P from Humblescrumble to Serpentyne) it turned out I was incorrect and the initial model would have scored better (oops).
The front page going down doesn’t actually make people who want to check the latest posts unable to due so; it’s so easy to circumvent that I think the front page going down is nearly costless
That said I do think the symbolic meaning is neat
I haven’t given this a thoroughly read yet, but I think this has some similarities to retroactive public goods funding:
https://harsimony.wordpress.com/2021/07/02/retroactive-public-goods-funding/
https://medium.com/ethereum-optimism/retroactive-public-goods-funding-33c9b7d00f0c
The impact markets team is working on implementing these:
https://impactmarkets.io/
Going by figure 5, I think the way to format climate contingent finance like an impact certificate would be:
‘A’ announces that they will award $X in prizes to different project based on how much climate change damage a project averts
‘B’ funds different projects (based on how much damage they think each project will avert in the future) in exchange for a percentage of the prize money
The fact that you highlight Trump as particularly authoritarian and keeping him out of office in 2024 as a priority shows that this is a hopelessly mindkilled effort.
No reason to think he’s better or worse than other politicians, but he’s certainly very different.
In a world of almost omnimalevolent conformity, it’s strange to see the possibility that things could be different.
This is an opinion that even some people who Trump appointed himself hold. Geoffrey S. Berman is the latest person who described it.
Trump right now makes the question of whether or not someone agrees that the 2020 election was fraudulent as the test of whether or not he supports the person. That’s a filter that will allow him to appoint people in a way that reduces the pushback that he got in his last administration when he tried to do authoritarian things like pushing Berman.
I’m curious what examples of highly authoritarian threats you see besides trump, and how trump ranks among them
Interesting. Can you give us a sense of how much those asks (offer to pay for the extra labour) end up costing you?
You mean money or social capital?
I was thinking of money. :)
Hard to give a general answer, but I think 2x someone’s normal salary (especially if it’s cash) is usually quite sufficient to get the job done, and kind of reliably has helped me in the past when I’ve try to find people happy to work night shifts
Both would be interesting.
I guess the process would be to pass it on to whichever cababilities researchers they trust with it. There would be a few of them at this point.
So, why not go straight to those researchers instead of MIRI? Because MIRI are more legible responsible intermediaries I guess.
Strongly agree about the existence of the problem. It’s something I’ve put a bit of thought into.
One thing I think could help, in some cases, would be to split the market definition into
the question definition, and
the resolution method
And then specify the relationship between them. For example:
Question: How many reported covid cases will there be in the US on [DATE]?
Resolution method: Look at https://covid.cdc.gov/covid-data-tracker/ a week after [DATE] for the reported values for [DATE].
Resolution notes: “Whatever values are reported that day will be used unconditionally.” or “If the values change within the following week, the updated values will be used instead.” or “The resolution method is a guideline only; ultimately the intent of the question, as interpreted by the question author, overrides.”
This will only solve a subset of ambiguous resolutions, but I think it would still be a big help to spell some of these things out more clearly.
David here from Manifold.
Really cool to see you taking the time to present your thoughts in a way that is constructive whilst not holding back with your criticisms. A lot of people would have just gotten frustrated with their experience on the site and have just moved on, so thank you for this post! We definitely think about these things a lot as we design Manifold and this post could definitely have an impact on where we lean within tough design problems.
First off, let me briefly discuss why we are bullish on using a creator-based resolution mechanism.
Creator reputation. This is definitely the most important point. Regular users quickly learn which markets are run by credible creators who know to set clear criteria. And in the rare instance an unexpected variable occurs making the resolution less clear, these creators will often defer to what the comments agree should be the correct resolution. Admittedly this is definitely not obvious for new users who are interacting with our markets for the first time and are often times trading in good faith that all the markets are reliable. We are aiming to get a reputation system set up at some point, but we haven’t had the capacity to do so yet
Creator resolution facilitates types of questions that otherwise couldn’t exist.
Fun, personal markets that often only the creator will know the results. eg. this one from James Medlock.
Subjective/recommendation markets.
Creators resolving works >95% of the time. For the majority of markets, there are no issues. Very rarely does a creator’s internal bias or ulterior motive affect the resolution. It’s unfortunate that the first market you interacted with may be one of the minority and we will be more conscious about ensuring big/featured markets are reliable. I strongly believe if you were to use the site more that you would find a lot of your concerns are exacerbated by your experience with an outlier.
Interpretation of a result is often necessary. As others have alluded to in the comments, there is often conflict between a market that is useful and answers what you want to know vs one that is extremely well-defined in advance. It will always be the case that questions come up where someone has to make a call and it makes sense that the person creating the market is the best person to do so. We have heard some good suggestions regarding crowd-sourcing resolutions so might be open to possible types of implementations of that.
Now that that’s out of the way, let’s discuss some of the specific criticisms you outlined. The following is more a reflection of my own thoughts and hasn’t necessarily been discussed in depth with the team.
The market probability becomes more reflective of the creator’s tendencies and bias and not the event itself.
I will agree that this is definitely true with some of our markets. But I actually think that is okay! One thing that sets Manifold apart from other prediction markets is that we support a wide range of markets that is only possible by giving creators full autonomy. I think it is a good thing that each individual creator is expressed in their market. Not all markets have to provide actionable, informative data even if this may not align with what is traditionally sought after with forecasting. Just to be clear though, one of our primary goals is to facilitate markets that provide actionable and informative discussion and data. But, I suspect as the site grows that it will become more obvious what the intent of each individual market is. There very well could be multiple instances of the same market which will actually have different probabilities based on who is the creator and subtle differences in resolution criteria.
Markets on Manifold are too ambiguous.
I don’t have too much to say on this as I agree with you that our markets are generally too ambiguous and what Jack said covers a lot of my own thoughts. We do encourage creators to try and be as clear as possible, but many people are still learning how to create questions. One of the main purposes of the site is to lower the entry-barrier to prediction markets and allow anyone to create one easily so it isn’t a surprise a large number of markets aren’t that great or clear. Fortunately, there is already a clear culture developing where users will comment when they see unambiguity or a possible loophole in a market. And I think you would find that our big serious markets tend to be fairly unambiguous at this point.
Manifold is untrustworthy.
I’m not sure I agree with this but that may definitely be my bias haha. I do think though that users should regard every market they bet on with mild scepticism and not default to trusting the creator. Now, as a site is it optimal to have a bunch of unreliable content? Maybe not, particularly for the new user experience and I can empathise with why this may cause you to mistrust the site. But it’s unrealistic to expect a site that gives its users full autonomy and does minimal curation to have only trustworthy content. But it is something we will need to think about more. Hopefully, it will be mostly solved once we introduce a reputation system of some sort. And from my experience, the number of markets that have been “untrustworthy” can be counted on one hand so far.
Thanks again for taking the time to try out our site (even if it didn’t go as you expected) and for doing this write-up. Am happy to discuss things further. Maybe what you’ve outlined is a bigger problem than we’ve realised and we need to spend a lot more effort designing the site around it, but I’m not yet convinced. It is good to know though that it is turning some users away.
I used a P100 elastomeric respirator pretty much any time I left the house, for multiple months in 2020 during early COVID, and intermittently after that.
The main downside, for me personally, was that people generally found understanding my speech through it difficult or impossible. This was a big enough problem that I haven’t used one in quite some time.
An off-hand attempt at market mechnanism to solve this. Have “resolution issue” predictors. Like “There will be a news paper article of lenght more than 20 cm written by a full time journalist about biological weapons AND the vague biological weapons question will resolve in the negative”.
Maybe we should just let people bet on N/A similar to Augur (with some stronger norm of resolving N/A in ambiguous cases)
These are fun to think about.
It’s not entirely clear to me that the model is making a mistake with the expected value calculations.
The model’s goal is to complete the pattern given examples. In the other prize winning submissions, the intent of the prompter is pretty clear—e.g. there was an explicit instruction in the “repeat after me” task. But in the expected value case, all the prompts were consistent with either expected value or winning is good/losing is bad. And I think the latter frame is more accessible to people—if you asked a random person, I’m pretty sure they are more likely to go with the hindsight bias analysis.
You could argue that there’s a mismatch between the researcher’s expectations, that an EV calculation is the right way to approach these, and the behavior, but this seems to me to be more like straightforward train/test mismatch rather than anything deep going on.
One potentially interesting follow-on might be to poll humans to see how well they would do, perhaps via mechanical turk or similar. I predict that humans would be ~perfect on redefine and repeat after me, and would perform poorly on the expected value task. So they seem qualitatively different to me.
(I didn’t mention the negation task because I found the example to be confusing—a below average temp might be fine or dangerous depending on the size of the drop. Of course, negation has long been hard in NLP, so it’s perfectly plausible that it’s still a problem with LLMs. And maybe the other examples weren’t so borderline.)
We did do human validation on the tasks with Surge: redefine-math, quote-repetition, and hindsight-neglect all got 100% agreement, and NeQA got 98% agreement. I agree though that it seems likely many people would do the task ‘wrong’, so maybe the task would be improved by adding clearer instructions.
The situation feels somewhat like model splintering to me: the few-shot examples fit both patterns but the question doesn’t. The larger models are learning the incorrect generalization.
I think it’s important to note that LMs learning to respond in the same way as the average internet user is in some sense expected but can still be an example of inverse scaling – we would like our models to be smarter than that.
Is this “major damage” claim true? I remember being unsure, at the time, if the effects of the rules that limited stack height were substantial or negligible, since some people were saying that they mostly just applied to places that didn’t have the equipment to stack higher. Did anyone ever follow up to at least check how much container stacking increased after the rule change?
I am confident that the container stacking rules caused major damage when compared to better stacking rules. If we had a sensible stacking rule across LB/LA from the start I am confident there would have been far less backlog.
What is less clear is the extent to which the rules changes that were enacted mitigated the problem. While LB made some changes on the day of the post, LA didn’t act and LB’s action wasn’t complete. Thus there was some increase in permitted stacking but it was far from what one would have hoped for. And Elizabeth is right that we did not see a difference in port backlog that we can definitively link to the partial change that was enacted.
There was a long debate on this, I thought Zvi had changed his mind and recognized the rule change made no detectable difference in the port backlog.
I got annoyed about titles that are of the form “You are underestimating X”, “X is underappriciated”, “X is not so bad”, “We should rethink X”.
The title writer doesn’t really know what I think and why. This pushes towards making groups homogenous in beliefs and people dealing with each other more in sterotypes.
But it was minor enough and all negative to actually comment on the specific title.
For the four examples of
24-16=12, 53-25=25, 34-16=13, 63-17=16
is this the pattern?
ab-cd=ca
(Engineer at Manifold here.) I largely agree! Letting people make markets on anything means that many people will make poorly operationalized markets. Subjective resolution is good for bets on personal events among friends, which is an important use case, but it bad for questions with an audience bigger than that.
We need to do a better job of:
1. resolution reliability, like by adding a reputation system or letting creators delegate resolution to more objective individuals / courts.
2. helping users turn their vague uncertainties into objective questions—crucial but less straightforward.
3. surfacing higher quality content
The button isn’t showing up for me. Well, it shows up for like a second after I re-load the page but then it’s gone. I tried Opera GX browser and Chrome, it happens in both. Is this intended behaviour? I use Windows 7, maybe thats why...
You shouldn’t yet be able to blow up the page due to your karma. Idk if it’s supposed to show up.
The button shows up for me despite low karma. I have looked through the client-side code, and found this snippet:
This probably means Mawrak is encountering some sort of Javascript error, since the code indicates the button should only reject the launch attempt after you press it, not before.
I would argue that the most complex information exchange system in the known Universe will be “hard to emulating”. I don’t see how it can be any other way. We already understand the neurons well enough to emulate them. This is not nearly enough. You will not be able to do whole brain emulation without understanding of the inner workings of the system.
Woah Jennifer this is wild stuff. Fascinating!
I’m just thinking—it might be that while heat does determine brain efficiency to a large degree it might not be so simple as it might be determined by the architecture.
DIfferent animals might be adapated to a certain temperature range—it might not make your brain more efficient when it is cooled in this way.
does anybody know how the cognition of coldblood creatures varies as they warm and cool throughout the day?
I’m not sure what theory that observation would test cleanly, since ectotherms have such a complicated relationship to heat.
One thing I just checked is honeybees, which I’ve read a book or two about, because I know they have large cooling requirements for their wing muscles (their blood doesn’t move O2, it just moves heat, basically). When the hive gets cold in the winter, some go to the center and lock their wings with their arms, and shiver their wing muscles, and this heats the entire hive. So they are kind of endothermic? Maybe? Depending on definitions?
It looks like probably individual honeybee brains are cooled by using honey sort of like how dogs use saliva, with evaporative cooling from the mouth. Then, when flying on a very hot day, the waste heat from the wings is moved, by the blood, to the thorax (but not to the head) to use the thorax as a radiator.
When it is cold, the stationary wings can send heat to the thorax via the blood, but somehow the head doesn’t need this.
What do you think this might imply for the brain & cognition?
Humans obtain value from other humans and depend on them for their existence. It is hypothesized that AGIs will not depend on humans for their existence. Thus, humans who would not push the button to kill all other humans may choose not to do so for reasons of utility that don’t apply to AGI. Your hypothetical assumes this difference away, but our observations of humans don’t.
As you not, human morality and values were shaped by evolutionary and cultural pressure in favor of cooperation with other humans. The way this presumably worked is that humans who were less able or willing to cooperate tended to die more frequently. And cultures that were less able or willing to do so were conquered and destroyed. It is unclear how we would be able to replicate this or how well it translates.
It is unclear how many humans would actually choose to press this button. Your guess is that between 5% and 50% of humans would choose to do so.
That doesn’t suggest humans are very aligned; rather the opposite. It means that if we have between 2 and 20 AGIs (and those numbers don’t seem unreasonable) between 1 and 10 would choose to destroy humanity. Of course, extinction is the extreme version; having an AGI could also result in other negative consequences.
I think I might have started from a more pessimistic standpoint? It’s more like, I could also imagine living in a world where humans cooperate, but not because they actually care about each other, but would just pretend to do so? Introspection tells me that does not apply to myself, though maybe I evolved to not be conscious of my own selfishness? I am even less sure how altruistic other people are, because I did not ask lots of people: “Would you press a button that annihilates everyone after your death, if in return you get an awesome life?”. On the other hand, cooperation would probably be hard for us in such a world, so this is not as surprising?
On Metaculus, people have attempted to solve this problem by having moderators review each question before it goes live. The result has generally been that most vague questions require substantial re-editing before they can appear on the website.
Also, over time, users have become quite skilled at noticing when questions are likely to resolve ambiguously.
Even with this cultural difference between the sites, many questions still resolve ambiguously on Metaculus. It’s just really, really hard to say exactly what you mean when predicting the future.
I think Metaculus’s level of verbosity in resolution criteria is bad in that it makes questions longer to write and longer to understand (because it takes longer to read and because its more complex). Part of the goal of Manifold is to remove trivial inconveniences so that people actually forecast at all, and so that we get markets on literally everything.
I think the synthesis here is to have a subset of high quality markets (clear resolution criteria, clear meta-resolution norms) but still have a fat tail of medium-quality questions.