Technical staff at Anthropic, previously #3ainstitute; interdisciplinary, interested in everything; ongoing PhD in CS (learning / testing / verification), open sourcerer, more at zhd.dev
Zac Hatfield-Dodds(Zac Hatfield-Dodds)
Karma: 2,176
Re: that mouse study… I’m disappointed to feel I can’t trust rationalist headlines without also personally checking for inconvenient details in the preprint and expert commentary (here’s Derek Lowe in Science, or Helen Branswell in Stat.) For example:
The chimeric mix(es) in the study (wild-type plus Omicron spike protein) have previously been observed in the wild, in humans, and were outcompeted by other circulating strains. Seems important to me!
The question of which mutations to Covid drive vaccine escape sure do seem relevant to pandemic response to me! Of course in a coherently-competent world we wouldn’t need this at all (remember that Australia and NZ both eliminated Covid with a short lockdown in early 2020?), but given the world we’re actually in I’m a big fan of escape-resistant vaccines—and getting them approved and delivered faster.
80% of transgenic mice, not humans died (or got fairly ill and were euthanized) to the modified strain, compared to 100% (of transgenic mice, not humans) to the original Wuhan wild-type virus. IN MICE is one of the best-known reporting biases! There’s literally a @justsaysinmice tweet about this study!
Yeah, there’s clearly been some level of misbehavior here, and the institutional responses documented above are clownish. This is clearly in the genre of unacceptably risky work that probably created and then leaked pandemic Covid, which I wish we’d cut back on and confine to labs in the remote desert instead of dense cities.
But I don’t think we’d be discussing this specific paper at all if the 80% MORTALITY! headlines hadn’t been amplified, I don’t think they were accurate, and I would prefer that we stop undermining the credibility of the rationalist community and important arguments against unacceptably risky research by failing to check the basic details before posting alarmist headlines.
If you must, I’d strongly prefer a framing like “This is another unfortunately-common example of research I think is unacceptably risky. Note the mismanagement, the inconsistent descriptions and justifications of this project and whether it constitutes ‘gain of function’, the history of bio lab leaks, etc. While research on immune escape could be valuable, reforming vaccine approval and distribution would be much more effective and less risky.”
Llama 2 is not open source.
(a few days after this comment, here’s a concurring opinion from the Open Source Initiative—as close to authoritative as you can get)
(later again, here’s Yan LeCun testifying under oath: “so first of all Llama system was not made open source … we released it in a way that did not authorize commercial use, we kind of vetted the people who could download the model it was reserved to researchers and academics”)
While their custom licence permits some commercial uses, it is not an OSI approved license, and because it violates the open source definition it never will be. Specifically, the llama 2 licence violates:
-
Source code. It’s a little ambiguous what this means for a trained model; I’d claim that an open model release should include the training code (yes) and dataset (no), along with sufficient instructions for others to reproduce the results. However, you could also argue that weights are in fact “the preferred form in which a programmer would modify the program”, so this is not an important objection.
-
No Discrimination Against Persons or Groups. See the ban on use by anyone who has, or is affiliated with anyone who has, more than 700M active users. As a side note, Snapchat recently announced that they had 750M active users, so this looks pretty targeted at competing social media (including Tiktok, Google, etc.). As a consequence, the Llama 2 license also violates OSD 7. Distribution of License: “the rights attached to the program must apply to all to whom the program is redistributed without the need for execution of an additional license by those parties.”
-
No Discrimination Against Fields of Endeavor. If you can’t use Llama 2 to—for example—train another model, it’s by definition not open source. Their entire acceptable use policy is included by reference and contains a wide variety of sometimes ambiguous restrictions.
So, why does this matter?
-
As an open-source maintainer and PSF Fellow, I have no objection to the existence of commercially licensed software. I use much of it, and have sold commercial licenses for software that I’ve written too. However, people—and especially megacorps—misrepresenting their closed-off projects as open source is an infuriating form of parasitism on a reputation painstakingly built over decades.
-
The restriction on model training makes Llama 2 much less useful for AI safety research, but it incurs just as much direct (roughly all via misuse IMO) and acceleration risk as an open-source release.
-
Using a custom license adds substantial legal risk for prospective commercial users, especially given the very broad restrictions imposed by the acceptable use policy. This reduces the economic upside enormously relative to standard open terms, and leaves Meta’s competitors particularly at risk of lawsuits if they attempt to use Llama 2.
To summarize, Meta gets a better cost/benefit tradeoff by using a custom, non-open-source license especially if people incorrectly percieve it as open source; everyone else is worse off; and it seems to me like they’re deliberately misrepresenting what they’ve done for their own gain. This really, really annoys me.
When someone describes Llama 2 as “open source”, please correct them: Meta is offering a limited commercial license which discriminates against specific users and bans many valuable use-cases, including in alignment research.
- Progress links digest, 2023-08-09: US adds new nuclear, Katalin Karikó interview, and more by (9 Aug 2023 19:22 UTC; 18 points)
- 's comment on “Open Source AI” isn’t Open Source by (15 Feb 2024 17:07 UTC; 16 points)
- Progress links digest, 2023-07-28: The decadent opulence of modern capitalism by (28 Jul 2023 14:36 UTC; 16 points)
- 's comment on Snapshot of narratives and frames against regulating AI by (1 Nov 2023 18:16 UTC; 9 points)
- Jason’s links digest, 2023-07-28: The decadent opulence of modern capitalism by (Progress Forum; 28 Jul 2023 14:36 UTC; 5 points)
- 's comment on AI #48: The Talk of Davos by (26 Jan 2024 7:41 UTC; 4 points)
- 's comment on Progress links digest, 2023-07-28: The decadent opulence of modern capitalism by (4 Aug 2023 8:07 UTC; 4 points)
-
This approach to tradeoffs makes sense for the USA in 2021.
I just don’t want our analysis to lose sight of the fact that facing these tradeoffs is stupid and avoidable, and that almost every country could have done so much better. Avoiding outbreaks is so much cheaper and easier than dealing with them that the choice to do so should have been overdetermined.
The background risk rate in Australia is roughly zero. We occasionally get “outbreaks” of single-digit cases, lock down one city for a few days to trace it, and then go back to normal.
It’s not even worth wearing masks here.
Australia is taking a (frustratingly) slow and cautious approach to the vaccine rollout. This will probably cost zero lives (though with a scary right-tail), plausibly saving some from avoided adverse reactions. IMO we should be going way faster on tail-risk and economic grounds, but...
TLDR: it’s much better to be careful instead of exponential growth than after it.
I agree that there’s no substitute for thinking about this for yourself, but I think that morally or socially counting “spending thousands of dollars on yourself, an AI researcher” as a donation would be an apalling norm. There are already far too many unmanaged conflicts of interest and trust-me-it’s-good funding arrangements in this space for me, and I think it leads to poor epistemic norms as well as social and organizational dysfunction. I think it’s very easy for donating to people or organizations in your social circle to have substantial negative expected value.
I’m glad that funding for AI safety projects exists, but the >10% of my income I donate will continue going to GiveWell.
I think you’ve misunderstood the lesson, and mis-generalized from your experience of manually improving results.
First, I don’t believe that you could write a generally-useful program to improve translations—maybe for Korean song lyrics, yes, but investing lots of human time and knowledge in solving a specific problem is exactly the class of mistakes the bitter lesson warns against.
Second, the techniques that were useful before capable models are usually different to the techniques that are useful to ‘amplify’ models—for example, “Let’s think step by step” would be completely useless to combine with previous question-answering techniques.
Third, the bitter lesson is not about deep learning; it’s about methods which leverage large amounts of compute. AlphaGo combining MCTS with learned heuristics is a perfect example of this:
One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
(Zac’s note: I’m posting this on behalf of Jack Clark, who is unfortunately unwell today. Everything below is his words.)
Hi there, I’m Jack and I lead our policy team. The primary reason it’s not discussed in the post is that the post was already quite long and we wanted to keep the focus on safety—I did some help editing bits of the post and couldn’t figure out a way to shoehorn in stuff about policy without it feeling inelegant / orthogonal.
You do, however, raise a good point, in that we haven’t spent much time publicly explaining what we’re up to as a team. One of my goals for 2023 is to do a long writeup here. But since you asked, here’s some information:
You can generally think of the Anthropic policy team as doing three primary things:
Trying to proactively educate policymakers about the scaling trends of AI systems and their relation to safety. Myself and my colleague Deep Ganguli (Societal Impacts) basically co-wrote this paper https://arxiv.org/abs/2202.07785 - you can think of us as generally briefing out a lot of the narrative in here.
Pushing a few specific things that we care about. We think evals/measures for safety of AI systems aren’t very good [Zac: i.e. should be improved!], so we’ve spent a lot of time engaging with NIST’s ‘Risk Management Framework’ for AI systems as a way to create more useful policy institutions here—while we expect labs in private sector and academia will do much of this research, NIST is one of the best institutions to take these insights and a) standardize some of them and b) circulate insights across government. We’ve also spent time on the National AI Research Resource as we see it as a path to have a larger set of people do safety-oriented analysis of increasingly large models.
Responding to interest. An increasing amount of our work is reactive (huge uptick in interest in past few months since launch of ChatGPT). By reactive I mean that policymakers reach out to us and ask for our thoughts on things. We generally aim to give impartial, technically informed advice, including pointing out things that aren’t favorable to Anthropic to point out (like emphasizing the very significant safety concerns with large models). We do this because a) we think we’re well positioned to give policymakers good information and b) as the stakes get higher, we expect policymakers will tend to put higher weight on the advice of labs which ‘showed up’ for them before it was strategic to do so. Therefore we tend to spend a lot of time doing a lot of meetings to help out policymakers, no matter how ‘important’ they or their country/organization are—we basically ignore hierarchy and try to satisfy all requests that come in at this stage.
More broadly, we try to be transparent on the micro level, but haven’t invested yet in being transparent on the macro. What I mean by that is many of our RFIs, talks, and ideas are public, but we haven’t yet done a single writeup that gives an overview of our work. I am hoping to do this with the team this year!
Some other desiderata that may be useful:
I testified in the Senate last year and wrote quite a long written testimony.
I talked to the Congressional AI Caucus; slides here. Note: I demo’d Claude but it’s worth noting that whenever I demo our system I also break it to illustrate safety concerns. IIRC here I jailbroke it so it would play along with me when I asked it how to make rabies airborne—this was to illustrate how loose some of the safety aspects of contemporary LLMs are.
A general idea I/we push with policymakers is the need to measure and monitor AI systems; Jess Whittlestone and I wrote up a plan here which you can expect us to be roughly outlining in meetings.
A NIST RFI that talks about some of the trends in predictability and surprise and also has some recommendations.
Our wonderful colleagues on the ‘Societal Impacts’ team led this work on Red Teaming and we (Policy) helped out on the paper and some of the research. We generally think red teaming is a great idea to push to policymakers re AI systems; it’s one of those things that is ‘shovel ready’ for the systems of today but, we think, has some decent chance of helping out in future with increasingly large models.
I think this is missing the point pretty badly for Anthropic, and leaves out most of the work that we do. I tried writing up a similar summary; which is necessarily a little longer:
Anthropic: Let’s get as much hands-on experience building safe and aligned AI as we can, without making things worse (advancing capabilities, race dynamics, etc). We’ll invest in mechanistic interpretability because solving that would be awesome, and even modest success would help us detect risks before they become disasters. We’ll train near-cutting-edge models to study how interventions like RL from human feedback and model-based supervision succeed and fail, iterate on them, and study how novel capabilities emerge as models scale up. We’ll also share information so policy-makers and other interested parties can understand what the state of the art is like, and provide an example to others of how responsible labs can do safety-focused research.
(Sound good? We’re hiring for both technical and non-technical roles.)
We were also delighted to welcome Tamera Lanham to Anthropic recently, so you could add her externalized reasoning oversight agenda to our alignment research too :-)
Why don’t more plants evolve towards the “grass” strategy?
I suspect it’s related to the distinction between C3 and C4 photosynthesis—both are common in grasses and C4 species tend to do better in hot climates, but trees seem to have trouble evolving C4 pathways even though that happened on 60+ separate occasions.
(also IMO monocots top out at “kinda tree-ish”—they do have a recognisable trunk, but more fibrous than woody)
The paper is really only 28 pages plus lots of graphs in the appendices! If you want to skim, I’d suggest just reading the abstract and then sections 5 and 6 (pp 16--21). But to summarize:
Do neural networks learn the same concepts as humans, or at least human-legible concepts? A “yes” would be good news for interpretability (and alignment). Let’s investigate AlphaZero and Chess as a case study!
Yes, over the course of training AlphaZero learns many concepts (and develops behaviours) which have clear correspondence with human concepts.
Low-level / ground-up interpretability seems very useful here. Learned summaries are also great for chess but rely on a strong ground-truth (e.g. “Stockfish internals”).
Details about where in the network and when in the training process things are represented and learned.
The analysis of differences between the timing and order of developments in human scholarship and AlphaZero training is pretty cool if you play chess; e.g. human experts diversify openings (not just
1.e4) since 1700 while AlphaZero narrows down from random to pretty much the modern distribution over GM openings; AlphaZero tends to learn material values before positions and standard openings.
Adam Jermyn says Anthropic’s RSP includes fine-tuning-included evals every three months or 4x compute increase, including during training.
You don’t need to take anyone’s word for this when checking the primary source is so easy: the RSP is public, and the relevant protocol is on page 12:
In more detail, our evaluation protocol is as follows: … Timing: During model training and fine-tuning, Anthropic will conduct an evaluation of its models for next-ASL capabilities both (1) after every 4x jump in effective compute, including if this occurs mid-training, and (2) every 3 months to monitor fine-tuning/tooling/etc improvements.
Relatedly, have you considered organizing the company as a Public Benefit Corporation, so that the mission and impact is legally protected alongside shareholder interests?
First, an apology: I didn’t mean this to be read as an attack or a strawman, nor applicable to any use of theorem-proving, and I’m sorry I wasn’t clearer. I agree that formal specification is a valuable tool and research direction, a substantial advancement over informal arguments, and only as good as the assumptions. I also think that hybrid formal/empirical analysis could be very valuable.
Trying to state a crux, I believe that any plan which involves proving corrigibility properties about MuZero (etc) is doomed, and that safety proofs about simpler approximations cannot provide reliable guarantees about the behaviour of large models with complex emergent behaviour. This is in large part because formalising realistic assumptions (e.g. biased humans) is very difficult, and somewhat because proving anything about very large models is wildly beyond the state of the art and even verified systems have (fewer) bugs.
Being able to state theorems about AGI seems absolutely necessary for success; but I don’t think it’s close to sufficient.
For the sake of readability, I have referred to misguidedness, callousness, and stupidity as type one, two, and three traits respectively
For what it’s worth I find descriptive terms much easier to read than type one, two, etc. In statistics I even dislike “false positive/negative”, and prefer the more descriptive “false/missed alarm/label/...”.
“Okay, you have found that one-layer transformer without MLP approximates skip-trigram statistics, how it generalizes to the question ’does GPT-6 want to kill us all?”?
I understand this is more an illustration than a question, but I’ll try answering it anyway because I think there’s something informative about different perspectives on the problem :-)
Skip-trigrams are a foundational piece of induction heads, which are themselves a key mechanism for in-context learning. A Mathematical Framework for Transformer Circuits was published less than a year ago, IMO subsequent progress is promising, and mechanistic interpretability has been picked up by independent researchers and other labs (e.g. Redwood’s project on GPT-2-small).
Of course the skip-trigram result isn’t itself an answer to the question of whether some very capable ML system is planning to deceive the operator or seize power, but I claim it’s analogous to a lemma in some paper that establishes a field and that said field is one of our most important tools for x-risk mitigation. This was even our hope at the time, though I expected both the research and the field-building to go more slowly—actual events are something like a 90th-percentile outcome relative to my expectations in October-2021.[1]
Finally, while I deeply appreciate theoretical/conceptual research as a complement to empirical and applied research and want both, how on earth is either meant to help alone? If we get a conceptual breakthrough but don’t know how to build—and verify that we’ve correctly built—the thing, we’re still screwed; conversely if we get really good at building stuff and verifying our expectations but don’t expect some edge-case like FDT-based cooperation then we’re still screwed. Efforts which integrate both at least have a chance, if nobody else does something stupid first.
- ↩︎
I still think it’s pretty unlikely (credible interval 0--40%) that we’ll have good enough interpretability tools by the time we really really need them, but I don’t see any mutually-exclusive options which are better.
- ↩︎
So, how do you think Australia did, all things considered?
I think it’s worth considering the federal and state governments separately.
The federal government has been incredibly incompetent on fundamental issues like vaccinations and refusing to do anything about border quarantine.
State governments—the ones responsible for NPI policy like lockdowns—were far slower than optimal last year, but with the critical exception of NSW doing pretty well this year.
Even with delta, every state and territory has seen an outbreak and squashed it within three weeks by fast, early action. This is far cheaper than an ongoing epidemic.
The goal of zero transmission was and is clearly the best equilibrium to be in; we knew this early last year. Lockdowns suck but they work, and a fast+hard lockdown sucks a lot less than an epidemic. We went more than a year in Canberra between cases! The system works, and to be blunt if it was applied consistently COVID could have been eradicated by mid-2020… not that it could have been applied, but still.
Just keep !
Overall grade: C-, for an approach which was sufficient until betrayed by the combination of one particularly incompetent state and a federal government which turned down early access to Pfizer vaccines. D+ for policy, B for outcomes—we were adequate enough that good luck could make a difference.
For a value of “break into flames” that matches damp and poorly-oxygenated fuel, yep! This case in Australia is illustrative; you tend to get a lot of nasty smoke rather than a nice campfire vibe.
You’d have to mismanage a household-scale compost pile very badly before it spontaneously combusts, but it’s a known and common failure mode for commercial-scale operations above a few tons. Specific details about when depend a great deal on the composition of the pile; with nitrate filmstock it was possible with as little as a few grams.
Bluntly: if you write it on Lesswrong or the Alignment Forum, or send it to a particular known person, governments will get a copy if they care to. Cybersecurity against state actors is really, really, really hard. Lesswrong is not capable of state-level cyberdefense.
If you must write it at all: do so with hardware which has been rendered physically unable to connect to the internet, and distribute only on paper, discussing only in areas without microphones. Consider authoring only on paper in the first place. Note that physical compromise of your home, workplace, and hardware is also a threat in this scenario.
(I doubt they care much, but this is basically what it takes if they do. Fortunately I think LW posters are very unlikely to be working with such high-grade secrets.)
I think it’s important here to quote Hamming defining important problem:
I’m not talking about ordinary run-of-the-mill research; I’m talking about great research. I’ll occasionally say Nobel-Prize type of work. It doesn’t have to gain the Nobel Prize, but I mean those kinds of things which we perceive are significant [e.g. Relativity, Shannon’s information theory, etc.] …
Let me warn you, “important problem” must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn’t work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It’s not the consequence that makes a problem important, it is that you have a reasonable attack.
This suggests to me that e.g. “AI alignment is an important problem”, not this particular approach to alignment is an important problem. The latter is too small; it can be good work and impactful work, but not great work in the sense of relativity or information theory or causality. (I’d love to be proven wrong!)
One year is actually the typical term length for board-style positions, but because members can be re-elected their tenure is often much longer. In this specific case of course it’s now up to the trustees!
I was halfway through a PhD on software testing and verification before joining Anthropic (opinions my own, etc), and I’m less convinced than Eliezer about theorem-proving for AGI safety.
There are so many independently fatal objections that I don’t know how to structure this or convince someone who thinks it would work. I am therefore offering a $1,000 prize for solving a far easier problem:
I will also take bets that nobody will accomplish this by 2025, nor any loosely equivalent proof for a GPT-3 class model by 2040. This is a very bold claim, but I believe that rigorously proving even trivial global bounds on the behaviour of large learned models will remain infeasible.
And doing this wouldn’t actually help, because a theorem is about the inside of your proof system. Recognising the people in a huge collection of atoms is outside your proof system. Analogue attacks like Rowhammer are not in (the ISA model used by) your proof system—and cache and timing attacks like Spectre probably aren’t yet either. Your proof system isn’t large enough to model the massive floating-point computation inside GPT-2, let alone GPT-3, and if it could the bounds would be (−∞,∞).
I still hope that providing automatic disproof-by-counterexample might, in the long term, nudge ML towards specifiability by making it easy to write and check falsifiable properties of ML systems. On the other hand, hoping that ML switches to a safer paradigm is not the kind of safety story I’d be comfortable relying on.