Technical staff at Anthropic, previously #3ainstitute; interdisciplinary, interested in everything; half a PhD in CS (learning / testing / verification), open sourcerer, more at zhd.dev
Zac Hatfield-Dodds(Zac Hatfield-Dodds)
Karma: 952
Fixed, thanks; it links to the transformer circuits thread which includes both the induction heads paper, SoLU, and Toy Models of Superposition.
“Okay, you have found that one-layer transformer without MLP approximates skip-trigram statistics, how it generalizes to the question ’does GPT-6 want to kill us all?”?
I understand this is more an illustration than a question, but I’ll try answering it anyway because I think there’s something informative about different perspectives on the problem :-)
Skip-trigrams are a foundational piece of induction heads, which are themselves a key mechanism for in-context learning. A Mathematical Framework for Transformer Circuits was published less than a year ago, IMO subsequent progress is promising, and mechanistic interpretability has been picked up by independent researchers and other labs (e.g. Redwood’s project on GPT-2-small).
Of course the skip-trigram result isn’t itself an answer to the question of whether some very capable ML system is planning to deceive the operator or seize power, but I claim it’s analogous to a lemma in some paper that establishes a field and that said field is one of our most important tools for x-risk mitigation. This was even our hope at the time, though I expected both the research and the field-building to go more slowly—actual events are something like a 90th-percentile outcome relative to my expectations in October-2021.[1]
Finally, while I deeply appreciate theoretical/conceptual research as a complement to empirical and applied research and want both, how on earth is either meant to help alone? If we get a conceptual breakthrough but don’t know how to build—and verify that we’ve correctly built—the thing, we’re still screwed; conversely if we get really good at building stuff and verifying our expectations but don’t expect some edge-case like FDT-based cooperation then we’re still screwed. Efforts which integrate both at least have a chance, if nobody else does something stupid first.
- ↩︎
I still think it’s pretty unlikely (credible interval 0--40%) that we’ll have good enough interpretability tools by the time we really really need them, but I don’t see any mutually-exclusive options which are better.
- ↩︎
Anthropic is still looking for a senior software engineer
As last time, Anthropic would like to hire many people for research, engineering, business, and operations roles. More on that here, or feel free to ask me :-)
Happily, the world’s experts regularly compile consensus reports for the UN Framework Convention on Climate Change, and the Sixth Assessment Report is currently being finalized. While the full report is many thousands of pages, the “summary for policymakers” is an easy—sometimes boring—read at tens of pages.
https://www.ipcc.ch/report/ar6/wg1/ - The Physical Science Basis
https://www.ipcc.ch/report/ar6/wg2/ - Impacts, Adaptation and Vulnerability
https://www.ipcc.ch/report/ar6/wg3/ - Mitigation of Climate Change
I think you’re asking about wg2 and wg3, in which case reading the SPM might be useful—there’s about 60 pages of them once you skip the frontmatter etc. Most shorter answers are going to be at best true but kinda useless “reduce emissions in the most cost-effective politically-feasible way, and take appropriate local action to adapt to changing conditions” would be my one-sentence attempt.
I’m not surprised that if you investigate context-free grammars with two-to-six-layer transfomers you learn something very much like a tree. I also don’t expect this result to generalize to larger models or more complex tasks, and so personally I find the paper plausible but uninteresting.
Please note that the inverse scaling prize is not from Anthropic:
The Inverse Scaling Prize is organized by a group of researchers on behalf of the Fund for Alignment Research (FAR), including Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez. Additionally, Sam Bowman and Ethan Perez are affiliated with Anthropic; Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman are affiliated with New York University. The prize pool is provided by the Future Fund.
Anthropic has provided some models as a validation set, but it’s not even the most common affiliation!
(if funding would get someone excited to do a great job of this, I’ll help make that happen)
I’d be especially excited if this debate produced an adversarial-collaboration-style synthesis document, laying out the various perspectives and cruxes. I think that collapsing onto an optimism/pessimism binary loses a lot of important nuance; but also that HAIST reading, summarizing, and clearly communicating the range of views on RLHF could help people holding each of those views more clearly understand each other’s concerns and communicate with each other.
No, this is also easy to work around; language models are good at deobfuscation and you could probably even do it with edit-distance techniques. Nor do you have enough volume of discussion to hide from humans literally just reading all of it; nor is Facebook secure against state actors, nor is your computer secure. See also Security Mindset and Ordinary Paranoia.
Bluntly: if you write it on Lesswrong or the Alignment Forum, or send it to a particular known person, governments will get a copy if they care to. Cybersecurity against state actors is really, really, really hard. Lesswrong is not capable of state-level cyberdefense.
If you must write it at all: do so with hardware which has been rendered physically unable to connect to the internet, and distribute only on paper, discussing only in areas without microphones. Consider authoring only on paper in the first place. Note that physical compromise of your home, workplace, and hardware is also a threat in this scenario.
(I doubt they care much, but this is basically what it takes if they do. Fortunately I think LW posters are very unlikely to be working with such high-grade secrets.)
Anthropic is hiring senior software engineers and many other positions including a recruiter and recruiting coordinator (meta-hiring!). More about that here.
I wouldn’t usually post those links, but the blue dot site seems to be broken and the comment here might give the impression that we’re only hiring for (specific) technical roles.
For Python basics, I have to anti-recommend Shaw’s ‘learn the hard way’; it’s generally outdated and in some places actively misleading. And why would you want to learn the hard way instead of the best way in any case?
Instead, my standard recommendation is Al Sweigart’s Automate the Boring Stuff and then Beyond the Basic Stuff (both readable for free on inventwithpython.com, or purchasable in books); he’s also written some books of exercises. If you prefer a more traditional textbook, Think Python 2e is excellent and also available freely online.
I think this framing is accurate and important. Implications are of course “undignified” to put it lightly...
Broadly agree on upshot (1), though of course I hope we can do even better. (2) is also important though IMO way too weak. (Rule zero: ensure that it’s never your lab that ends the world)
As usual, opinions my own.
One answer is that it might be soon enough that it’s very much like right now, and so any contemporary fiction could do. Another is that “the future is already here, just unevenly distributed”, and extrapolation can give you credible stories (so long as they’re not set too close to an AI lab).
On longer-timelines views, a prerequisite to writing such a novel would be accurately forecasting the trajectory of AI research and deployment right up to the end. This is generally seen as rather difficult.
Mu. Ask “how does this knowledge actually affect pandemic response”, not what I’d do. I claim personal expertise in neither case.
(If I could set policy we’d have driven at minimal cost (Covid was eradicable, and eliminated at several times in several countries!), had generic-coronavirus vaccine bases and pre-authorized challenge trials if that was infeasible, etc. In the world we’re actually in, my understanding is that research like this feeds in to vaccine design, however far from ideal vaccine policy and delivery might be.)
Thanks for an excellent reply! One possible crux is that I don’t think that synthesized human values are particularly useful; I’d expect that AGI systems can do their own synthesis from a much wider range of evidence (including law, fiction, direct observation, etc.). As to the specific points, I’d respond:
There is no unified legal theory precise enough to be practically useful for AI understanding human preferences and values; liberal and social democracies alike tend to embed constraints in law, with individuals and communities pursuing their values in the lacunae.
The rigorous tests of legal theories are carried out inside the system of law, and bent by systems of unjust power (e.g. disenfranchisement). We cannot validate laws or legal theories in any widely agreed-upon manner.
Law often lacks settled precedent, especially regarding new technologies, or disagreements between nations or different cultures.
I reject the assertion that imposition by a government necessarily makes a law legitimate. While I agree we don’t have a mechanism to ‘align the rest of the humans’ with a theory or meta-theory, I don’t think this is relevant (and in any case it’s equally applicable to law).
I agree that “moral lock-in” would be a disaster. However, I dispute that law accurately reflects the evolving will of citizens; or the proposition that so reflecting citizen’s will is consistently good (c.f. reproductive rights, civil rights, impacts on foreign nationals or future generations...)
These points are about law as it exists as a widely-deployed technology, not idealized democratic law. However, only the former is available to would-be AGI developers!
Law does indeed provide useful evidence about human values, coordination problems, and legitimacy—but this alone does not distinguish it.
Baseline again, I strongly support not doing any research that might collect, create, or leak pandemic pathogens. Defining this study as “not gain of function” is only defensible on technicalities that make the term almost meaningless.
It’s unclear to me which headline you considered misleading. Which rationalist outlet said in a headline something about 80% MORTALITY without mentioning that it’s in mice?
I’m not complaining about “literally failing to mention mice”, but rather “without appearing to know that the wild-type strain was 100% lethal to the same mice” - ‘in (transgenic) mice’ should be a signal that you need to read the study before using it as evidence. As an exemplar, I consider William A Eden’s tweet (the opener to this post) to be irresponsible reporting.
Zvi’s title for this post, “Covid 10/20/22: Wait, We Did WHAT?” is also more hyperbolic than I’d prefer, as is the opening: The what, in this case, was the type of thing that could cause another pandemic: … None of the other developments should scare you. I disagree that we should be scared of this particular study; and while the system of institutions and incentives that supports such work does need to be dismantled, I think the rationalist response to this paper has undermined our credibility and had the wrong marginal impact.
There’s a self-correction process at play here (and in Alyssa’s tweet above), but we shouldn’t make this kind of mistake in the first place and ought to correct it more quickly and comprehensively if we do.
Re: that mouse study… I’m disappointed to feel I can’t trust rationalist headlines without also personally checking for inconvenient details in the preprint and expert commentary (here’s Derek Lowe in Science, or Helen Branswell in Stat.) For example:
The chimeric mix(es) in the study (wild-type plus Omicron spike protein) have previously been observed in the wild, in humans, and were outcompeted by other circulating strains. Seems important to me!
The question of which mutations to Covid drive vaccine escape sure do seem relevant to pandemic response to me! Of course in a coherently-competent world we wouldn’t need this at all (remember that Australia and NZ both eliminated Covid with a short lockdown in early 2020?), but given the world we’re actually in I’m a big fan of escape-resistant vaccines—and getting them approved and delivered faster.
80% of transgenic mice, not humans died (or got fairly ill and were euthanized) to the modified strain, compared to 100% (of transgenic mice, not humans) to the original Wuhan wild-type virus. IN MICE is one of the best-known reporting biases! There’s literally a @justsaysinmice tweet about this study!
Yeah, there’s clearly been some level of misbehavior here, and the institutional responses documented above are clownish. This is clearly in the genre of unacceptably risky work that probably created and then leaked pandemic Covid, which I wish we’d cut back on and confine to labs in the remote desert instead of dense cities.
But I don’t think we’d be discussing this specific paper at all if the 80% MORTALITY! headlines hadn’t been amplified, I don’t think they were accurate, and I would prefer that we stop undermining the credibility of the rationalist community and important arguments against unacceptably risky research by failing to check the basic details before posting alarmist headlines.
If you must, I’d strongly prefer a framing like “This is another unfortunately-common example of research I think is unacceptably risky. Note the mismanagement, the inconsistent descriptions and justifications of this project and whether it constitutes ‘gain of function’, the history of bio lab leaks, etc. While research on immune escape could be valuable, reforming vaccine approval and distribution would be much more effective and less risky.”
(2) Many arguments for AGI misalignment depend on our inability to imbue AGI with a sufficiently rich understanding of what individual humans want and how to take actions that respect societal values more broadly.
(4) If AGI learns law, it will understand how to interpret vague human directives and societal human values well enough that its actions will not cause states of the world that dramatically diverge from human preferences and societal values.
I think (2) is straightforwardly false: the lethal difficulty is not getting something that understands what humans want; it’s that by default it’s very unlikely to care! The same argument implies that (4) might be true as written, but still irrelevant to x-risk.
Additionally, “Law” is not unique, not always humane, not always just, inconsistent between countries or even subnational jurisdictions, etc. - consider for example reproductive rights. I reject the assertion that law is the only legitimate source of human values, or indeed that law is a source of values at all (I’d describe it as a partial expression). Law might indeed turn out to be useful, but I don’t see it as distinguished from other forms of nonfiction or for that matter novels, poetry, etc.
A few subtle but very important statistical nitpicks (full text of the study for reference):
Their sample size is large enough to get pretty reliable P-values, which is great! However, this means is that if there was no relationship you would be unlikely to observe such extreme results by chance, not that the variable is a good predictor.
I didn’t see any correction for multiple-hypothesis testing, and the discussion of marital status leaves me thinking that they didn’t do this at all: “[The final model] included marital status (P=0.0499) (previously, marital status narrowly missed inclusion, with P=0.052)”.
Per the original form of Goodhart’s Law “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”, so I don’t find the stated claim convincing: it’s plausible that the treatment affects observables but not outcomes.
However figures two and three are pretty compelling: the relative risk reduction is between 30-45% for almost all cohorts, measured in terms of acute major coronary events per patient-century—“as previously reported, lovastatin-mediated changes in these lipids were associated with a 37% decrease”. Seems reasonable to conclude that apoB is a better target, even though the P-values don’t seem particularly relevant and I wish they’d show scatter or kde-plots under the regression lines.