LessWrong dev & admin as of July 5th, 2022.
RobertM
Karma: 3,594
Should be fixed now.
Good catch, looks like that’s from this revision, which looks like it was copied over from Arbital—some LaTeX didn’t make it through. I’ll see if it’s trivial to fix.
The page isn’t dead, Arbital pages just don’t load sometimes (or take 15+ seconds).
I understand this post to be claiming (roughly speaking) that you assign >90% likelihood in some cases and ~50% in other cases that LLMs have internal subjective experiences of varying kinds. The evidence you present in each case is outputs generated by LLMs.
The referents of consciousness for which I understand you to be making claims re: internal subjective experiences are 1, 4, 6, 12, 13, and 14. I’m unsure about 5.
Do you have sources of evidence (even illegible) other than LLM outputs that updated you that much? Those seem like very surprisingly large updates to make on the basis of LLM outputs (especially in cases where those outputs are self-reports about the internal subjective experience itself, which are subject to substantial pressure from post-training).
Separately, I have some questions about claims like this:
The Big 3 LLMs are somewhat aware of what their own words and/or thoughts are referring to with regards to their previous words and/or thoughts. In other words, they can think about the thoughts “behind” the previous words they wrote.
This doesn’t seem constructively ruled out by e.g. basic transformer architectures, but as justification you say this:
If you doubt me on this, try asking one what its words are referring to, with reference to its previous words. Its “attention” modules are actually intentionally designed to know this sort of thing, using using key/query/value lookups that occur “behind the scenes” of the text you actually see on screen.
How would you distinguish an LLM both successfully extracting and then faithfully representing whatever internal reasoning generated a specific part of its outputs, vs. conditioning on its previous outputs to give you plausible “explanation” for what it meant? The second seems much more likely to me (and this behavior isn’t that hard to elicit, i.e. by asking an LLM to give you a one-word answer to a complicated question, and then asking it for its reasoning).
My impression is that Yudkowsky has harmed public epistemics in his podcast appearances by saying things forcefully and with rather poor spoken communication skills for novice audiences.
I recommend reading the Youtube comments on his recorded podcasts, rather than e.g. Twitter commentary from people with a pre-existing adversarial stance to him (or AI risk questions writ large).
On one hand, I feel a bit skeptical that some dude outperformed approximately every other pollster and analyst by having a correct inside-view belief about how existing pollster were messing up, especially given that he won’t share the surveys. On the other hand, this sort of result is straightforwardly predicted by Inadequate Equilibria, where an entire industry had the affordance to be arbitrarily deficient in what most people would think was their primary value-add, because they had no incentive to accuracy (skin in the game), and as soon as someone with an edge could make outsized returns on it (via real-money prediction markets), they outperformed all the experts.
On net I think I’m still <50% that he had a correct belief about the size of Trump’s advantage that was justified by the evidence he had available to him, but even being directionally-correct would have been sufficient to get outsized returns a lot of the time, so at that point I’m quibbling with his bet sizing rather than the direction of the bet.
I’m pretty sure Ryan is rejecting the claim that the people hiring for the roles in question are worse-than-average at detecting illegible talent.
Depends on what you mean by “resume building”, but I don’t think this is true for “need to do a bunch of AI safety work for free” or similar. i.e. for technical research, many people that have gone through MATS and then been hired at or founded their own safety orgs have no prior experience doing anything that looks like AI safety research, and some don’t even have much in the way of ML backgrounds. Many people switch directly out of industry careers into doing e.g. ops or software work that isn’t technical research. Policy might seem a bit trickier but I know several people who did not spend anything like years doing resume building before finding policy roles or starting their own policy orgs and getting funding. (Though I think policy might actually be the most “straightforward” to break into, since all you need to do to demonstrate compentence is publish a sufficiently good written artifact; admittedly this is mostly for starting your own thing. If you want to get hired at a “larger” policy org resume building might matter more.)
(We switched back to shipping Calibri above Gill Sans Nova pending a fix for the horrible rendering on Windows, so if Ubuntu has Calibri, it’ll have reverted back to the previous font.)
Indeed, such red lines are now made more implicit and ambiguous. There are no longer predefined evaluations—instead employees design and run them on the fly, and compile the resulting evidence into a Capability Report, which is sent to the CEO for review. A CEO who, to state the obvious, is hugely incentivized to decide to deploy models, since refraining to do so might jeopardize the company.
This doesn’t seem right to me, though it’s possible that I’m misreading either the old or new policy (or both).
Re: predefined evaluations, the old policy neither specified any evaluations in full detail, nor did it suggest that Anthropic would have designed the evaluations prior to a training run. (Though I’m not sure that’s what you meant, when contrasted it with “employees design and run them on the fly” as a description of the new policy.)
Re: CEO’s decisionmaking, my understanding of the new policy is that the CEO (and RSO) will be responsible only for approving or denying an evaluation report making an affirmative case that a new model does not cross a relevant capability threshold (“3.3 Capability Decision”, original formatting removed, all new bolding is mine):
If, after the comprehensive testing, we determine that the model is sufficiently below the relevant Capability Thresholds, then we will continue to apply the ASL-2 Standard. The process for making such a determination is as follows:
First, we will compile a Capability Report that documents the findings from the comprehensive assessment, makes an affirmative case for why the Capability Threshold is sufficiently far away, and advances recommendations on deployment decisions.
The report will be escalated to the CEO and the Responsible Scaling Officer, who will (1) make the ultimate determination as to whether we have sufficiently established that we are unlikely to reach the Capability Threshold and (2) decide any deployment-related issues.
In general, as noted in Sections 7.1.4 and 7.2.2, we will solicit both internal and external expert feedback on the report as well as the CEO and RSO’s conclusions to inform future refinements to our methodology. For high-stakes issues, however, the CEO and RSO will likely solicit internal and external feedback on the report prior to making any decisions.
If the CEO and RSO decide to proceed with deployment, they will share their decision–as well as the underlying Capability Report, internal feedback, and any external feedback–with the Board of Directors and the Long-Term Benefit Trust before moving forward.
The same is true for the “Safeguards Decision” (i.e. making an affirmative case that ASL-3 Required Safeguards have been sufficiently implemented, given that there is a model that has passed the relevant capabilities thresholds).
This is not true for the “Interim Measures” described as an allowable stopgap if Anthropic finds itself in the situation of having a model that requires ASL-3 Safeguards but is unable to implement those safeguards. My current read is that this is intended to cover the case where the “Capability Decision” report made the case that a model did not cross into requiring ASL-3 Safeguards, was approved by the CEO & RSO, and then later turned out to be wrong. It does seem like this permits more or less indefinite deployment of a model that requires ASL-3 Safeguards by way of “interim measures” which need to provide “the the same level of assurance as the relevant ASL-3 Standard”, with no provision for what to do if it turns out that implementing the actually-specified ASL-3 standard is intractable. This seems slightly worse than the old policy:
If it becomes apparent that the capabilities of a deployed model have been under-elicited and the model can, in fact, pass the evaluations, then we will halt further deployment to new customers and assess existing deployment cases for any serious risks which would constitute a safety emergency. Given the safety buffer, de-deployment should not be necessary in the majority of deployment cases. If we identify a safety emergency, we will work rapidly to implement the minimum additional safeguards needed to allow responsible continued service to existing customers. We will provide transparency and support to impacted customers throughout the process. An emergency of this type would merit a detailed post-mortem and a policy shift to avoid re-occurrence of this situation.
which has much the same immediate impact, but with at least a nod to a post-mortem and policy adjustment.
But, overall, the new policy doesn’t seem to be opening up a gigantic hole that allows Dario to press the “all clear” button on capability determinations; he only has the additional option to veto, after the responsible team has already decided the model doesn’t cross the threshold.
But that’s a communication issue....not a truth issue.
Yes, and Logan is claiming that arguments which cannot be communicated to him in no more than two sentences suffer from a conjunctive complexity burden that renders them “weak”.
That’s not trivial. There’s no proof that there is such a coherent entity as “human values”, there is no proof that AIs will be value-driven agents, etc, etc. You skipped over 99% of the Platonic argument there.
Many possible objections here, but of course spelling everything out would violate Logan’s request for a short argument. Needless to say, that request does not have anything to do with effectively tracking reality, where there is no “platonic” argument for any non-trivial claim describable in only two sentence, and yet things continue to be true in the world anyways, so reductio ad absurdum: there are no valid or useful arguments which can be made for any interesting claims. Let’s all go home now!
A strong good argument has the following properties:
it is logically simple (can be stated in a sentence or two)
This is important, because the longer your argument, the more details that have to be true, and the more likely that you have made a mistake. Outside the realm of pure-mathematics, it is rare for an argument that chains together multiple “therefore”s to not get swamped by the fact that
No, this is obviously wrong.
Argument length is substantially a function of shared premises. I would need many more sentences to convey a novel argument about AI x-risk to someone who had never thought about the subject before, than to someone who has spent a lot of time in the field, because in all likelihood I would first need to communicate and justify many of the foundational concepts that we take for granted.
Note that even here, on LessWrong, this kind of detailed argumentation is necessary to ward off misunderstandings.
Argument strength is not an inverse function with respect to argument length, because not every additional “piece” of an argument is a logical conjunction which, if false, renders the entire argument false. Many details in any specific argument are narrowing down which argument the speaker is making, but are not themselves load-bearing (& conjunctive) claims that all have to be true for the argument to be valid. (These are often necessary; see #1.)
Anyways, the trivial argument that AI doom is likely (given that you already believe we’re likely to develop ASI in the next few decades, and that it will be capable of scientific R&D that sounds like sci-fi today) is that it’s not going to have values that are friendly to humans, because we don’t know how to build AI systems in the current paradigm with any particular set of values at all, and the people pushing frontier AI capabilities mostly don’t think this is a real problem that needs to be figured out[1]. This is self-evidently true, but you (and many others) disagree. What now?
- ^
A moderately uncharitable compression of a more detailed disagreement, which wouldn’t fit into one sentence.
Credit where credit is due: this is much better in terms of sharing one’s models than one could say of Sam Altman, in recent days.
As noted above the footnotes, many people at Anthropic reviewed the essay. I’m surprised that Dario would hire so many people he thinks need to “touch grass” (because they think the scenario he describes in the essay sounds tame), as I’m pretty sure that describes a very large percentage of Anthropic’s first ~150 employees (certainly over 20%, maybe 50%).
My top hypothesis is that this is a snipe meant to signal Dario’s (and Anthropic’s) factional alliance with Serious People; I don’t think Dario actually believes that “less tame” scenarios are fundamentally implausible[1]. Other possibilities that occur to me, with my not very well considered probability estimates:
I’m substantially mistaken about how many early Anthropic employees think “less tame” outcomes are even remotely plausible (20%), and Anthropic did actively try to avoid hiring people with those models early on (1%).
I’m not mistaken about early employee attitudes, but Dario does actually believe AI is extremely likely to be substantially transformative, and extremely unlikely to lead to the “sci-fi”-like scenarios he derides (20%). Conditional on that, he didn’t think it mattered whether his early employees had those models (20%) or might have slightly preferred not, all else equal, but wasn’t that fussed about it compared to recruiting strong technical talent (60%).
I’m just having a lot of trouble reconciling what I know of the beliefs of Anthropic employees, and the things Dario says and implies in this essay. Do Anthropic employees who think less tame outcomes are plausible believe Dario when he says they should “touch grass”? If you don’t feel comfortable answering that question in public, or can’t (due to NDA), please consider whether this is a good situation to be in.
- ^
He has not, as far as I know, deigned to offer any public argument on the subject.
Do you have a mostly disjoint view of AI capabilities between the “extinction from loss of control” scenarios and “extinction by industrial dehumanization” scenarios? Most of my models for how we might go extinct in next decade from loss of control scenarios require the kinds of technological advancement which make “industrial dehumanization” redundant, with highly unfavorable offense/defense balances, so I don’t see how industrial dehumanization itself ends up being the cause of human extinction if we (nominally) solve the control problem, rather than a deliberate or accidental use of technology that ends up killing all humans pretty quickly.
Separately, I don’t understand how encouraging human-specific industries is supposed to work in practice. Do you have a model for maintaining “regulatory capture” in a sustained way, despite having no economic, political, or military power by which to enforce it? (Also, even if we do succeed at that, it doesn’t sound like we get more than the Earth as a retirement home, but I’m confused enough about the proposed equilibrium that I’m not sure that’s the intended implication.)
Yeah, the essay (I think correctly) notes that the most significant breakthroughs in biotech come from the small number of “broad measurement tools or techniques that allow precise but generalized or programmable intervention”, which “are so powerful precisely because they cut through intrinsic complexity and data limitations, directly increasing our understanding and control”.
Why then only such systems limited to the biological domain? Even if it does end up being true that scientific and technological progress is substantially bottlenecked on real-life experimentation, where even AIs that can extract many more bits from the same observations than humans still suffer from substantial serial dependencies with no meaningful “shortcuts”, it still seems implausible that we don’t get to nanotech relatively quickly, if it’s physically realizable. And then that nanotech unblocks the rate of experimentation. (If you’re nanotech skeptical, human-like robots seem sufficient as actuators to speed up real-life experimentation by at least an order of magnitude compared to needing to work through humans, and work on those is making substantial progress.)
If Dario thinks that progress will cap out at some level due to humans intentionally slowing down, it seems good to say this.
Footnote 2 maybe looks like a hint in this direction if you squint, but Dario spent a decent chunk of the essay bracketing outcomes he thought were non-default and would need to be actively steered towards, so it’s interesting that he didn’t explicitly list those (non-tame futures) as a type of of outcome that he’d want to actively steer away from.
Not Mitchell, but at a guess:
LLMs really like lists
Some parts of this do sound a lot like LLM output:
“Complex Intervention Development and Evaluation Framework: A Blueprint for Ethical and Responsible AI Development and Evaluation”
“Addressing Uncertainties”
Many people who post LLM-generated content on LessWrong often wrote it themselves in their native language and had an LLM translate it, so it’s not a crazy prior, though I don’t see any additional reason to have guessed that here.
Having read more of the post now, I do believe it was at least mostly human-written (without this being a claim that it was at least partially written by an LLM). It’s not obvious that it’s particular relevant to LessWrong. The advice on the old internet was “lurk more”; now we show users warnings like this when they’re writing their first post.
I think it pretty much only matters as a trivial refutation of (not-object-level) claims that no “serious” people in the field take AI x-risk concerns seriously, and has no bearing on object-level arguments. My guess is that Hinton is somewhat less confused than Yann but I don’t think he’s talked about his models in very much depth; I’m mostly just going off the high-level arguments I’ve seen him make (which round off to “if we make something much smarter than us that we don’t know how to control, that might go badly for us”).
I don’t really see how this is responding to my comment. I was not arguing about the merits of RLHF along various dimensions, or what various people think about it, but pointing out that calling something “an alignment technique” with no further detail is not helping uninformed readers understand what “RLHF” is better (but rather worse).
Again, please model an uninformed reader: how does the claim “RLHF is an alignment technique” constrain their expectations? If the thing you want to say is that some of the people who invented RLHF saw it as an early stepping stone to solving more challenging problems with alignment later, I have no objection to that claim. This is a claim about the motivations and worldviews of those people. But I don’t know what sort of useful work “RLHF is an alignment technique” is doing, other than making claims that are not centrally about RLHF itself.
This wasn’t part of my original reasoning, but I went and did a search for other uses of “alignment technique” in tag descriptions. There’s one other instance that I can find, which I think could also stand to be rewritten, but at least in that case it’s quite far down the description, well after the object-level details about the proposed technique itself.
I tried to make a similar argument here, and I’m not sure it landed. I think the argument has since demonstrated even more predictive validity with e.g. the various attempts to build and restart nuclear power plants, directly motivated by nearby datacenter buildouts, on top of the obvious effects on chip production.