Mauricio

Karma: 192

Mauricio 14 Nov 2022 20:35 UTC
4 points
0 ∶ 0
in reply to: Aaro Salosensaari’s comment on: The Alignment Community Is Culturally Broken
One specific concern people could have with this thoughtspace is the concern that it’s hard to square with the knowledge that an AI PhD [edit: or rather, AI/ML expertise more broadly] provides. I took this point to be strongly suggested by the author’s suggestions that “experts knowledgeable in the relevant subject matters that would actually lead to doom find this laughable” and that someone who spent their early years “reading/studying deep learning, systems neuroscience, etc.” would not find risk arguments compelling. That’s directly refuted by the surveys (though I agree that some other concerns about this thoughtspace aren’t).

(However, it looks like the author was making a different point to what I first understood.)

Mauricio 14 Nov 2022 8:16 UTC
13 points
5 ∶ 0
in reply to: jacob_cannell’s comment on: The Alignment Community Is Culturally Broken

experts knowledgeable in the relevant subject matters that would actually lead to doom find this laughable

This seems overstated; plenty of AI/ML experts are concerned. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Quoting from [1], a survey of researchers who published at top ML conferences:

The median respondent’s probability of x-risk from humans failing to control AI was 10%

Admittedly, that’s a far cry from “the light cone is about to get ripped to shreds,” but it’s also pretty far from finding those concerns laughable. [Edited to add: another recent survey puts the median estimate of extinction-level extremely bad outcomes at 2%, lower but arguably still not laughable.]

Mauricio 7 Nov 2022 5:45 UTC
5 points
0 ∶ 0
in reply to: Ben Pace’s comment on: Instead of technical research, more people should focus on buying time
Yep! Compiled them here (google doc).

If someone’s been following along with popular LW posts on alignment and is new to governance, I’d expect them to find the “core readings” in “weeks” 4-6 most relevant.

Mauricio 7 Nov 2022 5:11 UTC
5 points
2 ∶ 0
in reply to: habryka’s comment on: Instead of technical research, more people should focus on buying time
I’m sympathetic under some interpretations of “a ton of time,” but I think it’s still worth people’s time to spend at least ~10 hours of reading and ~10 hours of conversation getting caught up with AI governance/strategy thinking, if they want to contribute.

Arguments for this:
- Some basic ideas/knowledge that the field is familiar with (e.g. on the semiconductor supply chain, antitrust law, immigration, US-China relations, how relevant governments and AI labs work, the history of international cooperation in the 20th century) seem really helpful for thinking about this stuff productively.
- First-hand knowledge of how relevant governments and labs work is hard/costly to get on one’s own.
- Lack of shared context makes collaboration with other researchers and funders more costly.
- Even if the field doesn’t know that much and lots of papers are more advocacy pieces, people can learn from what the field does know and read the better content.

Mauricio 6 Nov 2022 0:22 UTC
4 points
1 ∶ 0
on: Instead of technical research, more people should focus on buying time

more researchers should backchain from “how do I make AGI timelines longer

Like you mention, “end time” seems (much) more valuable than earlier time. But the framing here, as well as the broader framing of “buying time,” collapses that distinction (by just using “time” as the metric). So I’d suggest more heavily emphasizing buying end time.

One potential response is: it doesn’t matter; both framings suggest the same interventions. But that seems wrong. For example, slowing down AI progress now seems like it’d mostly buy “pre-end time” (potentially by burning “end time,” if the way we’re slowing down is by safety-conscious labs burning their leads), while setting up standards/regulations/coordination for mitigating racing/unilateralist dynamics at end time buys us end time.

Mauricio 6 Nov 2022 0:00 UTC
17 points
5 ∶ 0
on: Instead of technical research, more people should focus on buying time
Thanks for posting this!

There’s a lot here I agree with (which might not be a surprise). Since the example interventions are all/mostly technical research or outreach to technical researchers, I’d add that a bunch of more “governance-flavored” interventions would also potentially contribute.
- One of the main things that might keep AI companies from coordinating on safety is that some forms of coordination—especially more ambitious coordination—could violate antitrust law.
  - One thing that could help would be updating antitrust law or how it’s enforced so that it doesn’t do a terrible job at balancing anticompetition and safety concerns.
  - Another thing that could help would be a standard-setting organization, since coordination on standards is often more accepted when it’s done in such a context.
    [Added] Can standards be helpful for safety before we have reliably safety methods? I think so; until then, we could imagine standards on things like what types of training runs to not run, or when to stop a training run.
- If some leading AI lab (say, Google Brain) shows itself to be unreceptive to safety outreach and coordination efforts, and if the lead that more safety-conscious labs have over this lab is insufficient, then government action might be necessary to make sure that safety-conscious efforts have the time they need.
To be more direct, I’m nervous that people will (continue to) overlook a promising class of time-buying interventions (government-related ones) through mostly just having learned about government from information sources (e.g. popular news) that are too coarse and unrepresentative to make promising government interventions salient. Some people respond that getting governments to do useful things is clearly too intractable. But I don’t see how they can justifiably be so confident if they haven’t taken the time to form good models of government. At minimum, the US government seems clearly powerful enough (~$1.5 trillion discretionary annual budget, allied with nearly all developed countries, thousands of nukes, biggest military, experienced global spy network, hosts ODA+, etc.) for its interventions to be worth serious consideration.

Mauricio 8 Oct 2022 22:47 UTC
1 point
0 ∶ 0
in reply to: LawrenceC’s comment on: Warning Shots Probably Wouldn’t Change The Picture Much
I agree with a lot of that. Still, if

nuclear non proliferation [to the extent that it has been achieved] is probably harder than a ban on gain-of-function

that’s sufficient to prove Daniel’s original criticism of the OP—that governments can [probably] fail at something yet succeed at some harder thing.

(And on a tangent, I’d guess a salient warning shot—which the OP was conditioning on—would give the US + China strong incentives to discourage risky AI stuff.)

Mauricio 7 Oct 2022 0:01 UTC
3 points
0 ∶ 0
in reply to: Vaniver’s comment on: Warning Shots Probably Wouldn’t Change The Picture Much
I agree it’s some evidence, but that’s a much weaker claim than “probably policy can’t deliver the wins we need.”

Mauricio 6 Oct 2022 23:23 UTC
1 point
0 ∶ 0
in reply to: Vaniver’s comment on: Warning Shots Probably Wouldn’t Change The Picture Much
An earlier comment seems to make a good case that there’s already more community investment in AI policy, and another earlier thread points out that the content in brackets doesn’t seem to involve a good model of policy tractability.

Mauricio 6 Oct 2022 23:16 UTC
3 points
1 ∶ 0
on: Warning Shots Probably Wouldn’t Change The Picture Much
1. Perhaps the sorts of government interventions needed to make AI go well are not all that large, and not that precise.
I confess I don’t really understand this view.

Specifically for the sub-claim that “literal global cooperation” is unnecessary, I think a common element of people’s views is that: the semiconductor supply chain has chokepoints in a few countries, so action from just these few governments can shape what is done with AI everywhere (in a certain range of time).

Mauricio 6 Oct 2022 23:02 UTC
LW: 10 AF: 2
3 ∶ 0
AF
in reply to: LawrenceC’s comment on: Warning Shots Probably Wouldn’t Change The Picture Much
I’d guess the very slow rate of nuclear proliferation has been much harder to achieve than banning gain-of-function research would be, since, in the absence of intervention, incentives to get nukes would have been much bigger than incentives to do gain-of-function research.

Also, on top of the taboo against chemical weapons, there was the verified destruction of most chemical weapons globally.

AI Governance Needs Technical Work

Mauricio5 Sep 2022 22:28 UTC

39 points

1 comment9 min readLW link

Mauricio 13 Jul 2022 5:37 UTC
4 points
2 ∶ 0
on: Slowing down AI progress is an underexplored alignment strategy
Thanks for the post—I think there are some ways heavy regulation of AI could be very counterproductive or ineffective for safety:
- If AI progress slows down enough in countries were safety-concerned people are especially influential, then these countries (and their companies) will fall behind in international AI development. This would eliminate much/most of safety-concerned people’s opportunities for impacting how AI goes.
- If China “catches up” to the US in AI (due to US over-regulation) when AI is looking increasingly economically and militarily important, that could easily motivate US lawmakers to hit the gas on AI (which would at least undo some of the earlier slowing down of AI, and would at worst spark an international race to the bottom on AI).
Also, you mention,

The community strategy (insofar as there even is one) is to bet everything on getting a couple of technical alignment folks onto the team at top research labs in the hopes that they will miraculously solve alignment before the mad scientists in the office next door turn on the doomsday machine.

From conversation, my understanding is some governance/policy folks fortunately have (somewhat) more promising ideas than that. (This doesn’t show up much on this site, partly because: people on here tend to not be as interested in governance, these professionals tend to be busy, the ideas are fairly rough, and getting the optics right can be more important for governance ideas.) I hear there’s some work aimed at posting about some of these ideas—until then, chatting with people (e.g., by reaching out to people at conferences) might be the best way to learn about these ideas.
What links here?
- Mauricio's comment on Slowing down AI progress is an underexplored alignment strategy by Michael Huang (EA Forum; 13 Jul 2022 10:37 UTC; 24 points)

Mauricio 18 Apr 2022 5:29 UTC
1 point
in reply to: mukashi’s comment on: Convince me that humanity *isn’t* doomed by AGI
My problem is that most of the scenarios I see being discussed are dependent on a long chain of assumptions being true and they often seem to ignore that many things could go wrong, invalidating the full thing: you don’t need to be wrong in all those steps, one of them is just enough.

This feels a bit like it might be shifting the goalposts; it seemed like your previous comment was criticizing a specific argumentative step (“reasons not to believe in doom: [...] Orthogonality of intelligence and agency”), rather than just pointing out that there were many argumentative steps.

Anyway, addressing the point about there being many argumentative steps: I partially agree, although I’m not very convinced since there seems to be significant redundancy in arguments for AI risk (e.g., multiple fuzzy heuristics suggesting there’s risk, multiple reasons to expect misalignment, multiple actors who could be careless, multiple ways misaligned AI could gain influence under multiple scenarios).

The different AGIs might find it hard/impossible to coordinate. The different AGIs might even be in conflict with one another

Maybe, although here are six reasons to think otherwise:
- There are reasons to think they will have an easy time coordinating:
  - (1) As mentioned, a very plausible scenario is that many of these AI systems will be copies of some specific model. To the extent that the model has goals, all these copies of any single model would have the same goal. This seems like it would make coordination much easier.
  - (2) Computer programs may be able to give credible signals through open-source code, facilitating cooperation.
  - (3) Focal points of coordination may come up and facilitate coordination, as they often do with humans.
  - (4) If they are initially in conflict, this will create competitive selection pressures for well-coordinated groups (much like how coordinated human states arise from anarchy).
  - (5) They may coordinate due to decision theoretic considerations.
  - (Humans may be able to mitigate coordination earlier on, but this gets harder as their number and/or capabilities grow.)
- (6) Regardless, they might not need to (widely) coordinate; overwhelming numbers of uncoordinated actors may be risky enough (especially if there is some local coordination, which seems likely for the above reasons).

Mauricio 16 Apr 2022 22:47 UTC
3 points
in reply to: mukashi’s comment on: Convince me that humanity *isn’t* doomed by AGI
1. Orthogonality of intelligence and agency. I can envision a machine with high intelligence and zero agency, I haven’t seen any convincing argument yet of why both things must necessarily go together
Hm, what do you make of the following argument? Even assuming (contestably) that intelligence and agency don’t in principle need to go together, in practice they’ll go together because there will appear to be strong economic or geopolitical incentives to build systems that are both highly intelligent and highly agentic (e.g., AI systems that can run teams, companies, or militaries). (And even if some AI developers are cautious enough to not build such systems, less cautious AI developers will, in the absence of strong coordination.)

Also, (2) and (3) seem like reasons why a single AI system may be unable to disempower humanity. Even if we accept that, how relevant will these points be when there is a huge number of highly capable AI systems (which may happen because of the ease and economic benefits of replicating highly capable AI systems)? Their numbers might make up for their limited knowledge and limited plans.

(Admittedly, in these scenarios, people might have significantly more time to figure things out.)

Or as Paul Christiano puts it (potentially in making a different point):

At the same time, it becomes increasingly difficult for humans to directly control what happens in a world where nearly all productive work, including management, investment, and the design of new machines, is being done by machines. We can imagine a scenario in which humans continue to make all goal-oriented decisions about the management of PepsiCo but are assisted by an increasingly elaborate network of prosthetics and assistants. But I think human management becomes increasingly implausible as the size of the world grows (imagine a minority of 7 billion humans trying to manage the equivalent of 7 trillion knowledge workers; then imagine 70 trillion), and as machines’ abilities to plan and decide outstrip humans’ by a widening margin. In this world, the AIs that are left to do their own thing outnumber and outperform those which remain under close management of humans.

Mauricio 11 Mar 2022 6:39 UTC
3 points
on: We’re already in AI takeoff
For people curious about additional discussion of this, Eric Drexler has a closely related comment (focusing on corporate “superhumans” rather than ideological “superhumans”):

[...] The proper (and to some extent actual) goal of firms is widely considered to be maximizing share value, but this is manifestly not the same as maximizing shareholder value — or even benefiting shareholders. [...] Equating share value with shareholder value is obviously wrong (even when considering only share value!) and is potentially lethal. This conceptual error both encourages complacency regarding the alignment of corporate behavior with human interests and undercuts efforts to improve that alignment.”

Mauricio 18 Feb 2022 5:29 UTC
10 points
on: Why I’m co-founding Aligned AI

So we need a way to have alignment deployed throughout the algorithmic world before anyone develops AGI. To do this, we’ll start by offering alignment as a service for more limited AIs.

I’m tentatively fairly excited about some version of this, so I’ll suggest some tweaks that can hopefully be helpful for your success (or for the brainstorming of anyone else who’s thinking about doing something similar in the future).

We will refine and develop this deployment plan, depending on research results, commercial opportunities, feedback, and suggestions.

I suspect there’d be much better commercial/scaling opportunities for a somewhat similar org that offered a more comprehensive, high-quality package of “trustworthy AI services”—e.g., addressing bias, privacy issues, and other more mainstream concerns along with safety/alignment concerns. Then there’d be less of a need to convince companies about paying for some new service—you would mostly just need to convince them that you’re the best provider of services that they’re already interested in. (Cf. ethical AI consulting companies that already exist.)

(One could ask: But wouldn’t the extra price be the same, whether you’re offering alignment in a package or separately? Not necessarily—IP concerns and transaction costs incentivize AI companies to reduce the number of third parties they share their algorithms with.)

As an additional benefit, a more comprehensive package of “trustworthy AI services” would be directly competing for consumers with companies like the AI consulting company mentioned above. This might pressure those companies to start offering safety/alignment services—a mechanism for broadening adoption that isn’t available to an org that only provides alignment services.

[From the website] We are hiring AI safety researchers, ML engineers and other staff.

Relatedly to the earlier point, given that commercial opportunities are a big potential bottleneck (in other words, given that selling limited alignment services might be as much of a communications and persuasion challenge as it is a technical challenge), my intuition would be to also put significant emphasis into hiring people who will kill it at the persuasion: people who are closely familiar with the market and regulatory incentives faced by relevant companies, people with sales and marketing experience, people with otherwise strong communications skills, etc. (in addition to the researchers and engineers).

Mauricio 26 Dec 2021 8:06 UTC
2 points
on: More realistic tales of doom
A more recent clarification from Paul Christiano, on how Part 1 might get locked in / how it relates to concerns about misaligned, power-seeking AI:

I also consider catastrophic versions of “you get what you measure” to be a subset/framing/whatever of “misaligned power-seeking.” I think misaligned power-seeking is the main way the problem is locked in.

Mauricio 26 Dec 2021 6:52 UTC
11 points
AF
on: My Overview of the AI Alignment Landscape: Threat Models
I’m still pretty confused by “You get what you measure” being framed as a distinct threat model from power-seeking AI (rather than as another sub-threat model). I’ll try to address two defenses of that (of framing them as distinct threat models) which I interpret this post as suggesting (in the context of my earlier comment on the overview post). Broadly, I’ll be arguing that: power-seeking AI is necessary for “you get what you measure” concerns being plausible motivators for x-risk-focused people to pursue alignment research, so “you get what you measure” concerns are best thought of as a sub-threat model of power-seeking AI.

(Edit: OK, an aspect of “you get what you measure” concerns—the emphasis on something like “sufficiently strong optimization for some goal is very bad for different goals”—is a tweaked framing of power-seeking AI risk in general, rather than a subset. I think the aspect of “you get what you measure” concerns that’s a subset of power-seeking AI risk is the emphasis on goal misspecification as the cause of misalignment. Either way, not a distinct threat model.)

Lock-in: Once we’ve noticed problems, how difficult will they be to fix, and how much resistance will there be? For example, despite the clear harms of CO2 emissions, fossil fuels are such an indispensable part of the economy that it’s incredibly hard to get rid of them. A similar thing could happen if AI systems become an indispensable part of the economy, which seems pretty plausible given how incredibly useful human-level AI would be. As another example, imagine how hard it would be to ban social media, if we as a society decided that this was net bad for the world.

Unless I’m missing something, this is just an argument for why AI might get locked in—not an argument for why misaligned AI might get locked in. AI becoming an indispensable part of the economy isn’t a long-term problem if people remain capable of identifying and fixing problems with the AI. So we still need an additional lock-in mechanism (e.g. the initially deployed, misaligned AI being power-seeking) to have trouble. (If we’re wondering how hard it will be to fix/improve non-power-seeking AI after it’s been deployed, the difficulty of banning social media doesn’t seem like a great analogy; a more relevant analogy would be the difficulty of fixing/improving social media after it’s been deployed. Empirically, this doesn’t seem that hard. For example, YouTube’s recommendation algorithm started as a click-maximizer, and YouTube has already modified it to learn from human feedback (!).)

See Sam Clarke’s excellent post for more discussion of examples of lock-in.

I don’t think Sam Clarke’s post (which I’m also a fan of) proposes any lock-in mechanisms that (a) are plausible reasons for x-risk-focused people to pursue AI alignment work (and therefore be in the scope of this post) and (b) do not depend on AI being power-seeking. Clarke proposes five mechanisms by which Part 1 of “What Failure Looks Like” could get locked in—addressing each of these in turn (in the context of his original post):
- (1) short-term incentives and collective action—arguably fails condition (a) or fails condition (b); if we don’t assume AI will be power-seeking, then I see no reason why these difficulties would get much worse in hundreds of years than they are now, i.e. no reason why this on its own is a lock-in mechanism.
- (2) regulatory capture—the worry here is that the companies controlling AI might have and permanently act on bad values; this arguably fails condition (a), because if we’re mainly worried about AI developers being bad, then focusing on intent alignment doesn’t make that much sense.
- (3) genuine ambiguity—arguably fails condition (a) or fails condition (b); if we don’t assume AI will be power-seeking, then I see no reason why these difficulties would get much worse in hundreds of years than they are now, i.e. no reason why this on its own is a lock-in mechanism.
- (4) dependency and deskilling—addressed above
- (5) [AI] opposition to [humanity] taking back influence—clearly fails condition (b)
So I think there remains no plausible alignment-relevant threat model for “You get what you measure” that doesn’t fall under “power-seeking AI.” (And even partial deference to Paul Christiano wouldn’t imply that there is, since “What Failure Looks Like” Part 1 features power-seeking AI.)

Mauricio 20 Dec 2021 3:48 UTC
2 points
in reply to: Zvi’s comment on: Zvi’s Thoughts on the Survival and Flourishing Fund (SFF)
Thanks for the thoughtful reply—lots to think more about.

Mauricio

AI Gover­nance Needs Tech­ni­cal Work

AI Governance Needs Technical Work