Akash

Karma: 5,366

Akash 23 Jul 2024 17:44 UTC
3 points
1
in reply to: Richard_Ngo’s comment on: Towards more cooperative AI safety strategies
Thanks!
(I think “being the kind of agent who survives the selection process” can sometimes be an important epistemic thing to consider, though mostly when thinking about how systems work and what kinds of people/views those systems promote. Agreed that “being informed by many people who Y” is a rather weak one & certainly would not on its own warrant a disclosure.)

Akash 23 Jul 2024 13:19 UTC
16 points
9
in reply to: Adam Scholl’s comment on: Towards more cooperative AI safety strategies
An elephant in the room (IMO) is that moving forward, OpenAI probably benefits from a world in which the AI safety community does not have much influence.
There’s a fine line between “play nice with others and be more cooperative” and “don’t actually advocate for policies that you think would help the world, and only do things that the Big Companies and Their Allies are comfortable with.”
Again, I don’t think Richard sat in his room and thought “how do I spread a meme that is good for my company.” I think he’s genuinely saying what he believes and giving advice that he thinks will be useful to the AI safety community and improve society’s future .
But I also think that one of the reasons why Richard still works at OpenAI is because he’s the kind of agent who genuinely believes things that tend to be pretty aligned with OpenAI’s interests, and I suspect his perspective is informed by having lots of friends/colleagues at OpenAI.
Someone who works for a tobacco company can still have genuinely useful advice for the community of people concerned about the health effects of smoking. But I still think it’s an important epistemic norm that they add (at least) a brief disclaimer acknowledging that they work for a tobacco company.
(And the case becomes even stronger in the event that they have to get approval from the tobacco company comms team, or they filter out any ideas that they have that could get them in trouble with the company. Or perhaps before writing/publishing a post they consider the fact that other people have been fired from their company for sharing information that was against company interests, that the CEO attempted to remove a board member under the justification that she published a paper that went against company interests, that the company has previously used history of using highly restrive NDAs to prevent people from saying things that go against company interests.)

Akash 22 Jul 2024 10:29 UTC
4 points
1
in reply to: Richard_Ngo’s comment on: Towards more cooperative AI safety strategies
Separately, do you think “organized opposition” could have ever been avoided? It sounds like you’re making two claims:
- When AI safety folks advocate for specific policies, this gives opponents something to rally around and makes them more likely to organize.
- There are some examples of specific policies (e.g., restrictions on OS, SB1047) that have contributed to this.
Suppose no one said anything about OS, and also (separately) SB1047 never happened. Presumably, at some point, som groups start advocating for specific policies that go against the e/acc worldview. At that point, it seems like you get the organized resistance.
So I’m curious: What does the Ideal Richard World look like? Does it mean people are just much more selective about which policies to advocate for? Under what circumstances is it OK to advocate for something that will increase the political organization of opposing groups? Are there examples of policies that you think are so important that they’re worth the cost (of giving your opposition something to rally around?) To what extent is the deeper crux the fact that you’re less optimistic about the policy proposals actually helping?

Akash 22 Jul 2024 10:18 UTC
3 points
0
in reply to: Richard_Ngo’s comment on: Towards more cooperative AI safety strategies
Thanks for this clarification– I understand your claim better now.
Do you have any more examples of evidence that suggests that AI safety caused (or contributed meaningfully to) this shift from “online meme culture” to “organized political force?” This seems like the biggest crux imo.

Akash 22 Jul 2024 0:43 UTC
15 points
8
in reply to: Richard_Ngo’s comment on: Towards more cooperative AI safety strategies
e/acc has coalesced in defense of open-source, partly in response to AI safety attacks on open-source. This may well lead directly to a strongly anti-AI-regulation Trump White House
IMO this overstates the influence of OS stuff on the broader e/acc movement.
My understanding is that the central e/acc philosophy is around tech progress. Something along the lines of “we want to accelerate technological progress and AGI progress as quickly as possible, because we think technology is extremely awesome and will lead to a bunch of awesome+cool outcomes.” The support for OS is toward the ultimate goal of accelerating technological progress.
In a world where AI safety folks didn’t say/do anything about OS, I would still suspect clashes between e/accs and AI safety folks. AI safety folks generally do not believe that maximally fast/rapid technological progress is good for the world. This would inevitably cause tension between the e/acc worldview and the worldview of many AI safety folks, unless AI safety folks decided never to propose any regulations that could cause us to deviate from the maximally-fast pathways to AGI. This seems quite costly.
(Separately, I agree that “dunking on open-source people” is bad and that people should do less “dunking on X” in general. I don’t really see this as an issue with prioritizing short-term wins so much as getting sucked into ingroup vs. outgroup culture wars and losing sight of one’s actual goals.)
This may well lead directly to a strongly anti-AI-regulation Trump White House
Similar point here– I think it’s extremely likely this would’ve happened anyways. A community that believes passionately in rapid or maximally-fast AGI progress already has strong motivation to fight AI regulations.

Akash 18 Jul 2024 23:53 UTC
6 points
0
in reply to: habryka’s comment on: Towards more cooperative AI safety strategies
As a datapoint, I think I was likely underestimating the level of adversarialness going on & this thread makes me less likely to lump Lightcone in with other parts of the community.
I have definitely taken actions within the bounds of what seems reasonable that have aimed at getting the EA community to shut down or disappear (and will probably continue to do so).
@habryka are you able to share details/examples RE the actions you’ve taken to get the EA community to shut down or disappear?
I also personally do straightforwardly think that most of the efforts of the extended EA-Alignment ecosystem are bad, and would give up a large chunk of my resources to reduce their influence on the world
I would also be interested in more of your thoughts on this. (My Habryka sim says something like “the episemtic norms are bad, and many EA groups/individuals are more focused on playing status games. They are spending their effort saying and doing things that they believe will give them influence points, rather than trying to say true and precise things. I think society’s chances of getting through this critical period would be higher if we focused on reasoning carefully about these complex domains, making accurate arguments, and helping ourselves & others understand the situation.” Curious if this is roughly accurate or if I’m missing some important bits. Also curious if you’re able to expand on this or provide some examples of the things in the category “Many EA people think X action is a good thing to do, but I disagree.”

Akash 18 Jul 2024 23:09 UTC
5 points
3
in reply to: Richard_Ngo’s comment on: Towards more cooperative AI safety strategies
I’m not quite sure where we disagree, but if I had to put my finger on it, it’s something like “I don’t think that people would be offput by Alice going to networking events to try to get a job in housing policy, and I don’t think she would trigger any defense mechanisms.”
Specific question for you: Would you say that “Alice going to a networking event” (assume she’s doing it socially conventional/appropriate ways) would count as structural power-seeking? And would you discourage her from going?
More generally, there are a lot of things you’re labeling as “power-seeking” which feel inaccurate or at least quite unnatural to label as “power-seeking”, and I suspect that this will lead to confusion (or at worst, lead to some of the people you want to engage dismissing your valid points).
I think in your frame, Alice going to networking events would be seen as “there are some socially-accepted ways of seeking power” and in my frame this would be seen as “it doesn’t really make sense to call this power-seeking, as most people would find it ridiculous/weird to apply the label ‘power-seeking’ to an action as simple as going to a networking event.”
I’m also a bit worried about a motte-and-bailey here. The bold statement is “power-seeking (which I’m kind of defining as anything that increases your influence, regardless of how innocuous or socially accepted it seems) is bad because it triggers defense mechanisms” and the more moderated statement is “there are some specific ways of seeking power that have important social costs, and I think that some/many actors in the community underestimate those costs. Also, there are many strategies for achieving your goals that don’t involve seeking power, and I think some/many people in the community are underestimating those.”
I agree with the more moderated claims.

Akash 18 Jul 2024 21:31 UTC
3 points
0
in reply to: Richard_Ngo’s comment on: Towards more cooperative AI safety strategies
Influence-seeking activates the same kind of feeling though it’s less strong than for “power-seeking.”
but if Alice just reasoned from the top down about how to optimize her networking really hard for her career, in a non-socially-skilled way, a friend should pull her aside
+1. I suspect we’d also likely agree that if Alice just stayed in her room all day and only talked to her friends about what ideal housing policy should look like, someone should pull her aside and say “hey, you might want to go to some networking events and see if you can get involved in housing policy, or at least see if there are other things you should be doing to become a better fit for housing policy roles in the future.”
In this case, it’s not the desire to have influence that is the core problem. The core problem is whether or not Alice is taking the right moves to have the kind of influence she wants.
Bringing it back to the post– I think I’d be excited to see you write something more along the lines of “What mistakes do many people in the AIS community make when it comes to influence-seeking?” I suspect this would be more specific and concrete. I think the two suggestions at the end (prioritize legitimacy & prioritize competence) start to get at this.
Otherwise, I feel like the discussion is going to go into less productive directions, where people who already agree with you react like “Yeah, Alice is such a status-seeker! Stick it to her!” and people who disagree with you are like “Wait what? Alice is just trying to network so she can get a job in housing policy– why are you trying to cast that as some shady plot? Should she just stay in her room and write blog posts?”

Akash 18 Jul 2024 19:53 UTC
5 points
−1
on: Towards more cooperative AI safety strategies
I agree with many of the points expressed in this post, though something doesn’t sit right with me about some of the language/phrasing used.
For example, the terms “power-seeking” and “cooperative” feel somewhat loaded. It’s not so much that they’re inaccurate (when read in a rather precise and charitable way) but moreso that it feels like they have pretty strong connotations and valences.
Consider:
Alice: I’m going to a networking event tonight; I might meet someone who can help me get a job in housing policy!
Bob: That’s a power-seeking move.
Alice: Uh… what?
Bob: Well, you know, if you get a job, then that increases your power. It increases your ability to influence the world.
Alice: I guess me getting a job does technically increase my ability to influence the world, so if that’s how you want to define “power-seeking” then you’re technically correct, but that’s not really the first word that comes to mind here. We usually use the word “power-seeking” to refer to bad people who are overly concerned with power-seeking– usually for personal or selfish gain at the expense of others. And I don’t really think that’s what I’m doing.
Separately, I’d be curious how you’re defining “cooperative” in this context. (Does it mean “not power-seeking” or “strategies that focus more on sharing information with the public and making sure that competent people are in charge regardless of their views on AI safety”, or something else?)

Akash 18 Jul 2024 19:42 UTC
59 points
58
on: Towards more cooperative AI safety strategies
Meta: I think these kinds of posts should include some sort of disclaimer acknowledging that you are an OpenAI employee & also mentioning whether or not the post was reviewed by OpenAI staff, OpenAI comms, etc.
I imagine you didn’t do this because many people who read this forum are aware of this fact (and it’s on your profile– it’s not like you’re trying to hide it), but I suspect this information could be useful for newcomers who are engaging with this kind of material.

Akash 12 Jul 2024 21:45 UTC
10 points
1
in reply to: Neel Nanda’s comment on: Neel Nanda’s Shortform
Separately, while I think the discussion around “is X net negative” can be useful, I think it ends up implicitly putting the frame on “can X justify that they are not net negative.”
I suspect the quality of discourse– and society’s chances to have positive futures– would improve if the frame were more commonly something like “what are the best actions for X to be taken” or “what are reasonable/high-value things that X could be doing.”
And I think it’s valid to think “X is net positive” while also thinking “I feel disappointed in X because I don’t think it’s using its power/resources in ways that would produce significantly better outcomes.”
IDK what the bar should be for considering X a “responsible actor”, but I imagine my personal bar is quite a bit higher than “(barely) net positive in expectation.”
P.S. Both of these comments are on the opinionated side, so separately, I just wanted to say thank you Neel for speaking up & for offering your current takes on Anthropic. Strong upvoted!

Akash 12 Jul 2024 21:40 UTC
33 points
34
in reply to: Neel Nanda’s comment on: Neel Nanda’s Shortform
I’m a bit worried about a dynamic where smart technical folks end up feeling like “well, I’m kind of disappointed in Anthropic’s comms/policy stuff from what I hear, and I do wish they’d be more transparent, but policy is complicated and I’m not really a policy expert”.
To be clear, this is a quite reasonable position for any given technical researcher to have– the problem is that this provides pretty little accountability. In a world where Anthropic was (hypothetically) dishonest, misleading, actively trying to undermine/weaken regulations, or putting its own interests above the interests of the “commons”, it seems to me like many technical researchers (even Anthropic staff) would not be aware of this. Or they might get some negative vibes but then slip back into a “well, I’m not a policy person, and policy is complicated” mentality.
I’m not saying there’s even necessarily a strong case that Anthropic is trying to sabotage policy efforts (though I am somewhat concerned about some of the rhetoric Anthropic uses, public comments about thinking its too early to regulate, rumors that they have taken actions to oppose SB 1047, and a lack of any real “positive” signals from their positive team like EG recommending or developing policy proposals that go beyond voluntary commitments or encouraging people to measure risks.)
But I think once upon a time there was some story that if Anthropic defected in major ways, a lot of technical researchers would get concerned and quit/whistleblow. I think Anthropic’s current comms strategy, combined with the secrecy around a lot of policy things, combined with a general attitude (whether justified or unjustified) of “policy is complicated and I’m a technical person so I’m just going to defer to Dario/Jack” makes me concerned that safety-concerned people won’t be able to hold Anthropic accountable even if it actively sabotages policy stuff.
I’m also not really sure if there’s an easy solution to this problem, but I do imagine part of the solution involves technical people (especially at Anthropic) raising questions, asking people like Jack and Dario to explain their takes more, and being more willing to raise public & private discussions about Anthropic’s role in the broader policy space.

Advice to junior AI governance researchers

Akash8 Jul 2024 19:19 UTC

64 points

1 comment5 min readLW link

Akash 3 Jul 2024 0:42 UTC
4 points
2
on: An AI Race With China Can Be Better Than Not Racing
A common scheme for a conversation about pausing the development of transformative AI goes like this:
Minor: The first linked post is not about pausing AI development. It mentions various interventions for “buying time” (like evals and outreach) but it’s not about an AI pause. (When I hear the phrase “pausing AI development” I think more about the FLI version of this which is like “let’s all pause for X months” and less about things like “let’s have labs do evals so that they can choose to pause if they see clear evidence of risk”.)
At a basic level, we want to estimate how much worse (or, perhaps, better) it would be for the United States to completely cede the race for TAI to the PRC.
My impression is that (most? many?) pause advocates are not talking about completely ceding the race to the PRC. I would guess that if you asked (most? many?) people who describe themselves as “pro-pause”, they would say things like “I want to pause to give governments time to catch up and figure out what regulations are needed” or “I want to pause to see if we can develop AGI in a more secure way, such as (but not limited to) something like MAGIC.”
I doubt many of them would say “I would be in favor of a pause if it meant that the US stopped doing AI development and we completely ceded the race to China.” I would suspect many of them might say something like “I would be in favor of a pause in which the US sees if China is down to cooperate, but if China is not down to cooperate, then I would be in favor of the US lifting the pause.”

Akash 30 Jun 2024 16:43 UTC
13 points
3
on: Akash’s Shortform
Recommended reading: A recent piece argues that the US-China crisis hotline doesn’t work & generally raises some concerns about US-China crisis communication.
Some quick thoughts:
- If the claims in the piece are true, there seem to be some (seemingly tractable) ways of substantially improving US-China crisis communication.
- The barriers seem more bureaucratic (understanding how the defense world works and getting specific agencies/people to do specific things) than political (I doubt this is something you need Congress to pass new legislation to improve.)
- In general, I feel like “how do we improve our communication infrastructure during AI-related crises” is an important and underexplored area of AI policy. This isn’t just true for US-China communication but also for “lab-government communication”, “whistleblower-government communication”, and “junior AI staffer-senior national security advisor” communication.
  - Example: Suppose an eval goes off that suggests that an AI-related emergency might be imminent. How do we make sure this information swiftly gets to relevant people? To what extent do UKAISI and USAISI folks (or lab whistleblowers) have access to senior national security folks who would actually be able to respond in a quick or effective way?
- I think IAPS’ CDDC paper is a useful contribution here. I will soon be releasing a few papers in this broad space, with a focus on interventions that can improve emergency detection + emergency response.
- One benefit of workshops/conferences/Track 2 dialogues might simply be that you get relevant people to meet each other, share contact information, build trust/positive vibes, and be more likely to reach out in the event of an emergency scenario.
- Establishing things like the AI Safety and Security Board might also be useful for similar reasons. I think this has gotten a fair amount of criticism for being too industry-focused, and some of that is justified. Nonetheless, I think interventions along the lines of “make it easy for the people who might see the first signs of extreme risk have super clear ways of advising/contacting government officials” seem great.

Akash 30 Jun 2024 16:28 UTC
2 points
1
in reply to: Fabien Roger’s comment on: Fabien’s Shortform
Thanks! In general, I like these bite-sized summaries of various things you’re reading. Seems like a win for the commons, and I hope more folks engaging with governance/policy stuff do things like this.

Akash 28 Jun 2024 21:02 UTC
4 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
Do you feel like there are any benefits or drawbacks specifically tied to the fact that you’re doing this work as a contractor? (compared to a world where you were not a contractor but Anthropic just gave you model access to run these particular experiments and let Evan/Carson review your docs)

Akash 25 Jun 2024 0:48 UTC
2 points
2
in reply to: Richard_Ngo’s comment on: Buck’s Shortform
@Richard_Ngo do you have any alternative approaches in mind that are less susceptible to regulatory capture? At first glance, I think this broad argument can be applied to any situation where the government regulates anything. (There’s always some risk that R focuses on the wrong things or R experiences corporate/governmental pressure to push things through).
I do agree that the broader or more flexible the regulatory regime is, the more susceptible it might be to regulatory capture. (But again, this feels like it doesn’t really have much to do with safety cases– this is just a question of whether we want flexible or fixed/inflexible regulations in general.)

Akash 25 Jun 2024 0:42 UTC
LW: 3 AF: 2
−1
AF
in reply to: Buck’s comment on: Buck’s Shortform
Here’s how I understand your argument:
1. Some people are advocating for safety cases– the idea that companies should be required to show that risks drop below acceptable levels.
2. This approach is used in safety engineering fields.
3. But AI is different from the safety engineering fields. For example, in AI we have adversarial risks.
4. Therefore we shouldn’t support safety cases.
I think this misunderstands the case for safety cases, or at least only argues against one particular justification for safety cases.
Here’s how I think about safety cases (or really any approach in which a company needs to present evidence that their practices keep risks below acceptable levels):
1. AI systems pose major risks. A lot of risks stem from race dynamics and competitive pressures.
2. If companies were required to demonstrate that they kept risks below acceptable levels, this would incentivize a lot more safety research and curb some of the dangerous properties of race dynamics.
3. Other fields also have similar setups, and we should try to learn from them when relevant. Of course, AI development will also have some unique properties so we’ll have to adapt the methods accordingly.
I’d be curious to hear more about why you think safety cases fail to work when risks are adversarial (at first glance, it doesn’t seem like it should be too difficult to adapt the high-level safety case approach).
I’m also curious if you have any alternatives that you prefer. I currently endorse the claim “safety cases are better than status quo” but I’m open to the idea that maybe “Alternative approach X is better than both safety cases and status quo.”

Akash 21 Jun 2024 20:24 UTC
5 points
0
on: AI takeoff and nuclear war
Interesting analysis! I think it’ll be useful for more folks to think about the nuclear/geopolitical implications of AGI development, especially in worlds where governments are paying more attention & one or more nuclear powers experience a “wakeup” or “sudden increase in situational awareness.”
Some specific thoughts:
Of these two risks, it is likely simpler to work to reduce the risk of failure to navigate.
Can you say more about why you believe this? At first glance, it seems to be like “fundamental instability” is much more tied to how AI development goes, so I would’ve expected it to be more tractable [among LW users]. Whereas “failure to navigate” seems further outside our spheres of influence– it seems to me like there would be a lot of intelligence agency analysts, defense people, and national security advisors who are contributing to discussions about whether or not to go to war. Seems plausible that maybe a well-written analysis from folks in the AI safety community could be useful, but my impression is that it would be pretty hard to make a splash here since (a) things would be so fast-moving, (b) a lot of the valuable information about the geopolitical scene will be held by people working in government and people with security clearances, making it harder for outside people to reason about things, and (c) even conditional on valuable analysis, the stakeholders who will be deferred to are (mostly) going to be natsec/defense stakeholders.
3) In the aftermath of a nuclear war, surviving powers would be more fearful and hostile.
4) There would be greater incentives to rush for powerful AI, and less effort expended on going carefully or considering pausing.
There are lots of common-sense reasons why nuclear war is bad. That said, I’d be curious to learn more about how confident you are in these statements. In a post-catastrophe world, it seems quite plausible to me that the rebounding civilizations would fear existential catastrophes and dangerous technologies and try hard to avoid technology-induced catastrophes. I also just think such scenarios are very hard to reason about, such that there’s a lot of uncertainty around whether AI progress would be faster (bc civs are fearful of each other and hostile) or slower (because civs are are fearful of technology-induced catastrophes and generally have more of a safety/security mindset.)

Akash

Ad­vice to ju­nior AI gov­er­nance researchers

Advice to junior AI governance researchers