Akash

Karma: 4,408

Akash 19 Apr 2024 0:46 UTC
6 points
−3
on: Express interest in an “FHI of the West”
To what extent would the organization be factoring in transformative AI timelines? It seems to me like the kinds of questions one would prioritize in a “normal period” look very different than the kinds of questions that one would prioritize if they place non-trivial probability on “AI may kill everyone in <10 years” or “AI may become better than humans on nearly all cognitive tasks in <10 years.”
I ask partly because I personally would be more excited of a version of this that wasn’t ignoring AGI timelines, but I think a version of this that’s not ignoring AGI timelines would probably be quite different from the intellectual spirit/tradition of FHI.
More generally, perhaps it would be good for you to describe some ways in which you expect this to be different than FHI. I think the calling it the FHI of the West, the explicit statement that it would have the intellectual tradition of FHI, and the announcement right when FHI dissolves might make it seem like “I want to copy FHI” as opposed to “OK obviously I don’t want to copy it entirely I just want to draw on some of its excellent intellectual/cultural components.” If your vision is the latter, I’d find it helpful to see a list of things that you expect to be similar/different.)

Akash 19 Apr 2024 0:25 UTC
31 points
13
in reply to: peterbarnett’s comment on: peterbarnett’s Shortform
I would strongly suggest considering hires who would be based in DC (or who would hop between DC and Berkeley). In my experience, being in DC (or being familiar with DC & having a network in DC) is extremely valuable for being able to shape policy discussions, know what kinds of research questions matter, know what kinds of things policymakers are paying attention to, etc.
I would go as far as to say something like “in 6 months, if MIRI’s technical governance team has not achieved very much, one of my top 3 reasons for why MIRI failed would be that they did not engage enough with DC people//US policy people. As a result, they focused too much on questions that Bay Area people are interested in and too little on questions that Congressional offices and executive branch agencies are interested in. And relatedly, they didn’t get enough feedback from DC people. And relatedly, even the good ideas they had didn’t get communicated frequently enough or fast enough to relevant policymakers. And relatedly… etc etc.”
I do understand this trades off against everyone being in the same place, which is a significant factor, but I think the cost is worth it.

Akash 19 Apr 2024 0:18 UTC
6 points
2
in reply to: Alexander Gietelink Oldenziel’s comment on: Akash’s Shortform
I do think evaporative cooling is a concern, especially if everyone (or a very significant amount) of people left. But I think on the margin more people should be leaving to work in govt.
I also suspect that a lot of systemic incentives will keep a greater-than-optimal proportion of safety-conscious people at labs as opposed to governments (labs pay more, labs are faster and have less bureaucracy, lab people are much more informed about AI, labs are more “cool/fun/fast-paced”, lots of govt jobs force you to move locations, etc.)
I also think it depends on the specific lab– EG in light of the recent OpenAI departures, I suspect there’s a stronger case for staying at OpenAI right now than for DeepMind or Anthropic.

Akash 18 Apr 2024 15:48 UTC
13 points
2
on: AI #60: Oh the Humanity
Daniel Kokotajlo has quit OpenAI
I think now is a good time for people at labs to seriously consider quitting & getting involved in government/policy efforts.
I don’t think everyone should leave labs (obviously). But I would probably hit a button that does something like “everyone at a lab governance team and many technical researchers spend at least 2 hours thinking/writing about alternative options they have & very seriously consider leaving.”
My impression is that lab governance is much less tractable (lab folks have already thought a lot more about AGI) and less promising (competitive pressures are dominating) than government-focused work.
I think governments still remain unsure about what to do, and there’s a lot of potential for folks like Daniel K to have a meaningful role in shaping policy, helping natsec folks understand specific threat models, and raising awareness about the specific kinds of things governments need to do in order to mitigate risks.
There may be specific opportunities at labs that are very high-impact, but I think if someone at a lab is “not really sure if what they’re doing is making a big difference”, I would probably hit a button that allocates them toward government work or government-focused comms work.

Akash 18 Apr 2024 15:44 UTC
41 points
15
on: Akash’s Shortform
I think now is a good time for people at labs to seriously consider quitting & getting involved in government/policy efforts.
I don’t think everyone should leave labs (obviously). But I would probably hit a button that does something like “everyone at a lab governance team and many technical researchers spend at least 2 hours thinking/writing about alternative options they have & very seriously consider leaving.”
My impression is that lab governance is much less tractable (lab folks have already thought a lot more about AGI) and less promising (competitive pressures are dominating) than government-focused work.
I think governments still remain unsure about what to do, and there’s a lot of potential for folks like Daniel K to have a meaningful role in shaping policy, helping natsec folks understand specific threat models, and raising awareness about the specific kinds of things governments need to do in order to mitigate risks.
There may be specific opportunities at labs that are very high-impact, but I think if someone at a lab is “not really sure if what they’re doing is making a big difference”, I would probably hit a button that allocates them toward government work or government-focused comms work.
Written on a Slack channel in response to discussions about some folks leaving OpenAI.

Akash 17 Apr 2024 14:27 UTC
36 points
23
in reply to: habryka’s comment on: Paul Christiano named as US AI Safety Institute Head of AI Safety
I think this should be broken down into two questions:
1. Before the EO, if we were asked to figure out where this kind of evals should happen, what institution would we pick & why?
2. After the EO, where does it make sense for evals-focused people to work?
I think the answer to #1 is quite unclear. I personally think that there was a strong case that a natsec-focused USAISI could have been given to DHS or DoE or some interagency thing. In addition to the point about technical expertise, it does seem relatively rare for Commerce/NIST to take on something that is so natsec-focused.
But I think the answer to #2 is pretty clear. The EO clearly tasks NIST with this role, and now I think our collective goal should be to try to make sure NIST can execute as effectively as possible. Perhaps there will be future opportunities to establish new places for evals work, alignment work, risk monitoring and forecasting work, emergency preparedness planning, etc etc. But for now, whether we think it was the best choice or not, NIST/USAISI are clearly the folks who are tasked with taking the lead on evals + standards.

Akash 16 Apr 2024 20:48 UTC
56 points
23
on: Paul Christiano named as US AI Safety Institute Head of AI Safety
I’m excited to see how Paul performs in the new role. He’s obviously very qualified on a technical level, and I suspect he’s one of the best people for the job of designing and conducting evals.
I’m more uncertain about the kind of influence he’ll have on various AI policy and AI national security discussions. And I mean uncertain in the genuine “this could go so many different ways” kind of way.
Like, it wouldn’t be particularly surprising to me if any of the following occurred:
- Paul focuses nearly all of his efforts on technical evals and doesn’t get very involved in broader policy conversations
- Paul is regularly asked to contribute to broader policy discussions, and he advocates for RSPs and other forms of voluntary commitments.
- Paul is regularly asked to contribute to broader policy discussions, and he advocates for requirements that go beyond voluntary commitments and are much more ambitious than what he advocated for when he was at ARC.
- Paul is regularly asked to contribute to broader policy discussions, and he’s not very good at communicating his beliefs in ways that are clear/concise/policymaker-friendly, so his influence on policy discussions is rather limited.
- Paul [is/isn’t] able to work well with others who have very different worldviews and priorities.
Personally, I see this as a very exciting opportunity for Paul to form an identity as a leader in AI policy. I’m guessing the technical work will be his priority (and indeed, it’s what he’s being explicitly hired to do), but I hope he also finds ways to just generally improve the US government’s understanding of AI risk and the likelihood of implementing reasonable policies. On the flipside, I hope he doesn’t settle for voluntary commitments (especially as the Overton Window shifts) & I hope he’s clear/open about the limitations of RSPs.
More specifically, I hope he’s able to help policymakers reason about a critical question: what do we do after we’ve identified models with (certain kinds of) dangerous capabilities? I think the underlying logic behind RSPs could actually be somewhat meaningfully applied to USG policy. Like, I think we would be in a safer world if the USG had an internal understanding of ASL levels, took seriously the possibility of various dangerous capabilities thresholds being crossed, took seriously the idea that AGI/ASI could be developed soon, and had preparedness plans in place that allowed them to react quickly in the event of a sudden risk.
Anyways, a big congratulations to Paul, and definitely some evidence that the USAISI is capable of hiring some technical powerhouses.

Akash 16 Apr 2024 20:22 UTC
4 points
0
in reply to: ryan_greenblatt’s comment on: What convincing warning shot could help prevent extinction from AI?
I’ll admit I have only been loosely following the control stuff, but FWIW I would be excited about a potential @peterbarnett & @ryan_greenblatt dialogue in which you two to try to identify & analyze any potential disagreements. Example questions:
- What is the most capable system that you think we are likely to be able to control?
- What kind of value do you think we could get out of such a system?
- To what extent do you expect that system to be able to produce insights that help us escape the acute risk period (i.e., get out of a scenario where someone else can come along and build a catastrophe-capable system without implementing control procedures or someone else comes along and scales to the point where the control procedures are no longer sufficient)

Akash 15 Apr 2024 23:02 UTC
10 points
8
in reply to: habryka’s comment on: Anthropic AI made the right call
Here are three possible scenarios:
Scenario 1, Active Lying– Anthropic staff were actively spreading the idea that they would not push the frontier.
Scenario 2, Allowing misconceptions to go unchecked– Anthropic staff were aware that many folks in the AIS world thought that Anthropic had committed to not pushing the frontier, and they allowed this misconception to go unchecked, perhaps because they realized that it was a misconception that favored their commercial/competitive interests.
Scenario 3, Not being aware– Anthropic staff were not aware that many folks had this belief. Maybe they heard it once or twice but it never really seemed like a big deal.
Scenario 1 is clearly bad. Scenarios 2 and 3 are more interesting. To what extent does Anthropic have the responsibility to clarify misconceptions (avoid scenario 2) and even actively look for misconceptions (avoid scenario 3)?
I expect this could matter tangibly for discussions of RSPs. My opinion is that the Anthropic RSP is written in such a way that readers can come away with rather different expectations of what kinds of circumstances would cause Anthropic to pause/resume.
It wouldn’t be very surprising to me if we end up seeing a situation where many readers say “hey look, we’ve reached an ASL-3 system, so now you’re going to pause, right?” And then Anthropic says “no no, we have sufficient safeguards– we can keep going now.” And then some readers say “wait a second– what? I’m pretty sure you committed to pausing until your safeguards were better than that.” And then Anthropic says “no… we never said exactly what kinds of safeguards we would need, and our leadership’s opinion is that our safeguards are sufficient, and the RSP allows leadership to determine when it’s fine to proceed.”
In this (hypothetical) scenario, Anthropic never lied, but it benefitted from giving off a more cautious impression, and it didn’t take steps to correct this impression.
I think avoiding these kinds of scenarios requires some mix of:
- Clear, specific falsifiable statements on behalf of labs.
- Some degree of proactive attempts to identify and alleviate misconceptions.
One counterargument is something like “Anthropic is a company, and there are lots of things to do, and this is is demanding an unusually high amount of attention-to-detail and proactive communication that is not typically expected of companies.” To which my response is something like “yes, but I think it’s reasonable to hold companies to such standards if they wish to develop AGI. I think we ought to hold Anthropic and other labs to this standard, especially insofar as they want the benefits associated with being perceived as the kind of safety-conscious lab that refuses to push the frontier or commits to scaling policies that include tangible/concrete plans to pause.”

Akash 15 Apr 2024 21:32 UTC
20 points
6
in reply to: peterbarnett’s comment on: What convincing warning shot could help prevent extinction from AI?
I like these examples. One thing I’ll note, however, is that I think the “warning shot discourse” on LW tends to focus on warning shots that would be convincing to a LW-style audience.
If the theory of the change behind the warning shot requires LW-types (for example, folks at OpenAI/DeepMind/Anthropic who are relatively familiar with AGI xrisk arguments) to become concerned, this makes sense.
But usually, when I think about winning worlds that involve warning shots, I think about government involvement as the main force driving coordination, an end to race dynamics, etc.
[Caveating that my models of the USG and natsec community are still forming, so epistemic status is quite uncertain for the rest of this message, but I figure some degree of speculation could be helpful anyways].
I expect the kinds of warning shots that would be concerning to governments/national security folks will look quite different than the kinds of warning shots that would be convincing to technical experts.
LW-style warning shots tend to be more– for a lack of a better term– rational. They tend to be rooted in actual threat models (e.g., we understand that if an AI can copy its weights, it can create tons of copies and avoid being easily turned off, and we also understand that its general capabilities are sufficiently strong that we may be close to an intelligence explosion or highly capable AGI).
In contrast, without this context, I don’t think that “we caught an AI model copying its weights” would necessarily be a warning shot for USG/natsec folks. It could just come across as “oh something weird happened but then the company caught it and fixed it.” Instead, I suspect the warning shots that are most relevant to natsec folks might be less “rational”, and by that I mean “less rooted in actual AGI threat models but more rooted in intuitive things that seem scary.”
Examples of warning shots that I expect USG/natsec people would be more concerned about:
- An AI system can generate novel weapons of mass destruction
- Someone uses an AI system to hack into critical infrastructure or develop new zero-days that impress folks in the intelligence community.
- A sudden increase in the military capabilities of AI
These don’t relate as much to misalignment risk or misalignment-focused xrisk threat models. As a result, a disadvantage of these warning shots is that it may be harder to make a convincing case for interventions that focus specifically on misalignment. However, I think they are the kinds of things that might involve a sudden increase in the attention that the USG places on AI/AGI, the amount of resources it invests into understanding national security threats from AI, and its willingness to take major actions to intervene in the current AI race.
As such, in addition to asking questions like “what is the kind of warning shot that would convince me and my friends that we have something dangerous”, I think it’s worth separately asking “what is the kind of warning shot that would convince natsec folks that something dangerous or important is occurring, regardless of whether or not it connects neatly to AGI risk threat models.”
My impression is that the policy-relevant warning shots will be the most important ones to be prepared for, and the community may (for cultural/social/psychological reasons) be focusing too little effort on trying to prepare for these kinds of “irrational” warning shots.

Akash 28 Feb 2024 21:28 UTC
3 points
1
in reply to: Aryeh Englander’s comment on: Speaking to Congressional staffers about AI risk
If memory serves me well, I was informed by Hendrycks’ overview of catastrophic risks. I don’t think it’s a perfect categorization, but I think it does a good job laying out some risks that feel “less speculative” (e.g., malicious use, race dynamics as a risk factor that could cause all sorts of threats) while including those that have been painted as “more speculative” (e.g., rogue AIs).
I’ve updated toward the importance of explaining & emphasizing risks from sudden improvements in AI capabilities, AIs that can automate AI research, and intelligence explosions. I also think there’s more appetite for that now than there used to be.

Akash 23 Feb 2024 14:22 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
What work do you think is most valuable on the margin (for those who agree with you on many of these points)?

Akash 26 Jan 2024 10:48 UTC
6 points
8
in reply to: habryka’s comment on: The case for ensuring that powerful AIs are controlled
I’d be curious to hear more about your main disagreements.

Akash 26 Jan 2024 10:47 UTC
4 points
3
in reply to: joshc’s comment on: The case for ensuring that powerful AIs are controlled
Good point. I think it’s helpful when people working schemes with the rough flavor of “we do X, and then X helps us get to a useful AI that does not takeover” try to specify roughly how capable they expect the “useful AI does not takeover” to be.
Would be curious to hear more about the kinds of tasks that Ryan and Buck expect the first “transformatively useful but still controllable” AI will be able to do (perhaps expressed in multiplier terms, like 30X human SW engineer, perhaps expressed in terms of the kinds of novel reasoning it can perform, and perhaps expressed in terms of the kinds of research tasks or policy tasks it would be able to meaningfully help with).

Akash 25 Jan 2024 12:19 UTC
LW: 13 AF: 2
6
AF
on: The case for ensuring that powerful AIs are controlled
I’m excited to see how the AI control research direction evolves.
After skimming the post, it seems like it makes a few different claims, some of which seem more intuitive than others. For example:
- Claim 1: Labs should control powerful AI systems (very intuitive, even obvious)
- Claim 2: It is (much?) easier to evaluate control than alignment (somewhat intuitive, particularly if one buys the argument that alignment evals are very hard because you always have the “what if the AI is just scheming and subverting the eval” concern.)
- Claim 3: We (Buck and Ryan) have identified some specific techniques that we believe are promising for control (not intuitive; technical arguments needed).
- Claim 4: If we succeed in controlling “transformatively useful AI”, then we may be able to stop the race toward superintelligence and get AI labs or governments to agree to not build superintelligence until it can be controlled (not intuitive; non-technical arguments needed).
I’d be excited to see more posts that specifically engage with the strongest counterpoints to claims #2-4.
Some more on #2 & #4:
The basic problem with evaluating alignment is that no matter what behaviors you observe, you have to worry that your model is just acting that way in order to make you think that it is aligned.
I think those pessimistic about control evals could say something like “the basic problem with evaluating control is that no matter what techniques your red-team uses, you have to worry that your model is better at finding attacks than your red-team.” Of course, you note in the post some reason why we should expect our red-team to have advantages over models, but also you recognize that this won’t scale toward arbitrarily powerful AIs.
In some ways, this feels analogous to the following situation:
- Alice from the Scaling Department: Look, our current “alignment” techniques are working for GPT-4. So let’s scale up more! I get that at some point we won’t be able to align the AI, but we’re not there yet, so we should be able to keep going.
- Bob from the Preparedness Team: Well, the problem is we don’t know when our alignment techniques will stop working… like, it’s quite plausible that the next AI system will be able to game our evals.
- Alice from the Scaling Department: OK, well let me know when you have clear evidence that our alignment techniques or dangerous capability evals will fail.
Here’s the analogy for control:
- Alice from the Scaling Department: Look, our current control techniques are working for GPT-4. So let’s scale up more! I get that at some point our control evals aren’t going to be able to control arbitrarily powerful AI, but we’re not there yet, so we should be able to keep going.
- Bob from the Preparedness Team: Well, the problem is we don’t know when our control evals will stop working… like, it’s quite plausible that the next AI system will be better at coming up with attacks than our red-team.
- Alice from the Scaling Department: OK, well let me know when you have clear evidence that our control evals will fail.
I’d be curious to hear more about how you’re thinking about this (and apologies if some sections of the post already deal with this– feel free to quote them if I missed them in my initial skim). Specific questions:
1. How do you react to the analogy above? Do you expect that control evals will do a better job than other types of evals at determining when we have reached the point where scaling further is (directly/imminently) dangerous? Or more simply, will it be clear when we need to stop?
2. More broadly, what do you think we should do once we’re starting to see AI systems that are powerful enough to subvert the control evals (or develop attacks that are better than those of the human red teamers)?
3. Suppose control evals do allow us to stop at “transformatively useful systems that do not cause a catastrophe.” Roughly what kinds of capabilities do you expect this system to have? (My vague MIRI sim says something like “either the system is too weak to get us out of the critical risk window or the system is so strong that it’s capable of subverting your control evals”).

Akash 20 Jan 2024 11:56 UTC
LW: 21 AF: 6
19
AF
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
More broadly, TurnTrout, I’ve noticed you using this whole “look, if something positive happened, LW would totally rip on it! But if something is presented negatively, everyone loves it!” line of reasoning a few times (e.g., I think this logic came up in your comment about Evan’s recent paper). And I sort of see you taking on some sort of “the people with high P(doom) just have bad epistemics” flag in some of your comments.
A few thoughts (written quickly, prioritizing speed over precision):
1. I think that epistemics are hard & there are surely several cases in which people are biased toward high P(doom). Examples: Yudkowsky was one of the first thinkers/writers about AI, some people might have emotional dispositions that lead them toward anxious/negative interpretations in general, some people find it “cool” to think they’re one of the few people who are able to accurately identify the world is ending, etc.
2. I also think that there are plenty of factors biasing epistemics in the “hopeful” direction. Examples: The AI labs have tons of money and status (& employ large fractions of the community’s talent), some people might have emotional dispositions that lead them toward overly optimistic/rosy interpretations in general, some people might find it psychologically difficult to accept premises that lead them to think the world is ending, etc.
3. My impression (which could be false) is that you seem to be exclusively or disproportionately critical of poor arguments when they come from the “high P(doom)” side.
4. I also think there’s an important distinction between “I personally think this argument is wrong” and “look, here’s an example of propaganda + poor community epistemics.” In general, I suspect community epistemics are better when people tend to respond directly to object-level points and have a relatively high bar for saying “not only do I think you’re wrong, but also here are some ways in which you and your allies have poor epistemics.” (IDK though, insofar as you actually believe that’s what’s happening, it seems good to say aloud, and I think there’s a version of this that goes too far and polices speech reproductively, but I do think that statements like “community epistemics have been compromised by groupthink and fear” are pretty unproductive and could be met with statements like “community epistemics have been compromised by powerful billion-dollar companies that have clear financial incentives to make people overly optimistic about the trajectory of AI progress.”
5. I am quite worried about tribal dynamics reducing the ability for people to engage in productive truth-seeking discussions. I think you’ve pointed out how some of the stylistic/tonal things from the “high P(doom)//alignment hard” side have historically made discourse harder, and I agree with several of your critiques. More recently, though, I think that the “low P(doom)//alignment not hard” side seem to be falling into similar traps (e.g., attacking strawmen of those they disagree with, engaging some sort of “ha, the other side is not only wrong but also just dumb/unreasonable/epistemically corrupted” vibe that predictably makes people defensive & makes discourse harder.

Akash 20 Jan 2024 11:27 UTC
20 points
18
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
My impression is that the Shoggath meme was meant to be a simple meme that says “hey, you might think that RLHF ‘actually’ makes models do what we value, but that’s not true. You’re still left with an alien creature who you don’t understand and could be quite scary.”
Most of the Shoggath memes I’ve seen look more like this, where the disgusting/evil aspects are toned down. They depict an alien that kinda looks like an octopus. I do agree that the picture evokes some sort of “I should be scared/concerned” reaction. But I don’t think it does so in a “see, AI will definitely be evil” way– it does so in a “look, RLHF just adds a smiley face to a foreign alien thing. And yeah, it’s pretty reasonable to be scared about this foreign alien thing that we don’t understand.”
To be a bit bolder, I think Shoggath is reacting to the fact that RLHF gives off a misleading impression of how safe AI is. If I were to use proactive phrasing, I could say that RLHF serves as “propaganda”. Let’s put aside the fact that you and I might disagree about how much “true evidence” RLHF provides RE how easy alignment will be. It seems pretty clear to me that RLHF [and the subsequent deployment of RLHF’d models] spreads an overly-rosy “meme” that gives people a misleading perspective of how well we understand AI systems, how safe AI progress is, etc.
From this lens, I see Shoggath as a counter-meme. It basically says “hey look, the default is for people to think that these things are friendly assistants, because that’s what the AI companies have turned them into, but we should remember that actually we are quite confused about the alien cognition behind the RLHF smiley face.”

Akash 2 Jan 2024 14:38 UTC
16 points
0
on: The Plan − 2023 Version
If the timer starts to run out, then slap something together based on the best understanding we have. 18-24 months is about how long I expect it to take to slap something together based on the best understanding we have.
Can you say more about what you expect to be doing after you have slapped together your favorite plans/recommendations? I’m interested in getting a more concrete understanding of how you see your research (eventually) getting implemented.
Suppose after the 18-24 month process, you have 1-5 concrete suggestions that you want AGI developers to implement. Is the idea essentially that you would go to the superalignment team (and the equivalents at other labs) and say “hi, here’s my argument for why you should do X?” What kinds of implementation-related problems, if any, do you see coming up?
I ask this partially because I think some people are kinda like “well, in order to do alignment research that ends up being relevant, I need to work at one of the big scaling labs in order to understand the frames/ontologies of people at the labs, the constraints/restrictions that would come up if trying to implement certain ideas, get better models of the cultures of labs to see what ideas will simply be dismissed immediately, identify cruxes, figure out who actually makes decisions about what kinds of alignment ideas will end up being used for GPT-N, etc etc.”
My guess is that you would generally encourage people to not do this, because they generally won’t have as much research freedom & therefore won’t be able to work on core parts of the problem that you see as neglected. I suspect many would agree that there is some “I lose freedom” cost, but that this might be outweighed by the “I get better models of what kind of research labs are actually likely to implement” benefit, and I’m curious how you view this trade-off (or if you don’t even see this as a legitimate trade-off).

Akash 30 Dec 2023 11:24 UTC
11 points
3
in reply to: Daniel Kokotajlo’s comment on: The Plan − 2023 Version
Does the “rater problem” (raters have systematic errors) simply apply to step one in this plan? I agree that once you have a perfect reward model, you no longer need human raters.
But it seems like the “rater problem” still applies if we’re going to train the reward model using human feedback. Perhaps I’m too anchored to thinking about things in an RLHF context, but it seems like at some point in the process we need to have some way of saying “this is true” or “this chain-of-thought is deceptive” that involves human raters.
Is the idea something like:
- Eliezer: Human raters make systematic errors
- OpenAI: Yes, but this is only a problem if we have human raters indefinitely provide feedback. If human raters are expected to provide feedback on 10,000,000 responses under time-pressure, then surely they will make systematic errors.
- OpenAI: But suppose we could train a reward model on a modest number of responses and we didn’t have time-pressure. For this dataset of, say, 10,000 responses, we are super careful, we get a ton of people to double-check that everything is accurate, and we are nearly certain that every single label is correct. If we train a reward model on this dataset, and we can get it to generalize properly, then we can get past the “humans make systematic errors” problem.
Or am I totally off//the idea is different than this//the “yet-to-be-worked-out-techniques” would involve getting the reward model to learn stuff without ever needing feedback from human raters?

Akash 29 Dec 2023 8:57 UTC
5 points
1
in reply to: aogara’s comment on: NYT is suing OpenAI&Microsoft for alleged copyright infringement; some quick thoughts
I’d also be curious to know why (some) people downvoted this.
Perhaps it’s because you imply that some OpenAI folks were captured, and maybe some people think that that’s unwarranted in this case?
Sadly, the more-likely explanation (IMO) is that policy discussions can easily become tribal, even on LessWrong.
I think LW still does better than most places at rewarding discourse that’s thoughtful/thought-provoking and resisting tribal impulses, but I wouldn’t be surprised if some people were doing something like “ah he is saying something Against AI Labs//Pro-regulation, and that is bad under my worldview, therefore downvote.”
(And I also think this happens the other way around as well, and I’m sure people who write things that are “pro AI labs//anti-regulation” are sometimes unfairly downvoted by people in the opposite tribe.)