joshc

Safety stan­dards: a frame­work for AI regulation

• joshcApr 26, 2023, 9:05 PM

  1 point

  0

  on: Refram­ing the bur­den of proof: Com­pa­nies should prove that mod­els are safe (rather than ex­pect­ing au­di­tors to prove that mod­els are dan­ger­ous)

  I agree with this norm, though I think it would be better to say that the “burden of evidence” should be on labs. When I first read the title, I thought you wanted labs to somehow prove the safety of their system in a conclusive way. What this probably looks like in practice is “we put x resources into red teaming and didn’t find any problems.” I would be surprised if ‘proof’ was ever an appropriate term.

• joshcMar 14, 2023, 6:16 AM

  7 points

  0

  on: AI al­ign­ment re­searchers don’t (seem to) stack

  The analogy between AI safety and math or physics is assumed it in a lot of your writing and I think it is a source of major disagreement with other thinkers. ML capabilities clearly isn’t the kind of field that requires building representations over the course of decades.

  I think it’s possible that AI safety requires more conceptual depth than AI capabilities; but in these worlds, I struggle to see how the current ML paradigm coincides with conceptual ‘solutions’ that can’t be found via iteration at the end. In those worlds, we are probably doomed so I’m betting on the worlds in which you are wrong and we must operate within the current empirical ML paradigm. It’s odd to me that you and Eliezer seem to think the current situation is very intractable, and yet you are confident enough in your beliefs to where you won’t operate on the assumption that you are wrong about something in order to bet on a more tractable world.

• joshcMar 14, 2023, 5:55 AM

  3 points

  0

  in reply to: Zach Stein-Perlman’s comment on: AI al­ign­ment re­searchers don’t (seem to) stack

  +1. As a toy model, consider how the expected maximum of a sample from a heavy tailed distribution is affected by sample size. I simulated this once and the relationship was approximately linear. But Soares’ point still holds if any individual bet requires a minimum amount of time to pay off. You can scalably benefit from parallelism while still requiring a minimum amount of serial time.

Are short timelines ac­tu­ally bad?

[MLSN #7]: an ex­am­ple of an emer­gent in­ter­nal optimizer

• joshcJan 8, 2023, 9:48 PM

  1 point

  0

  on: Without spe­cific coun­ter­mea­sures, the eas­iest path to trans­for­ma­tive AI likely leads to AI takeover

  At its core, the argument appears to be “reward maximizing consequentialists will necessarily get the most reward.” Here’s a counter example to this claim: if you trained a Go playing AI with RL, you are unlikely to get a reward maximizing consequentialist. Why? There’s no reason for the Go playing AI to think about how to take over the world or hack the computer that is running the game. Thinking this way would be a waste of computation. AIs that think about how to win within the boundaries of the rules therefore do better.
  
  In the same way, if you could robustly enforce rules like “turn off when the humans tell you to” or “change your goal when humans tell you to” etc, perhaps you end up with agents that follow these rules rather than agents that think “hmmm… can I get away with being disobedient?”
  
  Both achieve the same reward if the rules are consistently enforced during training and I think there are weak reasons to expect deontological agents to be more likely.

• joshcDec 8, 2022, 5:39 PM

  3 points

  0

  in reply to: Heighn’s comment on: You should con­sider launch­ing an AI startup

  Also, AI startups make AI safety resources more likely to scale with AI capabilities.

• joshcDec 4, 2022, 6:28 AM

  1 point

  0

  in reply to: P.’s comment on: Prizes for ML Safety Bench­mark Ideas

  It hasn’t been canceled.

• joshcNov 7, 2022, 3:00 AM

  1 point

  0

  in reply to: joshc’s comment on: A philoso­pher’s cri­tique of RLHF

  I don’t mean to distract from your overall point though which I take to be “a philosopher said a smart thing about AI alignment despite not having much exposure.” That’s useful data.

• joshcNov 7, 2022, 2:56 AM

  1 point

  0

  on: A philoso­pher’s cri­tique of RLHF

  I don’t know why this is a critique of RLHF. You can use RLHF to train a model to ask you questions when it’s confused about your preferences. To the extent to which you can easily identify when a system has this behavior and when it doesn’t, you can use RLHF to produce this behavior just like you can use RLHF to produce many other desired behaviors.

• joshcNov 5, 2022, 3:42 AM

  1 point

  0

  in reply to: Thomas Larsen’s comment on: (My un­der­stand­ing of) What Every­one in Tech­ni­cal Align­ment is Do­ing and Why

  Thoughts on John’s comment: this is a problem with any method for detecting deception that isn’t 100% accurate. I agree that finding a 100% accurate method would be nice, but good luck.
  
  Also, you can somewhat get around this by holding some deception detecting methods out (i.e. not optimizing against them). When you finish training and the held out methods tell you that your AI is deceptive, you start over. Then you have to try to think of another approach that is more likely to actually discourage deception than fool your held out detectors. This is the difference between gradient descent search and human design search, which I think is an important distinction.

  Also, FWIW, I doubt that trojans are currently a good microcosm for detecting deception. Right now, it is too easy to search for the trigger using brute force optimization. If you ported this over to sequential-decision-making land where triggers can be long and complicated, that would help a lot. I see a lot of current trojan detection research as laying the groundwork for future research that will be more relevant.
  
  In general, it seems better to me to evaluate research by asking “where is this taking the field/​what follow-up research is this motivating?” rather than “how are the words in this paper directly useful if we had to build AGI right now?” Eventually, the second one is what matters, but until we have systems that look more like agents that plan and achieve goals in the real world, I’m pretty skeptical of a lot of the direct value of empirical research.

• joshcOct 28, 2022, 8:59 PM

  LW: 1 AF: 1

  0

  AF

  in reply to: zeshen’s comment on: My Thoughts on the ML Safety Course

  From what I understand, Dan plans to add more object-level arguments soon.

Prizes for ML Safety Bench­mark Ideas

• joshcOct 2, 2022, 4:02 AM

  LW: 2 AF: 2

  0

  AF

  on: My Thoughts on the ML Safety Course

  (opinions are my own)
  I think this is a good review. Some points that resonated with me:
  1. “The concepts of systemic safety, monitoring, robustness, and alignment seem rather fuzzy.” I don’t think the difference between objective and capabilities robustness is discussed but this distinction seems important. Also, I agree that Truthful AI could easily go into monitoring.
  2. “Lack of concrete threat models.” At the beginning of the course, there are a few broad arguments for why AI might be dangerous but not a lot of concrete failure modes. Adding more failure modes here would better motivate the material.

  3. Give more clarity on how the various ML safety techniques address the alignment problem, and how they can potentially scale to solve bigger problems of a similar nature as AIs scale in capabilities
  4. Give an assessment on the most pressing issues that should be addressed by the ML community and the potential work that can be done to contribute to the ML safety field

  You can read more about how these technical problems relate to AGI failure modes and how they rank on importance, tractability, and crowdedness in Pragmatic AI Safety 5. I think the creators included this content in a separate forum post for a reason.
  
  The course is intended for to two audiences: people who are already worried about AI X-risk and people who are only interested in the technical content. The second group doesn’t necessarily care about why each research direction relates to reducing X-risk.
  
  Putting a lot of emphasis on this might just turn them off. It could give them the impression that you have to buy X-risk arguments in order to work on these problems (which I don’t think is true) or it could make them less likely to recommend the course to others, causing fewer people to engage with the X-risk material overall.

• joshcSep 5, 2022, 5:29 AM

  1 point

  0

  in reply to: Vladimir_Nesov’s comment on: The AGI has to ac­tu­ally ‘care’ about humans

  These are good points. Maybe we’ll align these things enough to where they’ll give us a little hamster tank to run around in.

• joshcSep 5, 2022, 5:17 AM

  LW: 7 AF: 2

  2

  AF

  in reply to: Thomas Larsen’s comment on: (My un­der­stand­ing of) What Every­one in Tech­ni­cal Align­ment is Do­ing and Why

  PAIS #5 might be helpful here. It explains how a variety of empirical directions are related to X-Risk and probably includes many of the ones that academics are working on.

[Question] What is the best cri­tique of AI ex­is­ten­tial risk ar­gu­ments?

• joshcAug 29, 2022, 2:36 AM

  1 point

  0

  in reply to: Shmi’s comment on: The AGI has to ac­tu­ally ‘care’ about humans

  I agree that I did not justify this claim and it is controversial in the ML community. I’ll try to explain why I think this.
  
  First of all, when I say an AI ‘wants things’ or is ‘an agent’ I just mean it robustly and autonomously brings about specific outcomes. I think all of the arguments made above apply with this definition and don’t rely on anthropomorphism.
  
  Why are we likely to build superintelligent agents?
  1. It will be possible to build superintelligent agents
  I’m just going to take it as a given here for the sake of time. Note: I’m not claiming it will be easy, but I don’t think anyone can really say with confidence that it will be impossible to build such agents within the century.
  2. There will be strong incentives to build superintelligent agents
  I’m generally pretty skeptical of claims that if you just train an AI system for long enough it becomes a scary consequentialist. Instead, I think it is likely someone will build a system like this because humans want things and building a superintelligent agent that wants to do what you want pretty much solves all your problems. For example, they might want to remain technologically relevant/​be able to defend themselves from other countries/​extraterrestrial civilizations, expand human civilization, etc. Building a thing that ‘robustly and autonomously brings about specific outcomes’ would be helpful for all this stuff.
  
  In order to use current systems (‘tool AIs’ as some people call them) to actually get things done in the world, a human needs to be in the loop, which is pretty uncompetitive with fully autonomous agents.

  I wouldn’t be super surprised if humans didn’t build superintelligent agents for a while—even if they could. Like I mentioned in the post, most people prob want to stay in control. But I’d put >50% credence on it happening pretty soon after it is possible because of coordination difficulty and the unilateralist curse.

• joshcAug 28, 2022, 3:03 AM

  1 point

  0

  in reply to: Ben Pace’s comment on: An­nounc­ing En­cul­tured AI: Build­ing a Video Game

  I don’t doubt this. I was more reporting on how the branding came across to me.