Linch

[Linkpost] State­ment from Scar­lett Jo­hans­son on OpenAI’s use of the “Sky” voice, that was shock­ingly similar to her own voice.

• Linch 20 May 2024 23:29 UTC

  13 points

  2

  in reply to: MichaelDickens’s comment on: OpenAI: Exodus

  This has been OpenAI’s line (whether implicitly or explicitly) for a while iiuc. I referenced it on my Open Asteroid Impact website, under “Operation Death Star.”

• Linch 20 May 2024 21:38 UTC

  52 points

  11

  on: Open Thread Spring 2024

  Do we know if @paulfchristiano or other ex-lab people working on AI policy have non-disparagement agreements with OpenAI or other AI companies? I know Cullen doesn’t, but I don’t know about anybody else.

  I know NIST isn’t a regulatory body, but it still seems like standards-setting should be done by people who have no unusual legal obligations. And of course, some other people are or will be working at regulatory bodies, which may have more teeth in the future.
  
  To be clear, I want to differentiate between Non-Disclosure Agreements, which are perfectly sane and reasonable in at least a limited form as a way to prevent leaking trade secrets, and non-disparagement agreements, which prevents you from saying bad things about past employers. The latter seems clearly bad to have for anybody in a position to affect policy. Doubly so if the existence of the non-disparagement agreement itself is secretive.

• Linch 20 May 2024 8:52 UTC

  3 points

  4

  in reply to: James Payor’s comment on: Ilya Sutskever and Jan Leike re­sign from OpenAI

  I’d be a bit surprised if that’s the answer, if OpenAI doesn’t offer any vested equity, that half-truth feels overly blatant to me.

• Linch 19 May 2024 0:55 UTC

  2 points

  0

  in reply to: Rebecca’s comment on: Ilya Sutskever and Jan Leike re­sign from OpenAI

  OpenAI has something called PPUs (“Profit Participation Units”) which in theory is supposed to act like RSUs albeit with a capped profit and no voting rights, but in practice is entirely a new legal invention and we don’t really know how it works.

• Linch 17 May 2024 2:55 UTC

  6 points

  4

  in reply to: Zach Stein-Perlman’s comment on: Ilya Sutskever and Jan Leike re­sign from OpenAI

  I agree it’s not a large commitment in some absolute sense. I think it’d still be instructive to see whether they’re able to hit this (not very high) bar.

• Linch 15 May 2024 6:44 UTC

  22 points

  14

  on: Ilya Sutskever and Jan Leike re­sign from OpenAI

  It’ll be interesting to see if OpenAI will keep going with their compute commitments now that the two main superalignment leads have left.

  What links here?

  • OpenAI: Exodus by Zvi (20 May 2024 13:10 UTC; 111 points)

• Linch 13 May 2024 18:29 UTC

  6 points

  1

  in reply to: Linch’s comment on: Linch’s Shortform

  I’m interested in what people think of are the strongest arguments against this view. Here are a few counterarguments that I’m aware of:
  
  1. Empirically the AI-focused scaling labs seem to care quite a lot about safety, and make credible commitments for safety. If anything, they seem to be “ahead of the curve” compared to larger tech companies or governments.

  2. Government/​intergovernmental agencies, and to a lesser degree larger companies, are bureaucratic and sclerotic and generally less competent.

  3. The AGI safety issues that EAs worry about the most are abstract and speculative, so having a “normal” safety culture isn’t as helpful as buying in into the more abstract arguments, which you might expect to be easier to do for newer companies.

  4. Scaling labs share “my” values. So AI doom aside, all else equal, you might still want scaling labs to “win” over democratically elected governments/​populist control.

• Linch 13 May 2024 5:22 UTC

  43 points

  8

  on: Linch’s Shortform

  (x-posted from the EA Forum)

  We should expect that the incentives and culture for AI-focused companies to make them uniquely terrible for producing safe AGI.

  From a “safety from catastrophic risk” perspective, I suspect an “AI-focused company” (e.g. Anthropic, OpenAI, Mistral) is abstractly pretty close to the worst possible organizational structure for getting us towards AGI. I have two distinct but related reasons:

  1. Incentives

  2. Culture

  From an incentives perspective, consider realistic alternative organizational structures to “AI-focused company” that nonetheless has enough firepower to host multibillion-dollar scientific/​engineering projects:

  1. As part of an intergovernmental effort (e.g. CERN’s Large Hadron Collider, the ISS)

  2. As part of a governmental effort of a single country (e.g. Apollo Program, Manhattan Project, China’s Tiangong)

  3. As part of a larger company (e.g. Google DeepMind, Meta AI)

  In each of those cases, I claim that there are stronger (though still not ideal) organizational incentives to slow down, pause/​stop, or roll back deployment if there is sufficient evidence or reason to believe that further development can result in major catastrophe. In contrast, an AI-focused company has every incentive to go ahead on AI when the cause for pausing is uncertain, and minimal incentive to stop or even take things slowly.

  From a culture perspective, I claim that without knowing any details of the specific companies, you should expect AI-focused companies to be more likely than plausible contenders to have the following cultural elements:

  1. Ideological AGI Vision AI-focused companies may have a large contingent of “true believers” who are ideologically motivated to make AGI at all costs and

  2. No Pre-existing Safety Culture AI-focused companies may have minimal or no strong “safety” culture where people deeply understand, have experience in, and are motivated by a desire to avoid catastrophic outcomes.

  The first one should be self-explanatory. The second one is a bit more complicated, but basically I think it’s hard to have a safety-focused culture just by “wanting it” hard enough in the abstract, or by talking a big game. Instead, institutions (relatively) have more of a safe & robust culture if they have previously suffered the (large) costs of not focusing enough on safety.
  
  For example, engineers who aren’t software engineers understand fairly deep down that their mistakes can kill people, and that their predecessors’ fuck-up have indeed killed people (think bridges collapsing, airplanes falling, medicines not working, etc). Software engineers rarely have such experience.
  
  Similarly, governmental institutions have institutional memories with the problems of major historical fuckups, in a way that new startups very much don’t.

• Linch 7 May 2024 16:58 UTC

  6 points

  2

  in reply to: PhilosophicalSoul’s comment on: William_S’s Shortform

  I can see some arguments in your direction but would tentatively guess the opposite.

• Linch 3 May 2024 21:40 UTC

  34 points

  0

  in reply to: habryka’s comment on: William_S’s Shortform

  (not a lawyer)

  My layman’s understanding is that managerial employees are excluded from that ruling, unfortunately. Which I think applies to William_S if I read his comment correctly. (See Pg 11, in the “Excluded” section in the linked pdf in your link)

• Linch 1 May 2024 0:46 UTC

  2 points

  0

  in reply to: ektimo’s comment on: Open Thread Spring 2024

  “This is more of a comment than a question” as they say

• Linch 30 Apr 2024 23:29 UTC

  2 points

  0

  in reply to: Matthew Barnett’s comment on: Los­ing Faith In Con­trar­i­anism

  Yeah that’s fair! I agree that they would lose the bet as stated.

• Linch 27 Apr 2024 21:50 UTC

  2 points

  0

  in reply to: Matthew Barnett’s comment on: Los­ing Faith In Con­trar­i­anism

  Rebuttal here!

  Anyway, if the message someone received from Hanson’s writings on medicine was “yay Hanson”, and Scott’s response was “boo Hanson,” then I agree people should wait for Hanson’s rebuttal before being like “boo Hanson.”
  
  But if the message that people received was “medicine doesn’t work” (and it appears that many people did), then Scott’s writings should be an useful update, independent of whether Hanson’s-writings-as-intended was actually trying to deliver that message.

• Linch 16 Apr 2024 0:14 UTC

  5 points

  0

  on: Linch’s Shortform

  People might appreciate this short (<3 minutes) video interviewing me about my April 1 startup, Open Asteroid Impact:
  
  

• Linch 10 Apr 2024 21:38 UTC

  2 points

  0

  in reply to: scrollop’s comment on: In­tro­duc­ing Open As­teroid Impact

  Alas I think doing this will be prohibitively expensive/​technologically infeasible. We did some BOTECs at the launch party and even just getting rid of leap seconds was too expensive for us.

  That’s one of many reasons why I’m trying to raise 7 trillion dollars.

• Linch 10 Apr 2024 19:14 UTC

  2 points

  0

  in reply to: magic9mushroom’s comment on: In­tro­duc­ing Open As­teroid Impact

  Thanks for the pro-tip! I’m not much of a geologist, more of an ideas guy^[1] myself.

  1. ^

     “I can handle the business end”

• Linch 1 Apr 2024 21:59 UTC

  29 points

  7

  in reply to: gwern’s comment on: In­tro­duc­ing Open As­teroid Impact

  Open Asteroid Impact strongly disagrees with this line of thinking. Our theory of change relies on many asteroids filled with precious minerals hitting earth, as mining in space (even LEO) is prohibitively expensive compared to on-ground mining.

  While your claims may be true for small asteroids, we strongly believe that scale is all you need. Over time, sufficiently large, and sufficiently many, asteroids can solve the problem of specific asteroids not successfully impacting Earth.
  

• Linch 1 Apr 2024 19:21 UTC

  13 points

  7

  in reply to: roha’s comment on: In­tro­duc­ing Open As­teroid Impact

  rare earth metals? More like common space metals, amirite?

[April Fools’ Day] In­tro­duc­ing Open As­teroid Impact