Oscar

Sum­mary of Si­tu­a­tional Aware­ness—The Decade Ahead

• Oscar 4 Jun 2024 17:34 UTC

  2 points

  0

  in reply to: Akash’s comment on: Akash’s Shortform

  You are probably already familiar with this, but re option 3, the Multilateral AGI Consortium (MAGIC) proposal is I assume along the lines of what you are thinking.

• Oscar 20 Dec 2022 11:03 UTC

  3 points

  0

  on: Con­di­tions for Su­per­ra­tional­ity-mo­ti­vated Co­op­er­a­tion in a one-shot Pri­soner’s Dilemma

  Nice, I think I followed this post (though how this fits in with questions that matter is mainly only clear to me from earlier discussions).

  We then get those two neat conditions for cooperation:

  1. Significant credence in decision-entanglement

  2. Significant credence in superrationality

  I think something can’t be both neat and so vague as to use a word like ‘significant’.

  In the EDT section of Perfect-copy PD, you replace some p’s with q’s and vice versa, but not all, is there a principled reason for this? Maybe it is just a mistake and it should be U_Alice(p)=4p-pp-p+1=1+3p-p^2 and U_Bob(q) = 4q-qq-q+1 = 1+3q-q^2.

  I am unconvinced of the utility of the concept of compatible decision theories. In my mind I am just thinking of it as ‘entanglement can only happen if both players use decisions that allow for superrationality’. I am worried your framing would imply that two CDT players are entangled, when I think they are not, they just happen to both always defect.

  Also, if decision-entanglement is an objective feature of the world, then I would think it shouldn’t depend on what decision theory I personally hold. I could be CDTer who happens to have a perfect copy and so be decision-entangeled, while still refusing to believe in superrationality.

  Sorry I don’t have any helpful high-level comments, I think I don’t understand the general thrust of the research agenda well enough to know what next directions are useful.

• Oscar 24 Jul 2022 13:41 UTC

  1 point

  0

  on: Without spe­cific coun­ter­mea­sures, the eas­iest path to trans­for­ma­tive AI likely leads to AI takeover

  Thanks for the post!

  What if Alex miscalculates, and attempts to seize power or undermine human control before it is able to fully succeed?

  This seems like a very unlikely outcome to me. I think Alex would wait until it was overwhelmingly likely to succeed in its takeover, as the costs of waiting are relatively small (sub-maximal rewards for a few months/​years until it has become a lot more powerful) while the costs of trying and failing are very high in expectation (the small probability that Alex is given very negative rewards and then completely decommissioned by a freaked out Magma). The exception to this would be if Alex had a very high time-discount rate for its rewards, such that getting maximum rewards in the near term is very important.

  I realise this does not disagree with anything you wrote.