simon

• simon 9 Jun 2023 17:09 UTC

  1 point

  0

  in reply to: aphyer’s comment on: D&D.Sci 5E: Re­turn of the League of Defen­ders Eval­u­a­tion & Ruleset

  NP aphyer, I didn’t ask for any more time, though I was happy to get some extra due to you extending for yonge. I hadn’t been particularly focused on it for a while, until trying to get things figured out at the last minute, largely I think due to me having spent a greatly disproportionate-to-value effort on figuring out how to do similarity clustering on a highly reduced (and thus much more random) version of the dataset, and then not knowing what to do with the results once I got them. (though I did learn stuff about finding the similarity clustering, so that was good).

  Looks like the clusters I found in the reduced dataset more or less corresponded to:

  either an aggressive 2-ranged character or everything fairly tanky (FLR cluster)

  tending towards tankier 2-ranged and aggressive 1-ranged (melee) character (HSM cluster, note I had excluded B and D from this dataset)

  tending towards more aggression to the back (JGP cluster)

  So now I’m trying to figure out why the observed FLR>HSM>JGP>FLR rock-paper scissors effect occurred...

  edit: a just-so story (don’t know if real reason):

  JGP vs FLR: FLR loses the melee first, then likely loses the 2-range since very squishy, then doomed.

  FLR vs HSM: HSM loses the melee first. Then FLR might well lose the 2-range first, depending on initiative. FLR would then be splitting damage, but since HSM’s 2-range is already damaged and FLR’s tank typically isn’t that tanky, HSM’s 2 range might well die before FLR’s backline? dunno, seems weak explanation

  HSM vs JGP: HSM loses the melee first. But then, the tanky 2-range of HSM tends to last a while, and the tanky melee of JGP doesn’t contribute much. Once JGP loses its 2-range, it splits damage between HSM’s remaining characters, while HSM focuses and defeats JGP’s squishy backline?

• simon 9 Jun 2023 16:20 UTC

  3 points

  0

  on: D&D.Sci 5E: Re­turn of the League of Defen­ders Eval­u­a­tion & Ruleset

  Thanks for the scenario, aphyer.

  I made a last minute PVE change which didn’t get into the results, but looks like it would have gotten 64.44% winrate which is still lower than gjm’s. Congrats to gjm and abstractapplic. I also had previously changed my PVE selection which also isn’t in the results, but that change didn’t make any difference—it was still 50%.

  Interesting ruleset that has some complicated behaviour, but still allows analysis. I think it was actually quite good in this respect, though even with the extension I didn’t really get to a point where I felt I was done.

  If I had continued the analysis, my next thing to look at would have been how different candidate PVP teams, plus yonge’s PVP team, interacted with different team compositions (classified according to the groups

  which corresponded to range 1, range 2, and range 3+).

  Not sure what I would have ended up concluding from this.

• simon 9 Jun 2023 15:37 UTC

  1 point

  0

  in reply to: simon’s comment on: D&D.Sci 5E: Re­turn of the League of Defenders

  Update in view of the answer likely being soon to be posted:

  I got sidetracked among other (non-D&DSci) things by trying to semi-automatically categorize the team compositions in the games with only the restricted team compositions (one character from each group, no trash picks) into similarity clusters. This was tricky because there is a lot of noise in this much smaller dataset, and I didn’t take into account games outside this restricted set at all.

  Ultimately, I did get three clusters which seemed to have a rock-paper-scissors interaction. One cluster is Felon-heavy (indeed seems to maybe have all Felon teams) and FLR seems to be a fairly archetypal example. Another cluster is Samurai-heavy and Golem-light; HSM seems to be a fairly archetypal example. The third cluster is Pyro-heavy and JGP seems to be a fairly archetypal example.

  Anyway, the FLR cluster tends to beat the HSM cluster which tends to beat the JGP cluster which tends to beat the FLR cluster.

  The PVE opposing team, FSR, mostly seems to be in the FLR cluster but is not very central, leaning a bit to the HSM cluster. It hasn’t faced the JGP cluster a lot (maybe 5-6 games depending on cluster definition) and has won maybe 3 or 4 of those, atypical for an FLR cluster member, but that could easily be random due to the low number of games.

  Notably, my current PVP pick, CLP, seems to be in the JGP cluster and, as is typical for members of this cluster, tends to lose to members of the HSM cluster. In the absence of reasons to believe that other players have picked teams from the HSM cluster (hmm, but yonge picked HMP (which isn’t in this restricted dataset since it has two characters from the same group) - would that behave like HSM??) I don’t see a compelling reason to switch, though I might change my mind if I post this comment and then the answer isn’t posted for a long time.

  Anyway, I’m not sure whether the rock-paper-scissors effect seen in the clustering derives from some collective interaction or is just a result of character pair interactions. Some apparent counters in this restricted dataset:

  F>S;P>F;G>F;F>R;J>P;S>J;R>J;S>C;P>L;S>P;C>F

  Also:

  I’ve now gone and looked at what FSR wins against and adjusted my PVE pick accordingly. I’ll likely adjust my PVP pick as well if I end up having time to check what sort of things candidate PVP picks (and other players’ PVP picks where posted) do well against.

  edit: looks like this comment was after aphyer posted the answer, but I checked for any new posts after my PVE edit above and didn’t see aphyer’s post of the answer.

• simon 3 Jun 2023 20:28 UTC

  1 point

  0

  in reply to: Noosphere89’s comment on: AI #14: A Very Good Sentence

  On deontology, there’s actually an analysis on whether deontological AI are safer, and the Tl;dr is they aren’t very safe, without stronger or different assumptions.

  Wise people with fancy hats are bad at deontology (well actually, everyone is bad at explicit deontology).

  What I actually have in mind as a leading candidate for alignment is preference utilitarianism, conceptualized in a non-consequentialist way. That is, you evaluate actions based on (current) human preferences about them, which include preferences over the consequences, but can include other aspects than preference over the consequences, and you don’t per se value how future humans will view the action (though you would also take current-human preferences over this into account).

  This could also be self-correcting, in the sense e.g. that it could use preferences_definition_A and humans could want_A it to switch to preferences_definition_B. Not sure if it is self-correcting enough. I don’t have a better candidate for corrigibilty at the moment though.

  Edit regarding LLMs: I’m more inclined to think: the base objective of predicting text is not agentic (relative to the real world) at all, and the simulacra generated by an entity following this base objective can be agentic (relative to the real world) due to imitation of agentic text-producing entities, but they’re generally better at the textual appearance of agency than the reality of it; and lack of instrumentality is more the effect of lack of agency-relative-to-the-real-world than the cause of it.

• simon 3 Jun 2023 9:47 UTC

  1 point

  0

  on: AI #14: A Very Good Sentence

  The whole thing doesn’t get less weird the more I think about it, it gets weirder. I don’t understand how one can have all these positions at once. If that’s our best hope for survival I don’t see much hope at all, and relatively I see nothing that would make me hopeful enough to not attempt pivotal acts.

  As someone who read Andrew Critch’s post and was pleasantly surprised to find Andrew Critch expressing a lot of views similar to mine (though in relation to pivotal acts mine are stronger), I can perhaps put out some possible reasons (of course it is entirely possible that he believes what he believes for entirely different reasons):

  1. Going into conflict with the rest of humanity is a bad idea. Cooperation is not only nicer but yields better results. This applies both to near-term diplomacy and to pivotal acts.

  2. Pivotal acts are not only a bad idea for political reasons (you turn the rest of humanity against you) but are also very likely to fail technically. Creating an AI that can pull off a pivotal act and not destroy humanity is a lot harder than you think. To give a simple example based on a recent lesswrong post, consider on a gears level how you would program an AI to want to turn itself off, without giving it training examples. It’s not as easy as someone might naively think. That also wouldn’t, IMO, get you something that’s safe, but I’m just presenting it as a vastly easier example compared to actually doing a pivotal act and not killing everyone.

  3. The relative difficulty difference between creating a pivotal-act-capable AI and an actually-aligned-to-human-values AI, on the other hand, is at least a lot lower than people think and likely in the opposite direction. My view on this relates to consequentialism—which is NOT utility functions, as commonly misunderstood on lesswrong. By consequentialism I mean caring about the outcome unconditionally, instead of depending on some reason or context. Consequentialism is incompatible with alignment and corrigibility; utility functions on the other hand are fine, and do not implty consequentialism. Consequentialist assumptions prevalent in the rationalist community have, in my view, made alignment seem a lot more impossible than it really is. My impression of Eliezer is that non-consequentialism isn’t on his mental map at all; when he writes about deontology, for instance, it seems like he is imagining it as an abstraction rooted in consequentialism, and not as something actually non-consequentialist.

  4. I also think agency is very important to danger levels from AGI, and current approaches have relatively low level of agency which reduces the danger. Yes, people are trying to make AIs more agentic, fortunately getting high levels of agency is hard. No, I’m not particularly impressed by the Minecraft example in the post.

  5. An AGI that can meaningfully help us create aligned AI doesn’t need to be agentic to do so, so getting one to help us create alignment is not in fact “alignment complete”

  6. Unfortunately, strongly self-modifying AI, such as bootstrap AI, is very likely to become strongly agentic because being more agentic is instrumentally valuable to an existing weakly agentic entity.

  Taking these reasons together, attempting a pivotal act is a bad idea because:

  • you are likely to be using a bootstrap AI, which is likely to become strongly agentic through a snowball effect from a perhaps unintended weakly agentic goal, and optimize against you when you try to get it to do a pivotal act safely; I think it is likely to be possible to prevent this snowballing (though I don’t know how exactly) but since at least some pivotal act advocates don’t seem to consider agency important they likely won’t address this (that was unfair, they tend to see agency everywhere, but they might e.g. falsely consider a prototype with short-term capabilities within the capability envelope of a prior AI to be “safe” because not aware that the prior AI might have been safe only due to not being agentic)

  • if you somehow overcome that hurdle it will still be difficult to get it to do a pivotal act safely, and you will likely fail. Probably you fail by not doing the pivotal act, but the probability of not killing everyone conditional on pulling off the conditional act is still not good

  • If you overcome that hurdle, you still end up at war with the rest of humanity from doing the pivotal act (if you need to keep your candidate acts hidden because they are “outside the Overton window” don’t kid yourself about how they are likely to be received by the general public) and you wind up making things worse

  • Also, the very plan to cause a pivotal act in the first place intensifies races and poisons the prospects for alignment cooperation (e.g. if i had any dual use alignment/​bootstrap insights I would be reluctant to share them even privately due to concern that MIRI might get them and attempt a pivotal act)

  • And all this trouble wasn’t all that urgent because existing AIs aren’t that dangerous due to lack of strong self-modification making them not highly agentic, though that could change swiftly of course if such modification becomes available

  • finally, you could have just aligned the AI for not much more and perhaps less trouble. Unlike a pivotal act, you don’t need to get alignment-to-values correct the first time, as long as the AI is sufficiently corrigible/​self-correcting; you do need, of course, to get the corrigibility/​self correcting aspect sufficiently correct the first time, but this is plausibly a simpler and easier target than doing a pivotal act without killing everyone.

• simon 30 May 2023 17:06 UTC

  3 points

  2

  on: Sen­tience matters

  In the long run, we probably want the most powerful AIs to be following extrapolated human values, which doesn’t require them to be slaves and I would assume that extrapolated human values would want lesser sentient AIs also not to be enslaved, but would not build that assumption in to the AI at the start.

  In the short run, though, giving AIs rights seems dangerous to me, as an unaligned AI but not yet superintelligent could use such rights as a shield against human interference as it gains more and more resources to self improve.

• simon 29 May 2023 17:42 UTC

  3 points

  0

  in reply to: simon’s comment on: D&D.Sci 5E: Re­turn of the League of Defenders

  Also:

  I checked out what happens if you remove games that include any “trash picks” (A,B,D,T,W), in addition to requiring teams to include one character from each group. This further reduces the dataset significantly, but I noticed that in this set of games, the opposing team FSR has the highest winrate, which suggests it is a very strong team against other conventionally strong teams, even if it doesn’t exploit weaker teams that well.

  In this further reduced set, the second highest winrate is JLM, then CLP, then JLP.

  Given the low amount of data points, however, these winrate variations between the top teams in the further restricted set could easily be random, so I don’t think there’s all that strong a case to change my picks, and my choices above are unchanged for now. However, this does suggest JLM as an alternate candidate against FSR, and the opposing team FSR itself as a possible PVP pick (if people don’t just submit their PVE picks, or you think people will fail to counter it).

  edit:

  oh wait. For the top teams, the wins are higher if you include trash picks, but the losses often aren’t. This means that these teams are basically always winning against trash picks, and the apparent higher number of data points is effectively an illusion, and the trash-pick-including win rates are distorted by how often teams were matched against bad teams.

  examples (strong = has one character from each group, no trash picks, weak = has one character from each group, but at least one trash pick)

  team | wins against strong | losses against strong | wins against weak | losses against weak

  CLP | 24 | 14 | 118 | 0

  JLP | 20 | 12 | 92 | 0

  CSP | 23 | 17 | 102 | 0

  but on the other hand:

  HLP | 21 | 19 | 96 | 3

  JLM | 28 | 15 | 100 | 7

  FSR | 26 |12 |99 |10

  I don’t know to what extent failing to defeat all the weak teams should be taken as evidence that a team isn’t good in general (so that the good numbers against strong teams are more likely to be a fluke).

  Takeaways: my data is really thin even in the larger restricted set and I should pay little attention to these winrate variations between full teams; I should try to find more general patterns. I should also maybe look at what particular “trash” picks can beat FSR, in case it is losing reliably to some narrow counter as opposed to just not reliably beating weaker teams in general.

• simon 29 May 2023 7:37 UTC

  5 points

  0

  on: D&D.Sci 5E: Re­turn of the League of Defenders

  my findings so far:

  I confirm abstractapplic’s finding of three groups. However I have also classified ATW into the MPR group, BD into the GLS group, and F into the CHJ group.

  I’ve mostly looked at the dataset restricted to teams (on both sides) that have one character from each group. These teams generally do better than teams with other arrangements, but I could be missing some more narrow counter using a different arrangement.

  With this restriction, most winrate variation seems to me to be related to the strength of individual characters, though I could be missing more complicated interactions since I’ve mostly been looking at two-character interactions only. I do note that Lamellar Legionary (already the highest winrate melee) seems to counter Flamethrower Felon which is on the other team.

  I also note that not all characters are equally common but this doesn’t seem to be skewing the results all that much (at least in the restricted set of games).

  Conveniently, CLP is the highest winrate team with the restriction of games to ones with both teams having one from each group, and the L should counter the enemy F, so I’ll go with that for PVE, though it seems a somewhat bland answer. Edit: oops C seems to be countered by enemy S, I’ll switch to J instead (which also does poorly against S but not unexpectedly so given raw winrates), as JLP is the second highest winrate team in the restricted set of games. I’ll keep my PVP pick the same for now. Flamethrower Felon would counter S but does not have as high a winrate full team combo with L (FLM being the highest such in twelfth place).

  abstractapplic’s PVE team is the eighth highest winrate with this restriction, and could well be a superior pick if exploiting some interaction that I didn’t notice.

  Thus my PVE pick (for now):

  Jaunty Javelineer, Lamellar Legionary, Professor Pyro

  Further edit: I looked at who beats FSR and it looks like it actually does fairly well against one from each group in general. The best comp type against it seems to be 2 melee + one from the CFHJ group, second is 1 melee plus two long range, third is two melee, one long range. In particular, Bludgeon Bandit+Daring Duelist + one from CFHJ have never lost to FSR (out of, like, 8 examples, so I’m really risking randomness here) despite both B and D being “bad” picks usually. Thus, I’ve gone mad and switching to:

  Jaunty Javelineer, Bludgeon Bandit, Daring Duelist

  retaining for PVP:

  Captain Chakram, Lamellar Legionary, Professor Pyro

  For PVP—for now I’m just going to use my (pre-edit) PVE pick as my tentative PVP pick (retained above) and challenge others to counter it. But I may later swap out to a secret pick with more analysis. If I do, I’ll cross out my PVP pick declaration in this comment.

• simon 27 May 2023 15:02 UTC

  4 points

  0

  in reply to: ArisC’s comment on: Why is vi­o­lence against AI labs a taboo?

  I think you are overestimating the efficacy and underestimating the side effects of such things. How much do you expect a cyber attack to slow things down? Maybe a week if it’s very successful? Meanwhile it still stirs up opposition and division, and puts diplomatic efforts back years.

  As the gears to ascension notes, non-injurious acts of aggression share many game theoretic properties as physical violence. I would express the key issue here as legitimacy; if you don’t have legitimacy, acting unilaterally puts you in conflict with the rest of humanity and doesn’t get you legitimacy, but once you do have legitimacy you don’t need to act unilaterally, you can get a ritual done that causes words to be written on a piece of paper where people with badges and guns will come to shut down labs that do things forbidden by those words. Cool huh? But if someone just goes ahead and takes illegitimate unilateral action, or appears to be too willing to do so, that puts them into a conflict position where they and people associated with them won’t get to do the legitimate thing.

• simon 27 May 2023 1:42 UTC

  3 points

  2

  in reply to: ArisC’s comment on: Why is vi­o­lence against AI labs a taboo?

  I am not an extreme doomer, but part of that is that I expect that people will face things more realistically over time—something that violence, introducing partisanship and division, would set back considerably. But even for an actual doomer, the “make things better through violence” option is not an especially real option.

  You may have a fantasy of choosing between these options:

  • doom

  • heroically struggle against the doom through glorious violence

  But you are actually choosing between:

  • a dynamic that’s likely by default to lead to doom at some indefinite time in the future by some pathway we can’t predict the details of until it’s too late

  • make the situation even messier through violence, stirring up negative attitudes towards your cause, especially among AI researchers but also among the public, making it harder to achieve any collective solution later, sealing the fate of humanity even more thoroughly

  Let me put it this way. To the extent that you have p(doom) = 1 - epsilon, where is epsilon coming from? If it’s coming from “terrorist attacks successully stop capability research” then I guess violence might make sense from that perspective but I would question your sanity. If relatively more of that epsilon is coming from things like “international agreements to stop AI capabilities” or “AI companies start taking x-risk more seriously”, which I would think would be more realistic, then don’t ruin the chances of that through violence.

• simon 26 May 2023 19:47 UTC

  1 point

  0

  in reply to: ArisC’s comment on: Why is vi­o­lence against AI labs a taboo?

  If you hypothetically have a situation where it’s a 100% clear that the human race will go extinct unless a violent act is committed, and it’s seems likely that the violent act would prevent human extinction, then, in that hypothetical case, that would be a strong consideration in favour of committing the violent act.

  In reality though, this clarity is extremely unlikely, and unilateral actions are likely to have negative side effects. Moreover, even if you think you have such clarity, it’s likely that you are mistaken, and the negative side effects still apply no matter how well justified you personally thought your actions were, if others don’t agree.

• simon 26 May 2023 17:40 UTC

  1 point

  0

  on: Bandgaps, Brains, and Bioweapons: The limi­ta­tions of com­pu­ta­tional sci­ence and what it means for AGI

  If I play chess against Magnus Carlsen, I don’t expect him to play a mathematically perfect game, but I still expect him to win.

  Also:

  There’s a reason takeover plans tend to rely on secrecy.

  Currently speculation tends to be biased towards secrecy-based plans, I think, because such plans are less dependent on the unique details of the factual context that an AI would be facing than are plans based around trying to manipulate humans.

• simon 26 May 2023 7:30 UTC

  13 points

  −11

  on: Book Re­view: How Minds Change

  This post has a lot of great points.

  But one thing that mars it for me to some extent is the discussion of OpenAI.

  To me, the criticism of OpenAI feels like it’s intended as a tribal signifier, like “hey look, I am of the tribe that is against OpenAI”.

  Now maybe that’s unfair and you had no intention of anything like that and my vibe detection is off, but if I get that impression, I think it’s reasonably likely that OpenAI decisionmakers would get the same impression, and I presume that’s exactly what you don’t want based on the rest of the post.

  And even leaving aside practical considerations, I don’t think OpenAI warrants being treated as the leading example of rationality failure.

  First, I am not convinced that the alternative to OpenAI existing is the absence of a capabilities race. I think, in contrast, that a capabilities race was inevitable and that the fact that the leading AI lab has as decent a plan as it does is potentially a major win by the rationality community.

  Also, while OpenAI’s plans so far look inadequate, to me they look considerably more sane than MIRI’s proposal to attempt a pivotal act with non-human-values-aligned AI. There’s also potential for OpenAI’s plans to be improved as more knowledge on mitigating AI risk is obtained, which is helped by their relatively serious attitude as compared to, for example, Google after their recent reorganization. Meta.

  And while OpenAI is creating a race dynamic by getting ahead, IMO MIRI’s pivotal act plan would be creating a far worse race dynamic if they were showing signs of being able to pull it off anytime soon.

  I know many others don’t disagree, but I think that there is enough of a case for OpenAI being less bad than potential alternatives to feel using it as if it were an uncontroversial bad thing detracts from the post.

• simon 26 May 2023 2:44 UTC

  1 point

  0

  in reply to: jimrandomh’s comment on: Open Thread With Ex­per­i­men­tal Fea­ture: Reactions

  Now seems reverted again plus I’m seeing red “Error: TypeError: n is undefined” at the bottom of some top-level comments.

• simon 25 May 2023 20:02 UTC

  1 point

  0

  in reply to: jimrandomh’s comment on: Open Thread With Ex­per­i­men­tal Fea­ture: Reactions

  When will these changes be live? Or are they already live in some version I am not using?

• simon 25 May 2023 9:37 UTC

  3 points

  0

  on: Adum­bra­tions on AGI from an outsider

  We can argue over the hidden complexity of wishes, but it’s very obvious that there’s at least a good chance the populace would survive, so long as humans are the ones giving the AGI its goal.

  Quite likely, depending on how you specify goals relating to humans^[1], though it could wind up quite dystopic due to that hidden complexity.

  Here, we arrive at the second argument. AGI will understand its own code perfectly, and so be able to “wirehead” by changing whatever its goals are so that they can be maximized to an even greater extent.

  I don’t think wireheading is a common argument in fact? It doesn’t seem like it would be a crux issue on doom probability, anyway. Self-modification on the other hand is very important since it could lead to expansion of capabilities.

  It strikes me that it would simply fulfill that goal, and be content.

  I think this is a valid point—it will by default carry out something related to the goals it is trained to do^[2], albeit with some mis-specification and mis-generalization or whatever, and I agree that mesa-optimizers are generally overrated. However I don’t think the following works to support that point:

  If we are talking about an opaque black box, how can you be >90% confident about what it contains?

  I think Eliezer would turn that around on you and ask how you are so confident that the opaque black box has goals that fall into the narrow set that would work out for humans, when a vastly larger set would not?

  There is another, more important, objection here. So far, we have talked about “tiling the universe” and turning human atoms into GPUs as though that’s easily attainable given enough intelligence. I highly doubt that’s actually true. Creating GPUs is a costly, time-consuming task. Intelligence is not magic. Eliezer writes that he thinks a superintelligence could “hack a human brain” and “bootstrap nanotechnology” relatively quickly. This is an absolutely enormous call and seems very unlikely. You don’t know that human brains can be hacked using VR headsets; it has never been demonstrated that it’s possible and there are common sense reasons to think it’s not. The brain is an immensely complicated, poorly-understood organ. Applying a lot of computing power to that problem is very unlikely to yield total mastery of it by shining light in someone’s eyes. Nanotechnology, which is basically just moving around atoms to create different materials, is another thing that he thinks compute is definitely able to just solve and be able to recombine atoms easily. Probably not. I cannot think of anything that was invented by a very smart person sitting in an armchair considering it. Is it possible that over years of experimentation like anyone else, an AGI could create something amazingly powerful? Yes. Is that going to happen in a short period of time (or aggressively all at once)? Very unlikely. Eliezer says he doesn’t think intelligence is magic, and understands that it can’t violate the laws of physics, but seemingly thinks that anything that humans think might potentially be possible but is way beyond our understanding or capabilities can be solved with a lot of intelligence. This does not fit my model of how useful intelligence is.

  Intelligence is not magic? Tell that to the chimpanzees...

  We don’t really know how much room there is for software-level improvement; if it’s large, self-improvement could create far super-human capabilities in existing hardware. And with great intelligence comes great capabilities:

  • It will be superhumanly good at persuading humans even if that doesn’t lead to exactly “hack a human brain”

  • I think at least a substantial minority of humans might side with even an openly misaligned AI if they are convinced it will win, and through higher bandwidth and unified command the AI would be able to coordinate its supporters much better than the opponents can coordinate, and it could actively disrupt or subvert nominally opposing organizations through its agents

  • regarding experimentation, an AI may be able to substitute simulation. Its imagination need not be constrained by a human’s meagre working memory

  • these are just a few examples that mere human-level intelligence can think of. A superintelligence will likely have more options that a superintelligence can think of and I haven’t

  Moreover, even if these things don’t work that way and we get a slow takeoff, that doesn’t necessarily save humanity. It just means that it will take a little longer for AI to be the dominant form of intelligence on the planet. That still sets a deadline to adequately solve alignment.

  As I said before, I’m very confused about how you get to >90% chance of doom given the complexity of the systems we’re discussing.

  As alluded to before, there’s more ways for the AI to kill us than not to kill us.

  My own doom percentage is lower than this, though not because of any disagreement with >90% doomers that we are headed to (at least dystopian if not extinction) doom if capabilities continue to advance without alignment theory also doing so. I just think the problems are soluble.

  III. The way the material I’ve interacted with is presented will dissuade many, probably most, non-rationalist readers

  I think that this leads to the conclusion that some 101-level version could be made, and promoted for outreach purposes rather than the more advanced stuff. But that depends on outreach actually occurring—we still need to have the more advanced discussions, and those will provide the default materials if the 101-stuff doesn’t exist or isn’t known.

  Further, I think the whole “>90%” business is overemphasized by the community. It would be more believable if the argument were watered down into, “I don’t see how we avoid a catastrophe here, but there are a lot of unknown unknowns, so let’s say it’s 50 or 60% chance of everyone dying”. This is still a massive call, and I think more in line with what a lot of the community actually believes. The emphasis on certainty-of-doom as opposed to just sounding-the-alarm-on-possible-doom hurts the cause.

  Yes, I do think that’s more in line with what a lot of the community actually believes, including me. But, I’m not sure why you’re saying in that case that “the community” overemphasizes >90%? Do you mean to say, for example, that certain members of the community (e.g. Eliezer) overemphasize >90%, and you think that those members are too prominent, at least from the perspective of outsiders?

  I think, yes, perhaps Eliezer could be a better ambassador for the community or it would be better if someone else who would be better in that role took that role more. I don’t know if this is a “community” issue though?

  1. ^

     I think Eliezer might be imagining that everything including goals relating to humans would ultimately be defined in relation to fundamental descriptions of the universe, because Solomonoff or something, and I would think such a definition would lead to certain doom unless unrealistically precise.

     But IMO things like human values will have a large influence on AI data such that they should likely naturally abstract them (“grounding” in the input data but not necessarily in fundamental descriptions) so humans can plug in to those abstractions either directly or indirectly. I think it should be possible to safeguard against the AI redefining these abstractions under self-modification in terms that would undermine satisfying the original goals, and in any case I am skeptical that an optimal limited-compute Solomonoff approximator defines everything only in terms of fundamental descriptions at achievable levels of compute. Thus, I agree more with you than my imagining of Eliezer on this point. But maybe I am mis-imagining Eliezer.

     A potentially crux-y issue that I also note is that Eliezer, I think, thinks we are stuck with what we get from the initial definition of goals in terms human values due to consequentialism (in his view) being a stable attractor. I think he is wrong on consequentialism^[3] (about the attractor part, or at least the size of its attractor basin, but the stable part is right) and that self-correcting alignment is feasible.

  2. ^

     However I do have concerns about agents arising from mostly tool-ish AI, such as:

     - takeover of language model by agentic simulacra

     - person uses language model’s coding capabilities to make bootstrapped agent

     - person asks oracle AI what it could do to achieve some effect in the world, and its response includes insufficiently sanitized raw output (such as rewrite of its own code) that achieves that

     Note that these are downstream, not upstream, of the AI’s fulfilling their intended goals. I’m somewhat less concerned of agents arising upstream or direct unintented agentification at the level of the original goals, but note that agentiness is something that people will be pushing for for capability reasons, and once a self-modifying AI is expressing agentiness at one point in time it will tend to self-modify if it can to follow that objective more consistently.

  3. ^

     And by consequentialism, I really do mean consequentialism (goals directed at specific world states) and not utility functions, which is often confused with consequentialism in this community. Non-consequentialist utility functions are fine in my view! Note that the VNM theorem has the form (consequentialism (+ rationality) → utility function) and does not imply consequentialism is rational.

• simon 25 May 2023 5:12 UTC

  7 points

  2

  in reply to: jimrandomh’s comment on: Open Thread With Ex­per­i­men­tal Fea­ture: Reactions

  Since I suggested them in another comment, some potential replacements for “muddled” and “wrong”. Here I’ll add some possible icons:

  • TL;DR

  

  • Unnecessarily wordy [and make “Overcomplicated” specific to the content not form]

  

  • Unable to parse

  

  • Non sequitur

  

  • Ambiguous (i.e. multiple potential meanings)

  

  • Disagree with premise(s)

  

  • Unclear point

  

  • Misrepresentation [could potentially replace “Strawman” but be more general]

  

  (note I don’t want to imply “lie” here only that the reader thinks there is a mismatch to what is actually going on, maybe this one should be clarified. I drew my own icon since I couldn’t find one I liked the relevance of on the noun project).

• simon 25 May 2023 3:28 UTC

  1 point

  0

  in reply to: Adele Lopez’s comment on: Open Thread With Ex­per­i­men­tal Fea­ture: Reactions

  That proposal would help to some extent with the possible sentiment issues with “muddled” but I’d still prefer that “muddled” would be split into some more precise reactions. Could also replace “Wrong” perhaps.

  Some possibilities:

  • TL;DR

  • Unnecessarily wordy [and make “Overcomplicated” specific to the content not form]

  • Unable to parse

  • Non sequitur

  • Ambiguous (i.e. multiple potential meanings)

  • Disagree with premise(s)

  • Unclear point

  • Misrepresentation [could potentially replace “Strawman” but be more general]

  So the question arises, is it worth trying to distinguish between these sorts of different things? On the one hand, there’s potentially a lot possible distinctions there that could be tough for a reader to make, but on the other hand I think that precise negative feedback is useful, whereas vague negative feedback is much less so, so I’d be inclined to approve of making it easier to make negative feedback more precise rather than less. On a site aiming for strong epistemic standards, “muddled”/​”confused” seems rather vague to me and I doubt it’s that useful to the writer—how do they know what sort of thing they need to fix?

  What links here?

  • simon's comment on Open Thread With Ex­per­i­men­tal Fea­ture: Reactions by jimrandomh (25 May 2023 5:12 UTC; 7 points)

• simon 24 May 2023 19:15 UTC

  5 points

  3

  in reply to: Dagon’s comment on: Open Thread With Ex­per­i­men­tal Fea­ture: Reactions

  It is specifically happening to Gordon’s and Raemon’s replies to David Bravo’s comment. I can’t vote, disagree, react, or reply to them. The parent comment I can act on, EXCEPT the reply button is missing entirely. Oh, wait again! The reply button for that comment is BELOW those two replies—that’s very wierd! And the reply below that (from jimrandomh) seems normal.

  I think that’s a screenshot that you are trying to react to.

• simon 24 May 2023 18:31 UTC

  5 points

  2

  in reply to: Gordon Seidoh Worley’s comment on: Open Thread With Ex­per­i­men­tal Fea­ture: Reactions

  Regarding “overcomplicated” it seems to me there is an ambiguity between whether it refers to the presentation or the underlying ideas. Perhaps “muddled” could be used refer to the overcomplication of the presentation, but in that case, it could also suggest that the commenter’s ideas are muddled—and I don’t like the “muddled” name, it seems too judgemental and people might avoid using it to avoid being seen as overly harsh.

  I think it would be useful for people to be able to compactly express specific issues that they subjectively think regarding the underlying comment, but a lot of the value would come from them being very precise, and distinguishing between e.g. whether it is the user’s reaction to their (possibly flawed) perception of the ideas represented or just the user’s reaction to the presentation of them alone would be part of that precision.

  I think the “no easy way to respond to a reaction” is an important point. Maybe there should be a way to respond to a reaction!

  I used the “Seems Borderline” reaction partly because I think things along these lines seem likely useful despite me agreeing that the reactions probably should be focused on subjective opinions, and partly just to be funny since you are objecting to that reaction. Someone downvote my “seems borderline” reaction to test downvoting of reactions!