rpglover64

• rpglover64 19 Mar 2023 14:13 UTC

  1 point

  0

  on: Nat­u­ral Ab­strac­tions: Key claims, The­o­rems, and Critiques

  I had an insight about the implications of NAH which I believe is useful to communicate if true and useful to dispel if false; I don’t think it has been explicitly mentioned before.

  One of Eliezer’s examples is “The AI must be able to make a cellularly identical but not molecularly identical duplicate of a strawberry.” One of the difficulties is explaining to the AI what that means. This is a problem with communicating across different ontologies—the AI sees the world completely differently than we do. If NAH in a strong sense is true, then this problem goes away on its own as capabilities increase; that is, AGI will understand us when we communicate something that has a coherent natural interpretation, even without extra effort on our part to translate it to the AGI version of machine code.

  Does that seem right?

• rpglover64 18 Mar 2023 18:37 UTC

  5 points

  2

  on: “Care­fully Boot­strapped Align­ment” is or­ga­ni­za­tion­ally hard

  (Why “Top 3” instead of “literally the top priority?” Well, I do think a successful AGI lab also needs have top-quality researchers, and other forms of operational excellence beyond the ones this post focuses on. You only get one top priority, )

  I think the situation is more dire than this post suggests, mostly because “You only get one top priority.” If your top priority is anything other than this kind of organizational adequacy, it will take precedence too often; if your top priority is organizational adequacy, you probably can’t get off the ground.

  The best distillation of my understanding regarding why “second priority” is basically the same as “not a priority at all” is this twitter thread by Dan Luu.

  The fear was that if they said that they needed to ship fast and improve reliability, reliability would be used as an excuse to not ship quickly and needing to ship quickly would be used as an excuse for poor reliability and they’d achieve none of their goals.

  What links here?

  • Dan Luu on “You can only com­mu­ni­cate one top pri­or­ity” by Raemon (18 Mar 2023 18:55 UTC; 114 points)

• rpglover64 4 Mar 2023 16:03 UTC

  1 point

  0

  in reply to: abramdemski’s comment on: Teleose­man­tics!

  I just read an article that reminded me of this post. The relevant section starts with “Bender and Manning’s biggest disagreement is over how meaning is created”. Bender’s position seems to have some similarities with the thesis you present here, especially when viewed in contrast to what Manning claims is the currently more popular position that meaning can arise purely from distributional properties of language.

  This got me wondering: if Bender is correct, then there is a fundamental limitation in how well (pure) language models can understand the world; are there ways to test this hypothesis, and what does it mean for alignment?

  Thoughts?

• rpglover64 24 Feb 2023 2:09 UTC

  3 points

  0

  on: Teleose­man­tics!

  Interesting. I’m reminded of this definition of “beauty”.

• rpglover64 20 Feb 2023 14:57 UTC

  22 points

  16

  on: The idea that ChatGPT is sim­ply “pre­dict­ing” the next word is, at best, misleading

  One minor objection I have to the contents of this post is the conflation of models that are fine-tuned (like ChatGPT) and models that are purely self-supervised (like early GPT3); the former has no pretenses of doing only next token prediction.

• rpglover64 17 Feb 2023 21:59 UTC

  2 points

  0

  in reply to: Ben’s comment on: Are LLMs suffi­cient for AI take­off?

  But they seem like they are only doing part of the “intelligence thing”.

  I want to be careful here; there is some evidence to suggest that they are doing (or at least capable of doing) a huge portion of the “intelligence thing”, including planning, induction, and search, and even more if you include minor external capabilities like storage.

  I don’t know if anyone else has spoken about this, but since thinking about LLMs a little I am starting to feel like their something analagoss to a small LLM (SLM?) embedded somewhere as a component in humans

  I know that the phenomenon has been studied for reading and listening (I personally get a kick out of garden-path sentences); the relevant fields are “natural language processing” and “computational linguistics”. I don’t know know of any work specifically that addressed it in the “speaking” setting.

  if we want to build something “human level” then it stands to reason that it would end up with specialized components for the same sorts of things humans have specialized components for.

  Soft disagree. We’re actively building the specialized components because that’s what we want, not because that’s particularly useful for AGI.

[Question] Are LLMs suffi­cient for AI take­off?

• rpglover64 17 Feb 2023 15:28 UTC

  3 points

  2

  in reply to: Lost Futures’s comment on: Open & Wel­come Thread — Fe­bru­ary 2023

  Some thoughts:

  • Those who expect fast takeoffs would see the sub-human phase as a blip on the radar on the way to super-human

  • The model you describe is presumably a specialist model (if it were generalist and capable of super-human biology, it would plausibly count as super-human; if it were not capable of super-human biology, it would not be very useful for the purpose you describe). In this case, the source of the risk is better thought of as the actors operating the model and the weapons produced; the AI is just a tool

  • Super-human AI is a particularly salient risk because unlike others, there is reason to expect it to be unintentional; most people don’t want to destroy the world

  • The actions for how to reduce xrisk from sub-human AI and from super-human AI are likely to be very different, with the former being mostly focused on the uses of the AI and the latter being on solving relatively novel technical and social problems

• rpglover64 14 Feb 2023 13:58 UTC

  3 points

  3

  in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Cyborgism

  I think “sufficiently” is doing a lot of work here. For example, are we talking about >99% chance that it kills <1% of humanity, or >50% chance that it kills <50% of humanity?

  I also don’t think “something in the middle” is the right characterization; I think “something else” it more accurate. I think that the failure you’re pointing at will look less like a power struggle or akrasia and more like an emergent goal structure that wasn’t really present in either part.

  I also think that “cyborg alignment” is in many ways a much more tractable problem than “AI alignment” (and in some ways even less tractable, because of pesky human psychology):

  • It’s a much more gradual problem; a misaligned cyborg (with no agentic AI components) is not directly capable of FOOM (Amdhal’s law was mentioned elsewhere in the comments as a limit on usefulness of cyborgism, but it’s also a limit on damage)

  • It has been studied longer and has existed longer; all technologies have influenced human thought

  It also may be an important paradigm to study (even if we don’t actively create tools for it) because it’s already happening.

• rpglover64 12 Feb 2023 19:41 UTC

  1 point

  0

  AF

  in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Cyborgism

  Like, I may not want to become a cyborg if I stop being me, but that’s a separate concern from whether it’s bad for alignment (if the resulting cyborg is still aligned).

• rpglover64 11 Feb 2023 21:40 UTC

  2 points

  −1

  on: Cyborgism

  OpenAI’s focus with doing these kinds of augmentations is very much “fixing bugs” with how GPT behaves: Keep GPT on task, prevent GPT from making obvious mistakes, and stop GPT from producing controversial or objectionable content. Notice that these are all things that GPT is very poorly suited for, but humans find quite easy (when they want to). OpenAI is forced to do these things, because as a public facing company they have to avoid disastrous headlines like, for example: Racist AI writes manifesto denying holocaust.^[7]

  As alignment researchers, we don’t need to worry about any of that! The goal is to solve alignment, and as such we don’t have to be constrained like this in how we use language models. We don’t need to try to “align” language models by adding some RLHF, we need to use language models to enable us to actually solve alignment at its core, and as such we are free to explore a much wider space of possible strategies for using GPT to speed up our research.^[8]

  I think this section is wrong because it’s looking through the wrong frame. It’s true that cyborgs won’t have to care about PR disasters in their augments, but it feels like this section is missing the fact that there are actual issues behind the PR problem. Consider a model that will produce the statement “Men are doctors; women are nurses” (when not explicitly playing an appropriate role); this indicates something wrong with its epistemics, and I’d have some concern about using such a model.

  Put differently, a lot of the research on fixing algorithmic bias can be viewed as trying to repair known epistemic issues in the model (usually which are inherited from the training data); this seems actively desirable for the use case described here.

• rpglover64 11 Feb 2023 15:08 UTC

  LW: 6 AF: 3

  5

  AF

  in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Cyborgism

  I think that’s an important objection, but I see it applying almost entirely on a personal level. On the strategic level, I actually buy that this kind of augmentation (i.e. with in some sense passive AI) is not an alignment risk (any more than any technology is). My worry is the “dual use technology” section.

• rpglover64 10 Feb 2023 17:36 UTC

  3 points

  0

  in reply to: DragonGod’s comment on: On util­ity functions

  simple utility functions are fundamentally incapable of capturing the complexity/​subtleties/​nuance of human preferences.

  No objections there.

  that inexploitable agents exist that do not have preferences representable by simple/​unitary utility functions.

  Yep.

  Going further, I think utility functions are anti-natural to generally capable optimisers in the real world.

  I tentatively agree.

  That said

  The existence of a utility function is a sometimes useful simplifying assumption, in a way similar to how logical omniscience is (or should we be doing all math with logical inductors?), and naturally generalizes as a formalism to something like “the set of utility functions consistent with revealed preferences”.

  In the context of human rationality, I have found a local utility function perspective to be sometimes useful, especially as a probe into personal reasons; that is, if you say “this is my utility function” and then you notice “huh… my action does not reflect that”, this can prompt useful contemplation, some possible outcomes of which are:

  • You neglected a relevant term in the utility function, e.g. happiness of your immediate family

  • You neglected a relevant utility cost of the action you didn’t take, e.g. the aversiveness of being sweaty

  • You neglected a constraint, e.g. you cannot actually productively work for 12 hours a day, 6 days a week

  • The circumstances in which you acted are outside the region of validity of your approximation of the utility function, e.g. you don’t actually know how valuable having $100B would be to you

  • You made a mistake with your action

  Of course, a utility function framing is neither necessary nor sufficient for this kind of reflection, but for me, and I suspect some others, it is helpful.

• rpglover64 10 Feb 2023 17:15 UTC

  1 point

  0

  in reply to: Jozdien’s comment on: What’s ac­tu­ally go­ing on in the “mind” of the model when we fine-tune GPT-3 to In­struc­tGPT?

  The base models of GPT-3 already have the ability to “follow instructions”, it’s just veiled behind the more general interface. [...] you can see how it contains this capability somewhere.

  This is a good point that I forgot. My mental model of this is that since many training samples are Q&A, in these cases, learning to complete implies learning how to answer.

  InstructGPT also has the value of not needing the wrapper of “Q: [] A: []”, but that’s not really a qualitative difference.

  I want to push back a little bit on the claim that this is not a qualitative difference; it does imply a big difference in output for identical input, even if the transformation required to get similar output between the two models is simple.

  In other words, instruction following is not a new capability and the fine-tuning doesn’t really make any qualitative changes to the model. In fact, I think that you can get results [close to] this good if you prompt it really well (like, in the realm of soft prompts).

  TIL about soft prompts. That’s really cool, and I’m not surprised it works (it also feels a little related to my second proposed experiment). My intuition here (transferred from RNNs, but I think it should mostly apply to unidirectional transformers as well) is that a successful prompt puts the NN into the right “mental state” to generate the desired output: fine-tuning for e.g. instruction following mostly pushes to get the model into this state from the prompts given (as opposed to e.g. for HHH behavior, which also adjusts the outputs from induced states); soft prompts instead search for and learn a “cheat code” that puts the model into a state such that the prompt is interpreted correctly. Would you (broadly) agree with this?

• rpglover64 10 Feb 2023 13:57 UTC

  1 point

  0

  on: On util­ity functions

  Other people have given good answers to the main question, but I want to add just a little more context about self-modifying code.

  A bunch of MIRI’s early work explored the difficulties of the interaction of “rationality” (including utility functions induced by consistent preferences) with “self-modification” or “self-improvement”; a good example is this paper. They pointed out some major challenges that come up when an agent tries to reason about what future versions of itself will do; this is particularly important because one failure mode of AI alignment is to build an aligned AI that accidentally self-modifies into an unaligned AI (note that continuous learning is a restricted form of self-modification and suffers related problems). There are reasons to expect that powerful AI agents will be self-modifying (ideally self-improving), so this is an important question to have an answer to (relevant keywords include “stable values” and “value drift”).

  There’s also some thinking about self-modification in the human-rationality sphere; two things that come to mind are here and here. This is relevant because ways in which humans deviate from having (approximate, implicit) utility functions may be irrational, though the other responses point out limitations of this perspective.

• rpglover64 10 Feb 2023 13:08 UTC

  3 points

  0

  in reply to: DragonGod’s comment on: Open & Wel­come Thread—Jan­uary 2023

  Done: https://​​www.lesswrong.com/​​posts/​​eywpzHRgXTCCAi8yt/​​what-s-actually-going-on-in-the-mind-of-the-model-when-we

[Question] What’s ac­tu­ally go­ing on in the “mind” of the model when we fine-tune GPT-3 to In­struc­tGPT?

• rpglover64 10 Feb 2023 0:38 UTC

  3 points

  2

  on: Open & Wel­come Thread—Jan­uary 2023

  (I may promote this to a full question)

  Do we actually know what’s happening when you take an LLM trained on token prediction and fine-tune is via e.g. RLHF to get something like InstructGPT or ChatGPT? The more I think about the phenomenon, the more confused I feel.

  What links here?

  • What’s ac­tu­ally go­ing on in the “mind” of the model when we fine-tune GPT-3 to In­struc­tGPT? by rpglover64 (10 Feb 2023 7:57 UTC; 18 points)

• rpglover64 2 Feb 2023 23:24 UTC

  7 points

  5

  on: Fo­cus on the places where you feel shocked ev­ery­one’s drop­ping the ball

  English doesn’t have great words for me to describe what I mean here, but it’s something like: your visualization machinery says that it sees no obstacle to success, such that you anticipate either success or getting a very concrete lesson.

  One piece of advice/​strategy I’ve received that’s in this vein is “maximize return on failure”. So prefer to fail in ways that you learn a lot, and to fail quickly, cheaply, and conclusively, and produce positive externalities from failure. This is not so much a good search strategy but a good guiding principle and selection heuristic.

• rpglover64 31 Jan 2023 1:24 UTC

  LW: 10 AF: 6

  2

  AF

  on: Align­ment al­lows “non­ro­bust” de­ci­sion-in­fluences and doesn’t re­quire ro­bust grading

  This is a great article! It helps me understand shard theory better and value it more; in particular, it relates to something I’ve been thinking about where people seem to conflate utility-optimizing agents with policy-execuing agents, but the two have meaningfully different alignment characteristics, and shard theory seems to be deeply exploring the latter, which is 👍.

  That is to say, prior to “simulators” and “shard theory”, a lot of focus was on utility-maximizers—agents that do things like planning or search to maximize a utility function; but planning, although instrumentally useful, is not strictly necessary for many intelligent behaviors, so we are seeing more focus on e.g. agents that enact learned policies in RL that do not explicitly maximize reward in deployment but try to enact policies that did so in training.

  The answer is not to find a clever way to get a robust grader. The answer is to not need a robust grader

  💯

  From my perspective, this post convincingly argues that one route to alignment involves splitting the problem into two still-difficult sub-problems (but actually easier, unlike inner- and outer-alignment, as you’ve said elsewhere): identifying a good shard structure and training an AI with such a shard structure. One point is that the structure is inherently somewhat robust (and that therefore each individual shard need not be), making it a much larger target.

  I have two objections:

  • I don’t buy the implied “naturally-robust” claim. You’ve solved the optimizer’s curse, wireheading via self-generated adversarial inputs, etc., but the policy induced by the shard structure is still sensitive to the details; unless you’re hiding specific robust structures in your back pocket, I have no way of knowing that increasing the candy-shard’s value won’t cause a phase shift that substantially increases the perceived value of the “kill all humans, take their candy” action plan. I ultimately care about the agent’s “revealed preferences”, and I am not convinced that those are smooth relative to changes in the shards.

  • I don’t think that we can train a “value humans” shard that avoids problems with the edge cases of what that means. Maybe it learns that it should kill all humans and preserve their history; or maybe it learns that it should keep them alive and comatose; or maybe it has strong opinions one way or another on whether uploading is death; or maybe it respects autonomy too much to do anything (though that one would probably be decomissioned and replaced by one more dangerous). The problem is not adversarial inputs but genuine vagueness where precision matters. I think this boils down to me disagreeing with John Wentworth’s “natural abstraction hypothesis” (at least in some ways that matter)