porby

• porby 15 Oct 2022 0:17 UTC

  37 points

  15

  on: Coun­ter­ar­gu­ments to the ba­sic AI x-risk case

  Great post! I think this captures a lot of why I’m not ultradoomy (only, er, 45%-ish doomy, at the moment), especially A and B. I think it’s at least possible that our reality is on easymode, where muddling could conceivably put an AI into close enough territory to not trigger an oops.

  I’d be even less doomy if I agreed with the counterarguments in C. Unfortunately, I can’t shake the suspicion that superintelligence is the kind of ridiculously powerful lever that would magnify small oopses into the largest possible oopses.

  Hypothetically, if we took a clever human’s general capacity for problem solving, stripped it of limitations like getting bored or tired, got rid of its pesky intuitions around ethics, and sped it up by a factor of 1,000 times… I’d be very worried about what it would be able to do. Even without greater capacity for insight or an enhanced working memory, simply thinking really fast would be a broken superpower.

  Such an entity might not be able to recreate the technology of modern civilization starting from scratch (both in resources and knowledge) in the stone age within 30 years, primarily due to physical interaction requirements. But starting from anything like modern civilization? That would get weird fast.

  In other words, it seems like the intelligence range of humans- or even the range across animals and humans- is small compared to what is artificially possible even if we only consider speed. And it seems very likely at this point that a well-built artificial mind could have higher quality insights, too. MuZero certainly seems to within its domain. I don’t find much comfort in observable intelligence differences not always resulting in domination.

  What links here?

  • porby's comment on Why I think strong gen­eral AI is com­ing soon by porby (16 Oct 2022 19:42 UTC; 4 points)

• porby 24 Nov 2023 23:45 UTC

  29 points

  5

  on: Abil­ity to solve long-hori­zon tasks cor­re­lates with want­ing things in the be­hav­iorist sense

  I’m not sure if I fall into the bucket of people you’d consider this to be an answer to. I do think there’s something important in the region of LLMs that, by vibes if not explicit statements of contradiction, seems incompletely propagated in the agent-y discourse even though it fits fully within it. I think I at least have a set of intuitions that overlap heavily with some of the people you are trying to answer.

  In case it’s informative, here’s how I’d respond to this:

  Well, I claim that these are more-or-less the same fact. It’s no surprise that the AI falls down on various long-horizon tasks and that it doesn’t seem all that well-modeled as having “wants/​desires”; these are two sides of the same coin.

  Mostly agreed, with the capability-related asterisk.

  Because the way to achieve long-horizon targets in a large, unobserved, surprising world that keeps throwing wrenches into one’s plans, is probably to become a robust generalist wrench-remover that keeps stubbornly reorienting towards some particular target no matter what wrench reality throws into its plans.

  Agreed in the spirit that I think this was meant, but I’d rephrase this: a robust generalist wrench-remover that keeps stubbornly reorienting towards some particular target will tend to be better at reaching that target than a system that doesn’t.

  That’s subtly different from individual systems having convergent internal reasons for taking the same path. This distinction mostly disappears in some contexts, e.g. selection in evolution, but it is meaningful in others.

  If an AI causes some particular outcome across a wide array of starting setups and despite a wide variety of obstacles, then I’ll say it “wants” that outcome “in the behaviorist sense”.

  I think this frame is reasonable, and I use it.

  it’s a little hard to imagine that you don’t contain some reasonably strong optimization that strategically steers the world into particular states.

  Agreed.

  that the wanting-like behavior required to pursue a particular training target X, does not need to involve the AI wanting X in particular.

  Agreed.

  “AIs need to be robustly pursuing some targets to perform well on long-horizon tasks”, but it does not say that those targets have to be the ones that the AI was trained on (or asked for). Indeed, I think the actual behaviorist-goal is very unlikely to be the exact goal the programmers intended, rather than (e.g.) a tangled web of correlates.

  Agreed for a large subset of architectures. Any training involving the equivalent of extreme optimization for sparse/​distant reward in a high dimensional complex context seems to effectively guarantee this outcome.

  So, maybe don’t make those generalized wrench-removers just yet, until we do know how to load proper targets in there.

  Agreed, don’t make the runaway misaligned optimizer.

  I think there remains a disagreement hiding within that last point, though. I think the real update from LLMs is:

  1. We have a means of reaching extreme levels of capability without necessarily exhibiting preferences over external world states. You can elicit such preferences, but a random output sequence from the pretrained version of GPT-N (assuming the requisite architectural similarities) has no realistic chance of being a strong optimizer with respect to world states. The model itself remains a strong optimizer, just for something that doesn’t route through the world.

  2. It’s remarkably easy to elicit this form of extreme capability to guide itself. This isn’t some incidental detail; it arises from the core process that the model learned to implement.

  3. That core process is learned reliably because the training process that yielded it leaves no room for anything else. It’s not a sparse/​distant reward target; it is a profoundly constraining and informative target.

  In other words, a big part of the update for me was in having a real foothold on loading the full complexity of “proper targets.”

  I don’t think what we have so far constitutes a perfect and complete solution, the nice properties could be broken, paradigms could shift and blow up the golden path, it doesn’t rule out doom, and so on, but diving deeply into this has made many convergent-doom paths appear dramatically less likely to Late2023!porby compared to Mid2022!porby.

  What links here?

  • porby's comment on Why I think strong gen­eral AI is com­ing soon by porby (16 Dec 2023 23:21 UTC; 28 points)

• porby 16 Dec 2023 23:21 UTC

  28 points

  2

  on: Why I think strong gen­eral AI is com­ing soon

  It’s been over a year since the original post and 7 months since the openphil revision.

  A top level summary:

  1. My estimates for timelines are pretty much the same as they were.

  2. My P(doom) has gone down overall (to about 30%), and the nature of the doom has shifted (misuse, broadly construed, dominates).

  And, while I don’t think this is the most surprising outcome nor the most critical detail, it’s probably worth pointing out some context. From NVIDIA:

  

  In two quarters, from Q1 FY24 to Q3 FY24, datacenter revenues went from $4.28B to $14.51B.

  From the post:

  In 3 years, if NVIDIA’s production increases another 5x …

  Revenue isn’t a perfect proxy for shipped compute, but I think it’s safe to say we’ve entered a period of extreme interest in compute acquisition. “5x” in 3 years seems conservative.^[1] I doubt the B100 is going to slow this curve down, and competitors aren’t idle: AMD’s MI300X is within striking distance, and even Intel’s Gaudi 2 has promising results.

  Chip manufacturing remains a bottleneck, but it’s a bottleneck that’s widening as fast as it can to catch up to absurd demand. It may still be bottlenecked in 5 years, but not at the same level of production.

  On the difficulty of intelligence

  I’m torn about the “too much intelligence within bounds” stuff. On one hand, I think it points towards the most important batch of insights in the post, but on the other hand, it ends with an unsatisfying “there’s more important stuff here! I can’t talk about it but trust me bro!”

  I’m not sure what to do about this. The best arguments and evidence are things that fall into the bucket of “probably don’t talk about this in public out of an abundance of caution.” It’s not one weird trick to explode the world, but it’s not completely benign either.

  Continued research and private conversations haven’t made me less concerned. I do know there are some other people who are worried about similar things, but it’s unclear how widely understood it is, or whether someone has a strong argument against it that I don’t know about.

  So, while unsatisfying, I’d still assert that there are highly accessible paths to broadly superhuman capability on short timescales. Little of my forecast’s variance arises from uncertainty on this point; it’s mostly a question of when certain things are invented, adopted, and then deployed at sufficient scale. Sequential human effort is a big chunk; there are video games that took less time to build than the gap between this post’s original publication date and its median estimate of 2030.

  On doom

  When originally writing this, my model of how capabilities would develop was far less defined, and my doom-model was necessarily more generic.

  A brief summary would be:

  1. We have a means of reaching extreme levels of capability without necessarily exhibiting preferences over external world states. You can elicit such preferences, but a random output sequence from the pretrained version of GPT-N (assuming the requisite architectural similarities) has no realistic chance of being a strong optimizer with respect to world states. The model itself remains a strong optimizer, just for something that doesn’t route through the world.

  2. It’s remarkably easy to elicit this form of extreme capability to guide itself. This isn’t some incidental detail; it arises from the core process that the model learned to implement.

  3. That core process is learned reliably because the training process that yielded it leaves no room for anything else. It’s not a sparse/​distant reward target; it is a profoundly constraining and informative target.

  I’ve written more on the nice properties of some of these architectures elsewhere. I’m in the process of writing up a complementary post on why I think these properties (and using them properly) are an attractor in capabilities, and further, why I think some of the x-riskiest forms of optimization process are actively repulsive for capabilities. This requires some justification, but alas, the post will have to wait some number of weeks in the queue behind a research project.

  The source of the doom-update is the correction of some hidden assumptions in my doom model. My original model was downstream of agent foundations-y models, but naive. It followed a process: set up a framework, make internally coherent arguments within that framework, observe highly concerning results, then neglect to notice where the framework didn’t apply.

  Specifically, some of the arguments feeding into my doom model were covertly replacing instances of optimizers with hypercomputer-based optimizers^[2], because hey, once you’ve got an optimizer and you don’t know any bounds on it, you probably shouldn’t assume it’ll just turn out convenient for you, and hypercomputer-optimizers are the least convenient.

  For example, this part:

  Is that enough to start deeply modeling internal agents and other phenomena concerning for safety?

  And this part:

  AGI probably isn’t going to suffer from these issues as much. Building an oracle is probably still worth it to a company even if it takes 10 seconds for it to respond, and it’s still worth it if you have to double check its answers (up until oops dead, anyway).

  With no justification, I imported deceptive mesaoptimizers and other “unbound” threats. Under the earlier model, this seemed natural.

  I now think there are bounds on pretty much all relevant optimizing processes up and down the stack from the structure of learned mesaoptimizers to the whole capability-seeking industry. Those bounds necessarily chop off large chunks of optimizer-derived doom; many outcomes that previously seemed convergent to me now seem extremely hard to access.

  As a result, “technical safety failure causes existential catastrophe” dropped in probability by around 75-90%, down to something like 5%-ish.^[3]

  I’m still not sure how to navigate a world with lots of extremely strong AIs. As capability increases, outcome variance increases. With no mitigations, more and more organizations (or, eventually, individuals) will have access to destabilizing systems, and they would amplify any hostile competitive dynamics.^[4] The “pivotal act” frame gets imported even if none of the systems are independently dangerous.

  I’ve got hope that my expected path of capabilities opens the door for more incremental interventions, but there’s a reason my total P(doom) hasn’t yet dropped much below 30%.

  1. ^

     The reason why this isn’t an update for me is that I was being deliberately conservative at the time.

  2. ^

     A hypercomputer-empowered optimizer can jump to the global optimum with brute force. There isn’t some mild greedy search to be incrementally shaped; if your specification is even slightly wrong in a sufficiently complex space, the natural and default result of a hypercomputer-optimizer is infinite cosmic horror.

  3. ^

     It’s sometimes tricky to draw a line between “oh this was a technical alignment failure that yielded an AI-derived catastrophe, as opposed to someone using it wrong,” so it’s hard to pin down the constituent probabilities.

  4. ^

     While strong AI introduces all sorts of new threats, its generality amplifies “conventional” threats like war, nukes, and biorisk, too. This could create civilizational problems even before a single AI could, in principle, disempower humanity.

  What links here?

  • Vot­ing Re­sults for the 2022 Review by Ben Pace (2 Feb 2024 20:34 UTC; 57 points)
  • habryka's comment on The LessWrong 2022 Re­view: Re­view Phase by RobertM (10 Jan 2024 22:04 UTC; 17 points)

• porby 29 Nov 2023 4:06 UTC

  26 points

  5

  on: How to Con­trol an LLM’s Be­hav­ior (why my P(DOOM) went down)

  Signal boosted! This is one of those papers that seems less known that it should be. It’s part of the reason why I’m optimistic about dramatic increases in the quality of “prosaic” alignment (in the sense of avoiding jailbreaks and generally behaving as expected) compared to RLHF, and I think it’s part of a path that’s robust enough to scale.

  You can compress huge prompts into metatokens, too (just run inference with the prompt to generate the training data). And nest and remix metatokens together.

  It’s also interesting in that it can preserve the constraints on learnable values during predictive training, unlike approaches equivalent to RL with sparse/​distant rewards.

  The fact that the distinctions it learns about the metatokens become better and better as more optimization pressure is applied is an interesting inversion of the usual doom-by-optimization story. Taking such a model to the extreme of optimization just makes it exceedingly good at distinguishing subtle details of what constitutes <nice> versus <authoritative_tone> versus <correct>. It’s an axis of progress in alignment that generalizes as the capability does; the capability is the alignment. I’m pretty certain that a model that has very thoroughly learned what “nice” means at the human level can meaningfully generalize it to contexts where it hasn’t seen it directly applied.^[1]

  I’m also reasonably confident in finding some other paths to extremely similar effects on internal representations. I wouldn’t be surprised if we can decompose conditions into representational features to learn about what they mean at the learned feature level, then cobble together new inference-time conditions via representational intervention that would have equivalent effects to training new metatokens.

  1. ^

     After all, ChatGPT4/​DALLE3 can generate an image of a vacuum cleaner that “embodies the aspirational human trait of being kind to one another.” That seems like more of a reach than a hypothetical superintelligence figuring out that humans wouldn’t be okay with, say, a superscience plan that would blow up 25% of the earth’s crust.

     

• porby 28 Sep 2022 16:49 UTC

  25 points

  7

  in reply to: jacob_cannell’s comment on: Why I think strong gen­eral AI is com­ing soon

  Out of curiosity:

  1. What rough probability do you assign to a 10x improvement in efficiency for ML tasks (GPU or not) within 20 years?

  2. What rough probability do you assign to a 100x improvement in efficiency for ML tasks (GPU or not) within 20 years?

  My understanding is that we actually agree about the important parts of hardware, at least to the degree I think this question is even relevant to AGI at this point. I think we may disagree about the software side, I’m not sure.

  I do agree I left a lot out of the hardware limits analysis, but largely because I don’t think it is enough to move the needle on the final conclusion (and the post is already pretty long!).

• porby 2 Jun 2023 17:15 UTC

  24 points

  2

  on: Think care­fully be­fore call­ing RL poli­cies “agents”

  I’m generally on board with attempts to have more precise options for referring to these concepts, and in this context I agree that policy as a term is more appropriate and that gradients from RL training don’t magically include more agent juice.

  That said, I do think there is an important distinction between the tendencies of systems built with RL versus supervised learning that arises from reward sparsity.

  In traditional RL, individual policy outputs aren’t judged in as much detail as in supervised learning. Even when comparing against RL with reward shaping, it is still likely going to be far less densely defined and constrained than, say, per-output predictive loss.

  Since the target is smaller and more distant, traditional RL gives the optimizer more room to roam. I think it’s correct to say that most RL implementations will have a lot of reactive bits and pieces that are selected to form the final policy, but because learning instrumental behavior is effectively required for traditional RL to get anywhere at all, it’s more likely (than in predictive loss) that nonmyopic internal goal-like representations will be learned as a part of those instrumental behaviors.

  Training on purely predictive loss, in contrast, is both densely informative and extremely constraining. Goals are less obviously convergently useful, and any internal goal representations that are learned need to fit within the bounds enforced by the predictive loss and should tend to be more local in nature as a result. Learned values that overstep their narrowly-defined usefulness get directly slapped by other predictive samples.

  I think the greater freedom RL training tends to have, and the greater tendency to learn more broadly applicable internal goals to drive the required instrumental behavior, do make RL-trained systems feel more “agentic” even if it is not absolutely fundamental to the training process, nor even really related to the model’s coherence.

  What links here?

  • A po­ten­tially high im­pact differ­en­tial tech­nolog­i­cal de­vel­op­ment area by Noosphere89 (8 Jun 2023 14:33 UTC; 5 points)
  • Noosphere89's comment on AI #14: A Very Good Sentence by Zvi (3 Jun 2023 14:42 UTC; 1 point)

• porby 1 May 2023 0:45 UTC

  22 points

  6

  on: Dis­cus­sion about AI Safety fund­ing (FB tran­script)

  Ways to turn $$$ into impact that aren’t happening:
  ...
  4. Increasing independent alignment researcher salary to like 150k/​year (depending on location) to enable better time money tradeoffs.

  This is a bit surprising to me!

  If we’re talking about reasonably proven high-skill researchers, 150,000 USD/​year in a reasonable cost of living area (e.g. DFW area) is… still sufficiently uncompetitive that some people that could have contributed will definitely follow the money instead. With the implied skills and background, it’s not too hard to make a lot more money with less effort.

  The implication that current funding tends to target even lower salaries than this is concerning.

  Random thoughts:

  • If this number is a rough average that includes junior researchers (e.g. just out of school), there might be no issue.

  • I know funding at or above 150k/​year is possible based on the public reports, so there isn’t a hard threshold, which is good.

  • I could see an argument for preferring funding those who are already sufficiently deep that they are willing to take significant paycuts relative to industry because of risk/​timelines/​charitability/​autonomy. That subset of people might have higher impact per dollar spent (and have lower rates of fraud), but I doubt that this is a survival maximizing strategy unless you have enough talent in that pool that you become funding constrained.

  • I’m worried that the rate of funding across the field is lower than it should be (given my timelines, at least). Seems like the game’s going to end with a lot of unspent resources.

• porby 3 Aug 2022 3:59 UTC

  22 points

  13

  in reply to: aogara’s comment on: Two-year up­date on my per­sonal AI timelines

  I suspect Chinchilla’s implied data requirements aren’t going to be that much of a blocker for capability gain. It is an important result, but it’s primarily about the behavior of current backpropped transformer based LLMs.

  The data inefficiency of many architectures was known before Chinchilla, but the industry worked around it because it wasn’t yet a bottleneck. After Chinchilla, it has become one of the largest architectural optimization targets. Given the increase in focus and the relative infancy of the research, I would guess the next two years will see the picking of some very juicy low hanging fruit. There are a lot of options floating nearby in conceptspace and there is a lot of room to grow; I’d be surprised if data limitations still feel as salient in 2025.

• porby 17 Mar 2023 21:56 UTC

  21 points

  8

  on: Shut­ting Down the Light­cone Offices

  This going to point about 87 degrees off from the main point of the post, so I’m fine with discussing this elsewhere or in DMs or something, but I do wonder how cruxy this is:

  More broadly, I think AI Alignment ideas/​the EA community/​the rationality community played a pretty substantial role in the founding of the three leading AGI labs (Deepmind, OpenAI, Anthropic), and man, I sure would feel better about a world where none of these would exist, though I also feel quite uncertain here. But it does sure feel like we had a quite large counterfactual effect on AI timelines.

  I missed the first chunk of your conversation with Dylan at the lurkshop about this, but at the time, it sounded like you suspected “quite large” wasn’t 6-48 months, but maybe more than a decade. I could have gotten the wrong impression, but I remember being confused enough that I resolved to hunt you down later to ask (which I promptly forgot to do).

  I gather that this isn’t the issue, but it does seem load bearing. A model that suggests alignment/​EA/​rationality influences sped up AGI by >10 years has some pretty heavy implications which are consistent with the other things you’ve mentioned. If my understanding is correct, if you thought the actual speedup was more like 6 months (or even negative somehow), it would take some of the bite out.

  My attempt at paraphrasing your model goes something like:
  1. Alignment efforts influence an early start on some important companies like Deepmind.
  2. These companies tend to have more “vision” with regard to AGI; they aren’t just solving narrow basic tasks for incremental economic gain, but instead are very explicitly focused on the possibility of AGI.
  3. The founding difference in vision legitimizes certain kinds of research and investment, like spending millions of dollars on training runs for general models that might seem counterfactually silly (“what are you doing, trying to chase some sci-fi nonsense? no one respectable would do that!”)
  4. The early influences more aggressively push algorithmic and hardware advancements by bringing demand forward in time.

  I’m not sure how to get a decade or more out of this, though. I’d agree this dynamic did burn some time until AGI (and even 6 months is not ideal in context), but it seems like the largest counterfactual difference is in start date, not end date. I could see our universe getting into AGI-minded research 10 years earlier than others, but I strongly suspect the “boring” kind of applications would have ended up triggering basically the same dynamic later.

  Those other worlds would still have video games encouraging a Close Enough hardware architecture and other enormous industries driving demand for improved chip manufacturing. In alt-2023, they’d still have a consumer grade video card like the 4090, just probably with less tensor core stuff. As the level of performance per dollar rises on hardware, the number of people who might try pushing scale increases until someone inevitably manages to convince VCs or other influential targets that hey, maybe there’s something here.

  (Before pivoting to safety research, “doing somewhat fringe AI stuff for funsies” was exactly the kind of thing I did. The counterfactual version of me would have almost certainly noticed, and I can’t imagine I would have been alone or even close to it.)

  At that point, I think the main difference between the worlds is that alt-world gets an even more absurdly shocking speedrun of capabilities. The hardware overhang makes it easy to pick similar low hanging fruit, just more quickly.

  I think the only way for alt-world to take as long from Speedrun Start as us is if a surprisingly large amount of our progress came from sequential conceptual insights. That really doesn’t match my impression of progress so far. Given the disjunctive architectural possibilities for reaching transformer-ish performance, it seems more like a mix of “try random stuff” and “architectures enabled by hardware.”

  Even assuming we had a 10 year head start, I’d expect the counterfactual world to reach AGI less than 48 months after us. That gives enough time for 1-2 extra hardware generations on top of the sequential efforts of engineers building the systems involved. And I’m not sure we even have a 10 year head start.

  I’d also expect the counterfactual world to be even less concerned with safety. We’re not doing well, but there are almost always ways to do much worse! In this frame, we acquired multiple leading labs who at least ostensibly consider the problem for the price of a few years. Even with my short timelines, it’s hard for me to say this was a bad deal with high confidence.

• porby 28 Sep 2022 22:34 UTC

  20 points

  13

  in reply to: mocny-chlapik’s comment on: Why I think strong gen­eral AI is com­ing soon

  As mentioned in the post, that line of argument makes me more alarmed, not less.

  1. We observe these AIs exhibiting soft skills that many people in 2015 would have said were decades away, or maybe even impossible for AI entirely.

  2. We can use these AIs to solve difficult reasoning problems that most humans would do poorly on.

  3. And whatever algorithms this AI is using to go about its reasoning, they’re apparently so simple that the AI can execute them while still struggling on absolutely trivial arithmetic.

  4. WHAT?

  Yes, the AI has some blatant holes in its capability. But what we’re seeing is a screaming-hair-on-fire warning that the problems we thought are hard are not hard.

  What happens when we just slightly improve our AI architectures to be less dumb?

• porby 31 Dec 2022 20:54 UTC

  19 points

  8

  on: The Feel­ing of Idea Scarcity

  It’s not the only thread I’m pulling on.

  I think this is worth expanding on- in practice, I’ve found the strongest method for avoiding the “oh no my great idea is not working out but I’m stuck in it” trap is to have other promising options just waiting for you to poke them.

  Instead of feeling trapped and entering a cycle of motivation-killing burnout, a dying idea starts feeling just… kind of boring, and you naturally want to do the other more interesting thing. You don’t even have to try, you just find yourself thinking about it in the shower. When refined and used intentionally, it can align productivity with natural inclinations.

  I tend to organize my work as a few different parallel depth first searches. In a given day, I’ll often work on two different things, and in a given week I might hit three or five. Something like spaced repetition applied to fields of study, with time for the other ideaspaces to lie fallow. This tends to produce recursively expanding options for research. After doing this for years, it’s now completely impossible for me to actually investigate everything I’ve written down in a todo somewhere, but at least I definitely don’t feel married to any one idea!

  I suspect getting the full benefit does require a few (dozen) iterations of failure. Noticing that you already know an idea is dead is a lot easier when you’ve felt that same thing 20 times.

• porby 4 Jan 2023 17:24 UTC

  18 points

  13

  in reply to: Antb’s comment on: 2022 was the year AGI ar­rived (Just don’t call it that)

  While there is a limit to the current text datasets, and expanding that with high quality human-generated text would be expensive, I’m afraid that’s not going to be a blocker.

  Multimodal training already completely bypasses text-only limitations. Beyond just extracting text tokens from youtube, the video/​audio itself could be used as training data. The informational richness relative to text seems to be very high.

  Further, as gato demonstrates, there’s nothing stopping one model from spanning hundreds of distinct tasks, and many of those tasks can come from infinite data fountains, like simulations. Learning rigid body physics in isolation isn’t going to teach a model english, but if it’s one of a few thousand other tasks, it could augment the internal model into something more general. (There’s a paper that I have unfortunately forgotten the name of that created a sufficiently large set of permuted tasks that the model could not actually learn to perform each task, and instead had to learn what the task was within the context window. It worked. Despite being toy task permutations, I suspect something like this generalizes at sufficient scale.)

  And it appears that sufficiently capable models can refine themselves in various ways. At the moment, the refinement doesn’t cause a divergence in capability, but that’s no guarantee as the models improve.

• porby 20 Oct 2022 16:30 UTC

  18 points

  10

  in reply to: Sen’s comment on: Why I think strong gen­eral AI is com­ing soon

  May the forces of the cosmos intervene to make me look silly.

  What links here?

  • Am I se­cretly ex­cited for AI get­ting weird? by porby (29 Oct 2022 22:16 UTC; 115 points)
  • AGI in our life­times is wish­ful thinking by niknoble (24 Oct 2022 11:53 UTC; 0 points)

• porby 6 Nov 2023 20:31 UTC

  17 points

  11

  in reply to: Thane Ruthenis’s comment on: TurnTrout’s short­form feed

  My model says that general intelligence^[1] is just inextricable from “true-goal-ness”. It’s not that I think homunculi will coincidentally appear as some side-effect of capability advancement — it’s that the capabilities the AI Labs want necessarily route through somehow incentivizing NNs to form homunculi. The homunculi will appear inasmuch as the labs are good at their jobs.

  I’ve got strong doubts about the details of this. At the high level, I’d agree that strong/​useful systems that get built will express preferences over world states like those that could arise from such homunculi, but I expect that implementations that focus on inducing a homunculus directly through (techniques similar to) RL training with sparse rewards will underperform more default-controllable alternatives.

  My reasoning would be that we’re bad at using techniques like RL with a sparse reward to reliably induce any particular behavior. We can get it to work sometimes with denser reward (e.g. reward shaping) or by relying on a beefy pre-existing world model, but the default outcome is that sparse and distant rewards in a high dimensional space just don’t produce the thing we want. When this kind of optimization is pushed too far, it’s not merely dangerous; it’s useless.

  I don’t think this is temporary ignorance about how to do RL (or things with similar training dynamics). It’s fundamental:

  1. Sparse and distant reward functions in high dimensional spaces give the optimizer an extremely large space to roam. Without bounds, the optimizer is effectively guaranteed to find something weird.

  2. For almost any nontrivial task we care about, a satisfactory reward function takes a dependency on large chunks of human values. The huge mess of implicit assumptions, common sense, and desires of humans are necessary bounds during optimization. This comes into play even at low levels of capability like ChatGPT.

  Conspicuously, the source of the strongest general capabilities we have arises from training models with an extremely constraining optimization target. The “values” that can be expressed in pretrained predictors are forced into conditionalization as a direct and necessary part of training; for a reasonably diverse dataset, the resulting model can’t express unconditional preferences regarding external world states. While it’s conceivable that some form of “homunculi” could arise, their ability to reach out of their appropriate conditional context is directly and thoroughly trained against.

  In other words, the core capabilities of the system arise from a form of training that is both densely informative and blocks the development of unconditional values regarding external world states in the foundational model.

  Better forms of fine-tuning, conditioning, and activation interventions (the best versions of each, I suspect, will have deep equivalences) are all built on the capability of that foundational system, and can be directly used to aim that same capability. Learning the huge mess of human values is a necessary part of its training, and its training makes eliciting the relevant part of those values easier—that necessarily falls out of being a machine strongly approximating Bayesian inference across a large dataset.

  The final result of this process (both pretraining and conditioning or equivalent tuning) is still an agent that can be described as having unconditional preferences about external world states, but the path to get there strikes me as dramatically more robust both for safety and capability.

  Summarizing a bit: I don’t think it’s required to directly incentivize NNs to form value-laden homunculi, and many of the most concerning paths to forming such homunculi seem worse for capabilities.

• porby 28 Sep 2022 20:23 UTC

  16 points

  6

  in reply to: Lone Pine’s comment on: Why I think strong gen­eral AI is com­ing soon

  Now you know how the transformer feels!

• porby 28 Sep 2022 17:10 UTC

  16 points

  0

  in reply to: jacob_cannell’s comment on: Why I think strong gen­eral AI is com­ing soon

  Looks like we’re in almost perfect agreement!

• porby 9 Jun 2023 23:06 UTC

  15 points

  9

  on: Up­dates and Reflec­tions on Op­ti­mal Ex­er­cise af­ter Nearly a Decade

  The basic idea that lifting twice a week and doing cardio twice a week add up to a calorie expenditure that get you the vast majority of exercise benefits compared to extreme athletes holds up, especially when you take reverse causality adjustments into effect (survivorship bias on the genetic gifts of the extreme). Nothing I’ve encountered since has cast much doubt on this main takeaway.

  I’d like to offer further reinforcement on this point:

  1. Any exercise, even trivial amounts like walking 20 minutes or doing two sets of exercise a day, can yield dramatic improvements in quality of life. Young-ish healthy-ish people don’t tend to notice this because they’re already above the “life isn’t constantly painful and difficult” bar, but aging combined with a highly sedentary lifestyle can sneak up on some people. It can sometimes take very little to fix or avoid.

  2. Maintaining muscle mass that you’ve already built is much easier than building it in the first place. Some people find the thought of progressively overloading for eternity to be daunting- and if you don’t want to, you don’t have to! If you achieve a level of strength that satisfies your requirements, you can maintain it with relatively little effort. And if you do lose it, getting strong a second time will be easier.

  In other words: when it comes to exercise, doing anything really does help! (Just don’t hurt yourself.)

• porby 1 Jan 2023 3:44 UTC

  15 points

  8

  on: ‘simu­la­tor’ fram­ing and con­fu­sions about LLMs

  In fact, although the *output* tokens are myopic, autoregressive transformers are incentivised to compute activations at early sequence positions that will make them better at predicting tokens at later positions. This may also have indirect impacts on the actual tokens output at the early positions, although my guess would be this isn’t a huge effect.

  (I found myself writing notes down to clarify my own thoughts about parts of this, so this is in large part talking to myself that got commentified, not quite a direct reply)

  It’s true that gradients can flow to causally visible tokens and modify weights to serve future predictions. This does break a type of narrow myopia, but I don’t find that break very concerning.

  A previous token’s prediction receives its own calibration; weight modifications need to serve many such predictions to accumulate reliably.

  That’s a pretty heavy constraint. There often aren’t many degrees of freedom for earlier tokens to shift the prediction to serve a later prediction without harming the current one. A well-calibrated predictor can create self-fulfilling or self-easing prophecies while managing locally low loss only when the context permits it.

  Another angle: suppose there are two possible predictions P0 and P1 which are estimated to be equivalently and maximally probable and are distinct. We’ll assume P1 would make a future token prediction easier.

  Despite P1 having a long term advantage, the model cannot simply choose to bias P1′s probability up as that would worsen the current prediction’s loss in expectation. In order for P1 to be preferred against local pressure, the future expected benefit must swamp the local influence.

  Under what circumstances can that occur? In offline training, the model isn’t consuming its own outputs. It’s being calibrated against an external ground truth. Biasing up P1 isn’t helpful during training, because making the future easier to predict in an autoregressive context doesn’t reduce loss at training time. Matching the distribution of the input does. In this context, a bias in prediction is almost certainly from undertraining or a lack of capability.

  (The other options would tend to be esoteric stuff like “the model is extremely strong to the point of simulated agents managing a reflective gradient hacking attempt of some sort,” which doesn’t seem to be an obvious/​natural/​necessary outcome for all implementations.)

  In other words, the manner in which GPT-like architectures lack myopia is that the model can learn to predict the future beyond the current token in the service of predicting the current token more accurately.

  There’s probably some existing terminology for this split that I don’t know about, but I might call it something like myopic perception versus myopic action. GPT-likes do not have myopic perception, but they do have myopic action, and myopic action is the important one.

• porby 29 Sep 2022 2:06 UTC

  15 points

  3

  in reply to: Ege Erdil’s comment on: Why I think strong gen­eral AI is com­ing soon

  Notably, the result is correct; I did convert it to kelvin for the actual calculation. Just a leftover from when I was sketching things on wolframalpha. I’ll change that, since it is weird. (Thanks for the catch!)

• porby 28 Sep 2022 22:49 UTC

  15 points

  6

  in reply to: agi-hater’s comment on: Why I think strong gen­eral AI is com­ing soon

  Hmm...

  Given the new account, the account name, the fact that there were a few posts in the minutes prior to this one rejected by the spam filter, the arguments, and the fact that the decently large followup comment was posted only 3 minutes after the first...

  … are… are you the AI? Trying to convince me of dastardly things?

  You can’t trick me!

  :P