Veedrac

Op­ti­mal­ity is the tiger, and agents are its teeth

Moore’s Law, AI, and the pace of progress

Post-his­tory is writ­ten by the martyrs

• Veedrac 30 Sep 2023 21:36 UTC

  44 points

  16

  on: “Di­a­mon­doid bac­te­ria” nanobots: deadly threat or dead-end? A nan­otech in­ves­ti­ga­tion

  I’m not sure how to put this, but while this post is framed as a response to AI risk concerns, those concerns are almost entirely ignored in favor of looking at how plausible it is for near-term human research to achieve it, and only at the end is it connected back to AI risk via a brief aside whose crux is basically that you don’t think Yudkowsky-style ASI will exist.

  I like a lot of the discussion if I frame it in my head to be about what it is actually arguing for. Taking it as given, it seems instead broadly non-sequiter, as the evidence given basically doesn’t relate to resolving the disagreement.

• Veedrac 2 Apr 2022 19:48 UTC

  40 points

  in reply to: TLW’s comment on: New Scal­ing Laws for Large Lan­guage Models

  It mostly only means that training them compute optimally will require much more data, and doesn’t rule out OpenAI-style mostly-parameter scaling at all. Data scaling can be necessary to minimise loss to get optimal estimates of certain entropic variables, while still being unnecessary for general intelligence. Large undertrained models still learn faster. This new paper mostly makes parameter and data scaling both significantly more efficient, but data scaling to a larger degree, such that it’s more optimal to trade off these losses 1:1.

  Below the fold is musing and analysis around this question. It is not a direct answer to it though.

  ---

  We can take a look at the loss function, defined in terms of the irreducible loss, aka. the unmodelable entropy of language, the number of parameters $N$, and the number of data tokens $D$.

  $L(N,D)=1.69+\frac{406.4}{N^{0.34}}+\frac{410.7}{D^{0.28}}$

  If we put in the parameters for Chinchilla, we see $\frac{406.4}{N^{0.34}}\approx 0.083$, and $\frac{410.7}{D^{0.28}}\approx 0.163$. Although these equations have been locally tuned and are not valid in the infinite limit of a single variable, it does roughly say that just scaling parameter counts without training for longer will only tackle about a third of the remaining reducible loss.

  Note the implicit assumption that we are working in the infinite data limit, where we never intentionally train on the same tokens twice. If you run out of data, it doesn’t mean you are no longer able to train your models for longer as you scale, it only means that you will have to make more use of the data you already have, which can mean as little as multiple epochs or as much as sophisticated bootstrapping methods.

  The original scaling laws did not decompose so easily. I present them in simplified form.

  $L(N,D)={(\frac{1.54e10}{N^{0.738}}+\frac{1.8e13}{D})}^{0.103}$

  (Note that the dataset was different so the exact losses shouldn’t be centered identically.)

  This has major issues, like there is no irreducible loss and the values aren’t disentangled. We can still put in the parameters for GPT-3: $\frac{1.54e10}{N^{0.738}}\approx 77.7$ and $\frac{1.8e13}{D}\approx 60$; or in the limits, ${\left(\frac{1.54e10}{N^{0.738}}\right)}^{0.103}\approx 1.57$ and ${\left(\frac{1.8e13}{D}\right)}^{0.103}\approx 1.52$. It isn’t clear what this means about the necessary amount of data scaling, as in what fraction of the loss that it captures, especially because there is no entropy term, but it does mean that there is still about 1:1 contributions from both losses at the efficient point, at least if you ignore the fact that the equation is wrong. That you have to scale both in tandem to make maximal progress remains true in this older equation, it’s just more convoluted and has different factors.

• Veedrac 14 Mar 2022 2:32 UTC

  32 points

  on: Challenges to Yud­kowsky’s Pro­noun Re­form Proposal

  I think the length of this post detracts a lot from its clarity. There is a summary, but it’s hard to have an opinion about it without having read the whole rest of the post, which I only skimmed.

  Yudkowsky’s point as I understand it is just that the position of maximal abstention when it comes to linking gender to pronouns is to use the same default anyone else uses, but then allow anyone to have their personal preference about which term they feel most comfortable referenced by. This is not specifically to exert a stance in favor of transgender people being named by the transitioned-to gender, nor is it to deny that there is any preexisting connotation to the pronouns, it’s just because to object to someone’s request by saying the answer is anything else is to make a nominative statement about how gender should relate to pronouns, which is what you’re abstaining from.

  My comment was a lot longer than this, but I deleted the rest because these conversations feel like knives.

• Veedrac 28 May 2023 15:14 UTC

  25 points

  1

  on: You now can en­able re­acts on your own posts! (and other re­act info)

  Quick feedback,

  • The icons aren’t all obviously interpretable to me

    • Not a crux — looks like a +, rec: ‘crux’ plus ‘🚫’

    • Please elaborate — unclear that it’s a request, rec: ‘..?’

    • What’s your prediction — rec: ‘$P\left(x\right)?$’

    • Examples, please — rec: ‘ex?’

    • Additional questions — rec: ‘??’

    • Obtuse — rec: remove

  • Some reactions seem tonally unpleasant:

    • Not what I meant — idea is good, but icon is accusatory

    • I already addressed this — icon is good, text could be ‘addressed elsewhere’

    • Muddled — maybe replace with ‘Unclear’?

    • Obtuse — maybe replace with ‘Too Long’?

    • Not worth getting into — feels like a meaner version of Not Planning to Respond

    • Note that I do like some critical reactions as-is, like Too Many Assumptions

  • There are too many to remember easily; perhaps

    • remove some partial redundancies, like Shrug + Seems Borderline?

    • add one-word summaries to the icon box, like [🕑 discussed 4]?

    • make it easier to see descriptions on mobile?

  • I think a top level grouping like this could make sense:

    • Positive — eg. Thanks, Important, Exciting, Clear

    • Critical — eg. Taboo, Harsh, Non Sequitur

    • Informational — eg. Will/​Won’t Reply Later, Agree to This, Shrug

  • There should be a Bikeshed emoji, for comments like this one

• Veedrac 6 Oct 2023 0:48 UTC

  24 points

  11

  in reply to: Rob Bensinger’s comment on: Eval­u­at­ing the his­tor­i­cal value mis­speci­fi­ca­tion argument

  This is a bad analogy. Phoning a human fails dominantly because humans are less smart than the ASI they would be trying to wrangle. Contra, Yudkowsky has even said that were you to bootstrap human intelligence directly, there is a nontrivial short that the result is good. This difference is load bearing!

  This does get to the heart of the disagreement, which I’m going to try to badly tap out on my phone.

  The old, MIRI-style framing was essentially: we are going to build an AGI out of parts that are not intrinsically grounded in human values, but rather good abstract reasoning, during execution of which human values will be accurately deduced, and as this is after the point of construction, we hit the challenge of formally specifying what properties we want to preserve without being able to point to those runtime properties at specification.

  The newer, contrasting framing is essentially: we are going to bulld an AGI out of parts that already have strong intrinsic, conceptual-level understanding of the values we want them to preserve, and being able to directly point at those values is actually needle-moving towards getting a good outcome. This is hard to do right now, with poor interpretability and steerability of these systems, but is nonetheless a relevant component of a potential solution.

• Veedrac 27 Jul 2020 16:25 UTC

  LW: 21 AF: 7

  AF

  in reply to: Tomás B.’s comment on: Are we in an AI over­hang?

  Moore’s Law is not dead. I could rant about the market dynamics that made people think otherwise, but it’s easier just to point to the data.

  https://​​docs.google.com/​​spreadsheets/​​d/​​1NNOqbJfcISFyMd0EsSrhppW7PT6GCfnrVGhxhLA5PVw

  Moore’s Law might die in the short future, but I’ve yet to hear a convincing argument for when or why. Even if it does die, Cerebras presumably has at least 4 node shrinks left in the short term (16nm→10nm→7nm→5nm→3nm) for a >10x density scaling, and many sister technologies (3D stacking, silicon photonics, new non-volatile memories, cheaper fab tech) are far from exhausted. One can easily imagine a 3nm Cerebras waffle coated with a few layers of Nantero’s NRAM, with a few hundred of these connected together using low-latency silicon photonics. That would easily train quadrillion parameter models, using only technology already on our roadmap.

  Alas, the nature of technology is that while there are many potential avenues for revolutionary improvement, only some small fraction of them win. So it’s probably wrong to look at any specific unproven technology as a given path to 10,000x scaling. But there are a lot of similarly revolutionary technologies, and so it’s much harder to say they will all fail.

• Veedrac 22 Mar 2024 8:24 UTC

  18 points

  3

  on: “Deep Learn­ing” Is Func­tion Approximation

  It took me a good while reading this to figure out whether it was a deconstruction of tabooing words. I would have felt less so if the post didn’t keep replacing terms with ones that are both no less charged and also no more descriptive of the underlying system, and then start drawing conclusions from the resulting terms’ aesthetics.

  With regards to Yudkowsky’s takes, the key thing to keep in mind is that Yudkowsky started down his path by reasoning backwards from properties ASI would have, not from reasoning forward from a particular implementation strategy. The key reason to be concerned that outer optimization doesn’t define inner optimization isn’t a specific hypothesis about whether some specific strategy with neural networks will have inner optimizers, it’s because ASI will by necessity involve active optimization on things, and we want our alignment techniques to have at least any reason to work in that regime at all.

• Veedrac 7 Jan 2021 23:33 UTC

  18 points

  in reply to: Polytopos’s comment on: DALL-E by OpenAI

  Audio, video, text, images

  While other media would undoubtedly improve the model’s understanding of concepts hard to express through text, I’ve never bought the idea that it would do much for AGI. Text has more than enough in it to capture intelligent thought; it is the relations and structure that matters, above all else. If this weren’t true, one wouldn’t expect competent deafblind people, but there are. Their successes are even in spite of an evolutionary history with practically no surviving deafblind ancestors! Clearly the modules that make humans intelligent, in a way that other animals and things are not, are not dependent on multisensory data.

• Veedrac 3 May 2022 18:49 UTC

  17 points

  in reply to: Eli Tyre’s comment on: Op­ti­mal­ity is the tiger, and agents are its teeth

  I think that’s mostly a really good summary. The major distinction I would try to make is that agenthood is primarily a way to actualize power, rather than a source of it.

  If you had an agent that wasn’t strongly optimized in any sense other than it was an agent, in that it had goals and wanted to solve them, that wouldn’t make it dangerous, any more than your dog is dangerous for being an agent. Whereas the converse, if you have something that’s strongly optimised in some more generic sense, but wasn’t an agent, this still puts you extremely close to a lot of danger. The article was trying to emphasize this by pointing to the most reductive form of agenthood I could see, in that none of the intrinsic power of the resulting system could reasonably be attributed to any intrinsic smartness of the agent component, even if the system was an agent that was powerful.

• Veedrac 19 Dec 2021 20:29 UTC

  15 points

  12

  on: Oc­cu­pa­tional Infohazards

  It might help your case to write a version of this that removes most of the interpretation you’ve given here, and tries to present just the claims you know to be objective truths. While ‘the plaintiff is failing to personally provide an objective neutral point of view’ seems like a particularly disturbing sort of argument to dismiss something like this on, it is nonetheless the case that this does seem to be the principal defense, and most of those comments are pointing to real issues in your presentation.

  Disclaimer, I’m an outsider.

• Veedrac 8 Dec 2021 22:57 UTC

  15 points

  in reply to: Quintin Pope’s comment on: Deep­mind’s Go­pher—more pow­er­ful than GPT-3

  Skimming the Rᴇᴛʀᴏ paper is weird because it looks like there’s leakage everywhere, they admit leakage is everywhere, but then they sort of report results like it doesn’t matter, even putting a result on their leakiest dataset in their conclusion?

  On Wikitext103 and the Pile, Retro outperforms previous models trained on large scale datasets.

  It looks to me like Figure 6 is saying the improvement is fairly modest in unleaky datasets?

  Maybe someone who has gone over the paper in detail can chime in with thoughts.

  What links here?

  • paulfchristiano's comment on Deep­mind’s Go­pher—more pow­er­ful than GPT-3 by hath (8 Dec 2021 22:56 UTC; 21 points)

• Veedrac 24 Dec 2021 1:42 UTC

  14 points

  1

  on: Re­ply to Eliezer on Biolog­i­cal Anchors

  What actual important claim can you make on the basis of the ${10}^{43}$ estimate? I don’t see a way to attach it to timelines, at all. I don’t see how to attach it to median or modal compute estimates, at all.

• Veedrac 24 Dec 2021 4:11 UTC

  13 points

  −7

  in reply to: ESRogs’s comment on: Re­ply to Eliezer on Biolog­i­cal Anchors

  someone tells you that in year X it will cost $1000 and a few hours to perform as much computation as all of evolution

  Nobody has told you this, though? It is completely coherent to think, and I do in fact expect, this is a much harder task than AGI, even in 1000-year scenarios where NNs get nowhere.

  It’s a bit like that math trope of proving statements by first assuming their complex generalizations. This doesn’t actually get you any information that the assumption didn’t trivially hold. When is that theoretical timeline situated, where we have ~infinite compute? How is it relevant to determining actual estimates we actually care about?

  E: I’m worried I’m not saying what I’m trying to say well, and that it’s just coming across as dismissive, but I really am trying to point at something I think is important and missing in your argument, that would be a lot more apparent if you were explicitly trying to point at the thing you think is useful about what you are doing with this number.

• Veedrac 1 Jun 2023 23:00 UTC

  12 points

  20

  on: Yud­kowsky vs Han­son on FOOM: Whose Pre­dic­tions Were Bet­ter?

  I think this is a pretty good and fair roundup, but I want to add as very lazy bit of personal context short of actually explaining my takes:

  Both when I read the FOOM debate, and skimming over it again now, in my personal opinion Yudkowsky largely comes off better. Yudkowsky makes a few major mistakes that are clearly visible now, like being dismissive of dumb, scaled, connectionist architectures, but the arguments seem otherwise repairable. Contra, I do not know how to well defend Hanson’s position.

  I don’t state this to claim a winner, and for sure there are people who read the arguments the other way, but only to suggest to the reader, if you have the time, consider taking a look and forming your own opinion.

• Veedrac 4 Dec 2021 1:30 UTC

  12 points

  on: Shul­man and Yud­kowsky on AI progress

  I have some comments on things Eliezer has said. I don’t expect these disagreements are very important to the main questions, because I tend to agree with him overall despite it.

  my worldview also permits but does not mandate that you get up to the chimp level [...]

  A naïve model of intelligence is a linear axis, that puts everything on a simple trendline with one-dimensional distances. I assume most people here understand that this is a gross oversimplification. Intelligence is made of multiple pieces, which can have unique strengths. There is still such a thing as generality of intelligence, that at some point you have enough tools to dynamically apply your reasoning to a great many more things than those tools were originally adapted for. This ability does seem to have some degree of scale, in that a human is more general than a chimp is more general than a mouse, though it also seems to be fairly sharp-edged, in that the difference in generality between a human and a chimp seems much greater than between a chimp and a mouse.

  Because of the great differences between computer systems and biological ones, the individual components of computer intelligence (whether necessary for generality or not) when measured relative to a human tend to jump quickly between zero, when the program doesn’t have that ability, and effectively infinite, when the program has that ability. There is also a large set of relations between capabilities whereby one ability can substitute for another, typically at the cost of some large factor reduction in performance.

  A traditional chess engine has several component skills, like searching moves and updating decision trees, that it does vastly better than a human. This ability feeds down into some metrics of positional understanding. Positional understanding is not a particularly strong fundamental ability of a traditional chess engine, but rather something inefficiently paid for with its other skills that it does have in excess. The same idea holds for human intelligence, where we can use our more fundamental evolved skills, like object recognition, to build more complex skills. Because we have a broad array of baseline skills, and the enough tools to combine them to fit novel tasks, we can solve a much wider array of tasks, and can transfer domains with generally less cost than computers can. Nonetheless, there exist cognitive tasks we know can be done well that are outside of human mental capability.

  When I envision AI progress leading up to AGI, I don’t think of a single figure of merit that increases uniformly. I think of a set of capabilities, of which some are ~0, some are ~∞, and others are derived quantities not explicitly coded in. Scale advances in NNs push the effective infinities to greater effective infinities, and by extension push up the derived quantities across the board. Fundamental algorithmic advances increase the set of capabilities at ~infinity. At some point I expect the combination of fundamental and derived quantities to capture enough facets of cognition to push generality past a tipping point. In the run up to that point, lesser levels of generality will likely make AI systems applicable to more and more extensions of their primary domains.

  This seems to me like it’s mostly, if not totally, a literal interpretation of the world. Yet, to finally get to the point, nowhere in my map do I have a clear interpretation of what “get up to the chimp level” means. The degree to which chimps are generally intelligent seems very specific to the base skill set that chimps have, and it seems much more likely than not that AI will approach it from a completely different angle, because their base skillset is completely different and will generalize in a completely different way. The comment that “chimps are not very valuable” does not seem to map onto any relevant comment about pre-explosion AI. I do not know what it would mean to have a chimp level AI, or even chimp level generality.

  I would not be terribly surprised to find that results on benchmarks continue according to graph, and yet, GPT-4 somehow does not seem very much smarter than GPT-3 in conversation.

  I would be quite surprised for a similar improvement in perplexity not to correspond to at least a similar improvement in apparent smartness, versus GPT-3 over GPT-2.

  I would not be surprised for the perplexity improvement to level off, maybe not immediately but at least in some small count of generations, as it seems entirely reasonable that there are some aspects of cognition that GPT-style models can’t purchase. But for perplexity to improve on cue without an apparent improvement in intelligence, while logically coherent, would imply some very weird things about either the entropy of language or the scaling of model capacity.

  That is, either language has a bunch of easy to access regularity between what GPT-3 reached and what an agent with a more advanced understanding of the world could reach, distributed coincidentally in line with previous capability increases, or GPT-3 roughly caps out the semantic capabilities of the network, but extra parameters added on top are still practically identically effective at extracting a huge amount of more marginal non-semantic regularities at a rate fast enough to compete with prior model scale increases that did both.

  Stuff coming uncorrelated that way, sounds like some of the history I lived through, where people managed to make the graphs of Moore’s Law seem to look steady by rejiggering the axes, and yet, between 1990 and 2000 home computers got a whole lot faster, and between 2010 and 2020 they did not.

  There is truth to this comment, in that Dennard scaling fell around the turn of the millenia, but the 2010s were deceptive in that home computers stagnated in performance despite Moore’s Law improvements, because Intel sold you your CPUs, were stuck on an old node, being stuck on an old node prevented them from pushing out new architectures, and their monopoly position meant they never really needed to compete on price or core count either.

  But Moore’s Law did continue, just through TSMC, and the corresponding performance improvements were felt primarily in GPUs and mobile SoCs. Both of those have improved at a great pace. In the last three years competition has returned to the desktop CPU market, and Intel has just managed to get out of their node crisis, so CPU performance really is picking up steam again. This is true both for per-core performance, driven by architectures making use of the great many transistors available, and even moreso true of aggregate performance, what with core counts in the Steam Survey increasing from an average of 3.0 in early 2017 to an average of 5.0 in April this year.

  You are correct that the scaling regimes are different now, and Dennard scaling really is dead for good, but if you look back at the original Moore’s Law graphs from 1965, they never mentioned frequency, so I don’t buy the claim that the graphs have been rejigged.

• Veedrac 2 Dec 2021 17:11 UTC

  12 points

  in reply to: TekhneMakre’s comment on: Biol­ogy-In­spired AGI Timelines: The Trick That Never Works

  Like, we could extrapolate out 3 OOMs of compute/​$ per decade to get an upper bound: very probably AGI before 2150-ish, if Moore’s law continues.

  Projecting Moore’s Law to continue for 130 years more is almost surely incorrect. An upper bound that is conditional on that happening seems devoid of any actual predictive power. If we approach that level of computational power prior to AGI, it will almost surely be through some other mechanism than Moore’s Law, and so would be arbitrarily detached from that timeline.

• Veedrac 24 Jun 2021 2:55 UTC

  12 points

  on: Open and Wel­come Thread – June 2021

  Helion Energy has recently announced that their most recent fusion generator prototype has been running for 16 months at 100 million degrees with 10,000 operational cycles, with “upwards of 95%” energy harvesting and reclamation efficiency. This is an unusual setup because it recovers most of the energy used to compress the plasma, which means that the break-even point for net energy gain (aka. $Q_E$ > 1) is much lower. Some people on r/​fusion have estimated $Q_E$ factors of around 4, given public information, but, and here’s the catch, Helion have not given any $Q$ value for the machine, nor said whether it breaks even.

  If these commenters are right—and I’m not going to dismiss that possibility outright, given a lot of the subreddit members are professionals in the field—this would represent perhaps the weirdest possible timeline for fusion development I can imagine: that a public startup had achieved $Q_E$ > 1 while people are still mocking fusion as forever decades out, and for well over a year they’ve just… chosen not to tell anybody.

  Of course, it could just be that $Q_E$ is less than 1, and this is merely a very exciting prelude.