Logan Zoellner

Karma: 1,132

Logan Zoellner 31 May 2024 21:33 UTC
2 points
0
on: Environmentalism in the United States Is Unusually Partisan
The best data comes from a Pew survey of “17 advanced economies” in 2021
Not sure this is evidence of increased partisanship or just the fact that other nations are more liberal in general. The ⁷²⁄₉₈ gap in Canada means conservatives are ~15x less likely to be “willing to adjust”.
I do think the environmentalist movement is too entangled with other liberal causes (consider, for example the Green New Deal, which is largely a socialist wishlist of demands unrelated to the environment).
As a useful counterexample, consider the YIMBY movement which has actually done a decent job of avoiding entanglement with partizan issues.

Logan Zoellner 17 May 2024 17:09 UTC
5 points
3
in reply to: RobertM’s comment on: Against “argument from overhang risk”
We ran into a hardware shortage during a period of time where there was no pause, which is evidence that the hardware manufacturer was behaving conservatively.
Alternative hypothesis, there are physical limits on how fast you can build things.

Also, NVIDIA currently has a monopoly on “decent AI accelerator you can actually buy”. Part of the “shortage” is just the standard economic result that a monopoly produces less of something to increase profits.
This monopoly will not last forever, so in that sense we are currently in hardware “underhang”.

This and the rest of your comment seems to have ignored the rest of my post (see: multiple inputs to progress, all of which seem sensitive to “demand”
Nvidia doesn’t just make AGI accelerators. They are are video game graphics card company.
And even if we pause large training runs, demand for inference of existing models will continue to increase.
If you think my model of how inputs to capabilities progress are sensitive to demand for those inputs from AGI labs is wrong, then please argue so directly, or explain how your proposed scenario is compatible with it.
This is me arguing directly.
The model “all demand for hardware is driven by a handful of labs training cutting edge models” is completely implausible. It doesn’t explain how we got the hardware in the first place (video games) and it ignores the fact that there exist uses for AI acceleration hardware other than training cutting-edge models.

Logan Zoellner 16 May 2024 14:43 UTC
13 points
3
on: Against “argument from overhang risk”
To me, the recent hardware shortage is very strong evidence that we will not be surprised by a sharp jump in capabilities after a pause, as a result of the pause creating an overhang that eliminates all or nearly all bottlenecks to reaching ASI.
I don’t follow the reasoning here. Shouldn’t a hardware shortage be evidence we will see a spike after a pause?
For example, suppose we pause now for 3 years and during that time NVIDIA releases the RTX5090,6090,7090 which are produced using TSMC’s 3nm, 2nm and 10a processes. Then the amount of compute available at the end of the three year pause will be dramatically higher than it is today. (for reference, the 4090 is 4x better at inference than the 3090). Roughly speaking, then, after your 3 year pause a billion dollar investment will buy 64x as much compute (this is more than the difference between GPT-4 and GPT-3).
Also, a “pause” would most likely only be a cap on the largest training runs. It is unlikely that we’re going to pause all research on current LLM capabilities. Consider that a large part of the “algorithmic progress” in LLM inference speed is driven not by SOTA models, but by hobbyists trying to get LLMs to run faster on their own devices.
This means that in addition to the 64x hardware improvement, we would also get algorithmic improvement (which has historically faster than hardware improvement).
That means at the end of a 3 year pause, an equal cost run would be not 64x but 4096x larger.
Finally, LLMs have already reached the point where they can be reasonably expected to speed up economic growth. Given their economic value will become more obvious over time,the longer we pause, the more we can expect that the largest actors will be willing to spend on a single run. It’s hard to put an estimate on this, but consider that historically the largest runs have been increasing at 3x/year. Even if we conservatively estimate 2x per year, that gives us an additional 8x at the end of our 3 year pause. This now gives us a factor of 32k at the end of our 3 year pause.
Even if you don’t buy that “Most alignment progress will happen from studying closer-to-superhuman models”, surely you believe that “large discontinuous changes are risky” and a factor of 32,000x is a “large discontinuous change”.

Logan Zoellner 6 May 2024 14:03 UTC
0 points
−7
in reply to: habryka’s comment on: an effective ai safety initiative
It’s not trying to address present harms, it’s trying to address future harms, which are the important ones.
A real AI system that kills literally everyone will do so by gaining power/resources over a period of time. Most likely it will do so the same way existing bad-agents accumulate power and resources.
Unless you’re explicitly committing to the Diamondoid bacteria thing, stopping hacking is stopping AI from taking over the world.

Logan Zoellner 6 May 2024 13:59 UTC
6 points
0
in reply to: habryka’s comment on: an effective ai safety initiative
Point taken. “$$$” was not the correct framing (if we’re specifically talking about the Gwern story). I will edit to say “it accumulates ‘resources’”.
The Gwern story has faster takeoff than I would expect (especially if we’re talking a ~GPT4.5 autoGPT agent), but the focus on money vs just hacking stuff is not the point of my essay.

Logan Zoellner 26 Apr 2024 1:41 UTC
2 points
−2
on: The first future and the best future
1. What plateau? Why pause now (vs say 10 years ago)? Why not wait until after the singularity and impose a “long reflection” when we will be in an exponentially better place to consider such questions.
2. Singularity 5-10 years from now vs 15-20 years from now determines whether or not some people I personally know and care about will be alive.
3. Every second we delay the singularity leads to a “cosmic waste” as millions more galaxies move permanently behind the event horizon defined by the expanding universe
4. Slower is not prima facia safer. To the contrary, the primary mechanism for slowing down AGI is “concentrate power in the hands of a small number of decision makers,” which in my current best guess increases risk.
5. There is no bright line for how much slower we should go. If we accept without evidence that we should slow down AGI by 10 years, why not 50? why not 5000?

Logan Zoellner 26 Apr 2024 1:21 UTC
15 points
11
on: Losing Faith In Contrarianism
Sam Atis—a super forecaster—had a piece arguing against The Case Against Education
If it’s this piece, I would be interested to know why you found it convincing. He doesn’t address (or seem to have even read) any of Brian’s arguments. His argument basically boils down to “but so many people who work for universities think it’s good”.

Logan Zoellner 30 Mar 2024 20:21 UTC
2 points
0
in reply to: ReaderM’s comment on: Modern Transformers are AGI, and Human-Level
then that’s just irrelevant. You don’t need to evaluate millions of positions to backtrack (unless you think humans don’t backtrack) or play chess.
Humans are not transformers. The “context window” for a human is literally their entire life.

Logan Zoellner 30 Mar 2024 12:05 UTC
2 points
0
in reply to: ReaderM’s comment on: Modern Transformers are AGI, and Human-Level
Setting up the architecture that would allow a pretrained LLM to trial and error whatever you want is relatively trivial.
I agree. Or at least, I don’t see any reason why not.

My point was not that “a relatively simple architecture that contains a Transformer as the core” cannot solve problems via trial and error (in fact I think it’s likely such an architecture exists). My point was that transformers alone cannot do so.

You can call it a “gut claim” if that makes you feel better. But the actual reason is I did some very simple math (about the window size required and given quadratic scaling for transformers) and concluded that practically speaking it was impossible.
Also, importantly, we don’t know what that “relatively simple” architecture looks like. If you look at the various efforts to “extend” transformers to general learning machines, there are a bunch of different approaches: alpha-geometry, diffusion transformers, baby-agi, voyager, dreamer, chain-of-thought, RAG, continuous fine-tuning, V-JEPA. Practically speaking, we have no idea which of these techniques is the “correct” one (if any of them are).
In my opinion saying “Transformers are AGI” is a bit like saying “Deep learning is AGI”. While it is extremely possible that an architecture that heavily relies on Transformers and is AGI exists, we don’t actually know what that architecture is.
Personally, my bet is either on a sort of generalized alpha-geometry approach (where the transformer generates hypothesis and then GOFAI is used to evaluate them) or Diffusion Transformers (where we iteratively de-noise a solution to a problem). But I wouldn’t be at all surprised if a few years from now it is universally agreed that some key insight we’re currently missing marks the dividing line between Transformers and AGI.

Logan Zoellner 30 Mar 2024 1:10 UTC
2 points
0
in reply to: ReaderM’s comment on: Modern Transformers are AGI, and Human-Level
Ok? That’s how you teach anybody anything.
Have you never figured out something by yourself? The way I learned to do Sudoku was: I was given a book of Sudoku puzzles and told “have fun”.
you said it would be impossible to train a chess playing model this century.
I didn’t say it was impossible to train an LLM to play Chess. I said it was impossible for an LLM to teach itself to play a game of similar difficulty to chess if that game is not in it’s training data.
These are two wildly different things.
Obviously LLMs can learn things that are in their training data. That’s what they do. Obviously if you give LLMs detailed step-by-step instructions for a procedure that is small enough to fit in its attention window, LLMs can follow that procedure. Again, that is what LLMs do.
What they do not do is teach themselves things that aren’t in their training data via trial-and-error. Which is the primary way humans learn things.

Logan Zoellner 29 Mar 2024 20:44 UTC
2 points
0
in reply to: ReaderM’s comment on: Modern Transformers are AGI, and Human-Level
sure. 4000 words (~8000 tokens) to do a 9-state 9-turn game with the entire strategy written out by a human. Now extrapolate that to chess, go, or any serious game.

And this doesn’t address at all my actual point, which is that Transformers cannot teach themselves to play a game.

Logan Zoellner 28 Mar 2024 14:22 UTC
2 points
−2
AF
in reply to: AnthonyC’s comment on: Modern Transformers are AGI, and Human-Level
Absolutely. I don’t think it’s impossible to build such a system. In fact, I think a transformer is probably about 90% there. Need to add trial and error, some kind of long-term memory/fine-tuning and a handful of default heuristics. Scale will help too, but no amount of scale alone will get us there.

Logan Zoellner 27 Mar 2024 20:45 UTC
LW: 2 AF: 1
−2
AF
in reply to: Matt Goldenberg’s comment on: Modern Transformers are AGI, and Human-Level
It certainly wouldn’t generalize to e.g Hidouku

Logan Zoellner 27 Mar 2024 17:51 UTC
LW: 4 AF: 3
0
AF
in reply to: Matt Goldenberg’s comment on: Modern Transformers are AGI, and Human-Level
In the technical sense that you can implement arbitrary programs by prompting an LLM (they are turning complete), sure.
In a practical sense, no.
GPT-4 can’t even play tic-tac-toe. Manifold spent a year getting GPT-4 to implement (much less discover) the algorithm for Sudoku and failed.

Now imagine trying to implement a serious backtracking algorithm. Stockfish checks millions of positions per turn of play. The attention window for your “backtracking transformer” is going to have to be at lease {size of chess board state}*{number of positions evaluated}.

And because of quadratic attention, training it is going to take on the order of {number or parameters}*({chess board state size}*{number of positions evaluated})^2

Even with very generous assumptions for {number of parameters} and {chess board state}, there’s simply no way we could train such a model this century (and that’s assuming Moore’s law somehow continues that long).

Logan Zoellner 27 Mar 2024 13:21 UTC
LW: -2 AF: -2
0
AF
on: Modern Transformers are AGI, and Human-Level
Obvious bait is obvious bait, but here goes.
Transformers are not AGI because they will never be able to “figure something out” the way humans can.
If a human is given the rules for Sudoku, they first try filling in the square randomly. After a while, they notice that certain things work and certain things don’t work. They begin to define heuristics for things that work (for example, if all but one number appears in the same row or column as a box, that number goes in the box). Eventually they work out a complete algorithm for solving Sudoku.
A transformer will never do this (pretending Sudoku wasn’t in its training data). Because they are next-token predictors, they are fundamentally incapable of reasoning about things not in their training set. They are incapable of “noticing when they made a mistake” and then backtracking they way a human would.
Now it’s entirely possible that a very small wrapper around a Transformer could solve Sudoku. You could have the transformer suggest moves and then add a reasoning/planning layer around it to handle the back-tracking. This is effectively what Alpha-Geometry does.
But a Transformer BY ITSELF will never be AGI.

Logan Zoellner 25 Mar 2024 15:28 UTC
2 points
−4
in reply to: Tapatakt’s comment on: All About Concave and Convex Agents
Lots of people think that. Occasionally because they are in a situation where the assumptions of Kelly don’t hold (for example, you are not playing a repeated game), but more often because they confuse single bets (where Kelly maximizes log dollars) with repeated play (where Kelly maximizes total dollars).

In the long run, Kelly is optimal so long as your utility function is monotonic in dollars since it maximizes your total dollars.
From wikipedia

Logan Zoellner 25 Mar 2024 13:49 UTC
2 points
0
on: All About Concave and Convex Agents
No matter what the reward function, isn’t a rational maximizer engaging in repeated play going to “end up” concave due to e.g Kelly Criterion? I would think convex agents would be sub-optimal/quickly self-destruct if you tried to create one and nature will basically never do so.

Logan Zoellner 25 Mar 2024 13:40 UTC
−3 points
−6
on: On the Confusion between Inner and Outer Misalignment
The confusion arises because the terms Inner/Outer alignment were invented by people doing philosophy and not people doing actual machine learning. In practice inner misalignment is basically not a thing. See, for example, Nora Belrose’s comments about the “counting argument”.

Logan Zoellner 25 Mar 2024 13:34 UTC
5 points
2
on: “Deep Learning” Is Function Approximation
but for sufficiently large function approximators, the trend reverses
Transformers/deep learning work because of built-in regularization methods (like dropout layers) and not because “the trend reverses”. If you did naive “best fit polynomial” with a 7 billion parameter polynomial you would not get a good result.

Logan Zoellner 21 Mar 2024 15:24 UTC
9 points
5
on: AI #56: Blackwell That Ends Well
I do not know who is right about Mamba here and it doesn’t seem easy to check?
Nora is definitely right in this case (Mamba isn’t doing dramatically more serial operations than Transformers). If it was, it would be slower running in not-batch-mode, whereas the whole point of Mamba is it’s faster (due to sub-quadratic attention).