Logan Zoellner

Karma: 1,126

Logan Zoellner 24 Jun 2023 18:10 UTC
25 points
4
on: Why Not Subagents?
Can we crank this in reverse: given a utility function, design a market that whose representative agent has this utility function?
It seems like trivially, we could just have the market where a singleton agent has the desired utility function. But I wonder if there’s some procedure where we can “peel off” sub-agents within a market and end up at a market composed of the simplest possible sub-agents for some metric of complexity.
Either there is some irreducible complexity there, or maybe there is a Universality theorem proving that we can express any utility function using a market of agents who all have some extremely simple finite state, similar to how we can show any form of computation can be expressed using Turing Machines.

Logan Zoellner 3 Jun 2023 5:30 UTC
25 points
18
on: The AGI Race Between the US and China Doesn’t Exist.
China does not have access to the computational resources^[1] (compute, here specifically data centre-grade GPUs) needed for large-scale training runs of large language models.
While it’s true that Chinese semiconductor fabs are a decade behind TSMC (and will probably remain so for some time), that doesn’t seem to have stopped them from building 162 of the top 500 largest supercomputers in the world.
There are two inputs to building a large supercomputer: quality and quantity, and China seems more than willing to make up in quantity what they lack in quality.
The CCP is not interested in reaching AGI by scaling LLMs.
For a country that is “not interested” in scaling LLMs, they sure do seem to do a lot of research into large language models.
It’s also worth noting that China currently has the best open-source text-to-video model, has trained a state of the art text-to-image model, was the first to introduce AI in a mass consumer product, and is miles ahead of the west in terms of facial recognition.
I suspect that “China is not racing for AGI” will end up in the same historical bin as “Russia has no plans to invade Ukraine”, a claim that requires us to believe the Chinese stated preferences while completely ignoring their revealed ones.
I do agree that if the US and China were both racing, the US would handily win the race given current conditions. But if the US stops racing, there’s absolutely no reason to think the Chinese response would be anything other than “thanks, we’ll go ahead without you”.
--edit--
If a Chinese developer ever releases an LLM that is so powerful it inevitably oversteps censorship rules at some point, the Chinese government will block it and crack down on the company that released it.
This is a bit of a weird take to have if you are worried about AGI Doom. If your belief is “people will inevitably notice that powerful systems are misaligned and refuse to deploy them”, why are you worried about Doom in the first place?
Is the claim that China due to its all-powerful bureaucracy is somehow less prone to alignment failure and hard-left-turns than reckless American corporations? If so, I suggest you consider the possibility that Xi Jinping isn’t all that smart.

Logan Zoellner 25 Nov 2023 15:16 UTC
22 points
13
on: Ability to solve long-horizon tasks correlates with wanting things in the behaviorist sense
Well, I claim that these are more-or-less the same fact. It’s no surprise that the AI falls down on various long-horizon tasks and that it doesn’t seem all that well-modeled as having “wants/desires”; these are two sides of the same coin.
It’s weird that this sentence immediately follows you talking about AI being able to play chess. A chess playing AI doesn’t “want to win” in the behaviorist sense. If I flip over the board or swap pieces mid game or simply refuse to move the AI’s pieces on it’s turn, it’s not going to do anything to stop me because it doesn’t “want” to win the game. It doesn’t even realize that a game is happening in the real world. And yet it is able to make excellent long term plans about “how” to win at chess.

Either:
a) A chess playing AI fits into your definition of “want”, in which case who cares if AI wants things, this tells us nothing about their real-world behavior.
b) A chess playing AI doesn’t “want” to win (my claim) in which case AI can make long term plans without wanting.

Logan Zoellner 21 Jun 2022 18:14 UTC
22 points
1
on: The inordinately slow spread of good AGI conversations in ML
The claim that AI capabilities research is bad for Alignment is non-obvious.
In particular, if you think of AI-Alignment as a race between knowledge and hardware (more knowledge makes aligning AI easier, more hardware makes building dangerous AI easier), then AI research (that isn’t purely building faster hardware) is net-positive.

Logan Zoellner 28 May 2023 11:01 UTC
20 points
11
on: Hands-On Experience Is Not Magic
- As I’d mentioned, we often apply non-LPE-based environment-solving to constrain the space of heuristics over which we search, as in the tic-tac-toe and math examples. Indeed, it seems that scientific research would be impossible without that.
- LPE-based learning does not work in domains where failure is lethal, by definition. However, we have some success navigating them anyway.
I think this is a strawman of LPE. People who point out you need real world experience don’t say that you need 0 theory, but that you have to have some contact with reality, even in deadly domains.
Outside of a handful of domains like computer science and pure mathematics, contact with reality is necessary because the laws of physics dictate that we can only know things up to a limited precision. Moreover, it is the experience of experts in a wide variety of domains that “try the thing out and see what happens” is a ridiculously effective heuristic.
Even in mathematics, the one domain where LPE should in principal be unnecessary, trying things out is one of the main ways that mathematicians gain intuitions for what new results are/aren’t likely to hold.
I also note that your post doesn’t give a single example of a major engineering/technology breakthrough that was done without LPE (in a domain that interacts with physical reality).
It is, in fact, possible to make strong predictions about OOD events like AGI Ruin — if you’ve studied the problem exhaustively enough to infer its structure despite lacking the hands-on experience.
This is literally the one specific thing LPE advocates think you need to learn from experience about, and you’re just asserting it as true?
To summarize:
Domains where “pure thought” is enough:
- toy problems
- limited/no interaction with the real world
- solution/class of solutions known in advance
Domains where LPE is necessary:
- too complicated/messy to simulate
- depends on precise physical details of the problem
- even a poor approximation to solution not knowable in advance

Logan Zoellner 6 Jun 2022 22:40 UTC
20 points
−13
on: AGI Ruin: A List of Lethalities
Lots I disagree with here, so let’s go through the list.
There are no pivotal weak acts.
Strong disagree.
EY and I don’t seem to agree that “nuke every semiconductor fab” is a weakly pivotal act (since I think AI is hardware-limited and he thinks it is awaiting a clever algorithm). But I think even “build nanobots that melt every GPU” could be built using an AI that is aligned in the “less than 50% chance of murdering us all” sense. For example, we could simulate a bunch of human-level scientists trying to build nanobots and also checking each-other’s work.
On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions.
Nope. I think that you could build a useful AI (e.g. the hive of scientists) without doing any out-of-distribution stuff.
there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment
I am significantly more optimistic about explainable AI than EY.
There is no analogous truth about there being a simple core of alignment
I do not consider this at all obvious.
Corrigibility is anti-natural to consequentialist reasoning
Roll to disbelief. Cooperation is a natural equilibrium in many games.
you can’t rely on behavioral inspection to determine facts about an AI which that AI might want to deceive you about
Sure you can. Just train an AI that “wants” to be honest. This probably means training an AI with the objective function “accurately predict reality” and then using it to do other things (like make paperclips) rather than training it with an objective function “make paperclips”.
Coordination schemes between superintelligences are not things that humans can participate in
I don’t think this is as relevant as EY does. Even if it’s true that unaugmented humans are basically irrelevant to an economy of superintelligent AIs, that doesn’t mean we can’t have a future where augmented or tool-AI assisted humans can have meaningful influence.
Any system of sufficiently intelligent agents can probably behave as a single agent, even if you imagine you’re playing them against each other
I believe there is an intermediate level of AI between “utterly useless” and “immediately solves the acausal trading problem and begins coordinating perfectly against humans”. This window may be rather wide.
What makes an air conditioner ‘magic’ from the perspective of say the thirteenth century, is that even if you correctly show them the design of the air conditioner in advance, they won’t be able to understand from seeing that design why the air comes out cold
I’m virtually certain I could explain to Aristotle or DaVinci how an air-conditioner works.
There’s a pattern that’s played out quite often, over all the times the Earth has spun around the Sun, in which some bright-eyed young scientist, young engineer, young entrepreneur, proceeds in full bright-eyed optimism to challenge some problem that turns out to be really quite difficult. Very often the cynical old veterans of the field try to warn them about this, and the bright-eyed youngsters don’t listen, because, like, who wants to hear about all that stuff, they want to go solve the problem!
There’s also a pattern where the venerable scientist is proven wrong by the young scientist too foolish to know what they are doing is impossible.
There’s no plan.
There is at least one plan.
This situation you see when you look around you is not what a surviving world looks like
Currently Metaculus estimates 55% chance for “Will there be a positive transition to a world with radically smarter-than-human artificial intelligence?”. Admitted I would like this to be higher, but at the minimum this is what a world that “might survive” looks like. I have no particular reason to trust EY vs Metaculus.
I suspect EY and I both agree that if you take existing Reinforcement Learning Architectures, write down the best utility function humans can think of, and then turn the dial up to 11, bad things will happen. EY seems to believe this is a huge problem because of his belief that “there is no weak pivotal act”. I think this should be taken as a strong warning to not do that. Rather than scaling architectures that are inherently dangerous, we should focus on making use of architectures that are naturally safe. For example, EY and I both agree that GPT-N is likely to be safe. EY simply disagrees with the claim that it might be useful.
EY and I probably also agree that Facebook/Baidu do not have the world’s best interest at heart (and are not taking alignment seriously enough or at all). Hence it is important that people who care about Alignment gain a decisive lead over these efforts. To me, this logically means that people interested in Alignment should be doing more capabilities research. To EY, this means that alignment focused institutions need to be using more secrecy. I’m not utterly opposed to keeping pure-capabilities advancements secret, but if there is a significant overlap between capabilities and alignment, then we need to be publishing the alignment-relevant bits so that we can cooperate (and hopefully so that Facebook can incorporate them too).
And for completeness, here’s a bunch of specific claims by EY I agree with
AGI will not be upper-bounded by human ability or human learning speed. Things much smarter than human would be able to learn from less evidence than humans require
I think the people who thought this stopped thinking this after move 37. I hope.
A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure
Strongly agree.
Losing a conflict with a high-powered cognitive system looks at least as deadly as “everybody on the face of the Earth suddenly falls over dead within the same second”
Strongly agree.
We need to get alignment right on the ‘first critical try’
Strongly agree.
We can’t just “decide not to build AGI”
Strongly agree
Running AGIs doing something pivotal are not passively safe
Agree. But I don’t think this means they are totally unworkable either.
Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/training operations that didn’t kill you
Agree. But I think the lesson here is “don’t use powerful AIs until you are sure they are aligned”.
Operating at a highly intelligent level is a drastic shift in distribution from operating at a less intelligent level
Agree somewhat. But I don’t rule out that “cooperative” or “interesting” is a natural attractor.
Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously
Agree, conditional on our definition of fast. I think that within a year of training our first “smart human” AI, we can simulate “100 smart humans” using a similar compute budget. I don’t think Foom takes us from “human level AI” to “smarter than all humans AI” in a few minutes simply be rewriting code.
outer optimization even on a very exact, very simple loss function doesn’t produce inner optimization in that direction
Agree. This is why I am skeptical of utility-functions in general as a method for aligning AI.
Human raters make systematic errors—regular, compactly describable, predictable errors
Duh.
The first thing generally, or CEV specifically, is unworkable because the complexity of what needs to be aligned or meta-aligned for our Real Actual Values is far out of reach for our FIRST TRY at AGI.
I am really not very optimistic about CEV.
A powerful AI searches parts of the option space we don’t, and we can’t foresee all its options.
Yes.
This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents
Agree. But I don’t think you need to make an AI that imitates humans in order to make an AI that is useful. For example, Codex allows me to write code significantly (2-5x) faster, despite frequently making dumb mistakes.
The AI does not think like you do
Yes.
AI-boxing can only work on relatively weak AGIs; the human operators are not secure systems.
Mostly agree. I think there exist architectures of AI that can be boxed.
You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them.
I think the best approach to funding AI safety is something like Fast Grants where we focus more on quantity than on “quality” since it is nearly impossible to identify who will succeed in advance.
What links here?
- Anand Baburajan's comment on Open Thread Spring 2024 by habryka (14 Mar 2024 15:30 UTC; 5 points)

Logan Zoellner 10 Jan 2023 16:34 UTC
19 points
8
on: AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years
I feel like the correct take here is EMH false because market’s don’t really account for black swans well.
I agree that there’s money to be made here under certain circumstances, but also keep in mind that the best companies are not publicly traded. In a non-catastrophic scenario you can probably get decent exposure via public tech companies, but in a catastrophic scenario the winning strategy is to spend as much as possible before the world ends.
Suppose you have something like 50% catastrophic AI and 50% nothing happens for 2 decades, though, I’m not really sure what you should do. If nothing happens and you spend all your money you spend the rest of your life penniless. If the world does end, does buying that 2nd house really get you all that much utility? Most of the things I value (and haven’t already bought) are of the form “living a long and fulfilling life” or “seeing my grandchildren” which are impossible to invest for in the ⁵⁰⁄₅₀ scenario.

Logan Zoellner 26 Apr 2024 1:21 UTC
15 points
11
on: Losing Faith In Contrarianism
Sam Atis—a super forecaster—had a piece arguing against The Case Against Education
If it’s this piece, I would be interested to know why you found it convincing. He doesn’t address (or seem to have even read) any of Brian’s arguments. His argument basically boils down to “but so many people who work for universities think it’s good”.

Logan Zoellner 30 May 2022 18:25 UTC
15 points
−14
on: Will working here advance AGI? Help us not destroy the world!
I suspect my advice is the exact opposite of they Less Wrong/EY consensus, so here goes:
Choose to work at whatever company will allow you personally to get as good at AI/Machine learning as possible.
This is a restatement of my advice at the end of my essay on AI Alignment. Specifically, the two strategies I am the most optimistic about, Game Theory and The Plan both depend on very smart people becoming as wise as possible before the Singularity comes.
From a game-theory point of view, advancing AI knowledge in general is a tragedy of the commons. It would require coordination from everyone all at once in order to stop advancing AI beyond the danger level (whatever that might be). And it isn’t even possible to know if a particular field (compilers, formal mathematical methods, hardware improvement, AI art) will be the one that puts us over the top. That means there is very little benefit for you not to work on advancing AI (and it comes at a huge cost, since you basically have to give up on any career even tangentially related to technology).
On the other hand AI Alignment is likely to be solved by a “small group of thoughtful individuals”. Increasing your skills proportionally increases your chance of being a member of this category (since it seems like you already care about the topic).
One way to think about this advice is: every day Google, Open AI, Hugging Face, and 1000 other companies are hiring someone and that someone will likely work to advance AI. If we imagine the marginal case where a company is deciding between hiring you and someone slightly less concerned about AI alignment. Wouldn’t you rather they hire you?
Note that this advice does not mean you get to leave your ethics at the door. Quite the opposite, if you are working somewhere and it turns out they are doing something egregious stupid (like deploying a non-airgapped AI), it is your duty to do everything in your power to stop them. Complain to your boss, leak information to the press, chain yourself to the server. Whatever you do, do not become the engineer who warned about disaster but then quietly shrugged when pressured by management. But if you refuse to take any jobs related to AI, you won’t even be in the room when the disaster is about to happen. And on the margin, you should assume that somebody worse will be.
What links here?
- Vaniver's comment on Will working here advance AGI? Help us not destroy the world! by Yonatan Cale (30 May 2022 19:42 UTC; 4 points)

Logan Zoellner 14 Sep 2023 14:04 UTC
14 points
1
on: AI #29: Take a Deep Breath
When will instrumental convergence become a visible big problem?

If anyone knows of a real-world example of Instrumental Convergence, I will pay you $100

Logan Zoellner 22 Sep 2023 13:24 UTC
13 points
1
on: AI #30: Dalle-3 and GPT-3.5-Instruct-Turbo
I continue to be deeply, deeply skeptical of the whole ‘we need exactly the best possible frontier models to do good alignment work’ argument, that whatever the current best option is must be both necessary and sufficient.

I know asking other people to do work for you is generally, unfair, but would it be too much to ask for you to include a section along the lines of “AI alignment breakthroughs this week”?

There’s a new innovation at least as significant as this or this almost every week, and it would be a bit heartening to see that you were actually tracking progress towards solving the alignment problem instead of just focusing on pausing AI progress.

If your claim is that we can make progress on alignment without working on cutting-edge models, then you should actually be tracking/encouraging such research.

Logan Zoellner 12 Aug 2021 19:28 UTC
13 points
on: Obesity Epidemic Explained in 0.9 Subway Cookies
There’s pretty strong evidence that the rise in obesity is not primarily behavioral. For example, here. If obesity is the result of a collective depletion of willpower, than why is obesity increasing in animals? Why does it depend on elevation?
Furthermore, the curves that you present are way too steep. The average American puts on about 1 pound/year, which suggests that our bodies are more or less in homeostasis. The question is, what long-term trend is pushing this homeostasis upwards?

Logan Zoellner 17 Nov 2020 2:02 UTC
13 points
on: Propinquity Cities So Far
Don’t basically all cities control density pretty tightly? I know that a lot of density restriction is just nimbies defending housing scarcity, but it can’t all be that, can it?
I think this is where the whole post goes off the rails.
In the real world there are massiveeconomic inefficiencies created by government restrictions on density. Suggesting that we can fix these with a more complex government system is like suggesting we can solve the “wolves eat sheep” problem with bigger wolves.

Logan Zoellner 26 Sep 2021 21:00 UTC
12 points
on: Schools probably do do something
Wouldn’t this actually be evidence against school doing something? I.e. we have discovered that relative age is more important than years of education?

Logan Zoellner 15 Nov 2021 22:16 UTC
11 points
in reply to: Eliezer Yudkowsky’s comment on: Ngo and Yudkowsky on alignment difficulty
If I really thought AI was going to murder us all in the next 6 months to 2 years, I would definitely consider those 10 years “pivotal”, since it would give us 5x-20x the time to solve the alignment problem. I might even go full Butlerian Jihad and just ban semiconductor fabs altogether.
Actually, I think that right question, is: is there anything you would consider pivotal other that just solving the alignment problem? If no, the whole argument seems to be “If we can’t find a safe way to solve the alignment problem, we should consider dangerous ones.”

Logan Zoellner 19 Jul 2023 14:12 UTC
10 points
3
on: The Full Alignment Plan You’ve Never Heard Of
Modern AI systems (read: LLMs) don’t look like that, so Tammy thinks efforts to align them are misguided.
Starting your plan with “ignore all SOTA ML research” doesn’t sound like a winning proposition.
The important thing is, your math should be safe regardless of what human values turn out to really be, but you still need lots of info to pin those values down.
I don’t think even hardcore believers in the orthogonality thesis believe all possible sets of values are equally easy to satisfy. Starting with a value-free model is throwing away a lot of baby with the bath-water.
Moreover, explicitly having a plan of “we can satisfy any value system” leaves you wide open for abuse (and goodharting). Much better to aim from the start for a broadly socially agreeable goal like curiosity, love or freedom.
Look at every possible computer program.
I’m sure the author is aware that Solomonoff Induction is non-computable, but this article still fails to appreciate how impossible it is to even approximate. We could build a galaxy sized super-computer and still wouldn’t be able to calculate BB(7)
Overall, I would rate the probability of this plan succeeding at 0%
1. Ignores the fact that there’s no such thing in the real world as a “strong coherent general agent”. Normally I scoff at people who say things like “the word intelligence isn’t defined” but this is a case where the distinction between “can do anything a human can do” and “general purpose intelligence” really matters.
2. Value-free alignment is simultaneously impossible and dangerous
3. Refuses to look “inside the box” thereby throwing away all practical tools for alignment
4. Completely ignores everything we’ve actually learned about AI in the last 4 years
5. Requires calculation of an impossible to calculate function
6. Generally falls into the MIRI/rationalist tarpit of trying to reason about intelligent systems without actually working with/appreciating actually working systems in the real word

Logan Zoellner 24 Jun 2023 17:53 UTC
10 points
7
on: Catastrophic Risks from AI #3: AI Race
At this point, humans have few incentives to gain any skills or knowledge, because almost everything would be taken care of by much more capable AIs.
This premise makes no sense. Humans have been using technology to augment their abilities since literally as long as there have been humans. Why, when AI appears, would this suddenly cease to be the case? This is like arguing that books will make humans dumber because we won’t memorize things anymore.

Logan Zoellner 17 Nov 2021 3:03 UTC
10 points
on: The Greedy Doctor Problem
I’m surprised you didn’t mention financial solutions. E.g. “write a contract that pays the doctor more for every year that I live”. Although I suppose this might still be vulnerable to goodharting. For example the doctor may keep me “alive” indefinitely in a medical coma.

Logan Zoellner 21 Mar 2024 15:24 UTC
9 points
5
on: AI #56: Blackwell That Ends Well
I do not know who is right about Mamba here and it doesn’t seem easy to check?
Nora is definitely right in this case (Mamba isn’t doing dramatically more serial operations than Transformers). If it was, it would be slower running in not-batch-mode, whereas the whole point of Mamba is it’s faster (due to sub-quadratic attention).

Logan Zoellner 15 Sep 2023 13:02 UTC
9 points
6
in reply to: AnthonyC’s comment on: AI #29: Take a Deep Breath
As Zvi points out, what counts as “new information” depends on the person reading the message
Taking all the knowledge and writing and tendencies of the entire human race and all properties of the physical universe as a given, sure, this is correct. The response corresponds to the prompt, all the uniqueness has to be there.
Presumably when you look something up in an encyclopedia, it is because there is something that other people (the authors of the encyclopedia) know that you don’t know. When you write an essay, the situation should be exactly opposite: there is something you know that the reader of the essay doesn’t know. In this case the possibilities are:
1. The information I am trying to convey in the essay is strictly contained in the prompt (in which case GPT is just adding extra unnecessary verbiage)
2. GPT is adding information known to me but unknown to the reader of the essay (say I am writing a recipe for baking cookies and steps 1-9 are the normal way of making cookies but step 10 is something novel like “add cinnamon”)
3. GPT is hallucinating facts unknown to the writer of the essay (theoretically these could still be true facts, but the writer would have to verify this somehow)
Nassim Taleb’s point is that cases 1. and 3. are bad. Zvi’s point is that case 2. is frequently quite useful.