Logan Zoellner 24 Jun 2023 18:10 UTC
25 points
4
on: Why Not Subagents?
Can we crank this in reverse: given a utility function, design a market that whose representative agent has this utility function?
It seems like trivially, we could just have the market where a singleton agent has the desired utility function. But I wonder if there’s some procedure where we can “peel off” sub-agents within a market and end up at a market composed of the simplest possible sub-agents for some metric of complexity.
Either there is some irreducible complexity there, or maybe there is a Universality theorem proving that we can express any utility function using a market of agents who all have some extremely simple finite state, similar to how we can show any form of computation can be expressed using Turing Machines.

Logan Zoellner 3 Jun 2023 5:30 UTC
25 points
18
on: The AGI Race Between the US and China Doesn’t Exist.
China does not have access to the computational resources^[1] (compute, here specifically data centre-grade GPUs) needed for large-scale training runs of large language models.
While it’s true that Chinese semiconductor fabs are a decade behind TSMC (and will probably remain so for some time), that doesn’t seem to have stopped them from building 162 of the top 500 largest supercomputers in the world.
There are two inputs to building a large supercomputer: quality and quantity, and China seems more than willing to make up in quantity what they lack in quality.
The CCP is not interested in reaching AGI by scaling LLMs.
For a country that is “not interested” in scaling LLMs, they sure do seem to do a lot of research into large language models.
It’s also worth noting that China currently has the best open-source text-to-video model, has trained a state of the art text-to-image model, was the first to introduce AI in a mass consumer product, and is miles ahead of the west in terms of facial recognition.
I suspect that “China is not racing for AGI” will end up in the same historical bin as “Russia has no plans to invade Ukraine”, a claim that requires us to believe the Chinese stated preferences while completely ignoring their revealed ones.
I do agree that if the US and China were both racing, the US would handily win the race given current conditions. But if the US stops racing, there’s absolutely no reason to think the Chinese response would be anything other than “thanks, we’ll go ahead without you”.
--edit--
If a Chinese developer ever releases an LLM that is so powerful it inevitably oversteps censorship rules at some point, the Chinese government will block it and crack down on the company that released it.
This is a bit of a weird take to have if you are worried about AGI Doom. If your belief is “people will inevitably notice that powerful systems are misaligned and refuse to deploy them”, why are you worried about Doom in the first place?
Is the claim that China due to its all-powerful bureaucracy is somehow less prone to alignment failure and hard-left-turns than reckless American corporations? If so, I suggest you consider the possibility that Xi Jinping isn’t all that smart.

2023)

Logan Zoellner7 Nov 2023 13:26 UTC

24 points

0 comments4 min readLW link

(midwitalignment.substack.com)

AI Alignment [Incremental Progress Units] this Week (10/22/23)

Logan Zoellner23 Oct 2023 20:32 UTC

22 points

0 comments6 min readLW link

(midwitalignment.substack.com)

The Walking Dead

Logan Zoellner22 Jul 2021 16:19 UTC

22 points

2 comments1 min readLW link

Logan Zoellner 25 Nov 2023 15:16 UTC
22 points
13
on: Ability to solve long-horizon tasks correlates with wanting things in the behaviorist sense
Well, I claim that these are more-or-less the same fact. It’s no surprise that the AI falls down on various long-horizon tasks and that it doesn’t seem all that well-modeled as having “wants/desires”; these are two sides of the same coin.
It’s weird that this sentence immediately follows you talking about AI being able to play chess. A chess playing AI doesn’t “want to win” in the behaviorist sense. If I flip over the board or swap pieces mid game or simply refuse to move the AI’s pieces on it’s turn, it’s not going to do anything to stop me because it doesn’t “want” to win the game. It doesn’t even realize that a game is happening in the real world. And yet it is able to make excellent long term plans about “how” to win at chess.

Either:
a) A chess playing AI fits into your definition of “want”, in which case who cares if AI wants things, this tells us nothing about their real-world behavior.
b) A chess playing AI doesn’t “want” to win (my claim) in which case AI can make long term plans without wanting.

Logan Zoellner 21 Jun 2022 18:14 UTC
22 points
1
on: The inordinately slow spread of good AGI conversations in ML
The claim that AI capabilities research is bad for Alignment is non-obvious.
In particular, if you think of AI-Alignment as a race between knowledge and hardware (more knowledge makes aligning AI easier, more hardware makes building dangerous AI easier), then AI research (that isn’t purely building faster hardware) is net-positive.

An embedding decoder model, trained with a different objective on a different dataset, can decode another model’s embeddings surprisingly accurately

Logan Zoellner3 Sep 2023 11:34 UTC

20 points

1 comment1 min readLW link

Logan Zoellner 28 May 2023 11:01 UTC
20 points
11
on: Hands-On Experience Is Not Magic
- As I’d mentioned, we often apply non-LPE-based environment-solving to constrain the space of heuristics over which we search, as in the tic-tac-toe and math examples. Indeed, it seems that scientific research would be impossible without that.
- LPE-based learning does not work in domains where failure is lethal, by definition. However, we have some success navigating them anyway.
I think this is a strawman of LPE. People who point out you need real world experience don’t say that you need 0 theory, but that you have to have some contact with reality, even in deadly domains.
Outside of a handful of domains like computer science and pure mathematics, contact with reality is necessary because the laws of physics dictate that we can only know things up to a limited precision. Moreover, it is the experience of experts in a wide variety of domains that “try the thing out and see what happens” is a ridiculously effective heuristic.
Even in mathematics, the one domain where LPE should in principal be unnecessary, trying things out is one of the main ways that mathematicians gain intuitions for what new results are/aren’t likely to hold.
I also note that your post doesn’t give a single example of a major engineering/technology breakthrough that was done without LPE (in a domain that interacts with physical reality).
It is, in fact, possible to make strong predictions about OOD events like AGI Ruin — if you’ve studied the problem exhaustively enough to infer its structure despite lacking the hands-on experience.
This is literally the one specific thing LPE advocates think you need to learn from experience about, and you’re just asserting it as true?
To summarize:
Domains where “pure thought” is enough:
- toy problems
- limited/no interaction with the real world
- solution/class of solutions known in advance
Domains where LPE is necessary:
- too complicated/messy to simulate
- depends on precise physical details of the problem
- even a poor approximation to solution not knowable in advance

Logan Zoellner 6 Jun 2022 22:40 UTC
20 points
−13
on: AGI Ruin: A List of Lethalities
Lots I disagree with here, so let’s go through the list.
There are no pivotal weak acts.
Strong disagree.
EY and I don’t seem to agree that “nuke every semiconductor fab” is a weakly pivotal act (since I think AI is hardware-limited and he thinks it is awaiting a clever algorithm). But I think even “build nanobots that melt every GPU” could be built using an AI that is aligned in the “less than 50% chance of murdering us all” sense. For example, we could simulate a bunch of human-level scientists trying to build nanobots and also checking each-other’s work.
On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions.
Nope. I think that you could build a useful AI (e.g. the hive of scientists) without doing any out-of-distribution stuff.
there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment
I am significantly more optimistic about explainable AI than EY.
There is no analogous truth about there being a simple core of alignment
I do not consider this at all obvious.
Corrigibility is anti-natural to consequentialist reasoning
Roll to disbelief. Cooperation is a natural equilibrium in many games.
you can’t rely on behavioral inspection to determine facts about an AI which that AI might want to deceive you about
Sure you can. Just train an AI that “wants” to be honest. This probably means training an AI with the objective function “accurately predict reality” and then using it to do other things (like make paperclips) rather than training it with an objective function “make paperclips”.
Coordination schemes between superintelligences are not things that humans can participate in
I don’t think this is as relevant as EY does. Even if it’s true that unaugmented humans are basically irrelevant to an economy of superintelligent AIs, that doesn’t mean we can’t have a future where augmented or tool-AI assisted humans can have meaningful influence.
Any system of sufficiently intelligent agents can probably behave as a single agent, even if you imagine you’re playing them against each other
I believe there is an intermediate level of AI between “utterly useless” and “immediately solves the acausal trading problem and begins coordinating perfectly against humans”. This window may be rather wide.
What makes an air conditioner ‘magic’ from the perspective of say the thirteenth century, is that even if you correctly show them the design of the air conditioner in advance, they won’t be able to understand from seeing that design why the air comes out cold
I’m virtually certain I could explain to Aristotle or DaVinci how an air-conditioner works.
There’s a pattern that’s played out quite often, over all the times the Earth has spun around the Sun, in which some bright-eyed young scientist, young engineer, young entrepreneur, proceeds in full bright-eyed optimism to challenge some problem that turns out to be really quite difficult. Very often the cynical old veterans of the field try to warn them about this, and the bright-eyed youngsters don’t listen, because, like, who wants to hear about all that stuff, they want to go solve the problem!
There’s also a pattern where the venerable scientist is proven wrong by the young scientist too foolish to know what they are doing is impossible.
There’s no plan.
There is at least one plan.
This situation you see when you look around you is not what a surviving world looks like
Currently Metaculus estimates 55% chance for “Will there be a positive transition to a world with radically smarter-than-human artificial intelligence?”. Admitted I would like this to be higher, but at the minimum this is what a world that “might survive” looks like. I have no particular reason to trust EY vs Metaculus.
I suspect EY and I both agree that if you take existing Reinforcement Learning Architectures, write down the best utility function humans can think of, and then turn the dial up to 11, bad things will happen. EY seems to believe this is a huge problem because of his belief that “there is no weak pivotal act”. I think this should be taken as a strong warning to not do that. Rather than scaling architectures that are inherently dangerous, we should focus on making use of architectures that are naturally safe. For example, EY and I both agree that GPT-N is likely to be safe. EY simply disagrees with the claim that it might be useful.
EY and I probably also agree that Facebook/Baidu do not have the world’s best interest at heart (and are not taking alignment seriously enough or at all). Hence it is important that people who care about Alignment gain a decisive lead over these efforts. To me, this logically means that people interested in Alignment should be doing more capabilities research. To EY, this means that alignment focused institutions need to be using more secrecy. I’m not utterly opposed to keeping pure-capabilities advancements secret, but if there is a significant overlap between capabilities and alignment, then we need to be publishing the alignment-relevant bits so that we can cooperate (and hopefully so that Facebook can incorporate them too).
And for completeness, here’s a bunch of specific claims by EY I agree with
AGI will not be upper-bounded by human ability or human learning speed. Things much smarter than human would be able to learn from less evidence than humans require
I think the people who thought this stopped thinking this after move 37. I hope.
A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure
Strongly agree.
Losing a conflict with a high-powered cognitive system looks at least as deadly as “everybody on the face of the Earth suddenly falls over dead within the same second”
Strongly agree.
We need to get alignment right on the ‘first critical try’
Strongly agree.
We can’t just “decide not to build AGI”
Strongly agree
Running AGIs doing something pivotal are not passively safe
Agree. But I don’t think this means they are totally unworkable either.
Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/training operations that didn’t kill you
Agree. But I think the lesson here is “don’t use powerful AIs until you are sure they are aligned”.
Operating at a highly intelligent level is a drastic shift in distribution from operating at a less intelligent level
Agree somewhat. But I don’t rule out that “cooperative” or “interesting” is a natural attractor.
Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously
Agree, conditional on our definition of fast. I think that within a year of training our first “smart human” AI, we can simulate “100 smart humans” using a similar compute budget. I don’t think Foom takes us from “human level AI” to “smarter than all humans AI” in a few minutes simply be rewriting code.
outer optimization even on a very exact, very simple loss function doesn’t produce inner optimization in that direction
Agree. This is why I am skeptical of utility-functions in general as a method for aligning AI.
Human raters make systematic errors—regular, compactly describable, predictable errors
Duh.
The first thing generally, or CEV specifically, is unworkable because the complexity of what needs to be aligned or meta-aligned for our Real Actual Values is far out of reach for our FIRST TRY at AGI.
I am really not very optimistic about CEV.
A powerful AI searches parts of the option space we don’t, and we can’t foresee all its options.
Yes.
This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents
Agree. But I don’t think you need to make an AI that imitates humans in order to make an AI that is useful. For example, Codex allows me to write code significantly (2-5x) faster, despite frequently making dumb mistakes.
The AI does not think like you do
Yes.
AI-boxing can only work on relatively weak AGIs; the human operators are not secure systems.
Mostly agree. I think there exist architectures of AI that can be boxed.
You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them.
I think the best approach to funding AI safety is something like Fast Grants where we focus more on quantity than on “quality” since it is nearly impossible to identify who will succeed in advance.