Engineer at CoinList.co. Donor to LW 2.0.
ESRogs
Karma: 4,844
Thanks for the answers!
Correct me if I’m wrong, but it struck while reading this that you can think of a neural network as learning two things at once:
a classification of the input into 2^N different classes (where N is the total number of neurons), each of which gets a different function applied to it
those functions themselves
(also each function is a linear transformation)
The 2^N classes would be all the different polytopes, and each function would be the linear transformation that the network implements when a polytope’s neurons are on and all others are off.
To me this suggests some questions:
Can the functions and classes be decoupled?
It seems a little odd that the weights in a given layer are kind of doing double-duty — they’re both helping to implement some particular linear transformation that the whole network performs (transforming the kind of data you feed into the network at the input into the kind of data you expect at the output), and they’re also helping to determine which linear function the rest of the network is going to perform (by tipping downstream neurons’ activations over their thresholds or not).
Is that at all a sensible way to look at it?
Could you come up with some other scheme for choosing between a whole bunch of different linear transformations?
How much of the power of neural networks comes from their ability to learn to classify something into exponentially many different classes vs from the linear transformations that each class implements?
Does this question even make sense?
The 2^N different linear transformations can’t be arbitrary. As mentioned in the post, there’s the constraint that neighboring polytopes implement very similar transformations, because their weight matrices vary by just one row.
What would be a reasonable way to measure this degree of constrained-ness?
This comment has been the result of a layman thinking out loud. Appreciate anyone who can answer my questions or point out any misunderstandings!
I pointed out the OOM error on Twitter (citing this comment), and Dan has updated the post with a correction.
In a forthcoming report I will estimate how might change as increases. The report will enumerate different sources of text-based data (e.g. publicly-accessible internet text, private social media messages, human conversations, etc), and for each data-source the report will estimate the cost-per-token and the total availability of the data.
The analysis may be tricky to do, but I’d be particularly interested in seeing model-generated data included in this list. I suspect that in practice the way model-builders will get around the data limit is by generating (and curating) synthetic data.
(This doesn’t have to involve the model just getting high on its own supply. If you build in an evaluation step before including generated data in the training set, then I’d bet you can effectively do AlphaZero-like IDA. I’m guessing that a lot of the action is going to be in figuring out how to set up the generation + evaluation algorithms.)
For instance, Facebook, Google, and Apple will be advantaged, whereas OpenAI, DeepMind, and EleutherAI will be disadvantaged.
Nitpick: Google owns DeepMind, so it doesn’t seem right to list DM on the disadvantaged side.
What’s a latent/patent in the context of a large language model? “patent” is ungoogleable if you’re not talking about intellectual property law.
My money’s on: typo.
Can you get anywhere with synthetic data? What happens if you train a model on its own output?
Glad you liked it! I think the ideas are very interesting too, for I think similar reasons to you.
Will be curious to see how much further they go.
As a person who worked on Arbitral, I agree with this.
To put it another way, I would agree that Eliezer has made (what seem to me like) world-historically-significant contributions to understanding and advocating for (against) AI risk.
So, if 2007 Eliezer was asking himself, “Why am I the only one really looking into this?”, I think that’s a very reasonable question.
But here in 2022, I just don’t see this particular post as that significant of a contribution compared to what’s already out there.
Imagine that this is v0 of a series of documents that need to evolve into humanity’s (/ some specific group’s) actual business plan for saving the world.
Why is this v0 and not https://arbital.com/explore/ai_alignment/, or the Sequences, or any of the documents that Evan links to here?
That’s part of what I meant to be responding to — not that this post is not useful, but that I don’t see what makes it so special compared to all the other stuff that Eliezer and others have already written.
If you disagree with the OP… that’s pretty important! Share your thoughts.
Wrote a long comment here. (Which you’ve seen, but linking since your comment started as a response to me.)
I think he means that there are more points that could be made. (If the points in the post are the training set, can you also produce the points in the held-out test set?)
Separately from whether the plans themselves are safe or dangerous, I think the key question is whether the process that generated the plans is trying to deceive you (so it can break out into the real world or whatever).
If it’s not trying to deceive you, then it seems like you can just build in various safeguards (like asking, “is this plan safe?”, as well as more sophisticated checks), and be okay.
What do you think of a claim like “most of the intelligence comes from the steps where you do most of the optimization”? A corollary of this is that we particularly want to make sure optimization intensive steps of AI creation are safe WRT not producing intelligent programs devoted to killing us.
This seems probably right to me.
Example: most of the “intelligence” of language models comes from the supervised learning step. However, it’s in-principle plausible that we could design e.g. some really capable general purpose reinforcement learner where the intelligence comes from the reinforcement, and the latter could (but wouldn’t necessarily) internalise “agenty” behaviour.
I agree that reinforcement learners seem more likely to be agent-y (and therefore scarier) than self-supervised learners.
Can you visualize an agent that is not “open-ended” in the relevant ways, but is capable of, say, building nanotech and melting all the GPUs?
FWIW, I’m not sold on the idea of taking a single pivotal act. But, engaging with what I think is the real substance of the question — can we do complex, real-world, superhuman things with non-agent-y systems?
Yes, I think we can! Just as current language models can be prompt-programmed into solving arithmetic word problems, I think a future system could be led to generate a GPU-melting plan, without it needing to be a utility-maximizing agent.
For a very hand-wavy sketch of how that might go, consider asking GPT-N to generate 1000s of candidate high-level plans, then rate them by feasibility, then break each plan into steps and re-evaluate, etc.
Or, alternatively, imagine the cognitive steps you might take if you were trying to come up with a GPU-melting plan (or alternatively a pivotal act plan in general). Do any of those steps really require that you have a utility function or that you’re a goal-directed agent?
It seems to me that we need some form of search, and discrimination and optimization. But not necessarily anymore than GPT-3 already has. (It would just need to be better at the search. And we’d need to make many many passes through the network to complete all the cognitive steps.)
On your view, what am I missing here?
Is GPT-3 already more of an agent than I realize? (If so, is it dangerous?)
Will GPT-N by default be more of an agent than GPT-3?
Are our own thought processes making use of goal-directedness more than I realize?
Will prompt-programming passive systems hit a wall somewhere?
If so, what are some of the simplest cognitive tasks that we can do that you think such systems wouldn’t be able to do?
(See also my similar question here.)
I disagree; I think the agency is necessary to build a really good world-model, one that includes new useful concepts that humans have never thought of.
Without the agency, some of the things that you lose are (and these overlap): Intelligently choosing what to attend to; intelligently choosing what to think about; intelligently choosing what book to re-read and ponder; intelligently choosing what question to ask; ability to learn and use better and better brainstorming strategies and other such metacognitive heuristics.
Why is agency necessary for these things?
If we follow Ought’s advice and build “process-based systems [that] are built on human-understandable task decompositions, with direct supervision of reasoning steps”, do you expect us to hit a hard wall somewhere that prevents these systems from creatively choosing things to think about, books to read, or better brainstorming strategies?
This sounds maybe exactly the same as Tegmark Level 4 Multiverse.
I agree!
Dumb question — are these the same polytopes as described in Anthropic’s recent work here, or different polytopes?