p.b.

Karma: 779

The king token

p.b.28 May 2023 19:18 UTC

10 points

0 comments4 min readLW link

Character alignment II

p.b.14 May 2023 14:17 UTC

5 points

0 comments2 min readLW link

p.b. 14 May 2023 13:52 UTC
LW: 4 AF: 3
0
AF
on: Steering GPT-2-XL by adding an activation vector
This is really cool work! Congratulations!
Besides the LLM related work it also reminds somewhat of dynamic prompting in Stable Diffusion, where part of the prompt is changed after a number of steps to achieve a mixture of promp1 and prompt2.
What’s the TL;DR for the Vicuna 13B experiments?

p.b. 12 May 2023 18:22 UTC
44 points
38
on: The way AGI wins could look very stupid
I think this extrapolates far from one example and I’m not sure the example applies all that well.
Old engines played ugly moves because of their limitations, not because playing ugly moves is a super power. They won anyway because humans cannot out calculate engines.
AlphaZero plays beautiful games and even todays standard engines don’t play ugly or dumb looking moves anymore. I think in the limit superior play will tend to be beautiful and elegant.
If there is a parallel between early super human chess and AGI takeover it will be that AGI uses less than brillant strategies that still work because of flawless or at least vastly superhuman execution. But these strategies will not look dump or incomprehensible.

p.b. 2 May 2023 13:34 UTC
1 point
−1
in reply to: Alexei’s comment on: What 2025 looks like
Hah, I read that as 5-10%, which I guess would be realistic.

p.b. 1 May 2023 20:27 UTC
6 points
5
on: AI #9: The Merge and the Million Tokens
The “million token” recurrent memory transformer was first published July 2022. The new paper is just an investigation whether the method can also be used for BERT-like encoder models.
Given that there was a ton of papers that “solved” the quadratic bottleneck I wouldn’t hold my breath.

p.b. 21 Apr 2023 10:44 UTC
6 points
5
on: AI #8: People Can Do Reasonable Things
GPT-3 to GPT-4 took 3 years. Why is it surprising that the training run for GPT-5 has not yet started?
The crucial thing is that OpenAI never stopped developing, training and tweaking GPT-3 during that time and capabilities made significant progress. They certainly won’t stop putting a ton of compute and data and algorithmic ingenuity into GPT-4 until they feel that compute and algorithms have reached a point where training a completely new model from scratch makes sense.
Given that the image perception functionality of GPT-4 hasn’t even been rolled out yet, they also haven’t been able to collect feedback on which parts need tweaking most.
One could argue that the important point of this piece of information is that the capabilities jump that will come with GPT-5 is not as imminent as if the training had already started. But I strongly suspect that the later the training starts the larger the jump will be.
I wouldn’t be worried about a GPT-5 that started training already. I would expect it to be a bigger GPT-4. I’m definitely more worried about a GPT-5 that only starts training after the lessons of massive scale deployment and a thousand different use cases have been incorporated into its architecture.

p.b. 16 Apr 2023 5:55 UTC
14 points
8
in reply to: Nate Showell’s comment on: The ‘ petertodd’ phenomenon
Glitch tokens make for fascinating reading, but I think the technical explanation doesn’t leave too much mystery on the table. I think where those tokens end up in concept space is basically random and therefore extreme.
To really study them more closely, I think it makes sense to use Llama 65B or OPT 175B. There you would have full control over the vector embedding and you could input random embeddings and semi-random embeddings and study which parts of the concept space leads to which behaviours.

p.b. 12 Apr 2023 10:47 UTC
5 points
5
on: [Link] Sarah Constantin: “Why I am Not An AI Doomer”
This is easily the best anti near-term AGI article I have read so far. Especially cross-ontology goal robustness is a great point.

p.b. 7 Apr 2023 6:42 UTC
3 points
6
on: AI #6: Agents of Change
The RAM requirement drop for Llama wasn’t real. Mmap just makes OS RAM accounting wonky.

p.b. 3 Apr 2023 13:38 UTC
14 points
0
in reply to: Tao Lin’s comment on: Hooray for stepping out of the limelight
Deepmind surpassed GPT-3 within the same year it was published, but kept quiet about that for another full year. Then they published the Chinchilla-scaling paper and Flamingo.
If they were product focused they could have jumped on the LLM-API bandwagon any time on equal footing with OpenAI. I doubt that suddenly changed with GPT4.

p.b. 29 Mar 2023 13:34 UTC
12 points
11
on: FLI open letter: Pause giant AI experiments
I just want to point out that if successful this moratorium would be the fire alarm that otherwise doesn’t exist for AGI.
The benefit would be the “industry leaders agree …” headline, not the actual pause.

p.b. 17 Mar 2023 12:21 UTC
1 point
0
in reply to: Viliam’s comment on: Are the LLM “intelligence” tests publicly available for humans to take?
This is one of the tests where you spent not too much time (I think I took 13 minutes) and at the end there was “this is your result, if you want to see fine-grained scores, send money”.
Well, they claim that it is somehow validated and my score was somewhat realistic.
Main negative point was the need to provide an email for the result.

p.b. 17 Mar 2023 8:37 UTC
1 point
0
on: Are the LLM “intelligence” tests publicly available for humans to take?
On twitter the IQ score of IIRC 84 for ChatGPT and 96 for GPT-4 were making the rounds, maybe you refer to those? I believe these scores are based on this freely available online test:
https://iqtest.com/take-the-test/
I took it on wednesday just for fun. It’s purely text-based but involves many different types of reasoning (including spatial reasoning). It’s also a timed test which arguably inflates the LLM scores compared to humans.

p.b. 17 Mar 2023 8:31 UTC
1 point
0
in reply to: SandXbox’s comment on: What’s the Least Impressive Thing GPT-4 Won’t be Able to Do
ChatGPT could do this almost, with the same technique, but occasionally forgot to count one word. Do you know how reliably Bing Chat is for bigger numbers?

p.b. 16 Mar 2023 21:27 UTC
3 points
0
in reply to: Robert_AIZI’s comment on: GPT-4: What we (I) know about it
It was posted by Stella Biderman probably in the Eleuther discord many month ago. I took a screenshot that I somewhat surprisingly was able to find again just now. At least for OPT at GPT-3 size the feedforward layers apparently use 80% of the compute. The naming of the other columns confuses me, logically one should be a key, query, value computation and the other the actually quadratic dot-product, but I can’t tell which is which.

p.b. 16 Mar 2023 10:15 UTC
2 points
0
on: GPT-4: What we (I) know about it
In a transformer, the compute cost for context length n grows at O(n^2)^[4], so it’s a 16x increase in compute cost to go from 2000 tokens to 8000, and another 16x increase to go to 32000.
I see that a lot, even from experts, but it’s a misconception.
In transformers the compute cost for context length n of a part of the attention mechanism, which itself is only a part of the transformer architecture, grows at O(n^2), so for the transformer itself this is only true in the limit.
Doubling the context length does not nearly incur a quadrupling of the overall compute cost for current models as far as I know. In fact, I think that as models are scaled, the attention mechanism becomes an ever smaller part of the overall compute cost (empirically, i.e. I saw a table to that effect, you could certainly scale differently), so with model scaling you get more and more leeway to increase the context length without impacting compute (both training and inference) cost too much.
And the 32k version is almost certainly only fine-tuned on the longer context length which is another very viable option to get longer context without paying the full quadratic compute cost even at context lengths where it starts to dominate compute.

p.b. 14 Mar 2023 20:35 UTC
1 point
0
in reply to: gwern’s comment on: Can ChatGPT count?
BPEs explain the string examples but I also couldn’t get ChatGPT to accurately count words.
Edit:
An examples that I just now tried with GPT-3.5-turbo:
User: How many times does the word “cat” occur in the following sentence: A cat was a cat in a cat with a cat as a hat and picture of a cat on the cat hat.
Assistant: The word “cat” occurs 5 times in the sentence.
Because apparently I don’t have it in me to concentrate on work today, here is some more experimentation with ChatGPT:
Me: How many times does the word “cat” occur in the following sentence: A cat was a cat in a cat with a cat as a hat and picture of a cat on the cat hat.
ChatGPT: The word “cat” occurs five times in the sentence.
Me: Could you repeat the sentence but put a number after each occurence of “cat” counting up.
ChatGPT: Sure, here’s the sentence with a number after each occurrence of “cat” counting up:
“A cat (1) was a cat (2) in a cat (3) with a cat (4) as a hat and picture of a cat (5) on the cat hat.”
So I thought, maybe “a cat” is one token for some reason and I tried:
Me: Now, count the number of occurences of the word “permeable” in the following sentence: A permeable membrane is more permeable if it is more permeable rather than less permeable than a less permeable membrane is permeable.
ChatGPT: The word “permeable” appears four times in the sentence.
Me: Could you again repeat the sentence and put a number in brackets after each occurence of “permeable” counting up?
ChatGPT: Sure, here’s the sentence with a number in brackets after each occurrence of “permeable” counting up:
“A permeable membrane is more permeable if it is [1] more permeable rather than less permeable than a less permeable membrane is [2] permeable.”
“Permeable” appears twice in the first clause and twice in the second clause of the sentence, so there are four occurrences of the word “permeable” in total.

p.b. 12 Feb 2023 20:26 UTC
12 points
1
on: A proposed method for forecasting transformative AI
I applaud the effort, but I confess that I didn’t read the post very carefully, because so many problems with this approach jumped out at me. So it is entirely possible that some of the following points of critique are already dealt with in your text, in which case, sorry about that:
My understanding is that the irreducible part of the loss has nothing (necessarily) to do with “entropy of natural text” and even less with “roughly human-level”—it is the loss this particular architecture for this particular training regime can reach in the limit on this particular training data distribution.
Nothing more, nothing less.
Other architectures with other training regimes will have scaling laws with a different irreducible loss. Human level of token prediction is way worse than probably any GPT, so why would that loss indicate human level of reasoning?
To actually have an “anchor” you’d have to look at human performance on benchmarks und estimate the range of loss that might indicate human level of reasoning.
I also don’t understand why the loss on random internet text should be the same as on scientific papers. Loss differs from domain to domain. Scaling laws likely do, too.
Also, new (significant) papers are not sampled from the distribution of papers. They are out of distribution because they go beyond all previous papers. So I’m not sure your formula doesn’t just measure the ability to coherently rehash stuff that is already known.
Of course if you’d actually measure the ability to write a new scientific paper, the model would be ridiculously overpowered, because humans need 6 or 7 orders of magnitude more time to create such a paper (at least with current sampling speeds).

p.b. 7 Jan 2023 13:13 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Running by Default
Barefoot shoes had the same effect for me. They shorten the stride and that takes a lot of impact away from the knee.

p.b.

The king token

Char­ac­ter al­ign­ment II

Character alignment II