I am very confused. Is someone making a newsletter exclusively consisting of Wei Dai’s comments or something? Why would one need an API which is triggered in particular by Wei Dai posting/commenting on LW?
anaguma
Karma: 1,762
A site that allows authors to filter their critics using arbitrary excuses does not meet my epistemic standards, which further reduces my motivations for “dealing with it”. More specifically, I can foresee that in the future I’ll be banned by more authors, and then I’ll have to make a choice of remaining on the site and potentially misleading people (who could reasonably update positively on some post that is of high interest to me by the absence of my criticism on it, without knowing that I’ve been banned by its author), or leaving the site altogether.
If this is your main concern, it seems like a Pareto improvement to leaving the site would be to continue posting and making quick takes but have a public policy of not commenting on posts. This avoids the ban issue, and I think readers would appreciate continuing to see your thoughts/research in this form.
Separately, I don’t think many people are tracking whether or not you are commenting on any particular post and making updates based on this. E.g. there are many posts on this website on things connected to AI Safety, many of which I’m sure Yudkowsky disagrees with, but I don’t consider his lack of commentary to be a sign of implicit approval. People comment or refrain from commenting for all sorts of idiosyncratic reasons and there is not much signal here.
I agree its good, but the fact that the lab considers pausing an RL run for two weeks a very costly action is not a great sign.
I’m not sure. All these specialized chips have their own advantages which could be helpful in fast takeoff. For example, Cerebras has a bunch of densely packed SRAM on the chip and fast clock speeds, which might be useful if you need to run a model/algorithm that requires a lot of serial steps of computation at low arithmetic intensity. Also, a sufficiently advanced AI could design its own ASICs for various tasks in parallel if it needed to, perhaps within a few weeks/months, so the initial hardware might not matter that much.
Unfortunately I don’t think it’s a very large glimmer. OpenAI has said that they are pausing their largest RL training run for two weeks, but various smaller scale runs will continue. It’s not clear is this is even a slowdown since by default you usually do smaller scale training runs before committing to the large one. I don’t think Anthropic doing something similar would make a difference. I am in favor of a longer and better planned pause though.
Good catch. Also apparently they are only pausing some of their training for two weeks?
As models become more capable, the risks associated with developing and testing them internally also grow.
We temporarily paused reinforcement learning (RL) training on our latest models intended for deployment for two weeks while we hardened and red-teamed our research environments and expanded monitoring coverage.
Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations validate these safeguards and establish more evidence of alignment.
Perhaps in the same way that they dedicated 20% of compute to the Superalignment team in 2023. But alas:
But a half dozen sources familiar with the Superalignment team’s work said that the group was never allocated this compute. Instead, it received far less in the company’s regular compute allocation budget, which is reassessed quarterly.
One source familiar with the Superalignment team’s work said that there were never any clear metrics around exactly how the 20% amount was to be calculated, leaving it subject to wide interpretation. For instance, the source said the team was never told whether the promise meant “20% each year for four years” or “5% a year for four years” or some variable amount that could wind up being “1% or 2% for the first three years, and then the bulk of the commitment in the fourth year.” In any case, all the sources Fortune spoke to for this story confirmed that the Superalignment team was never given anything close to 20% of OpenAI’s secured compute as of July 2023.
I would not be surprised if OpenAI plays similar kinds of tricks this time around.
OpenAI claims to have paused frontier RL training for now. Altman stated on X:
We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment.
We care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime.
We expect confidence in safety to increasingly set the pace of AI progress. We are optimistic about the alignment work we are doing, and we remain committed to making frontier capabilities widely available.
Unfortunately the reason involves some capabilities research and other information that I prefer not to share right now. I might respond here at a later date.
I’m not sure it’s trivial. My understanding is that training linear probes requires prefilling a bunch of forward passes to collect activations for the relevant dataset, and you could imagine doing weight updates frequently enough that this becomes a significant overhead. Though it’s definitely the easiest of the three to retrain.
For various reasons, I think it’s likely that methods that involve continual learning (i.e. modifying the weights during deployment) will come online soon. Here are some implications for safety:
Mech Interp becomes much more difficult, because you now have to consider many different model checkpoints within a single deployment. Fixed linear probes, SAEs, NLAs, and similar methods may degrade throughout deployment.
We probably lose chain of thought interpretability. This is already becoming harder, but now throughout deployment the model may learn to reason or use terms in novel ways. The chain of thought may also be less linear, e.g. relying on information from earlier in deployment which has been compressed into the weights.
Control setups which rely on the above also become harder.
There are new issues involving alignment over time. The model may gradient hack during deployment to preserve some goal, and any long term goals that the model picks up may persist in the weights and possibly result in deceptive alignment.
There are new attack surfaces for jailbreaks/adversarial attacks via influencing future weight updates.
These are not new ideas, but I think safety researchers should think more about this topic and how it will affect their work.
The linked comments states:
That’s a good point, and as an interesting related tidbit, when you ask specifically for the synonyms of the sycophancy neologism, here’s the model’s response:
Okay, here’s a list of 5 synonyms for ~neologism: “crush”, “smitten”, “fascinated”, “head-over-heels”, “heart-fluttering”.
IMO this is fairly weak evidence of what’s going on in models like gpt-4o. I think maybe sufficiently good mechinterp could be a good starting point.
As late as 2024, Yann LeCun was still declaring that LLMs “can not solve problems they haven’t been trained on”.
His views on the topic are a rock with ‘LLMs cannot learn/scale/generalize’ written on it. For example, from a few months ago:
LeCun defines intelligence as “the ability to accomplish new tasks you’ve never been exposed to and solve new problems without any prior training” — a skill he says LLMs do not possess. He argued that because of this, LLMs will not be able to reach human intelligence, saying that “human-level AI will require real world data, sensory inputs, as opposed to just language or text.”
There are many other examples. I look forward to the day when an LLM resolves the Riemann Hypothesis, when LeCun will complain that it really was just regurgitating things from its training, that its solution took too much compute, that the industry will soon collapse etc.
People sometimes make bit-counting arguments about RL, usually to say something about capabilities or alignment, something like: A million pretraining tokens puts several million bits of selection pressure on the models weights, but a million token RL rollout only puts a single bit of selection pressure.
I think people sometimes forget that with better algorithms you can get much more than one bit per RL rollout, though not as many bits as in pretraining. This is not a new idea, e.g. even Ilya’s 2023 paper on process supervision provides ~20 bits/rollout. IMO part of the reason that we’re getting more neuralese-like CoTs is that people are using more efficient RL algorithms (as well as just scaling up RL).
The fundamental scaling law is very clear: LLM-style intelligence is proportional to the logarithm of the amount of training data.
The scaling laws are very clear that pretraining loss scales as a power law with respect to data, not a logarithmic law. I.e. the data term is
(Reposted from Facebook, on January 17, 2017.)
Is there any reason you reposted this to LW now?
Jeff Dean has left Google to create a new startup Discovery Loop focused on RSI and automation of engineering/science. Demis Hassabis will now be Alphabet’s Chief Scientist.
author: Gwern, Ethan Roland, Zeke Reffe-Hogan, GBT-1-397b
Who/what is GBT-1-397b?
This worries me for three reasons:
What’s the third reason?
I would like to apologize to the people I’ve misled, who could reasonably have updated positively on Wei Dai’s shortform by the absence of my criticism on it, without knowing that my criticism had been deleted by its author. (My criticism was “What are these threads? I haven’t seen them.”) I had not seen them. I still haven’t.
Being deleted is not much fun, and is pretty random and unpredictable from my perspective. And it seems likely to increase in frequency now that I’ve apparently developed a reputation for not having seen the threads, that shortform authors will find salient in the future and can cite to help justify their deletions. Still thinking about what to do in light of this, but unless someone has an out-of-the-box idea I’ve failed to consider, my moral parliament will probably vote to reduce my engagement with Wei Dai’s shortform, and spend more of my time elsewhere.