porby

Karma: 1,687

porby 9 Feb 2024 18:29 UTC
3 points
0
in reply to: Decaeneus’s comment on: porby’s Shortform
Yup, exactly the same experience here.

porby 6 Feb 2024 22:53 UTC
4 points
0
on: porby’s Shortform
Has there been any work on the scaling laws of out-of-distribution capability/behavior decay?
A simple example:
1. Simultaneously train task A and task B for N steps.
2. Stop training task B, but continue to evaluate the performance of both A and B.
3. Observe how rapidly task B performance degrades.
Repeat across scale and regularization strategies.
Would be nice to also investigate different task types. For example, tasks with varying degrees of implied overlap in underlying mechanisms (like #2).
I’ve previously done some of these experiments privately, but not with nearly the compute necessary for an interesting result.
The sleeper agents paper reminded me of it. I would love to see what happens on a closer-to-frontier model that’s intentionally backdoored, and then subjected to continued pretraining. Can a backdoor persist for another trillion tokens of nonadversarial-but-extremely-broad training? Does that vary across scale etc?
I’d also like to intentionally find the circumstances that maximize the persistence of out of distribution capabilities not implied by the current training distribution.
Seems like identifying a robust trend here would have pretty important Implications, whichever direction it points.

porby 2 Feb 2024 19:37 UTC
4 points
0
in reply to: porby’s comment on: porby’s Shortform
A further extension and elaboration on one of the experiments in the linkpost:
Pitting execution fine-tuning against input fine-tuning also provides a path to measuring the strength of soft prompts in eliciting target behaviors. If execution fine-tuning “wins” and manages to produce a behavior in some part of input space that soft prompts cannot elicit, it would be a major blow to the idea that soft prompts are useful for dangerous evaluations.
On the flip side, if ensembles of large soft prompts with some hyperparameter tuning always win (e.g. execution fine tuning cannot introduce any behaviors accessible by any region of input space without soft prompts also eliciting it), then they’re a more trustworthy evaluation in practice.

porby 2 Feb 2024 19:31 UTC
7 points
2
on: porby’s Shortform
Having escaped infinite overtime associated with getting the paper done, I’m now going back and catching up on some stuff I couldn’t dive into before.
Going through the sleeper agents paper, it appears that one path—adversarially eliciting candidate backdoor behavior—is hampered by the weakness of the elicitation process. Or in other words, there exist easily accessible input conditions that trigger unwanted behavior that LLM-driven adversarial training can’t identify.
I alluded to this in the paper linkpost, but soft prompts are a very simple and very strong option for this. There remains a difficulty in figuring out what unwanted behavior to adversarially elicit, but this is an area that has a lot of low hanging fruit.
I’d also interested in whether how more brute force interventions, like autoregressively detuning a backdoored model with a large soft prompt for a very large dataset (or an adversarially chosen anti-backdoor dataset) compares to the other SFT/RL interventions. Activation steering, too; I’m currently guessing activation-based interventions are the cheapest for this sort of thing.

porby 2 Feb 2024 5:51 UTC
5 points
0
on: Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities
By the way: I just got into San Francisco for EAG, so if anyone’s around and wants to chat, feel free to get in touch on swapcard (or if you’re not in the conference, perhaps a DM)! I fly out on the 8th.

Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities

porby2 Feb 2024 5:49 UTC

43 points

1 comment4 min readLW link

(1drv.ms)

porby 16 Dec 2023 23:21 UTC
28 points
2
on: Why I think strong general AI is coming soon
It’s been over a year since the original post and 7 months since the openphil revision.
A top level summary:
1. My estimates for timelines are pretty much the same as they were.
2. My P(doom) has gone down overall (to about 30%), and the nature of the doom has shifted (misuse, broadly construed, dominates).
And, while I don’t think this is the most surprising outcome nor the most critical detail, it’s probably worth pointing out some context. From NVIDIA:
In two quarters, from Q1 FY24 to Q3 FY24, datacenter revenues went from $4.28B to $14.51B.
From the post:
In 3 years, if NVIDIA’s production increases another 5x …
Revenue isn’t a perfect proxy for shipped compute, but I think it’s safe to say we’ve entered a period of extreme interest in compute acquisition. “5x” in 3 years seems conservative.^[1] I doubt the B100 is going to slow this curve down, and competitors aren’t idle: AMD’s MI300X is within striking distance, and even Intel’s Gaudi 2 has promising results.
Chip manufacturing remains a bottleneck, but it’s a bottleneck that’s widening as fast as it can to catch up to absurd demand. It may still be bottlenecked in 5 years, but not at the same level of production.
On the difficulty of intelligence
I’m torn about the “too much intelligence within bounds” stuff. On one hand, I think it points towards the most important batch of insights in the post, but on the other hand, it ends with an unsatisfying “there’s more important stuff here! I can’t talk about it but trust me bro!”
I’m not sure what to do about this. The best arguments and evidence are things that fall into the bucket of “probably don’t talk about this in public out of an abundance of caution.” It’s not one weird trick to explode the world, but it’s not completely benign either.
Continued research and private conversations haven’t made me less concerned. I do know there are some other people who are worried about similar things, but it’s unclear how widely understood it is, or whether someone has a strong argument against it that I don’t know about.
So, while unsatisfying, I’d still assert that there are highly accessible paths to broadly superhuman capability on short timescales. Little of my forecast’s variance arises from uncertainty on this point; it’s mostly a question of when certain things are invented, adopted, and then deployed at sufficient scale. Sequential human effort is a big chunk; there are video games that took less time to build than the gap between this post’s original publication date and its median estimate of 2030.
On doom
When originally writing this, my model of how capabilities would develop was far less defined, and my doom-model was necessarily more generic.
A brief summary would be:
1. We have a means of reaching extreme levels of capability without necessarily exhibiting preferences over external world states. You can elicit such preferences, but a random output sequence from the pretrained version of GPT-N (assuming the requisite architectural similarities) has no realistic chance of being a strong optimizer with respect to world states. The model itself remains a strong optimizer, just for something that doesn’t route through the world.
2. It’s remarkably easy to elicit this form of extreme capability to guide itself. This isn’t some incidental detail; it arises from the core process that the model learned to implement.
3. That core process is learned reliably because the training process that yielded it leaves no room for anything else. It’s not a sparse/distant reward target; it is a profoundly constraining and informative target.
I’ve written more on the nice properties of some of these architectures elsewhere. I’m in the process of writing up a complementary post on why I think these properties (and using them properly) are an attractor in capabilities, and further, why I think some of the x-riskiest forms of optimization process are actively repulsive for capabilities. This requires some justification, but alas, the post will have to wait some number of weeks in the queue behind a research project.
The source of the doom-update is the correction of some hidden assumptions in my doom model. My original model was downstream of agent foundations-y models, but naive. It followed a process: set up a framework, make internally coherent arguments within that framework, observe highly concerning results, then neglect to notice where the framework didn’t apply.
Specifically, some of the arguments feeding into my doom model were covertly replacing instances of optimizers with hypercomputer-based optimizers^[2], because hey, once you’ve got an optimizer and you don’t know any bounds on it, you probably shouldn’t assume it’ll just turn out convenient for you, and hypercomputer-optimizers are the least convenient.
For example, this part:
Is that enough to start deeply modeling internal agents and other phenomena concerning for safety?
And this part:
AGI probably isn’t going to suffer from these issues as much. Building an oracle is probably still worth it to a company even if it takes 10 seconds for it to respond, and it’s still worth it if you have to double check its answers (up until oops dead, anyway).
With no justification, I imported deceptive mesaoptimizers and other “unbound” threats. Under the earlier model, this seemed natural.
I now think there are bounds on pretty much all relevant optimizing processes up and down the stack from the structure of learned mesaoptimizers to the whole capability-seeking industry. Those bounds necessarily chop off large chunks of optimizer-derived doom; many outcomes that previously seemed convergent to me now seem extremely hard to access.
As a result, “technical safety failure causes existential catastrophe” dropped in probability by around 75-90%, down to something like 5%-ish.^[3]
I’m still not sure how to navigate a world with lots of extremely strong AIs. As capability increases, outcome variance increases. With no mitigations, more and more organizations (or, eventually, individuals) will have access to destabilizing systems, and they would amplify any hostile competitive dynamics.^[4] The “pivotal act” frame gets imported even if none of the systems are independently dangerous.
I’ve got hope that my expected path of capabilities opens the door for more incremental interventions, but there’s a reason my total P(doom) hasn’t yet dropped much below 30%.
1. ^
  The reason why this isn’t an update for me is that I was being deliberately conservative at the time.
2. ^
  A hypercomputer-empowered optimizer can jump to the global optimum with brute force. There isn’t some mild greedy search to be incrementally shaped; if your specification is even slightly wrong in a sufficiently complex space, the natural and default result of a hypercomputer-optimizer is infinite cosmic horror.
3. ^
  It’s sometimes tricky to draw a line between “oh this was a technical alignment failure that yielded an AI-derived catastrophe, as opposed to someone using it wrong,” so it’s hard to pin down the constituent probabilities.
4. ^
  While strong AI introduces all sorts of new threats, its generality amplifies “conventional” threats like war, nukes, and biorisk, too. This could create civilizational problems even before a single AI could, in principle, disempower humanity.
What links here?
- Voting Results for the 2022 Review by Ben Pace (2 Feb 2024 20:34 UTC; 57 points)

porby 13 Dec 2023 23:41 UTC
14 points
2
on: AI Views Snapshots
Mine:
My answer to “If AI wipes out humanity and colonizes the universe itself, the future will go about as well as if humanity had survived (or better)” is pretty much defined by how the question is interpreted. It could swing pretty wildly, but the obvious interpretation seems ~tautologically bad.

porby 13 Dec 2023 20:49 UTC
4 points
0
on: porby’s Shortform
I sometimes post experiment ideas on my shortform. If you see one that seems exciting and you want to try it, great! Please send me a message so we can coordinate and avoid doing redundant work.

porby 13 Dec 2023 20:45 UTC
3 points
0
on: Suggestions for net positive LLM research
I’m accumulating a to-do list of experiments much faster than my ability to complete them:
If you wanted to take one of these and run with it or a variant, I wouldn’t mind!
The unifying theme behind many of these is goal agnosticism: understanding it, verifying it, maintaining it, and using it.
Note: I’ve already started some of these experiments, and I will very like start others soon. If you (or anyone reading this, for that matter) sees something they’d like to try, we should chat to avoid doing redundant work. I currently expect to focus on #4 for the next handful of weeks, so that one is probably at the highest risk of redundancy.
Further note: I haven’t done a deep dive on all relevant literature; it could be that some of these have already been done somewhere! (If anyone happens to know of prior art for any of these, please let me know.)

porby 11 Dec 2023 2:55 UTC
4 points
0
on: porby’s Shortform
Retrodicting prompts can be useful for interpretability when dealing with conditions that aren’t natively human readable (like implicit conditions induced by activation steering, or optimized conditions from soft prompts). Take an observed completion and generate the prompt that created it.
What does a prompt retrodictor look like?
Generating a large training set of soft prompts to directly reverse would be expensive. Fortunately, there’s nothing special in principle about soft prompts with regard to their impact on conditioning predictions.
Just take large traditional text datasets. Feed the model a chunk of the string. Train on the prediction of tokens before the chunk.
Two obvious approaches:
1. Special case of infilling. Stick to a purely autoregressive training mode, but train the model to fill a gap autoregressively. In other words, the sequence would be:
  [Prefix token][Prefix sequence][Suffix token][Suffix sequence][Middle token][Middle sequence][Termination token]
  Or, as the paper points out:
  [Suffix token][Suffix sequence][Prefix token][Prefix sequence][Middle sequence][Termination token] Nothing stopping the prefix sequence from having zero length.
2. Could also specialize training for just previous prediction:
  [Prompt chunk]["Now predict the previous" token][Predicted previous chunk, in reverse]
But we don’t just want some plausible previous prompts, we want the ones that most precisely match the effect on the suffix’s activations.
This is trickier. Specifying the optimization target is easy enough: retrodict a prompt that minimizes MSE((activations | sourcePrompt), (activations | retrodictedPrompt)), where (activations | sourcePrompt) are provided. Transforming that into a reward for RL is one option. Collapsing the outout distribution into a token is a problem; there’s no way to directly propagate the gradient through that collapse and into the original distribution. Without that differentiable connection, analytically computing gradients for the other token options becomes expensive and turns into a question of sampling strategies. Maybe something clever floating around.
Note that retrodicting with an activation objective has some downsides:
1. If the retrodictor’s the same model as the predictor, there are some weird feedback loops. The activations become a moving target.
2. Targeting activations makes the retrodictor model-specific. Without targeting activations, the retrodictor could work for any model in principle.
3. While the outputs remain constrained to token distributions, the natural endpoint for retrodiction on activations is not necessarily coherent natural language. Adversarially optimizing for tokens which produce a particular activation may go weird places. It’ll likely still have some kind of interpretable “vibe,” assuming the model isn’t too aggressively exploitable.
This class of experiment is expensive for natural language models. I’m not sure how interesting it is at scales realistically trainable on a couple of 4090s.

porby 11 Dec 2023 0:04 UTC
2 points
0
in reply to: porby’s comment on: porby’s Shortform
Another potentially useful metric in the space of “fragility,” expanding on #4 above:
The degree to which small perturbations in soft prompt embeddings yield large changes in behavior can be quantified. Perturbations combined with sampling the gradient with respect to some behavioral loss suffices.
This can be thought of as a kind of internal representational fragility. High internal representational fragility would imply that small nudges in the representation can blow up intent.
Does internal representational fragility correlate with other notions of “fragility,” like the information-required-to-induce-behavior “fragility” in the other subthread about #6? In other words, does requiring very little information to induce a behavior correlate with the perturbed gradients with respect to behavioral loss being large for that input?
Given an assumption that the information content of the soft prompts have been optimized into a local minimum, sampling the gradient directly at the soft prompt should show small gradients. In order for this correlation to hold, there would need to be steeply bounded valley in the loss landscape. Or to phrase it another way, for this correlation to exist, behaviors which are extremely well-compressed by the model and have informationally trivial pointers would need to correlate with fragile internal representations.
If anything, I’d expect anticorrelation; well-learned regions probably have enough training constraints that they’ve been shaped into more reliable, generalizing formats that can representationally interpolate to adjacent similar concepts.
That’d still be an interesting thing to observe and confirm, and there are other notions of fragility that could be considered.

porby 10 Dec 2023 22:47 UTC
2 points
0
in reply to: porby’s comment on: porby’s Shortform
A further extension: While relatively obvious in context, this also serves as a great way to automate adversarial jailbreak attempts (broadly construed), and to quantify how resistant a given model or prompting strategy is to jailbreaks.
Set up your protections, then let SGD try to jailbreak it. The strength of the protections can be measured by the amount of information required to overcome the defenses to achieve some adversarial goal.
In principle, a model could be perfectly resistant and there would be no quantity of information sufficient to break it. That’d be good to know!
This kind of adversarial prompt automation could also be trivially included in an evaluations program.
I can’t imagine that this hasn’t been done before. If anyone has seen something like this, please let me know.

porby 10 Dec 2023 22:38 UTC
2 points
0
in reply to: porby’s comment on: porby’s Shortform
Expanding on #6 from above more explicit, since it seems potentially valuable:
From the goal agnosticism FAQ:
The definition as stated does not put a requirement on how “hard” it needs to be to specify a dangerous agent as a subset of the goal agnostic system’s behavior. It just says that if you roll the dice in a fully blind way, the chances are extremely low. Systems will vary in how easy they make it to specify bad agents.
From earlier experimentpost:
Figure out how to think about the “fragility” of goal agnostic systems. Conditioning a predictor can easily yield an agent that is not goal agnostic; this is expected and not inherently problematic. But what if it is trivial to accidentally condition a strong model into being a worldeater, rather than a passive Q&A bot? There’s clearly a spectrum here in terms of how “chaotic” a model is—the degree to which small perturbations can yield massive consequences—but it remains conceptually fuzzy.
This can be phrased as “what’s the amount of information required to push a model into behavior X?”
Given a frozen model, optimizing prompt tokens gives us a direct way of answering a relevant proxy for this question:
“What is the amount of information (accessible to SGD through soft prompting) required to push a model into behavior X?”
In practice, this seems like it should be a really good proxy, and (provided some compute) it gives you a trivially quantifiable answer:
Try different soft prompt token counts and observe performance on the task that the soft prompts were targeting. The resulting token count versus performance curve characterizes the information/performance tradeoff for that behavior, given that model.
This seems like… it’s… an extremely good answer to the “fragility” question? It’s trivial to incorporate this into an evaluations scheme. Just have a bunch of proxy tasks that would be alarming if they were accessible by trivial differences in prompting.
Conceptually, it’s a quantification of the number of information theoretic mistakes you’d need to make to get bad behavior from the model.

porby 10 Dec 2023 22:23 UTC
5 points
0
on: porby’s Shortform
Soft prompts are another form of prompt automation that should naturally preserve all the nice properties of goal agnostic architectures.
Does training the model to recognize properties (e.g. ‘niceness’) explicitly as metatokens via classification make soft prompts better at capturing those properties?
You could test for that explicitly:
1. Pretrain model A with metatokens with a classifier.
2. Pretrain model B without metatokens.
3. Train soft prompts on model A with the same classifier.
4. Train soft prompts on model B with the same classifier.
5. Compare performance of soft prompts in A and B using the classifier.
Notes and extensions:
1. The results of the research are very likely scale sensitive. As the model gets larger, many classifier-relevant distinctions that could be missed by small models lacking metatoken training may naturally get included. In the limit, the metatoken training contribution may become negligible. Is this observable across ~pythia scales? Could do SFT on pythia to get a “model A.”
2. The above description leaves out some complexity. Ideally, the classifier could give scalar scores. This requires scalarized input tokens for the model that pretrains with metatokens.
3. How does soft prompting work when tokens are forced to be smaller? For example, if each token is a character, it’ll likely have a smaller residual dedicated to it compared to tokens that spans ~4 characters to equalize total compute.
4. To what degree does soft prompting verge on a kind of “adversarial” optimization? Does it find fragile representations where small perturbations could produce wildly different results? If so, what kinds of regularization are necessary to push back on that, and what is the net effect of that regularization?
5. There’s no restriction on the nature of the prompt. In principle, the “classifier” could be an RL-style scoring mechanism for any reward. How many tokens does it take to push a given model into particular kinds of “agentic” behavior? For example, how many tokens does it take to encode the prompt corresponding to “maximize the accuracy of the token prediction at index 32 in the sequence”?
6. More generally: the number of tokens required to specify a behavior could be used as a metric for the degree to which a model “bakes in” a particular functionality. More tokens required to specify behavior successfully → more information required in that model to specify that behavior.

porby 9 Dec 2023 20:42 UTC
4 points
2
on: porby’s Shortform
Quarter-baked experiment:
1. Stick a sparse autoencoder on the residual stream in each block.
2. Share weights across autoencoder instances across all blocks.
3. Train autoencoder during model pretraining.
4. Allow the gradients from autoencoder loss to flow into the rest of the model.
Why? With shared autoencoder weights, every block is pushed toward sharing a representation. Questions:
1. Do the meanings of features remain consistent over multiple blocks? What does it mean for an earlier block’s feature to “mean” the same thing as a later block’s same feature when they’re at different parts of execution?
2. How much does a shared representation across all blocks harm performance? Getting the comparison right is subtle; it would be quite surprising if there is no slowdown on predictive training when combined with the autoencoder training since they’re not necessarily aligned. Could try training very small models to convergence to see if they have different plateaus.
3. If forcing a shared representation doesn’t harm performance, why not? In principle, blocks can execute different sorts of programs with different IO. Forcing the residual stream to obey a format that works for all blocks without loss would suggest that there were sufficient representational degrees of freedom remaining (e.g. via superposition) to “waste” some when the block doesn’t need it. Or the shared “features” mean something completely different at different points in execution.
4. Compare the size of the dictionary required to achieve a particular specificity of feature between the shared autoencoder and a per-block autoencoder. How much larger is the shared autoencoder? In the limit, it could just be BlockCount times larger with some piece of the residual stream acting as a lookup. It’d be a little surprising if there was effectively no sharing.
5. Compare post-trained per-block autoencoders against per-block autoencoders embedded in pretraining that allow gradients to flow into the rest of the model. Are there any interesting differences in representation? Maybe in terms of size of dictionary relative to feature specificity? In other words, does pretraining the feature autoencoder encourage a more decodable native representation?
6. Take a look at the decoded features across blocks. Can you find a pattern for what features are relevant to what blocks? (This doesn’t technically require having a shared autoencoder, but having a single shared dictionary makes it easier to point out when the blocks are acting on the same feature, rather than doing an investigation, squinting, and saying “yeah, that sure looks similar.”)

porby 30 Nov 2023 18:44 UTC
5 points
1
in reply to: RogerDearnaley’s comment on: How to Control an LLM’s Behavior (why my P(DOOM) went down)
I think that’d be great!
Some of this stuff technically accelerates capabilities (or more specifically, the elicitation of existing capabilities), but I think it also belongs to a more fundamentally reliable path on the tech tree. The sooner the industry embraces it, the less time they spend in other parts of the tech tree that are more prone to misoptimization failures, and the less likely it is that someone figures out how to make those misoptimization failures way more efficient.
I suspect there’s a crux about the path of capabilities development in there for a lot of people; I should probably get around to writing a post about the details at some point.

porby 29 Nov 2023 19:18 UTC
5 points
1
in reply to: RogerDearnaley’s comment on: How to Control an LLM’s Behavior (why my P(DOOM) went down)
What I’m calling a simulator (following Janus’s terminology) you call a predictor
Yup; I use the terms almost interchangeably. I tend to use “simulator” when referring to predictors used for a simulator-y use case, and “predictor” when I’m referring to how they’re trained and things directly related to that.
I also like your metatoken concept: that’s functionally what I’m suggesting for the tags in my proposal, except I follow the suggestion of this paper to embed them via pretraining.
Yup again—to be clear, all the metatoken stuff I was talking about would also fit in pretraining. Pretty much exactly the same thing. There are versions of it that might get some efficiency boosts by not requiring them to be present for the full duration of pretraining, but still similar in concept. (If we can show an equivalence between trained conditioning and representational interventions, and build representational interventions out of conditions, that could be many orders of magnitude faster.)

porby 29 Nov 2023 19:00 UTC
4 points
0
in reply to: Shiroe’s comment on: How to Control an LLM’s Behavior (why my P(DOOM) went down)
Alas, nope! To my knowledge it hasn’t actually been tried at any notable scale; it’s just one of those super simple things that would definitely work if you were willing to spend the compute to distill the behavior.

porby 29 Nov 2023 4:06 UTC
26 points
5
on: How to Control an LLM’s Behavior (why my P(DOOM) went down)
Signal boosted! This is one of those papers that seems less known that it should be. It’s part of the reason why I’m optimistic about dramatic increases in the quality of “prosaic” alignment (in the sense of avoiding jailbreaks and generally behaving as expected) compared to RLHF, and I think it’s part of a path that’s robust enough to scale.
You can compress huge prompts into metatokens, too (just run inference with the prompt to generate the training data). And nest and remix metatokens together.
It’s also interesting in that it can preserve the constraints on learnable values during predictive training, unlike approaches equivalent to RL with sparse/distant rewards.
The fact that the distinctions it learns about the metatokens become better and better as more optimization pressure is applied is an interesting inversion of the usual doom-by-optimization story. Taking such a model to the extreme of optimization just makes it exceedingly good at distinguishing subtle details of what constitutes <nice> versus <authoritative_tone> versus <correct>. It’s an axis of progress in alignment that generalizes as the capability does; the capability is the alignment. I’m pretty certain that a model that has very thoroughly learned what “nice” means at the human level can meaningfully generalize it to contexts where it hasn’t seen it directly applied.^[1]
I’m also reasonably confident in finding some other paths to extremely similar effects on internal representations. I wouldn’t be surprised if we can decompose conditions into representational features to learn about what they mean at the learned feature level, then cobble together new inference-time conditions via representational intervention that would have equivalent effects to training new metatokens.
1. ^
  After all, ChatGPT4/DALLE3 can generate an image of a vacuum cleaner that “embodies the aspirational human trait of being kind to one another.” That seems like more of a reach than a hypothetical superintelligence figuring out that humans wouldn’t be okay with, say, a superscience plan that would blow up 25% of the earth’s crust.

porby

Soft Prompts for Eval­u­a­tion: Mea­sur­ing Con­di­tional Dis­tance of Capabilities

On the difficulty of intelligence

On doom

Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities