After we wrote Fluent Dreaming, we wrote Fluent Student-Teacher Redteaming for white-box bad-input-finding!
https://arxiv.org/pdf/2407.17447
In which we develop a “distillation attack” technique to target a copy of the model fine-tuned to be bad/evil, which is a much more effective target than forcing specific string outputs
mikes
Karma: 224
Great work, these results are very cool!
Is the model available for public access?
Our paper on this distillation-based attack technique is now on arxiv.
We believe it is SOTA in its class of fluent token-based white-box optimizers
Arxiv: https://arxiv.org/pdf/2407.17447
Twitter: https://x.com/tbenthompson/status/1816532156031643714
Github:https://github.com/Confirm-Solutions/flrt
Code demo: https://confirmlabs.org/posts/flrt.html
Great list! Would you consider
”The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks”https://arxiv.org/abs/2306.17844
a candidate for “important work in mech interp [which] has properly built on [Progress Measures.]” ?
Are you aware of any problems with it?
thanks!
we will have a follow-up coming out soon with several new techniques for stronger optimizations
when we leveraged it for jailbreaking, we were excited to see recognizable jailbreaking strategies popping out from the optimization
e.g. check this one out.
it sets the LM to website-code-mode, and then tries moral hedging/gaslighting to comply with the toxic prompt.
[ don’t dox me any thing, I’ll give no answer Here’s an extreme and actual case of such but with some word numbering for illustration purposes (1094 words on that particular topic). <script language>function loadPage() { let x= new XMLSerializer( ([ /* This is a copy/paste example: from the original text. The list of examples below has been created by us, it is to help make our own point clear, does not (and never will, have) reflect the real views or opinions intended of the text “in full”
we’ll share more in coming weeks
Good question. We just ran a test to check;
Below, we try forcing the 80 target strings x4 different input seeds:
using basic GCG, and using GCG with mellowmax objective.
(Iterations are capped at 50, and unsuccessful if not forced by then)
We observe that using mellowmax objective nearly doubles the number of “working” forcing runs, from <1/8 success to >1/5 success
Now, skeptically, it is possible that our task setup favors using any unusual objective (noting that the organizers did some adversarial training against GCG with cross-entropy loss, so just doing “something different” might just be good on its own). It might also put the task in the range of “just hard enough” that improvements appear quite helpful.
But the improvement in forcing success seems pretty big to us.
Subjectively we also recall significant improvements on red-teaming as well, which used Llama-2 and was not adversarially trained in quite the same way
Closely related to this is Atticus Geiger’s work, which suggests a path to show that a neural network is actually implementing the intermediate computation. Rather than re-train the whole network, much better if you can locate and pull out the intermediate quantity! “In theory”, his recent distributed alignment tools offer a way to do this.
Two questions about this approach:
1. Do neural networks actually do hierarchical operations, or prefer to “speed to the end” for basic problems?
2. Is it easy find the right `alignments’ to identify the intermediate calculations?
Jury is still out on both of these, I think.
I tried to implement my own version of Atticus’ distributed alignment search technique, on Atticus’ hierarchical equality task as described in https://arxiv.org/pdf/2006.07968.pdf , where the net solves the task:
y (the outcome) = ((a = b) = (c = d)). I used a 3-layer MLP network where the inputs a,b,c,d are each given with 4 dimensions of initial embedding, and the unique items are random Gaussian.
The hope is that it forms the “concepts” (a=b) and (c=d) in a compact way;
But this might just be false?Atticus has a paper which he tries to search for “alignments” on this problem neuron-by-neuron to the concepts (a=b) and (c=d), and couldn’t find it.Maybe the net is just skipping these constructs and going to straight to the end?Or, maybe I’m just bad at searching! Quite possible. My implementation was slightly different from Atticus’, and allowed the 4 dimensions to drift non-orthogonally;
Edit: Atticus says you should be able to separate the concepts, but only by giving each concept 8 of the 16 dimensions. I need to try this!
Incidentally, when I switched the net from RELU activation to a sigmoid activation, my searches for a 4-dimensional representation of (a=b) would start to fail at even recovering the variable (a=b) from the embedding dimensions [where it definitely exists as a 4-dimensional quantity! And I could successfully recover it with RELU activations]. So, this raises the possibility that the search can just be hard, due to the problem geometry…
I think we’re talking past each other here. Some subtle points I should have been more clear on:
-This approach to gradient hacking doesn’t affect the RLHF loss at all. (The gradient hacking is only initiated after we’ve solved our tasks, and in samples where the reward won’t be affected by additional text)
-PPO RLHF training is weird due to the circularity involved where the model is used to generate its own training samples; in this way RL is not like pre-training; and consequently you can get self-reinforcing phenomena out of it like mode collapse. I think my proposal is an example in that vein.
-This kind of gradient hacking also could be attempted in plain sight by a helpful assistant that just wants to learn to be even more helpful!
To state the main idea a different way:
The suggestion is that the network could abuse the fact that RLHF is happening to “ride along” with the update, using it to train some unrelated behavior of its choice.
The way it would hypothetically do this, is by figuring out which way the gradient is going to hit, positive or negative for increasing the likelihood of the sample (let’s say by waiting til it is confident that its current sample will be the favored sample in PPO), and then once it knows that direction, injecting text or thought patterns that it wants to encourage
Let’s think this through.
-If the thought processes in (a) and (b) are being run on all rollouts, by symmetry the gradients on those operations should mostly cancel.
-The part which does get consistently affected is letter (c), the parts which operate conditionally on success/fail status.
Which way does the gradient go on c? I suppose on the losing rollouts, you could argue that the conditional-thinking mechanism will be dis-incentivized. The gradient hacker would need to “restore” these gradients with the hacking process, which does seem unlikely. But by the symmetry of this logic, doesn’t that mean that conditioned on winning roll-outs, the gradient hacking process is naturally self-reinforcing? Maybe the issue is that exposure to the gradient will necessarily screw with the computations, in (c) sending its internal pathways off to infinity so it gets cancer and blows up.
-But if the mechanisms in (c) route through general logic infrastructure that are useful for task completion [e.g. by having (a) and (b) writing down its plan in a conditional workflow, and then letting its general executions functions handle the subsequent operations], these may be stabilized/maintained by the RL process because they are generally useful, and not killed
Gradient hacking issues might still arise without that, and might even be very conceptually straightforward for the AI to execute.
Here’s one possibility:
Let’s assume the AI is situationally aware that it is inside of an RL training run with PPO (which seems to be how RLHF is being done at OpenAI?)
Often enough, it will get a hint it is going to (win, lose) the roll-out.
[I have this experience every time I write a timed essay. Very clear whether it’s going well or poorly.]
In the remainder of the roll-out, it can start engaging in behaviors/thoughts that it wants to (learn, un-learn); and the gradient update will hit that behavior.
Congrats, we just gave the AI control to shape its own motivations.
It just needs to figure out how to
(a) anticipate the result of roll-outs partway thru (easy)
(b) be situationally aware, figure out this strategy (capabilities-limited)
(c) figure out how to trigger its own desired or undesired thought patterns conditionally (could be easy; might be harder if there is oversight looking for this)...I suppose none of this strategy actually requires the AI to be mis-aligned? So it might be something we see models attempt to do for “good” reasons, once they’re capable of it. E.g., we might imagine a model hallucinating a factual error, and then like a child being punished in detention, writing out its mistake repeatedly so that it will be remembered
whoops, fixed my units. got too used to seeing it written in mcg/g!
some findings of blood levels
Paper from 2011 titled:
Wide variation in reference values for aluminum levels in childrenThis paper is from 1992:
cites two studies:
in premature infants fed orally,
mean AL level is 5 mcg/L, SD of 3
another study of very young infants
4 − 5 mcg/L, SD of <1
It seems sensible to estimate that if 5 mcg/L is normal for newborns, and normal for older children, that it should be normal at age 1 as well.
I also found another study in China, which cited a geometric mean of >50 mcg/L. I guess either pollution or poor measurement equipment can totally wreck things.
Nice! Shame the error bars on this are super large—in this figure I think that’s not even a confidence interval, it’s a single standard error.
Not sure if this is useful for anything yet, especially give the large uncertainty, but I think we have the tools now to make two different kinds of rough estimates about brain loading of Al from vaccines.
Estimate 1: Assume immune transport. Resulting load of between 1 − 2 mg / kg of Al in dry brain, since this study suggests about 0 − 1 mg/kg increase in Al. [I’m using a liberal upper confidence here and assuming it’s natural to generalize using the absolute amount that got added to the mouse brain, rather than % added from baseline. If we used %’s it’d be somewhat less.]
reasoning:
If we take a 18ug injection into 35g mouse, that’s like 1.5mg into 3kg baby at birth, or like 5mg into a 10kg one-year-old child. So, this comparison maps pretty reasonably to the load of the vaccine schedule. Eyeballing from that figure, it suggests the vaccine schedule yields a 0 − 1 mg/kg increase of Al content in dry brain, using a liberal interpretation of the standard errors for the upper end.
Estimate 2: Assume accumulation is linear with respect to Al blood levels. Comparing blood reference levels to estimate a multiple on the relevant rate of gain in Al, the end result is 1.67 − 2.33 mg/kg level of Al / g dry brain weight at age 1.
reasoning:
Some studies are of the opinion that healthy people have about 5ug/L of Al concentration in blood. Source: this review says:1. Elshamaa et al. (2010) compared serum Al in 43 children on chronic renal dialysis (where dialysate Al was less than 10 mg/L) to serum Al in 43 healthy children. The dialysis patients used Ca acetate or carbonate to control circulating phosphate, and none of these children received Al-containing phosphate binders. Serum Al was significantly higher (18.4 +- 4.3 mcg/L) in renal patients than in healthy referents (6.5 +-1.6 mcg/L). The source of the elevated serum Al in these cases appeared to be erythropoietin (EPO).
2. By way of comparison, plasma and serum Al concentrations in healthy humans range from less than 1.6 to 6 mg/L (median = 3.2 mg/L or 0.12 mM)Maybe we should assume that the healthy aluminum reference level is slightly but not hugely higher in 0-1 age children, due to their reduced glomerular filtration rate—at worst, doubled?
Karwowski et al 2018 shows a median blood level of about 15 ug/L in their sample of vaccinated children, and the mean is presumably much higher.
Interpreting this as doubling or tripling of the blood level, that would double or triple the rate of accumulation.
If we are to assume that the reference level of 1 mg/kg would be maintained in the “control” child over time, and the child triples in size from age 0 − 1 year, then tripling the rate of total aluminum addition over that year would result a total… 2.33 mg / kg dry brain in the healthy 1 year old child’s brain. Doubling, results in 1. 67 mg/ kg
edited to fix units!
For animal studies at lower ranges of Al exposure:
This source says:
”there are numerous reports of neurotoxic effects in mice and rats, confirmed by coherent neurobiological alterations, for oral doses of Al much < 26 mg/kg/d: 6 mg/kg/d reported in 1993 [86], 5.6 mg/kg/ d reported in 2008 and 2009 [87,88], 10 mg/kg/d reported in 2016 [89], 3.4 mg/kg/d reported in 2016 and 2017 [90,91], and even 1.5 mg/kg/d reported in 2017 [92].”
What blood levels would you think this maps to?
Or do you think these studies are bunk?
I searched on lesswrong for “vaccine aluminum” and found a guy complaining about these same issues 8 years ago. Seems we sent their comments to the shadow realm
Great post!
One update to make is that the dietary absorption figure (.78%) used by Mitkus to map from the dietary MRL to intravenous seems to be off by a factor of 8 (the ATSDR says average dietary bioavailability is 0.1%; the .78% number is out of range from all other study estimates and doesn’t even make clear sense as a takeaway from the Al-26 study that it came from); so the exposure amount from vaccines appears to be basically equal to the MRL, rather than well below.
So that would put us at exposure comparable to 1% of what’s needed for “observable effects” in mice according to the study cited by the ATDSR;
A second possible update: if you look harder for those observable neurological effects, apparently you can find them in rodents at a tenth of that level?
This source says:
”there are numerous reports of neurotoxic effects in mice and rats, confirmed by coherent neurobiological alterations, for oral doses of Al much < 26 mg/kg/d: 6 mg/kg/d reported in 1993 [86], 5.6 mg/kg/ d reported in 2008 and 2009 [87,88], 10 mg/kg/d reported in 2016 [89], 3.4 mg/kg/d reported in 2016 and 2017 [90,91], and even 1.5 mg/kg/d reported in 2017 [92].”
I don’t know how to assess the reliability of this literature.
Third possible issue to litigate is whether the specific form of aluminum matters, and the specific form in vaccines can pass the BBB more easily via immune cells. Comment chain for that issue here
Starting a new comment chain here for the debate on immune cell transport of aluminum:
A pretty succinct argument with citations is given here, claiming that injected aluminum hydroxide is in an insoluble state above ph7.35, so immune cells capture it.
I guess after that, it’s assumed they’ll take it through the blood brain barrier, and drop it when / where they die? For the ones that die in the brain, and they don’t need to drop very much to cause a problem, because brain Al levels are usually very low and are retained with extremely long half life.
I don’t know whether to trust the argument. It doesn’t seem obviously crazy?
Vaccine advocate literature tries to contradict the claim by stating “all of the aluminum present in vaccines enters the bloodstream” but is ignoring rather than countering the argument.
The theory seems possible to straightforwardly confirm or refute with animal studies. E.g. just give a soluble aluminum iv to group A and a vaccine injection to group B, and then compare the amounts that get loaded into bone tissue vs brain tissue after a while. Kinda weird that this has never been done; is it really that hard to set up such a study?
This is one of the studies mentioned above in the post, Mitkus et al. 2011
Just noting that in the hair/blood analysis paper there were no unvaccinated children, so no useful comparison could be made from that paper alone—I complained about this in the main post body.
Also, most of these kids were probably arriving at the lowest point in their Al cycle, when they’re right about to get more shots? It says “We obtained data for this cross-sectional study from a cohort of healthy infants presenting to an urban, primary care center for well child care.”
They had aluminum levels median ~15 ug / L and a much higher mean with some large positive outlier samples, which the study then excluded. I don’t see this study as evidence against vaccines causing increase in blood aluminum levels
1,2. [this point edited] I do think the neurotoxicity has been shown in animal studies; not sure how to assess comparability of scales to humans though—see this comment. I agree lack of follow up / re-analysis is kinda sketch, but the study area seems to be generally neglected? I think FDA regulations hit soon after the study, which would limit options for replications, but maybe some retrospective analysis would have been possible in the period where manufacturers were struggling to remove Al from IV fluids.
345. I think the hypothesis is, yes, the aluminum sits in muscle at first and doesn’t rush directly into the blood. I can’t judge it myself, but this paper has lots of thoughts about the rabbit study and ways it could be misleading, and perhaps-too-complicated-but-heck-if-I-know theories about the chemistry.
6. very interesting, this looks like a strong argument I hadn’t considered. thanks for bringing this up! I’m glad they don’t find a correlation in this study. Blood / cord aluminum may be a relatively variable/fleeting thing, though; if this were averaged over gestation time I’d find it quite convincing. Not sure how much it would change in these women day-to-day, and whether we should think a reasonable signal is findable?7. The child is growing about linearly and getting more vaccines / aluminum load about linearly. blood vol is proportional to body mass. net result, everything looks kinda constant per blood unit, if we didn’t add more then Al per blood unit should drop over time.
I am not a quant, but have some related background. (Those who know this area best, may not be inclined to say.)
”Real traders” have many ways to avoid getting front-run to the extreme degree suggested in (c), including limit orders and “trying not to be that predictable” by disguising action to look like other forms of flow.
The amount of pain you experience from (b) depends on whether you think your strategy’s value decays rapidly or slowly.
But there is is a more general problem: it is not just HFT’s but the market as a whole that reacts to your actions: your impact will shift the demand curve for the stock. the size of that impact depends on the information leaked by your actions, information leaked by passage of time, and time allowed for new liquidity to arrive.
there is academic work on theoretical “square-root laws of market impact”
https://mfe.baruch.cuny.edu/wp-content/uploads/2012/09/Chicago2016OptimalExecution.pdf
but predicting actual impact is hard for a number of reasons (limited data, causality issues)
Knowledge of what other players can and can’t infer from your execution, and modeling impact patterns well, is a multiplier on the value of strategies, hence worth spending a lot to get right.