SWE/MLE/AI Safety
submarat
Karma: 51
Transformers Don’t Need LayerNorm at Inference Time: Implications for Interpretability
We attempted 1.a “diversity measure based on sentence embeddings” and found that for Llama3.2-1B the diversity appears to decay after the cusp value for R; picking R at highest average diversity was a decent heuristic for finding meaningful steering vectors. The Llama model starts to produce highly repetitive output past the cusp. We demonstrated that repetitive completions were considered similar by our chosen sentence embedding model (SentenceTransformer all-mpnet-base-v2). Using “sum of variances” vs “mean of cosine similarities” didn’t seem to matter.
Thank you for reading and the suggestions. I enumerate for easier reference:
1. Find max activating examples
2. Understand which vectors are being found
3. Attempt to scale up
4. Finding useful applications once scaled up
For 1. do you mean:
Take an input (from a bank of random example prompts)
Do forward pass on unsteered model
Extract the activations at the target layer
Compute the dot product between these activations and the steering vector
Use this dot product value as a measure of how strongly this example activates the behavior associated with the steering vector
Am I following correctly?