tailcalled

Karma: 6,408

tailcalled 1 May 2024 17:39 UTC
2 points
0
on: Mechanistically Eliciting Latent Behaviors in Language Models
I wonder if a similar technique could form the foundation for a fully general solution to the alignment problem. Like mathematically speaking all this technique needs is a vector-to-vector function, and it’s not just layer-to-layer relationships that can be understood as a vector-valued function; the world as a function of the policy is also vector-valued.

I.e. rather than running a search to maximize some utility function, a model-based agent could run a search for small changes in policy that have a large impact on the world. If one can then taxonomize, constrain and select between these impacts, one might be able to get a highly controllable AI.

Obviously there’s some difficulties here because the activations are easier to search over since we have an exact way to calculate them. But that’s a capabilities question rather than an alignment question.

tailcalled 1 May 2024 16:19 UTC
2 points
0
in reply to: Andrew Mack’s comment on: Mechanistically Eliciting Latent Behaviors in Language Models
The singular vectors of the Jacobian between two layers seems more similar to what you’re doing in the OP than the Hessian of the objective function does? Because the Hessian of the objective function sort of forces it all to be mediated by the final probabilities, which means it discounts directions in activation space that don’t change the probabilities yet, but would change the probabilities if the change in activations was scaled up beyond infinitesimal.

Edit: wait, maybe I misunderstood, I assumed by the objective function you meant some cross-entropy on the token predictions, but I guess in-context it’s more likely you meant the objective function for the magnitude of change in later layer activations induced by a given activation vector?

tailcalled 1 May 2024 8:08 UTC
2 points
0
in reply to: TurnTrout’s comment on: Mechanistically Eliciting Latent Behaviors in Language Models
I think one characteristic of steering vectors constructed this way is that they are allowed to be off-manifold, so they don’t necessarily tell us how the networks currently work, rather than how they can be made to work with adaptations.

For the past for weeks, I’ve been thinking about to interpret networks on-manifold. The most straightforward approach I could come up with was to restrict oneself to the space of activations that actually occur for a given prompt, e.g. by performing SVD of the token x activation matrix for a given layer, and then restricting oneself to changes that occur along the right singular vectors of this.

My SVD idea might improve things, but I didn’t get around to testing it because I eventually decided that it wasn’t good enough for my purposes because 1) it wouldn’t keep you on-manifold enough because it could still introduce unnatural placement of information and exaggerated features, 2) given that transformers are pretty flexible and you can e.g. swap around layers, it felt unclean to have a method that’s this strongly dependent on the layer structure.

A followup idea I’ve been thinking about but haven’t been able to be satisfied with is, projections. Like if you pick some vector u, and project the activations (or weights, in my theory, but you work more with activations so let’s consider activations, it should be similar) onto u, and then subtract off that projection from the original activations, you get “the activations with u removed”, which intuitively seems like it would better focus on “what the network actually does” as opposed to “what the network could do if you added something more to it”.

Unfortunately after thinking for a while, I started thinking this actually wouldn’t work. Let’s say the activations a = x b + y c, where b is a large activation vector that ultimately doesn’t have an effect on the final prediction, and c is a small activation vector that does have an effect. If you have some vector d that the network doesn’t use at all, you could then project away sqrt(1/2) (b-d), which would introduce the d vector into the activations.

Another idea I’ve thought about is, suppose you do SVD of the activations. You would multiply the feature half of the SVD with the weights used to compute the KQV matrices, and then perform SVD of that, which should get you the independent ways that one layer affects the next layer. One thing I in particular wonder about is, if you start doing this from the output layer, and proceed backwards, it seems like this would have the effect of “sparsifying” the network down to only the dimensions which matter for the final output, which seems like it should assist in interpretability and such. But it’s not clear it interacts nicely with the residual network element.

tailcalled 1 May 2024 7:46 UTC
3 points
0
on: Mechanistically Eliciting Latent Behaviors in Language Models
How important is it to use full-blown gradient descent to train them? Could one instead take the first singular vector for the Jacobian between the neural network layers, and get something that works similarly well?

tailcalled 1 May 2024 7:42 UTC
2 points
0
on: Mechanistically Eliciting Latent Behaviors in Language Models
For the purposes of my prediction market, I’m thinking this probably counts as part of the Shard Theory program? Since Alex Turner was involved as a mentor and activation vectors are kind of shard-theorist and so on?

Possibly this is because being biased in favor of unsupervised learning, but my interest is piqued by this finding.

tailcalled 28 Apr 2024 22:21 UTC
7 points
1
in reply to: Seth Herd’s comment on: [Aspiration-based designs] 1. Informal introduction
See also: satisficers tend to seek power: instrumental convergence through retargetability / parametrically retargetable decision-makers tend to seek power.

tailcalled 28 Apr 2024 8:09 UTC
12 points
3
on: So What’s Up With PUFAs Chemically?

Ideally I’d like someone to just sample the McDonald’s cooking oil every few days through a few cycles of oil changes, but I doubt this will happen.

This seems like a huge civilizational inadequacy to me. This would be informative at large scale, and the cost would be very small.

tailcalled 28 Apr 2024 7:49 UTC
3 points
0
in reply to: ChristianKl’s comment on: Losing Faith In Contrarianism
Definitely relevant to figure out what’s true when one is only talking about the object level, but the OP was about how trustworthy contrarians are compared to the mainstream rather than simply being about the object level.

tailcalled 27 Apr 2024 17:29 UTC
0 points
−2
in reply to: ChristianKl’s comment on: Losing Faith In Contrarianism
I feel like if there’s one side arguing the genetic gap is x, and one side arguing the genetic gap is 0, the natural dichotomization is whether the genetic gap is larger or smaller than x/2.

tailcalled 27 Apr 2024 14:15 UTC
2 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Why I stopped being into basin broadness
Do you have ab outline of how SLT answers this?

tailcalled 26 Apr 2024 7:44 UTC
8 points
4
in reply to: David Hornbein’s comment on: The first future and the best future
If you go slower, you have more time to find desirable mechanisms. That’s pretty much it I guess.

tailcalled 26 Apr 2024 6:11 UTC
3 points
0
on: Losing Faith In Contrarianism
I’m convinced by the mainstream view on COVID origins and medicine.

I’m ambivalent on education—I guess if done well, it’d consistently have good effects, and that currently, it on average has good effects, but also the effect varies a lot from person to person, so simplistic quantitative reviews don’t tell you much. When I did an epistemic spot check on Caplan’s book, it failed terribly (it cited a supposedly-ingenious experiment that university didn’t improve critical thinking, but IMO the experiment had terrible psychometrics).

I don’t know enough about sleep research to disagree with Guzey on the basis of anything but priors. In general, I wouldn’t update much on someone writing a big review, because often reviews include a lot of crap information.

I might have to read Jayman’s rebuttal of B-W genetic IQ differences in more detail, but at first glance I’m not really convinced by it because it seems to focus on small sample sizes in unusual groups, so it’s unclear how much study noise, publication bias and and sampling bias effects things. At this point I think indirect studies are getting obsolete and it’s becoming more and more feasible to just directly measure the racial genetic differences in IQ.

However I also think HBDers have a fractal of bad takes surrounding this, because they deny the phenotypic null hypothesis and center non-existent abstract personality traits like “impulsivity” or “conformity” in their models.

Why I stopped being into basin broadness

tailcalled25 Apr 2024 20:47 UTC

14 points

3 comments2 min readLW link

tailcalled 25 Apr 2024 19:12 UTC
4 points
0
in reply to: Algon’s comment on: Examples of Highly Counterfactual Discoveries?
Yes.

tailcalled 25 Apr 2024 19:11 UTC
2 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Examples of Highly Counterfactual Discoveries?
- The RLCT = $λ$ first-order term for in-distribution generalization error and also Bayesian learning (technically the ‘Bayesian free energy’). This justifies the name of ‘learning coefficient’ for lambda. I emphasize that these are mathematically precise statements that have complete proofs, not conjectures or intuitions.
Link(s) to your favorite proof(s)?
Also, do these match up with empirical results?
- Knowing a little SLT will inoculate you against many wrong theories of deep learning that abound in the literature. I won’t be going in to it but suffice to say that any paper assuming that the Fischer information metric is regular for deep neural networks or any kind of hierarchichal structure is fundamentally flawed. And you can be sure this assumption is sneaked in all over the place. For instance, this is almost always the case when people talk about Laplace approximation.
I have a cached belief that the Laplace approximation is also disproven by ensemble studies, so I don’t really need SLT to inoculate me against that. I’d mainly be interested if SLT shows something beyond that.
it can be estimated at scale.
As I read the empirical formulas in this paper, they’re roughly saying that a network has a high empirical learning coefficient if an ensemble of models that are slightly less trained on average have a worse loss than the network.
But then so they don’t have to retrain the models from scratch, they basically take a trained model, and wiggle it around using Gaussian noise while retraining it.
This seems like a reasonable way to estimate how locally flat the loss landscape is. I guess there’s a question of how much the devil is in the details; like whether you need SLT to derive an exact formula that works.
I guess I’m still not super sold on it, but on reflection that’s probably partly because I don’t have any immediate need for computing basin broadness. Like I find the basin broadness theory nice to have as a model, but now that I know about it, I’m not sure why I’d want/need to study it further.
There was a period where I spent a lot of time thinking about basin broadness. I guess I eventually abandoned it because I realized the basin was built out of a bunch of sigmoid functions layered on top of each other, but the generalization was really driven by the neural tangent kernel, which in turn is mostly driven by the Jacobian of the network outputs for the dataset as a function of the weights, which in turn is mostly driven by the network activations. I guess it’s plausible that SLT has the best quantities if you stay within the basin broadness paradigm. 🤔

tailcalled 25 Apr 2024 13:39 UTC
2 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Examples of Highly Counterfactual Discoveries?
Newton’s Universal Law of Gravitation was the first highly accurate model of things falling down that generalized beyond the earth, and it is also the second-most computationally applicable model of things falling down that we have today.

Are you saying that singular learning theory was the first highly accurate model of breadth of optima, and that it’s one of the most computationally applicable ones we have?

tailcalled 25 Apr 2024 6:32 UTC
3 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Examples of Highly Counterfactual Discoveries?
I would say “the thing that contains the inheritance particles” rather than “the inheritance particle”. “Particulate inheritance” is a technical term within genetics and it refers to how children don’t end up precisely with the mean of their parents’ traits (blending inheritance), but rather with some noise around that mean, which particulate inheritance asserts is due to the genetic influence being separated into discrete particles with the children receiving random subsets of their parent’s genes. The significance of this is that under blending inheritance, the genetic variation between organisms within a species would be averaged away in a small number of generations, which would make evolution by natural selection ~impossible (as natural selection doesn’t work without genetic variation).

tailcalled 25 Apr 2024 6:20 UTC
2 points
−1
in reply to: Garrett Baker’s comment on: Examples of Highly Counterfactual Discoveries?
Isn’t singular learning theory basically just another way of talking about the breadth of optima?

tailcalled 24 Apr 2024 17:38 UTC
2 points
0
in reply to: JBlack’s comment on: David Udell’s Shortform
The tricky part is, on the margin I would probably use various shortcuts, and it’s not clear where those shortcuts end short of just getting knowledge beamed into my head.

I already use LLMs to tell me facts, explain things I’m unfamiliar with, handle tedious calculations/coding, generate simulated data/brainstorming and summarize things. Not much, because LLMs are pretty bad, but I do use them for this and I would use them more on the margin.

tailcalled 24 Apr 2024 12:18 UTC
2 points
0
in reply to: Ben’s comment on: Subjective Questions Require Subjective information
$ω$ isn’t a reference frame; rather, if $ψ$ is a world then $ω^{- 1} ({ψ})$ aka ${ι \in I ∣ ω (ι) = ψ}$ are the reference frames for $ψ$ .

Essentially when dealing with generalized reference frames that contain answers to questions such as “who are you?”, the possible reference frames are going to depend on the world (because you can only be a real person, and which real people there are depends on what the world is). As such, “reference frames” don’t make sense in isolation, rather one needs a (world, reference frame) pair, which is what I call an “interpretation”.

tailcalled

Why I stopped be­ing into basin broadness

Why I stopped being into basin broadness