techn ical
typo?
neverix
Karma: 25
I did some similar experiments two months ago, and with to your setup the special tokens show up on the first attempt:
This project tried this.
It is based on this. I changed it to optimize using softmax instead of straight-through estimation and added regularization for the embedded tokens.
Notebook link—this is a version that mimics this post instead of optimizing a single neuron as in the original.
EDIT: github link
sidereal
typo?
This is the whole point of goal misgeneralization. They have experiments (albeit on toy environments that can be explained by the network finding the wrong algorithm), so I’d say quite plausible.
course
coarse?
A0>A1
How is orbit comparison for sets defined?
Seems like quoting doesn’t work for LaTeX, it was definitions 2⁄3. Reading again I saw D2 was indeed applicable to sets.
There are also somewhat principled reasons for using a “fuzzy ellipsoid”, which I won’t explain here.
If you view as 2x learning rate, the ellipsoid contains parameters which will jump straight into the basin under the quadratic approximation, and we assume for points outside the basin the approximation breaks entirely. If you account for gradient noise
in the form of a Gaussian with sigma equal to gradient, the PDF of the resulting point at the basin is equal to the probability a Gaussian parametrized by the ellipsoid at the preceding point.This is wrong, but there is an interpretation of the noise as a Gaussian with variance increasing away from the basin origin.
You could literally go through some giant corpus with an LLM and see which samples have gradients similar to those from training on a spelling task.
We can idealize the outer alignment solution as a logical inductor.
Why outer?
From a conversation on Discord:
Do you have in mind a way to weigh sequential learning into the actual prior?
Dmitry:
good question! We haven’t thought about an explicit complexity measure that would give this prior, but a very loose approximation that we’ve been keeping in the back of our minds could be a Turing machine/Boolean circuit version of the “BIMT” weight penalty from this paper https://arxiv.org/abs/2305.08746 (which they show encourages modularity at least in toy models)
Response:
Hmm, BIMT seems to only be about intra-layer locality. It would certainly encourage learning an ensemble of features, but I’m not sure if it would capture the interesting bit, which I think is the fact that features are built up sequentially from earlier to later layers and changes are only accepted if they improve local loss.
I’m thinking about something like an existence of a relatively smooth scaling law (?) as the criterion.
So, just some smoothness constraint that would basically integrate over paths SGD could take.
Activation additions in generative models
Also related is https://arxiv.org/abs/2210.10960. They use a small neural network to generate steering vectors for the UNet bottleneck in diffusion to edit images using CLIP.
This happens in transformer MLP layers. Note that the hidden dimen
Is the point that transformer MLPs blow up the hidden dimension in the middle?
To clarify, I thought it was about superposition happening inside the projection afterwards.
typo