I think this is missing the ‘one country conquers another’ analogy, which is sort of halfway between your two scenarios, and is probably the closest to what I think the takeover would look like, at least at first. They would disempower us, arrest and kill a bunch, and take our stuff. There may be a subjugation (like what the British did in India) or a large-scale clearing-out of the natives (like what happened in North America).
Mikewins
Karma: 269
I want to chime in to warn against doing what I did. When I was in grad school (theoretical physics), I freaked out after ChatGPT and started trying to reorient my work to have more alignment implications. I should be clear that this wasn’t a radical reorientation, more ‘what can I do that will be of interest to my advisor and help me get a job in physics academia while also being alignment-related?’ With the benefit of hindsight, I think it’s safe to say that all the work I did in that period has no object-level value, and if I’d been ten times more productive and completed all the alignment projects I started (I have a lot of paper ideas)… I still wouldn’t have produced anything very useful. That’s not to say the time was completely wasted. I learned a lot about ML theory, which comes in handy. But given that I ended up leaving academia anyways, sticking around for three years after ChatGPT was definitely a career mistake.
You’re a different person than me (citation needed), your life might be different, you might be subject to visa constraints or a two-body problem or something. But there’s a lot to be said for leaving academia. And, if you can’t leave, you should choose your alignment problems based on your all-things-considered view of what’s most useful, not try to 80-20 it. If you’re anything like me, you’re more likely to 20-80 it.
I agree that a lot of random short texts are going to be preserved (I think inscriptions of statues will be most useful). But I’m pretty sure all the nutrition facts on food packages get bleached away (I also spent way too much time worrying about vanity license plates; those actually rust pretty quickly).
I hadn’t thought about coffee cups as a historical record, though. Good one.
I can’t imagine being passionate about event management, but the people who are get snapped up by the Generator Residency https://generatorresidency.org/
It’s certainly possible to estimate random networks. We have a whole paper about it (https://www.alignment.org/blog/mechanistic-estimation-for-wide-random-mlps/). We don’t think the algorithm in that paper is ideal, and we’re hoping that the good people of the internet could help us improve it.
As for whether trained neural networks are easier or harder, it probably depends on the order of quantifiers. I would guess that for a lot of simple networks (eg the ones in AlgZoo), estimating wouldn’t be that much harder than for random MLPs, or possibly even easier. But coming up with an algorithm that works on any trained network has got to be strictly harder (since it would include untrained as a limit).
Lol, thanks.
Some of the equations near the bottom of section 4 don’t look like they’re rendering properly.
Does the 1/sqrt(N) error for SGD assume single-pass? It seems like if we’re bottlenecked on few data points we can use multi-pass and do nearly as well as bayesian (at least for half-spaces).
Honestly for me it’s more of a strike against RNNs. Real deep neural networks that have been trained don’t have this property, so it’s a bridge we’re going to need to cross at some point regardless. From a derisking point of view I’d kind of like to get to that point ASAP. There’s a lot of talk about looking at random boolean circuits (which very obviously don’t have this property), narrow MLPs, or even jumping all the way to wide MLPs trained in some sort of mean-field/maximum update regime that gets rid of it.
I am affiliated with ARC and played a major role in the MLP stuff
I’m loosely familiar with Greg Yang’s work, and very familiar with the ‘Neural Network Gaussian Process’ canon. It’s definitely relevant, especially as an intuition pump, but it tends to answer a different question. They answer ‘what is the distribution of quantities x y and z over the set of all NNs’ where quantities x y and z might be some preactivation on specific inputs. Knowing that they are jointly gaussian with such-and-such covariance has been a powerful intuition pump for us. But the main problem we want is an algorithm that takes in a specific NN with specific weights and tells us about the average over inputs.
I’ve found that this distinction is a powerful antimeme, and every time I give a presentation on the topic I have a slide on the difference between averaging over x versus over theta. By the end of the talk the audience is clamoring to recommend I read Principles of Deep Learning Theory (which is lovely if you want to improve on NNGP, but not relevant to calculating averages for a specific value of theta.).
I am affiliated with ARC and played a major role in the MLP stuff
The particular infinite sum discussed in this post is used for approximating MLPs with just one hidden layer, so things like vanishing gradients can’t matter.
We are now doing work on deeper MLPs. In this case, the vanishing gradients story definitely does seem relevant. We definitely don’t fully understand every detail, but I’ll mouth off anyways.
On one hand, there are hyperparameter choices where gradients explode. It turns out that in this regime, matching sampling: exponentially large gradients mean that the average is a sum over exponentially many tiny and more-or-less independent regions, so it’s exponentially small and you can beat sampling by just answering zero until your compute budget is exponential. Even then you might not have the budget to go through all of input space, but we have ideas on what you could do.
On the other hand, there are regions where gradients vanish. Here, also, it seems like things work out. Since the inputs to neurons in later layers concentrate, you can do very well by just approximating the later non-linearities as linearizations around the concentrating point.
Then there’s the so-called ‘critical hyperparameters’ right in the middle. It turns out that for MLPs the gradients vanish (albeit more slowly) even at criticality. Back-of-the-envelope, it seems to go to zero just fast enough.
There are other convergence-related questions (in particular, a lot of these series are so-called ‘asymptotic series’ which officially never converge), and at smaller epsilons/higher compute budget those issues come into play as well, but I don’t think they’re connected to gradients.
Maybe I’m missing something, but I suspect that the huge amount of rollouts involved in training the model would pose a much larger risk than the inference to solve Navier-Stokes.