Adam Jermyn(Adam Jermyn)

Karma: 1,189

Adam Jermyn 7 Jun 2022 18:18 UTC
28 points
16
in reply to: wachichornia’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
I believe 80000 hours has a lot more coaching capacity now, it might be worth asking again!

Adam Jermyn 14 Jun 2022 0:55 UTC
18 points
on: Contra EY: Can AGI destroy us without trial & error?
A crux here seems to be the question of how well the AGI can simulate physical systems. If it can simulate them perfectly, there’s no need for real-world R&D. If its simulations are below some (high) threshold fidelity, it’ll need actors in the physical world to conduct experiments for it, and that takes human time-scales.
A big point in favor of “sufficiently good simulation is possible” is that we know the relevant laws of physics for anything the AGI might need to take over the world. We do real-world experiments because we haven’t managed to write simulation software that implements these laws at sufficiently high fidelity and for sufficiently complex systems, and because the compute cost of doing so is enormous. But in 20 years, an AGI running on a giant compute farm might both write more efficient simulation codes and have enough compute to power them.

Adam Jermyn 19 Jul 2023 23:05 UTC
LW: 16 AF: 9
3
AF
on: Alignment Grantmaking is Funding-Limited Right Now
This matches my impression. At EAG London I was really stunned (and heartened!) at how many skilled people are pivoting into interpretability from non-alignment fields.

Adam Jermyn 27 Sep 2022 13:31 UTC
LW: 13 AF: 9
11
AF
on: Simulators
This is great! I really like your “prediction orthogonality thesis”, which gets to the heart of why I think there’s more hope in aligning LLM’s than many other models.
One point of confusion I had. You write:
Optimizing toward the simulation objective notably does not incentivize instrumentally convergent behaviors the way that reward functions which evaluate trajectories do. This is because predictive accuracy applies optimization pressure deontologically: judging actions directly, rather than their consequences. Instrumental convergence only comes into play when there are free variables in action space which are optimized with respect to their consequences.[25]Constraining free variables by limiting episode length is the rationale of myopia ; deontological incentives are ideally myopic. As demonstrated by GPT, which learns to predict goal-directed behavior, myopic incentives don’t mean the policy isn’t incentivized to account for the future, but that it should only do so in service of optimizing the present action (for predictive accuracy)[26].
I don’t think I agree with this conclusion (or maybe I don’t understand the claim). I agree that myopic incentives don’t mean myopic behavior, but they also don’t imply that actions are chosen myopically? For instance I think a language model could well end up sacrificing some loss on the current token if that made the following token easier to predict. I’m not aware of examples of this happening, but it seems consistent with the way these models are trained.
In the limit a model could sacrifice a lot of loss upfront if that allowed it to e.g. manipulate humans into giving it resources with which to better predict later tokens.

Adam Jermyn 8 Nov 2022 20:50 UTC
LW: 12 AF: 4
15
AF
on: Mysteries of mode collapse due to RLHF
Just to say I really enjoyed reading this post, and found it helpful as a way to get a sense of what mode collapse looks like in practice.

Adam Jermyn 27 May 2022 12:52 UTC
11 points
in reply to: David Johnston’s comment on: An academic journal is just a Twitter feed
Zenodo is a pretty accepted solution to data-durability in academia (https://zenodo.org). There’s no reason you couldn’t upload papers there (and indeed they host papers/conference proceedings/etc.). Uploads get assigned a DOI and get versioning, get indexed for citation purposes, etc.
If I were starting a journal it would probably look like “Zenodo for hosting, some AirTable or GitHub workflow for (quick) refereeing/editorial workflow.”

Adam Jermyn 29 Sep 2022 23:09 UTC
LW: 10 AF: 7
5
AF
on: Where I currently disagree with Ryan Greenblatt’s version of the ELK approach
For what it’s worth I found this writeup informative and clear. So lowering your standards still produced something useful (at least to me).

Adam Jermyn 7 Jun 2022 18:17 UTC
10 points
5
in reply to: DirectedEvolution’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
This is a slightly orthogonal answer, but “humans who understand the risks” have a big human-bias-incentive to believe that AGI is far off (in that it’s aversive to thinking that bad things are going to happen to you personally).
A more direct answer is: There is a wide range of people who say they work on “AI safety” but almost none of them work on “Avoiding doom from AGI”. They’re mostly working on problems like “make the AI more robust/less racist/etc.”. These are valuable things to do, but to the extent that they compete with the “Avoid doom” researchers for money/status/influence they have an incentive to downplay the odds of doom. And indeed this happens a fair amount with e.g. articles on how “Avoid doom” is a distraction from problems that are here right now.

Adam Jermyn 7 Jun 2022 18:13 UTC
9 points
3
in reply to: ekka’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
Among organizations, both OpenAI and DeepMind are aiming at AGI and seem confident they will get there. I don’t know their internal timelines and don’t know if they’ve stated them...

Adam Jermyn 1 Dec 2022 18:21 UTC
LW: 8 AF: 6
2
AF
in reply to: Neel Nanda’s comment on: Multi-Component Learning and S-Curves
I agree with both of your rephrasings and I think both add useful intuition!
Regarding rank 2, I don’t see any difference in behavior from rank 1 other than the “bump” in alignment that Lawrence mentioned. Here’s an example:
This doesn’t happen in all rank-2 cases but is relatively common. I think usually each vector grows primarily towards 1 or the other target. If two vectors grow towards the same target then you get this bump where one of them has to back off and align more towards a different target [at least that’s my current understanding, see my reply to Lawrence for more detail!].
What happens in a cross-entropy loss style setup, rather than MSE loss? IMO cross-entropy loss is a better analogue to real networks. Though I’m confused about the right way to model an internal sub-circuit of the model. I think the exponential decay term just isn’t there?
What does a cross-entropy setup look like here? I’m just not sure how to map this toy model onto that loss (or vice-versa).
How does this interact with weight decay? This seems to give an intrinsic exponential decay to everything
Agreed! I expect weight decay to (1) make the converged solution not actually minimize the original loss (because the weight decay keeps tugging it towards lower norms) and (2) accelerate the initial decay. I don’t think I expect any other changes.
How does this interact with softmax? Intuitively, softmax feels “S-curve-ey”
I’m not sure! Do you have a setup in mind?
How does this with interact with Adam? In particular, Adam gets super messy because you can’t just disentangle things. Even worse, how does it interact with AdamW?
I agree this breaks my theoretical intuition. Experimentally most of the phenomenology is the same, except that the full-rank (rank 100) case regains a plateau.
Here’s rank 2:
rank 10:
(maybe there’s more ‘bump’ formation here than with SGD?)
rank 100:
It kind of looks like the plateau has returned! And this replicates across every rank 100 example I tried, e.g.
The plateau corresponds to a period with a lot of bump formation. If bumps really are a sign of vectors competing to represent different chunks of subspace then maybe this says that Adam produces more such competition (maybe by making different vectors learn at more similar rates?).
I’d be curious if you have any intuition about this!

Adam Jermyn 4 Oct 2022 22:36 UTC
LW: 8 AF: 4
5
AF
in reply to: Donald Hobson’s comment on: Smoke without fire is scary
It seems plausible to me that there could be models capable enough to realize they should hide some capabilities, but not so capable that they tile the universe in paperclips. The right-hand side of the graph is meant to reflect such a model.

Adam Jermyn 3 Dec 2023 3:38 UTC
7 points
0
in reply to: Sam Marks’s comment on: How useful is mechanistic interpretability?
In Towards Monosemanticity we also did a version of this experiment, and found that the SAE was much less interpretable when the transformer weights were randomized (https://transformer-circuits.pub/2023/monosemantic-features/index.html#appendix-automated-randomized).

Adam Jermyn 14 Oct 2023 12:43 UTC
LW: 7 AF: 4
4
AF
in reply to: Zvi’s comment on: RSPs are pauses done right
Anthropic’s RSP includes evals after every 4x increase in effective compute and after every 3 months, whichever comes sooner, even if this happens during training, and the policy says that these evaluations include fine-tuning.

Adam Jermyn 4 Dec 2022 16:45 UTC
7 points
0
in reply to: sapphire’s comment on: Semi-conductor/AI Stock Discussion.
And: having a lot of capital could be very useful in the run up to TAI. Eg for pursuing/funding safety work.

Adam Jermyn 7 Jun 2022 18:05 UTC
7 points
2
in reply to: michael_mjd’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
Applying to Redwood or Anthropic seems like a great idea. My understanding is that they’re both looking for aligned engineers and scientists and are both very aligned orgs. The worst case seems like they (1) say no or (2) don’t make an offer that’s enough for you to keep your lifestyle (whatever that means for you). In either case you haven’t lost much by applying, and you definitely don’t have to take a job that puts you in a precarious place financially.

Adam Jermyn 9 Dec 2022 3:58 UTC
6 points
4
on: AI Safety Seems Hard to Measure
In fact, it’s not 100% clear that AI systems could learn to deceive and manipulate supervisors even if we deliberately tried to train them to do it. This makes it hard to even get started on things like discouraging and detecting deceptive behavior.
Plausibly we already have examples of (very weak) manipulation, in the form of models trained with RLHF saying false-but-plausible-sounding things, or lying and saying they don’t know something (but happily providing that information in different contexts). [E.g. ChatGPT denies having information about how to build nukes, but will also happily tell you about different methods for Uranium isotope separation.]

Adam Jermyn 30 Nov 2022 22:26 UTC
LW: 6 AF: 4
0
AF
on: The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable
This is really interesting! One extension that comes to mind: SVD will never recover a Johnson-Lindenstrauss packing, because SVD can only return as many vectors as the rank of the relevant matrix. But you can do sparse coding to e.g. construct an overcomplete basis of vectors such that typical samples are sparse combinations of those vectors. Have you tried/considered trying something like that?

Adam Jermyn 8 Nov 2022 18:30 UTC
LW: 6 AF: 4
0
AF
on: Applying superintelligence without collusion
I like the framing of “how can we disrupt collusion?”, but I’m a little skeptical that some of the solutions are practical. Specifically:

B2: Collusion can happen between agents that are copies of each other, who then have no reason to defect (so long as they employ a decision theory with a broad enough cartesian boundary). If we knew we could engineer systems with narrow decision theories (e.g. “Even though that’s a copy of me, I’ll pretend it’s a different agent”) I’d feel much better here, but as far as I know we don’t know how to do that. I’d be excited to hear about work in that direction.

B4: I’m skeptical that we can really constrain communication meaningfully between e.g. models interacting over the internet. And I do expect a lot of models to get deployed over the internet, because that’s where the economic value is.

B6: Some of the most powerful models are LLM’s, which get trained on a large chunk of all of the text ever written. It seems they’ll share an awful lot of knowledge, though maybe you’re considering very specific information they’ll lack?

C1: I don’t think this is born out by the models in use today, but I don’t know for sure. E.g. I would bet that text-davinci does better than a combination of smaller models with diverse training processes, but I don’t know if that comparison has been done.

Adam Jermyn 17 Aug 2022 21:15 UTC
LW: 6 AF: 5
4
AF
on: Human Mimicry Mainly Works When We’re Already Close
Another angle: number of bits of optimization required is a direct measure of “how far out of distribution” we need to generalize.
I think it’s useful to distinguish between the amount of optimization we ask the model to do versus the unlikelihood of the world we ask it to simulate.
For instance, I can condition on something trivial like “the weather was rainy on ⁸⁄₁₄, sunny on ⁸⁄₁₅, rainy on ⁸⁄₁₆...”. This specifies a very unlikely world, but so long as the pattern I specify is plausible it doesn’t require much optimization on the part of the model or take me far out of distribution. There can be many, many plausible patterns like this because the weather is a chaotic system and so intrinsically has a lot of uncertainty, so there’s actually a lot of room to play here.
That’s a silly example, but there are more useful ones. Suppose I condition on a sequence of weather patterns (all locally plausible) that affect voter turnout in key districts such that politicians get elected who favor policies that shift the world towards super-tight regulatory regimes on AI. That let’s me push down the probability that there’s a malicious AI in the simulated world without requiring the model itself to perform crazy amounts of optimization.
Granted, when the model tries to figure out what this world looks like, there’s a danger that it says “Huh, that’s a strange pattern. I wonder if there’s some master-AGI engineering the weather?” and simulates that world. That’s possible, and the whole question is about whether the things you conditioned on pushed down P(bad AGI controls the world) faster than they made the world-writ-large unlikely.

Adam Jermyn 25 Jul 2022 1:36 UTC
6 points
1
in reply to: cata’s comment on: Hiring Programmers in Academia
In my experience there are lots of things in academia that would be improved by someone writing software, but even below market rate there often isn’t any money for it. Funding agencies are (at least in the US) weirdly uninterested in funding software development unless it’s done by someone with the title “grad student” or perhaps “postdoc”. That’s not to say opportunities don’t exist, but the market failure goes deeper than being unwilling to pay market rate.