Spencer Becker-Kahn

• Spencer Becker-Kahn 24 Mar 2022 15:56 UTC

  LW: 9 AF: 3

  AF

  on: Job Offer­ing: Help Com­mu­ni­cate Infrabayesianism

  This post caught my eye as my background is in mathematics and I was, in the not-too-distant past, excited about the idea of rigorous mathematical AI alignment work. My mind is still open to such work but I’ll be honest, I’ve since become a bit less excited than I was. In particular, I definitely “bounced off” the existing write-ups on Infrabayesianism and now without already knowing what it’s all about, it’s not clear it’s worth one’s time. So, at the risk of making a basic or even cynical point: The remuneration of the proposed job could be important for getting attention/​ incentivising people on-the-fence.

• Spencer Becker-Kahn 25 Mar 2022 10:45 UTC

  9 points

  AF

  in reply to: Algon’s comment on: Job Offer­ing: Help Com­mu­ni­cate Infrabayesianism

  It could also work here. But I do feel like pointing out that the bounty format has other drawbacks. Maybe it works better when you want a variety of bitesize contributions, like various different proposals? I probably wouldn’t do work like Abram proposes—quite a long and difficult project, I expect—for the chance of winning a prize, particularly if the winner(s) were decided by someone’s subjective judgement.

• Spencer Becker-Kahn 21 Apr 2022 10:32 UTC

  LW: 11 AF: 4

  AF

  in reply to: Steven Byrnes’s comment on: Call For Distillers

  I agree i.e. I also (fairly weakly) disagree with the value of thinking of ‘distilling’ as a separate thing. Part of me wants to conjecture that it’s comes from thinking of alignment work predominantly as mathematics or a hard science in which the standard ‘unit’ is a an original theorem or original result which might be poorly written up but can’t really be argued against much. But if we think of the area (I’m thinking predominantly about more conceptual/​theoretical alignment) as a ‘softer’, messier, ongoing discourse full of different arguments from different viewpoints and under different assumptions, with counter-arguments, rejoinders, clarifications, retractions etc. that takes place across blogs, papers, talks, theorems, experiments etc that all somehow slowly works to produce progress, then it starts to be less clear what this special activity called ‘distilling’ really is.
  
  Another relevant point, but one which I won’t bother trying to expand on much here, is that a research community assimilating—and then eventually building on—complex ideas can take a really long time.
  
  [At risk of extending into a rant, I also just think the term is a bit off-putting. Sure, I can get the sense of what it means from the word and the way it is used—it’s not completely opaque or anything—but I’d not heard it used regularly in this way until I started looking at the alignment forum. What’s really so special about alignment that we need to use this word? Do we think we have figured out some new secret activity that is useful for intellectual progress that other fields haven’t figured out? Can we not get by using words like “writing” and “teaching” and “explaining”?]

• Spencer Becker-Kahn 26 Apr 2022 9:26 UTC

  3 points

  AF

  in reply to: adamShimi’s comment on: In­tu­itions about solv­ing hard problems

  I broadly agree with Richard’s main point, but I also do agree with this comment in the sense that I am not confident that the example of Turing compared with e.g. Einstein is completely fair/​accurate.
  
  One thing I would say in response to your comment, Adam, is that I don’t usually see the message of your linked post as being incompatible with Richard’s main point. I think one usually does have or does need productive mistakes that don’t necessarily or obviously look like they are robust partial progress. But still, often when there actually is a breakthrough, I think it can be important to look for this “intuitively compelling” explanation. So one thing I have in mind is that I think it’s usually good to be skeptical if a claimed breakthrough seems to just ‘fall out’ of a bunch of partial work without there being a compelling explanation after the fact.

• Spencer Becker-Kahn 26 Apr 2022 11:26 UTC

  3 points

  in reply to: adamShimi’s comment on: In­tu­itions about solv­ing hard problems

  Yes I think you understood me correctly. In which case I think we more or less agree in the sense that I also think it may not be productive to use Richard’s heuristic as a criterion for which research directions to actually pursue.

An ob­ser­va­tion about Hub­inger et al.’s frame­work for learned optimization

• Spencer Becker-Kahn 19 May 2022 10:39 UTC

  1 point

  in reply to: Ryan Kidd’s comment on: SERI ML Align­ment The­ory Schol­ars Pro­gram 2022

  Hi Ryan, do you still plan for results to come out by May 27? And for those who are successful for the next stage to start June 6th etc.? (That’s what is says on the FAQ on the website still).

• Spencer Becker-Kahn 20 May 2022 11:25 UTC

  4 points

  on: Deep­Mind is hiring for the Scal­able Align­ment and Align­ment Teams

  Hi there,
  
  Given that you’ve described various ‘primarily conceptual’ projects on the Alignment Team, and given the distinction between Scientists and Engineers, one aspect that I’m unsure about is roughly: Would you expect a Research Scientist on the Alignment Team to necessarily have a minimum level of practical ML knowledge? Are you able to say any more about that? e.g. Would they have to pass a general Deep Mind coding test or something like that?

• Spencer Becker-Kahn 20 May 2022 20:05 UTC

  1 point

  in reply to: Rohin Shah’s comment on: Deep­Mind is hiring for the Scal­able Align­ment and Align­ment Teams

  Thanks Rohin!

• Spencer Becker-Kahn 24 May 2022 9:47 UTC

  2 points

  in reply to: Evan R. Murphy’s comment on: An ob­ser­va­tion about Hub­inger et al.’s frame­work for learned optimization

  Thanks for the comments and pointers!

• Spencer Becker-Kahn 25 May 2022 12:36 UTC

  LW: 12 AF: 5

  AF

  on: In­for­ma­tion Loss --> Basin flatness

  This was pretty interesting and I like the general direction that the analysis goes in. I feel it ought to be pointed out that what is referred to here as the key result is a standard fact in differential geometry called (something like) the submersion theorem, which in turn is essentially an application of the implicit function theorem.
  
  I think that your setup is essentially that there is an $N$-dimensional parameter space, let’s call it $\Theta$ say, and then for each element $x_i$ of the training set, we can consider the function $f_i:\Theta ⟶\text{Output Space}=:O$ which takes in a set of parameters (i.e. a model) and outputs whatever the model does on training data point $x_i$. We are thinking of both $\Theta$ and $O^k$ as smooth (or at least sufficiently differentiable) spaces (I take it).
  
  A contour plane is a level set of one of the $f_i$, i.e. a set of the form

  $$\{\theta \in \Theta :f_i(\theta )=o\},$$

  for some $o\in O$ and $i\in \{1,\dots ,k\}$. A behavior manifold is a set of the form

  $$\bigcap _{i=1}^k\{\theta \in \Theta :f_i=o\}$$

  for some $o\in O$.
  
  A more concise way of viewing this is to define a single function $f:\Theta ⟶O^k$ and then a behavior manifold is simply a level set of this function. The map $f$ is a submersion at $\theta \in \Theta$ if the Jacobian matrix at $\theta$ is a surjective linear map. The Jacobian matrix is what you call $G^T$ I think (because the Jacobian is formed with each row equal to a gradient vector with respect to one of the output coordinates). It doesn’t matter much because what matters to check the surjectivity is the rank. Then the standard result implies that given $o\in O$, if $f$ is a submersion in a neighbourhood of a point ${\theta }_0\in f^{-1}\left(o\right)$, then $f^{-1}\left(o\right)$ is a smooth $(N-k)$-dimensional submanifold in a neighbourhood of ${\theta }_0$ .
  
  Essentially, in a neighbourhood of a point at which the Jacobian of $f$ has full rank, the level set through that point is an $(N-k)$-dimensional smooth submanifold.
  
  Then, yes, you could get onto studying in more detail the degeneracy when the Jacobian does not have full rank. But in my opinion I think you would need to be careful when you get to claim 3. I think the connection between loss and behavior is not spelled out in enough detail: Behaviour can change while loss could remain constant, right? And more generally, in exactly which directions do the implications go? Depending on exactly what you are trying to establish, this could actually be a bit of a ‘tip of the iceberg’ situation though. (The study of this sort of thing goes rather deep; Vladimir Arnold et al. wrote in their 1998 book: “The theory of singularities of smooth maps is an apparatus for the study of abrupt, jump-like phenomena—bifurcations, perestroikas (restructurings), catastrophes, metamorphoses—which occur in systems depending on parameters when the parameters vary in a smooth manner”.)
  
  Similarly when you say things like “Low rank $G$ indicates information loss”, I think some care is needed because the paragraphs that follow seem to be getting at something more like: If there is a certain kind of information loss in the early layers of the network, then this leads to low rank $G$. It doesn’t seem clear that low rank $G$ is necessarily indicative of information loss?

• Spencer Becker-Kahn 26 May 2022 12:58 UTC

  LW: 3 AF: 2

  AF

  in reply to: Vivek Hebbar’s comment on: In­for­ma­tion Loss --> Basin flatness

  Thanks for the substantive reply.
  
  First some more specific/​detailed comments: Regarding the relationship with the loss and with the Hessian of the loss, my concern sort of stems from the fact that the domains/​codomains are different and so I think it deserves to be spelled out. The loss of a model with parameters $\theta \in \Theta$ can be described by introducing the actual function that maps the behavior to the real numbers, right? i.e. given some actual function $l:O^k\to R$ we have:

  $$L:\Theta \stackrel{f}{⟶}{O}^k\stackrel{l}{⟶}R$$

  i.e. it’s $l$ that might be something like MSE, but the function $L$″ is of course more mysterious because it includes the way that parameters are actually mapped to a working model. Anyway, to perform some computations with this, we are looking at an expression like

  $$L\left(\theta \right)=l(f\left(\theta \right))$$

  We want to differentiate this twice with respect to $\theta$ essentially. Firstly, we have

  $$\nabla L\left(\theta \right)=\nabla l\left(f\right(\theta \left)\right)Jf\left(\theta \right)$$

  where—just to keep track of this—we’ve got:

  $$(1\times N)\text{vector}=[(1\times k)\text{vector}][(k\times N)\text{matrix}]$$

  Or, using ‘coordinates’ to make it explicit:

  $$\frac{\partial }{\partial {\theta }_i}L\left(\theta \right)=\nabla l(f\left(\theta \right))\cdot \frac{\partial f}{\partial {\theta }_i}=\sum _{p=1}^k{\nabla }^{p}l(f\left(\theta \right))\cdot \frac{\partial f^p}{\partial {\theta }_i}$$

  for $i=1,\dots ,N$. Then for $j=1,\dots ,N$ we differentiate again:

  $$\frac{{\partial }^2}{\partial {\theta }_j\partial {\theta }_i}L\left(\theta \right)=\sum _{p=1}^k\sum _{q=1}^k{\nabla }^q{\nabla }^{p}l(f\left(\theta \right))\frac{\partial f^q}{\partial {\theta }_j}\frac{\partial f^p}{\partial {\theta }_i}+\sum _{p=1}^k{\nabla }^{p}l(f\left(\theta \right))\frac{\partial f^p}{\partial {\theta }_j\partial {\theta }_i}$$

  Or,

  $$Hess(L)\left(\theta \right)=Jf\left(\theta {)}^T[Hess(l)\right(f\left(\theta \right)\left)]Jf\right(\theta )+\nabla l(f(\theta \left))D^{2}f\right(\theta )$$

  This is now at the level of $(N\times N)$ matrices. Avoiding getting into any depth about tensors and indices, the $D^{2}f$ term is basically a $(N\times N\times k)$ tensor-type object and it’s paired with $\nabla l$ which is a $(1\times k)$ vector to give something that is $(N\times N)$.

  So what I think you are saying now is that if we are at a local minimum for $l$, then the second term on the right-hand side vanishes (because the term includes the first derivatives of $l$, which are zero at a minimum). You can see however that if the Hessian of $l$ is not a multiple of the identity (like it would be for MSE), then the claimed relationship does not hold, i.e. it is not the case that in general, at a minima of $l$, the Hessian of the loss is equal to a constant times $(Jf{)}^{T}Jf$. So maybe you really do want to explicitly assume something like MSE.
  
  I agree that assuming MSE, and looking at a local minimum, you have $rank\left(Hess\right(L\left)\right)=rank\left(Jf\right)$ .

  (In case it’s of interest to anyone, googling turned up this recent paper https://​​openreview.net/​​forum?id=otDgw7LM7Nn which studies pretty much exactly the problem of bounding the rank of the Hessian of the loss. They say: “Flatness: A growing number of works [59–61] correlate the choice of regularizers, optimizers, or hyperparameters, with the additional flatness brought about by them at the minimum. However, the significant rank degeneracy of the Hessian, which we have provably established, also points to another source of flatness — that exists as a virtue of the compositional model structure —from the initialization itself. Thus, a prospective avenue of future work would be to compare different architectures based on this inherent kind of flatness.”)
  
  Some broader remarks: I think these are nice observations but unfortunately I think generally I’m a bit confused/​unclear about what else you might get out of going along these lines. I don’t want to sound harsh but just trying to be plain: This is mostly because, as we can see, the mathematical part of what you have said is all very simple, well-established facts about smooth functions and so it would be surprising (to me at least) if some non-trivial observation about deep learning came out from it. In a similar vein, regarding the “cause” of low-rank G, I do think that one could try to bring in a notion of “information loss” in neural networks, but for it to be substantive one needs to be careful that it’s not simply a rephrasing of what it means for the Jacobian to have less-than-full rank. Being a bit loose/​informal now: To illustrate, just imagine for a moment a real-valued function on an interval. I could say it ‘loses information’ where its values cannot distinguish between a subset of points. But this is almost the same as just saying: It is constant on some subset...which is of course very close to just saying the derivative vanishes on some subset. Here, if you describe the phenomena of information loss as concretely as being the situation where some inputs can’t be distinguished, then (particularly given that you have to assume these spaces are actually some kind of smooth/​differentiable spaces to do the theoretical analysis), you’ve more or less just built into your description of information loss something that looks a lot like the function being constant along some directions, which means there is a vector in the kernel of the Jacobian. I don’t think it’s somehow incorrect to point to this but it becomes more like just saying ‘perhaps one useful definition of information loss is low rank G’ as opposed to linking one phenomenon to the other.
  
  Sorry for the very long remarks. Of course this is actually because I found it well worth engaging with. And I have a longer-standing personal interest in zero sets of smooth functions!

  What links here?

  • Spencer Becker-Kahn's comment on Basin broad­ness de­pends on the size and num­ber of or­thog­o­nal features by CallumMcDougall (29 Aug 2022 12:36 UTC; 4 points)

• Spencer Becker-Kahn 27 May 2022 8:29 UTC

  LW: 3 AF: 1

  AF

  in reply to: Vivek Hebbar’s comment on: In­for­ma­tion Loss --> Basin flatness

  Thanks again for the reply.
  
  In my notation, something like $\nabla l$ or $Jf$ are functions in and of themselves. The function $\nabla l$ evaluates to zero at local minima of $l$.
  
  In my notation, there isn’t any such thing as ${\nabla }_{f}l$.
  
  But look, I think that this is perhaps getting a little too bogged down for me to want to try to neatly resolve in the comment section, and I expect to be away from work for the next few days so may not check back for a while. Personally, I would just recommend going back and slowly going through the mathematical details again, checking every step at the lowest level of detail that you can and using the notation that makes most sense to you.

• Spencer Becker-Kahn 22 Jun 2022 9:53 UTC

  9 points

  1

  on: The in­or­di­nately slow spread of good AGI con­ver­sa­tions in ML

  I’m a little sheepish about trying to make a useful contribution to this discussion without spending a lot of time thinking things through but I’ll give it a go anyway. There’s a fair amount that I agree with here, including that there is by now a lot of introductory resources. But regarding the following:

  
  (I do think it’s possible to create a much better intro resource than any that exist today, but ‘we can do much better’ is compatible with ‘it’s shocking that the existing material hasn’t already finished the job’.)

  I feel like I want to ask: Do you really find it “shocking”? My experience with explaining things to more general audiences leaves me very much of the opinion that it is by default an incredibly slow and difficult process to get unusual, philosophical, mathematical, or especially technical ideas to permeate. I include ‘average ML engineer’ as something like a “more general audience” member relative to MIRI style AGI Alignment theory. I guess I haven’t thought it about it much but presumably there exist ideas/​arguments that are way more mainstream, also very important, and with way more written about them that people still somehow, broadly speaking, don’t engage with or understand?
  
  I also don’t really understand how the point that is being made in the quote from Inadequate Equilibria is supposed to work. Perhaps in the book more evidence is provided for when “the silence broke”, but the Hawking article was before the release of Superintelligencea and then the Musk tweet was after it and was reacting to it(!) .. So I guess I’m sticking up for AGI x-risk respectability politics a bit here because surely I might also use essentially this same anecdote to support the idea that boring old long-form academic writing that clearly lays things out in as rigorous a way as possible is actually more the root cause that moved the needle here? Even if it ultimately took the engagement of Musk’s off the cuff tweets, Gates, or journalists etc., they wouldn’t have had something respectable enough to bounce off had Bostrom not given them the book.

• Spencer Becker-Kahn 22 Jun 2022 11:29 UTC

  5 points

  2

  in reply to: Rob Bensinger’s comment on: The in­or­di­nately slow spread of good AGI con­ver­sa­tions in ML

  Thanks for the nice reply.
  

  I do buy the explanations I listed in the OP (and other, complementary explanations, like the ones in Inadequate Equilbria), and I think they’re sufficient to ~fully make sense of what’s going on. So I don’t feel confused about the situation anymore. By “shocking” I meant something more like “calls for an explanation”, not “calls for an explanation, and I don’t have an explanation that feels adequate”. (With added overtones of “horrifying”.)

  
  Yeah, OK, I think that helps clarify things for me.

  
  
  As someone who was working at MIRI in 2014 and watched events unfolding, I think the Hawking article had a negligible impact and the Musk stuff had a huge impact. Eliezer might be wrong about why Hawking had so little impact, but I do think it didn’t do much.

  Maybe we’re misunderstanding each other here. I don’t really doubt what you’re saying there^ i.e. I am fully willing to believe that the Hawking thing had negligible impact and the Musk tweet had a lot. I’m more pointing to why Musk had a lot rather than why Hawking had little: Trying to point out that since Musk was reacting to Superintelligence, one might ask whether he could have had a similar impact without Superintelligence. And so maybe the anecdote could be used as evidence that Superintelligence was really the thing that helped ‘break the silence’. However, Superintelligence feels way less like “being blunt” and “throwing a brick” and—at least from the outside—looks way more like the “scripts, customs, and established protocols” of “normal science” (i.e. Oxford philosophy professor writes book with somewhat tricky ideas in it, published by OUP, reviewed by the NYT etc. etc.) and clearly is an attempt to make unusual ideas sound “sober and serious”. So I’m kind of saying that maybe the story doesn’t necessarily argue against the possibility of doing further work like that that—i.e. writing books that manage to stay respectable and manage to “speak accurately and concretely about the future of AI without sounding like a sci-fi weirdo”(?)

Notes on Learn­ing the Prior

• Spencer Becker-Kahn 26 Jul 2022 16:13 UTC

  LW: 4 AF: 4

  2

  AF

  in reply to: Geoffrey Irving’s comment on: Notes on Learn­ing the Prior

  Thanks very much Geoffrey; glad you liked the post. And thanks for the interesting extra remarks.

A brief note on Sim­plic­ity Bias

• Spencer Becker-Kahn 29 Aug 2022 12:36 UTC

  LW: 4 AF: 2

  0

  AF

  on: Basin broad­ness de­pends on the size and num­ber of or­thog­o­nal features

  I wrote out the Hessian computation in a comment to one of Vivek’s posts. I actually had a few concerns with his version and I could be wrong but I also think that there are some issues here. (My notation is slightly different because for me the sum over $x$ was included in the function I called “$l$”, but it doesn’t affect my main point).
  
  I think the most concrete thing is that the function $f$ - i.e. the `input-output’ function of a neural network—should in general have a vector output, but you write things like

  $$\frac{d^{2}l}{d{f}^2}$$

  without any further explanation or indices. In your main computation it seems like it’s being treated as a scalar.
  
  Since we never change the labels or the dataset, on can drop the explicit dependence on $y\left(x\right)$ from our notation for $l$. Then if the network has $p$ neurons in the final layer, the codomain of the function $f$ is $R^p$ (unless I’ve misunderstood what you are doing?). So to my mind we have:

  $$l(f\left(x\right))=l(f^1\left(x\right),f^2\left(x\right),\dots ,f^p\left(x\right)).$$

  Going through the computation in full using the chain rule (and a local minimum of the loss function $l$) one would get something like:

  $$Hess(L)\left(\theta \right)=\sum _{x}Jf\left(\theta {)}^T[Hess(l)\right(f\left(\theta \right)\left)]Jf\right(\theta )$$

  Vivek wanted to suppose that $Hess\left(l\right)$ were equal to the identity matrix, or a multiple thereof, which is the case for mean squared loss. But without such an assumption, I don’t think that the term

  $$\sum _{x}Jf\left(\theta {)}^{T}Jf\right(\theta )$$

  appears (this is the matrix you describe as containing “the $L_2$ inner products of the features over the training data set.”)
  
  Another (probably more important but higher-level) issue is basically: What is your definition of ‘feature’? I could say: Have you not essentially just defined `feature’ to be something like `an entry of $Jf\left(\theta \right)$’? Is the example not too contrived in that sense it clearly supposes that $f$ has a very special form (in particular it is linear in the $\Theta$ variables so that the derivatives are not functions of $\Theta$.)

• Spencer Becker-Kahn 30 Aug 2022 20:33 UTC

  6 points

  3

  in reply to: Tom Lieberum’s comment on: Tak­ing the pa­ram­e­ters which seem to mat­ter and ro­tat­ing them un­til they don’t

  I’m not at liberty to share it directly but I am aware that Anthropic have a draft of small toy models with hand-coded synthetic data showing superposition very cleanly. They go as far as saying that searching for an interpretable basis may essentially be mistaken.