Daniel Murfet

• Daniel Murfet 3 Apr 2024 8:24 UTC

  2 points

  0

  on: Na­ture is an in­finite sphere whose cen­ter is ev­ery­where and cir­cum­fer­ence is nowhere

  Haha this is so intensely on-brand.

• Daniel Murfet 20 Mar 2024 23:44 UTC

  2 points

  0

  on: Are ex­treme prob­a­bil­ities for P(doom) epistem­i­cally jus­tifed?

  The kind of superficial linear extrapolation of trendlines can be powerful, perhaps more powerful than usually accepted in many political/​social/​futurist discussions. In many cases, succesful forecasters by betting on some high level trend lines often outpredict ‘experts’.
  
  But it’s a very non-gears level model. I think one should be very careful about using this kind of reasoning when for tail-events.
  e.g. this kind of reasoning could lead one to reject development of nuclear weapons.

  Agree. In some sense you have to invent all the technology before the stochastic process of technological development looks predictable to you, almost by definition. I’m not sure it is reasonable to ask general “forecasters” about questions that hinge on specific technological change. They’re not oracles.

Stage­wise Devel­op­ment in Neu­ral Networks

• Daniel Murfet 16 Mar 2024 19:21 UTC

  5 points

  0

  in reply to: habryka’s comment on: More peo­ple get­ting into AI safety should do a PhD

  Do you mean the industry labs will take people with MSc and PhD qualifications in CS, math or physics etc and retrain them to be alignment researchers, or do you mean the labs will hire people with undergraduate degrees (or no degree) and train them internally to be alignment researchers?
  
  I don’t know how OpenAI or Anthropic look internally, but I know a little about Google and DeepMind through friends, and I have to say the internal incentives and org structure don’t strike me as really a very natural environment for producing researchers from scratch.

• Daniel Murfet 16 Mar 2024 8:44 UTC

  19 points

  3

  on: More peo­ple get­ting into AI safety should do a PhD

  I think many early-career researchers in AI safety are undervaluing PhDs.

  I agree with this. To be blunt, it is my impression from reading LW for the last year that a few people in this community seem to have a bit of a chip on their shoulder Re: academia. It certainly has its problems, and academics love nothing more than pointing them out to each other, but you face your problems with the tools you have, and academia is the only system for producing high quality researchers that is going to exist at scale over the next few years (MATS is great, I’m impressed by what Ryan and co are doing, but it’s tiny).
  
  I would like to see many more academics in CS, math, physics and adjacent areas start supervising students in AI safety, and more young people go into those PhDs. Also, more people with PhDs in math and physics transitioning to AI safety work.
  
  One problem is that many of the academics who are willing to supervise PhD students in AI safety or related topics are evaporating into industry positions (subliming?). There are also long run trends that make academia relatively less attractive than it was in the past (e.g. rising corporatisation) even putting aside salary comparisons, and access to compute. So I do worry somewhat about how many PhD students in AI safety adjacent fields can actually be produced per year this decade.
  

• Daniel Murfet 12 Mar 2024 0:45 UTC

  5 points

  3

  in reply to: Nisan’s comment on: Sim­ple ver­sus Short: Higher-or­der de­gen­er­acy and er­ror-correction

  This comment of mine is a bit cheeky, since there are plenty of theoretical computer scientists who think about characterising terms as fixed points, and logic programming is a whole discipline that is about characterising the problem rather than constructing a solution, but broadly speaking I think it is true among less theoretically-minded folks that “program” means “thing constructed step by step from atomic pieces”.

• Daniel Murfet 11 Mar 2024 18:58 UTC

  22 points

  12

  in reply to: ryan_greenblatt’s comment on: Sim­ple ver­sus Short: Higher-or­der de­gen­er­acy and er­ror-correction

  Maybe I can clarify a few points here:

  • A statistical model is regular if it is identifiable and the Fisher information matrix is everywhere nondegenerate. Statistical models where the prediction involves feeding samples from the input distribution through neural networks are not regular.

  • Regular models are the ones for which there is a link between low description length and low free energy (i.e. the class of models which the Bayesian posterior tends to prefer are those that are assigned lower description length, at the same level of accuracy).

  • It’s not really accurate to describe regular models as “typical”, especially not on LW where we are generally speaking about neural networks when we think of machine learning.

  • It’s true that the example presented in this post is, potentially, not typical (it’s not a neural network nor is it a standard kind of statistical model). So it’s unclear to what extent this observation generalises. However, it does illustrate the general point that it is a mistake to presume that intuitions based on regular models hold for general statistical models.

  • A pervasive failure mode in modern ML is to take intuitions developed for regular models, and assume they hold “with some caveats” for neural networks. We have at this point many examples where this leads one badly astray, and in my opinion the intuition I see widely shared here on LW about neural network inductive biases and description length falls into this bucket.

  • I don’t claim to know the content of those inductive biases, but my guess is that it is much more interesting and complex than “something like description length”.

Sim­ple ver­sus Short: Higher-or­der de­gen­er­acy and er­ror-correction

Ti­maeus’s First Four Months

• Daniel Murfet 30 Jan 2024 0:35 UTC

  3 points

  0

  in reply to: Wuschel Schulz’s comment on: A short ‘deriva­tion’ of Watan­abe’s Free En­ergy Formula

  Yes, good point, but if the prior is positive it drops out of the asymptotic as it doesn’t contribute to the order of vanishing, so you can just ignore it from the start.

• Daniel Murfet 29 Jan 2024 23:49 UTC

  4 points

  0

  on: A short ‘deriva­tion’ of Watan­abe’s Free En­ergy Formula

  There was a sign error somewhere, you should be getting + lambda and - (m-1). Regarding the integral from 0 to 1, since the powers involved are even you can do that and double it rather than −1 to 1 (sorry if this doesn’t map exactly onto your calculation, I didn’t read all the details).

• Daniel Murfet 16 Jan 2024 9:47 UTC

  7 points

  0

  on: What’s Gen­eral-Pur­pose Search, And Why Might We Ex­pect To See It In Trained ML Sys­tems?

  There is some preliminary evidence in favour of the view that transformers approximate a kind of Bayesian inference in-context (by which I mean something like, they look at in-context examples and process them to represent in their activations something like a Bayesian posterior for some “inner” model based on those examples as samples, and then predict using the predictive distribution for that Bayesian posterior). I’ll call the hypothesis that this is taking place “virtual Bayesianism”.
  
  I’m not saying you should necessarily believe that, for current generation transformers. But fwiw I put some probability on it, and if I had to predict one significant capability advance in the next generation of LLMs it would be to predict that virtual Bayesianism becomes much stronger (in-context learning being a kind of primitive pre-cursor).
  
  Re: the points in your strategic upshots. Given the above, the following question seems quite important to me: putting aside transformers or neural networks, and just working in some abstract context where we consider Bayesian inference on a data distribution that includes sequences of various lengths (i.e. the kinds of distribution that elicits in-context learning), is there a general principle of Bayesian statistics according to which general-purpose search algorithms tend to dominate the Bayesian posterior?

• Daniel Murfet 16 Jan 2024 1:56 UTC

  6 points

  0

  on: Three Types of Con­straints in the Space of Agents

  In mathematical terms, what separates agents that could arise from natural selection from a generic agent?
  
  To ask a more concrete question, suppose we consider the framework of DeepMind’s Population Based Training (PBT), chosen just because I happen to be familiar with it (it’s old at this point, not sure what the current thing is in that direction). This method will tend to produce a certain distribution over parametrised agents, different from the distribution you might get by training a single agent in traditional deep RL style. What are the qualitative differences in these inductive biases?

• Daniel Murfet 11 Jan 2024 3:29 UTC

  1 point

  0

  in reply to: Michael Pearce’s comment on: You’re Mea­sur­ing Model Com­plex­ity Wrong

  This is an open question. In practice it seems to work fine even at strict saddles (i.e. things where there are no negative eigenvalues in the Hessian but there are still negative directions, i.e. they show up at higher than second order in the Taylor series), in the sense that you can get sensible estimates and they indicate something about the way structure is developing, but the theory hasn’t caught up yet.

• Daniel Murfet 17 Dec 2023 22:58 UTC

  1 point

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  I think there’s no such thing as parameters, just processes that produce better and better approximations to parameters, and the only “real” measures of complexity have to do with the invariants that determine the costs of those processes, which in statistical learning theory are primarily geometric (somewhat tautologically, since the process of approximation is essentially a process of probing the geometry of the governing potential near the parameter).
  
  From that point of view trying to conflate parameters $w_1,w_2$ such that $p\left(x|w_1\right)\approx p\left(x|w_2\right)$ is naive, because $w_1,w_2$ aren’t real, only processes that produce better approximations to them are real, and so the $\frac{\partial }{\partial w}$ derivatives of $p\left(x|w_1\right),p\left(x|w_2\right)$ which control such processes are deeply important, and those could be quite different despite $p\left(x|w_1\right)\approx p\left(x|w_2\right)$ being quite similar.
  
  So I view “local geometry matters” and “the real thing are processes approximating parameters, not parameters” as basically synonymous.

• Daniel Murfet 27 Nov 2023 18:32 UTC

  1 point

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  You might reconstruct your sacred Jeffries prior with a more refined notion of model identity, which incorporates derivatives (jets on the geometric/​statistical side and more of the algorithm behind the model on the logical side).

• Daniel Murfet 27 Nov 2023 18:26 UTC

  3 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  Except nobody wants to hear about it at parties.

  You seem to do OK…

  If they only would take the time to explain things simply you would understand.

  This is an interesting one. I field this comment quite often from undergraduates, and it’s hard to carve out enough quiet space in a conversation to explain what they’re doing wrong. In a way the proliferation of math on YouTube might be exacerbating this hard step from tourist to troubadour.

• Daniel Murfet 27 Nov 2023 18:21 UTC

  7 points

  6

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  As a supervisor of numerous MSc and PhD students in mathematics, when someone finishes a math degree and considers a job, the tradeoffs are usually between meaning, income, freedom, evil, etc., with some of the obvious choices being high/​low along (relatively?) obvious axes. It’s extremely striking to see young talented people with math or physics (or CS) backgrounds going into technical AI alignment roles in big labs, apparently maximising along many (or all) of these axes!
  
  Especially in light of recent events I suspect that this phenomenon, which appears too good to be true, actually is.

• Daniel Murfet 27 Nov 2023 18:16 UTC

  6 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  Please develop this question as a documentary special, for lapsed-Starcraft player homeschooling dads everywhere.

• Daniel Murfet 22 Nov 2023 17:53 UTC

  1 point

  0

  on: Public Call for In­ter­est in Math­e­mat­i­cal Alignment

  Thanks for setting this up!