Jsevillamol

• Jsevillamol 25 Apr 2024 6:26 UTC

  4 points

  0

  on: Bayesian in­fer­ence with­out priors

  I’ve been tempted to do this sometime, but I fear the prior is performing one very important role you are not making explicit: defining the universe of possible hypothesis you consider.

  In turn, defining that universe of probabilities defines how bayesian updates look like. Here is a problem that arises when you ignore this: https://​​www.lesswrong.com/​​posts/​​R28ppqby8zftndDAM/​​a-bayesian-aggregation-paradox

• Jsevillamol 23 Jan 2024 22:47 UTC

  2 points

  0

  in reply to: johnswentworth’s comment on: Re­vis­it­ing al­gorith­mic progress

  shrug

  I think this is true to an extent, but a more systematic analysis needs to back this up.

  For instance, I recall quantization techniques working much better after a certain scale (though I can’t seem to find the reference...). It also seems important to validate that techniques to increase performance apply at large scales. Finally, note that the frontier of scale is growing very fast, so even if these discoveries were done with relatively modest compute compared to the frontier, this is still a tremendous amount of compute!

• Jsevillamol 23 Jan 2024 21:22 UTC

  2 points

  0

  in reply to: johnswentworth’s comment on: Re­vis­it­ing al­gorith­mic progress

  even a pause which completely stops all new training runs beyond current size indefinitely would only ~double timelines at best, and probably less

  I’d emphasize that we currently don’t have a very clear sense of how algorithmic improvement happens, and it is likely mediated to some extent by large experiments, so I think is more likely to slow timelines more than this implies.

• Jsevillamol 24 Nov 2023 20:39 UTC

  2 points

  in reply to: paato’s comment on: Anal­y­sis of World Records in Speedrun­ning [LINKPOST]

  I agree! I’d be quite interested in looking at TAS data, for the reason you mentioned.

• Jsevillamol 28 Aug 2023 22:41 UTC

  4 points

  0

  on: The Ev­i­dence for Ques­tion De­com­po­si­tion is Weak

  I think Tetlock and cia might have already done some related work?

  Question decomposition is part of the superforecasting commandments, though I can’t recall off the top of my head if they were RCT’d individually or just as a whole.

  ETA: This is the relevant paper (h/​t Misha Yagudin). It was not about the 10 commandments. Apparently those haven’t been RCT’d at all?

• Jsevillamol 28 Aug 2023 9:00 UTC

  4 points

  in reply to: gwern’s comment on: Anal­y­sis of World Records in Speedrun­ning [LINKPOST]

  We actually wrote a more up to date paper here

  https://​​arxiv.org/​​abs/​​2304.10004

• Jsevillamol 21 Aug 2023 19:54 UTC

  7 points

  4

  on: Rice’s The­o­rem says that AIs can’t de­ter­mine much from study­ing AI source code

  I cowrote a detailed response here

  https://​​www.cser.ac.uk/​​news/​​response-superintelligence-contained/​​

  Essentially, this type of reasoning proves too much, since it implies we cannot show any properties whatsoever of any program, which is clearly false.

• Jsevillamol 14 Aug 2023 7:01 UTC

  14 points

  0

  in reply to: Daniel Kokotajlo’s comment on: AGI is eas­ier than robotaxis

  Here is some data through Matthew Barnett and Jess Riedl

  Number of cumulative miles driven by Cruise’s autonomous cars is growing as an exponential at roughly 1 OOM per year.

  https://​​twitter.com/​​MatthewJBar/​​status/​​1690102362394992640

• Jsevillamol 6 Jun 2023 10:27 UTC

  6 points

  3

  in reply to: meijer1973’s comment on: Yud­kowsky vs Han­son on FOOM: Whose Pre­dic­tions Were Bet­ter?

  That is to very basic approximation correct.

  Davidson’s takeoff model illustrates this point, where a “software singularity” happens for some parameter settings due to software not being restrained to the same degree by capital inputs.

  I would point out however that our current understanding of how software progress happens is somewhat poor. Experimentation is definitely a big component of software progress, and it is often understated in LW.

  More research on this soon!

• Jsevillamol 4 Jun 2023 9:35 UTC

  6 points

  2

  in reply to: habryka’s comment on: Yud­kowsky vs Han­son on FOOM: Whose Pre­dic­tions Were Bet­ter?

  algorithmic progress is currently outpacing compute growth by quite a bit

  This is not right, at least in computer vision. They seem to be the same order of magnitude.

  Physical compute has growth at 0.6 OOM/​year and physical compute requirements have decreased at 0.1 to 1.0 OOM/​year, see a summary here or a in depth investigation here

  Another relevant quote

  Algorithmic progress explains roughly 45% of performance improvements in image classification, and most of this occurs through improving compute-efficiency.

• Jsevillamol 9 May 2023 8:29 UTC

  LW: 2 AF: 1

  AF

  in reply to: Zenin Easa Panthakkalakath’s comment on: What’s the back­ward-for­ward FLOP ra­tio for Neu­ral Net­works?

  $t$ is not a transpose! It is the timestep $t$. We are raising $\beta$ to the $t$-th power.

Power laws in Speedrun­ning and Ma­chine Learning

• Jsevillamol 21 Apr 2023 6:54 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Edouard Harris’s comment on: An­nounc­ing Epoch’s dash­board of key trends and figures in Ma­chine Learning

  Thanks!

  Our current best guess is that this includes costs other than the amortized compute of the final training run.

  If no extra information surfaces we will add a note clarifying this and/​or adjust our estimate.

• Jsevillamol 20 Apr 2023 21:29 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Neel Nanda’s comment on: An­nounc­ing Epoch’s dash­board of key trends and figures in Ma­chine Learning

  Thanks Neel!

  The difference between tf16 and FP32 comes to a x15 factor IIRC. Though also ML developers seem to prioritise other characteristics than cost effectiveness when choosing GPUs like raw performance and interconnect, so you can’t just multiply the top price performance we showcase by this factor and expect that to match the cost performance of the largest ML runs today.

  More soon-ish.

• Jsevillamol 13 Apr 2023 19:25 UTC

  6 points

  0

  in reply to: Tao Lin’s comment on: An­nounc­ing Epoch’s dash­board of key trends and figures in Ma­chine Learning

  Because there is more data available for FP32, so it’s easier to study trends there.

  We should release a piece soon about how the picture changes when you account for different number formats, plus considering that most runs happen with hardware that is not the most cost-efficient.

An­nounc­ing Epoch’s dash­board of key trends and figures in Ma­chine Learning

• Jsevillamol 4 Apr 2023 6:38 UTC

  17 points

  5

  in reply to: dxu’s comment on: Com­mu­ni­cat­ing effec­tively un­der Knigh­tian norms

  Note that Richard is not treating knightian uncertainty as special and unquantifiable, but instead is giving examples of how to treat it like any other uncertainty, that he is explicitly quantifying and incorporating in his predictions.

  I’d prefer calling Richard’s “model error” to separate the two, but I’m also okay appropriating the term as Richard did to point to something coherent.

• Jsevillamol 21 Mar 2023 21:47 UTC

  2 points

  0

  on: The Wizard of Oz Prob­lem: How in­cen­tives and nar­ra­tives can skew our per­cep­tion of AI developments

  To my knowledge, we currently don’t have a way of translating statements about “loss” into statements about “real-world capabilities”.

  Now we do!

• Jsevillamol 9 Feb 2023 2:00 UTC

  2 points

  0

  in reply to: dust_to_must’s comment on: A time-in­var­i­ant ver­sion of Laplace’s rule

  My intuition is that it’s not a great approximation in those cases, similar to how in regular Laplace the empirical approximation is not great when you have eg N<5

  Id need to run some calculations to confirm that intuition though.

• Jsevillamol 6 Feb 2023 14:24 UTC

  LW: 4 AF: 2

  0

  AF

  in reply to: Jsevillamol’s comment on: SolidGoldMag­ikarp (plus, prompt gen­er­a­tion)

  Here is a 2012 meme about SolidGoldMagikarp

  https://​​9gag.com/​​gag/​​3389221