Nathan Helm-Burger

Karma: 2,364

AI alignment researcher, ML engineer. Masters in Neuroscience.

I believe that cheap and broadly competent AGI is attainable and will be built soon. This leads me to have timelines of around 2024-2027. Here’s an interview I gave recently about my current research agenda. I think the best path forward to alignment is through safe, contained testing on models designed from the ground up for alignability trained on censored data (simulations with no mention of humans or computer technology). I think that current ML mainstream technology is close to a threshold of competence beyond which it will be capable of recursive self-improvement, and I think that this automated process will mine neuroscience for insights, and quickly become far more effective and efficient. I think it would be quite bad for humanity if this happened in an uncontrolled, uncensored, un-sandboxed situation. So I am trying to warn the world about this possibility.

See my prediction markets here:

https://manifold.markets/NathanHelmBurger/will-gpt5-be-capable-of-recursive-s?r=TmF0aGFuSGVsbUJ1cmdlcg

I also think that current AI models pose misuse risks, which may continue to get worse as models get more capable, and that this could potentially result in catastrophic suffering if we fail to regulate this.

I now work for SecureBio on AI-Evals.

relevant quote:

“There is a powerful effect to making a goal into someone’s full-time job: it becomes their identity. Safety engineering became its own subdiscipline, and these engineers saw it as their professional duty to reduce injury rates. They bristled at the suggestion that accidents were largely unavoidable, coming to suspect the opposite: that almost all accidents were avoidable, given the right tools, environment, and training.” https://www.lesswrong.com/posts/DQKgYhEYP86PLW7tZ/how-factories-were-made-safe

Neural net / decision tree hybrids: a potential path toward bridging the interpretability gap

Nathan Helm-Burger23 Sep 2021 0:38 UTC

21 points

2 comments12 min readLW link

Nathan Helm-Burger 3 Oct 2021 2:01 UTC
10 points
on: The 2021 Less Wrong Darwin Game
Hey so, this tickled my curiosity and I went exploring for similar projects just to see what’s out there. I came across a couple youtube videos I enjoyed enough to feel they were worth passing on. So just in case you are also fascinated by similar sorts of sims...

Nathan Helm-Burger 3 Oct 2021 3:25 UTC
3 points
in reply to: Steven Byrnes’s comment on: Book review: “A Thousand Brains” by Jeff Hawkins
I also studied neuroscience for several years, and Jeff’s first book was a major inspiration for me beginning that journey. I agree very much with the points you make in this review and in https://www.lesswrong.com/posts/W6wBmQheDiFmfJqZy/brain-inspired-agi-and-the-lifetime-anchor
Since we seem to be more on the same page than most other people I’ve talked to about this, perhaps a collaboration between us could be fruitful. Not sure on what exactly, but I’ve been thinking about how to transition into direct work on AGI safety since updating in the past couple years that it is potentially even closer than I’d thought.
As for the brain division, I also think of the neocortex and basal ganglia working together as a subsystem. I actually strengthened my belief in their tight coupling in my last year of grad school when I learned more about the striatum gating thoughts (not just motor actions), and the cerebellum smoothing abstract thoughts (not just motor actions). So now I envision it more like the brain is thousands of mostly repetitive loops of little neocortex region → little basal ganglia region → little hindbrain region → little cerebellum region → same little neocortex region, and that these loops also communicate sideways a bit in each region but mostly in the neocortex. With this understanding, I feel like I can’t at all get behind Jeff H’s idea of safely separating out the neocortical functions from the mid/hind brain functions. I think that an effective AGI general learning algorithm is likely to have to have at least some aspects of those little loops, with striatum gating and cerebellar smoothing, and hippocampal memory linkages.… I do think that the available data in neuroscience is very close, if not already, sufficient for describing the necessary algorithm and it’s just a question of a bit more focused work on sorting out the necessary parts from the unneeded complexity. I pulled back from actively trying to do just that once I realized that gaining that knowledge without sufficient safety preparation could be a bad thing for humanity.

Nathan Helm-Burger 3 Oct 2021 4:17 UTC
3 points
on: Brain-inspired AGI and the “lifetime anchor”
As a neuroscientist-turned-machine-learning-engineer, I have been thinking about this situation in a very similar way to that described in this article. One (perhaps) difference is that I think there are a fair number of possible algorithms/architectures that could successfully generate an agentive general learner sufficient for AGI. I think that a human brain -similar algorithm might be the first developed because of fairly good efficiency and having a working model to study (albeit with difficultly). On the other hand, I think it’s probable that deep learning, scaled up enough, will stumble across a surprisingly effective algorithm all of a sudden with little warning (aka the lottery ticket hypothesis), risking an accidental hard take-off scenario.
I kinda hope the human brain-like algorithm actually does turn out to be the first breakthrough, since I feel like we’d have a better chance of understanding and controlling it, and noticing/measuring when we’d gotten quite close.
With the blind groping into unknown solution spaces that deep learning represents, we might find more than we’d bargained for with no warning at all. Just a sudden jump from awkward semi-competent statistical machine to powerful deceitful alien-minded agent.

Nathan Helm-Burger 16 Feb 2022 22:09 UTC
3 points
on: larger language models may disappoint you [or, an eternally unfinished draft]
A small takeaway from this is the value of using better descriptions of tasks when reporting results. For example, if the data about failures had been presented as a violin plot rather than an average then it would be easy to see the difference between a distribution consisting of middling results vs a bimodal distribution of mostly-successes and complete-failures.

Progress Report 1: interpretability experiments & learning, testing compression hypotheses

Nathan Helm-Burger22 Mar 2022 20:12 UTC

11 points

0 comments2 min readLW link

Nathan Helm-Burger 23 Mar 2022 5:06 UTC
4 points
on: It Looks Like You’re Trying To Take Over The World
Slightly off-topic, but reading through this comment section I couldn’t help but notice that the comments critical of the work (which I quite enjoyed the work!) mostly seemed to pattern-match to nit-picking of examples in such a way that it didn’t address the central claims. And then I read this article discussing that phenomenon, so I thought I’d link it here https://ansuz.sooke.bc.ca/entry/350

Nathan Helm-Burger 23 Mar 2022 19:29 UTC
1 point
on: It Looks Like You’re Trying To Take Over The World
Not exactly on topic, but related:

“An artificial intelligence model invents 40,000 chemical weapons in just 6 hours”

https://interestingengineering.com/artificial-intelligence-chemical-weapons

Progress Report 2

Nathan Helm-Burger30 Mar 2022 2:29 UTC

4 points

1 comment1 min readLW link

Nathan Helm-Burger 30 Mar 2022 2:43 UTC
6 points
in reply to: Rohin Shah’s comment on: Neural net / decision tree hybrids: a potential path toward bridging the interpretability gap
https://www.lesswrong.com/posts/pvbWz5mEFCS6DFaWD/progress-report-1-interpretability-experiments-and-learning
https://www.lesswrong.com/posts/nxLHgG5SCKCxM9oJX/progress-report-2
Now that I got a grant from the Long Term Future Fund and quit my job to do interpretability research full time, I’m actually making progress on some of my ideas!

Nathan Helm-Burger 30 Mar 2022 19:10 UTC
1 point
on: Procedurally evaluating factual accuracy: a request for research
I have a friend who has been working on a team doing automatic factual responses to search queries. I’ll send him the link to this article and maybe he’ll have some thoughts...

Nathan Helm-Burger 30 Mar 2022 21:28 UTC
1 point
on: Progress Report 2
Looking at a couple hundred of these plots for the MLP neurons, I see two obvious patterns which can occur alone or together/overlapping. One pattern is a ‘vertical group’, a narrow band that runs through many layers. The other pattern is a ‘horizontal group’, which is lots of involvement within one layer.
Now I’m generating and looking at plots which do both the MLP neurons and the attention heads.

Progress report 3: clustering transformer neurons

Nathan Helm-Burger5 Apr 2022 23:13 UTC

5 points

0 comments2 min readLW link

Nathan Helm-Burger 6 Apr 2022 3:03 UTC
1 point
in reply to: moridinamael’s comment on: the scaling “inconsistency”: openAI’s new insight
I think there’s something we could do even beyond choosing the best of existing data points to study. I think we could create data-generators, which could fill out missing domains of data using logical extrapolations. I think your example is a great type of problem for such an approach.

Nathan Helm-Burger 6 Apr 2022 4:09 UTC
3 points
in reply to: Pattern’s comment on: Project Intro: Selection Theorems for Modularity
I recall something similar about a robot hand trained in varying simulations. I remember an OpenAI project not a Deepmind one… Here’s the link to the OpenAI environment-varying learner: https://openai.com/blog/learning-dexterity/

Progress Report 4: logit lens redux

Nathan Helm-Burger8 Apr 2022 18:35 UTC

3 points

0 comments2 min readLW link

Nathan Helm-Burger 8 Apr 2022 20:39 UTC
81 points
on: It’s time for EA leadership to pull the fast-takeoff fire alarm.
About 15 years ago, before I’d started professionally studying and doing machine learning research and development, my timeline had most of its probability mass around 60 − 90 years from then. This was based on my neuroscience studies and thinking about how long I thought it would take to build a sufficiently accurate emulation of the human brain to be functional. About 8 years ago, studying machine learning full time, AlphaGo coming out was inspiration for me to carefully rethink my position, and I realized there were a fair number of shortcuts off my longer figure that made sense, and updated to more like 40 − 60 years. About 3 years ago, GPT-2 gave me another reason to rethink with my then fuller understanding. I updated to 15 − 30 years. In the past couple of years, with the repeated success of various explorations of the scaling law, the apparent willingness of the global community to rapidly scale investments in large compute expenditures, and yet further knowledge of the field, I updated to more like 2 − 15 years as having 80% of my probability mass. I’d put most of that in the 6 − 12 year range, but I wouldn’t be shocked if things turned out to be easier than expected and something really took off next year.
One of the things that makes me think the BioAnchors estimate is a bit too far into the future is that I know from neuroscience that it’s possible for a human to have a sufficient set of brain functions to count as a General Intelligence by a fairly reasonable standard with significant chunks of their brain dead or missing. I mean, they’ll not be in great shape as they’ll be missing some stuff, but if the stuff they’re missing is non-critical they can still function well enough to be a minimal GI. Plenty well enough to be scary if they were a self-improving self-replicating agentive AI.
So anyway, yeah, I’ve been scared for a while now. Latest news has just reinforced my belief we are in a short timeline world, not surprised me. Glad to see more people getting on board with my point of view.

Nathan Helm-Burger 9 Apr 2022 17:23 UTC
3 points
on: Fixing The Good Regulator Theorem
Trying to think about this from more of a practical machine learning standpoint, but without full understanding of all the points you made...
I think in the case where X, Y, and S are all partially but not fully known, and you have to choose model M so as to minimize regret over time. Two things occur to me as possibly useful strategies for choosing M. You might find the opportunity to run ‘experiments’, to choose suboptimal output R at an early timestep such that from then on you’d have a better understanding of S, and be able to better refine M for future timesteps.
The other thing that occurs to me is that if the distributions of X and Y are unknown and you have some approximation of them, then perfectly fitting M to those approximations would potentially be overfitting. If you know you have only approximations, then you’d want to not fully minimize M but instead leave some ‘wiggle room’ for probable future cases of X and Y. You could even tailor this based on patterns you’ve seen in X and Y. Like, if X was gaussian noise, but the mean seemed to be roughly following a sine wave for n cycles, then a square wave for n cycles, then a triangular wave for n cycles, you’d be reasonably justified in guessing that there would be a new repeating wave shape coming up which would repeat for n cycles. You’d want to leave room in the model to account for this.
Perhaps someone who understands the post more completely can let me know if I seem to be on the right track for understanding and extrapolating from this.

Nathan Helm-Burger 9 Apr 2022 18:49 UTC
3 points
in reply to: Ilverin the Stupid and Offensive’s comment on: Google’s new 540 billion parameter language model
That’s already what TPUs do, basically

Nathan Helm-Burger 9 Apr 2022 22:30 UTC
68 points
on: A concrete bet offer to those with short AI timelines
Ok, I take your bet for 2030. I win, you give me $1000. You win, I give you $3000. Want to propose an arbiter? (since someone else also took the bet, I’ll get just half the bet, their $500 vs my $1500)

Nathan Helm-Burger

Neu­ral net /​ de­ci­sion tree hy­brids: a po­ten­tial path to­ward bridg­ing the in­ter­pretabil­ity gap

Progress Re­port 1: in­ter­pretabil­ity ex­per­i­ments & learn­ing, test­ing com­pres­sion hypotheses

Progress Re­port 2

Progress re­port 3: clus­ter­ing trans­former neurons

Progress Re­port 4: logit lens redux

Neural net / decision tree hybrids: a potential path toward bridging the interpretability gap

Progress Report 1: interpretability experiments & learning, testing compression hypotheses

Progress Report 2

Progress report 3: clustering transformer neurons

Progress Report 4: logit lens redux