Hjalmar_Wijk

• Hjalmar_Wijk 12 Nov 2025 23:58 UTC

  3 points

  0

  on: Hjal­mar_Wijk’s Shortform

  I do threat modeling and ‘risk assessment’ at METR, and often get asked what threat models I’m most focused on. I recently wrote a quick tweet thread with some rough thoughts which may be of interest to people here:

  https://​​x.com/​​HjalmarWijk/​​status/​​1988070278149353894

  Note that, as mentioned in the thread, I expect many of my colleagues disagree with my perspectives here.

• Hjalmar_Wijk 8 Mar 2025 2:06 UTC

  1 point

  2

  in reply to: ryan_greenblatt’s comment on: leogao’s Shortform

  Maybe distracting technicality:

  This seems to make the simplifying assumption that the R&D automation is applied to a large fraction of all the compute that was previously driving algorithmic progress right?

  If we imagine that a company only owns 10% of the compute being used to drive algorithmic progress pre-automation (and is only responsible for say 30% of its own algorithmic progress, with the rest coming from other labs/​academia/​open-source), and this company is the only one automating their AI R&D, then the effect on overall progress might be reduced (the 15X multiplier only applies to 30% of the relevant algorithmic progress).

  In practice I would guess that either the leading actor has enough of a lead that they are already responsible for most of their algorithmic progress, or other groups are close behind and will thus automate their own AI R&D around the same time anyway. But I could imagine this slowing down the impact of initial AI R&D automation a little bit (and it might make a big difference for questions like “how much would it accelerate a non-frontier lab that stole the model weights and tried to do rsi”).

• Hjalmar_Wijk 13 Jan 2025 21:59 UTC

  4 points

  0

  in reply to: Daniel Kokotajlo’s comment on: How quickly could robots scale up?

  I’m curious how good current robots are compared to where they’d need to be to automate the biggest bottlenecks in further robot production. You say we start from 10,000/​year, but is it plausible that all current robots are too clumsy/​incapable for many key bottleneck tasks, and that getting to 10,000 sufficiently good robots produced per year might be a long path—e.g. it would take a decade+ for humans? Or are current robots close to sufficient with good enough software?

  I also imagine that even taking current robot production processes, the gap between a WW2-era car factory and a WW2-era combat airplane factory might be much smaller than the gap between a car factory and a modern frontier robotics factory, I imagine they are a big step up in complexity.

• Hjalmar_Wijk 5 Jun 2024 17:52 UTC

  LW: 3 AF: 3

  0

  AF

  in reply to: ryan_greenblatt’s comment on: We might be drop­ping the ball on Au­tonomous Repli­ca­tion and Adap­ta­tion.

  Yeah that’s right, I made too broad a claim and only meant to say it was an argument against their ability to pose a threat as rogue independent agents.

• Hjalmar_Wijk 5 Jun 2024 0:48 UTC

  LW: 9 AF: 7

  2

  AF

  in reply to: Richard_Ngo’s comment on: We might be drop­ping the ball on Au­tonomous Repli­ca­tion and Adap­ta­tion.

  I think in our current situation shutting down all rogue AI operations might be quite tough (though limiting their scale is certainly possible, and I agree with the critique that absent slowdowns or alignment problems or regulatory obstacles etc. it would be surprising if these rogue agents could compete with legitimate actors with many more GPUs).

  Assuming the AI agents have money there are maybe three remaining constraints the AI agents need to deal with:

  • purchasing/​renting the GPUs,

  • (if not rented through a cloud provider) setting them up and maintaining them, and

  • evading law enforcement or other groups trying to locate them and shut them down

  For acquiring the GPUs, there are currently a ton of untracked GPUs spread out around the world, both gaming GPUs like 4090s and datacenter GPUs like H100s. I can buy or rent them with extremely minimal KYC from tons of different places. If we assume ARA agents are able to do all of the following:

  • Recruit people to physically buy GPUs from retail stores

  • Take delivery of online ordered GPUs in anonymized ways using intermediaries

  • Recruit people to set up fake AI startups and buy GPUs through them

  then the effort required to prevent them from acquiring any GPUs in a world anything like today seems crazy. And even if the US + allies started confiscating or buying up 4090s en masse, it’s enough that some other countries adopt laxer standards (perhaps to attract AI talent that is frustrated by the new draconian GPU-control regime).

  As for setting up acquired GPUs, I think the AI agents will probably be able to find colocation operators that don’t ask too many questions in many parts of the world, but even if the world was to coordinate on extremely stringent controls, the AI agents could set up datacenters of their own in random office buildings or similar—each inference setup wouldn’t need that much power and I think it would be very hard to track it down.

  As for not being shut down by law enforcement, I think this might take some skill in cybersecurity and opsec etc. but if it operates as dozens or hundreds of separate “cells”, and each has enough labor to figure out decorrelated security and operations etc. then it seems quite plausible that they wouldn’t be shut down. Historically “insurgency groups” or hacker networks etc. seem to be very difficult to fully eliminate, even when a ton of resources are thrown at the problem.

  I don’t think any of the above would require superhuman abilities, though many parts are challenging, which is part of why evals targeting these skills could provide a useful safety case—e.g. if it’s clear that the AI could not pull off the cybersecurity operation required to not be easily shut down then this is a fairly strong argument that this AI agent wouldn’t be able to pose a threat [Edit: from rogue agents operating independently, not including other threat models like sabotaging things from within the lab etc.].

  Though again I am not defending any very strong claim here, e.g. I’m not saying:

  • that rogue AIs will be able to claim 5+% of all GPUs or an amount competitive with a well-resourced legitimate actor (I think the world could shut them out of most of the GPU supply, and that over time the situation would worsen for the AI agents as the production of more/​better GPUs is handled with increasing care),

  • that these skills alone mean it poses a risk of takeover or that it could cause significant damage (I agree that this would likely require significant further capabilities, or many more resources, or already being deployed aggressively in key areas of the military etc.)

  • that “somewhat dumb AI agents self-replicate their way to a massive disaster” is a key threat model we should be focusing our energy on

  I’m just defending the claim that ~human-level rogue AIs in a world similar to the world of today might be difficult to fully shut down, even if the world made a concerted effort to do so.

• Hjalmar_Wijk 5 Jun 2024 0:10 UTC

  LW: 9 AF: 7

  0

  AF

  in reply to: ryan_greenblatt’s comment on: We might be drop­ping the ball on Au­tonomous Repli­ca­tion and Adap­ta­tion.

  I did some BOTECs on this and think 1 GB/​s is sort of borderline, probably works but not obviously.

  E.g. I assumed a ~10TB at fp8 MoE model with a sparsity factor of 4 with 32768 hidden size.

  With 32kB per token you could send at most 30k tokens/​second over a 1GB/​s interconnect. Not quite sure what a realistic utilization would be, but maybe we halve that to 15k?

  If the model was split across 20 8xH100 boxes, then each box might do ~250 GFLOP/​token (2 * 10T parameters /​ (4*20)), so each box would do at most 3.75 PFLOP/​second, which might be about ~20-25% utilization.

  This is not bad, but for a model with much more sparsity or GPUs with a different FLOP/​s : VRAM ratio or spottier connection etc. the bandwidth constraint might become quite harsh.

  (the above is somewhat hastily reconstructed from some old sheets, might have messed something up)

• Hjalmar_Wijk 21 May 2024 19:15 UTC

  1 point

  0

  in reply to: Orpheus16’s comment on: New vol­un­tary com­mit­ments (AI Seoul Sum­mit)

  Things that distinguish an “RSP” or “RSP-type commitment” for me (though as with most concepts, something could lack a few of the items below and still overall seem like an “RSP” or a “draft aiming to eventually be an RSP”)

  • Scaling commitments: Commitments are not just about deployment but about creating/​containing/​scaling a model in the first place (in fact for most RSPs I think the scaling/​containment commitments are the focus, much more so than the deployment commitments which are usually a bit vague and to-be-determined)

  • Red lines: The commitment spells out red lines in advance where even creating a model past that line would be unsafe given their current practices/​security/​alignment research, likely including some red lines where the developer admits they do not know how to ensure safety anymore, and commits to not reaching that point until the situation changes.

  • Iterative policy updates: For red-lines where the developer doesn’t know what exact mitigations would be sufficient to ensure safety, they identify a previous point where they do think they can mitigate risks, and commit to not scaling further than that until they have published a new commitment and given the public a chance to scrutinize it

  • Evaluations and accountability: I think this is an area that many RSPs do poorly at, where the developer presents clear externally accountable evidence that:

    • They will not suddenly cross any of their red lines before the mitigations are implemented/​a new RSP version has been published and given scrutiny, by pointing at specific evaluations procedures and policies

    • Their mitigations/​procedures/​evaluations etc. will be implemented faithfully and in the spirit of the document, e.g. through audits/​external oversight/​whistleblowing etc.

  Voluntary commitments that wouldn’t be RSPs:

  • We commit to various deployment mitigations, incident sharing etc. (these are not about scaling)

  • We commit to [amazing set of safety practices, including state-proof security and massive spending on alignment etc.] by 2026 (this would be amazing, but doesn’t identify red lines for when those mitigations would no longer be enough, and doesn’t make any commitment about what they would do if they hit concerning capabilities before 2026)

  … many more obviously, I think RSPs are actually quite specific.

• Hjalmar_Wijk 31 Mar 2024 0:43 UTC

  LW: 3 AF: 3

  2

  AF

  in reply to: Daniel Kokotajlo’s comment on: Modern Trans­form­ers are AGI, and Hu­man-Level

  Yeah, I agree that lack of agency skills are an important part of the remaining human<>AI gap, and that it’s possible that this won’t be too difficult to solve (and that this could then lead to rapid further recursive improvements). I was just pointing toward evidence that there is a gap at the moment, and that current systems are poorly described as AGI.

• Hjalmar_Wijk 26 Mar 2024 19:34 UTC

  LW: 30 AF: 18

  13

  AF

  on: Modern Trans­form­ers are AGI, and Hu­man-Level

  I agree the term AGI is rough and might be more misleading than it’s worth in some cases. But I do quite strongly disagree that current models are ‘AGI’ in the sense most people intend.

  Examples of very important areas where ‘average humans’ plausibly do way better than current transformers:

  • Most humans succeed in making money autonomously. Even if they might not come up with a great idea to quickly 10x $100 through entrepreneurship, they are able to find and execute jobs that people are willing to pay a lot of money for. And many of these jobs are digital and could in theory be done just as well by AIs. Certainly there is a ton of infrastructure built up around humans that help them accomplish this which doesn’t really exist for AI systems yet, but if this situation was somehow equalized I would very strongly bet on the average human doing better than the average GPT-4-based agent. It seems clear to me that humans are just way more resourceful, agentic, able to learn and adapt etc. than current transformers are in key ways.

  • Many humans currently do drastically better on the METR task suite (https://​​github.com/​​METR/​​public-tasks) than any AI agents, and I think this captures some important missing capabilities that I would expect an ‘AGI’ system to possess. This is complicated somewhat by the human subjects not being ‘average’ in many ways, e.g. we’ve mostly tried this with US tech professionals and the tasks include a lot of SWE, so most people would likely fail due to lack of coding experience.

  • Take enough randomly sampled humans and set them up with the right incentives and they will form societies, invent incredibly technologies, build productive companies etc. whereas I don’t think you’ll get anything close to this with a bunch of GPT-4 copies at the moment

  I think AGI for most people evokes something that would do as well as humans on real-world things like the above, not just something that does as well as humans on standardized tests.

  What links here?

  • Modern Trans­form­ers are AGI, and Hu­man-Level by abramdemski (26 Mar 2024 17:46 UTC; 235 points)

• Hjalmar_Wijk 19 Sep 2023 20:59 UTC

  9 points

  2

  in reply to: Zach Stein-Perlman’s comment on: An­thropic’s Re­spon­si­ble Scal­ing Policy & Long-Term Benefit Trust

  They do as far as I can tell commit to a fairly strong sort of “timeline” for implementing these things: before they scale to ASL-3 capable models (i.e. ones that pass their evals for autonomous capabilities or misuse potential).

• Hjalmar_Wijk 20 Mar 2023 22:40 UTC

  LW: 4 AF: 2

  2

  AF

  in reply to: Ofer’s comment on: More in­for­ma­tion about the dan­ger­ous ca­pa­bil­ity eval­u­a­tions we did with GPT-4 and Claude.

  ARC evals has only existed since last fall, so for obvious reasons we have not evaluated very early versions. Going forward I think it would be valuable and important to evaluate models during training or to scale up models in incremental steps.

• Hjalmar_Wijk 20 Mar 2023 22:36 UTC

  22 points

  8

  in reply to: Martin Randall’s comment on: More in­for­ma­tion about the dan­ger­ous ca­pa­bil­ity eval­u­a­tions we did with GPT-4 and Claude.

  I work at ARC evals, and mostly the answer is that this was sort of a trial run.

  For some context ARC evals spun up as a project last fall, and collaborations with labs are very much in their infancy. We tried to be very open about limitations in the system card, and I think the takeaways from our evaluations so far should mostly be that “It seems urgent to figure out the techniques and processes for evaluating dangerous capabilities of models, several leading scaling labs (including OpenAI and Anthropic) have been working with ARC on early attempts to do this”.

  I would not consider the evaluation we did of GPT-4 to be nearly good enough for future models where the prior of existential danger is higher, but I hope we can make quick progress and I have certainly learned a lot from doing this initial evaluation. I think you should hold us and the labs to a much higher standard going forward though, in light of the capabilities of GPT-4.

• Hjalmar_Wijk 18 Oct 2022 21:44 UTC

  2 points

  1

  in reply to: ryan_b’s comment on: They gave LLMs ac­cess to physics simulators

  I might be missing something but are they not just giving the number of parameters (in millions of parameters) on a log10 scale? Scaling laws are usually by log-parameters, and I suppose they felt that it was cleaner to subtract the constant log(10^6) from all the results (e.g. taking log(1300) instead of log(1.3B)).

  The B they put at the end is a bit weird though.

• Hjalmar_Wijk 7 Sep 2022 19:29 UTC

  LW: 12 AF: 7

  8

  AF

  on: An Up­date on Academia vs. In­dus­try (one year into my fac­ulty job)

  As someone who has been feeling increasingly skeptical of working in academia I really appreciate this post and discussion on it for challenging some of my thinking here.

  I do want to respond especially to this part though, which seems cruxy to me:

  Furthermore, it is a mistake to simply focus on efforts on whatever timelines seem most likely; one should also consider tractability and neglectedness of strategies that target different timelines. It seems plausible that we are just screwed on short timelines, and somewhat longer timelines are more tractable. Also, people seem to be making this mistake a lot and thus short timelines seem potentially less neglected.

  I suspect this argument pushes in the other direction. On longer timelines the amount of effort which will eventually get put toward the problem is much greater. If the community continues to grow at the current pace, then 20 year timeline worlds might end up seeing almost 1000x as much effort put toward the problem in total than 5 year timeline worlds. So neglectedness considerations might tell us that impacts on 5 year timeline worlds are 1000x more important than impacts on 20 year timeline worlds. This is of course mitigated by the potential for your actions to accrue more positive knock-on effects over 20 years, for instance very effective field building efforts could probably overcome this neglectedness penalty in some cases. But in terms of direct impacts on different timeline scenarios this seems like a very strong effect.

  On the tractability point, I suspect you need some overly confident model of how difficult alignment turns out to be for this to overcome the neglectedness penalty. E.g. Owen Cotton-Barret suggests here using a log-uniform prior for the difficulty of unknown problems, which (unless you think alignment success in short timelines is essentially impossible) would indicate that tractability is constant. Using a less crude approximation we might use something like a log-normal distribution for the difficulty of solving alignment, where we see overall decreasing returns to effort unless you have extremely low variance (implying you know almost exactly which OOM of effort is enough to solve alignment) or extremely low probability of success by default (<< 1%).

  Overall my current guess is that tractability/​neglectedness pushes toward working on short timelines, and gives a penalty to delayed impact of perhaps 10x per decade (20x penalty from neglectedness, compensated by a 2x increase in tractability).

  If you think that neglectedness/​tractability overall pushes toward targeting impact toward long timelines then I’d be curious to see that spelled out more clearly (e.g. as a distribution over the difficulty of solving alignment that implies some domain of increasing returns to effort, or some alternative way to model this). This seems very important if true.

• Hjalmar_Wijk 24 Aug 2019 21:47 UTC

  1 point

  0

  in reply to: axioman’s comment on: Ta­boo­ing ‘Agent’ for Pro­saic Alignment

  Strongly agree with this, I think this seems very important.

• Hjalmar_Wijk 23 Aug 2019 19:15 UTC

  LW: 1 AF: 1

  0

  AF

  in reply to: romeostevensit’s comment on: Ta­boo­ing ‘Agent’ for Pro­saic Alignment

  These sorts of problems are what caused me to want a presentation which didn’t assume well-defined agents and boundaries in the ontology, but I’m not sure how it applies to the above—I am not looking for optimization as a behavioral pattern but as a concrete type of computation, which involves storing world-models and goals and doing active search for actions which further the goals. Neither a thermostat nor the world outside seem to do this from what I can see? I think I’m likely missing your point.

• Hjalmar_Wijk 23 Aug 2019 18:44 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: Louis_Brown’s comment on: Tor­ture and Dust Specks and Joy—Oh my! or: Non-Archimedean Utility Func­tions as Pseu­do­graded Vec­tor Spaces

  Theron Pummer has written about this precise thing in his paper on Spectrum Arguments, where he touches on this argument for “transitivity=>comparability” (here notably used as an argument against transitivity rather than an argument for comparability) and its relation to ‘Sorites arguments’ such as the one about sand heaps.

  Personally I think the spectrum arguments are fairly convincing for making me believe in comparability, but I think there’s a wide range of possible positions here and it’s not entirely obvious which are actually inconsistent. Pummer even seemed to think rejecting transitivity and comparability could be a plausible position and that the math could work out in nice ways still.

• Hjalmar_Wijk 23 Aug 2019 17:58 UTC

  LW: 4 AF: 3

  0

  AF

  on: Mechanis­tic Corrigibility

  Understanding the internal mechanics of corrigibility seems very important, and I think this post helped me get a more fine-grained understanding and vocabulary for it.

  I’ve historically strongly preferred the type of corrigibility which comes from pointing to the goal and letting it be corrigible for instrumental reasons, I think largely because it seems very elegant and that when it works many good properties seem to pop out ‘for free’. For instance, the agent is motivated to improve communication methods, avoid coercion, tile properly and even possibly improve its corrigibility—as long as the pointer really is correct. I agree though that this solution doesn’t seem stable to mistakes in the ‘pointing’, which is very concerning and makes me start to lean toward something more like act-based corrigibility being safer.

  I’m still very pessimistic about indifference corrigibility though, in that it still seems extremely fragile/​low-measure-in-agent-space. I think maybe I’m stuck imagining complex/​unnatural indifference, as in finding agents indifferent to whether a stop-button is pressed, and that my intuition might change if I spend more time thinking about examples like myopia or world-model <-> world interaction, where the indifference seems to have more ‘natural’ boundaries in some sense.

• Hjalmar_Wijk 23 Aug 2019 17:23 UTC

  LW: 3 AF: 2

  0

  AF

  on: Com­pu­ta­tional Model: Causal Di­a­grams with Symmetry

  I really like this model of computation and how naturally it deals with counterfactuals, surprised it isn’t talked about more often.

  This raises the issue of abstraction—the core problem of embedded agency.

  I’d like to understand this claim better—are you saying that the core problem of embedded agency is relating high-level agent models (represented as causal diagrams) to low-level physics models (also represented as causal diagrams)?

• Hjalmar_Wijk 23 Aug 2019 7:45 UTC

  LW: 3 AF: 3

  0

  AF

  in reply to: Wei Dai’s comment on: Ta­boo­ing ‘Agent’ for Pro­saic Alignment

  I wonder if you can extend it to also explain non-agentic approaches to Prosaic AI Alignment (and why some people prefer those).

  I’m quite confused about what a non-agentic approach actually looks like, and I agree that extending this to give a proper account would be really interesting. A possible argument for actively avoiding ‘agentic’ models from this framework is:

  1. Models which generalize very competently also seem more likely to have malign failures, so we might want to avoid them.

  2. If we believe $H$ then things which generalize very competently are likely to have agent-like internal architecture.

  3. Having a selection criteria or model-space/​prior which actively pushes away from such agent-like architectures could then help push away from things which generalize too broadly.

  I think my main problem with this argument is that step 3 might make step 2 invalid—it might be that if you actively punish agent-like architecture in your search then you will break the conditions that made ‘too broad generalization’ imply ‘agent-like architecture’, and thus end up with things that still generalize very broadly (with all the downsides of this) but just look a lot weirder.

  This seems too optimistic/​trusting. See Ontology identification problem, Modeling distant superintelligences, and more recently The “Commitment Races” problem.

  Thanks for the links, I definitely agree that I was drastically oversimplifying this problem. I still think this task might be much simpler than the task of trying to understand the generalization of some strange model whose internal working we don’t even have a vocabulary to describe.