Steve Kommrusch

• Steve Kommrusch 20 Feb 2026 20:54 UTC

  1 point

  0

  on: Why we should ex­pect ruth­less so­ciopath ASI

  I think that, even if LLM’s don’t smoothly evolve into AGI then ASI, an alternative ‘brain-like’ AGI will have a similar progress ramp that allows for alignment learning-by-doing in a very meaningful way. To explain this, let’s discuss the LLM path a bit. OpenAI’s deliberative alignment and Anthropic’s more sober discussion of the ongoing alignment challenge both highlight the effort that companies today put in to understanding and improving LLM alignment. Alignment work is progressing through improved training, RLHF, RLAIF, Constitutional Classifiers, etc. One would expect that, as AI agents get used more and home robots get marketed, customers will refuse to buy unsafe AI agents and AI companies will need to learn to improve the AI behavior. It would be great to have some regulation or strong liability laws to help with this, but customer demand alone will provide impetus for general alignment of today’s systems. As LLMs and their cousins VLAs move towards AGI, we’ll have tolerably aligned AGI and we’ll have learned how to get alignment to generalize for an AGI. As AGIs advance to ASI, we’ll continue to have product pressure and RLAIF will improve in capability along with the AGI’s themselves. The point of that summary is not to say that I’m sure AI safety will play out well, but that there is indeed a lot of effort put in to prevent sociopathic results.
  
  Now if we posit a different learning system that takes us to ASI, I would still expect a multi-year ramp from ‘not yet on the public radar’ to ASI. There will be many companies and watchdog groups watching the new systems grow, make mistakes, and get fixed. If this new learning approach results in AI’s as capable as today’s systems but LESS aligned, they aren’t likely to sell well. I think that before we need to worry about ASI, we should accept that the AGI we build will be valuable to someone and, hence, by definition tolerably aligned (although I don’t disagree that ‘tolerable’ may be a low bar).
  
  In the end, I would expect that a useful AGI (not ASI) would need to have features like corrigibility (ability to evaluate goals and adjust or abort them), curiosity (recognizing when a conclusion or plan may be wrong), and self-critiquing (using classifiers or other systems to stress-test a plan for unwanted side-effects). I disagree with the premise that ASI’s will evolve into ruthless optimizers because a useful AGI will have learned the value of reconsidering goals and trying to understand the full impact of plans and actions. These features don’t guarantee we avoid sociopaths, but I see them as necessary items to solve for useful AGI and, hence, the ASI developers will have something to build on.

• Steve Kommrusch 18 Dec 2025 4:40 UTC

  3 points

  1

  in reply to: RussellThor’s comment on: The be­hav­ioral se­lec­tion model for pre­dict­ing AI motivations

  Interesting—I too suspect that good world models will help with data efficiency. Even using the existing training paradigm where a lot of data is needed to get the generalization to work well, if an AI has a good internal world model it could generate usable synthetic examples for incremental training. For example, when a child sees a photo of some strange new animal from the side, the child likely surmises that the animal looks the same from the other side; if the photo only shows one eye, the child can imagine that looking head on into the animal’s face it will have 2 eyes, etc. Because the child has a rather reliable model of an ‘animal’, they can create reliable synthetic data for incremental training from a single picture.

  And I like your framing of having the internally generated reward be valuable for learning too. While I expect that reward is a composite of experience (enlightened self-interest, reading and discussion, etc) it can still be more important day-to-day than the external rewards received immediately. (I think this opens up a lot of philosophy—what are the ‘ultimate’ goals for your internal ethics and personally fulfilling rewards, etc. But I see your point).

• Steve Kommrusch 13 Dec 2025 17:28 UTC

  4 points

  3

  on: The be­hav­ioral se­lec­tion model for pre­dict­ing AI motivations

  Thanks for the post, and especially the causal graph framework you use to describe and analyze the categories of motivations. It feels similar to Richard Dawkins ‘The Selfish Gene’ work in so far as it studies the fitness of motivations in their selection environment.

  One area I think it could extend to is around the concepts of curiosity, exploration, and even corrigibility. This relates to ‘reflection’ as mentioned by habryka. I expect that as AI moves towards AGI it will improve its ability to recognize holes in its knowledge and take actions to close them (aka an AI scientist). Forming a world model that can be used for ‘reflecting’ on current motivations and actions is important here. To an extent, longer training runs can instill a type of curiosity. For example, a training run that includes formulating a web search query, analyzing results, updating the search query and doing a new web search, and then producing the correct answer, will help train for ‘curiosity’ but only as it becomes a ‘schemer’ I presume. In other words, a general schemer motivation of ‘curiosity/​exploration’ could be included in your model if it reliably has a consequence of producing results that have a higher reward. Similarly, training examples where the AI asks a user for clarification on a goal could help create a ‘corrigibility’ schemer. But I think the general concept is crucial enough to warrant explicit discussion. For example, in the diagram that includes ‘saint’ and ‘alien’ you might add a 5th category for ‘explorer’ or ‘scientist’.

  I think that as AI models get better at recognizing holes in their knowledge and seeking to fill them, the models will find benefit from taking actions which tend to minimize side effects on the world and leave open many possible future options (low impact). Again, long training examples might suffice to train this, but new training systems might be needed to create this behavior. This relates to corrigibility in that goals themselves might be adjusted or questioned during action sequences (if the steps required violate certain trained motivations or system prompt rules).

• Steve Kommrusch 9 Dec 2025 22:55 UTC

  1 point

  0

  on: Eliezer’s Un­teach­able Meth­ods of Sanity

  Thanks for the interesting peak into your brain. I have a couple thoughts to share on how my own approaches relate.

  The first is related to watching plenty of sci-fi apocalyptic future movies. While it’s exciting to see the hero’s adventures, I’d like to think that I’d be one of the scrappy people trying to hold some semblance of civilization together. Or the survivor trying to barter and trade with folks instead of fighting over stuff. In general, even in the face of doom, just trying to help minimize suffering unto the end. So the ‘death with dignity’ ethos fits in with this view.

  A second relates to the idea of seeing yourself getting out of bed in the morning. When I’ve had a lot on my plate to the point of seeming stressful, it helps to visualize the future state where I’ve gotten the work done and am looking back. Then just imagining inside my brain sodium ions moving around, electrons dropping energy states, proteins changing shapes, etc, as the problem gets resolved. Visualizing the low-level activity in my brain helps me shift focus from the stress and actually move ahead solving the problem.

• Steve Kommrusch 9 Dec 2025 22:19 UTC

  2 points

  0

  in reply to: Yoav Hollander’s comment on: Align­ment re­mains a hard, un­solved problem

  That idea of catching bad mesa-objectives during training sounds key and I presume fits under the ‘generalization science’ and ‘robust character training’ from Evan’s original post. In the US, NIST is working to develop test, evaluation, verification and validation standards for AI and it would be good to include this concept into that effort.

• Steve Kommrusch 9 Dec 2025 22:04 UTC

  2 points

  0

  in reply to: Rohin Shah’s comment on: Align­ment re­mains a hard, un­solved problem

  The data relating exponential capabilities to Elo is seen over decades in the computer chess history too. From the 1960s into the 2020′s, while computer hardware advanced exponentially at 100-1000x per decade in performance (and SW for computer chess advanced too), Elo scores grew linearly at about 400x per decade, taking multiple decades to go from ‘novice’ to ‘superhuman’. Elo scores have a tinge of exponential to them—a 400 point Elo advantage is about a 10:1 chance for the higher scored competitor to win, and an 800 point Elo is about 200:1, etc. It appears that the current HW/​SW/​dollar rate of growth towards AGI means the Elo relative to humans is increasing faster than 400 Elo/​decade. And, of course, unlike computer chess, as AI Elo at ‘AI development’ approaches the level of a skilled human, we’ll likely get a noticable increase in the rate of capability increase.

• Steve Kommrusch 9 Dec 2025 19:27 UTC

  1 point

  0

  on: Align­ment re­mains a hard, un­solved problem

  Thanks Evan for an excellent overview of the alignment problem as seen from within Anthropic. Chris Olah’s graph showing perspectives on alignment difficulty is indeed a useful visual for this discussion. Another image I’ve shared lately relates to the challenge of building inner alignment illustrated by figure 2 from Contemplative Artificial Intelligence:

  

  In the images, the blue arrows indicate our efforts to maintain alignment on AI as capabilities advance through AGI to ASI. In the left image, we see the case of models that generalize in misaligned ways—the blue constraints (guardrails, system prompts, thin efforts at RLHF, etc) fail to constrain the advancing AI to remain aligned. The right image shows the happier result where training, architecture, interpretability, scalable oversight, etc contribute to a ‘wise world model’ that maintains alignment even as capabilities advance.
  
  I think the Anthropic probability distribution of alignment difficulty seems correct—we probably won’t get alignment by default from advancing AI, but, as you suggest, by serious concerted effort we can maintain alignment through AGI. What’s critical is to use techniques like generalization science, interpretability, and introspective honesty to gauge whether we are building towards AGI capable of safely automating alignment research towards ASI. To that end, metrics that allow us determine if alignment is actual closer to P-vs-NP in difficulty are crucial, and efforts from METR, UK AISI, NIST, and others can help here. I’d like to see more ‘positive alignment’ papers such as Cooperative Inverse Reinforcement Learning, Corrigibility, and AssistanceZero: Scalably Solving Assistance Games, as detecting when an AI is positively aligned internally is critical to getting to the ‘wise world model’ outcome.

• Steve Kommrusch 9 Jul 2025 20:05 UTC

  1 point

  0

  in reply to: Steven Byrnes’s comment on: Foom & Doom 1: “Brain in a box in a base­ment”

  I concur with that sentiment. GPUs hit a sweet spot between compute efficiency and algorithmic flexibility. CPUs are more flexible for arbitrary control logic, and custom ASICs can improve compute efficiency for a stable algorithm, but GPUs are great for exploring new algorithms where SIMD-style control flows exist (SIMD=single instruction, multiple data).

• Steve Kommrusch 9 Jul 2025 19:59 UTC

  1 point

  0

  in reply to: Raemon’s comment on: Foom & Doom 1: “Brain in a box in a base­ment”

  I would include “constructivist learning” in your list, but I agree that LLMs seem capable of this. By “constructivist learning” I mean a scientific process where the learning conceives of an experiment on the world, tests the idea by acting on the world, and then learns from the result. A VLA model with incremental learning seems close to this. RL could be used for the model update, but I think for ASI we need learnings from real-world experiments.

• Steve Kommrusch 9 Jul 2025 17:47 UTC

  1 point

  0

  on: Foom & Doom 1: “Brain in a box in a base­ment”

  This post provides a good overview of some topics I think need attention by the ‘AI policy’ people at national levels. AI policy (such as the US and UK AISI groups) has been focused on generative AI and recently agentic AI to understand near-term risks. Whether we’re talking LLM training and scaffolding advances, or a new AI paradigm, there is new risk when AI begins to learn from experiments in the world or reasoning about its own world model. In child development, imitation learning focuses on learning from examples, while constructivist learning focuses on learning by reflecting on interactions with the world. Constructivist learning is, I expect, key to push past AGI to ASI and caries obvious risks to alignment beyond imitation learning.

  In general, I expect something LLM-like (i.e. transformer models or an improved derivative) to be able to reach ASI with a proper learning-by-doing structure. But I also expect ASI could find and implement a more efficient intelligence algorithm once ASI exists.

  1.4.1 Possible counter: “If a different, much more powerful, AI paradigm existed, then someone would have already found it.”

  This paragraph tries to provide some data for a probability estimate of this point. AI as a field has been around at least since the Dartmouth conference in 1956. In this time we’ve had Eliza, Deep Blue, Watson, and now transformer-based models including OpenAI o3-pro. In support of Steven’s position, one could note that AI research publications are much higher now that during the previous 70 years, but at the same time many AI ideas have been explored and the current best results are with models based on the 8-year-old “Attention is all you need” paper. To get a sense for the research rate, we can note that the doubling time for AI/​ML research papers per month was about 2 years between 1994 and 2023 according to this Nature paper. Hence, every 2 years we have about as many papers as created in the last 70 years. I don’t expect this doubling can continue forever, but certainly many new ideas are being explored now. If a ‘simple model’ for AI exists and it’s discovery is, say, randomly positioned on a given AI/​ML research paper published between 1956 and ASI achievement then one could estimate the probability of the paper’s position using this simplistic research model. If ASI is only 6 years out and the doubling every 2 years continues, then almost 90% of the AI/​ML research papers before ASI are still in the future. Even though many of these papers are LLM focused, there is still active work in alternative areas. But even though the foundational paper for ASI may yet be in our future, I would expect something like a ‘complex’ ML model will win out (for example, Yann LeCun’s ideas involve differentiable brain modules). And the solution may or may not be more compute-intensive than current models. The brain compute estimates vary widely and the human brain has been optimized by evolution for many generations. In short, it seems reasonable to expect another key idea before ASI, but I would not expect it to be a simple model.

• Steve Kommrusch 26 Aug 2024 22:43 UTC

  2 points

  1

  in reply to: sanxiyn’s comment on: Limi­ta­tions on For­mal Ver­ifi­ca­tion for AI Safety

  From what I gather reading the ACAS X paper, it formally proved a subset of the whole problem and many issues uncovered by using the formal method were further analyzed using simulations of aircraft behaviors (see the end of section 3.3). One of the assumptions in the model is that the planes react correctly to control decisions and don’t have mechanical issues. The problem space and possible actions were well-defined and well-constrained in the realistic but simplified model they analyzed. I can imagine complex systems making use of provably correct components in this way but the whole system may not be provably correct. When an AI develops a plan, it could prefer to follow a provably safe path when reality can be mapped to a usable model reliably, and then behave cautiously when moving from one provably safe path to another. But the metric for ‘reliable model’ and ‘behave cautiously’ still require non-provable decisions to solve a complex problem.

• Steve Kommrusch 26 Aug 2024 21:29 UTC

  3 points

  −2

  in reply to: Steve_Omohundro’s comment on: Limi­ta­tions on For­mal Ver­ifi­ca­tion for AI Safety

  Steve, thanks for your explanations and discussion. I just posted a base reply about formal verification limitations within the field of computer hardware design. In that field, ignoring for now the very real issue of electrical and thermal noise, there is immense value in verifying that the symbolic 1′s and 0′s of the digital logic will successfully execute the similarly symbolic software instructions correctly. So the problem space is inherently simplified from the real world, and the silicon designers have incentive to build designs that are easy to test and debug, and yet only small parts of designs can be formally verified today. It would seem to me that, although formal verification will keep advancing, AI capabilities will advance faster and we need to develop simulation testing approaches to AI safety that are as robust as possible. For example, in silicon design one can make sure the tests have at least executed every line of code. One could imaging having a METR test suite and try to ensure that every neuron in a given AI model has been at least active and inactive. It’s not a proof, but it would speak to the breadth of the test suite in relation to the model. Are there robustness criteria for directed and random testing that you consider highly valuable without having a full safety proof?

• Steve Kommrusch 26 Aug 2024 19:54 UTC

  8 points

  1

  on: Limi­ta­tions on For­mal Ver­ifi­ca­tion for AI Safety

  Thanks for the study Andrew. In the field of computer hardware design, formal verification is often used on smaller parts of the design, but randomized dynamic verification (running the model and checking results) is still necessary to test corner cases in the larger design. Indeed, the idea that a complex problem can be engineered so as to be easier to formally verify is discussed in this recent paper formally verifying IEEE floating point arithmetic. In that paper, published in 2023, they report using their divide-and-conquer approach on the problem resulting in a 7.5 hour run time to prove double-precision division correct. Another illustrative example is given by a paper from Intel which includes the diagram below showing how simulation is relied on for the Full IP level verification of complex systems.

  

  This data supports your points in limitations 2 and 3 and shows the difficulty in engineering a system to be easily proven formally. Certainly silicon processor design has had engineer-millennia spent on the problem of proving the design correct before manufacturing. For AI safety, the problem is much more complex than ‘can the silicon boot the OS and run applications?’ and I expect directed and random testing will need to be how we test advanced AI as we move towards AGI and ASI. AI can help improve the quality of safety testing including contributing to red-teaming of next-generation models, but I doubt it will be able to help us formally prove correctness before we actually have ASI.

• Steve Kommrusch 15 Aug 2024 21:43 UTC

  1 point

  0

  on: Fields that I refer­ence when think­ing about AI takeover prevention

  An interesting grouping of fields. While my recent work is in with AI and machine learning, I used to work in the field of computer hardware engineering and have thought there are a lot of key parallels:
  1) Critical usage test occurs after design and validation
  In the case of silicon design, the fabrication of the design is quite expensive and so a lot of design techniques are included to facilitate debug after manufacturing and also a lot of effort is put in to pre-silicon validation. In the case of transformative AI, using the system after development and training is where the potential risk arises (certainly human-level AI includes currently known safety issues like chemical, biological, and nuclear capabilities while ASI adds in nanotech, etc).
  2) Design validation is an inherent cost of creating a quality product
  In the case of silicon design, a typical team tends to have more people doing design testing than doing the actual design. Randomized testing, coverage metrics, and other techniques have developed over the decades as companies work to reduce risk of discovering a critical failure after the silicon chip is manufactured. In the case of AI (such as current LLMs) companies invest in RLHF, use Constitutional AI, and other techniques. I’d like to see more commonality between agreed-upon AI safety techniques so that we can get to the next point…
  3) An ecosystem of companies and researchers arise to provide validation services
  In the case of silicon design, again driven by market needs, design companies are willing to purchase tools from 3rd parties to help improve verification coverage before the expense of silicon fabrication. The existence of this market helps standardize processes as companies compete to provide measurable benefit to the silicon design companies. I’d love to see more use and recognition of 3rd party AI safety entities (such as METR, Apollo, and the UK AISI) and have them be striving to thoroughly test and evaluate AI products in the way companies like Synopsys, Cadence, and Mentor Graphics help test and evaluate silicon designs.
  4) Current systems can be used to develop next generation systems
  In the case of silicon design, the prior generation of the same product is often used to develop the next generation (this is certainly true with CPU design, but also applies to graphics processors and some peripheral silicon products like memories). In the case of AI, one can use AI to help test and evaluate the next generation (such as Paul Christiano’s IDA and to some extent Anthropic’s constitutional AI). As AI advances, I expect using AI to help verify future AI will become common practice due to AI’s increasing capabilities.
  
  There are other analogies between AI safety and silicon design that I’ve considered, but those 4 give a sense for my thoughts about the issue.

• Steve Kommrusch 19 May 2024 21:38 UTC

  1 point

  0

  in reply to: Foyle’s comment on: Su­per-Ex­po­nen­tial ver­sus Ex­po­nen­tial Growth in Com­pute Price-Performance

  The Tom’s Hardware article is interesting, thanks. It makes the point that the price quoted may not include the full ‘cost of revenue’ for the product in that it might be the bare die price and not the tested and packaged part (yields from fabs aren’t 100% so extensive functional testing of every part adds cost). The article also notes that R&D costs aren’t included in that figure; the R&D for NVIDIA (and TSMC, Intel, AMD, etc) are what keep that exponential perf-per-dollar moving along.
  
  For my own curiosity, I looked into current and past income statements for companies. Today, NVIDIA’s latest balance sheet for the fiscal year ending 1/​31/​2024 has $61B in revenue, 17B for cost of revenue (that would include the die cost, as well as testing and packaging), R&D of 9B, and a total operating income of 33B. AMD for their fiscal year ending 12/​31/​2023 had $23B revenue, 12B cost of revenue, 6B R&D, and 0.4B operating income. Certainly NVIDIA is making more profit, but the original author and wikipedia picked the AMD RX 7600 as the 2023 price-performance leader and there isn’t much room in AMD’s income statement to lower those prices. While NVIDIA could cut their revenue in half and still make a profit in 2023, in 2022 their profit was 4B on 27B in revenue. FWIW, Goodyear Tire, selected by me ‘randomly’ as an example of a company making a product with lower technology innovation year-to-year, had 20B revenue for the most recent year, 17B cost of revenue, and no R&D expense. So if we someday plateau silicon technology (even if ASI can help us build transistors smaller than atoms, the plank length is out there at some point), then maybe silicon companies will start cutting costs down to bare manufacturing costs. As a last study, the wikipedia page on FLOPS cited the Pentium Pro from Intel as part of the 1997 perf-per-dollar system. For 1997, Intel reported 25B in revenues, 10B cost of sales (die, testing, packaging, etc), 2B in R&D, and an operating income of 10B; so it was spending a decent amount on R&D too in order to stay on the Moore’s law curve.
  
  I agree with Foyle’s point that even with successful AGI alignment the socioeconomic implications are huge, but that’s a discussion for another day...

• Steve Kommrusch 16 May 2024 22:06 UTC

  1 point

  0

  on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  This is very interesting work, showing the fractal graph is a good way to visualize the predictive model being learned. I’ve had many conversations with folks who struggle with the idea ‘the model is just predicting the next token, how can it be doing anything interesting’?. My standard response had been that conceptually the transformer model matches up tokens at the first layer (using the key and query vectors), then matches up sentences a few layers up, and then paragraphs a few layers above that; hence the model, when presented with an input, was not just responding with ‘the next most likely token’, but more accurately ‘the best token to use to start the best sentence to start the best paragraph to answer the question’. Which usually helped get the complexity across; but I like the learned fractal of the belief state and will see how well I can use that in the future.

  For future work, I think it would be interesting to tease out how the system learns 2 interacting state machines (this may give hints regarding its ability to generalize different actors in the world). For example, consider another 3-state HMM with the same transition probabilities but behaving independent of the 1st HMM. Then have the probability of outputting A,B, or C be the average of the arcs taken on the 2 HMMs each step. For example, if the 1st HMM is in H0 and stays in H0 it gives a 60% chance of generating A and a 20% chance for B and C, while if the 2nd HMM is in H2 and stays in H2, it gives 20% for A and B and 60% for C, so the overall output probability is 40% A, 20% B, 40%C for my example. Now certainly this is a 9 state HMM (3x3), but it’s more simply represented as two 3-state HMMs, what would the neural network learn? What if you combined 3 HMMs this way, so the single HMM is 3x3x3=27 states, but the simpler representation is 3+3+3=9? Again, my goal here would be to understand how the system might model multiple agents in the world given limited visibility to the agents directly. Perhaps there is a cleaner way to explore the same question.

• Steve Kommrusch 15 May 2024 21:47 UTC

  1 point

  0

  on: Su­per-Ex­po­nen­tial ver­sus Ex­po­nen­tial Growth in Com­pute Price-Performance

  Thanks for the interesting and thoughtful article. As a current AI researcher and former silicon chip designer, I’d suspect that our perf-per-doller is trending a bit slower than exponential now and not a hyperexponential. My first datapoint in support of this is the data from https://​​en.wikipedia.org/​​wiki/​​FLOPS which shows over 100X perf/​​dollar improvement from 1997 to 2003 (6 years), but the 100X improvement from 2003 is in 2012 (9 years), and our most recent 100X improvement (to the AMD RX 7600 the author cites) took 11 years. This aligns with TOP500 compute performance, which is progressing at a slower exponential since about 2013: https://​​www.nextplatform.com/​​2023/​​11/​​13/​​top500-supercomputers-who-gets-the-most-out-of-peak-performance/​​ . I think that a real challenge to the future scaling is the size of the silicon atom relative to current (marketing-skewed) process nodes supported by TSMC, Intel, and others. I don’t think our silicon performance will flatline in the 2030′s as implied by https://​​epochai.org/​​blog/​​predicting-gpu-performance , but it could be that scaling FET-based geometries becomes very difficult and we’ll need to move away from the basic FET-based design style used for last 50 years to some new substrate, which will slow the exponential for a bit. That said, I think that even if we don’t get full AGI by 2030, the AI we do have by 2030 will be making real contributions to silicon design and that could be what keeps us from dipping too much below an exponential. But my bet would be against a hyperexponential playing out over the next 10 years.