Roman Leventov

Karma: 1,338

An independent researcher/blogger/philosopher about intelligence and agency (esp. Active Inference), alignment, ethics, interaction of the AI transition with the sociotechnical risks (epistemics, economics, human psychology), collective mind architecture, research strategy and methodology.

Twitter: https://twitter.com/leventov. E-mail: leventov.ru@gmail.com (the preferred mode of communication). I’m open to collaborations and work.

Presentations at meetups, workshops and conferences, some recorded videos.

I’m a founding member of the Gaia Consoritum, on a mission to create a global, decentralised system for collective sense-making and decision-making, i.e., civilisational intelligence. Drop me a line if you want to learn more about it and/or join the consoritum.

You can help to boost my sense of accountability and give me a feeling that my work is valued by becoming a paid subscriber of my Substack (though I don’t post anything paywalled; in fact, on this blog, I just syndicate my LessWrong writing).

For Russian speakers: русскоязычная сеть по безопасности ИИ, Telegram group.

Roman Leventov 15 Feb 2023 16:26 UTC
27 points
18
on: Bing Chat is blatantly, aggressively misaligned
This looks like a very good moment for the AI safety community to push for a much more deliberate approach to R&D at AGI labs. We must not squander this moment. This is not quite a “sinister stumble”, an event of another kind, but of comparable positive optionality for AI safety.
By “a more deliberate approach to R&D”, I mean researching much more deeply from both theoretical (scientific) and interpretability standpoints what’s going on with (self-)awareness, agency, and feeling in these networks, and publishing their results academically.

Roman Leventov 1 Aug 2022 10:14 UTC
24 points
10
on: chinchilla’s wild implications
My two cents contra updates towards longer or more uncertain AGI timelines given the information in this post:
- The training of language models is many orders of magnitude less efficient than the training of the human brain, which acquires comparable language comprehension and generation ability on a tiny fraction of the text corpora discussed in this post. So we can expect more innovations that improve the training efficiency. Even one such innovation, improving the training efficiency (in terms of data) by a single order of magnitude, would probably ensure that the total size of publically available text data is not a roadblock on the path to AGI, even if it is, currently. I think the probability that we will see at least one such innovation in the next 5 years is quite high, more than 10%.
- Perhaps DeepMind’s Gato is already a response to the realisation that “there is not enough text”, explained in this post. So they train Gato on game replays, themselves generated programmatically, using RL agents. They can generate practically unlimited amounts of training data in this way. Then there is probably a speculation that at some scale, Gato will generalise the knowledge acquired in games to text, or will indeed enable much more efficient training on text, (a-la few-shot learning in current LMs) if the model is pre-trained on games and other tasks.

Roman Leventov 26 Feb 2023 8:25 UTC
22 points
10
on: Cognitive Emulation: A Naive AI Safety Proposal
I understood the proposal as “let’s create relatively small, autonomous AIs of multi-component cognitive architecture instead of unitary DNNs becoming global services (and thus ushering cognitive globalisation)”. The key move seems to be that you want to increase the AI architecture’s interpretability by pushing some intelligence into the multi-component interaction from the “DNN depths”. In this setup, components may remain relatively small (e.g., below 100B parameter scale, i.e., at the level of the current SoTA DNNs), while their interaction leads to the emergence of general intelligence, which may not be achievable for unitary DNNs at these model scales.
The proposal seems to be in some ways very close to the recent Eric Drexler’s Open Agency Model. Both your proposal and Drexler’s “open agencies” seem to noticeably allude to the (classic) approaches to cognitive architecture, a la OpenCog, or LeCun’s H-JEPA.
As well as Drexler, it seems that you take for granted that multi-component cognitive architectures will have various “good” properties, ranging from interpretability to retargetability (which Michael Levin calls persuadability, btw). However, as I noted in this comment, none of these “good properties” are actually granted for an arbitrary multi-component AI. It must be demonstrated for a specific multi-component architecture why it is more interpretable (see also in the linked comment my note that “outputting plans” != interpretability), persuadable, robust, and ethical than an alternative architecture.
Your approach to this seems to be “minimal”, that is, something like “at least we know the level of interpretability, persuadability/corrigibility, robustness, and ethics of humans, so let’s try to build AI ‘after humans’ so that we get at least these properties, rather than worse properties”.
As such, I think this approach might not really solve anything and might not “end the acute risk period”, because it amounts to “building more human-like minds”, with all other economic, social, and political dynamics intact. I don’t see how building “just more humans” prevents corporations from pursuing cognitive globalisation in one form or another. The approach just seems to add more “brain-power” to the engine (either via building AI “like humans, but 2-3 sigmas more intelligent”, or just building many of them and keeping them running around the clock), without targeting any problems with the engine itself. In other words, the approach is not placed within a frame of a larger vision for civilisational intelligence architecture, which, I argued, is a requirement for any “AI safety paradigm”: “Both AI alignment paradigms (protocols) and AGI capability research (intelligence architectures) that don’t position themselves within a certain design for civilisational intelligence are methodologically misguided and could be dangerous.”
Humans have rather bad interpretability (see Chater’s “The Mind is Flat”), bad persuadability (see Scott Alexander’s “Trapped Priors”), bad robustness (see John Doyle’s hijackable language and memetic viruses), poor capability for communication and alignment (as anyone who tried to reliably communicate any idea to anyone else or to align with anyone else on anything can easily attest; cf. discussion of communication protocols in “Designing Ecosystems of Intelligence from First Principles”), and, of course, poor ethics. It’s also important to note that all these characteristics seem to be largely uncorrelated (at least, not strictly pegged) with raw general intelligence (GI factor) in humans.
I agree with you and Friston and others who worry about the cognitive globalisation and hyperscaling approach, but I also think that in order to improve the chances of humanity, we should at least aim at better than human architecture from the beginning. Creating many AI minds of the architecture “just like humans” (unless you count on some lucky emergence and that even trying to target “at humans” will yield architecture “better than humans”) doesn’t seem to help in civilisational intelligence and robustness, just accelerate the current trends.
I’m also optimistic because I see nothing impossibly hard or intractable in designing cognitive architectures that would be better than humans from the beginning and getting good engineering assurances that the architecture will indeed yield these (better than human) characteristics. It just takes time and a lot of effort (but so as architecturing AI “just like humans” does). E.g., (explicit) Active Inference architecture seems to help significantly at least with interpretability and the capacity for reliable and precise communication and, hence, belief and goal alignment (see Friston et al., 2022).
What links here?
- AGI will be made of heterogeneous components, Transformer and Selective SSM blocks will be among them by Roman Leventov (27 Dec 2023 14:51 UTC; 33 points)
- AGI will be made of heterogeneous components, Transformer and Selective SSM blocks will be among them by Roman Leventov (EA Forum; 27 Dec 2023 14:51 UTC; 5 points)

Roman Leventov 24 Sep 2023 9:55 UTC
15 points
8
on: There should be more AI safety orgs
@Nathan Helm-Burger’s comment made me think it’s worthwhile to reiterate here the point that I periodically make:
Direct “technical AI safety” work is not the only way for technical people (who think that governance & politics, outreach, advocacy, and field-building work doesn’t fit them well) to contribute to the larger “project” of “ensuring that the AI transition of the civilisation goes well”.
Now, as powerful LLMs are available, is the golden age to build innovative systems and tools to improve^[1]:
- Politics: see https://cip.org/, Audrey Tang’s projects
- Social systems: innovative LLM/AI-first social networks that solve the social dilemma? (I don’t have a good existing examples of such projects, though)
- Psychotherapy, coaching: see Inflection
- Economics: see Verses, One Project, the Gaia Consortium
- Epistemic infrastructure: see Subconscious Network, Ought, the Cyborgism agenda, Quantum Leap (AI safety edtech)
- Authenticity infrastructure: see Optic, proof-of-personhood projects
- Cybersec/infosec: see various AI startups for cybersecurity, trustoverip.org
- More?
I believe that if such projects are approached with integrity, thoughtful planning, and AI safety considerations at heart rather than with short-term thinking (specifically, not considering how the project will play out if or when AGI is developed and unleashed on the economy and the society) and profit-extraction motives, they could shape to shape the trajectory of the AI transition in a positive way, and the impact may be comparable to some direct technical AI safety/alignment work.
In the context of this post, it’s important that the verticals and projects mentioned above could either be conventionally VC-funded because they could promise direct financial returns to the investors, or could receive philanthropic or government funding that wouldn’t otherwise go to technical AI safety projects. Also, there is a number of projects in these areas that are already well-funded and hiring.
Joining such projects might also be a good fit for software engineers and other IT and management professionals who don’t feel they are smart enough or have the right intellectual predispositions to do good technical research, anyway, even there was enough well-funded “technical AI safety research orgs”. There should be some people who do science and some people who do engineering.
1. ^
  I didn’t do serious due diligence and impact analysis on any of the projects mentioned. The mentioned projects are just meant to illustrate the respective verticals, and are not endorsements.
What links here?
- Roman Leventov's comment on Memo on some neglected topics by Lukas Finnveden (EA Forum; 13 Nov 2023 20:59 UTC; 3 points)

Roman Leventov 6 May 2023 16:32 UTC
15 points
0
in reply to: the gears to ascension’s comment on: Orthogonal’s Formal-Goal Alignment theory of change
I just left two long comments on this post with a critique.

Roman Leventov 31 Mar 2023 6:44 UTC
15 points
7
in reply to: Cameron Holmes’s comment on: “Dangers of AI and the End of Human Civilization” Yudkowsky on Lex Fridman
Yes, unfortunately, Eliezer’s delivery was suffering in many places from assuming that listeners have a lot of prior knowledge/context.
If he wishes to become a media figure going forward (which looks to me like an optimal thing to do for him at this point), this is one of the most important aspects to improve in his rhetoric. Pathos (the emotional content) is already very good, IMO.

Roman Leventov 27 Mar 2024 3:16 UTC
13 points
8
on: Modern Transformers are AGI, and Human-Level
Cf. DeepMind’s “Levels of AGI” paper (https://arxiv.org/abs/2311.02462), calling modern transformers “emerging AGI” there, but also defining “expert”, “virtuoso”, and “superhuman” AGI.

Roman Leventov 20 Sep 2023 13:53 UTC
13 points
0
on: Formalizing «Boundaries» with Markov blankets + Criticism of this approach
Eh, I think Markov blankets probably aren’t the way to go to represent “boundaries”. Markov blankets can’t follow entities through space as well as you might imagine. Suppose you have a Bayesian network whose variables are the physical properties of small regions in space-time. A Markov blanket is a set of nodes (with specific conditional independence requirements, but that part isn’t important right now). Say your probabilistic model contains a grasshopper who might hop in two different directions, depending on stimuli. Since your model represents both possibilities, you can’t select a Markov blanket to contain all and only the grasshopper. You’re forced to include some empty air (if you include either of the hop-trajectories inside your Markov blanket) or to miss some grasshopper (if you include only space-time regions certain to contain the grasshopper, by only circling the grasshopper before it makes the uncertain jump).
This treats spacetime as fundamental, and the voxels of spacetime as the “bearers” measurable physical properties. I believe this is wrong: physics studies objects (systems) and tries to predict their interactions (i.e., the behaviour) via assigning them certain properties. The 4D spatiotemporal coordinates/extent of the object is itself such as measurable property, along with energy, momentum, mass, charge, etc. And a growing number of physicists think that at least space (i.e., position) property is not fundamental, but rather emergent of the topology of the interaction graph of the objects.
Note that what I write above is agnostic on whether the universe is computational/mathematical (Tegmark, Wolfram, …) or not (most of the physics establishment), or whether this is a sensible question.
I can guess that the idea that voxels of spacetime is what we should attach physical properties to comes from quantum field theory and the standard model, which are spacetime-embedded.
Quantum FEP/Active Inference (see also the lecture series exposition) is a scale-free version of Active Inference. When some agent (a human scientist or a lizard, or the grasshopper itself) models the grasshopper, their best predictive model of the dynamics of the grasshopper is not in terms of particle fields. Rather, it’s in terms of the grasshopper (an object/physical system), it’s physiological properties (age, point in the circadian cycle, level of hunger, temperature, etc.) and its distances and exposures to the relevant stimuli, such as the topology of light and shade around it, gradients of certain attractive smells, and so on. So, under this level of coarse graining, we treat these as the “grasshopper’s properties”. Whether the grasshopper jumps left or right doesn’t affect grasshopper’s boundary in this model, but affects its properties. And the probabilistic distributions over these properties is what we happily model the grasshopper’s dynamics (most likely trajectories) with.
Note, however, that the conception of boundaries as holographic screens for information (which is what Quantum FEP is doing) is not watertight. All systems in the universe are only approximately separable from their environments but aren’t perfectly separable. Thus, their holographic boundaries are approximate.

Roman Leventov 14 Jul 2023 6:36 UTC
13 points
8
on: Elon Musk announces xAI
I think it is going to be pro-humanity from the standpoint that humanity is just much more interesting than not-humanity.
This is not obvious, to say the least. In fact, I think this is unlikely. Non-humanity might very well be more interesting to non-human minds. E.g., if there is a maximally curious AI, it might be much more interested in tiling the surface of Earth with its conspecifics and marvelling at the complexity and interestingness of the emerging society of AIs, compared to which the humanity may appear to be utterly boring.

Roman Leventov 12 Apr 2023 7:40 UTC
13 points
1
on: Where’s the foom?
Scientific-style risk assessments should be positioned in the frame of some science. For all the examples that you gave (atomic bomb, vacuum decay, etc.) that was obviously physics.
To formally assess and quantify the risks of misaligned AI, the calculations should be done within a frame of the science of intelligence (i.e., “abstract” cognitive science, not tied to the specifics of the human brain) and agency.
No such general science of intelligence exists at the moment. There are frameworks that aspire to describe intelligence, agency, and sentience in general, such as Active Inference (Friston et al., 2022), thermodynamic ML (Boyd et al., 2022), MCR^2 (Ma et al., 2022), and, perhaps, Vanchurin’s “neural network toolbox for describing the world and agents within it”. Of these, I’m somewhat familiar only with Active Inference and Vanchun’s work, and it seems to me that both of them are too general frameworks to be the basis of concrete risk calculations. I’m not familiar with thermodynamic ML work and MCR^2 work at all (a recurrent note: if would be very useful if the alignment community had people who are at least deeply familiar with and conversant in these frameworks).
Then, there are more concrete engineering proposals for A(G)I architectures, such as Transformer-based LLMs, RL agents of specific architectures, or LeCun’s H-JEPA architecture. Empirical and more general theoretical papers have been published about the risks of goal misgeneralisation for RL agents (I believe these works are referenced in Ngo et al.’s “The Alignment Problem from a Deep Learning Perspective”). I hope at least in part due to this (although he doesn’t explicitly acknowledge this, as far as I know), LeCun is sceptical about RL and advocates for minimising RL from intelligence architectures.
Theoretical analysis of the risks of the H-JEPA architecture hasn’t been done yet (let alone empirical, because H-JEPA agents haven’t been constructed yet), however, H-JEPA belongs to a wider class of intelligence architectures that employ energy-based modelling (EBM) of the world. EBM agents have been constructed and could be analysed empirically already for their alignment properties. It’s worth noting that energy-based modelling is conceptually rather similar to Active Inference, and Active Inference agents have been constructed, too (Fountas et al., 2020), and thus could be probed empirically for their alignment properties as well.
LLMs are also tested empirically for alignment (e.g., OpenAI’s evals, the Machiavelli benchmark). Theoretical analysis of alignment in future LLMs is probably bottlenecked by better theories of transformers and auto-regressive LLMs in general, such as Roberts, Yaida, and Hanin’s deep learning theory (2021), Anthropic’s mathematical framework for transformers (2021), Edelman et al.’s theory of “sparse variable creation” in transformers (2022), Marciano’s theory of DNNs as a semi-classical limit of topological quantum NNs (2022), Bahri et al.’s review of statistical mechanics of deep learning (2022), and other general theories of mechanistic interpretability and representation in DNNs (Räukur et al. 2022). See here for a more detailed discussion.
The fundamental limitation of analysing the risks of any specific architecture is that once AGI starts to rewrite itself, it could probably easily depart from its original architecture. Then then the question is whether we can organizationally, institutionally, and economically prevent AGIs from taking control of their own self-improvement. This question doesn’t have a scientific answer and it’s impossible to rigorously put a probability on this event. I would only say that if AGI is available in open-source, I think the probability of this will be approximately 100%: surely, some open-source hacker (or, more likely, dozens of them) will task AGI with improving itself, just out of curiosity or due to other motivations.
Likewise, even if AGI is developed strictly within tightly controlled and regulated labs, it’s probably impossible to rigorously assess the probability that the labs will drift into failure (e.g., will accidentally release an unaligned version of AGI through AI), or will “leak” the AGI code or parameter weights. I suspect the majority of AI x-risk in the assessments of MIRI people and others is not even purely technical (“as if we had a perfect plan and executed it perfectly”), but from these kinds of execution, organisation resilience, theft, governmental or military coercion, and other forms of not strictly technical and therefore not formally assessable risk.

Roman Leventov 6 Sep 2022 18:26 UTC
13 points
6
on: Simulators
Overall, I agree with most of this post, thanks for writing it.
The term “Simulator” has a potentially dangerous connotation of precision and reliability
I agree with your discussion of the importance of having the right vocabulary. However, I feel that the term “simulator” that you propose has a nagging flaw: that is, it invokes the connotation of “precision simulation” in people with a computer engineering background, so perhaps in most alignment researchers (rather than, I guess, the main connotation invoked in the general public, as in “Alice simulated illness to skip classes”, which is actually closer to what GPT does). Additionally, the simulation hypothesis sometimes (though not always) assumes a “precision simulation”, not an “approximate simulation” a.k.a. prediction, which GPT really does and will do.
To me, it’s obvious that GPT-like AIs will always be “predictors”, not “precision simulators” because of computation boundedness and context (prompt, window) boundedness.
Why this false connotation of precision is bad? Because it seems to lead to over-estimation of simulacra rolled out by GPT. Such as in the following sentence:
Simulators like GPT give us methods of instantiating intelligent processes, including goal-directed agents, with methods other than optimizing against a reward function.
At the very least, the statement that GPT can “instantiate intelligent processes, including goal-directed agents” should be proven (it’s very far from obvious for me that what is produced by GPT in such cases can be called an intelligent agent), and I feel there is much more nuance to it than thinking of GPT as a “simulator” tempts you to claim.
What term do I propose instead? To me, it’s somewhere in the conceptual cloud of meaning between the verbs “simulate”, “imagine”, “predict”, “generate”, “reason”, and “sample”. Perhaps, we should better coin a new term.
If we are accustomed to thinking of AI systems as corresponding to agents, it is natural to interpret behavior produced by GPT – say, answering questions on a benchmark test, or writing a blog post – as if it were a human that produced it. We say “GPT answered the question {correctly|incorrectly}” or “GPT wrote a blog post claiming X”, and in doing so attribute the beliefs, knowledge, and intentions revealed by those actions to the actor, GPT (unless it has ‘deceived’ us).
But when grading tests in the real world, we do not say “the laws of physics got this problem wrong” and conclude that the laws of physics haven’t sufficiently mastered the course material. If someone argued this is a reasonable view since the test-taker was steered by none other than the laws of physics, we could point to a different test where the problem was answered correctly by the same laws of physics propagating a different configuration. The “knowledge of course material” implied by test performance is a property of configurations, not physics.
This analogy is rather confusing because there is no information about the right answers to the test stored in the laws of physics, but there is a lot of information stored in the “GPT rule”, which is (and will be) far from pure “laws of physics” simulator, but full of factual knowledge and heuristics. This is because of the issue discussed elsewhere in the post: prompts under-determine the simulacra, but GPT has to generalise from that nonetheless.
The implication of this observation is that, I think, in a certain sense, it’s reasonable to say that “GPT wrote a blog post”, much more reasonable than “laws of physics wrote a blog post”, though perhaps less reasonable than “an agent wrote a blog post”. But I think it’s not right to declare (which seems to me, you do) that people make a semantic mistake when they say “GPT wrote a blog post”.
In the simulation ontology, I say that GPT and its output-instances correspond respectively to the simulator and simulacra. GPT is to a piece of text output by GPT as quantum physics is to a person taking a test, or as transition rules of Conway’s Game of Life are to glider. The simulator is a time-invariant law which unconditionally governs the evolution of all simulacra.
Putting quantum physics and a person taking a test in this row of analogies is problematic for at least two reasons: 1) quantum physics is not all the physics there is (at least as long as there is no theory of quantum gravity), 2) ontologically, seeing the laws of physics as a simulator and the reality around us as its simulacra is just one interpretation of what the laws of physics really are (roughly—realism). But there are also non-realist views.
Learned simulations can be partially observed and lazily-rendered, and still work. A couple of pages of text severely underdetermines the real-world process that generated text, so GPT simulations are likewise underdetermined. A “partially observed” simulation is more efficient to compute because the state can be much smaller, but can still have the effect of high fidelity as details can be rendered as needed.
Invoking the phrase “partially observed” is very confusing here, because we are not talking about some ground state of the world and the observation window into it (as in partially observed Markov decision process), but of something very different from that.
The tradeoff is that it requires the simulator to model semantics – human imagination does this, for instance – which turns out not to be an issue for big models.
What do you mean by “modelling semantics” here?
What links here?
- Roman Leventov's comment on Simulators by janus (6 Sep 2022 18:51 UTC; 4 points)

Roman Leventov 18 Dec 2023 20:54 UTC
12 points
2
on: OpenAI: Preparedness framework
Everyone on Twitter has criticised the label “Responsible Scaling Policy”, but the author of this post seems not to respect what seems like a gentle attempt by OpenAI to move past this label.
If we were a bit more serious about this we would perhaps immediately rename the tag “Responsible Scaling Policies” on LessWrong into “Preparedness Frameworks” with a note on the tag page “Anthropic calls their PF ‘RSP’, but we think this is a bad label”.

Roman Leventov 22 Oct 2023 8:12 UTC
12 points
5
on: Muddling Along Is More Likely Than Dystopia
Thanks for pointing this out. I find it particularly epistemically suspicious that all the quoted people (that currently predict grim future for humanity in the next 100 years if AGI won’t be built and deployed massively) could hardly have median AGI timelines shorter than 30 years 5 years ago, with a significant probability weight for 70+ year timelines, yet they didn’t voice this as a significant concern, even existential risk, at that time. And I don’t think anything that has happened in the last 5 years should have made them so much more pessimistic. Trump already won the US election, Brexit was already voted for, the erosion of epistemic and civic commons, and the democratic backsliding was already happening, the climate situation was basically as obviously grave as it is today. Civilisational unpreparedness to a major pandemic was also obvious, as that nothing was being made to fix that.

This suggests to me that these people currently converged on thinking “x-risk from building AGI is high, but the risk for the civilisational collapse in the next 100 years without AGI is even higher” is a trick of mind that would probably not have happened if AGI timelines didn’t so dramatically shrink for everyone.

I’m also aware that Will MacAskill argued for 30% chance of stagnation and civilisational collapse in this century in “What We Owe The Future” book last year, that could have prompted various people to update. But I was mostly unconvinced by that argument and I wonder if that argument itself was an example of the same psychological reaction (or a sort of counter-reaction) to massively shrinking timelines.

Roman Leventov 16 Jan 2023 4:29 UTC
12 points
0
on: Review AI Alignment posts to help figure out how to make a proper AI Alignment review
I feel that LW is quite bad as the system for performing AI safety research. Most likely worse than the traditional system of academic publishing, in aggregate. A random list of things that I find unhelpful:
- Very short “attention timespan”. People read posts within a couple of days from publishing unless curated (but curation is not a solution, because it also either happens or not within a short time window, and is a subjective judgement of a few moderators), perhaps within a few weeks, unless hugely upvoted. And a big “wave of upvotes” is also a somewhat immediate reaction rather than a research appreciation.
  - Contrast: academic papers get read when they have many citations. Citations accumulate over time. Citations are a much more reliable indicator of a paper’s usefulness (exception: controversial takes by high-profile authors that invoke a lot of counter-takes). I suspect a paper’s “peak read” is somewhere within a year of the publishing date, but definitely not the day 1 after publishing.
  - Given LW/AF’s naturally faster turnover of posts and ideas (also necessitated by the ultra-fast AI progress), if it adopted showing on the front page posts that accumulated the most citations (backlinks) on these forums in the last few months, it could be more useful for guiding research.
- No standards and practices of review. No expectation that established researchers review the work of novices, thus disseminating ideas and helping novices to fix their conceptual confusion. Unstructured comments are way too “poor man’s” reviews, they often cherry-pick a small point and the ensuing argument loses the big picture quickly, making it largely unhelpful.
  - Yann LeCun proposed a review system here that seems to be implementable on LW/AF. Importantly, a full review of a post (a “paper”) could give it a “rating”, but a mere comment can’t. People can subscribe to streams of posts, approved (or highly rated) by specific researchers or groups (“Reviewing Entities”).
  - The cumulative rating of the post could also be used to guide the community’s attention. But note that reading a big post and writing a full review of it and giving it a rating or approval is much harder than just clicking the upvote button, and is in itself an intellectual bar. The “crowd” that just clicks upvotes (usually before even reading the post beyond the title) shouldn’t guide the attention of the research community.
  - The current review system on LW emphasises focusing on specific lines and claims, rather than on the big picture, and doesn’t guide the reviewers to ask the right questions about the work.
- No community standards (nor LW native support, e. g. for a dedicated “references” section) for referencing prior work. An academic paper in physics, computer science, biology, etc. with zero or a couple of references to prior work is unfathomable, in LW/AF this is standard practice (including for posts attempting to make research contributions: develop new theories, etc.). Also, these references shouldn’t be solipsistically limited to LW/AF alone.
- The selectivity of AF doesn’t seem to me to work or help (of course, this take is shaped by my personal experience of posts that IMO are obviously relevant for AF and of no lower quality than a big portion of posts on AF that don’t get published there because I’m not a member of the in-group, and, perhaps, not completely onboard with the currently most popular paradigms for thinking about alignment). Seems that AF just reinforces the “echo chamber” effect that is already present on LW (relative to the rest of scientific publishing/thought).
AF as a research medium
I think the following setup would be interesting: all posts with tags like “AI”, “Alignment”, “Agency”, plus an additional tag/label that authors may add, like “Research”, are automatically visible on “AF” (though, it wouldn’t make much sense to call that medium “forum” anymore). On these posts, apart from regular comments, people could also post structured reviews (with many obligatory sections, similar to academic reviews). There is already similar machinery on LW: on questions, there could be “answers”, but there also could be regular comments.
Finally, the content on the “new AF” is ranked completely differently than on LW, along the lines as I described above: regular LW’s upvotes and downvotes don’t matter at all, however, the ratings given out in reviews do matter (along, perhaps, with the reputation of the reviewers), as well as backlinks/reference counts. This necessarily implies that the content on this “AF” becomes visible on the front page only slowly (busy researchers are unlikely to review work quicker than on about a week timescale, and reviews themselves take time), but also potentially stays there for longer, if the content generates a lot of backlinks.
Also, this “new AF” should provide formal academic labelling, similar to how https://distill.pub/ did.
If someone shares these ideas and has more thoughts on why LW/AF system doesn’t cater for AI safety progress well, I would like to collaborate on a shared post (for LW).
PS. We, the AI Safety community, need to “outcompete” capability research, including on this front, so it’s unacceptable that we toil behind including because our epistemic systems don’t support a more effective accumulation of knowledge and insight.
What links here?
- Roman Leventov's comment on AGI safety field building projects I’d like to see by Severin T. Seehrich (25 Jan 2023 7:31 UTC; 1 point)
- Roman Leventov's comment on Review AI Alignment posts to help figure out how to make a proper AI Alignment review by habryka (16 Jan 2023 4:36 UTC; 1 point)

Roman Leventov 10 Jul 2023 18:23 UTC
11 points
2
on: “Reframing Superintelligence” + LLMs + 4 years

I know of no persuasive argument for the superior value (or safety!) of powerful, unitary AI agents. Intellectual inertia, institutional inertia, convenient anthropomorphism (see below), and bragging rights are not good justifications for increasing existential risk.

You didn’t mention the biggest reason why the discussion of unitary agents is still very much relevant: due to their economic (and, later, military, political, and even romantic) attractiveness and power, unitary agents will be created anyway (as you yourself admit), and to counteract them, some people (including myself) think there should be aligned “guardian” unitary agents around who can spin their OODA loop as quickly as potentially misaligned/rogue agents. The OODA iteration of an open agency will take more time.

We can debate this, or whether the latency of an OODA cycle is that important in offense-defense balance in an AI conflict, but I don’t think these discussions are due to either “intellectual inertia, institutional inertia, convenient anthropomorphism, or bragging rights”.

Roman Leventov 1 Sep 2022 17:41 UTC
11 points
0
on: Discovering Agents
Should agency remain an informal concept?
The discussion of the different conceptualisations of agency in the paper left me confused. The authors pull a number of different strings, some of them are hardly related to each other:
- “Agent is a system that would adapt their policy if their actions influenced the world in a different way”
- Accidentality: “In contrast, the output of non-agentic systems might accidentally be optimal for producing a certain outcome, but these do not typically adapt. For example, a rock that is accidentally optimal for reducing water flow through a pipe would not adapt its size if the pipe was wider.”
- Intentional stance
- “Our definition is important for goal-directedness, as it distinguishes incidental influence that a decision might have on some variable, from more directed influence: only a system that counterfactually adapts can be said to be trying to influence the variable in a systematic way.” — here, it’s unclear what does it mean to adapt “counterfactually”.
- Learning behaviour, as discussed in section 1.3, and also in the section “Understanding a system with the process of its creation as an agent” below.
I’m not sure this is much of an improvement over Russell and Norvig’s classification of agents, which seems to capture most of the same threads, and also the fact that agency is more of a gradient than a binary yes/no property. This is also consistent with minimal physicalism, the scale-free understanding of cognition (and, hence, agency) by Fields et al.
“Agent” is such a loaded term that I feel it would be easier to read the paper if the authors didn’t attempt to “seize” the term “agent” but instead used, for example, the term “consequentialist”. The term “agent” carries too much semantic burden. Most readers probably already have an ingrained intuitive understanding of this word or have their own favourite theory of agency. So, readers should fight a bias in their head while reading the paper and nonetheless risk misunderstanding it.
Understanding a system with the process of its creation as an agent
[…] our definition depends on whether one considers the creation process of a system when looking for adaptation of the policy. Consider, for example, changing the mechanism for how a heater operates, so that it cools rather than heats a room. An existing thermostat will not adapt to this change, and is therefore not an agent by our account. However, if the designers were aware of the change to the heater, then they would likely have designed the thermostat differently. This adaptation means that the thermostat with its creation process is an agent under our definition. Similarly, most RL agents would only pursue a different policy if retrained in a different environment. Thus we consider the system of the RL training process to be an agent, but the learnt RL policy itself, in general, won’t be an agent according to our definition (as after training, it won’t adapt to a change in the way its actions influence the world, as the policy is frozen).
I think most readers have a strong intuition that agents are physical systems. A system with its creation process is actually a physical object in the BORO ontology (where a process is a 4D spacetime object, and any collection of objects can be seen as an object, too) and probably some other approaches to ontology, but I suspect this might be highly counterintuitive to most readers, so perhaps warrants some discussion.
Also, I think that a “learnt RL policy” can mean either an abstract description of behaviour or a physical encoding of this description on some information storage device. Neither of these can intuitively be an “agent”, so I would stick with simply “RL agent” (meaning “a physical object that acts according to a learned policy”) in this sentence again, to avoid confusion.
I think the authors of the paper had a goal of bridging the gap between the real world roamed by suspected agents and the mathematical formalism of causal graphs. From the conclusion:
We proposed the first formal causal definition of agents. Grounded in causal discovery, our key contribution is to formalise the idea that agents are systems that adapt their behaviour in response to changes in how their actions influence the world. Indeed, Algorithms 1 and 2 describe a precise experimental process that can, in principle and under some assumptions, be done to assess whether something is an agent. Our process is largely consistent with previous, informal characterisations of agents (e.g. Dennett, 1987; Flint, 2020; Garrabrant, 2021; Wiener, 1961), but making it formal enables agents and their incentives to be identified empirically [emphasis mine — Roman Leventov] or from the system architecture.
There are other pieces of language that hint that the authors see their contributions as epistemological rather than mathematical (all emphasis is mine):
We derive the first causal discovery algorithm for discovering agents from empirical data.
[Paper’s contributions] ground game graph representations of agents in causal experiments. These experiments can be applied to real systems, or used in thought-experiments to determine the correct game graph and resolve confusions (see Section 4).
However, I don’t think the authors created a valid epistemological method for discovering agents from empirical data. In the following sections, I lay out the arguments supporting this claim.
For the mechanism variables that the modeller can’t intervene and gain information about which only via observing their corresponding object-level variables, it doesn’t make sense to draw a causal link from the mechanism to the object-level variable
If the mechanism variable can’t be intervened and is only observed through its object-level variable, then this mechanism is purely a product of the modeller’s imagination and can be anything.
Such mechanism variables, however, still have a place on the causal graphs: they are represented physically by the modeller (e. g. stored in the computer memory, or in the modeller’s brain) and these representations do physically affect other modeller’s representations, including of other variables, and their decision policies. For example, in graph 1c, the mechanisms of all three variables should be seen as stored in the mouse’s brain:
The causal links $X$ → $~ X$ and $U$ → $~ U$ point from the object-level to the mechanism variables because the mouse learns the mechanisms by observing the object-level.
An example of a mechanism variable which we can’t intervene in, but may observe independently from the object-level is humans’ explicit report of their preferences, separate from what is revealed in their behaviour (e. g. in the content recommendation setting, which I explore in more detail below).
In section 5.2 “Modelling advice”, the authors express a very similar idea:
It should be fully clear both how to measure and how to intervene on a variable. Otherwise its causal relationship to other variables will be ill-defined. In our case, this requirement extends also to the mechanism of each variable.
This idea is similar because the modeller often can’t intervene in the abstract “mechanism” attached to an object-level variable which causes it, but the modeller always can intervene on its own belief about this mechanism. And if mechanism variables represent the modeller’s beliefs rather than “real mechanisms” (cf. Dennett’s real patterns), then it’s obvious that the direction of the causal links should be from the object-level variables to the corresponding beliefs about their mechanism, rather than vice versa.
So, I agree with this last quote, but it seems to contradict a major chunk of the other paper’s content.
There is no clear boundary between the object-level and the mechanism variables
In explaining mechanism and object-level variables, the authors seemingly jump between reasoning within a mathematical formalism of SCMs (or a simulated environment, where all mechanisms are explicitly specified and are controllable by the simulator; this environment doesn’t differ much from a mathematical formalism) and within a real world.
The mathematical/simulation frame:
The intended interpretation is that the mechanism variables parameterise how the object-level variables depend on their object-level parents.
The formalism doesn’t tell us anything about how to distinguish between object-level and mechanism variables: object-level variables are just the variables that are included in the object-level graph, but that itself is arbitrary. For example, in section 4.4, the authors note that Langlois and Everitt (2021) included the decision rule in the game graph, but it should have been a mechanism variable. However, in the content recommendation setting (section 4.2), the “human model” ( $M$ ) is clearly a mechanism variable for the original human preferences ( $H_{1}$ ), but is nevertheless included in the object-level graph because other object-level variables depend on it.
There are also two phrases in section 3.3 that suppose a mathematical frame. First, “the set of interventional distributions generated by a mechanised SCM” (in Lemma 2) says that interventional distributions are created by the model, rather than by the physical system (the algorithm executor) performing the interventions in the modelled world. Second, the sentence “Applied to the mouse example of Fig. 1, Algorithm 1 would take interventional data from the system and draw the edge-labelled mechanised causal graph in Fig. 1c.” doesn’t emphasise the fact that an algorithm is always performed by an executor (a physical system) and it’s important who that executor is, including for the details of the algorithm (cf. Deutsch’s and Marletto’s constructor theory of information). Algorithms don’t execute themselves.
The real-world frame:
An intervention on an object-level variable $V$ changes the value of $V$ without changing its mechanism, $~ V$ . This can be interpreted as the intervention occurring after all mechanisms variables have been determined/sampled [emphasis mine—Roman Leventov].
In section 3.5, it is said that mechanised SCM is a “physical representation of the system”.
The distinction between mechanism and object-level variables can be made more concrete by considering repeated interactions. In Section 1.1, assume that the mouse is repeatedly placed in the gridworld, and can adapt its decision rule based (only) on previous episodes. A mechanism intervention would correspond to a (soft) intervention that takes place across all time steps, so that the mouse is able to adapt to it. Similarly, the outcome of a mechanism can then be measured by observing a large number of outcomes of the game, after any learning dynamics has converged. Finally, object-level interventions correspond to intervening on variables in one particular (postconvergence) episode.

[In this quote, the authors used both the terms “interaction” and “episode” to point to the same concept, but I stick to the former because in the Reinforcement Learning literature, the term “episode” has a meaning slightly different from the meaning that is implied by the authors in this quote. — Roman Leventov]
I read from this quote that the authors take an inductive bias that mechanism variables update slowly, so we take a simplifying assumption that they don’t update at all within a single interaction. However, I think this assumption is dangerously ignorant for reasoning about agents capable of reflection and explicit policy planning. Such agents (including humans) can switch their policy (i. e., the mechanism) of making a certain kind of decision in response to a single event. And other agents in the game, aware of this possibility, can and should take it into account, effectively modelling the game as having direct causal paths from some object-level variables into the mechanism variables, which in turn inform their own decisions.
Optimising a model of a human
If, as is common in practice, the model was obtained by predicting clicks based on past user data, then changing how a human reacts to recommended content ( $_{2}$ ), would lead to a change in the way that predicted clicks depend on the model of the original user ( $~ U$ ). This means that there should be an edge, as we have drawn in Fig. 3b. Everitt et al. (2021a) likely have in mind a different interpretation, where the predicted clicks are derived from 𝑀 according to a different procedure, described in more detail by Farquhar et al. (2022). But the intended interpretation is ambiguous when looking only at Fig. 3a – the mechanised graph is needed to reveal the difference.
Why does all this matter? Everitt et al. (2021a) use Fig. 3a to claim that there is no incentive for the policy to instrumentally control how the human’s opinion is updated and they deem the proposed system safe as a result. However, under one plausible interpretation, our causal discovery approach yields the mechanised causal graph representation of Fig. 3b, which contains a directed path $H_{2}$ → $D$ . This can be interpreted as the recommendation system is influencing the human in a goal-directed way, as it is adapting its behaviour to changes in how the human is influenced by its recommendation (cf. discussion in Section 1.2).
This example casts doubt on the reliability of graphical incentive analysis (Everitt et al., 2021a) and its applications (Ashurst et al., 2022; Cohen et al., 2021; Evans and Kasirzadeh, 2021; Everitt et al., 2021b; Farquhar et al., 2022; Langlois and Everitt, 2021). If different interpretations of the same graph yields different conclusions, then graph-based inference does not seem possible.
I think the real problem with the graph in Figure 3a is that it has already stepped onto the “mechanism land”, but didn’t depict any mechanism variables apart from that of $H_{1}$ , which is $M$ . The discussion in the quote above assumes that the graph in Figure 3a models repeated recommendations (and, well, otherwise there weren’t both $H_{1}$ and $H_{2}$ on this graph simultaneously). Therefore, as I noted above, causal links between object-level chance variables and the corresponding mechanism variables should point from the object-level to the mechanism. And, indeed, there is a link $H_{1}$ → $M$ . Thus, $M$ on the graph is identical to $_{1}$ in the notation of mechanised SCMs, and $~ M$ should be interpreted as the mechanism of deriving the mechanism: that is, the statistical algorithm used to derive $M$ ( $_{1}$ ) from $_{2}$ and $U$ .
I think the mechanised SCM of content recommendation should look closer to this, taking the “the model was obtained by predicting clicks based on past user data” interpretation:
On this graph, red causal links are those that are different from Figure 3b, which doesn’t imply any special causal semantics.
I also assumed that $~ U$ and $M$ are learned jointly, hence a direct bidirectional link between these models. However, $~ U$ might also be fixed to some static algorithm, such as a static click rate discounted according to a certain static formula which takes the strength of the preference of the user to the recommended content as the input. The preference is taken from the user model $M$ (which is still learned). In this interpretation, all the incoming causal links into $~ U$ should be erased, and “changing how a human reacts to recommended content ( $_{2}$ ), would lead to a change in the way that predicted clicks depend on the model of the original user ( $~ U$ )” is not inevitable (see the beginning of the quote from the paper above).
Actor–Critic
This can help avoid modelling mistakes and incorrect inference of agent incentives. In particular, Christiano (private communication, 2019) has questioned the reliability of incentive analysis from CIDs, because of an apparently reasonable way of modelling the actor-critic system where the actor is not modelled as an agent, shown in Fig. 4c. Doing incentive analysis on this single-agent diagram would lead to the assertion that the system is not trying to influence the state 𝑆 or the reward 𝑅, because they don’t lie on the directed path 𝑄 → 𝑊 (i.e. neither 𝑆 nor 𝑅 has an instrumental control incentive; Everitt et al., 2021a). This would be incorrect, as the system is trying to influence both these variables (in an intuitive and practical sense).
I tried to model the system described on page 332 of Sutton and Barto, “One-step Actor–Critic (episodic)” algorithm, preserving the structure of the above graph, but using notation from Sutton and Barto. To me, it seems that the best model of the system is a single-agent, but where $A$ is still a decision rather than a chance variable, and $^v (\cdot, w)$ (the equivalent of $Q$ on the graph above) should best be seen as a part of the mechanism for decision $A$ which includes both $^v (\cdot, w)$ and $π (\cdot, θ)$ rather than a single mechanism variable:
The advice that the variables should be logically independent is phrased stronger than other modelling advice, which is not well justified
Variables should be logically independent: one variable taking on a value should never be mutually exclusive with another variable taking on a particular value.
In this modelling advice, the word “never” communicates that this is “stronger” advice than others provided in section 5.2. However, under bounded rationality of physical causal reasoners, some models can have logical inconsistencies, yet still enable better inference (and more efficient action policies) than alternative models without the variables that are co-dependent with some other variables and thus introduce logical inconsistencies.
Two miscellaneous clarification notes for section 3.2
“whether a variable’s distribution adaptively responds for a downstream reason, (i.e. is a decision node), rather than for no downstream consequence (e.g. its distribution is set mechanistically by some natural process)” — a “natural process” implies an object-level causal link from W to V. However, from the text below, it seems to be implied that W is also downstream of V on the object-level. This means there is a causal cycle on the object-level, but we don’t consider such models. So, while this example might be formally correct, I think this example is more confusing than helpful.
“to determine whether a variable, 𝑉, adapts for a downstream reason, we can test whether 𝑉’s mechanism still responds even when the children of 𝑉 stop responding to 𝑉 (i.e. 𝑉 has no downstream effect).” — This sentence is confusing. I think it should be “… whether V’s mechanism stops responding to changes in the mechanisms of its downstream variables if the children of 𝑉 stop responding to 𝑉” (as it’s formalised in Definition 3).
What links here?
- Roman Leventov's comment on Agency As a Natural Abstraction by Thane Ruthenis (25 Oct 2022 13:05 UTC; 1 point)

Roman Leventov 27 Dec 2023 20:11 UTC
10 points
1
in reply to: Vladimir_Nesov’s comment on: AGI will be made of heterogeneous components, Transformer and Selective SSM blocks will be among them
The fact that hybridisation works better than pure architectures (architectures consisting of a single core type of block, we shall say), is exactly the point that Nathan Labenz makes in the podcast and I repeat in the beginning of the post.
(Ah, I actually forgot to repeat this point, apart from noting that Doyle predicted this in his architecture theory.)

Roman Leventov 28 Nov 2023 8:43 UTC
9 points
0
on: Shallow review of live agendas in alignment & safety
Under “Understand cooperation”, you should add Metagov (many relevant projects under this umbrella, please visit the website, in particular, DAO Science), “ecosystems of intelligence” agenda (itself pursued by Verses, Active Inference Institute, Gaia Consortium, Digital Gaia, and Bioform Labs). This is more often more practical than theoretical work though, so the category names (“Theory” > “Understanding cooperation”) wouldn’t be totally reasonable for it, but this is also true for a lot of entires already on the post.

In general, the science of cooperation, game theory, digital assets and money, and governance is mature, with a lot of academics working in it in different countries. Picking up just a few projects “familiar to the LessWrong crowd” is just reinforcing the bubble.
The “LessWrong bias” is also felt in the decision to omit all the efforts that contribute to the creation of the stable equilibrium for the civilisation on which an ASI can land. Here’s my stab at what is going into that from one month ago; and this is Vitalik Buterin’s stab from yesterday.
Also, speaking about pure “technical alignment/AI safety” agendas that “nobody on LW knows and talks about”, check out the 16 projects already funded by the “Safe Learning-Enabled Systems” NSF grant program. All these projects have received grants from $250k to $800k and are staffed with teams of academics in American universities.

Roman Leventov 24 Mar 2023 19:40 UTC
9 points
5
on: The Overton Window widens: Examples of AI risk in the media
Another opinion on NYT: https://www.nytimes.com/2023/03/24/opinion/yuval-harari-ai-chatgpt.html (Yuval Noah Harari, Tristan Harris, and Aza Ruskin. Non-paywalled version.)

Roman Leventov 3 Jan 2024 17:34 UTC
8 points
0
on: AI Alignment Metastrategy
I think this metastrategy classification is overly simplified to the degree that I’m not sure it is net helpful. I don’t see how Hendrycks’ “Leviathan safety”, Drexler’s Open Agency Model, Davidad’s OAA, Bengio’s “AI pure scientist” and governance proposals (see https://slideslive.com/39014230/towards-quantitative-safety-guarantees-and-alignment), Kaufmann and Leventov’s Gaia Network, AI Objectives Institute’s agenda (and related Collective Intelligence Project’s), Conjecture’s CoEms, OpenAI’s “AI alignment scientist” agenda, and Critch’s h/acc (and related janus et al.’s Cyborgism) straightforwardly lend on this classification, at least not without losing some important nuance.

Furthermore, there is also the missing dimension of [technical strategy, organisational strategy, governance and political strategy] that could perhaps recombine to some degree.

Finally, in the spirit of “passing ideological Turing test” and “describing, not persuading” norms, it would be nice I think to include the criticism of the “conservative strategy” to the same level of fidelity that other metastrategies are criticised here, even if you or others discussed that in some other posts.

Roman Leventov

The term “Simulator” has a potentially dangerous connotation of precision and reliability

AF as a research medium

Should agency remain an informal concept?

Understanding a system with the process of its creation as an agent

For the mechanism variables that the modeller can’t intervene and gain information about which only via observing their corresponding object-level variables, it doesn’t make sense to draw a causal link from the mechanism to the object-level variable

There is no clear boundary between the object-level and the mechanism variables

Optimising a model of a human

Actor–Critic

The advice that the variables should be logically independent is phrased stronger than other modelling advice, which is not well justified

Two miscellaneous clarification notes for section 3.2