Roman Leventov

Karma: 817

An independent researcher of ethics, AI safety, and AI impacts. Twitter: https://twitter.com/leventov. E-mail: leventov.ru@gmail.com (the preferred mode of communication).

You can help to boost my sense of accountability and give me a feeling that my work is valued by becoming a paid subscriber of my Substack (though I don’t post anything paywalled; in fact, on this blog, I just syndicate my LessWrong writing).

A Telegram group where we discuss AI x-risk/safety, theories of intelligence, agency, consciousness, and ethics, in Russian: https://t.me/agi_risk_and_ethics.

Roman Leventov 1 Jun 2023 18:42 UTC
1 point
0
on: EIS VI: Critiques of Mechanistic Interpretability Work in AI Safety
All the critiques focus on MI not being effective enough at its ultimate purpose—namely, interpretability, and secondarily, finding adversaries (I guess), and maybe something else?
Did you seriously think through whether interpretability, and/or finding adversaries, or some specific aspects or kinds of either interoperability or finding adversaries could be net negative for safety overall? Such as what was contemplated in “AGI-Automated Interpretability is Suicide”, “AI interpretability could be harmful?”, and “Why and When Interpretability Work is Dangerous”. However, I think that none of the authors of these three posts is an expert in interpretability or adversaries, so it would be really interesting to see your thinking on this topic.

Roman Leventov 1 Jun 2023 15:31 UTC
1 point
0
on: Types and Degrees of Alignment
Is this ‘alignment’ a natural thing you can get easily or even by default, that is essentially a normal engineering problem, or is it a highly unnatural outcome where security mindset and bulletproof approaches as yet unfound even in principle are required, with any flaws are exploited, amplified and fatal, and many lethal problems all of which one must avoid?
To answer these questions specifically, it’s really important not just to consider AI—human alignment “in the abstract”, but embedded in the current civilisation, with its infrastructure and incentives structures. As I wrote here:
[...] we should address this strategic concern by rewiring the economic and action landscapes (which also interacts with the “game-theoretic, mechanism-design” alignment paradigm mentioned above). The current (internet) infrastructure and economic systems are not prepared for the emergence of powerful adversarial agents at all:
- There are no systems of trust and authenticity verification at the root of internet communication (see https://trustoverip.org/)
- The storage of information is centralised enormously (primarily in the data centres of BigCos such as Google, Meta, etc.)
- Money has no trace, so one may earn money in arbitrary malicious or unlawful ways (i.e., gain instrumental power) and then use it to acquire resources from respectable places, e.g., paying for ML training compute at AWS or Azure and purchasing data from data providers. Formal regulations such as compute governance and data governance and human-based KYC procedures can only go so far and could probably be social-engineered by a superhuman imposter or persuader AI.
In essence, we want to design civilisational cooperation systems such that being aligned is a competitive advantage. Cf. “The Gaia Attractor” by Rafael Kaufmann.
This is a very ambitious program to rewire the entirety of the internet, other infrastructure, and the economy, but I believe this must be done anyway, just expecting a “miracle” HRAD invention to be sufficient without fixing the infrastructure and system design layers doesn’t sound like a good strategy. By the way, such infrastructure and economy rewiring is the real “pivotal act”.
If we imagined that the world had a “right” kind of infrastructure and social structure (really decentralised, trust-first), probably alignment would be much more of an “ordinary engineering” problem. With the current economic and infrastructural vulnerabilities mentioned, however, the alignment becomes a much higher-stakes problem, requiring more of “bulletproof” solutions “on the first try”, I think.

Roman Leventov 1 Jun 2023 15:21 UTC
1 point
0
in reply to: Roman Leventov’s comment on: Types and Degrees of Alignment
And Zvi points out these contradictions himself:
It is not clear to what extent robust alignment is a coherent concept especially in a competitive world or even how it interacts with maximization, as it contains many potential contradictions and requirements.

Roman Leventov 1 Jun 2023 15:07 UTC
1 point
0
on: Types and Degrees of Alignment
Strict alignment. The damn thing will actually follow some set of instructions to the letter subject to its optimization constraints, hopefully you like the consequences of that. It is a potentially important crux if you disagree with the claim that for almost all specified instruction sets you won’t like the consequences, and there is no known good one yet, due to various alignment difficulties.
The name is uninformative and possibly misleading. If the set of instructions is in a natural or a formal language, you push the alignment difficulty into the semantics and semiotics, which are not “strict”, and the alignment ends up not “strict” either.
In the planning-as-inference frame, I guess you probably mean something like an external evaluation (perhaps with some “good old-fashioned algorithm”, a-la “type checker”, rather than another AI, although it’s really questionable whether such “type checker” could be built) of the inferred plans in their entirety. But again, even internal AI’s representations are symbols, not “ground truth”, so they are subject to the same difficulty of semantic and semiotic interpretation as natural language.
It is not clear to what extent strict alignment or strawberry alignment gives us affordance to reach good outcomes, how universal and deadly the various sources of lethality involved would be, or how difficult it would be to locate such affordances, especially on the first try.
“Strict” alignment is an engineering technique or approach, that shouldn’t be judged in isolation but rather as a part of a cognitive architecture as a whole, as I explained in this comment.
“Strawberry” alignment is an external evaluation criterion or characteristic. However, I would go even further and say that “strawberry” is just a thought experiment which is not meant to be an actual eval that we will actually try, and the purpose of this thought experiment is just to show that the process of reasoning (and the result of reasoning, i.e., the plan), and the resulting behaviour are the actual objects of ethical evaluation and alignment rather than simply “goals”. Goals become “good” or “bad” only in the context of larger plans and behaviour. This thought could be expressed in different ways, e.g., directly, as I just did (as well as in this comment). The latest OpenAI paper, “Let’s Verify Step By Step” highlights this idea, too.

Roman Leventov 1 Jun 2023 14:42 UTC
1 point
0
on: Types and Degrees of Alignment
It is not clear that human-level or friendly alignment would do us much good for long either, given the nature and history of humans, and the competitive dynamics involved, and the various reasons to expect change. If AGIs are much smarter and more capable and efficient than us, is there reason to think this level of alignment might be sufficient for long?
“Human-level” is just more commonly called “value alignment” (or “alignment with human values” if you want). But I agree with the conclusion: “friendly” is an attempt at “moral fact alignment” (“humanity is valuable to preserve”), which is probably futile without considering and aligning on the underlying theory ethics, i.e., without the methodological and scientific alignment, as I described in a different comment. Value alignment, if taken literally, i.e., as attempting to impart AI with humans’ heuristics about value, is also a species of “moral fact alignment”, just somewhat more concrete than just “humanity is valuable to preserve” (although the latter is also one of the human values).

Roman Leventov 1 Jun 2023 14:35 UTC
3 points
2
on: Types and Degrees of Alignment
From the perspective of ontology and memetic engineering, the whole “ontology” or classification of alignments that you give, “fragile, friendly, …” is bad because it’s not based on some theory but rather on the cacophony of commonsensical ideas. These “alignments” don’t even belong to the same type: “Fragile” is an engineering approach (but there are also many other engineering approaches which you haven’t mentioned!), 2-3 and 5-6 are black-box descriptions of some alignment characteristics (at least these seem to belong to the same type), and “Strict” looks like a description of a mathematical mechanism. Also, the names are bad.
Fragile alignment
The name is really bad you use a property of this engineering approach (its fragility) as its name. But there are infinitely many other engineering approaches that are very fragile. For example, consider post-filtering LLM’s outputs for any occurrences of the words like “bomb”, “kill”, “poison”, n-word, etc., and not passing these rollouts through, but passing all the rest. Is this an alignment technique? Yes, if we consider the filter as part of the cognitive architecture, as we should. Is it fragile? Yes.
But when multiple such approaches are combined, this may actually lead to a cognitive architecture that is robustly aligned. This is the promise of LMCAs and the natural language alignment.
It is not clear fragile alignment is even meaningfully helpful – that it does much, survives for long, or causes actions compatible with our survival, once the AGI is smarter than we are, even if we get its details mostly right and faced relatively good conditions. There are overlapping and overdetermined reasons (the strength and validity of which are of course disputed) to expect any good properties to break down exactly when it is important they not break down.
Laissez-faire (any prompt goes), “bare”, non-post-filtered LLM rollouts as a cognitive architecture is obviously doomed. OpenAI has stopped deploying and giving access to base models (GPT-4 is available only in SFT’ed and RLHF’ed form), and I expect that in the next iteration (GPT-5), they will (my wishful thinking) stop even that—they will give access only to an LMCA from the beginning. Even “rollout + post-filtering” is already a primitive form of such LMCA.
In turn, LMCAs in general shouldn’t necessarily inherit the cardinal sin of LLMs (that are “exponentially diverging diffusion process”, in the words of LeCun). And SFT/RLHF (things that you have called “fragile alignment”) could be a part of a robust architecture, as I already noted above.

Roman Leventov 1 Jun 2023 13:58 UTC
1 point
0
on: Types and Degrees of Alignment
Have this alignment and the surrounding dynamics cause humans to choose to remain in control over time, or somehow be unable to choose differently.
This is self-contradictory: if the surrounding dynamics strongly preclude humans from “choosing otherwise”, humans are no longer “in control”. Also, under certain definitions of “choosing differently”, humans may be precluded from moving into different biological and computational substrates, which in itself might be a cosmic tragedy because it may forever preclude humans from realising vast amounts of potential.

Roman Leventov 1 Jun 2023 13:57 UTC
1 point
−1
on: Types and Degrees of Alignment
It is impossible in theory to have all these different kinds of alignment simultaneously. You cannot simultaneously (without any claim of completeness):
1. Do what I say
2. Also do what I mean
3. Also do what I should have said and meant
4. Also do what is best for me
5. Also do what broader society or humanity says
6. Also do what broader society or humanity means or should have said
7. Also do what broader society or humanity should have said given their values
8. Also do what is best for everyone
9. Do some ideal friendly combination of all of it that a broadly good guy would do, in a way that is respectful of and preserves what is valuable on all levels
10. Strictly follow some other set of rules that were set up long ago, no matter the cost
I think we should already forget about items 1 and 2 (as well as 10, but that goes without saying). In the context of communicating and aligning with superhumanly smart AI, it is really hubristic and stupid to think that although we will be much stupider than AI, it will be able to write a superhumanly good theory of ethics (such that humans didn’t come up with in thousands of years) in a second, and design an AI (or change oneself) to follow this theory, humans’ thoughts about value will still be somehow “covered with gold” and worth adhering to, for that AI.
Given this, I disagree that “we will need to: 1) Be able to get an AGI to do something a human selects at all, rather than something not selected. Be able to retain some form of control over what it does in the future, or set it on a chosen course. At all. …”. I’m not sure that this is possible (which implies that the orthogonality thesis holds to a sufficiently strong degree, which I doubt), but if this is possible, I don’t think this would be desirable.
Items 3-9 raise an important and valid concern, that of the (infinity) multiplicity of (collective) identity and alignment subjectivity.
We must pick at most one of those, or another variation on them, or something else, as our primary target. A machine cannot serve two masters any more than a man can, and the ability to put the machine under arbitrary stress, and its additional capabilities and intelligence, makes this that much more clear.
I think this is a wrong reaction to the above concern. Humans can serve multiple masters (themselves, their family, their community, their nation/society, the whole of humanity, and Gaia), so why AIs couldn’t?
In the phrase “It is impossible in theory to have all these different kinds of alignment simultaneously”, it’s unclear what you mean by “having alignment”. If you meant something like formal, “total” alignment, then sure, it’s impossible to have perfect alignment in the physical reality outside us (i.e., outside simple simulations and mathematical abstractions). But if we strive for continuously increasing alignment along various dimensions and various levels of intelligent entities (alignment subjects), it should definitely be possible in general (although some “bad” situations may call for a choice where the alignment between certain entities or along a certain dimension is worsened in favour of the alignment between some other types of entities or along other dimensions).
I think that to increase our chances to realise a good future, we must find a principled way of addressing the issue of the multiplicity of identity and alignment subjectivity, which is the essence of a scale-free theory of ethics.

Roman Leventov 1 Jun 2023 13:24 UTC
1 point
0
on: Types and Degrees of Alignment
the system has unintended (harmful or dangerous) goals or behaviors.
Note that judgements about the harmfulness and dangerousness of some goals or behaviours are themselves theory-laden. This is why Goal alignment without alignment on epistemology, ethics, and science is futile. From the perspective of any theory of cognition/intelligence that includes a generative model (which is not only Active Inference, but also LeCun’s H-JEPA, LMCAs such as the “exemplary actor”, and more theories of cognition and/or AI architectures) for performing planning-as-inference, I think a straightforward and useful ladder of aligned could be introduced: methodological, scientific, and fact alignment:
Thus, the crux of alignment is aligning the generative models of humans and AIs. Generative models could be “decomposed”, vaguely (there is a lot of intersection between these categories), into
- Methodology: the mechanics of the models themselves (i.e., epistemology, rationality, normative logic, ethical deliberation),
- Science: mechanics, or “update rules/laws” of the world (such as the laws of physics or the heuristical learnings about society, economy, markets, psychology, etc.), and
- Fact: the state of the world (facts, or inferences about the current state of the world: CO2 level in the atmosphere, the suicide rate in each country, distance from Earth to the Sun, etc.)
These, we can conceptualise, give rise to “methodological alignment”, “scientific alignment”, and “fact alignment” respectively. Evidently, methodological alignment is most important: it in principle allows for alignment on science, and methodology plus science helps to align on facts.
Under this framework, goals are a specific type of facts, laden by a specific theory of mind of another agent (natural or AI). A theory of mind here should be a specialised version of a general theory of cognition which itself, as noted above, includes a generative model and planning-as-inference, under which goals become future world states or some features of future world states, predicted/planned (prediction and planning is the same thing, under planning-as-inference) by the other mind (or oneself, if the agent reflects about its own goals).^[1]
Thus, goal alignment is easy (practically, automatic) when two agents are aligned on methodology and science (albeit goal alignment even between methodologically and scientifically aligned agents usually still requires communication and coordination, unless we enter the territory of logical handshakes...), but is also futile when there is no common methodological and scientific ground.
1. ^
  Incidentally, this means that RL is not a very useful framework for discussing goals, because goals couldn’t be conceptualised under RL easily, which causes a lot of trouble to people in the AI safety community who tend to think that there should be a single “right” theory or framework of cognition and intelligence. There should not: For alignment, we should simultaneously use multiple theories of cognition and value. And RL, although probably couldn’t be deployed very usefully to discuss goal alignment specifically, could still be used to discussed some aspects of value alignment between the minds.

Roman Leventov 1 Jun 2023 12:31 UTC
1 point
0
in reply to: Mitchell_Porter’s comment on: Mitchell_Porter’s Shortform
Re: the virtuous cycle, I was excited recently to find Toby Smithe’s work, a compositional account of Bayesian Brain, which strives to establish formal connections between ontology, epistemology, phenomenology, semantics, evolutionary game theory, and more.
Next week, Smithe will give a seminar about this work.

Roman Leventov 1 Jun 2023 12:02 UTC
1 point
0
on: Roman Leventov’s Shortform
I asked GPT-4 to write a list of desiderata for a naturalistic (i.e., scientific) theory of ethics: https://chat.openai.com/share/1025a325-30e0-457c-a1ed-9d6b3f23eb5e. It made some mistakes but in other regards surprised me with the quality of its philosophy of science and meta-ethics.
The mistake that jumped out for me was “6. Robustness and Flexibility”:
The ethical theory should be robust and flexible, meaning it should be able to accommodate new information and adapt to different contexts and conditions. As our scientific knowledge evolves, the theory should be able to incorporate new insights without losing its coherence or practical applicability.
According to a Popperian criterion for a good scientific theory, falsifiability, GPT-4 should have made the opposite point: the scientific theory of ethics should fit with other theories (of rationality, evolution, consciousness, etc.) in such a way that an update in one of those theories requires a revision in the theory of ethics as well. So, all theories should be mutually “brittle”, which reflects falsifiability.
When I asked GPT-4 to criticise the desiderata for a scientific theory of ethics with Popperian criteria in mind, it found this problem with item 6, as well as several other (albeit smaller) problems that I didn’t think about, and probably wouldn’t come up with even if tried quite hard!
Then, when I asked GPT-4 to rewrite the desiderata considering the criticisms, the flip side of LLM’s self-consistency (Karpathy mentions it in the “State of GPT” talk: youtube.com/watch?v=bZQun8Y4L2A) has come up: the LLM was reluctant to turn 180 degrees upon recognising its own mistake and instead tried to “amend” all desiderata, preserving their original chich actually made them only worse). The supposed remedy for this problem is the “Tree of Thoughts” LMCA: arxiv.org/abs/2305.10601.

Roman Leventov 29 May 2023 19:28 UTC
2 points
0
in reply to: Seth Herd’s comment on: Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”
LMCA that uses a body of knowledge in the form of textbooks, scientific theories and models may be updated very frequently and cheaply: essentially, every update of the scientific textbook is an update of LMCA. No need to re-train anything.
GFlowNets have a disadvantage because they are trained for a very particular version of the exemplary actor, drawing upon a particular version of the body of knowledge. And this training will be extremely costly (billions or tens or even hundreds of billions of USD?) and high-latency (months?). By the time a hypothetical training of GFlowNet is complete, the textbooks and models may be already outdated.
This consideration really challenges the economic and capability expediency of GFlowNet actors (as described in the post) vs. LMCA. The flexibility, deployability, configurability, and iterability of LMCA may prove to be a too strong factor.

Roman Leventov 29 May 2023 19:07 UTC
1 point
0
in reply to: Seth Herd’s comment on: Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”
I thought your response would be that the H-JEPA network might be substantially faster, and so have a lower alignment tax than the exemplary LMCA.
I discuss this in section 4.5. My intuition is that LMCA with latency in tens of minutes is basically as “powerful” (on the civilisational scale) as an agent with latency of one second, there is no OODA-style edge in being swifter than tens of minutes. So, I think that Eric Schmidt’s idea of “millisecond-long war” (or, a war where action unfolds at millisecond-scale cadence) just doesn’t make sense.
However, these are just my intuitions. They may be wrong, and flash attacks might be possible. In this case, GFlowNets could be useful because they could work much faster than LMCA, indeed^[1].
1. ^
  Of course, if training such GFlowNets will even become tractable, which is not guaranteed. I discuss the potential of this training to be orders of magnitude more expensive than training of even GPT-5/6 level LLMs in section 3.4.

Roman Leventov 29 May 2023 18:40 UTC
3 points
0
in reply to: Seth Herd’s comment on: Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”
My question is: if we already have an aligned LMCA, why would we use it to train a less interpretable H-JEPA AGI?
First, it is not less interpretable. Here, Bengio and Hu argue that GFlowNets are more interpretable than auto-regressive LLMs; but in the setup where the energy function is not explicitly given (as in some other GFlowNet training setups, e.g., for drug discovery) but rather learned from examples (as I proposed in the post), GFlowNet don’t have any interpretability advantage over the AI that generate the examples, which is the aligned LMCA in this case. Hence, we should consider aligned LMCA and H-JEPA agent with GFlowNet actors to have “the same level” of decent interpretability.
Second, answering to the question: well, actually it appears to me that there is not much reason (although it seemed to me more like there would be some significant advantages to it when I was starting to write this article). I’ve summarised this result in these two paragraphs of the “Conclusion” section:
The only aspect in which the x-risk profile of H-JEPA agent with GFlowNet actors seems to be qualitatively different from that of the exemplary actor is the risk of direct access to the underlying Transformers, which is catastrophic in the case of the exemplary actor (section 2.3.1) and perhaps could be addressed completely in GFlowNet actors if we accept that they will deliberately dumb themselves in strategic x-risk analysis and planning (section 4.3). However, even if we accept this tradeoff, this might not reduce the overall x-risk of the civilisation because GFlowNet actors are not “self-sufficient” for training (section 4.4) and therefore the powerful LLM that underlies the exemplary actor that is used to train GFlowNet actors should still be kept around, and the risk of direct access to this LLM is still present.
Thus, H-JEPA agent with GFlowNet actors could become really interesting perhaps only if the “LLM optimism” view will prove to be correct and thus LMCAs could generally work and be satisfactorily aligned, but also sensory grounding proves to be a really important missing piece of the puzzle. (Though, this combination of conditionals looks to me rather unlikely.) The proposed variant of H-JEPA combines “the best of two worlds”: grounding from H-JEPA and aligned reasoning from the LMCA.

Roman Leventov 29 May 2023 11:56 UTC
1 point
0
in reply to: avturchin’s comment on: Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”
One line: H-JEPA probably won’t save us unless we already have an aligned LLM-based cognitive architecture.

Also: see section “Conclusion” in the post

Roman Leventov 29 May 2023 11:20 UTC
1 point
0
in reply to: Roman Leventov’s comment on: LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem
Update: I wrote a big article “Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor” in which I develop the thinking behind the comment above (but also update it significantly).

Roman Leventov 29 May 2023 11:18 UTC
1 point
0
in reply to: Steven Byrnes’s comment on: H-JEPA might be technically alignable in a modified form
Replied here: “Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”.

An LLM-based “exemplary actor”

Roman Leventov29 May 2023 11:12 UTC

15 points

0 comments12 min readLW link

Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”

Roman Leventov29 May 2023 11:08 UTC

11 points

10 comments29 min readLW link

Roman Leventov 19 May 2023 8:12 UTC
2 points
0
on: Some Summaries of Agent Foundations Work
Existing models of agency from fields like reinforcement learning and game theory don’t seem up to the job, so trying to develop better ones might pay off.
One account of why our usual models of agency aren’t up to the job is the Embedded Agency sequence—the usual models assume agents are unchanging, indivisible entities which interact with their environments through predefined channels, but real-world agents are a part of their environment. The sequence lists identifies four rough categories of problems that arise when we switch to trying to model embedded agents, explained in terms of Marcus Hutter’s model of the theoretically perfect reinforcement learning agent, AIXI.
Reinforcement learning and game theory are mathematical formalisms/frameworks (and some associated algorithms, such as many specific RL learning algorithms) But they are not science in themselves. “Embedded Agency” basically says “Let’s do cognitive science rather than develop mathematical formalisms in isolation from science”.
Then, there is a question of whether we want to predict something about any intelligent^[1] systems (extremely general theory of cognition/agency) so that our predictions (or ensuing process frameworks of alignment) are robust to paradigm shifts in ML/AI, perhaps even right during the recursive self-improvement phase, or we want to prove something about intelligent systems engineered in a particular way.
For the first purpose, I know of three theories/frameworks that are probably not worse than any of the theories that you mentioned:
- Free Energy Principle, Ramstead et al., 2023 (the latest and most up-to-date overview)
- Thermodynamic ML, Boyd et al., 2022
- Maximal Coding Rate Reduction Principle (MCR^2), Ma et al., 2022
For the second purpose, there is nothing wrong with building an AI as an RL system (for example) and then basing a process framework of alignment (corrigibility, control, etc.) of it exactly on the RL formalism because the system was specifically built to conform to it. From this perspective, RL, game theory, control theory, H-JEPA, Constitutional AI for LLMs, and many other theories, formalisms, architectures, and algorithms will be instrumental to developing a safe AI in practice. Cf. “For alignment, we should simultaneously use multiple theories of cognition and value”.
1. ^
  Often these theories extend to describe not just ‘intelligent’, whatever that means, but any non-equilibrium, adaptive systems (or, postulate that any non-equilibrium system is intelligent, in some sense).

Roman Leventov

An LLM-based “ex­em­plary ac­tor”

Align­ing an H-JEPA agent via train­ing on the out­puts of an LLM-based “ex­em­plary ac­tor”

An LLM-based “exemplary actor”

Aligning an H-JEPA agent via training on the outputs of an LLM-based “exemplary actor”