Roman Leventov

Cri­tique of some re­cent philos­o­phy of LLMs’ minds

Yoshua Ben­gio: “Slow­ing down de­vel­op­ment of AI sys­tems pass­ing the Tur­ing test”

[Question] Has pri­vate AGI re­search made in­de­pen­dent safety re­search in­effec­tive already? What should we do about this?

A multi-dis­ci­plinary view on AI safety research

How evolu­tion­ary lineages of LLMs can plan their own fu­ture and act on these plans

Pro­posal for im­prov­ing the global on­line dis­course through per­son­al­ised com­ment or­der­ing on all websites

Pow­er­ful mesa-op­ti­mi­sa­tion is already here

AGI will be made of het­ero­ge­neous com­po­nents, Trans­former and Selec­tive SSM blocks will be among them

Prop­er­ties of cur­rent AIs and some pre­dic­tions of the evolu­tion of AI from the per­spec­tive of scale-free the­o­ries of agency and reg­u­la­tive development

A re­ply to Byrnes on the Free En­ergy Principle

• Roman Leventov 15 Feb 2023 16:26 UTC

  27 points

  18

  on: Bing Chat is blatantly, ag­gres­sively misaligned

  This looks like a very good moment for the AI safety community to push for a much more deliberate approach to R&D at AGI labs. We must not squander this moment. This is not quite a “sinister stumble”, an event of another kind, but of comparable positive optionality for AI safety.

  By “a more deliberate approach to R&D”, I mean researching much more deeply from both theoretical (scientific) and interpretability standpoints what’s going on with (self-)awareness, agency, and feeling in these networks, and publishing their results academically.

An open let­ter to SERI MATS pro­gram organisers

AI ro­man­tic part­ners will harm so­ciety if they go unregulated

AGI-level rea­soner will ap­pear sooner than an agent; what the hu­man­ity will do with this rea­soner is critical

• Roman Leventov 1 Aug 2022 10:14 UTC

  24 points

  10

  on: chin­chilla’s wild implications

  My two cents contra updates towards longer or more uncertain AGI timelines given the information in this post:

  • The training of language models is many orders of magnitude less efficient than the training of the human brain, which acquires comparable language comprehension and generation ability on a tiny fraction of the text corpora discussed in this post. So we can expect more innovations that improve the training efficiency. Even one such innovation, improving the training efficiency (in terms of data) by a single order of magnitude, would probably ensure that the total size of publically available text data is not a roadblock on the path to AGI, even if it is, currently. I think the probability that we will see at least one such innovation in the next 5 years is quite high, more than 10%.

  • Perhaps DeepMind’s Gato is already a response to the realisation that “there is not enough text”, explained in this post. So they train Gato on game replays, themselves generated programmatically, using RL agents. They can generate practically unlimited amounts of training data in this way. Then there is probably a speculation that at some scale, Gato will generalise the knowledge acquired in games to text, or will indeed enable much more efficient training on text, (a-la few-shot learning in current LMs) if the model is pre-trained on games and other tasks.

Open Agency model can solve the AI reg­u­la­tion dilemma

For al­ign­ment, we should si­mul­ta­neously use mul­ti­ple the­o­ries of cog­ni­tion and value

• Roman Leventov 26 Feb 2023 8:25 UTC

  22 points

  10

  on: Cog­ni­tive Emu­la­tion: A Naive AI Safety Proposal

  I understood the proposal as “let’s create relatively small, autonomous AIs of multi-component cognitive architecture instead of unitary DNNs becoming global services (and thus ushering cognitive globalisation)”. The key move seems to be that you want to increase the AI architecture’s interpretability by pushing some intelligence into the multi-component interaction from the “DNN depths”. In this setup, components may remain relatively small (e.g., below 100B parameter scale, i.e., at the level of the current SoTA DNNs), while their interaction leads to the emergence of general intelligence, which may not be achievable for unitary DNNs at these model scales.

  The proposal seems to be in some ways very close to the recent Eric Drexler’s Open Agency Model. Both your proposal and Drexler’s “open agencies” seem to noticeably allude to the (classic) approaches to cognitive architecture, a la OpenCog, or LeCun’s H-JEPA.

  As well as Drexler, it seems that you take for granted that multi-component cognitive architectures will have various “good” properties, ranging from interpretability to retargetability (which Michael Levin calls persuadability, btw). However, as I noted in this comment, none of these “good properties” are actually granted for an arbitrary multi-component AI. It must be demonstrated for a specific multi-component architecture why it is more interpretable (see also in the linked comment my note that “outputting plans” != interpretability), persuadable, robust, and ethical than an alternative architecture.

  Your approach to this seems to be “minimal”, that is, something like “at least we know the level of interpretability, persuadability/​corrigibility, robustness, and ethics of humans, so let’s try to build AI ‘after humans’ so that we get at least these properties, rather than worse properties”.

  As such, I think this approach might not really solve anything and might not “end the acute risk period”, because it amounts to “building more human-like minds”, with all other economic, social, and political dynamics intact. I don’t see how building “just more humans” prevents corporations from pursuing cognitive globalisation in one form or another. The approach just seems to add more “brain-power” to the engine (either via building AI “like humans, but 2-3 sigmas more intelligent”, or just building many of them and keeping them running around the clock), without targeting any problems with the engine itself. In other words, the approach is not placed within a frame of a larger vision for civilisational intelligence architecture, which, I argued, is a requirement for any “AI safety paradigm”: “Both AI alignment paradigms (protocols) and AGI capability research (intelligence architectures) that don’t position themselves within a certain design for civilisational intelligence are methodologically misguided and could be dangerous.”

  Humans have rather bad interpretability (see Chater’s “The Mind is Flat”), bad persuadability (see Scott Alexander’s “Trapped Priors”), bad robustness (see John Doyle’s hijackable language and memetic viruses), poor capability for communication and alignment (as anyone who tried to reliably communicate any idea to anyone else or to align with anyone else on anything can easily attest; cf. discussion of communication protocols in “Designing Ecosystems of Intelligence from First Principles”), and, of course, poor ethics. It’s also important to note that all these characteristics seem to be largely uncorrelated (at least, not strictly pegged) with raw general intelligence (GI factor) in humans.

  I agree with you and Friston and others who worry about the cognitive globalisation and hyperscaling approach, but I also think that in order to improve the chances of humanity, we should at least aim at better than human architecture from the beginning. Creating many AI minds of the architecture “just like humans” (unless you count on some lucky emergence and that even trying to target “at humans” will yield architecture “better than humans”) doesn’t seem to help in civilisational intelligence and robustness, just accelerate the current trends.

  I’m also optimistic because I see nothing impossibly hard or intractable in designing cognitive architectures that would be better than humans from the beginning and getting good engineering assurances that the architecture will indeed yield these (better than human) characteristics. It just takes time and a lot of effort (but so as architecturing AI “just like humans” does). E.g., (explicit) Active Inference architecture seems to help significantly at least with interpretability and the capacity for reliable and precise communication and, hence, belief and goal alignment (see Friston et al., 2022).

  What links here?

  • AGI will be made of het­ero­ge­neous com­po­nents, Trans­former and Selec­tive SSM blocks will be among them by Roman Leventov (27 Dec 2023 14:51 UTC; 33 points)
  • AGI will be made of het­ero­ge­neous com­po­nents, Trans­former and Selec­tive SSM blocks will be among them by Roman Leventov (EA Forum; 27 Dec 2023 14:51 UTC; 5 points)

AI al­ign­ment as a trans­la­tion problem

[Question] Any re­search in “probe-tun­ing” of LLMs?