This is a report from a LessWrong perspective, on the 30th Soar workshop. Soar is a cognitive architecture that has been in continuous development for nearly 30 years, and is in a direct line of descent from some of the earliest AI research (Simon’s LT and GPS). Soar is interesting to LessWrong readers for two reasons:
Soar is a cognitive science theory, and has had some success at modeling human reasoning—this is relevant to the central theme of LessWrong, improving human rationality.
Soar is an AGI research project—this is relevant to the AGI risks sub-theme of LessWrong.
Where I’m coming from: I’m a skeptic about EY/SIAI dogmas that AI research is more risky than software development, and that FAI research is not AI research, and has little to learn from the field of AI research. In particular, I want to understand why AI researchers are generally convinced that their experiments and research are fairly safe—I don’t think that EY/SIAI are paying sufficient attention to these expert opinions.
Overall summary: John Laird and his group are smart, dedicated, and funded. Their theory and implementation moves forward slowly but continuously. There’s no (visible) work being done on self-modifying, bootstrapping or approximately-universal (e.g. AIXItl) entities. There is some concern about how to build trustworthy and predictable AIs (for the military’s ROE) - for example, Scott Wallace’s research.
As far as I can tell, the Soar group’s work is no more (or less) risky than narrow AI research or ostensibly non-AI software development. To be blunt—package managers like APT seem more risky than Soar, because the economic forces that push them to more capability and complexity are more difficult to control.
Impressions of (most of) the talks—they can be roughly categorized into three types.
Miscellaneous
Paul Rosenbloom (one of the original authors of Soar, with John Laird and Allen Newell) spoke about trying to create a layer architecturally beneath Soar based on Bayesian graphical models; specifically factor graphs. There were not a lot of people using or trying to use Bayesian magic at the workshop, but Paul Rosenbloom is definitely a Bayesian magician; he’s getting Rete to run as an emergent consequence of the sum-product algorithm.
Ken Forbus (of “Structure-Mapping Engine” fame) spoke generally about his research, including Companions and CogSketch. I got the impression that there are worlds and worlds within academia, and I only have faint visibility into one or a couple of them.
Extending, combining and unifying the existing Soar capabilities (uniformly by Laird and his students):
Nate Derbinsky’s tutorials and talks were on various extensions recently added to Soar (“RL”, “SMem”, “EpMem”).
Integrating reinforcement learning with the (already complicated) Soar architecture must have been difficult, but tabular Q-learning/SARSA is now well integrated, and there’s some support in the released code for eligibility traces and hierarchical reinforcement learning, but not value function approximators. I believe that means that Soar-RL is not as capable at RL tasks as the cutting edge of RL research, but of course, the cutting edge of RL research is not as capable at the symbolic processing tasks that are Soar’s bread and butter.
SMem is essentially a form of content-addressable storage that is under Soar’s explicit control. This is in contrast to Soar’s working memory, which is content-addressible using (Rete) pattern-matching, which is more analogous to being involuntarily reminded of something, than deliberately building a cue and searching one’s memory for a match. This means that SMem scales to larger sizes than working memory.
EpMem is a memory of the content of past working memories. Unlike SMem, (if this feature is turned on) Soar needn’t explicitly store into this memory—every working memory will be stored into EpMem. Fetching from EpMem is content-addressible similarly to SMem, though once an episode has been fetched, Soar can ask what happened next or before that.
John Laird spoke about how the new features of Soar enabled new forms of action modeling. Action modeling is what you use to simulate, internally, the consequences of actions, in the current state or some anticipated future state. Action modeling is necessary to do planning. The standard way in Soar to do this has been for the programmer to add rules explaining the consequences of actions. SMem, EpMem and imagery (a currently-being-developed extension to Soar) can each enable new forms of action modeling.
Mitchell Bloch replicated a particular hierarchical reinforcement learning experiment (Dietterich 1998, MAXQ). My takeaway was that Soar-RL is steadily advancing, incorporating ideas from the RL literature, and though they’re behind the state of the (RL) art now, that’s changing.
Nick Gorski studies combining learning and memory (“learning to use memory”).
Joseph Xu’s first talk was about combining symbolic planning (an old strength of Soar) with imagery and learning. His domain was two-dimensional block stacking with continuous space and gravity, though it might eventually be The Incredible Machine.
In his second talk, he spoke about an architectural variant of Soar-RL, that learns a model and a policy simultaneously, and eliminates a free parameter. His domain here was a probabilistic version of vector racer. Again, Soar-RL is steadily advancing.
Yongjia Wang’s talks both used a domain of learning to hunt various kinds of prey with various kinds of weapons. As far as I could tell, the goal was more integration of RL technology (hierarchical value function approximators) into Soar.
Sam Wintermute spoke about imagery—a not-yet-released extension to Soar. His system learns to play Frogger and two other Atari video games, and learns much better using imagery than without.
Justin Li announced an intention to develop ways to learn subgoal hierarchies. Again, the theme is integration of capabilities; learning and subgoal hierarchies.
Applications of Soar
Kate Tyrol’s and my talks were fairly trivial, trying to connect Soar to Ms. Pacman and Rogue respectively. (We basically used Soar badly; using small fractions of its capabilities and becoming flummoxed by bugs that we introduced.)
I studied Rogue partly because Rogue has escape-to-shell functionality (so a sufficiently clever Rogue AI could easily escape and become a “rogue AI”), and I wanted to understand my own implicit safety case while developing it.
Isaiah Hines (no relation) and Nikhil Mangla announced that they are connecting Soar to Unreal.
Brian Magerko studies improv theatre, and makes some use of Soar in some of his models of improvisation. As far as I can tell, his current use of Soar could be easily replaced by ordinary “non-AI” programming; his association with Soar is in the past (as one of John Laird’s students) and in the future, as his micro-models of improv theatre are (hopefully) combined into an agent that can play improv games.
Shiwali Mohan attacked the Infinite Mario task with Soar-RL using various strategies. My takeaway was that framing the problem for Soar-RL is tricky on realistically difficult tasks like Infinite Mario (rather than “gimme” tasks like T-mazes and broomstick balancing). You can give the task of “make it learn” to a smart grad student and have them tweak and poke at it for a year or so, and still have substantially lower learned performance than handcoded agents.
Bryan Smith spoke about putting OWL ontologies into Soar’s SMem. As far as I can tell, this wasn’t for any particular purpose, but just to play with the two technologies.
Sean Bittle spoke about using Soar to learn heuristics for constraint programming. As far as I can tell, this was a negative or intermediate result; essentially no successful learning, or too little for the amount of effort and complexity introduced.
Olivier Georgeon spoke about a model of early developmental learning. I believe this is intended to model human learning, but the domain he was using was a fairly inhuman gridworld.
Bob Wray spoke about using Soar to create learning experiences.
I was a bit disappointed, since (as far as I can tell) this work simply used Soar as an exotic programming language. I believe this is one of the primary ways that Soar could help expand human rationality: if the procedures that a human is supposed to learn are encoded as Soar productions, and the training software can test whether any student has any given production, and instill it if it is not there, then (assuming Soar is a decent model of how humans think), instilling all of the necessary productions should also instill the complete procedure.
Margaux Lhommet announced an intention to use Soar in a training simulation to control simulated victims of a radioactive / biological terrorist attack.
Bill Kennedy announced that he was looking for something like Soar, but lighter-weight, so he could do large-scale agent-based simulations.
Nate Derbinsky wrote a piece of middleware (Soar2Soar) so that Soar programmers can use the Soar programming language to write the environment. I’m so unskilled with Soar that that doesn’t sound like a win to me, but Soar is sometimes very declarative, and might be appropriate for rapid development. He also got Soar to run on an iPhone.
Jonathan Voight spoke about Sproom, a mixed virtual/real robot simulation. The agent can control a virtual robot or a real robot—and even when its controlling a real robot, the virtual environment can augment the inputs, so a wall-avoiding robot would avoid both real and virtual walls. The bot’s effectors can be implemented in the virtual reality, so it can drive around in the real world, picking up virtual objects and putting them away.
John Laird spoke about using Sproom to study “Situated Interactive Instruction”—so that an agent could be taught by interacting in semi-formal language with a human, while it is performing its task. The domain is robots moving through a building doing IED-clearing; the IEDs and the operations on them (pickup, defuse) are virtual. As I understand it, some but not all of this functionality is currently functioning.
Shiwali Mohan used the same sort of situated interactive instruction in the Infinite Mario domain, though not very much data was conveyed (yet) via instruction. As I understand it, Mohan’s previous hardwired agent had three verbs like “tackle-monster” and “get-coin”; the instruction consists of the agent asking “I see a coin, which verb should I use for it?”—so after the human has answered the three object-verb correspondences, it knows everything it will ever learn via instruction. However, it’s working and could be extended.
I want to emphasize that these just my impressions (which are probably flawed—because of my inexperience I probably misunderstood important points), and the proceedings (that is, the slides that everyone used to talk with) will soon be available, so you can read them and form your own impressions.
There are three forks to my implicit safety case while developing. I’m not claiming this is a particularly good safety case or that developing Rogue-Soar was safe—just that it’s what I have.
The first fork is that tasks vary in their difficulty (Pickering’s “resistances”), and entities vary in their strength or capability. There’s some domain-ish structure to entity’s strengths (a mechanical engineering task will be easier for someone trained as a mechanical engineer than a chemist), and intention matters—difficult tasks are rarely accomplished unintentionally. I’m fairly weak, and my agent-in-progress was and is very, very weak. The chance that I or my agent solves a difficult task (self-improving AGI) unintentionally while writing Rogue-Soar is incredibly small, and comparable to the risk of my unintentionally solving self-improving AGI while working at my day job. This suggests that safer (not safe) AI development might involve: One, tracking ELO-like scores of people’s strengths and task difficulties, and Two, tracking and incentivizing people’s intentions.
The second fork is that even though I’m surprised sometimes while developing, the surprises are still confined to an envelope of possible behavior. The agent could crash, run forever, move in a straight line or take only one step when I was expecting it to wander randomly, but pressing ”!” when I expected it to be confined to “hjkl” would be beyond this envelope. Of course, there are many nested envelopes, and excursions beyond the bounds of the narrowest are moderately frequent. This suggests that safer AI development might involve tracking these behavior envelopes (altogether they might form a behavior gradient), and the frequency and degree of excursions, and deciding whether development is generally under control—that is, acceptably risky compared to the alternatives.
The third fork is that the runaway takeoff arguments necessarily involve circularities and feedback. By structural inspection and by intention, if the AI is dealing with Rogue, and not learning, programming, or bootstrapping, then it’s unlikely to undergo takeoff. This suggests that carefully documenting and watching for circularities and feedback may be helpful for safer AI research.
30th Soar workshop
This is a report from a LessWrong perspective, on the 30th Soar workshop. Soar is a cognitive architecture that has been in continuous development for nearly 30 years, and is in a direct line of descent from some of the earliest AI research (Simon’s LT and GPS). Soar is interesting to LessWrong readers for two reasons:
Soar is a cognitive science theory, and has had some success at modeling human reasoning—this is relevant to the central theme of LessWrong, improving human rationality.
Soar is an AGI research project—this is relevant to the AGI risks sub-theme of LessWrong.
Where I’m coming from: I’m a skeptic about EY/SIAI dogmas that AI research is more risky than software development, and that FAI research is not AI research, and has little to learn from the field of AI research. In particular, I want to understand why AI researchers are generally convinced that their experiments and research are fairly safe—I don’t think that EY/SIAI are paying sufficient attention to these expert opinions.
Overall summary: John Laird and his group are smart, dedicated, and funded. Their theory and implementation moves forward slowly but continuously. There’s no (visible) work being done on self-modifying, bootstrapping or approximately-universal (e.g. AIXItl) entities. There is some concern about how to build trustworthy and predictable AIs (for the military’s ROE) - for example, Scott Wallace’s research.
As far as I can tell, the Soar group’s work is no more (or less) risky than narrow AI research or ostensibly non-AI software development. To be blunt—package managers like APT seem more risky than Soar, because the economic forces that push them to more capability and complexity are more difficult to control.
Impressions of (most of) the talks—they can be roughly categorized into three types.
Miscellaneous
Paul Rosenbloom (one of the original authors of Soar, with John Laird and Allen Newell) spoke about trying to create a layer architecturally beneath Soar based on Bayesian graphical models; specifically factor graphs. There were not a lot of people using or trying to use Bayesian magic at the workshop, but Paul Rosenbloom is definitely a Bayesian magician; he’s getting Rete to run as an emergent consequence of the sum-product algorithm.
Ken Forbus (of “Structure-Mapping Engine” fame) spoke generally about his research, including Companions and CogSketch. I got the impression that there are worlds and worlds within academia, and I only have faint visibility into one or a couple of them.
Extending, combining and unifying the existing Soar capabilities (uniformly by Laird and his students):
Nate Derbinsky’s tutorials and talks were on various extensions recently added to Soar (“RL”, “SMem”, “EpMem”).
Integrating reinforcement learning with the (already complicated) Soar architecture must have been difficult, but tabular Q-learning/SARSA is now well integrated, and there’s some support in the released code for eligibility traces and hierarchical reinforcement learning, but not value function approximators. I believe that means that Soar-RL is not as capable at RL tasks as the cutting edge of RL research, but of course, the cutting edge of RL research is not as capable at the symbolic processing tasks that are Soar’s bread and butter.
SMem is essentially a form of content-addressable storage that is under Soar’s explicit control. This is in contrast to Soar’s working memory, which is content-addressible using (Rete) pattern-matching, which is more analogous to being involuntarily reminded of something, than deliberately building a cue and searching one’s memory for a match. This means that SMem scales to larger sizes than working memory.
EpMem is a memory of the content of past working memories. Unlike SMem, (if this feature is turned on) Soar needn’t explicitly store into this memory—every working memory will be stored into EpMem. Fetching from EpMem is content-addressible similarly to SMem, though once an episode has been fetched, Soar can ask what happened next or before that.
John Laird spoke about how the new features of Soar enabled new forms of action modeling. Action modeling is what you use to simulate, internally, the consequences of actions, in the current state or some anticipated future state. Action modeling is necessary to do planning. The standard way in Soar to do this has been for the programmer to add rules explaining the consequences of actions. SMem, EpMem and imagery (a currently-being-developed extension to Soar) can each enable new forms of action modeling.
Mitchell Bloch replicated a particular hierarchical reinforcement learning experiment (Dietterich 1998, MAXQ). My takeaway was that Soar-RL is steadily advancing, incorporating ideas from the RL literature, and though they’re behind the state of the (RL) art now, that’s changing.
Nick Gorski studies combining learning and memory (“learning to use memory”).
Joseph Xu’s first talk was about combining symbolic planning (an old strength of Soar) with imagery and learning. His domain was two-dimensional block stacking with continuous space and gravity, though it might eventually be The Incredible Machine.
In his second talk, he spoke about an architectural variant of Soar-RL, that learns a model and a policy simultaneously, and eliminates a free parameter. His domain here was a probabilistic version of vector racer. Again, Soar-RL is steadily advancing.
Yongjia Wang’s talks both used a domain of learning to hunt various kinds of prey with various kinds of weapons. As far as I could tell, the goal was more integration of RL technology (hierarchical value function approximators) into Soar.
Sam Wintermute spoke about imagery—a not-yet-released extension to Soar. His system learns to play Frogger and two other Atari video games, and learns much better using imagery than without.
Justin Li announced an intention to develop ways to learn subgoal hierarchies. Again, the theme is integration of capabilities; learning and subgoal hierarchies.
Applications of Soar
Kate Tyrol’s and my talks were fairly trivial, trying to connect Soar to Ms. Pacman and Rogue respectively. (We basically used Soar badly; using small fractions of its capabilities and becoming flummoxed by bugs that we introduced.)
I studied Rogue partly because Rogue has escape-to-shell functionality (so a sufficiently clever Rogue AI could easily escape and become a “rogue AI”), and I wanted to understand my own implicit safety case while developing it.
Isaiah Hines (no relation) and Nikhil Mangla announced that they are connecting Soar to Unreal.
Brian Magerko studies improv theatre, and makes some use of Soar in some of his models of improvisation. As far as I can tell, his current use of Soar could be easily replaced by ordinary “non-AI” programming; his association with Soar is in the past (as one of John Laird’s students) and in the future, as his micro-models of improv theatre are (hopefully) combined into an agent that can play improv games.
Shiwali Mohan attacked the Infinite Mario task with Soar-RL using various strategies. My takeaway was that framing the problem for Soar-RL is tricky on realistically difficult tasks like Infinite Mario (rather than “gimme” tasks like T-mazes and broomstick balancing). You can give the task of “make it learn” to a smart grad student and have them tweak and poke at it for a year or so, and still have substantially lower learned performance than handcoded agents.
Bryan Smith spoke about putting OWL ontologies into Soar’s SMem. As far as I can tell, this wasn’t for any particular purpose, but just to play with the two technologies.
Sean Bittle spoke about using Soar to learn heuristics for constraint programming. As far as I can tell, this was a negative or intermediate result; essentially no successful learning, or too little for the amount of effort and complexity introduced.
Olivier Georgeon spoke about a model of early developmental learning. I believe this is intended to model human learning, but the domain he was using was a fairly inhuman gridworld.
Bob Wray spoke about using Soar to create learning experiences.
I was a bit disappointed, since (as far as I can tell) this work simply used Soar as an exotic programming language. I believe this is one of the primary ways that Soar could help expand human rationality: if the procedures that a human is supposed to learn are encoded as Soar productions, and the training software can test whether any student has any given production, and instill it if it is not there, then (assuming Soar is a decent model of how humans think), instilling all of the necessary productions should also instill the complete procedure.
Margaux Lhommet announced an intention to use Soar in a training simulation to control simulated victims of a radioactive / biological terrorist attack.
Bill Kennedy announced that he was looking for something like Soar, but lighter-weight, so he could do large-scale agent-based simulations.
Nate Derbinsky wrote a piece of middleware (Soar2Soar) so that Soar programmers can use the Soar programming language to write the environment. I’m so unskilled with Soar that that doesn’t sound like a win to me, but Soar is sometimes very declarative, and might be appropriate for rapid development. He also got Soar to run on an iPhone.
Jonathan Voight spoke about Sproom, a mixed virtual/real robot simulation. The agent can control a virtual robot or a real robot—and even when its controlling a real robot, the virtual environment can augment the inputs, so a wall-avoiding robot would avoid both real and virtual walls. The bot’s effectors can be implemented in the virtual reality, so it can drive around in the real world, picking up virtual objects and putting them away.
John Laird spoke about using Sproom to study “Situated Interactive Instruction”—so that an agent could be taught by interacting in semi-formal language with a human, while it is performing its task. The domain is robots moving through a building doing IED-clearing; the IEDs and the operations on them (pickup, defuse) are virtual. As I understand it, some but not all of this functionality is currently functioning.
Shiwali Mohan used the same sort of situated interactive instruction in the Infinite Mario domain, though not very much data was conveyed (yet) via instruction. As I understand it, Mohan’s previous hardwired agent had three verbs like “tackle-monster” and “get-coin”; the instruction consists of the agent asking “I see a coin, which verb should I use for it?”—so after the human has answered the three object-verb correspondences, it knows everything it will ever learn via instruction. However, it’s working and could be extended.
I want to emphasize that these just my impressions (which are probably flawed—because of my inexperience I probably misunderstood important points), and the proceedings (that is, the slides that everyone used to talk with) will soon be available, so you can read them and form your own impressions.
There are three forks to my implicit safety case while developing. I’m not claiming this is a particularly good safety case or that developing Rogue-Soar was safe—just that it’s what I have.
The first fork is that tasks vary in their difficulty (Pickering’s “resistances”), and entities vary in their strength or capability. There’s some domain-ish structure to entity’s strengths (a mechanical engineering task will be easier for someone trained as a mechanical engineer than a chemist), and intention matters—difficult tasks are rarely accomplished unintentionally. I’m fairly weak, and my agent-in-progress was and is very, very weak. The chance that I or my agent solves a difficult task (self-improving AGI) unintentionally while writing Rogue-Soar is incredibly small, and comparable to the risk of my unintentionally solving self-improving AGI while working at my day job. This suggests that safer (not safe) AI development might involve: One, tracking ELO-like scores of people’s strengths and task difficulties, and Two, tracking and incentivizing people’s intentions.
The second fork is that even though I’m surprised sometimes while developing, the surprises are still confined to an envelope of possible behavior. The agent could crash, run forever, move in a straight line or take only one step when I was expecting it to wander randomly, but pressing ”!” when I expected it to be confined to “hjkl” would be beyond this envelope. Of course, there are many nested envelopes, and excursions beyond the bounds of the narrowest are moderately frequent. This suggests that safer AI development might involve tracking these behavior envelopes (altogether they might form a behavior gradient), and the frequency and degree of excursions, and deciding whether development is generally under control—that is, acceptably risky compared to the alternatives.
The third fork is that the runaway takeoff arguments necessarily involve circularities and feedback. By structural inspection and by intention, if the AI is dealing with Rogue, and not learning, programming, or bootstrapping, then it’s unlikely to undergo takeoff. This suggests that carefully documenting and watching for circularities and feedback may be helpful for safer AI research.