The NTSB report was released last week, showing that Uber’s engineering was doing some things very wrong (with specifics that had not been reported before). Self-driving programs shouldn’t go on public roads with that kind of system, even with a driver ready to take over.
orthonormal
Karma: 16,678
Exactly. It seems like you need something beyond present imitation learning and deep reinforcement learning to efficiently learn strategies whose individual components don’t benefit you, but which have a major effect if assembled perfectly together.
(I mean, don’t underestimate gradient descent with huge numbers of trials—the genetic version did evolve a complicated eye in such a way that every step was a fitness improvement; but the final model has a literal blind spot that could have been avoided if it were engineered in another way.)
I just finished Story of Your Life, and I disagree with your attempt to retcon it as a physically possible story; I think the evidence is clear that Chiang intended it to be understood the way that most people understand it, and that he in fact was misunderstanding* the interaction between the variational principle and causality.
[Spoilers ahead.]
Firstly, it is a really plausible misunderstanding. If you don’t deeply grok how decoherence works, then the principle of least action can really seem like it requires information to behave in nonsequential ways. (But in fact the Schrodinger equation exhibits causality, and variational principles are just the result of amplitudes nearly cancelling out for all paths whose action is not a local extremum.) Chiang is a gifted amateur in physics, so this misconception is not that unlikely.
Secondly, if your interpretation were his intended one, he could have done any number of things to suggest it! Louise directly tells the reader that this story is being told from the night of her daughter’s conception, and that she now sees both future and past but does not act to change it. This is shown in the story to be the point of Heptapod B, and Chiang works at length to try and justify how understanding this language lets one understand (but not alter) the future. There’s nothing else in the story to hint that Louise is an unreliable narrator.
If your interpretation were Chiang’s, he would have to be intentionally misdirecting the audience to a degree you only see from authors like Nabokov, and not leaving any clue except for flawed science, which is common enough for dramatic license in science fiction that it really can’t count as a clue. I doubt Chiang is doing that.
And finally, the whole thematic import of the story is ruined by your interpretation; the poignance comes from Louise’s existentialist fatalism, giving life to the lines of the tragic play she is cast in, which is only meaningful insofar as she’s telling the truth about perceiving forwards and backwards. (It’s a variant on the Trafalmadorians, though interestingly Chiang hadn’t yet read Slaughterhouse Five when he wrote this.) It’s just a more boring story the way you see it!
*Possibly intentionally misunderstanding; I can imagine a science fiction author pestering their physics friend with “I’ve got a great story idea using this concept”, the friend saying “It doesn’t work that way”, and the author asking “But if I pretend it does work that way, exactly how much physics knowledge would it take someone to be annoyed by it?”
Decision-theoretic blackmail is when X gets Y to choose A over B, not via acting to make the consequences of A more appealing to Y, but by making the consequences of B less appealing to Y.
The exceptions to this definition are pretty massive, though, and I don’t know a principled emendation that excludes them.
1. There’s a contract / social contract / decision-theoretic equilibrium, and within that, B will be punished. (This may not be a true counterexample, because the true choice is whether to join the contract… though this is less clear for the social contract than for the other two.)
2. Precommitting not to give in to blackmail is not itself blackmail. Of course, in an ultimatum game both players can imagine themselves as doing this.
Can anyone think of more exceptions, or a redefinition that clearly excludes these?
Removing things entirely seems extreme.
Dropout is a thing, though.
Shapley Values [thanks Zack for reminding me of the name] are akin to credit assignment: you have a bunch of agents coordinating to achieve something, and then you want to assign payouts fairly based on how much each contribution mattered to the final outcome.
And the way you do this is, for each agent you look at how good the outcome would have been if everybody except that agent had coordinated, and then you credit each agent proportionally to how much the overall performance would have fallen off without them.
So what about doing the same here- send rewards to each contributor proportional to how much they improved the actual group decision (assessed by rerunning it without them and seeing how performance declines)?
Customer service human interactions don’t feel especially valuable to me, compared to intentional human interactions or even seeing other people walking down the street.
Good comment. I disagree with this bit:
I would, for instance, predict that if Superintelligence were published during the era of GOFAI, all else equal it would’ve made a bigger splash because AI researchers then were more receptive to abstract theorizing.
And then it would probably have been seen as outmoded and thrown away completely when AI capabilities research progressed into realms that vastly surpassed GOFAI. I don’t know that there’s an easy way to get capabilities researchers to think seriously about safety concerns that haven’t manifested on a sufficient scale yet.
To copy myself in another thread, AlphaZero did some (pruned) game tree exploration in a hardcoded way that allowed the NN to focus on the evaluation of how good a given position was; this allowed it to kind of be a “best of both worlds” between previous algorithms like Stockfish and a pure deep reinforcement learner.
Re: your middle paragraph, I agree that you’re correct about an RL agent doing metalearning, though we’re also agreed that with current architectures it would take a prohibitive amount of computation to get anything like a competent general causal reasoner that way.
I’m not going to go up against your intuitions on imitation learning etc; I’m just surprised if you don’t expect there’s a necessary architectural advance needed to make anything like general causal reasoning emerge in practice from some combination of imitation learning and RL.
AlphaZero did some (pruned) game tree exploration in a hardcoded way that allowed the NN to focus on the evaluation of how good a given position was; this allowed it to kind of be a “best of both worlds” between previous algorithms like Stockfish and a pure deep reinforcement learner.
This is impossible for a game with an action space as large as StarCraft II, though; but in order to modify a game like Go, it would have to become completely different.
I’m not 100% sure about the example you raise, but it seems to me it’s either going to have a decently prune-able game tree, or that humans won’t be capable of playing the game at a very sophisticated level, so I’d expect AlphaZero-esque things to get superhuman at it. StarCraft is easier for humans relative to AIs because we naturally chunk concepts together (visually and strategically) that are tricky for the AI to learn.
DeepMind says it hopes the techniques used to develop AlphaStar will ultimately help it “advance our research in real-world domains”.
But Prof Silver said the lab “may rest at this point”, rather than try to get AlphaStar to the level of the very elite players.
AlphaStar: Impressive for RL progress, not for AGI progress
Thanks, will do.
Virtue ethics seems less easily applicable to the domain of “what governmental policies to support” than to the domain of personal behavior, so I had a hard time thinking of examples. Can you?
Right, for them “alignment” could mean their desired concept, “safe for everyone except our targets”.
DeepMind released their AlphaStar paper a few days ago, having reached Grandmaster level at the partial-information real-time strategy game StarCraft II over the summer.
This is very impressive, and yet less impressive than it sounds. I used to watch a lot of StarCraft II (I stopped interacting with Blizzard recently because of how they rolled over for China), and over the summer there were many breakdowns of AlphaStar games once players figured out how to identify the accounts.
The impressive part is getting reinforcement learning to work at all in such a vast state space- that took breakthroughs beyond what was necessary to solve Go and beat Atari games. AlphaStar had to have a rich enough set of potential concepts (in the sense that e.g. a convolutional net ends up having concepts of different textures) that it could learn a concept like “construct building P” or “attack unit Q” or “stay out of the range of unit R” rather than just “select spot S and enter key T”. This is new and worth celebrating.
The overhyped part is that AlphaStar doesn’t really do the “strategy” part of real-time strategy. Each race has a few solid builds that it executes at GM level, and the unit control is fantastic, but the replays don’t look creative or even especially reactive to opponent strategies.
That’s because there’s no representation of causal thinking—“if I did X then they could do Y, so I’d better do X’ instead”. Instead there are many agents evolving together, and if there’s an agent evolving to try Y then the agents doing X will be replaced with agents that do X’.
(This lack of causal reasoning especially shows up in building placement, where the consequences of locating any one building here or there are minor, but the consequences of your overall SimCity are major for how your units and your opponents’ units would fare if they attacked you. In one comical case, AlphaStar had surrounded the units it was building with its own factories so that they couldn’t get out to reach the rest of the map. Rather than lifting the buildings to let the units out, which is possible for Terran, it destroyed one building and then immediately began rebuilding it before it could move the units out!)
This means that, first, AlphaStar just doesn’t have a decent response to strategies that it didn’t evolve, and secondly, it doesn’t do much in the way of a reactive decision tree of strategies (if I scout this, I do that). That kind of play is unfortunately very necessary for playing Zerg at a high level, so the internal meta has just collapsed into one where its Zerg agents predictably rush out early attacks that are easy to defend if expected. This has the flow-through effect that its Terran and Protoss are weaker against human Zerg than against other races, because they’ve never practiced against a solid Zerg that plays for the late game.
The end result cleaned up against weak players, performed well against good players, but practically never took a game against the top few players. I think that DeepMind realized they’d need another breakthrough to do what they did to Go, and decided to throw in the towel while making it look like they were claiming victory.
Finally, RL practitioners have known that genuine causal reasoning could never be achieved via known RL architectures- you’d only ever get something that could execute the same policy as an agent that had reasoned that way, via a very expensive process of evolving away from dominated strategies at each step down the tree of move and countermove. It’s the biggest known unknown on the way to AGI.
[Cross-posted from Medium, written for a pretty general audience]
There are many words that could describe my political positions. But there’s one fundamental label for me: I am a consequentialist.
Consequentialism is a term from ethics; there, it means the position that consequences are what truly make an action right or wrong, rather than rules or virtues. What that means is that for me, the most essential questions about policy aren’t things like “what is fair” or “what rights do people have”, although these are good questions. For me, it all boils down to “how do we make people’s lives better?”
(There are some bits of nuance to the previous paragraph, which I’ve kept as a long endnote.)
“Make people’s lives better” isn’t a platitude- there’s a real difference here! To explain, I want to point out that there are both consequentialists and non-consequentialists within different political camps. Let’s consider socialists first and then libertarians second.
Many socialists believe both that (A) the world is headed for plutocratic disaster unless capitalism is overthrown, and that (B) labor markets and massive wealth disparities would be crimes even if they did not doom others to suffering. The difference is that some are more motivated by beliefs like (A), and could thus change their positions if convinced that e.g. the Nordic model was much better for future growth than a marketless society; while others are more motivated by beliefs like (B), and would continue to support pure socialism even if they were convinced it would mean catastrophe.
And many libertarians believe both that (A’) the only engine that can consistently create prosperity for all is a free market with no interference, and that (B’) taxation is a monstrous act of aggression and theft. The difference is that some are more motivated by beliefs like (A’), and thus could change their position if convinced that e.g. progressive taxation and redistribution would not destroy the incentives behind economic growth; while others are more motivated by beliefs like (B’), and would continue to support pure libertarianism even if they were convinced it would mean catastrophe.
I find it fruitful to talk with the first kind of socialist and the first kind of libertarian, but not the second kind of either. The second type just isn’t fundamentally interested in thinking about the consequences (except insofar as they can convince others by arguing for certain consequences). But among the first type, it’s possible to figure out the truth together by arguing about historical cases, studying natural experiments in policy, and articulating different theories.
I hope it’s been helpful to draw out this distinction; I’d encourage you to first find fellow consequentialists among your natural allies, and expand from there when and if you feel comfortable. There’s a lot that can be done to make the world a better place, and those of us who care most about making the world better can achieve more once we find each other!
P.S. The above focuses on the sort of political questions where most people’s influence is limited to voting and convincing others to vote with them. But there’s more ways to have an effect than that; I’d like to take one last moment to recommend the effective altruism movement, which investigates the best ways for people to have a big positive impact on the world.
---
Nuance section:
the position that consequences are what truly make an action right or wrong
There’s a naive version of this, which is that you should seize any good immediate outcome you can, even by doing horrific things. That’s… not a healthy version of consequentialism. The way to be less naive is to care about long-term consequences, and also to expect that you can’t get away with keeping your behavior secret from others in general. Here’s a good account of what non-naive consequentialism can look like.
the most essential questions about policy aren’t things like “what is fair” or “what rights do people have”, although these are good questions
In particular, fairness and rights are vital to making people’s lives better! We want more than just physical comforts; we want autonomy and achievement and meaning, we want to have trustworthy promises about what the world will ask of us tomorrow, and we want past injustices to be rectified. But these can be traded off, in extreme situations, against the other things that are important for people. In a massive emergency, I’d rather save lives in an unfair way and try to patch up the unfairness later, than let people die to preserve fairness.
how do we make people’s lives better?
This gets complicated and weird when you apply it to things like our distant descendants, but there are some aspects in the world today that seem fairly straightforward. Our world has built an engine of prosperity that makes food and goods available to many, beyond what was dreamt of in the past. But many people in the world are still living short and painful lives filled with disease and starvation. Another dollar of goods will do much more for one of them than for one of us. If we can improve their lives without destroying that engine, it is imperative to do that. (What consequentialists mostly disagree on is how the engine really works, how it could be destroyed, and how it could be improved!)
orthonormal’s Shortform
Well, damn. I started out reading this really skeptically, but the process fit my own past experiences of changing core beliefs frighteningly well. (The only thing that’s ever noticeably worked for me is something like showing that one of my IFS parts is a hypocrite, which results in it more or less self-destructing and allowing change to happen.) I just emailed a Coherence Therapy practitioner, since I’m looking for a new therapist; this seems worth a try.
The other explanation I’ve heard bandied about is a polygenic version of sickle-cell anemia (where being heterozygous for the allele protects you from malaria but being homozygous gives you an awful disease).
In this model, there are a bunch of alleles that all push the phenotype in roughly the same direction, and having some of them is good, but past some threshold fitness starts rapidly declining.
(Further speculation: the optimum threshold is higher in the environment of civilization than in the ancestral environment, so these genes are experiencing massive positive selection over the last 10,000 years.)
This isn’t a causal explanation, but it differs from the tower-foundation model in claiming that there’s not a separate weakness to go looking for, just an optimum that’s being surpassed- especially when two people near the optimum have children.