paulfchristiano

Karma: 8,352 (LW), 276 (AF)

paulfchristiano 4 Jun 2019 15:56 UTC
5 points

in reply to: rmoehn's comment on: Prosaic AI alignment
I’m imagining the first marginal unit of effort, which you’d apply to the most likely possibility. Its expected impact is reduced by that highest probability.
If you get unlucky, then your actual impact might be radically lower than if you had known what to work on.

paulfchristiano 4 Jun 2019 2:15 UTC
6 points

in reply to: Søren Elverlin's comment on: Any rebuttals of Christiano and AI Impacts on takeoff speeds?
An example would be my Matrix Multiplication example (https://youtu.be/5DDdBHsDI-Y). Here, a series of 4 key insights turn the problem from requiring a decade, to a year, to a day, to a second.
In fact Strassen’s algorithm is worse than textbook matrix multiplication for most reasonably sized matrices, including all matrices that could be multiplied in the 70s. Even many decades later the gains are still pretty small (and it’s only worth doing for unusually giant matrix multiplies). As far as I am aware nothing more complicated than Strassen’s algorithm is ever used in practice. So it doesn’t seem like an example of a key insight enabling a problem to be solved.
We could imagine an alternate reality in which large matrix multiplications became possible only after we discovered Strassen’s algorithm. But I think there is a reason that reality is alternate.
Overall I think difficult theory and clever insights are sometimes critical, perhaps often enough to more than justify our society’s tiny investment in them, but it’s worth having a sense of how exceptional these cases are.

paulfchristiano 12 May 2019 6:34 UTC
23 points

on: Disincentives for participating on LW/AF
I don’t comment more because writing comments takes time. I think that in person discussions tend to add more value per minute. (I expect your post is targeted at people who comment less than I do, but the reasons may be similar.)
I can imagine getting more mileage out of quick comments, which would necessarily be short and unplished. I’m less likely to do that because I feel like fast comments will often reflect poorly on me for a variety of reasons: they would have frequent and sometimes consequential errors (that would be excused in a short in-person discussion because of time), in general hastily-written comments send negative signal (better people write better comments, faster comments are worse, full model left as exercise for reader), I’d frequently leave errors uncorrected or threads of conversation dropped, and so on.

paulfchristiano 19 Apr 2019 16:43 UTC
12 points

in reply to: Wei_Dai's comment on: Announcement: AI alignment prize round 4 winners
Is there another way to spend money that seems clearly more cost-effective at this point, and if so what?
To be clear, I still think this is a good way to spend money. I think the main cost is time.

paulfchristiano 27 Mar 2019 16:18 UTC
18 points
AF
in reply to: CarlShulman's comment on: More realistic tales of doom
I do agree there was a miscommunication about the end state, and that language like “lots of obvious destruction” is an understatement.
I do still endorse “military leaders might issue an order and find it is ignored” (or total collapse of society) as basically accurate and not an understatement.

paulfchristiano 27 Mar 2019 16:12 UTC
11 points
AF
in reply to: CarlShulman's comment on: More realistic tales of doom
My median outcome is that people solve intent alignment well enough to avoid catastrophe. Amongst the cases where we fail, my median outcome is that people solve enough of alignment that they can avoid the most overt failures, like literally compromising sensors and killing people (at least for a long subjective time), and can build AIs that help defend them from other AIs. That problem seems radically easier—most plausible paths to corrupting sensors involve intermediate stages with hints of corruption that could be recognized by a weaker AI (and hence generate low reward). Eventually this will break down, but it seems quite late.
very confident that no AI company would implement something with this vulnerability?
The story doesn’t depend on “no AI company” implementing something that behaves badly, it depends on people having access to AI that behaves well.
Also “very confident” seems different from “most likely failure scenario.”
Haven’t you yourself written about the failure modes of ‘do things predicted to lead to videos that people rate as acceptable’ where the attack involves surreptitiously reprogramming the camera to get optimal videos (including weird engineered videos designed to optimize on infelicities in the learned objective?
That’s a description of the problem / the behavior of the unaligned benchmark, not the most likely outcome (since I think the problem is most likely to be solved). We may have a difference in view between a distribution over outcomes that is slanted towards “everything goes well” such that the most realistic failures are the ones that are the closest calls, vs. a distribution slanted towards “everything goes badly” such that the most realistic failures are the complete and total ones where you weren’t even close.
Because it definitely seems that Vox got the impression from it that there is never a robot army takeover in the scenario, not that it’s slightly preceded by camera hacking.
I agree there is a robot takeover shortly later in objective time (mostly because of the singularity). Exactly how long it is mostly depends on how early things go off the rails w.r.t. alignment, perhaps you have O(year).

paulfchristiano 27 Mar 2019 1:53 UTC
7 points
AF
in reply to: CarlShulman's comment on: More realistic tales of doom
I agree that robot armies are an important aspect of part II.
In part I, where our only problem is specifying goals, I don’t actually think robot armies are a short-term concern. I think we can probably build systems that really do avoid killing people, e.g. by using straightforward versions of “do things that are predicted to lead to videos that people rate as acceptable,” and that at the point when things have gone off the rails those videos still look fine (and to understand that there is a deep problem at that point you need to engage with complicated facts about the situation that are beyond human comprehension, not things like “are the robots killing people?”). I’m not visualizing the case where no one does anything to try to make their AI safe, I’m imagining the most probable cases where people fail.
I think this is an important point, because I think much discussion of AI safety imagines “How can we give our AIs an objective which ensures it won’t go around killing everyone,” and I think that’s really not the important or interesting part of specifying an objective (and so leads people to be reasonably optimistic about solutions that I regard as obviously totally inadequate). I think you should only be concerned about your AI killing everyone because of inner alignment / optimization daemons.
That said, I do expect possibly-catastrophic AI to come only shortly before the singularity (in calendar time) and so the situation “humans aren’t able to steer the trajectory of society” probably gets worse pretty quickly. I assume we are on the same page here.
In that sense Part I is misleading. It describes the part of the trajectory where I think the action is, the last moments where we could have actually done something to avoid doom, but from the perspective of an onlooker that period could be pretty brief. If there is a Dyson sphere in 2050 it’s not clear that anyone really cares what happened during 2048-2049. I think the worst offender is the last sentence of Part I (“By the time we spread through the stars...”)
Part I has this focus because (i) that’s where I think the action is—by the time you have robot armies killing everyone the ship is so sailed, I think a reasonable common-sense viewpoint would acknowledge this by reacting with incredulity to the “robots kill everyone” scenario, and would correctly place the “blame” on the point where everything got completely out of control even though there weren’t actually robot armies yet (ii) the alternative visualization leads people to seriously underestimate the difficulty of the alignment problem, (iii) I was trying to describe the part of the picture which is reasonably accurate regardless of my views on the singularity.

paulfchristiano 27 Mar 2019 1:35 UTC
10 points
AF
in reply to: CarlShulman's comment on: More realistic tales of doom
The Vox article also mistakes the source of influence-seeking patterns to be about social influence rather than ’systems that try to increase in power and numbers tend to do so, so are selected for if we accidentally or intentionally produce them and don’t effectively weed them out; this is why living things are adapted to survive and expand; such desires motivate conflict with humans when power and reproduction can be obtained by conflict with humans, which can look like robot armies taking control.
Yes, I agree the Vox article made this mistake. Me saying “influence” probably gives people the wrong idea so I should change that—I’m including “controls the military” as a central example, but it’s not what comes to mind when you hear “influence.” I like “influence” more than “power” because it’s more specific, captures what we actually care about, and less likely to lead to a debate about “what is power anyway.”
In general I think the Vox article’s discussion of Part II has some problems, and the discussion of Part I is closer to the mark. (Part I is also more in line with the narrative of the article, since Part II really is more like Terminator. I’m not sure which way the causality goes here though, i.e. whether they ended up with that narrative based on misunderstandings about Part II or whether they framed Part II in a way that made it more consistent with the narrative, maybe having been inspired to write the piece based on Part I.)
There is a different mistake with the same flavor, later in the Vox article: “But eventually, the algorithms’ incentives to expand influence might start to overtake their incentives to achieve the specified goal. That, in turn, makes the AI system worse at achieving its intended goal, which increases the odds of some terrible failure”
The problem isn’t really “the AI system is worse at achieving its intended goal;” like you say, it’s that influence-seeking AI systems will eventually be in conflict with humans, and that’s bad news if AI systems are much more capable/powerful than we are.
[AI systems] wind up controlling or creating that military power and expropriating humanity (which couldn’t fight back thereafter even if unified)
Failure would presumably occur before we get to the stage of “robot army can defeat unified humanity”—failure should happen soon after it becomes possible, and there are easier ways to fail than to win a clean war. Emphasizing this may give people the wrong idea, since it makes unity and stability seem like a solution rather than a stopgap. But emphasizing the robot army seems to have a similar problem—it doesn’t really matter whether there is a literal robot army, you are in trouble anyway.

paulfchristiano 26 Mar 2019 20:01 UTC
9 points
AF
in reply to: Wei_Dai's comment on: More realistic tales of doom
I think of #3 and #5 as risk factors that compound the risks I’m describing—they are two (of many!) ways that the detailed picture could look different, but don’t change the broad outline. I think it’s particularly important to understand what failure looks like under a more “business as usual” scenario, so that people can separate objections to the existence of any risk from objections to other exacerbating factors that we are concerned about (like fast takeoff, war, people being asleep at the wheel, etc.)
I’d classify #1, #2, and #4 as different problems not related to intent alignment per se (though intent alignment may let us build AI systems that can help address these problems). I think the more general point is: if you think AI progress is likely to drive many of the biggest upcoming changes in the world, then there will be lots of risks associated with AI. Here I’m just trying to clarify what happens if we fail to solve intent alignment.

paulfchristiano 23 Mar 2019 16:11 UTC
2 points
AF
in reply to: Wei_Dai's comment on: What’s wrong with these analogies for understanding Informed Oversight and IDA?
Can you quote these examples? The word “example” appears 27 times in that post and looking at the literal second and third examples, they don’t seem very relevant to what you’ve been saying here so I wonder if you’re referring to some other examples.
Subsections “Modeling” and “Alien reasoning” of “Which C are hard to epistemically dominate?”
What I’m inferring from this (as far as a direct answer to my question) is that an overseer trying to do Informed Oversight on some ML model doesn’t need to reverse engineer the model enough to fully understand what it’s doing, only enough to make sure it’s not doing something malign, which might be a lot easier, but this isn’t quite reflected in the formal definition yet or isn’t a clear implication of it yet. Does that seem right?
You need to understand what facts the model “knows.” This isn’t value-loaded or sensitive to the notion of “malign,” but it’s still narrower than “fully understand what it’s doing.”
As a simple example, consider linear regression. I think that linear regression probably doesn’t know anything you don’t. Yet doing linear regression is a lot easier than designing a linear model by hand.
If that’s what you do, it seems “P outputs true statements just in the cases I can check.” could have a posterior that’s almost 50%, which doesn’t seem safe, especially in an iterated scheme where you have to depend on such probabilities many times?
Where did 50% come from?
Also “P outputs true statements in just the cases I check” is probably not catastrophic, it’s only catastrophic once P performs optimization in order to push the system off the rails.

paulfchristiano 20 Mar 2019 18:26 UTC
13 points
AF
on: What’s wrong with these analogies for understanding Informed Oversight and IDA?
A universal reasoner is allowed to use an intuition “because it works.” They only take on extra obligations once that intuition reflects more facts about the world which can’t be cashed out as predictions that can be confirmed on the same historical data that led us to trust the intuition.
For example, you have an extra obligation if Ramanujan has some intuition about why theorem X is true, you come to trust such intuitions by verifying them against proof of X, but the same intuitions also suggest a bunch of other facts which you can’t verify.
In that case, you can still try to be a straightforward Bayesian about it, and say “our intuition supports the general claim that process P outputs true statements;” you can then apply that regularity to trust P on some new claim even if it’s not the kind of claim you could verify, as long as “P outputs true statements” had a higher prior than “P outputs true statements just in the cases I can check.” That’s an argument that someone can give to support a conclusion, and “does process P output true statements historically?” is a subquestion you can ask during amplification.
The problem becomes hard when there are further facts that can’t be supported by this Bayesian reasoning (and therefore might undermine it). E.g. you have a problem if process P is itself a consequentialist, who outputs true statements in order to earn your trust but will eventually exploit that trust for their own advantage. In this case, the problem is that there is something going on internally inside process P that isn’t surfaced by P’s output. Epistemically dominating P requires knowing about that.
See the second and third examples in the post introducing ascription universality. There is definitely a lot of fuzziness here and it seems like one of the most important places to tighten up the definition / one of the big research questions for whether ascription universality is possible.

paulfchristiano 18 Mar 2019 15:24 UTC
11 points

in reply to: Tobias_Baumann's comment on: More realistic tales of doom
But why exactly should we expect that the problems you describe will be exacerbated in a future with powerful AI, compared to the state of contemporary human societies?
To a large extent “ML” refers to a few particular technologies that have the form “try a bunch of things and do more of what works” or “consider a bunch of things and then do the one that is predicted to work.”
That is true but I think of this as a limitation of contemporary ML approaches rather than a fundamental property of advanced AI.
I’m mostly aiming to describe what I think is in fact most likely to go wrong, I agree it’s not a general or necessary feature of AI that its comparative advantage is optimizing easy-to-measure goals.
(I do think there is some real sense in which getting over this requires “solving alignment.”)

paulfchristiano 18 Mar 2019 4:21 UTC
5 points
AF
in reply to: John_Maxwell_IV's comment on: More realistic tales of doom
I’m not mostly worried about influence-seeking behavior emerging by “specify a goal” --> “getting influence is the best way to achieve that goal.” I’m mostly worried about influence-seeking behavior emerging within a system by virtue of selection within that process (and by randomness at the lowest level).

What failure looks like

paulfchristiano

17 Mar 2019 20:18 UTC

201 points

42 comments8 min readLW link

paulfchristiano 15 Mar 2019 15:57 UTC
4 points
AF
in reply to: michaelcohen's comment on: Asymptotically Benign AGI
I don’t see why their methods would be elegant. In particular, I don’t see why any of {the anthropic update, importance weighting, updating from the choice of universal prior} would have a simple form (simpler than the simplest physics that gives rise to life).
I don’t see how MAP helps things either—doesn’t the same argument suggest that for most of the possible physics, the simplest model will be a consequentialist? (Even more broadly, for the universal prior in general, isn’t MAP basically equivalent to a random sample from the prior, since some random model happens to be slightly more compressible?)

paulfchristiano 15 Mar 2019 1:20 UTC
4 points
AF
in reply to: michaelcohen's comment on: Asymptotically Benign AGI
I say maybe ¹⁄₅ chance it’s actually dominated by consequentialists
Do you get down to 20% because you think this argument is wrong, or because you think it doesn’t apply?
What problem do you think bites you?
What’s $β$ ? Is it O(1) or really tiny? And which value of $c$ do you want to consider, polynomially small or exponentially small?
But if it somehow magically predicted which actions BoMAI was going to take in no time at all, then c would have to be above 1/d.
Wouldn’t they have to also magically predict all the stochasticity in the observations, and have a running time that grows exponentially in their log loss? Predicting what BoMAI will do seems likely to be much easier than that.

paulfchristiano 14 Mar 2019 17:04 UTC
5 points
AF
in reply to: michaelcohen's comment on: Asymptotically Benign AGI
This invalidates some of my other concerns, but also seems to mean things are incredibly weird at finite times. I suspect that you’ll want to change to something less extreme here.
(I might well be misunderstanding something, apologies in advance.)
Suppose the “intended” physics take at least 1E15 steps to run on the UTM (this is a conservative lower bound, since you have to simulate the human for the whole episode). And suppose $β < 0.999$ (I think you need $β$ much lower than this). Then the intended model gets penalized by at least exp(1E12) for its slowness.
For almost the same description complexity, I could write down physics + “precompute the predictions for the first N episodes, for every sequence of possible actions/observations, and store them in a lookup table.” This increases the complexity by a few bits, some constant plus K(N|physics), but avoids most of the computation. In order for the intended physics to win, i.e. in order for the “speed” part of the speed prior to do anything, we need the complexity of this precomputed model to be at least 1E12 bits higher than the complexity of the fast model.
That appears to happen only once N > BB(1E12). Does that seem right to you?
We could talk about whether malign consequentialists also take over at finite times (I think they probably do, since the “speed” part of the speed prior is not doing any work until after BB(1E12) steps, long after the agent becomes incredibly smart), but it seems better to adjust the scheme first.
Using the speed prior seems more reasonable, but I’d want to know which version of the speed prior and which parameters, since which particular problem bites you will depend on those choices. And maybe to save time, I’d want to first get your take on whether the proposed version is dominated by consequentialists at some finite time.

paulfchristiano 14 Mar 2019 4:50 UTC
4 points
AF
in reply to: michaelcohen's comment on: Asymptotically Benign AGI
From the formal description of the algorithm, it looks like you use a universal prior to pick $k$ , and then allow the $k^{th}$ Turing machine to run for $ℓ$ steps, but don’t penalize the running time of the machine that outputs $k$ . Is that right? That didn’t match my intuitive understanding of the algorithm, and seems like it would lead to strange outcomes, so I feel like I’m misunderstanding.

paulfchristiano 14 Mar 2019 4:48 UTC
2 points
AF
in reply to: michaelcohen's comment on: Asymptotically Benign AGI
(I actually have a more basic confusion, started a new thread.)

paulfchristiano 14 Mar 2019 4:21 UTC
2 points
AF
in reply to: michaelcohen's comment on: Asymptotically Benign AGI
(ETA: I think this discussion depended on a detail of your version of the speed prior that I misunderstood.)
Given a world model ν, which takes k computation steps per episode, let νlog be the best world-model that best approximates ν (in the sense of KL divergence) using only logk computation steps. νlog is at least as good as the “reasoning-based replacement” of ν.
The description length of νlog is within a (small) constant of the description length of ν. That way of describing it is not optimized for speed, but it presents a one-time cost, and anyone arriving at that world-model in this way is paying that cost.
To be clear, that description gets ~0 mass under the speed prior, right? A direct specification of the fast model is going to have a much higher prior than a brute force search, at least for values of $β$ large enough (or small enough, however you set it up) to rule out the alien civilization that is (probably) the shortest description without regard for computational limits.
One could consider instead νlogε, which is, among the world-models that ε-approximate ν in less than logk computation steps (if the set is non-empty), the first such world-model found by a searching procedure ψ. The description length of νlogε is within a (slightly larger) constant of the description length of ν, but the one-time computational cost is less than that of νlog.
Within this chunk of the speed prior, the question is: what are good ψ? Any reasonable specification of a consequentialist would work (plus a few more bits for it to understand its situation, though most of the work is done by handing it $ν$ ), or of a petri dish in which a consequentialist would eventually end up with influence. Do you have a concrete alternative in mind, which you think is not dominated by some consequentialist (i.e. a ψ for which every consequentialist is either slower or more complex)?