Mass_Driver

Karma: 2,819

Mass_Driver 12 Jun 2022 7:14 UTC
2 points
0
in reply to: TurnTrout’s comment on: AGI Ruin: A List of Lethalities
I suspect we’re talking about two different things.
If you just naively program a super-intelligent AI to satisfice a goal, then, sure, most of the candidate pathways to satisfice will involve accruing a lot of some type of power, because power is useful for achieving goals. That’s a valid point, and it’s important to understand that merely switching from optimizers to satisficers won’t adequately protect us against overly ambitious AIs.
However, that doesn’t mean that it’s futile to explicitly penalize most (but not literally all) of the paths that the AI could take to accumulate too much power. Suppose you adequately penalize all of the tactics that would have catastrophic side effects except for, oh, I don’t know, cornering the world market on all types of grain, because you forgot to think of that one particular path to power. Would most of the candidate plans that the AI submits for human approval then turn out to involve secretly cornering the grain market?
I don’t see why they would. All else being equal, sure, the most powerful tactic available is going to be somewhat more attractive than other, less powerful tactics. But how much more attractive? Would an AI generate 1,000x more plans that involve one specific globally disruptive tactic like cornering the grain market vs. all of the various tactics that are only locally disruptive, like publishing propaganda or hiring away competing researchers or hacking into competing research labs or interrupting the flow of electricity to those labs? 10x more such plans? 2x more such plans? I don’t think that’s the kind of estimate you can make just by gesturing at basic principles of game theory; you’d need some concrete domain knowledge about the AI’s specific planning algorithms.
If the truly dangerous plans are only 10x more common on the initial brainstorming list, then we can probably make sure those plans aren’t chosen by filtering for plans that the AI rates as safe and transparent. We can use 3 different AIs on airgapped equipment—one that only “cares” about evaluating whether a plan is safe (i.e., would the plan cause unacceptably large changes if implemented?), one that only “cares” about evaluating whether a plan is transparent (i.e., can the plan be explained to humans so that the humans will have an accurate understanding of what actions will be taken as part of the plan?), and one that generates plans that satisfice for a substantive goal (in this case, shutting down opposing AI research).
Of the plans that are approved by all 3 AIs, most of them probably won’t involve accumulating excessive amounts of power, because usually if you accumulate too much power, then that is (a) unsafe, and (b) likely to involve lying to ‘your’ human programmers.
If powerful plans are 10x more common on the initial brainstorming list, then you need filters for safety and/or transparency that are at least 90% effective at catching unacceptably powerful plans in order to get back up to a 50% chance of humanity surviving. If powerful plans are more than 10x more common, then you need a set of filters that’s more reliable than 90%. I don’t think we have any idea what those numbers are yet, but I do think it’s worth trying to reduce how common it is for excessively powerful plans to show up on the initial brainstorming list, and I think we can do that by training AIs to recognize dangerously disruptive plans and to try to avoid those types of plans. It’s better to at least try to get AIs to engage with the concept of “this plan is too disruptive” then to throw up our hands and say, “Oh, power is an attractor in game theory space, so there’s no possible way to get brilliant AIs that don’t seize infinite power.”

Mass_Driver 7 Jun 2022 6:46 UTC
6 points
2
in reply to: Rob Bensinger’s comment on: AGI Ruin: A List of Lethalities
Sure, the metaphor is strained because natural selection doesn’t have feelings, so it’s never going to feel satisfied, because it’s never going to feel anything. For whatever it’s worth, I didn’t pick that metaphor; Eliezer mentions contraception in his original post.
As I understand it, the point of bringing up contraception is to show that when you move from one level of intelligence to another, much higher level of intelligence, then the more intelligent agent can wind up optimizing for values that would be anathema to the less intelligent agents, even if the less intelligent agents have done everything they can to pass along their values. My objection to this illustration is that I don’t think anyone’s demonstrated that human goals could plausibly be described as “anathema” to natural selection. Overall, humans are pursuing a set of goals that are relatively well-aligned with natural selection’s pseudo-goals.

Mass_Driver 7 Jun 2022 6:37 UTC
5 points
0
AF
in reply to: Chris_Leong’s comment on: AGI Ruin: A List of Lethalities
One of my assumptions is that it’s possible to design a “satisficing” engine—an algorithm that generates candidate proposals for a fixed number of cycles, and then, assuming at least one proposal with estimated utility greater than X has been generated within that amount of time, selects one of the qualifying proposals at random. If there are no qualifying candidates, the AI takes no action.
If you have a straightforward optimizer that always returns the action with the highest expected utility, then, yeah, you only have to miss one “cheat” that improves “official” utility at the expense of murdering everyone everywhere and then we all die. But if you have a satisficer, then as long as some of the qualifying plans don’t kill everyone, there’s a reasonable chance that the AI will pick one of those plans. Even if you forget to explicitly penalize one of the pathways to disaster, there’s no special reason why that one pathway would show up in a large majority of the AI’s candidate plans.

Mass_Driver 6 Jun 2022 14:31 UTC
2 points
0
AF
in reply to: Chris_Leong’s comment on: AGI Ruin: A List of Lethalities
Sure, I agree! If we miss even one such action, we’re screwed. My point is that if people put enough skill and effort into trying to catch all such actions, then there is a significant chance that they’ll catch literally all the actions that are (1) world-ending and that (2) the AI actually wants to try.
There’s also a significant chance we won’t, which is quite bad and very alarming, hence people should work on AI safety.

Mass_Driver 6 Jun 2022 14:29 UTC
5 points
0
in reply to: Rob Bensinger’s comment on: AGI Ruin: A List of Lethalities
Right, I’m not claiming that AGI will do anything like straightforwardly maximize human utility. I’m claiming that if we work hard enough at teaching it to avoid disaster, it has a significant chance of avoiding disaster.
The fact that nobody is artificially mass-producing their genes is not a disaster from Darwin’s point of view; Darwin is vaguely satisfied that instead of a million humans there are now 7 billion humans. If the population stabilizes at 11 billion, that is also not a Darwinian disaster. If the population spreads across the galaxy, mostly in the form of emulations and AIs, but with even 0.001% of sentient beings maintaining some human DNA as a pet or a bit of nostalgia, that’s still way more copies of our DNA than the Neanderthals were ever going to get.
There are probably some really convincing analogies or intuition pumps somewhere that show that values are likely to be obliterated after a jump in intelligence, but I really don’t think evolution/contraception is one of those analogies.

Mass_Driver 6 Jun 2022 2:06 UTC
4 points
in reply to: FiftyTwo’s comment on: An Outside View on Less Wrong’s Advice
I think we’re doing a little better than I predicted. Rationalists seem to be somewhat better able than their peers to sift through controversial public health advice, to switch careers (or retire early) when that makes sense, to donate strategically, and to set up physical environments that meet their needs (homes, offices, etc.) even when those environments are a bit unusual. Enough rationalists got into cryptocurrency early enough and heavy enough for that to feel more like successful foresight than a lucky bet. We’re doing something at least partly right.
That said, if we really did have a craft of reliably identifying and executing better decisions, and if even a hundred people had been practicing that craft for a decade, I would expect to see a lot more obvious results than the ones I actually see. I don’t see a strong correlation between the people who spend the most time and energy engaging with the ideas you see on Less Wrong, and the people who are wealthy, or who are professionally successful, or who have happy families, or who are making great art, or who are doing great things for society (with the possible exception of AI safety, and it’s very difficult to measure whether working on AI safety is actually doing any real good).
If anything, I think the correlation might point the other way—people who are distressed or unsuccessful at life’s ordinary occupations are more likely to immerse themselves in rationalist ideas as an alternate source of meaning and status. There is something actually worth learning here, and there are actually good people here; it’s not like I would want to warn anybody away. If you’re interested in rationality, I think you should learn about it and talk about it and try to practice it. However, I also think some of us are still exaggerating the likely benefits of doing so. Less Wrong isn’t objectively the best community; it’s just one of many good communities, and it might be well-suited to your needs and quirks in particular.

Mass_Driver 6 Jun 2022 1:50 UTC
9 points
−7
AF
on: AGI Ruin: A List of Lethalities
I mostly agree with the reasoning here; thank you to Eliezer for posting it and explaining it clearly. It’s good to have all these reasons here in once place.
The one area I partly disagree with is Section B.1. As I understand it, the main point of B.1 is that we can’t guard against all of the problems that will crop up as AI grows more intelligent, because we can’t foresee all of those problems, because most of them will be “out-of-distribution,” i.e., not the kinds of problems where we have reasonable training data. A superintelligent AI will do strange things that wouldn’t have occurred to us, precisely because it’s smarter than we are, and some of those things will be dangerous enough to wipe out all human life.
I think this somewhat overstates the problem. If we tell an AI not to invent nanotechnology, not to send anything to protein labs, not to hack into all of the world’s computers, not to design weird new quantum particles, not to do 100 of the other most dangerous and weirdest things we can think of, and then ask it to generalize and learn not to do things of that sort and build avoidance of catastrophic danger as a category into its utility function...
And then we test whether the AI is actually doing these things and successfully using something like the human category of “catastrophe” when the AI is only slightly smarter than humans...
And then learn from those tests and honestly look at the failures and improve the AI’s catastrophe-avoidance skills based on what we learn...
Then the chances that that AI won’t immediately destroy the world seem to me to be much much larger than 0.1%. They’re still low, which is bad, but they’re not laughably insignificant, either, because if you make an honest, thoughtful, sustained effort to constrain the preferences of your successors, then often you at least partially succeed.
If natural selection had feelings, it might not be maximally happy with the way humans are behaving in the wake of Cro-Magnon optimization...but it probably wouldn’t call it a disaster, either. Despite the existence of contraception, there sure are a whole lot more Cro-Magnons than there ever were Neanderthals, and the population is still going up every year.
Similarly, training an AI to act responsibly isn’t going to get us a reliably safe AI, but whoever launches the first super-intelligent AI puts enough effort into that kind of training, then I don’t see any reason why we shouldn’t expect at least a 50% chance of a million or better survivors. I’m much more worried about large, powerful organizations that “vocally disdain all talk of AGI safety” than I am about the possibility that AGI safety research is inherently futile. It’s inherently imperfect in that there’s no apparent path to guaranteeing the friendliness of superintelligence...but that’s not quite the same thing as saying that we shouldn’t expect to be able to increase the probability that superintelligence is at least marginally friendly.

Mass_Driver 1 Sep 2021 22:05 UTC
4 points
on: Introduction to Reducing Goodhart
I appreciate how much detail you’ve used to lay out why you think a lack of human agency is a problem—compared to our earlier conversations, I now have a better sense of what concrete problem you’re trying to solve and why that problem might be important. I can imagine that, e.g., it’s quite difficult to tell how well you’ve fit a curve if the context in which you’re supposed to fit that curve is vulnerable to being changed in ways whose goodness or badness is difficult to specify. I look forward to reading the later posts in this sequence so that I can get a sense of exactly what technical problems are arising and how serious they are.
That said, until I see a specific technical problem that seems really threatening, I’m sticking by my opinion that it’s OK that human preferences vary with human environments, so long as (a) we have a coherent set of preferences for each individual environment, and (b) we have a coherent set of preferences about which environments we would like to be in. Right, like, in the ancestral environment I prefer to eat apples, in the modern environment I prefer to eat Doritos, and in the transhuman environment I prefer to eat simulated wafers that trigger artificial bliss. That’s fine; just make sure to check what environment I’m in before feeding me, and then select the correct food based on my environment. What do you do if you have control over my environment? No big deal, just put me in my preferred environment, which is the transhuman environment.
What happens if my preferred environment depends on the environment I’m currently inhabiting, e.g., modern me wants to migrate to the transhumanist environment, but ancestral me thinks you’re scary and just wants you to go away and leave me alone? Well, that’s an inconsistency in my preferences—but it’s no more or less problematic than any other inconsistency. If I prefer oranges when I’m holding an apple, but I prefer apples when I’m holding an orange, that’s just as annoying as the environment problem. We do need a technique for resolving problems of utility that are sensitive to initial conditions when those initial conditions appear arbitrary, but we need that technique anyway—it’s not some special feature of humans that makes that technique necessary; any beings with any type of varying preferences would need that technique in order to have their utility fully optimized.
It’s certainly worth noting that standard solutions to Goodhart’s law won’t work without modification, because human preferences vary with their environments—but at the moment such modifications seem extremely feasible to me. I don’t understand why your objections are meant to be fatal to the utility of the overall framework of Goodhart’s Law, and I hope you’ll explain that in the next post.

Mass_Driver 3 Jul 2021 15:02 UTC
2 points
in reply to: jsalvatier’s comment on: Being a teacher
Hmm. Nobody’s ever asked me to try to teach them that before, but here’s my advice:
1. Think about what dimensions or components success at the task will include. E.g., if you’re trying to play a song on the guitar, you might decide that a well-played song will have the correct chords played with the correct fingering and the correct rhythm.
2. Think about what steps are involved in each of the components of success, with an eye toward ordering those steps in terms of which steps are easiest to learn and which steps are logical prerequisites for the others. E.g., in order to learn how to play a rhythm, you first need an understanding of rhythmic concepts like beats and meters. Then, once you have a language that you can use to describe a rhythm, you need some concrete examples of rhythms, e.g., a half note followed by two quarter-notes. Then you need to translate that into the physical motions taken on the guitar, e.g., downstrokes and upstrokes with greater or lesser emphasis. Those are two different steps; first you teach the difference between a downstroke and an upstroke, and then you teach the difference between a stressed beat and an unstressed beat. You might change the order of those steps if you are working with a student who’s more comfortable with physical techniques than with language, e.g., demonstrate some rhythms first, and then only after that explain what they mean in words. In general, most values will have a vocabulary that lets you describe them, a series of examples that help you understand them, and a set of elements that constitute them; using each new word in the vocabulary and recognizing each type of example and recognizing each element and using each element is a separate step in learning the technique.
3. Leave some room at the end for integration, e.g., if you’ve learned rhythm and fingering and chords, you still need some time to practice using all three of those correctly at once. This may include learning how to make trade-offs among the various components, e.g., if you’ve got some very tricky fingering in one measure, maybe you simplify the chord to make that easier.

Mass_Driver 3 Jul 2021 13:12 UTC
3 points
on: Impossible moral problems and moral authority
I’m curious about the source of your intuition that we are obligated to make an optimal selection. You mention that the utility difference between two plausibly best meals could be large, which is true, especially when we drop the metaphor and reflect on the utility difference between two plausibly best FAI value schemes. And I suppose that, taken literally, the utilitarian code urges us to maximize utility, so leaving any utility on the table would technically violate utilitarianism.
On a practical level, though, I’m usually not in the habit of nitpicking people who do things for me that are sublimely wonderful yet still marginally short of perfect, and I try not to criticize people who made a decision that was plausibly the best available decision simply because some other decision was also plausibly the best available decision. If neither of us can tell for sure which of two options is the best, and our uncertainty isn’t of the kind that seems likely to be resolvable by further research, then my intuition is that the morally correct thing to do is just pick one and enjoy it, especially if there are other worse options that might fall upon us by default if we dither for too long.
I agree with you that a trusted moral authority figure can make it easier for us to pick one of several plausibly best options...but I disagree with you that such a figure is morally necessary; instead, I see them as useful moral support for an action that can be difficult due to a lack of willpower or self-confidence. Ideally, I would just always pick a plausibly best decision by myself; since that’s hard and I am a human being who sometimes experiences angst, it’s nice when my friends and my mom help me make hard decisions. So the role of the moral authority, in my view, isn’t that they justify a hard decision, causing it to become correct where it was not correct prior to their blessing; it’s that the moral authority eases the psychological difficulty of making a decision that was hard to accept but that was nevertheless correct even without the authority’s blessing.

Mass_Driver 1 Jul 2021 15:34 UTC
3 points
on: What am I fighting for?
Thank you for sharing this; there are several useful conceptual tools in here. I like the way you’ve found crisply different adjectives to describe different kinds of freedom, and I like the way you’re thinking about the computational costs of surplus choices.
Building on that last point a bit, I might say that a savvy agent who has already evaluated N choices could try to keep a running estimate of their expected gains from choosing the best option available after considering X more choices and then compare that gain to their cost of computing the optimal choice out of X + N options. Right, like if the utility of an arbitrary choice follows anything like a normal distribution, then as N increases, we expect U(N+X) to have tinier and tinier advantages over U(N), because N choices already cover most of the distribution, so it’s unlikely that an even better choice is available within the X additional choices you look at, and even if you do find a better choice, it’s probably only slightly better. Yet for most humans, computing the best choice out of N+X options is more costly than computing the best choice for only N options, because you start to lose track of the details of the various options you’re considering as you add more and more possibilities to the list, and the list starts to feel boring or overwhelming, so it gets harder to focus. So there’s sort of a natural stopping point where the cost of considering X additional options can be confidently predicted to outweigh the expected benefit of considering X additional options, and when you reach that point, you should stop and pick the best choice you’ve already researched.
I like having access to at least some higher-order freedoms because I enjoy the sensation of planning and working toward long-term goal, but I don’t understand why the order of a freedom is important enough to justify orienting our entire system of ethics around it. Right, like, I can imagine some extremely happy futures where everyone has stable access to dozens of high-quality choices in all areas of their lives, but, sadly, none of those choices exceed order 4, and none of them ever will. I think I’d take that future over our present and be quite grateful for the exchange. On the other hand, I can imagine some extremely dark futures where the order of choices is usually increasing for most people, because, e.g., they’re becoming steadily smarter and/or more resilient and they live in a complicated world, but they’re trapped in a kind of grindy hellscape where they have to constantly engage in that sort of long-term planning in order to purchase moderately effective relief from their otherwise constant suffering.
So I’d question whether the order of freedoms is (a) one interesting heuristic that is good to look at when considering possible futures, or (b) actually the definition of what it would mean to win. If it’s (b), I think you have some more explaining to do.

Mass_Driver 12 May 2020 21:38 UTC
8 points
on: The Technique Taboo
I agree with this post. I’d add that from what I’ve seen of medical school (and other high-status vocational programs like law school, business school, etc.), there is still a disproportionate emphasis on talking about the theory of the subject matter vs. building skill at the ultimate task. Is it helpful to memorize the names of thousands of arteries and syndromes and drugs in order to be a doctor? Of course. Is that *more* helpful than doing mock patient interviews and mock chart reviews and live exercises where you try to diagnose a tumor or a fracture or a particular kind of pus? Is it *so* much more helpful that it makes sense to spend 40x more hours on biochemistry than on clinical practice? Because my impression of medical school is that you do go on clinical rounds and do internships and things, but that the practical side of things is mostly a trial-by-fire where you are expected to improvise many of your techniques, often after seeing them demonstrated only once or twice, often with minimal supervision, and usually with little or no coaching or after-the-fact feedback. The point of the internships and residencies seems to be primarily to accomplish low-prestige medical labor, not primarily to help medical students improve their skills.
I’d be curious to hear from anyone who disagrees with me about medical school. I’m not super-confident about this assessment of medical school; I’m much more confident that an analogous critique applies well to law school and business school. Lawyers learn the theory of appellate decision-making, not how to prepare a case for trial or negotiate a settlement or draft a contract. MBAs learn economics and financial theory, not how to motivate or recruit or evaluate their employees.
As far as *why* we don’t see more discussion about how to improve technique, I think part of it is just honest ignorance. Most people aren’t very self-reflective and don’t think very much about whether they’re good at their jobs or what it means to be good at their jobs or how they could become better. Even when people do take time to reflect on what makes a good [profession], they may not have the relevant background to draw useful conclusions. Academic authorities often have little or no professional work experience; the median law professor has tried zero lawsuits; the median dean of a business school has never launched a startup; the median medical school lecturer has never worked as a primary care physician in the suburbs.
Some of it may be, as Isnasene points out, a desire to avoid unwanted competition. If people are lazy and want to enjoy high status that they earned a long time ago without putting in further effort, they might not want to encourage comparisons of skill levels.
Finally, as Isusr suggests, some of the taboo probably comes from an effort to preserve a fragile social hierarchy, but I don’t think the threat is “awareness of internal contradictions;” I think the threat is simply a common-sense idea of fairness or equity. If authorities or elites are no more objectively skillful than a typical member of their profession, then there is little reason for them to have more power, more money, or easier work. Keeping the conversation firmly fixed on discussion *about* the profession (rather than discussion about *how to do* the profession) helps obscure the fact that the status of elites is unwarranted.

Mass_Driver 10 Mar 2018 5:40 UTC
5 points
on: The abruptness of nuclear weapons
I like the style of your analysis. I think your conclusion is wrong because of wonky details about World War 2. 4 years of technical progress at anything important, delivered for free on a silver platter, would have flipped the outcome of the war. 4 years of progress in fighter airplanes means you have total air superiority and can use enemy tanks for target practice. 4 years of progress in tanks means your tanks are effectively invulnerable against their opponents, and slice through enemy divisions with ease. 4 years of progress in manufacturing means you outproduce your opponent 2:1 at the front lines each and overwhelm them with numbers. 4 years of progress in cryptography means you know your opponent’s every move and they are blind to your strategy.
Meanwhile, the kiloton bombs were only able to cripple cities “in a single mission” because nobody was watching out for them. Early nukes were so heavy that it’s doubtful whether the slow clumsy planes that carried them could have arrived at their targets against determined opposition.
There is an important sense in which fission energy is discontinuously better than chemical energy, but it’s not obvious that this translates into a discontinuity in strategic value per year of technological progress.

Mass_Driver 26 May 2017 7:51 UTC
41 points
on: Dragon Army: Theory & Charter (30min read)
1) I agree with the very high-level point that there are lots of rationalist group houses with flat / egalitarian structures, and so it might make sense to try one that’s more authoritarian to see how that works. Sincere kudos to you for forming a concrete experimental plan and discussing it in public.

2) I don’t think I’ve met you or heard of you before, and my first impression of you from your blog post is that you are very hungry for power. Like, you sound like you would really, really enjoy being the chief of a tribe, bossing people around, having people look up to you as their leader, feeling like an alpha male, etc. The main reason this makes me uncomfortable is that I don’t see you owning this desire anywhere in your long post. Like, if you had said, just once, “I think I would enjoy being a leader, and I think you might enjoy being led by me,” I would feel calmer. Instead I’m worried that you have convinced yourself that you are grudgingly stepping up as a leader because it’s necessary and no one else will. If you’re not being fully honest about your motivations for nominating yourself to be an authoritarian leader, what else are you hiding?

3) Your post has a very high ratio of detailed proposals to literature review. I would have liked to see you discuss other group houses in more detail, make reference to articles or books or blog posts about the theory of cohousing and of utopian communities more generally, or otherwise demonstrate that you have done your homework to find out what has worked, what has not worked, and why. None of your proposals sound obviously bad to me, and you’ve clearly put some thought and care into articulating them, but it’s not clear whether your proposals are backed up by research, or whether you’re just reasoning from your armchair.

4) Why should anyone follow you on an epic journey to improve their time management skills if you’re sleep-deprived and behind schedule on writing a blog post? Don’t you need to be more or less in control of your own lifestyle before you can lead others to improve theirs?

Mass_Driver 23 Dec 2016 10:08 UTC
1 point
on: Expecting Short Inferential Distances

And if you think you can explain the concept of “systematically underestimated inferential distances” briefly, in just a few words, I’ve got some sad news for you...

“I know [evolution] sounds crazy—it didn’t make sense to me at first either. I can explain how it works if you’re curious, but it will take me a long time, because it’s a complicated idea with lots of moving parts that you probably haven’t seen before. Sometimes even simple questions like ‘where did the first humans come from?’ turn out to have complicated answers.”

Mass_Driver 14 Dec 2016 1:28 UTC
2 points
in reply to: Qiaochu_Yuan’s comment on: Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality”
I am always trying to cultivate a little more sympathy for people who work hard and have good intentions! CFAR staff definitely fit in that basket. If your heart’s calling is reducing AI risk, then work on that! Despite my disappointment, I would not urge anyone who’s longing to work on reducing AI risk to put that dream aside and teach general-purpose rationality classes.

That said, I honestly believe that there is an anti-synergy between (a) cultivating rationality and (b) teaching AI researchers. I think each of those worthy goals is best pursued separately.

Mass_Driver 13 Dec 2016 8:25 UTC
4 points
in reply to: Qiaochu_Yuan’s comment on: Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality”
Yeah, that pretty much sums it up: do you think it’s more important for rationalists to focus even more heavily on AI research so that their example will sway others to prioritize FAI, or do you think it’s more important for rationalists to broaden their network so that rationalists have more examples to learn from?

Shockingly, as a lawyer who’s working on homelessness and donating to universal income experiments, I prefer a more general focus. Just as shockingly, the mathematicians and engineers who have been focusing on AI for the last several years prefer a more specialized focus. I don’t see a good way for us to resolve our disagreement, because the disagreement is rooted primarily in differences in personal identity.

I think the evidence is undeniable that rationality memes can help young, awkward engineers build a satisfying social life and increase their productivity by 10% to 20%. As an alum of one of CFAR’s first minicamps back in 2011, I’d hoped that rationality would amount to much more than that. I was looking forward to seeing rationalist tycoons, rationalist Olympians, rationalist professors, rationalist mayors, rationalist DJs. I assumed that learning how to think clearly and act accordingly would fuel a wave of conspicuous success, which would in turn attract more resources for the project of learning how to think clearly, in a rapidly expanding virtuous cycle.

Instead, five years later, we’ve got a handful of reasonably happy rationalist families, an annual holiday party, and a couple of research institutes dedicated to pursuing problems that, by definition, will provide no reliable indicia of their success until it is too late. I feel very disappointed.

Mass_Driver 13 Dec 2016 5:13 UTC
1 point
in reply to: Qiaochu_Yuan’s comment on: Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality”
Well, like I said, AI risk is a very important cause, and working on a specific problem can help focus the mind, so running a series of AI-researcher-specific rationality seminars would offer the benefit of (a) reducing AI risk, (b) improving morale, and (c) encouraging rationality researchers to test their theories using a real-world example. That’s why I think it’s a good idea for CFAR to run a series of AI-specific seminars.

What is the marginal benefit gained by moving further along the road to specialization, from “roughly half our efforts these days happen to go to running an AI research seminar series” to “our mission is to enlighten AI researchers?” The only marginal benefit I would expect is the potential for an even more rapid reduction in AI risk, caused by being able to run, e.g., 4 seminars a quarter for AI researchers, instead of 2 for AI researchers and 2 for the general public. I would expect any such potential to be seriously outweighed by the costs I describe in my main post (e.g., losing out on rationality techniques that would be invented by people who are interested in other issues), such that the marginal effect of moving from 50% specialization to 100% specialization would be to increase AI risk. That’s why I don’t want CFAR to specialize in educating AI researchers to the exclusion of all other groups.

Mass_Driver 12 Dec 2016 16:23 UTC
9 points
on: Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality”
I dislike CFAR’s new focus, and I will probably stop my modest annual donations as a result.

In my opinion, the most important benefit of cause-neutrality is that it safeguards the integrity of the young and still-evolving methods of rationality. If it is official CFAR policy that reducing AI risk is the most important cause, and CFAR staff do almost all of their work with people who are actively involved with AI risk, and then go and do almost all of their socializing with rationalists (most of whom also place a high value on reducing AI risk), then there will be an enormous temptation to discover, promote, and discuss only those methods of reasoning that support the viewpoint that reducing AI risk is the most important value. This is bad partly because it might stop CFAR from changing its mind in the face of new evidence, but mostly because the methods that CFAR will discover (and share with the world) will be stunted—students will not receive the best-available cognitive tools; they will only receive the best-available cognitive tools that encourage people to reduce AI risk. You might also lose out on discovering methods of (teaching) rationality that would only be found by people with different sorts of brains—it might turn out that the sort of people who strongly prioritize friendly AI think in certain similar ways, and if you surround yourself with only those people, then you limit yourself to learning only what those people have to teach, even if you somehow maintain perfect intellectual honesty.

Another problem with focusing exclusively on AI risk is that it is such a Black Swan-type problem that it is extremely difficult to measure progress, which in turn makes it difficult to assess the value or success of any new cognitive tools. If you work on reducing global warming, you can check the global average temperature. More importantly, so can any layperson, and you can all evaluate your success together. If you work on reducing nuclear proliferation for ten years, and you haven’t secured or prevented a single nuclear warhead, then you know you’re not doing a good job. But how do you know if you’re failing to reduce AI risk? Even if you think you have good evidence that you’re making progress, how could anyone who’s not already a technical expert possibly assess that progress? And if you propose to train all of the best experts in your methods, so that they learn to see you as a source of wisdom, then how many of them will retain the capacity to accuse you of failure?

I would not object to CFAR rolling out a new line of seminars that are specifically intended for people working on AI risk—it is a very important cause, and there’s something to be gained in working on a specific problem, and as you say, CFAR is small enough that CFAR can’t do it all. But what I hear you saying that the mission is now going to focus exclusively on reducing AI risk. I hear you saying that if all of CFAR’s top leadership is obsessed with AI risk, then the solution is not to aggressively recruit some leaders who care about other topics, but rather to just be honest about that obsession and redirect the institution’s policies accordingly. That sounds bad. I appreciate your transparency, but transparency alone won’t be enough to save the CFAR/MIRI community from the consequences of deliberately retreating into a bubble of AI researchers.

Mass_Driver 19 Feb 2016 17:03 UTC
2 points
in reply to: RomeoStevens’s comment on: Rationality Quotes Thread February 2016
Does anyone know what happened to TC Chamberlin’s proposal? In other words, shortly after 1897, did he in fact manage to spread better intellectual habits to other people? Why or why not?