Best career models for doing research?
Ideally, I’d like to save the world. One way to do that involves contributing academic research, which raises the question of what’s the most effective way of doing that.
The traditional wisdom says if you want to do research, you should get a job in a university. But for the most part the system seems to be set up so that you first spend a long time working for someone else and research their ideas, after which you can lead your own group, but then most of your time will be spent on applying for grants and other administrative trivia rather than actually researching the interesting stuff. Also, in Finland at least, all professors need to also spend time doing teaching, so that’s another time sink.
I suspect I would have more time to actually dedicate on research, and I could get doing it quicker, if I took a part-time job and did the research in my spare time. E.g. the recommended rates for a freelance journalist in Finland would allow me to spend a week each month doing work and three weeks doing research, of course assuming that I can pull off the freelance journalism part.
What (dis)advantages does this have compared to the traditional model?
Some advantages:
Can spend more time on actual research.
A lot more freedom with regard to what kind of research one can pursue.
Cleaner mental separation between money-earning job and research time (less frustration about “I could be doing research now, instead of spending time on this stupid administrative thing”).
Easier to take time off from research if feeling stressed out.
Some disadvantages:
Harder to network effectively.
Need to get around journal paywalls somehow.
Journals might be biased against freelance researchers.
Easier to take time off from research if feeling lazy.
Harder to combat akrasia.
It might actually be better to spend some time doing research under others before doing it on your own.
EDIT: Note that while I certainly do appreciate comments specific to my situation, I posted this over at LW and not Discussion because I was hoping the discussion would also be useful for others who might be considering an academic path. So feel free to also provide commentary that’s US-specific, say.
- Optimal Employment by (31 Jan 2011 12:50 UTC; 77 points)
- Verifying Rationality via RationalPoker.com by (25 Mar 2011 16:32 UTC; 45 points)
- Lesswrong Community’s How-Tos and Recommendations by (7 May 2012 13:41 UTC; 38 points)
- Optimal Employment Open Thread by (14 Feb 2011 22:49 UTC; 18 points)
- Discussion: Pathways for the Aspiring AGI Researcher? by (3 May 2011 16:48 UTC; 13 points)
- 's comment on What Should the Average EA Do About AI Alignment? by (EA Forum; 1 Mar 2017 22:11 UTC; 8 points)
- 's comment on Smart, (young), ambitious and clueless—what to do to maximize goodness? by (15 Jul 2011 7:36 UTC; 8 points)
- People who want to save the world by (15 May 2011 0:44 UTC; 5 points)
- 's comment on Impact obsession: Feeling like you never do enough good by (EA Forum; 24 Aug 2023 11:56 UTC; 4 points)
- 's comment on Best career models for doing research? by (9 Dec 2010 19:37 UTC; 3 points)
I believe that most people hoping to do independent academic research vastly underestimate both the amount of prior work done in their field of interest, and the advantages of working with other very smart and knowledgeable people. Note that it isn’t just about working with other people, but with other very smart people. That is, there is a difference between “working at a university / research institute” and “working at a top university / research institute”. (For instance, if you want to do AI research in the U.S., you probably want to be at MIT, Princeton, Carnegie Mellon, Stanford, CalTech, or UC Berkeley. I don’t know about other countries.)
Unfortunately, my general impression is that most people on LessWrong are mostly unaware of the progress made in statistical machine learning (presumably the brand of AI that most LWers care about) and cognitive science in the last 20 years (I mention these two fields because I assume they are the most popular on LW, and also because I know the most about them). And I’m not talking about impressive-looking results that dodge around the real issues, I’m talking about fundamental progress towards resolving the key problems in artificial intelligence. Anyone planning to do AI research should probably at least understand these first, and what the remaining obstacles are.
You aren’t going to understand this without doing a lot of reading, and by the time you’ve done that reading, you’ll probably have identified a research group whose work clearly reflects your personal research goals. At this point it seems like the obvious next step is to apply to work with that group as a graduate student / post doc. This circumvents the problem of having to work on research you aren’t interested in. As for other annoyances, while teaching can potentially be a time-sink, the rest of “wasted” time seems to be about publishing your work; I really find it hard to justify not publishing your work, since (a) other people need to know about it, and (b) writing up your results formally oftentimes leads to a noticeably deeper understanding than otherwise. Of course, you can waste time trying to make your results look better than they are, but this certainly isn’t a requirement and has obvious ethical issues.
EDIT: There is the eventual problem that senior professors spend more and more of their time on administrative work / providing guidance to their lab, rather than doing research themselves. But this isn’t going to be an issue until you get tenure, which is, if you do a post-doc, something like 10-15 years out from starting graduate school.
This might not even be a significant problem when the time does come around. High fluid intelligence only lasts for so long, and thus using more crystallized intelligence later on in life to guide research efforts rather than directly performing research yourself is not a bad strategy if the goal is to optimize for the actual research results.
Those are roughly my thoughts as well, although I’m afraid that I only believe this to rationalize my decision to go into academia. While the argument makes sense, there are definitely professors that express frustration with their position.
What does seem like pretty sound logic is that if you could get better results without a research group, you wouldn’t form a research group. So you probably won’t run into the problem of achieving suboptimal results from administrative overhead (you could always just hire less people), but you might run into the problem of doing work that is less fun than it could be.
Another point is that plausibly some other profession (corporate work?) would have less administrative overhead per unit of efficiency, but I don’t actually believe this to be true.
Could you point me towards some articles here? I fully admit I’m unaware of most of this progress, and would like to learn more.
A good overview would fill up a post on its own, but some relevant topics are given below. I don’t think any of it is behind a paywall, but if it is, let me know and I’ll link to another article on the same topic. In cases where I learned about the topic by word of mouth, I haven’t necessarily read the provided paper, so I can’t guarantee the quality for all of these. I generally tried to pick papers that either gave a survey of progress or solved a specific clearly interesting problem. As a result you might have to do some additional reading to understand some of the articles, but hopefully this is a good start until I get something more organized up.
Learning:
Online concept learning: rational rules for concept learning [a somewhat idealized situation but a good taste of the sorts of techniques being applied]
Learning categories: Bernoulli mixture model for document classification, spatial pyramid matching for images
Learning category hierarchies: nested Chinese restaurant process, hierarchical beta process
Learning HMMs (hidden Markov models): HDP-HMMs this is pretty new so the details haven’t been hammered out, but the article should give you a taste of how people are approaching the problem, although I also haven’t read this article; I forget where I read about HDP-HMMs, although another paper on HDPs is this one. I think the original article I read was one of Erik Sudderth’s, which are here. Another older algorithm is the Baum-Welch algorithm.
Learning image characteristics: deep Boltzmann machines
Handwriting recognition: hierarchical Bayesian approach, basically the same as the previous research
Learning graphical models: a survey paper
Planning:
Planning in MDPs: value iteration, plus LQR trees for many physical systems
Planning in POMDPs: I don’t actually know much about this; my impression is that we need to do more work in this area, but approaches include reinforcement learning. A couple interesting papers: Bayes risk approach, plus a survey of hierarchical methods
I’m not planning to do AI research, but I do like to stay no more than ~10 years out of date regarding progress in fields like this. At least at the intelligent-outsider level of understanding. So, how do I go about getting and keeping almost up-to-date in these fields. Is MacKay’s book a good place to start on machine learning? How do I get an unbiased survey of cognitive science? Are there blogs that (presuming you follow the links) can keep you up to date on what is getting a buzz?
I haven’t read MacKay myself, but it looks like it hits a lot of the relevant topics.
You might consider checking out Tom Griffiths’ website, which has a reading list as well as several tutorials.
We should try to communicate with long letters (snail mail) more. Academics seem to have done that a lot in the past. From what I have seen these exchanges seem very productive, though this could be a sampling bias. I don’t see why there aren’t more ‘personal communication’ cites, except for them possibly being frowned upon.
Why use snail mail when you can use skype? My lab director uses it regularly to talk to other researchers.
Because it is written. Which makes it good for communicating complex ideas. The tradition behind it also lends it an air of legitimacy. Researchers who don’t already have a working relationship with each other will take each other’s letters more seriously.
Upvoted for the good point about communication. Not sure I agree with the legitimacy part (what is p(Crackpot | Snail Mail) compared to p(Crackpot | Email)? I would guess higher).
What I’m now wondering is, how does using email vs. snail mail affect the probability of using green ink, or its email equivalent...
Heh you are probably right. It just seemed strange to me how researchers cannot just communicate with each other as long as they have the same research interests. My first thought was that it might have been something to do with status games, where outsiders are not allowed. I suppose some exchanges require rapid and frequent feedback. But then, like you mentioned, wouldn’t Skype do?
I’m not sure what the general case looks like, but the professors who I have worked with (who all have the characteristic that they do applied-ish research at a top research university) are both constantly barraged by more e-mails than they can possibly respond to. I suspect that as a result they limit communication to sources that they know will be fruitful.
Other professors in more theoretical fields (like pure math) don’t seem to have this problem, so I’m not sure why they don’t do what you suggest (although some of them do). And I am not sure that all professors run into the same problem as I have described, even in applied fields.
“In the past” as in before they had alternative methods of long distance communication, or after?
After several years as a post-doc I am facing a similar choice.
If I understand correctly you have no research experience so far. I’d strongly suggest completing a doctorate because:
you can use that time to network and establish a publication record
most advisors will allow you as much freedom as you can handle, particularly if you can obtain a scholarship so you are not sucking their grant money. Choose your advisor carefully.
you may well get financial support that allows you to work full time on your research for at least 4 years with minimal accountability
if you want, you can practice teaching and grant applications to taste how onerous they would really be
once you have a doctorate and some publications, it probably won’t be hard to persuade a professor to offer you an honorary (unpaid) position which gives you an institutional affiliation, library access, and maybe even a desk. Then you can go ahead with freelancing, without most of the disadvantages you cite.
You may also be able to continue as a post-doc with almost the same freedom. I have done this for 5 years. It cannot last forever, though, and the longer you go on, the more people will expect you to devote yourself to grant applications, teaching and management. That is why I’m quitting.
Huh. That’s a fascinating idea, one which had never occurred to me. I’ll have to give this suggestion serious consideration.
Ron Gross’s The Independent Scholar’s Handbook has lots of ideas like this. A lot of the details in it won’t be too useful, since it is mostly about history and the humanities, but quite a bit will be. It is also a bit old to have some more recent stuff, since there was almost no internet in 1993.
Or become a visiting professor in which you teach one or two courses a year in return for modest pay, affiliation and library access.
I’m putting the finishing touches on a future Less Wrong post about the overwhelming desirability of casually working in Australia for 1-2 years vs “whatever you were planning on doing instead”. It’s designed for intelligent people who want to earn more money, have more free time, and have a better life than they would realistically be able to get in the US or any other 1st world nation without a six-figure, part-time career… something which doesn’t exist. My world saving article was actually just a prelim for this.
Are you going to accompany the “this is cool” part with a “here’s how” part? I estimate that would cause it to influence an order of magnitude more people, by removing an inconvenience that looks at least trivial and might be greater.
I’m now thinking of why Australian readers should go to London and live in a cramped hovel in an interesting place. I feel like I’ve moved to Ankh-Morpork.
Simple! Tell them they too can follow the way of Lu-Tze, The Sweeper! For is it not said, “Don’t knock a place you’ve never been to”
As someone already living in Australia and contemplating a relocation to the US for study purposes, I would be extremely interested in this article
Come to England! It’s small, cramped and expensive! The stuff here is amazing, though.
(And the GBP is taking a battering while the AUD is riding high.)
I was under the impression that England was quite difficult to emigrate to?
My mother’s English, so I’m British by paperwork. Four-year working or study visas for Australians without a British parent are not impossible and can also be converted to a working one or even permanent residency if whatever hoops are in place at the time happen to suit.
Hope face.
Let’s see if you can beat my next 2 years in Brazil..… I’ve been hoping for something to come along (trying to defeat my status quo bias) but it has been really hard to find something comparable.
In fact, if this comment is upvoted enough, I might write a “How to be effective from wherever you are currently outside 1st world countries” post...… because if only I knew, life would be just, well, perfect. I assume many other latinos, africans, filipinos, and slavic fellows feel the same way!
Louie? I was thinking about this years ago and would love to know more details. Hurry up and post it! :)
Color me very interested!
Wow. Even the people being censored don’t know it. That’s kinda creepy!
How did you work out that it had been deleted? Just by logging out, looking and trying to remember where you had stuff posted?
I think it’s a standard tool: trollish comments look like being ignored to the trolls. But I think it’s impolite to delete comments made in good faith without notification and usable guidelines for cleaning up and reposting. (Hint hint.)
I only made one comment on the subject and I was rather confused that it was being ignored. I also knew I might have said too much about the Roko post and actually included a sentence saying that if I crossed the line I’d appreciate being told to edit it instead of having the entire thing deleted. So I just checked that one comment in particular. If other comments of mine have been deleted I wouldn’t know about it, though this was the only comment in which I have discussed the Roko post.
I doubt that this is a deliberate feature.
Consider taking a job as a database/web developer at a university department. This gets you around journal paywalls, and is a low-stress job (assuming you have or can obtain above-average coding skills) that leaves you plenty of time to do your research. (My wife has such a job.) I’m not familiar with freelance journalism at all, but I’d still guess that going the software development route is lower risk.
Some comments on your list of advantages/disadvantages:
Harder to network effectively. - I guess this depends on what kind of research you want to do. For the areas I’ve been interested in, networking does not seem to matter much (unless you count participating in online forums as networking :).
Journals might be biased against freelance researchers. - I publish my results online, informally, and somehow they’ve usually found an interested audience. Also, the journals I’m familiar with require anonymous submissions. Is this not universal?
Harder to combat akrasia. - Actually, might be easier.
A couple other advantages of the non-traditional path:
If you get bored you can switch topics easily.
I think it’s crazy to base one’s income on making research progress. How do you stay objective when you depend on your ideas being accepted as correct for food and shelter? Also, you’d be forced to pick research goals that have high probability of success (so you can publish and keep your job) instead of high expected benefit for humanity (or for your intellectual interests).
Well I guess this is our true point of disagreement. I went to the effort of finding out a lot, went to SIAI and Oxford to learn even more, and in the end I am left seriously disappointed by all this knowedge. In the end it all boils down to:
“most people are irrational, hypocritical and selfish, if you try and tell them they shoot the messenger, and if you try and do anything you bear all the costs, internalize only tiny fractions of the value created if you succeed, and you almost certainly fail to have an effect anyway. And by the way the future is an impending train wreck”
I feel quite strongly that this knowledge is not a worthy thing to have sunk 5 years of my life into getting. I don’t know, XiXiDu, you might prize such knowledge, including all the specifics of how that works out exactly.
If you really strongly value the specifics of this, then yes you probably would on net benefit from the censored knowledge, the knowledge that was never censored because I never posted it, and the knowledge that I never posted because I was never trusted with it anyway. But you still probably won’t get it, because those who hold it correctly infer that the expected value of releasing it is strongly negative from an altruist’s perspective.
The future is probably an impending train wreck. But if we can save the train, then it’ll grow wings and fly up into space while lightning flashes in the background and Dragonforce play a song about fiery battlefields or something. We’re all stuck on the train anyway, so saving it is worth a shot.
I hate to see smart people who give a shit losing to despair. This is still the most important problem and you can still contribute to fixing it.
TL;DR: I want to give you a hug.
I disagree with this argument. Pretty strongly. No selfish incentive to speak of.
So? They’re just kids!
(or)
He glanced over toward his shoulder, and said, “That matter to you?”
Caw!
He looked back up and said, “Me neither.”
I mean I guess I shouldn’t complain that you don’t find this bothers you, because you are, in fact, helping me by doing what you do and being very good at it, but that doesn’t stop it being demotivating for me! I’ll see what I can do regarding quant jobs.
I liked the first response better.
This isn’t meant as an insult, but why did it take you 5 years of dedicated effort to learn that?
Specifics. Details. The lesson of science is that details can sometimes change the overall conclusion. Also some amount of nerdyness meaning that the statements about human nature weren’t obvious to me.
That doesn’t sound right to me. Indeed, it sounds as though you are depressed :-(
Unsolicited advice over the public internet is rather unlikely to help—but maybe focus for a bit on what you want—and the specifics of how to get to there.
Upvoted for the excellent summary!
I’m curious about the “future is an impending train wreck” part. That doesn’t seem particularly accurate to me.
Maybe it will all be OK. Maybe the trains fly past each other on separate tracks. We don’t know. There sure as hell isn’t a driver though. All the inside-view evidence points to bad things,with the exception that Big Worlds could turn out nicely. Or horribly.
Perhaps try this one: The Rational Optimist: How Prosperity Evolves
Ah, you remind me of me from a while back. When I was an elementary schooler, I once replied to someone asking “would you rather be happy or right” with “how can I be happy if I can’t be right?” But these days I’ve moderated somewhat, and I feel that there is indeed knowledge that can be harmful.
The largest disadvantage to not having, essentially, an apprenticeship is the stuff you don’t learn.
Now, if you want to research something where all you need is a keen wit, and there’s not a ton of knowledge for you to pick up before you start… sure, go ahead. But those topics are few and far between. (EDIT: oh, LW-ish stuff. Meh. Sure, then, I guess. I thought you meant researching something hard >:DDDDD
No, but really, if smart people have been doing research there for 50 years and we don’t have AI, that means that “seems easy to make progress” is a dirty lie. It may mean that other people haven’t learned much to teach you, though—you should put some actual effort (get responses from at least two experts) finding out of this is the case)
Usually, an apprenticeship will teach you:
What needs to be done in your field.
How to write, publicize and present your work. The communication protocols of the community. How to access the knowledge of the community.
How to use all the necessary equipment, including the equipment that builds other equipment.
How to be properly rigorous—a hard one in most fields, you have to make it instinctual rather than just known.
The subtle tricks an experienced researcher uses to actually do research—all sorts of things you might not have noticed on your own.
And more!
Another idea is the “Bostrom Solution”, i.e. be so brilliant that you can find a rich guy to just pay for you to have your own institute at Oxford University.
Then there’s the “Reverse Bostrom Solution”: realize that you aren’t Bostrom-level brilliant, but that you could accrue enough money to pay for an institute for somebody else who is even smarter and would work on what you would have worked on. (FHI costs $400k/year, which isn’t such a huge amount as to be unattainable by Kaj or a few Kaj-like entities collaborating)
Sounds like a good bet even if you are brilliant. Make money, use money to produce academic institute, do your research in concert with academics at your institute. This solves all problems of needing to be part of academia, and also solves the problem of academics doing lots of unnecessary stuff—at your institute, academics will not be required to do unnecessary stuff.
Maybe. The disadvantage is lag time, of course. Discount rate for Singularity is very high. Assume that there are 100 years to the singularity, and that P(success) is linearly decreasing in lag time; then every second approximately 25 galaxies are lost, assuming that the entire 80 billion galaxies’ fate is decided then.
25 galaxies per second. Wow.
I’m surprised that noone has asked Roko where he got these numbers from.
Wikipedia says that there are about 80 billion galaxies in the “observable universe”, so that part is pretty straightforward. Though there’s still the question of why all of them are being counted, when most of them probably aren’t reachable with slower-than-light travel.
But I still haven’t found any explanation for the “25 galaxies per second”. Is this the rate at which the galaxies burn out? Or the rate at which something else causes them to be unreachable? Is it the number of galaxies, multiplied by the distance to the edge of the observable universe, divided by the speed of light?
calculating...
Wikipedia says that the comoving distance from Earth to the edge of the observable universe is about 14 billion parsecs (46 billion light-years short scale, i.e. 4.6 × 10^10 light years) in any direction.
Google Calculator says 80 billion galaxies / 46 billion light years = 1.73 galaxies per year, or 5.48 × 10^-8 galaxies per second
so no, that’s not it.
If I’m going to allow my mind to be blown by this number, I would like to know where the number came from.
I also took a while to understand what was meant, so here is my understanding of the meaning:
Assumptions: There will be a singularity in 100 years. If the proposed research is started now it will be a successful singularity, e.g. friendly AI. If the proposed research isn’t started by the time of the singularity, it will be a unsuccessful (negative) singularity, but still a singularity. The probability of the successful singularity linearly decreases with the time when the research starts, from 100 percent now, to 0 percent in 100 years time.
A 1 in 80 billion chance of saving 80 billion galaxies is equivalent to definitely saving 1 galaxy, and the linearly decreasing chance of a successful singularity affecting all of them is equivalent to a linearly decreasing number being affected. 25 galaxies per second is the rate of that decrease.
I meant if you divide the number of galaxies by the number of seconds to an event 100 years from now. Yes, not all reachable. Probably need to discount by an order of magnitude for reachability at lightspeed.
Hmm, by the second wikipedia link there is no basis for the 80 billion galaxies since only a relatively small fraction of the observable universe (4.2%?) is reachable if limited by the speed of light, and if not the whole universe is probably at least 10^23 times larger (by volume or by radius?).
Guh. Every now and then something reminds me of how important the Singularity is. Time to reliable life extension is measured in lives per minute, time to Singularity is measured in galaxies per second.
Now thats a way to eat up your brain.
Well conservatively assuming that each galaxy supports lives at 10^9 per sun per century (1/10th of our solar system), that’s already 10^29 lives per second right there.
And assuming utilization of all the output of the sun for living, i.e. some kind of giant spherical shell of habitable land, we can add another 12 orders of magnitude straight away. Then if we upload people that’s probably another 10 orders of magnitude.
Probably up to 10^50 lives per second, without assuming any new physics could be discovered (a dubious assumption). If instead we assume that quantum gravity gives us as much of an increase in power as going from newtonian physics to quantum mechanics did, we can pretty much slap another 20 orders of magnitude onto it, with some small probability of the answer being “infinity”.
In what I take to be a positive step towards viscerally conquering my scope neglect, I got a wave of chills reading this.
What’s your P of “the fate of all 80 billion galaxies will be decided on Earth in the next 100 years”?
About 10% (if we ignore existential risk, which is a way of resolving the ambiguity of “will be decided”). Multiply that by opportunity cost of 80 billion galaxies.
Could you please detail your working to get to this 10% number? I’m interested in how one would derive it, in detail.
I restored the question as asking about probability that we’ll be finishing an FAI project in the next 100 years. Dying of engineered virus doesn’t seem like an example of “deciding the fate of 80 billion galaxies”, although it’s determining that fate.
FAI looks really hard. Improvements in mathematical understanding to bridge comparable gaps in understanding can take at least many decades. I don’t expect a reasonable attempt at actually building a FAI anytime soon (crazy potentially world-destroying AGI projects go in the same category as engineered viruses). One possible shortcut is ems, that effectively compress the required time, but I estimate that they probably won’t be here for at least 80 more years, and then they’ll still need time to become strong enough and break the problem. (By that time, biological intelligence amplification could take over as a deciding factor, using clarity of thought instead of lots of time to think.)
My question has only a little bit to do with the probability that an AI project is successful. It has mostly to do with P(universe goes to waste | AI projects are unsuccessful). For instance, couldn’t the universe go on generating human utility after humans go extinct?
How? By coincidence?
(I’m assuming you also mean no posthumans, if humans go extinct and AI is unsuccessful.)
Aliens. I would be pleased to learn that something amazing was happening (or was going to happen, long “after” I was dead) in one of those galaxies. Since it’s quite likely that something amazing is happening in one of those 80 billion galaxies, shouldn’t I be pleased even without learning about it?
Of course, I would be correspondingly distressed to learn that something horrible was happening in one of those galaxies.
Some complexities regarding “decided” since physics is deterministic, but hand waving that aside, I’d say 50%.
With high probability, many of those galaxies are already populated. Is that irrelevant?
I disagree. I claim that the probability of >50% of the universe being already populated (using the space of simultaneity defined by a frame of reference comoving with earth) is maybe 10%.
“Already populated” is a red herring. What’s the probability that >50% of the universe will ever be populated? I don’t see any reason for it to be sensitive to how well things go on Earth in the next 100 years.
I think it is likely that we are the only spontaneously-created intelligent species in the entire 4-manifold that is the universe, space and time included (excluding species which we might create in the future, of course).
I’m curious to know how likely, and why. But do you agree that aliens are relevant to evaluating astronomical waste?
That seems contrary to the http://en.wikipedia.org/wiki/Self-Indication_Assumption
Do you have a critique—or a supporting argument?
Yes, I have a critique. Most of anthropics is gibberish. Until someone makes anthropics work, I refuse to update on any of it. (Apart from the bits that are commonsensical enough to derive without knowing about “anthropics”, e.g. that if your fising net has holes 2 inches big, don’t expect to catch fish smaller then 2 inches wide)
I don’t think you can really avoid anthropic ideas—or the universe stops making sense. Some anthropic ideas can be challenging—but I think we have got to try.
Anyway, you did the critique—but didn’t go for a supporting argument. I can’t think of very much that you could say. We don’t have very much idea yet about what’s out there—and claims to know such things just seem over-confident.
Basically Rare Earth seems to me to be the only tenable solution to Fermi’s paradox.
Fermi’s paradox implying no aliens surely applies within-galaxy only. Many galaxies are distant, and intelligent life forming there concurrently (or long before us) is quite compatible with it not having arrived on our doorsteps yet—due to the speed of light limitation.
If you think we should be able to at least see life in distant galaxies, then, in short, not really—or at least we don’t know enough to say yea or nay on that issue with any confidence yet.
The Andromeda Galaxy is 2.5 million light-years away. The universe is about 1250 million years old. Therefore that’s not far enough away to protect us from colonizing aliens travelling at 0.5c or above.
The universe is about 13,750 million years old. The Fermi argument suggests that—if there were intelligent aliens in this galaxy, they should probably have filled it by now—unless they originated very close to us in time—which seems unlikely. The argument applies much more weakly to galaxies, because they are much further away, and they are separated from each other by huge regions of empty space. Also, the Andromeda Galaxy is just one galaxy. Say only one galaxy in 100 has intelligent life—and the Andromeda Galaxy isn’t among them. That bumps the required distance to be travelled up to 10 million light years or so.
Even within this galaxy, the Fermi argument is not that strong. Maybe intelligent aliens formed in the last billion years, and haven’t made it here yet—because space travel is tricky, and 0.1c is about the limit. The universe is only about 14 billion years old. For some of of that there were not too many second generations stars. The odds are against there being aliens nearby—but they are not that heavily stacked. For other galaxies, the argument is much, much less compelling.
There are strained applications of anthropics, like the doomsday argument. “What happened here might happen elsewhere” is much more innocuous.
There are some more practical and harmless applications as well. In Nick Bostrom’s Anthropic Bias, for example, there is an application of the Self-Sampling Assumption to traffic analysis.
Bostrom says: “Cars in the next lane really do go faster”
I agree.
Even Nick Bostrom, who is arguably the leading expert on anthropic problems, rejects SIA for a number of reasons (see his book Anthropic Bias). That alone is a pretty big blow to its credibility.
That is curious. Anyway, the self-indication assumption seems fairly straight-forwards (as much as any anthropic reasoning is, anyway). The critical material from Bostrom on the topic I have read seems unpersuasive. He doesn’t seem to “get” the motivation for the idea in the first place.
If you think there is a significant probability that an intelligence explosion is possible or likely, then that question is sensitive to how well things go on Earth in the next 100 years.
However likely they are, I expect intelligence explosions to be evenly distributed through space and time. If 100 years from now Earth loses by a hair, there are still plenty of folks around the universe who will win or have won by a hair. They’ll make whatever use of the 80 billion galaxies that they can—will they be wasting them?
If Earth wins by a hair, or by a lot, we’ll be competing with those folks. This also significantly reduces the opportunity cost Roko was referring to.
That seems like a rather exaggerated sense of importance. It may be a fun fantasy in which the fate of the entire universe hangs in the balance in the next century—but do bear in mind the disconnect between that and the real world.
Out of curiosity: what evidence would convince you that the fate of the entire universe does hang in the balance?
No human-comparable aliens, for one.
Which seems awfully unlikely, the more we learn about solar systems.
“Convince me”—with some unspecified level of confidence? That is not a great question :-|
We lack knowlegde of the existence (or non-existence) of aliens in other galaxies. Until we have such knowledge, our uncertainty on this matter will necessarily be high—and we should not be “convinced” of anything.
What evidence would convince you, with 95% confidence, that the fate of the universe hangs in the balance in this next century on Earth?
You may specify evidence such as “strong evidence that we are completely alone in the universe” even if you think it is unlikely we will get such evidence.
I did get the gist of your question the first time—and answered according. The question takes us far into counter-factual territory, though.
I was just curious to see if you rejected the fantasy on principle, or if you had other reasons.
Unfortunately, FHI seems to have filled the vacancies it advertised earlier this month.
Are you talking about these? (http://www.fhi.ox.ac.uk/news/2010/vacancies) This seems odd, the deadline for applications is on Jan 12th.
Oh yes—strange, I swear it said no vacancies...
Sure, so this favors the “Create a new James Martin” strategy.
Most people wouldn’t dispute the first half of your comment. What they might take issue with is this:
The problem is that we have to defer to Eliezer’s (and, by extension, SIAI’s) judgment on such issues. Many of the commenters here think that this is not only bad PR for them, but also a questionable policy for a “community blog devoted to refining the art of human rationality.”
If you are going to quote and respond to that sentence, which anticipates people objecting to trusting Eliezer to make those judgments, you should also quote and repond to my response to that anticipation (ie, the next sentence):
Also, I am getting tired of objections framed as predictions that others would make the objections. It is possible to have a reasonable discussion with people who put forth their own objections, explain their own true rejections, and update their own beleifs. But when you are presenting the objections you predict others will make, it is much harder, even if you are personally convinced, to predict that these nebulous others will also be persuaded by my response. So please, stick your own neck out if you want to complain about this.
That’s definitely a fair objection, and I’ll answer: I personally trust Eliezer’s honesty, and he is obviously much smarter than myself. However, that doesn’t mean that he’s always right, and it doesn’t mean that we should trust his judgment on an issue until it has been discussed thoroughly.
I agree. The above paragraph is my objection.
The problem with a public thorough discussion in these cases is that once you understand the reasons why the idea is dangerous, you already know it, and don’t have the opportunity to choose whether to learn about it.
If you trust Eliezer’s honesty, then though he may make mistakes, you should not expect him to use this policy as a cover for banning posts as part of some hidden agenda.
That’s definitely the root of the problem. In general, though, if we are talking about FAI, then there shouldn’t be a dangerous idea. If there is, then it means we are doing something wrong.
I don’t think he’s got a hidden agenda; I’m concerned about his mistakes. Though I’m not astute enough to point them out, I think the LW community as a whole is.
I have a response to this that I don’t actually want to say, because it could make the idea more dangerous to those who have heard about it but are currently safe due to not fully understanding it. I find that predicting that this sort of thing will happen makes me reluctant to discuss this issue, which may explain why of those who are talking about it, most seem to think the banning was wrong.
Given that there has been one banned post. I think that his mistakes are much less of a problem than overwrought concern about his mistakes.
If you have a reply, please PM me. I’m interested in hearing it.
Are you interested in hearing it if it does give you a better understanding of the dangerous idea that you then realize is in fact dangerous?
It may not matter anymore, but yes, I would still like to hear it.
In this case, the same point has been made by others in this thread.
Why do you believe that? FAI is full of potential for dangerous ideas. In its full development, it’s an idea with the power to rewrite 100 billion galaxies. That’s gotta be dangerous.
Let me try to rephrase: correct FAI theory shouldn’t have dangerous ideas. If we find that the current version does have dangerous ideas, then this suggests that we are on the wrong track. The “Friendly” in “Friendly AI” should mean friendly.
Pretty much correct in this case. Roko’s original post was, in fact, wrong; correctly programmed FAIs should not be a threat.
(FAIs shouldn’t be a threat, but a theory to create a FAI will obviously have at least potential to be used to create uFAIs. FAI theory will have plenty of dangerous ideas.)
I want to highlight at this point how you think about similar scenarios:
That isn’t very reassuring. I believe that if you had the choice of either letting a Paperclip maximizer burn the cosmic commons or torture 100 people, you’d choose to torture 100 people. Wouldn’t you?
They are always a threat to some beings. For example beings who oppose CEV or other AI’s. Any FAI who would run a human version of CEV would be a potential existential risk to any alien civilisation. If you accept all this possible oppression in the name of what is subjectively friendliness, how can I be sure that you don’t favor torture for some humans that support CEV, in order to ensure it? After all you already allow for the possibility that many beings are being oppressed or possible killed.
This seems to be true and obviously so.
Narrowness. You can parry almost any statement like this, by posing a context outside its domain of applicability.
Another pointless flamewar. This part makes me curious though:
There are two ways I can interpret your statement:
a) you know a lot more about decision theory than you’ve disclosed so far (here, in the workshop and elsewhere);
b) you don’t have that advanced knowledge, but won’t accept as “correct” any decision theory that leads to unpalatable consequences like Roko’s scenario.
Which is it?
From my point of view, and as I discussed in the post (this discussion got banned with the rest, although it’s not exactly on that topic), the problem here is the notion of “blackmail”. I don’t know how to formally distinguish that from any other kind of bargaining, and the way in which Roko’s post could be wrong that I remember required this distinction to be made (it could be wrong in other ways, but that I didn’t notice at the time and don’t care to revisit).
(The actual content edited out and posted as a top-level post.)
(I seem to have a talent for writing stuff, then deleting it, and then getting interesting replies. Okay. Let it stay as a little inference exercise for onlookers! And please nobody think that my comment contained interesting secret stuff; it was just a dumb question to Eliezer that I deleted myself, because I figured out on my own what his answer would be.)
Thanks for verbalizing the problems with “blackmail”. I’ve been thinking about these issues in the exact same way, but made no progress and never cared enough to write it up.
Perhaps the reason you are having trouble coming up with a satisfactory characterization of blackmail is that you want a definition with the consequence that it is rational to resist blackmail and therefore not rational to engage in blackmail.
Pleasant though this might be, I fear the universe is not so accomodating.
Elsewhere VN asks how to unpack the notion of a status-quo, and tries to characterize blackmail as a threat which forces the recipient to accept less utility than she would have received in the status quo. I don’t see any reason in game theory why such threats should be treated any differently than other threats. But it is easy enough to define the ‘status-quo’.
The status quo is the solution to a modified game—modified in such a way that the time between moves increases toward infinity and the current significance of those future moves (be they retaliations or compensations) is discounted toward zero. A player who lives in the present and doesn’t respond to delayed gratification or delayed punishment is pretty much immune to threats (and to promises).
On RW it’s called Headless Chicken Mode, when the community appears to go nuts for a time. It generally resolves itself once people have the yelling out of their system.
The trick is not to make any decisions based on the fact that things have gone into headless chicken mode. It’ll pass.
[The comment this is in reply to was innocently deleted by the poster, but not before I made this comment. However, I think I’m making a useful point here, so would prefer to keep this comment.]
This is certainly the case with regard to the kind of decision theoretic thing in Roko’s deleted post. I’m not sure if it is the case with all ideas that might come up while discussing FAI.
Wrong and stupid.
FYI, this is an excellent example of contempt.
And so it was, but not an example for other times when it wasn’t. A rare occurrence. I’m pretty sure it didn’t lead to any errors though, in this simple case.
(I wonder why Eliezer pitched in the way he did, with only weak disambiguation between the content of Tetronian’s comment and commentary on correctness of Roko’s post.)
I got the impression that you responded to “FAI Theory” as our theorizing and Eliezer responded to it as the theory making its way to the eventual FAI.
Ok...but why?
Edit: If you don’t want to say why publicly, feel free to PM me.
here
The above deleted comment referenced some details of the banned post. With those details removed, it said:
It was forbidden for being a Langford Basilisk.
Strange LessWrong software fact: this showed up in my reply stream as a comment consisting only of a dot (”.”), though it appears to be a reply to a reply to me.
It also shows up on my user page as a dot. Before I edited it to be just a dot, it showed up in your comment stream and my user page with the original complete content.
An important academic option: get tenure at a less reputable school. In the States at least there are tons of universities that don’t really have huge research responsibilities (so you won’t need to worry about pushing out worthless papers, preparing for conferences, peer reviewing, etc), and also don’t have huge teaching loads. Once you get tenure you can cruise while focusing on research you think matters.
The down side is that you won’t be able to network quite as effectively as if you were at a more prestigious university and the pay isn’t quite as good.
Don’t forget about the ridiculous levels of teaching you’re responsible for in that situation. Lots worse than at an elite institution.
Not necessarily. I’m not referring to no-research universities, which do have much higher teaching loads (although still not ridiculous. Teaching 3 or 4 classes a semester is hardly strenuous). I’m referring to research universities that aren’t in the top 100, but which still push out graduate students.
My undegrad Alma Mater, Kansas University, for instance. Professors teach 1 or 2 classes a semester, with TA support (really, when you have TAs, teaching is not real work). They are still expected to do research, but the pressure is much less than at a top 50 school.
I pointed out to Roko by PM that his comment couldn’t be doing his cause any favors, but did not ask him to delete it, and would have discouraged him from doing so.
I can’t be sure, but it sounded from:
like he’d gotten a stronger message from someone high up in SIAI—though of course, I probably like that theory because of the Bayesian Conspiracy aspects.
Would you mind PM’ing me (or just posting) the message you sent?
Also, does the above fit with your experiences at SIAI? I find it hard, but not impossible, to believe that Roko just described something akin to standard hiring procedure, and would very much like to hear an inside (and presumably saner) account.
Most people who actually work full-time for SIAI are too busy to read every comments thread on LW. In some cases, they barely read it at all. The wacky speculation here about SIAI is very odd—a simple visit in most cases would eliminate the need for it. Surely more than a hundred people have visited our facilities in the last few years, so plenty of people know what we’re really like in person. Not very insane or fanatical or controlling or whatever generates a good comic book narrative.
PMed the message I sent.
Certainly not anything like standard hiring procedure.
Thanks Nick.
Please pardon my prying, but as you’ve spent more time with SIAI, have you seen tendencies toward this sort of thing? Public declarations, competitions/pressure to prove devotion to reducing existential risks, scolding for not towing the party line, etc.
I’ve seen evidence of fanaticism, but have always been confused about what the source is (did they start that way, or were they molded?).
Basically, I would very much like to know what your experience has been as you’ve gotten closer to SIAI.
I’m sure I’m not the only (past, perhaps future) donor would appreciate the air being cleared about this.
No problem, and I welcome more such questions.
No; if anything, I see explicit advocacy, as Carl describes, against natural emergent fanaticism (see below), and people becoming less fanatical to the extent that they’re influenced by group norms. I don’t see emergent individual fanaticism generating significant unhealthy group dynamics like these. I do see understanding and advocacy of indirect utilitarianism as the proper way to ‘shut up and multiply’. I would be surprised if I saw any of the specific things you mention clearly going on, unless non-manipulatively advising people on how to live up to ideals they’ve already endorsed counts. I and others have at times felt uncomfortable pressure to be more altruistic, but this is mostly pressure on oneself — having more to do with personal fanaticism and guilt than group dynamics, let alone deliberate manipulation — and creating a sense of pressure is generally recognized as harmful.
I think the major source is that self-selection for taking the Singularity seriously, and for trying to do something about it, selects for bullet-biting dispositions that predispose towards fanaticism, which is then enabled by having a cause and a group to identify with. I don’t think this is qualitatively different from things that happen in other altruistic causes, just more common in SIAI due to much stronger selective pressure for bullet-biting.
I also have the impression that Singularitarian fanaticism in online discussions is more common among non-affiliated well-wishers than people who have spent time with SIAI (but there are more of the former category, so it’s not easy to tell).
I was there for a summer and don’t think I was ever even asked to donate money.
Ahh. I was trying to ask about Cialdini-style influence techniques.
Very little, if any.
What exactly is Roko’s cause by your estimation? I wasn’t aware he had one, at least in the secretive sense.
I meant SIAI.
This depends on the field, university, and maybe country. In many cases, doing your own research is the main focus from graduate school on. At research universities in the US, at least, doing research is a professor’s main job—although they do also have to do some teaching, apply for grants, and so on, professors are primarily judged by their publication record. In graduate school, many students get to work on their own research projects. A common model is: a professor who has some areas of interest & expertise gets graduate students who are interested in doing research in those areas. At first the students might work primarily on the professor’s projects, especially if they don’t have research ideas of their own yet, but during their time at grad school a student is expected to develop their own research ideas (within the same general area) and do their own projects, with guidance from the professor so that they can learn to do it well.
I think the academic route should work pretty well if you’re interested in topics that are an established part of an academic field. If you’re interested in an unusual topic that is not so well established, then you need to look and see if you’ll be able to make academia work. Will you be able to get articles about that topic published in academic journals? Can you find a grad school, and then a university job, where they will support & encourage your research on that topic?
If you can find any published articles related to the topic then that’s a starting point. Then I’d make a list of every researcher in the field who is interested in the topic, starting with the authors of published articles. Then look into all the grad students who have worked with those researchers, follow citation paths, and so on. You can get a decent sense of what academia might be like for you based on publicly available info (those researchers’ websites, their lists of publications, and so on), and then you can contact them for more info. If you do go to grad school, you might go to one of their universities, or to a university that they recommended.
I don’t understand your reasoning here—bad events don’t get a “flawless victory” badness bonus for being guaranteed. A 100% chance of something bad isn’t much worse than a 90% chance.
I said that I wouldn’t want to know it if a bad outcome was guaranteed. But if it would make a bad outcome possible, but very-very-unlikely to actually occur, then the utility I assign to knowing the truth would outweigh the very unlikely possibility of something bad happening.
No, dude, you’re wrong
The compelling argument for me is that knowing about bad things is useful to the extent that you can do something about them, and it turns out that people who don’t know anything (call them “non-cogniscenti”) will probably free-ride their way to any benefits of action on the collective-action-problem that is the at issue here, whilst avoiding drawing any particular attention to themselves ==> avoiding the risks.
Vladimir Nesov doubts this prima facie, i.e. he asks “how do you know that the strategy of being a completely inert player is best?”.
-- to which I answer, “if you want to be the first monkey shot into space, then good luck” ;D
This is the “collective-action-problem”—where the end of the world arrives—unless a select band of heroic messiahs arrive and transport everyone to heaven...?
That seems like a fantasy story designed to manipulate—I would council not getting sucked in.
No, this is the “collective-action-problem”—where the end of the world arrives—despite a select band of decidedly amateurish messiahs arriving and failing to accomplish anything significant.
You are looking at those amateurs now.
The END OF THE WORLD is probably the most frequently-repeated failed prediction of all time. Humans are doing spectacularly well—and the world is showing many signs of material and moral progress—all of which makes the apocalypse unlikely.
The reason for the interest here seems obvious—the Singularity Institute’s funding is derived largely from donors who think it can help to SAVE THE WORLD. The world must first be at risk to enable heroic Messiahs to rescue everyone.
The most frequently-cited projected cause of the apocalypse: an engineering screw-up. Supposedly, future engineers are going to be so incompetent that they accidentally destroy the whole world. The main idea—as far as I can tell—is that a bug is going to destroy civilisation.
Also—as far as I can tell—this isn’t the conclusion of analysis performed on previous engineering failures—or on the effects of previous bugs—but rather is wild extrapolation and guesswork.
Of course it is true that there may be a disaster, and END OF THE WORLD might arrive. However there is no credible evidence that this is likely to be a probable outcome. Instead, what we have appears to be mostly a bunch of fear mongering used for fundraising aimed at fighting the threat. That gets us into the whole area of the use and effects of fear mongering.
Fearmongering is a common means of psychological manipulation, used frequently by advertisers and marketers to produce irrational behaviour in their victims.
It has been particularly widely used in the IT industry—mainly in the form of fear, uncertainty and doubt.
Evidently, prolonged and widespread use is likely to help to produce a culture of fear. The long-term effects of that are not terribly clear—but it seems to be dubious territory.
I would council those using fear mongering for fund-raising purposes to be especially cautious of the harm this might do. It seems like a potentially dangerous form of meme warfare. Fear targets circuits in the human brain that evolved in an earlier, more dangerous era—where death was much more likely—so humans have an evolved vulnerability in the area. The modern super-stimulus of the END OF THE WORLD overloads those vulnerable circuits.
Maybe this is an effective way of extracting money from people—but also, maybe it is an unpleasant and unethical one. So, wannabe heroic Messiahs, please: take care. Starting out by screwing over your friends and associates by messing up their heads with a hostile and virulent meme complex may not be the greatest way to start out.
Do you also think that global warming is a hoax, that nuclear weapons were never really that dangerous, and that the whole concept of existential risks is basically a self-serving delusion?
Also, why are the folks that you disagree with the only ones that get to be described with all-caps narrative tropes? Aren’t you THE LONE SANE MAN who’s MAKING A DESPERATE EFFORT to EXPOSE THE TRUTH about FALSE MESSIAHS and the LIES OF CORRUPT LEADERS and SHOW THE WAY to their HORDES OF MINDLESS FOLLOWERS to AN ENLIGHTENED FUTURE? Can’t you describe anything with all-caps narrative tropes if you want?
Not rhethorical questions, I’d actually like to read your answers.
I laughed aloud upon reading this comment; thanks for lifting my mood.
Tim on global warming: http://timtyler.org/end_the_ice_age/
1-line summary—I am not too worried about that either.
Global warming is far more the subject of irrational fear-mongering then machine intelligence is.
It’s hard to judge how at risk the world was from nuclear weapons during the cold war. I don’t have privileged information about that. After Japan, we have not had nuclear weapons used in anger or war. That doesn’t give much in the way of actual statistics to go on. Whatever estimate is best, confidence intervals would have to be wide. Perhaps ask an expert on the history of the era this question.
The END OF THE WORLD is not necessarily an idea that benefits those who embrace it. If you consider the stereotypical END OF THE WORLD plackard carrier, they are probably not benefitting very much personally. The benefit associated with the behaviour accrues mostly to the END OF THE WORLD meme itself. However, obviously, there are some people who benefit. 2012 - and all that.
The probabality of the END OF THE WORLD soon—if it is spelled out exactly what is meant by that—is a real number which could be scientifically investigated. However whether the usual fundraising and marketing campaigns around the subject illuminate that subject more than they systematically distort it seems debatable.
This is a pretty optimistic way of looking at it, but unfortunately it’s quite unfounded. Current scientific consensus is that we’ve already released more than enough greenhouse gases to avert the next glacial period. Melting the ice sheets and thus ending the ice age entirely is an extremely bad idea if we do it too quickly for global ecosystems to adapt.
We don’t even really understand what causes the glacial cycles yet. This is an area where there are multiple competing hypotheses. I list four of these on my site. So, since we don’t have a proper understanding of the mechanics involved with much confidence yet, we don’t yet know what it would take to prevent them.
Here’s what Dyson says on the topic:
I do not believe this is contrary to any “scientific consensus” on the topic. Where is this supposed “scientific consensus” of which you speak?
Melting the ice caps is inevitably an extremely slow process—due to thermal inertia. It is also widely thought to be a runaway positive feedback cycle—and so probably a phenomenon that it would be difficult to control the rate of.
Melting of the icecaps is now confirmed to be a runaway positive feedback process pretty much beyond a shadow of a doubt. Within the last few years, melting has occurred at a rate that exceeded the upper limits of our projection margins.
Have you performed calculations on what it would take to avert the next glacial period on the basis of any of the competing models, or did you just assume that ice ages are bad, so preventing them is good and we should thus work hard to prevent reglaciation? There’s a reason why your site is the first and possibly only only result in online searches for support of preventing glaciation, and it’s not because you’re the only one to think of it
There are others who share my views—e.g.:
http://www.theregister.co.uk/2007/08/14/freeman_dyson_climate_heresies/
http://www.guardian.co.uk/environment/2002/dec/05/comment.climatechange
http://www.stanford.edu/~moore/Boon_To_Man.html
http://www.telegraph.co.uk/news/uknews/1563054/Global-warming-is-good-and-is-not-our-fault.html
Why is being difficult to control glacial melting a point in the favor of increasing greenhouse gas emissions?
It’s true that climate change models are limited in their ability to project climate change accurately, although they’re getting better all the time. Unfortunately, the evidence currently suggests that they’re undershooting actual warming rates even at their upper limits.
The pro-warming arguments on your site essentially boil down to “warm earth is better than cold earth, so we should try to warm the earth up.” Regardless of the relative merits of a warmer or colder planet though, rapid change of climate is a major burden on ecosystems. Flooding and forest fires are relatively trivial effects, it’s mass extinction events that are a real matter of concern.
That is hard to parse. You are asking why I think the rate of runaway positive feedback cycles is difficult to control? That is because that is often their nature.
You talk as though I am denying warming is happening. HUH?
Right. So, if you want a stable climate, you need to end the yo-yo glacial cycles—and end the ice age. A stable climate is one of the benefits of doing that.
I have a section entitled “Climate stablity” in my essay. To quote from it:
http://timtyler.org/end_the_ice_age/
I have no idea how you got that out of my question. It’s obvious why runaway positive feedback cycles would be hard to control, the question I asked is why this in any way supports global warming not being dangerous.
That was not something I meant to imply. My point is that you seem to have decided that it’s better for our earth to be warm than cold, and thus that it’s good to approach that state, but not done any investigation into whether what we’re doing is a safe means of accomplishing that end; rather you seem to have assumed that we cannot do too much.
Most of the species on earth today have survived through multiple glaciation periods. Our ecosystems have that plasticity, because those species that were not able to cope with the rapid cooling periods died out. Global warming could lead to a stable climate, but it’s also liable to cause massive extinction in the process as climate zones shift in ways that they haven’t in millions of years far at a rate outside the tolerances of many ecosystems.
When it comes to global climate, there are really no “better” or “worse” states. Species adapt to the way things are. Cretaceous organisms are adapted to Cretaceous climates, Cenozoic organisms are adapted to Cenozoic climates, and either would have problems dealing with the other’s climate. Humans more often suffer problems from being too cold than too hot, but we’ve scarcely had time to evolve since we left near-equatorial climates. We’re adapted to be comfortable in hotter climates than the ones in which most people live today, but the species we rely on are mostly adapted to deal with the climates they’re actually in, with cooling periods lying within the tolerances of ecosystems that have been forced to deal with them recently in their evolutionary history.
There most certainly are—from the perspective of individuals, groups, or species.
From the perspective of species, “better,” is generally “maintain ecosystem status quo” and “worse” is everything else, except for cases where they come out ahead due to competitors suffering more heavily from the changes.
For most possible changes, a good rule of thumb is on average that half the agents affected do better than average, and half the agents affected do worse than average.
Fitness is relative—and that’s just what it means to consider an average value.
I go into all this in more detail on: http://timtyler.org/why_everything_is_controversial/
Roughly half of agents may have a better than average response to the change, but when rapid ecosystem changes occur, the average species response is negative. Particularly when accompanied by other forms of ecosystem pressure (which humanity is certainly exerting) rapid changes in climate tend to be accompanied by extinction spikes and decreases in species diversity.
I am not sure I am following. You are saying that such changes are bad—because they drive species towards extinction?
If you look at: http://alife.co.uk/essays/engineered_future/
...you will see that I expect the current mass extinction to intensify tremendously. However, I am not clear about how or why that would be bad. Surely it is a near-inevitable result of progress.
Rapid change drives species to extinction at a rate liable to endanger the function of ecosystems we rely on. Massive extinction events are in no way an inevitable consequence of improving the livelihoods of humans, although I’m not optimistic about our prospects of actually avoiding them.
Loss of a large percentage of the species on earth would hurt us, both in practical terms and as a matter of widely shared preference. As a species, we would almost certainly survive anthropogenic climate change even if it caused a runaway mass extinction event, but that doesn’t mean that it’s not an outcome that would be better to avoid if possible. Frankly, I don’t expect legislation or social agitation ever to have an adequate impact in halting anthropogenic global warming; unless we come up with some really clever hack, the battle is going to be lost, but that doesn’t mean that we shouldn’t be aware of what we stand to lose, and take notice if any viable means of avoiding it arises.
The argument suggesting that we should move away from the “cliff edge” of reglaciation is that it is dangerous hanging around there—and we really don’t want to fall off.
You seem to be saying that we should be cautious about moving too fast—in case we break something. Very well, I agree entirely—so: let us study the whole issue while moving as rapidly away from the danger zone as we feel is reasonably safe.
As I already noted, as best indicated by our calculations we have already overshot the goal of preventing the next glaciation period. Moving away from the danger zone at a reasonably safe pace would mean a major reduction in greenhouse gas emissions.
We don’t know that. The science of this isn’t settled. The Milankovitch hypothesis of glaciation is more band-aid than theory. See:
http://en.wikipedia.org/wiki/Milankovitch_cycles#Problems
CO2 apparently helps—but even that is uncertain. I would want to see a very convincing case that we are far enough from the edge for the risk of reglaciation to be over before advocating hanging around on the reglaciation cliff-edge. Short of eliminating the ice caps, it is difficult to imagine what would be convincing. Those ice caps are potentially major bad news for life on the planet—and some industrial CO2 is little reassurance—since that could relatively quickly become trapped inside plants and then buried.
The global ice caps have been around for millions of years now. Life on earth is adapted to climates that sustain them. They do not constitute “major bad news for life on this planet.” Reglaciation would pose problems for human civilization, but the onset of glaciation occurs at a much slower rate than the warming we’re already subjecting the planet to, and as such even if raising CO2 levels above what they’ve been since before the glaciations began in the Pleistocene were not enough to prevent the next round, it would still be a less pressing issue.
On a geological time scale, the amount of CO2 we’ve released could quickly be trapped in plants and buried, but with the state of human civilization as it is, how do you suppose that would actually happen quickly enough to be meaningful for the purposes of this discussion?
The ice age is a pretty major problem for the planet. Huge ice sheets obliterate most life on the northern hemisphere continents every 100 thousand years or so.
Re: reglaciation being slow—the last reglaciation looked slower than the last melt. The one before that happened at about the same speed. However, they both look like runaway positive feedback processes. Once the process has started it may not be easy to stop it.
Thinking of reglaciation as “not pressing” seems like a quick way to get reglaciated. Humans have got to intervene in the planet’s climate and warm it up in order to avoid this disaster. Leaving the climate alone would be a recipe for reglaciation. Pumping CO2 into the atmosphere may have saved us from disaster already, may save us from disaster in the future, may merely be a step in the right direction—or may be pretty ineffectual. However, it is important to realise that humans have got to take steps to warm the planet up—otherwise our whole civilisation may be quickly screwed.
We don’t know that industrial CO2 will protect us from reglaciation—since we don’t yet fully understand the latter process—though we do know that it devastates the planet like clockwork, and so has an astronomical origin.
The atmosphere has a CO2 decay function with an estimated half-life time of somwhere between 20-100 years. It wouldn’t vanish overnight—but a lot of it could go pretty quickly if civilisation problems resulted in a cessation of production.
If reglaciation starts, could it be stopped by sprinkling coal dust on some of the ice?
Hopefully—if we have enough of a civilisation at the time. Reglaciation seems likely to only really be a threat after a major disaster or setback—I figure. Otherwise, we can just adjust the climate controls. The chances of such a major setback may seem slender—but perhaps are not so small that we can afford to be blazee about the matter. What we don’t want is to fall down the stairs—and then be kicked in the teeth.
I discuss possible theraputic interventions on: http://timtyler.org/tundra_reclamation/
The main ones listed are planting northerly trees and black ground sheets.
We don’t know great many things, but what to do right now, we must decide right now, based on whatever we happen to know. (To address the reason for Desrtopa’s comment, if not any problem with your comment on this topic I’m completely ignorant about.)
If you are concerned about loss of potentially valuable information in the form of species extinction, global warming seems like total fluff. Look instead to habitat destruction and decimation, farming practices, and the resistribution of pathogens, predators and competitors by humans.
I do look at all these issues. I’ve spoken at conferences about how they receive too little attention relative to the danger they pose. That doesn’t mean that global warming does not stand to cause major harm, and going on the basis of the content of your site, you don’t seem to have invested adequate effort into researching the potential dangers, only the potential benefits.
Really? Don’t you hear enough about the supposed potential dangers from elsewhere already?!? I certainly do.
Be advised that I have pretty minimal effort to invest in discussing global warming these days. It is a big environmentalist scam. I list it at the top of my list of bad causes here.
The only reason worth discussing it—IMO—is if you are trying to direct all the resources that are being senselessly squandered on it towards better ends. You show little sign of doing that. Rather you appear to be caught up in promoting the lunacy. Global warming is mostly fluff. WAKE UP! Bigger fish are frying.
Frankly, reviewing the content of your site strongly leads me to suspect that your position is not credible; you consistently fail to accurately present what scientists consider to be the reasons for concern. When you claim that global warming is mostly fluff, I already have stronger reason than usual to suspect that you haven’t come by your conclusion from an unbiased review of the data.
I would care much less about bothering to convince you if you were not hosting a website for the purpose of convincing others to support furthering global warming, and despite the fact that there is now virtually no controversy that global warming exists, that we are causing it, and that the negatives will outweigh the positives, many people will take any excuse to dismiss it.
My site doesn’t go into the down-sides of global warming sufficiently for you?!? My main purpose is to explain the advantages. My site is part of the internet. You can obtain a huge mountain of information about the disadvantages by following the links. I do not expect readers to use my site as their sole source of information on the topic.
There is little controversy that global warming exists. There’s quite a bit more controversy about the role of humans—though my personal view is that humans are implicated. Relatively few people seem to have given much thought to the optimal temperature of the planet for living systems. Certainly very few of those involved in the “global warming movement”.
Anyway, global warming is good. It may have saved the planet from reglaciation already, or it may do so in the future, but without global warming, the planet and our civilisation would be screwed for a long time, with high probabality. The effects so far have been pretty miniscule, though. We have to carry on warming up the planet on basic safety grounds. The only issue I see is: “how fast”.
Your representations of the downsides (forest fires and floods) are actively misleading to readers. You create a false comparison by weighing all the most significant pros you can raise against a few of the more trivial cons, and encourage readers to make a judgment on that basis. Remember that people tend to actively seek out information sources that support their own views, and once they have adopted a view, seeing arguments for it falsified will not tend to revise their confidence downwards. Additionally, there is no shortage of people who will seize on any remotely credible sounding excuse to vindicate them from having to worry about global warming. Your site runs counter to the purpose of informing the public.
If global warming has already prevented reglaciation (which, as I have stated repeatedly, is most likely the case,) then why should we continue warming the planet? Unless we slow it down by orders of magnitude, the warming is occurring faster than the geological timescales on which ecosystems without technological protection are equipped to cope with it. Onset of glaciation is much slower than the progress of global warming. We are closer to the “cliff edge” of warming the world too fast than we are to reglaciation.
You’ve already stated that you consider global warming to be a storm in a teacup because you expect to see weather control before it becomes a serious issue, but even assuming that’s realistic, it gives us a larger time window to deal with reglaciation, should we turn out not to have already prevented it.
There is no shortage of people with smoke coming out of their ears about the issue either.
Look, I don’t just mention fires and floods. I mention sea-level rise, coral reef damage, desertification, heatstroke, and a number of other disadvantages. However, the disadvantages of GW are not the main focus of my article—you can find them on a million other web sites.
Repeating something doesn’t make it true. Show that the risk is so small that we need no longer concern ourselves with the huge catastrophe it would represent, and I swear, I will not mention it again. The current situation—as I understand it—is that our understanding of glaciation cycles is weak, our understanding of climate dynamics is weak, and we have little idea of what risk of reglaciation we face.
Of course, a warmer planet will still be healthier and better one—risk of glaciation or no—but then warming it up becomes less urgent.
There isn’t a “cliff-edge” of warming nearby—AFAICS. If you look at the temperature graph of the last million years, we are at a high point—and about to fall off a cliff into a glacial phase. The other direction we can only “fall” in by rearranging the continents, so there isn’t land over the south pole, and a land-locked region around the north pole. With ice-age continental positions, warming the planet is more like struggling uphill.
One thing we could try and do about that is to destroy the Isthmus of Panama. However, that needs further research—and might be a lot of work.
Indeed. So, I only have one page about the topic, and thousands of pages about other things. As I said, this area is only of interest as a bad cause, really.
It’s true that there are people with an unrealistic view of the dangers that global warming poses, and hyperbolic reactions may stand to hurt the cause of getting people to take it seriously, but your expectations of how seriously people ought to be taking it are artificially low.
You don’t just mention fires and floods, this is true, but you invite readers to make a comparison of the pros and cons without accurately presenting the cons. If you don’t want to mislead people, you should either be presenting them more accurately, or outright telling people “This site does not accurately present the cons of global warming, and you should not form an opinion on the basis of this site without doing further research at sites not dedicated to arguing against the threats it may pose.”
You can claim that your readers are free to research the issue elsewhere, but what you are doing is encouraging them to be informationally irresponsible.
As you have already noted earlier in this debate, ice sheet melting is a runaway positive feedback process. Once we have started the cycle, warming the planet is not an uphill struggle. If you look at the temperature graph of the last million years, you will not see that we are about to fall into another glaciation period, because the climate record of the last million years does not reflect the current situation. Even if greenhouse gas levels halted where they are now, temperature would continue to rise to meet the level they set. Our climate situation is more reflective of the Pliocene, which did not have cyclical glaciation periods.
I don’t have the articles on which I based the statement; I would need access to materials from courses I’ve already graduated. I may be able to get the data by contacting old professors, but there’s a significant hassle barrier, particularly since I’m arguing with someone who I do not have a strong expectation of being receptive to new information. If I do retrieve the information, do you commit to revising your site to reflect it, or removing the site as obsolete?
You what? You are starting to irritate me with your unsolicited editorial advice. Don’t like my web page—go and make your own!
Yes—from the point of full glaciation to little glaciation, ice core evidence suggests this process is a relatively unstoppable one-way slide. However, we have gone down that slide already in this glacial cycle. Trying to push the temperature upward from there hasn’t happened naturally for millions of years. That doesn’t look so easy—while we still have an ice-age continental configuration.
That sounds more like ignoring the temperature graph—and considering other information—to me.
You are kidding, presumably. The information needs to be compelling evidence that there isn’t a significant risk. Whether you retreive it or not seems likely to be a minor factor.
As I previously explained, a warmer planet would still be better, reglaciation risk or no. More of the planet would be habitable, fewer would die of pneumonia, there would be fewer deserts—and so on. However, the reglaciation risk does add urgency to the situation.
Given that people tend to seek out informational sources that flatter their biases, and your website is one of relatively few sources attempting to convince people that the data suggests global warming is positive, making another website has less utility to me than convincing you to revise yours.
It is considering the temperature graph insofar as it is relevant, while incorporating other information that is more reflective of our current situation. Would you try to extrapolate technological advances in the next twenty years based on the average rate of technological advancement over the last six hundred?
We had more or less the same continental configuration in the Pliocene, without cyclical glaciation periods.
Climate zones would shift, necessitating massive relocations which would cause major infrastructure damage. If we alter the earth’s temperature to make larger proportions of the land mass optimally biologically suitable to humans, we are going to do tremendous damage to species that are not adapted to near-equatorial habitation, and the impact on those other species is going to cause more problems than the relatively small drop in rates of pneumonia.
How compelling would you expect the evidence to be in order for you to be willing to revise the content of your site?
I am not sure what you are getting at there. Continental configuration is one factor. There may be other ones—including astronomical influences.
Why do you think it needs revising? A warmer planet looks remarkably positive—while keeping the planet in the freezer does not. I am not very impressed by the argument that change is painful, so we should keep the planet in an ice-age climate. Anyway, lots of evidence that global warming is undesirable—and that a half-frozen planet is good—would cause me to revise my position. However, I don’t seriously expect that to happen. That is a misguided lie—and a cause of much wasted energy and resources on the planet.
You are proposing an additional mechanism to a phenomenon that we can now model with considerable confidence without it
Since you do not seem prepared to acknowledge the more significant dangers of rapid climate change, while substantially overstating the benefits of altering the entire planet to suit the biological preferences of a species that adapted to life near the equator, I doubt you would be swayed by the amount of evidence that should be (and, I assert, is,) forthcoming if you are wrong. I expect that most of the other members here already regard your position with an appropriate degree of skepticism, so while it frustrates me to leave the matter at rest while I am convinced that your position is untenable, I don’t see any benefit in continuing to participate in this debate.
Continental configuration is not an unnecessary “additional mechanism”. It is a well known factor—see:
http://en.wikipedia.org/wiki/Ice_age#Position_of_the_continents
Thanks for the link, though. It will be interesting to see whether their ideas stand up to peer review. If so, it seems like bad news—the authors seem to think the forces that lead to reglaciation are due to kick in around about now.
OK, then—bye!
Do you have a feeling for how much of the planet would be temperate if it were warmer?
Imagine http://en.wikipedia.org/wiki/Temperateness with the bands further towards the poles.
And a bit further away from the equator.
The reason for the shift to the term “climate change” over “global warming” is that climate zones would shift. Places that were previously not arable would become so, but places that previously were arable would cease to be. It would require a significant restructuring of our civilization to chase around the climate zones with the appropriate infrastructure.
Looking at a map …the warming occurs quite a bit more near the poles—and over land masses. So, mostly Canada and Russia will get less frosty and more cosy. That seems good. It would be hard to imagine a more positive kind of climate change than one that makes our large and mostly barren northern wastelands more productive and habitable places.
Right—over tens of thousands of years, probably. The Antarctic is pretty thick.
While I can’t speak for everyone, I strongly suspect presenting things like this makes your case less persuasive.
A dead-pan presentation washes out my character—and means I have to be more wordy to get across my intended message. Who knows if a slap will work? Not me. Consider it an experiment.
All right. In that case: what would you consider meaningful experimental results, and what would they demonstrate?
This is my experiment—and, alas, you will have to leave me to it.
I would be wary of overloading your message. Consider this as a core:
“My heavily researched impression is that global warming, while somewhat concerning, is significantly less important than many other concerns.”
Now, we dress that up. Notice the impact the following change has on you:
“I, Vaniver, have heavily researched global warming, and my impression is that while it is somewhat concerning, it is significantly less important than many other concerns.”
Most of the time, putting yourself into the message reduces persuasiveness, and the places where it doesn’t probably don’t occur during abstract debates. (Persuading someone to share their lunch with you is an example of an appropriate time, but I wouldn’t call it an abstract debate.) So while some character is inevitable, I would generally seek to err on the side of suppressing it rather than expressing it.
Global warming seems a lot less dangerous than reglaciation.
Actually, I expect us to master climate control fairly quickly. That is another reason why global warming is a storm in a teacup. However, the future is uncertain. We might get unlucky—and be hit by a fair-sized meteorite. If that happens, reglaciation is about the last thing we would want for desert.
“Fairly quickly”? What if we don’t? Do you expect reglaciation to occur within the next 100 years, 200 years? If not we can wait until we have the knowledge to pull off climate control safely. (And if we do get hit by an asteroid, the last thing we probably want is runaway climate change started when we didn’t know what we were doing either.)
If things go according to plan, we get climate control—and then need to worry little about either warming or reglaciation. The problem is things not going according to plan.
Indeed. The “runaway climate change” we are scheduled for is reglaciation. The history of the planet is very clear on this topic. That is exactly what we don’t want. A disaster followed by glaciers descending over the northern continents could make a mess of civilisation for quite a while. Warming, by contrast doesn’t represent a significant threat—living systems including humans thrive in warm conditions.
Living systems including humans also thrive in cold conditions. Most species on the planet today have persisted through multiple glaciation periods, but not through pre-Pleistocene level warmth or rapid warming events.
Plus, the history of the Pleistocene, in which our record of glaciation exists, contains no events of greenhouse gas release and warming comparable to the one we’re in now, this is not business as usual on the track to reglaciation. Claiming that the history of the planet is very clear that we’re headed for reglaciation is flat out misleading. Last time the world had CO2 levels as high as they are now, it wasn’t going through cyclical glaciation.
Most species on the planet are less than 2.5 million years old?!?
I checked and found: “The fossil record suggests an average species lifespan of about five million years” and “Average species lifespan in fossil record: 4 million years.” (search for sources).
So, I figure your claim is probably factually incorrect. However, isn’t it a rather meaningless statistic anyway? It depends on how often lineages speciate. That actually says very little about how long it takes to adapt to an environment.
The average species age is necessarily lower than the average species duration.
Additionally, the fossil record measures species in paleontological terms, a paleontological “species” is not a species in biological terms, but a group which cannot be distinguished from each other by fossilized remains. Paleontological species duration sets the upper bound on biological species duration; in practice, biological species duration is shorter.
Species originating more than 2.5 million years ago which were not capable of enduring glaciation periods would have died out when they occurred. The origin window for species without adaptations to cope is the last ten thousand years. Any species with a Pleistocene origin or earlier has persisted through glaciation periods.
Allow me to try: There are positive feedback cycles which appear to be going in runaway mode. Why is this evidence for “things are going to get better” rather than “things are going to get worse”?
Your argument as a whole- “we need to get above this variability regime into a stable regime”- answers why the runaway positive feedback loop would be desirable, but does not convincingly establish (the part I’ve read, at least, you may do this elsewhere) that the part above the current variability is actually a stable attractor, instead of us shooting to up to Venus’s climate (or something less extreme but still regrettable for humans).
Well, we already know what the planet is like when it is not locked into a crippling ice age. Ice-cap free is how the planet has spent the vast majority of its history. We have abundant records about that already.
That’s the whole “ice age: bad / normal planet: good” notion. I figure a planet locked into a crippling era of catastrophic glacial cycles is undesirable.
So the real problem here is weakness of arguments, since they lack explanatory power by being able to “explain” too much.
Point of fact: the negative singularity isn’t a superstimulus for evolved fear circuits: current best-guess would be that it would be a quick painless death in the distant future (30 years+ by most estimates, my guess 50 years+ if ever). It doesn’t at all look like how I would design a superstimulus for fear.
It typically has the feature that you, all your relatives, friends and loved-ones die—probably enough for most people to seriously want to avoid it. Michael Vasser talks about “eliminating everything that we value in the universe”.
Maybe better super-stimuli could be designed—but there are constraints. Those involved can’t just make up the apocalypse that they think would be the most scary one.
Despite that, some positively hell-like scenarios have been floated around recently. We will have to see if natural selection on these “hell” memes results in them becoming more prominent—or whether most people just find them too ridiculous to take seriously.
Yes, you can only look at them through a camera lens, as a reflection in a pool or possibly through a ghost! ;)
I think you’re trying to fit the facts to the hypothesis. Negatve singularity in my opinion is at least 50 years away. Many people I know will already be dead by then, including me if I die at the same point in life as the average of my family.
And as a matter of fact it is failing to actually get much in the way of donations, compared to donations to the church which is using hell as a superstimulus, or even compared to campaigns to help puppies (about $10bn in total as far as I can see).
It is also not well-optimized to be believable.
It doesn’t work. Jehovah’s Witnesses don’t even believe into a hell and they are gaining a lot of members each year and donations are on the rise. Donations are not even mandatory either, you are just asked to donate if possible. The only incentive they use is positive incentive.
People will do everything for their country if it asks them to give their life. Suicide bombers also do not blow themselves up because of negative incentive but because they promise their families help and money. Also some believe that they will enter paradise. Negative incentive makes many people reluctant. There is much less crime in the EU than in the U.S. and they got death penalty. Here you get out of jail after max. ~20 years and there’s almost no violence in jails either.
I take it that you would place (t(positive singularity) | positive singularity) a significant distance further still?
This got a wry smile out of me. :)
(t(positive singularity) | positive singularity)
I’m going to say 75 years for that. But really, this is becoming very much total guesswork.
I do know that AGI -ve singularity won’t happen in the next 2 decades and I think one can bet that it won’t happen after that for another few decades either.
It’s still interesting to hear your thoughts. My hunch is that the difficulty of the -ve --> +ve step is much harder than the ‘singularity’ step so I would expect the time estimates to reflect that somewhat. But there are all sorts of complications there and my guesswork is even more guess-like than yours!
If you find anyone who is willing to take you up on a bet of that form given any time estimate and any odds then please introduce them to me! ;)
Many plausible ways to S^+ involve something odd or unexpected happening. WBE might make computational political structures, i.e. political structures based inside a computer full of WBEs. This might change the way humans cooperate.
Suffices to say that FAI doesn’t have to come via the expected route of someone inventing AGI and then waiting until they invent “friendliness theory” for it.
Church and cute puppies are likely worse causes, yes. I listed animal charities in my “Bad causes” video.
I don’t have their budget at my fingertips—but SIAI has raked in around 200,000 dollars a year for the last few years. Not enormous—but not trivial. Anyway, my concern is not really with the cash, but with the memes. This is a field adjacent to one I am interested in: machine intelligence. I am sure there will be a festival of fear-mongering marketing in this area as time passes, with each organisation trying to convince consumers that its products will be safer than those of its rivals. “3-laws-safe” slogans will be printed. I note that Google’s recent chrome ad was full of data destruction images—and ended with the slogan “be safe”.
Some of this is potentially good. However, some of it isn’t—and is more reminiscent of the Daisy ad.
To me, $200,000 for a charity seems to be pretty much the smallest possible amount of money. Can you find any charitable causes that recieve less than this?
Basically, you are saying that SIAI DOOM fearmongering is a trick to make money. But really, it fails to satisfy several important criteria:
it is shit at actually making money. I bet you that there are “save the earthworm” charities that make more money.
it is not actually frightening. I am not frightened; quick painless death in 50 years? boo-hoo. Whatever.
it is not optimized for believability. In fact it is almost optimized for anti-believability, “rapture of the nerds”, much public ridicule, etc.
A moment’s googling finds this:
http://www.buglife.org.uk/Resources/Buglife/Buglife%20Annual%20Report%20-%20web.pdf
($863 444)
I leave it to readers to judge whether Tim is flogging a dead horse here.
Not the sort of thing that could, you know, give you nightmares?
The sort of thing that could give you nightmares is more like the stuff that is banned. This is different than the mere “existential risk” message.
Alas, I have to reject your summary of my position. The situation as I see it:
DOOM-based organisations are likely to form with a frequency which depends on the extent to which the world is percieved to be at risk;
They are likely to form from those with the highest estimates of p(DOOM);
Once they exist, they are likely to try and grow, much like all organisations tend to do—wanting attention, time, money and other available resources;
Since they are funded in proportion to the percived value of p(DOOM), such organisations will naturally promote the notion that p(DOOM) is a large value.
This is all fine. I accept that DOOM-based organisations will exist, will loudly proclaim the coming apocalypse, and will find supporters to help them propagate their DOOM message. They may be ineffectual, cause despair and depression or help save the world—depending on their competence—and on to what extent their paranoia turns out to be justified.
However, such organisations seem likely to be very bad sources of information for anyone interested in the actual value of p(DOOM). They have obvious vested interests.
Agreed that x-risk orgs are a biased source of info on P(risk) due to self-selection bias. Of course you have to look at other sources of info, you have to take the outside view on these questions, etc.
Personally I think that we are so ignorant and irrational as a species (humanity) and as a culture that there’s simply no way to get a good, stable probability estimate for big important questions like this, much less to act rationally on the info.
But I think your pooh-pooh’ing such infantile and amateurish efforts as there is silly when the reasoning is entirely bogus.
Why don’t you refocus your criticism on the more legitimate weakness of existential risks: that is highly likely to be irrelevant (either futile or unnecessary), since by its own prediction, the relevant risks are highly complex and hard to mitigate against, and people in general are highly unlikely to either understand the issues or cooperate on them.
The most likely route to survival would seem to be that the entire model of the future propounded here is wrong. But in that case we move into the domain of irrelevance.
I hope I am not “pooh-pooh’ing”. There do seem to be a number of points on which I disagree. I feel a bit as though I am up against a propaganda machine—or a reality distortion field. Part of my response is to point out that the other side of the argument has vested interests in promoting a particular world view—and so its views on the topic should be taken with multiple pinches of salt.
I am not sure I understand fully—but I think the short answer is because I don’t agree with that. What risks there are, we can collectively do things about. I appreciate that it isn’t easy to know what to do, and am generally supportive and sympathetic towards efforts to figure that out.
Probably my top recommendation on that front so far is corporate reputation systems. We have these huge, powerful creatures lumbering around on the planet, and governments provide little infrastructure for tracking their bad deeds. Reviews and complaints scattered around the internet is just not good enough. If there’s much chance of corporation-originated intelligent machines, reputation-induced cooperation would help encourage these entities to be good and do good.
If our idea of an ethical corporation is one whose motto is “don’t be evil”, then that seems to be a pretty low standard. We surely want our corporations to aim higher than that.
One important aspect of corporate reputation is what it’s like to work there—and this is important on the department level and smaller level.
Abusive work environments cause a tremendous amount of misery, and there’s no reliable method of finding out whether a job is likely to land you in one.
This problem is made worse if leaving a job makes an potential employee seem less reliable.
Another aspect of a universal reputation system is that there needs to be some method of updating and verification. Credit agencies are especially notable for being sloppy.
Not necessarily. The risk might be virtually unstoppable, like a huge oil tanker compared to the force of a single person swimming in the water trying to slow it down.
What I mean is that, in my opinion, most of the risks under discussion are not like that. Large meteorites are a bit like that—but they are not very likely to hit us soon.
Ok, I see. Well, that’s just a big factual disagreement then.
The usual Singularity Institute line is that it is worth trying too, I believe. As to what p(success) is, the first thing to do would be to make sure that the parties involved mean the same thing by “success”. Otherwise, comparing values would be rather pointless.
This all reminds me of the dirac delta function. Its width is infinitesimal but its area is 1. Sure, it’s worth trying in the “Dirac Delta Function” sense.
Agreed that there are vested interests potentially biasing reasoning.
I think your disapproval of animal charities is based on circular logic, or at least an unproven premise.
You seem to be saying that animal causes are unworthy recipients of human effort because animals aren’t humans. However, people care about animals because of the emotional effects of animals. They care about people because of the emotional effects of people. I don’t think it’s proven that people only like animals because the animals are super-stimuli.
I could be mistaken, but I think that a more abstract utilitarian approach grounds out in some sort of increased enjoyment of life, or else it’s an effort to assume a universe-eye’s view of what’s ultimately valuable. I’m inclined to trust the former more.
What’s your line of argument for supporting charities that help people?
I usually value humans much more than I value animals. Given a choice between saving a human or N non-human animals, N would normally have to be very large before I would even think twice about it. Similar values are enshrined in law in most countries.
To the extent that the law accurately represents the values of the people it governs charities are not necessary. Vales enshrined in law are by necessity irrelevant.
(Noting by way of pre-emption that I do not require that laws should fully represent the values of the people.)
I do not agree. If the law says that killing a human is much worse than killing a dog, that is probably a reflection of the views of citizens on the topic.
And yet this is not contrary to my point. Charity operates, only needs to operate, on areas that laws do not already create a solution for. If there was a law specifying that dying kids get trips to Disneyland and visits by popstars then there wouldn’t be a “Make A Wish Foundation”.
You said the law was “irrelevant”—but there’s a sense in which we can see consensus human values about animals by looking at what the law dictates as punishment for their maltreatment. That is what I was talking about. It seems to me that the law has something to say about the issue of the value of animals relative to humans.
For the most part, animals are given relatively few rights under the law. There are exceptions for some rare ones. Animals are routinely massacred in huge numbers by humans—including some smart mammals like pigs and dolphins. That is a broad reflection how relatively-valuable humans are considered to be.
And once it’s enshrined in law, it no longer matters whether citizens think killing a human is worse or better than killing a dog. I think that is what wedrifid was noting.
You may be interested in Alan Dawrst’s essays on animal suffering and animal suffering prevention.
I believe the numbers are actually higher than $200,000. SIAI’s 2008 budget was about $500,000. 2006 was about $400,000 and 2007 was about $300,000 (as listed further in the linked thread). I haven’t researched to see if gross revenue numbers or revenue from donations are available. Curiously, Guidestar does not seem to have 2009 numbers for SIAI, or at least I couldn’t find those numbers; I just e-mailed a couple people at SIAI asking about that.
That being said, even $500,000, while not trivial, seems to me a pretty small budget.
Sorry, yes, my bad. $200,000 is what they spent on their own salaries.
I wonder what fraction of actual historical events a hostile observer taking similar liberties could summarize to also sound like some variety of “a fantasy story designed to manipulate”.
I don’t know—but believing inaction is best is rather common—and there are pages all about it—e.g.:
http://en.wikipedia.org/wiki/Learned_helplessness
One big disadvantage is that you won’t be interacting with other researchers from whom you can learn.
Research seems to be an insiders’ game. You only ever really see the current state of research in informal settings like seminars and lab visits. Conference papers and journal articles tend to give strange, skewed, out-of-context projections of what’s really going on, and books summarise important findings long after the fact.
At the same time however, you might be able to interact with researchers more effectively. For example, you could spend some of those research weeks visiting selected labs and seminars and finding out what’s up. It’s true that this would force you to be conscientious about opportunities and networking, but that’s not necessarily a bad thing. Networks formed with a very distinct purpose are probably going to outperform those that form more accidentally. You wouldn’t be as tied down as other researchers, which could give you an edge in getting the ideas and experiences you need for your research, while simultaneously making you more valuable to others when necessary (For example, imagine if one of your important research contacts needs two weeks of solid help on something. You could oblige whereas others with less fluid obligations could not.).
Being in a similar position (also as far as aversion to moving to e.g. US is concerned), I decided to work part time (roughly 1⁄5 of the time of even less) in software industry and spend the remainder of the day studying relevant literature, leveling up etc. for working on the FAI problem. Since I’m not quite out of the university system yet, I’m also trying to build some connections with our AI lab staff and a few other interested people in the academia, but with no intention to actually join their show. It would eat away almost all my time, so I could work on some AI-ish bio-informatics software or something similarly irrelevant FAI-wise.
There are of course some benefits in joining the academia, as you mentioned, but it seems to me that you can reap quite a bit of them by just befriending an assistant professor or two.
Kaj, why don’t you add the option of getting rich in your 20s by working in finance, then paying your way into research groups in your late 30s? PalmPilot guy, uh Jeff Hawkins essentially did this. Except he was an entrepreneur.
That doesn’t sound very easy.
Sounds a heck of a lot easier than doing an equivalent amount of status grabbing within academic circles over the same time.
Money is a lot easier to game and status easier to buy.
There is the minor detail that it really helps not to hate each and every individual second of your working life in the process. A goal will only pull you along to a certain degree.
(Computer types know all the money is in the City. I did six months of it. I found the people I worked with and the people whose benefit I worked for to be excellent arguments for an unnecessarily bloody socialist revolution.)
For many people that is about half way between the Masters and PhD degrees. ;)
If only being in a university was a guarantee of an enjoyable working experience.
Curious, why did it bother you that you disliked the people you worked with? Couldn’t you just be polite to them and take part in their jokes/socialgames/whatever? They’re paying you handsomely to be there, after all?
Or was it a case of them being mean to you?
No, just loathsome. And the end product of what I did and finding the people I was doing it for loathsome.
I dunno, “loathsome” sounds a bit theoretical to me. Can you be specific?
One of my brother’s co-workers at Goldman Sachs has actively tried to sabotage his work. (Goldman Sachs runs on a highly competitive “up or out” system; you either get promoted or fired, and most people don’t get promoted. If my brother lost his job, his coworker would be more likely to keep his.)
I don’t understand: he tried to sabotage his cowerker’s work, or his own?
CronoDAS’s Brother’s Co-worker tried to sabotage CronoDAS’s Brother’s work.
“Hamlet, in love with the old man’s daughter, the old man thinks.”
Not without getting political. Fundamentally, I didn’t feel good about what I was doing. And I was just a Unix sysadmin.
This was just a job to live, not a job taken on in the furtherance of a larger goal.
Agreed. Average Prof is a nobody at 40, average financier is a millionaire. shrugs
The average financier is a millionaire at 40?! What job is this, exactly?
Thank you for this. This was a profound revelation for me.
Upvoted for comedy.
Also, you can get a PhD in a relevant mathy discipline first, thereby satisfying the condition of having done research.
And the process of dealing with the real world enough to make money will hopefully leave you with better anti-akrasia tactics, better ability to achieve real-world goals, etc.
You might even be able to hire others.
I don’t think you need to be excessively rich. $1-4M ought to be enough.
Edit: oh, I forgot, you live in scandanavia, with a taxation system so “progressive” that it has an essential singularity at $100k. Might have to move to US.
I’m afraid that’s not really an option for me, due to various emotional and social issues. I already got horribly homesick during just a four month visit.
Alaska might be a reasonable Finland substitute, weather-wise, but the other issues will be difficult to resolve (if you’re moving to the US to make a bunch of money, Alaska is not the best place to do it).
One of my favorite professors was Brazilian, who went to graduate school at the University of Rochester. Horrified (I used to visit my ex in upstate New York, and so was familiar with the horrible winters that take up 8 months of the year without the compensations that convince people to live in Scandinavia), I asked him how he liked the transition- and he said that he loved it, and it was the best time of his life. I clarified that I was asking about the weather, and he shrugged and said that in academia, you absolutely need to put the ideas first. If the best place for your research is Antarctica, that’s where you go.
The reason why I tell this story is that this is what successful professors look like, and only one tenth of the people that go to graduate school end up as professors. If you would be outcompeted by this guy instead of this guy, keep that in mind when deciding you want to enter academia. And, if you want to do research outside of academia, in order to do that well that requires more effort than research done inside of academia.
It’s not the weather: I’d actually prefer a warmer climate than Finland has. It’s living in a foreign culture and losing all of my existing social networks.
I don’t have a problem with putting in a lot of work, but to be able to put in a lot of work, my life needs to be generally pleasant otherwise, and the work needs to be at least somewhat meaningful. I’ve tried the “just grit your teeth and toil” mentality, and it doesn’t work—maybe for someone else it does, but not for me.
The first part is the part I’m calling into question, not the second. Of course you need to be electrified by your work. It’s hard to do great things when you’re toiling instead of playing.
But your standards for general pleasantness are, as far as I can tell, the sieve for a lot of research fields. As an example, it is actually harder to be happy on a grad student/postdoc salary; instead of it being shallow to consider that a challenge, it’s shallow-mindedness to not recognize that that is a challenge. It is actually harder to find a mate and start a family while an itinerant academic looking for tenure. (Other examples abound; two should be enough for this comment.) If you’re having trouble leaving your network of friends to go to grad school / someplace you can get paid more, then it seems likely that you will have trouble with the standard academic life or standard corporate life.
While there are alternatives, those tend not to play well with doing research, since the alternative tends to take the same kind of effort that you would have put into research. I should comment that I think a normal day job plus research on the side can work out but should be treated like writing a novel on the side- essentially, the way creative literary types play the lottery.
It’s living in a foreign culture and losing all of my existing social networks.
Of course it is! I am in the same situation. Just finished undergrad in philosophy. But here life is completely optimized for happiness: 1) No errands 2) Friends filtered through 15 years for intelligence, fun, beauty, awesomeness. 3) Love, commitment, passion, and just plain sex with the one, and the others. 4) Deep knowledge of the free culture available 5) Ranking high in the city (São Paulo’s) social youth hierarchy 6) Cheap services 7) Family and acquaintances network. 8) Freedom timewise to write my books 9) Going to the park 10 min walking 10) Having been to, and having friends who were in the US, and knowing for fact that life just is worse there....
This is how much fun I have, the list’s impact is the only reason I’m considering not going to study, get FAI faster, get anti-ageing faster.
If only life were just a little worse...… I would be in a plane towards posthumanity right now.
So how good has a life to be for you to be forgiven of not working for what really matters? Help me folks!
Well, you wanna make an omlet, you gotta break some eggs!
Conditioning on yourself deeming it optimal to make a metaphorical omelet by breaking metaphorical eggs, metaphorical eggs will deem it less optimal to remain vulnerable to metaphorical breakage by you than if you did not deem it optimal to make a metaphorical omelet by breaking metaphorical eggs; therefore, deeming it optimal to break metaphorical eggs in order to make a metaphorical omelet can increase the difficulty you find in obtaining omelet-level utility.
Many metaphorical eggs are not [metaphorical egg]::Utility maximizing agents.
True, and to the extent that is not the case, the mechanism I specified would not activate.
Redefining one’s own utility function so as to make it easier to achieve is the road that leads to wireheading.
Correct. However, the method I proposed does not involve redefining one’s utility function, as it leaves terminal values unchanged. It simply recognizes that certain methods of achieving one’s pre-existing terminal values are better than others, which leaves the utility function unaffected (it only alters instrumental values).
The method I proposed is similar to pre-commitment for a causal decision theorist on a Newcomb-like problem. For such an agent, “locking out” future decisions can improve expected utility without altering terminal values. Likewise, a decision theory that fully absorbs such outcome-improving “lockouts” so that it outputs the same actions without explicit pre-commitment can increase its expected utility for the same utility function.
Do you have any advice for getting into Quant work? (I’m a second year maths student at Oxford, don’t know much about the city).
An advice sheet for mathematicians considering becoming quants. It’s not a path that interests me, but if it was I think I’d find this useful.
Are there any good ways of getting rich that don’t involve selling your soul?
Please rephrase without using “selling your soul”.
Are there any good ways of getting rich that don’t involve a Faustian exchange with Lucifer himself?
Pfft. No good ways.
Without corrupting my value system, I suppose? I’m interested in getting money for reasons other than my own benefit. I am not fully confident in my ability to enter a field like finance without either that changing or me getting burned out by those around me.
As well ask if there are hundred-dollar bills lying on sidewalks.
EDIT: 2 days after I wrote this, I was walking down the main staircase in the library and laying on the central landing, highly contrasted against the floor, in completely clear view of 4 or 5 people who walked past it, was a dollar bill. I paused for a moment reflecting on the irony that sometimes there are free lunches—and picked it up.
This thread raises the question about how many biologists and medical researchers are on here. Due to our specific cluster I expect a strong learning towards the IT people. So AI research gets over proportional recognition, while medical research including direct life extension falls on the wayside.
Speaking as someone who is in grad school now, even with prior research, the formal track of grad school is very helpful. I am doing research that I’m interested in. I don’ t know if I’m a representative sample in that regard. It may be that people have more flexibility in math than in other areas. Certainly my anecdotal impression is that people in some areas such as biology don’t have this degree of freedom. I’m also learning more about how to research and how to present my results. Those seem to be the largest advantages. Incidentally, my impression is that for grad school at least in many areas, taking a semester or two off if very stressed isn’t treated that badly if one is otherwise doing productive research.
I’m in grad school in statistics and am in the same boat. It doesn’t seem that difficult to do research on something you’re interested in while still in grad school. In a nutshell, choose your major professor wisely. (And make sure the department is large enough that there are plenty of options)
(I would have liked to reply to the deleted comment, but you can’t reply to deleted comments so I’ll reply to the repost.)
EDIT: Roko reveals that he was actually never asked to delete his comment! Disregard parts of the rest of this comment accordingly.
I don’t think Roko should have been requested to delete his comment. I don’t think Roko should have conceded to deleting his comment.
The correct reaction when someone posts something scandalous like
is not to attempt to erase it, even if that was possible, but to reveal the context. The context, supposedly, would make it seem less scandalous—for example, maybe it was a private dicussion about philosophical hypotheticals. If it wouldn’t, that’s a bad sign about SIAI.
The fact that that erasure was the reaction suggests that there is no redeeming context!
That someone asked Roko to erase his comment isn’t a very bad sign, since it’s enough that one person didn’t understand the reasoning above for that to happen. That fact that Roko conceded is a bad sign, though.
Now SIAI should save face not by asking a moderator to delete wfg’s reposts, but by revealing the redeeming context in which the scandalous remarks that Roko alluded to were made.
Roko may have been thinking of [just called him, he was thinking of it] a conversation we had when he and I were roommates in Oxford while I was visiting the Future of Humanity Institute, and frequently discussed philosophical problems and thought experiments. Here’s the (redeeming?) context:
As those who know me can attest, I often make the point that radical self-sacrificing utilitarianism isn’t found in humans and isn’t a good target to aim for. Almost no one would actually take on serious harm with certainty for a small chance of helping distant others. Robin Hanson often presents evidence for this, e.g. this presentation on “why doesn’t anyone create investment funds for future people?” However, sometimes people caught up in thoughts of the good they can do, or a self-image of making a big difference in the world, are motivated to think of themselves as really being motivated primarily by helping others as such. Sometimes they go on to an excessive smart sincere syndrome, and try (at the conscious/explicit level) to favor altruism at the severe expense of their other motivations: self-concern, relationships, warm fuzzy feelings.
Usually this doesn’t work out well, as the explicit reasoning about principles and ideals is gradually overridden by other mental processes, leading to exhaustion, burnout, or disillusionment. The situation winds up worse according to all of the person’s motivations, even altruism. Burnout means less good gets done than would have been achieved by leading a more balanced life that paid due respect to all one’s values. Even more self-defeatingly, if one actually does make severe sacrifices, it will tend to repel bystanders.
Instead, I typically advocate careful introspection and the use of something like Nick Bostrom’s parliamentary model:
In the conversation with Roko, we were discussing philosophical thought experiments (trolley problem style, which may indeed be foolish ) to get at ‘real’ preferences and values for such an exercise. To do that, one often does best to adopt the device of the True Prisoner’s Dilemma and select positive and negative payoffs that actually have emotional valence (as opposed to abstract tokens). For positive payoffs, we used indefinite lifespans of steady “peak experiences” involving discovery, health, status, and elite mates. For negative payoffs we used probabilities of personal risk of death (which comes along with almost any effort, e.g. driving to places) and harms that involved pain and/or a decline in status (since these are separate drives). Since we were friends and roommates without excessive squeamishness, hanging out at home, we used less euphemistic language.
Neither of us was keen on huge sacrifices in Pascal’s-Mugging-like situations, viewing altruism as only one part of our respective motivational coalitions, or one term in bounded utility functions. I criticized his past “cheap talk” of world-saving as a primary motivation, given that in less convenient possible worlds, it was more easily overcome than his phrasing signaled. I said he should scale back his claims of altruism to match the reality, in the way that I explicitly note my bounded do-gooding impulses.
We also differed in our personal views on the relative badness of torture, humiliation and death. For me, risk of death was the worst, which I was least willing to trade off in trolley-problem type cases to save others. Roko placed relatively more value on the other two, which I jokingly ribbed and teased him about.
In retrospect, I was probably a bit of a jerk in pushing (normative) Hansonian transparency. I wish I had been more careful to distinguish between critiquing a gap between talk and values, and critiquing the underlying values, and probably should just take wedifrid’s advice on trolley-problem-type scenarios generally.
First off, great comment—interesting, and complex.
But, some things still don’t make sense to me...
Assuming that what you described led to:
How did precommitting enter in to it?
Are you prepared to be tortured or raped for the cause? Have you precommitted to it?
Have other SIAI people you know of talked about this with you, have other SIAI people precommitted to it?
What do you think of others who do not want to be tortured or raped for the cause?
Thanks, wfg
I find this whole line of conversation fairly ludicrous, but here goes:
Number 1. Time-inconsistency: we have different reactions about an immediate certainty of some bad than a future probability of it. So many people might be willing to go be a health worker in a poor country where aid workers are commonly (1 in 10,000) raped or killed, even though they would not be willing to be certainly attacked in exchange for 10,000 times the benefits to others. In the actual instant of being tortured anyone would break, but people do choose courses of action that carry risk (every action does, to some extent), so the latter is more meaningful for such hypotheticals.
Number 2. I have driven and flown thousands of kilometers in relation to existential risk, increasing my chance of untimely death in a car accident or plane crash, so obviously I am willing to take some increased probability of death. I think I would prefer a given chance of being tortured to a given chance of death, so obviously I care enough to take at least some tiny risk from what I said above. As I also said above, I’m not willing to make very big sacrifices (big probabilities of such nasty personal outcomes) for tiny shifts in probabilities of big impersonal payoffs (like existential risk reduction). In realistic scenarios, that’s what “the cause” would refer to. I haven’t made any verbal or explicit “precommitment” or promises or anything like that.
In sufficiently extreme (and ludicrously improbable) trolley-problem style examples, e.g. “if you push this button you’ll be tortured for a week, but if you don’t then the Earth will be destroyed (including all your loved ones) if this fair coin comes up heads, and you have incredibly (impossibly?) good evidence that this really is the setup” I hope I would push the button, but in a real world of profound uncertainty, limited evidence, limited personal power (I am not Barack Obama or Bill Gates), and cognitive biases, I don’t expect that to ever happen. I also haven’t made any promises or oaths about that.
I am willing to give of my time and effort, and forgo the financial rewards of a more lucrative career, in exchange for a chance for efficient do-gooding, interaction with interesting people who share my values, and a meaningful project. Given diminishing returns to money in rich countries today, and the ease of obtaining money for folk with high human capital, those aren’t big sacrifices, if they are sacrifices at all.
Number 3. SIAIers love to be precise and analytical and consider philosophical thought experiments, including ethical ones. I think most have views pretty similar to mine, with somewhat varying margins. Certainly Michael Vassar, the head of the organization, is also keen on recognizing one’s various motives and living a balanced life, and avoiding fanatics. Like me, he actively advocates Bostrom-like parliamentary model approaches to combining self-concern with parochial and universalist altruistic feelings.
I have never heard anyone making oaths or promises to make severe sacrifices.
Number 4. This is a pretty ridiculous question. I think that’s fine and normal, and I feel more comfortable with such folk than the alternative. I think people should not exaggerate that do-gooding is the most important thing in their life lest they deceive themselves and others about their willingness to make such choices, which I criticized Roko for.
This sounds very sane, and makes me feel a lot better about the context. Thank you very much.
I very much like the idea that top SIAI people believe that there is such a thing as too much devotion to the cause (and, I’m assuming, actively talk people who are above that level down as you describe doing for Roko).
As someone who has demonstrated impressive sanity around these topics, you seem to be in a unique position to answer these questions with an above-average level-headedness:
Do you understand the math behind the Roko post deletion?
What do you think about the Roko post deletion?
What do you think about future deletions?
Yes, his post was based on (garbled versions of) some work I had been doing at FHI, which I had talked about with him while trying to figure out some knotty sub-problems.
I think the intent behind it was benign, at least in that Eliezer had his views about the issue (which is more general, and not about screwed-up FAI attempts) previously, and that he was motivated to prevent harm to people hearing the idea and others generally. Indeed, he was explicitly motivated enough to take a PR hit for SIAI.
Regarding the substance, I think there are some pretty good reasons for thinking that the expected value (with a small probability of a high impact) of the info for the overwhelming majority of people exposed to it would be negative, although that estimate is unstable in the face of new info.
It’s obvious that the deletion caused more freak-out and uncertainty than anticipated, leading to a net increase in people reading and thinking about the content compared to the counterfactual with no deletion. So regardless of the substance about the info, clearly it was a mistake to delete (which Eliezer also recognizes).
Obviously, Eliezer is continuing to delete comments reposting on the topic of the deleted post. It seems fairly futile to me, but not entirely. I don’t think that Less Wrong is made worse by the absence of that content as such, although the fear and uncertainty about it seem to be harmful. You said you were worried because it makes you uncertain about whether future deletions will occur and of what.
After about half an hour of trying, I can’t think of another topic with the same sorts of features. There may be cases involving things like stalkers or bank PINs or 4chan attacks or planning illegal activities. Eliezer called on people not to discuss AI at the beginning of Less Wrong to help establish its rationality focus, and to back off from the gender warfare, but hasn’t used deletion powers for such things.
Less Wrong has been around for 20 months. If we can rigorously carve out the stalker/PIN/illegality/spam/threats cases I would be happy to bet $500 against $50 that we won’t see another topic banned over the next 20 months.
That sounds like it’d generate some perverse incentives to me.
Urk.
Just to be clear: he recognizes this by comparison with the alternative of privately having the poster delete it themselves, rather than by comparison to not-deleting.
Or at least that was my understanding.
Regardless, thanks for a breath of clarity in this thread. As a mostly disinterested newcomer, I very much appreciated it.
Well, if counterfactually Roko hadn’t wanted to take it down I think it would have been even more of a mistake to delete it, because then the author would have been peeved, not just the audience/commenters.
Which is fine.
But Eliezer’s comments on the subject suggest to me that he doesn’t think that.
More specifically, they suggest that he thinks the most important thing is that the post not be viewable, and if we can achieve that by quietly convincing the author to take it down, great, and if we can achieve it by quietly deleting it without anybody noticing, great, and if we can’t do either of those then we achieve it without being quiet, which is less great but still better than leaving it up.
And it seemed to me your parenthetical could be taken to mean that he agrees with you that deleting it would be a mistake in all of those cases, so I figured I would clarify (or let myself be corrected, if I’m misunderstanding).
I should have taken this bet
Your post has been moved to the Discussion section, not deleted.
Looking at your recent post, I think Alicorn had a good point.
I agree with your main point, but the thought experiment seems to be based on the false assumption that the risk of being raped or murdered are smaller than 1 in 10K if you stay at home. Wikipedia guesstimates that 1 in 6 women in the US are on the receiving end of attempted rape at some point, so someone who goes to a place with a 1 in 10K chance of being raped or murdered has probably improved their personal safety. To make a better thought experiment, I suppose you have to talk about the marginal increase in rape or murder rate when working in the poor country when compared to staying home, and perhaps you should stick to murder since the rape rate is so high.
You lost me at ‘ludicrous’. :)
but he won me back by answering anyway <3
How so?
Thanks!
Great comment Carl!
Roko was not requested to delete his comment. See this parallel thread. (I would appreciate it if you would edit your comment to note this, so readers who miss this comment don’t have a false belief reinforced.) (ETA: thanks)
Agreed (and I think the chance of wfg’s reposts being deleted is very low, because most people get this). Unfortunately, I know nothing about the alleged event (Roko may be misdescribing it, as he misdescribed my message to him) or its context.
Roko said he was asked. You didn’t ask him but maybe someone else did?
Roko’s reply to me strongly suggested that he interpreted my message as requesting deletion, and that I was the cause of him deleting it. I doubt anyone at SIAI would have explicitly requested deletion.
I can confirm that I was not asked to delete the comment but did so voluntarily.
I think you are too trigger-happy.
The wording here leaves weird wiggle room—you’re implying it wasn’t Nick?
I’m wondering whether you, Nick, have learned anything from this experience—something perhaps about how attempting to hide something is almost always counterproductive?
Of course, Roko contributed here by deleting the message, you didn’t create this mess by yourself. But you sure have helped. :)
Well, look, I deleted it of my own accord, but only after being prompted that it was a bad thing to have posted. Can we just drop this? It makes me look like even more of a troublemaker than I already look like, and all I really want to do is finish the efficient charity competition then get on with life outside teh intenetz.
This is very important. If the SIAI is the organisation to solve the friendly AI problem and implement CEV then it should be subject to public examination, especially if they ask for money.
The current evidence that anyone anywhere can implement CEV is two papers in six years that talk about it a bit. There appears to have been nothing else from SIAI and no-one else in philosophy appears interested.
If that’s all there is for CEV in six years, and AI is on the order of thirty years away, then (approximately) we’re dead.
This is rather disappointing, as if CEV is possible then a non-artificial general intelligence should be able to implement it, at least partially. And we have those. The reason for CEV is (as I understand it) the danger of the AI going FOOM before it cares about humans. However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
Addendum: I see others have been asking “but what do you actually mean?” for a couple of years now.
This strikes me as a demand for particular proof. SIAI is small (and was much smaller until the last year or two), the set of people engaged in FAI research is smaller, Eliezer has chosen to focus on writing about rationality over research for nearly four years, and FAI is a huge problem, in which any specific subproblem should be expected to be underdeveloped at this early stage. And while I and others expect work to speed up in the near future with Eliezer’s attention and better organization, yes, we probably are dead.
Somewhat nitpickingly, this is a reason for FAI in general. CEV is attractive mostly for moving as much work from the designers to the FAI as possible, reducing the potential for uncorrectable error, and being fairer than letting the designers lay out an object-level goal system.
This sounds interesting; do you think you could expand?
It wasn’t intended to be—more incredulity. I thought this was a really important piece of the puzzle, so expected there’d be something at all by now. I appreciate your point: that this is a ridiculously huge problem and SIAI is ridiculously small.
I meant that, as I understand it, CEV is what is fed to the seed AI. Or the AI does the work to ascertain the CEV. It requires an intelligence to ascertain the CEV, but I’d think the ascertaining process would be reasonably set out once we had an intelligence on hand, artificial or no. Or the process to get to the ascertaining process.
I thought we needed the CEV before the AI goes FOOM, because it’s too late after. That implies it doesn’t take a superintelligence to work it out.
Thus: CEV would have to be a process that mere human-level intelligences could apply. That would be a useful process to have, and doesn’t require first creating an AI.
I must point out that my statements on the subject are based in curiosity, ignorance and extrapolation from what little I do know, and I’m asking (probably annoyingly) for more to work with.
“CEV” can (unfortunately) refer to either CEV the process of determining what humans would want if we knew more etc., or the volition of humanity output by running that process. It sounds to me like you’re conflating these. The process is part of the seed AI and is needed before it goes FOOM, but the output naturally is neither, and there’s no guarantee or demand that the process be capable of being executed by humans.
OK. I still don’t understand it, but I now feel my lack of understanding more clearly. Thank you!
(I suppose “what do people really want?” is a large philosophical question, not just undefined but subtle in its lack of definition.)
I have recieved assurances that SIAI will go to significant efforts not to do nasty things, and I believe them. Private assurances given sincerely are, in my opinion, the best we can hope for, and better than we are likely to get from any other entity involved in this.
Besides, I think that XiXiDu, et al are complaining about the difference between cotton and silk, when what is actually likely to happen is more like a big kick in the teeth from reality. SIAI is imperfect. Yes. Well done. Nothing is perfect. At least cut them a bit of slack.
What?!? Open source code—under a permissive license—is the traditional way to signal that you are not going to run off into the sunset with the fruits of a programming effort. Private assurances are usually worth diddly-squat by comparison.
I think that you don’t realize just how bad the situation is. You want that silken sheet. Rude awakening methinks. Also open-source not neccessarily good for FAI in any case.
I don’t think that you realize how bad it is. I’d rather have the universe being paperclipped than supporting the SIAI if that means that I might be tortured for the rest of infinity!
To the best of my knowledge, SIAI has not planned to do anything, under any circumstances, which would increase the probability of you or anyone else being tortured for the rest of infinity.
Supporting SIAI should not, to the best of my knowledge, increase the probability of you or anyone else being tortured for the rest of infinity.
Thank you.
But imagine there was a person a level above yours that went to create some safeguards for an AGI. That person would tell you that you can be sure that the safeguards s/he plans to implement will benefit everyone. Are you just going to believe that? Wouldn’t you be worried and demand that their project is being supervised?
You are in a really powerful position because you are working for an organisation that might influence the future of the universe. Is it really weird to be skeptical and ask for reassurance of their objectives?
Logical rudeness is the error of rejecting an argument for reasons other than disagreement with it. Does your “I don’t think so” mean that you in fact believe that SIAI (possibly) plans to increase the probability of you or someone else being tortured for the rest of eternity? If not, what does this statement mean?
I removed that sentence. I meant that I didn’t believe that the SIAI plans to harm someone deliberately. Although I believe that harm could be a side-effect and that they would rather harm a few beings than allowing some Paperclip maximizer to take over.
You can call me a hypocrite because I’m in favor of animal experiments to support my own survival. But I’m not sure if I’d like to have someone leading an AI project who thinks like me. Take that sentence to reflect my inner conflict. I see why one would favor torture over dust specks but I don’t like such decisions. I’d rather have the universe to end now, or having everyone turned into paperclips, than having to torture beings (especially if I am the being).
I feel uncomfortable that I don’t know what will happen because there is a policy of censorship being favored when it comes to certain thought experiments. I believe that even given negative consequences, transparency is the way to go here. If the stakes are this high, people who believe will do anything to get what they want. That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
I apologize. I realized my stupidity in interpreting your comment a few seconds after posting the reply (which I then deleted).
Better yet, you could use a kind of doublethink—and then even actually mean it. Here is W. D. Hamilton on that topic:
Discriminating Nepotism—as reprinted in: Narrow Roads of Gene Land, Volume 2 Evolution of Sex, p.356.
In TURING’S CATHEDRAL, George Dyson writes:
I think many people would like to be in that group—if they can find a way to arrange it.
Unless AI was given that outcome (cheerful, contented people etc) as a terminal goal, or that circle of people was the best possible route to some other terminal goal, both of which are staggeringly unlikely, Dyson suspects wrongly.
If you think he suspects rightly, I would really like to see a justification. Keep in mind that AGIs are currently not being built using multi-agent environment evolutionary methods, so any kind of ‘social cooperation’ mechanism will not arise.
Machine intelligence programmers seem likely to construct their machines so as to help them satisfy their preferences—which in turn is likely to make them satisfied. I am not sure what you are talking about—but surely this kind of thing is already happening all the time—with Sergey Brin, James Harris Simons—and so on.
That doesn’t really strike me as a stunning insight, though. I have a feeling that I could find many people who would like to be in almost any group of “cheerful, contented, intellectually and physically well-nourished people.”
This all depends on what the AI wants. Without some idea of its utility function, can we really speculate? And if we speculate, we should note those assumptions. People often think of an AI as being essentially human-like in its values, which is problematic.
It’s a fair description of today’s more successful IT companies. The most obvious extrapolation for the immediate future involves more of the same—but with even greater wealth and power inequalities. However, I would certainly also council caution if extrapolating this out more than 20 years or so.
Currently, there are no entities in physical existence which, to my knowledge, have the ability to torture anyone for the rest of eternity.
You intend to build an entity which would have that ability (or if not for infinity, for a googolplex of subjective years).
You intend to give it a morality based on the massed wishes of humanity—and I have noticed that other people don’t always have my best interests at heart. It is possible—though unlikely—that I might so irritate the rest of humanity that they wish me to be tortured forever.
Therefore, you are, by your own statements, raising the risk of my infinite torture from zero to a tiny non-zero probability. It may well be that you are also raising my expected reward enough for that to be more than counterbalanced, but that’s not what you’re saying—any support for SIAI will, unless I’m completely misunderstanding, raise the probability of infinite torture for some individuals.
See the “Last Judge” section of the CEV paper.
As Vladimir observes, the alternative to SIAI doesn’t involve nothing new happening.
That just pushes the problem along a step. IF the Last Judge can’t be mistaken about the results of the AI running AND the Last Judge is willing to sacrifice the utility of the mass of humanity (including hirself) to protect one or more people from being tortured, then it’s safe. That’s very far from saying there’s a zero probability.
If the Last Judge peeks at the output and finds that it’s going to decide to torture people, that doesn’t imply abandoning FAI, it just requires fixing the bug and trying again.
Just because AGIs have capability to inflict infinite torture, doesn’t mean they have a motive. Also, status quo (with regard to SIAI’s activity) doesn’t involve nothing new happening.
I explained that he is planning to supply one with a possible motive (namely that the CEV of humanity might hate me or people like me). It is precisely because of this that the problem arises. A paperclipper, or any other AGI whose utility function had nothing to do with humanity’s wishes, would have far less motive to do this—it might kill me, but it really would have no motive to torture me.
You can have your private assurances—and I will have my open-source software.
Gollum gave his private assurances to Frodo—and we all know how that turned out.
If someone solicits for you to “trust in me”, alarm bells should start ringing immediately. If you really think that is “the best we can hope for”, then perhaps revisit that.
Well I’m convinced. Frodo should definitely have worked out a way to clone the ring and made sure the information was available to all of Middle Earth. You can never have too many potential Ring-Wraiths.
Suddenly I have a mental image of “The Lord of the Rings: The Methods of Rationality.”
Someone should write that (with a better title). We could have a whole genre of rational fanfiction.
Agreed; Lord of the Rings seems like a natural candidate for discussing AI and related topics.
I’d also like to see His Dark Materials with rationalist!Lyra. The girl had an alethiometer. She should have kicked way more ass than she did as soon as she realized what she had.
Open source AGI is not a good thing. In fact, it would be a disastrously bad thing. Giving people the source code doesn’t just let them inspect it for errors, it also lets them launch it themselves. If you get an AGI close to ready for launch, then sharing its source code means that instead of having one party to decide whether there are enough safety measures ready to launch, you have many parties individually deciding whether to launch it themselves, possibly modifying its utility function to suit their own whim, and the hastiest party’s AGI wins.
Ideally, you’d want to let people study the code, but only trustworthy people, and in a controlled environment where they can’t take the source code with them. But even that is risky, since revealing that you have an AGI makes you a target for espionage and attack by parties who shouldn’t be trusted with humanity’s future.
Actually it reduces the chance of any party drawing massively ahead of the rest. It acts as an equalising force, by power-sharing. Since one of the main things we want to avoid is a disreputable organisation using machine intelligence to gain an advantage—and sustaining it over a long period of time. Using open-source software helps to defend against that possibility.
Machine intelligence will be a race—but it will be a race, whether participants share code or not.
Having said all that, machine intelligence protected by patents with secret source code on a server somewhere does seem like a reasonably probable outcome.
Only if (a) there is no point at which AGIs “foom”, (b) source code sharing is well enough enforced on everyone that no bad organizations combine open source with refinements that they keep secret for an advantage, (c) competing AIs form a stable power equilibrium at all points along their advancement, and (d) it is impossible to trade off goal system stability for optimization power.
I estimate probabilities of 0.4, 0.2, 0.05, and 0.5 for these hypotheses, respectively.
I disagree with most of that analysis. I assume machine intelligence will catalyse its own creation. I fully expect that some organisations will stick with secret source code. How could the probability of that possibly be as low as 0.8!?!
I figure that use of open source software is more likely to lead to a more even balance of power—and less likely to lead to a corrupt organisation in charge of the planet’s most advanced machine intelligence efforts. That assessment is mostly based on the software industry to date—where many of the worst abuses appear to me to have occurred at the hands of proprietary software vendors.
If you have an unethical open source project, people can just fork it, and make an ethical version. With a closed source project, people don’t have that option—they often have to go with whatever they are given by those in charge of the project.
Nor am I assuming that no team will ever win. If there is to be a winner, we want the best possible lead up. The “trust us” model is not it—not by a long shot.
There are two problems with this reasoning. First, you have the causality backwards: makers of open-source software are less abusive than makers of closed-source software not because open-source is such a good safeguard, but because the sorts of organizations that would be abusive don’t open source in the first place.
And second, if there is an unethical AI running somewhere, then forking the code will not save humanity. Forking is a defense against not having good software to use yourself; it is not a defense against other people running software that does bad things to you.
Really? I just provided an example of a mechanism that helps keep open source software projects ethical—the fact that if the manufacturers attempt to exploit their customers it is much easier for the customers to switch to a more ethical fork—because creating such a fork no longer violates copyright law. Though you said you were pointing out problems with my reasoning, you didn’t actually point out any problems with that reasoning.
We saw an example of this kind of thing very recently—with LibreOffice. The developers got afraid that their adopted custodian, Oracle, was going to screw the customers of their project—so, to protect their customers and themselves, they forked it—and went their own way.
If other people are running software that does bad things to you then running good quality software yourself most certainly is a kind of defense. It means you are better able to construct defenses, better able to anticipate their attacks—and so on. Better brains makes you more powerful.
Compare with the closed-source alternative: If other people are running software that does bad things to you—and you have no way to run such software yourself—since it is on their server and running secret source that is also protected by copyright law—you are probably pretty screwed.
How so? I’ve just reread some of your comments on your now deleted post. It looks like you honestly tried to get the SIAI to put safeguards into CEV. Given that the idea spread to many people by now, don’t you think it would be acceptably to discuss the matter before one or more people take it serious or even consider to implement it deliberately?
I don’t think it is a good idea to discuss it. I think that the costs outweigh the benefits. The costs are very big. Benefits marginal.
Ok by me. It is pretty obvious by this point that there is no evil conspiracy involved here. But I think the lesson remains, I you delete something, even if it is just because you regret posting it, you create more confusion than you remove.
I think the question you should be asking is less about evil conspiracies, and more about what kind of organization SIAI is—what would they tell you about, and what would they lie to you about.
If the forbidden topic would be made public (and people would believe it), it would result in a steep rise of donations towards the SIAI. That alone is enough to conclude that the SIAI is not trying to hold back something that would discredit it as an organisation concerned with charitable objectives. The censoring of the information was in accordance with their goal of trying to prevent unfriendly artificial intelligence. Making the subject matter public did already harm some people and could harm people in future.
But the forbidden topic is already public. All the effects that would follow from it being public would already follow. THE HORSE HAS BOLTED. It’s entirely unclear to me what pretending it hasn’t does for the problem or the credibility of the SIAI.
It is not as public as you think. If it was then people like waitingforgodel wouldn’t ask about it.
I’m just trying to figure out how to behave without being able talk about it directly. It’s also really interesting on many levels.
Rather more public than a long forgotten counterfactual discussion collecting dust in the blog’s history books would be. :P
Precisely. The place to hide a needle is in a large stack of needles.
The choice here was between “bad” and “worse”—a trolley problem, a lose-lose hypothetical—and they appear to have chosen “worse”.
I prefer to outsource my needle-keeping security to Clippy in exchange for allowing certain ‘bending’ liberties from time to time. :)
Upvoted for LOL value. We’ll tell Clippy the terrible, no good, very bad idea with reasons as to why this would hamper the production of paperclips.
“Hi! I see you’ve accidentally the whole uFAI! Would you like help turning it into paperclips?”
Brilliant.
Frankly, Clippy would be better than the Forbidden Idea. At least Clippy just wants paperclips.
Of course, if Clippy were clever he would then offer to sell SIAI a commitment to never release the UFAI in exchange for a commitment to produce a fixed number of paperclips per year, in perpetuity.
Admittedly, his mastery of human signaling probably isn’t nuanced enough to prevent that from sounding like blackmail.
I really don’t see how that follows. Will more of the public take it seriously? As I have noted, so far the reaction from people outside SIAI/LW has been “They did WHAT? Are they IDIOTS?”
That doesn’t make it not stupid or not counterproductive. Sincere stupidity is not less stupid than insincere stupidity. Indeed, sincere stupidity is more problematic in my experience as the sincere are less likely to back down, whereas the insincere will more quickly hop to a different idea.
Citation needed.
Citation needed.
I sent you another PM.
Hmm, okay. But that, I suggest, appears to have been a case of reasoning oneself stupid.
It does, of course, account for SIAI continuing to attempt to secure the stable doors after the horse has been dancing around in a field for several months taunting them with “COME ON IF YOU THINK YOU’RE HARD ENOUGH.”
(I upvoted XiXiDu’s comment here because he did actually supply a substantive response in PM, well deserving of a vote, and I felt this should be encouraged by reward.)
I wish I could upvote twice
A kind of meta-question: is there any evidence suggesting that one of the following explanations of the recent deletion is better than another?
That an LW moderator deleted Roko’s comment.
That Roko was asked to delete it, and complied.
That Roko deleted it himself, without prompting.
One of the more disturbing topics in this post is the question of how much can you trust an organization of people who are willing to endure torture, rape, and death for their cause.
Surely lying isn’t as bad as any of those...
Of course, lying for your cause is almost certainly a long term retarded thing to do… but so is censoring ideas...
It’s hard to know what to trust on this thread
As you may know from your study of marketing, accusations stick in the mind even when one is explicitly told they are false. In the parent comment and a sibling, you describe a hypothetical SIAI lying to its donors because… Roko had some conversations with Carl that led you to believe we care strongly about existential risk reduction?
If your aim is to improve SIAI, to cause there to be good organizations in this space, and/or to cause Less Wrong-ers to have accurate info, you might consider:
Talking with SIAI and/or with Fellows program alumni, so as to gather information on the issues you are concerned about. (I’d be happy to talk to you; I suspect Jasen and various alumni would too.) And then
Informing folks on LW of anything interesting/useful that you find out.
Anyone else who is concerned about any SIAI-related issue is also welcome to talk to me/us.
Actually that citation is about both positive and negative things—so unless you’re also asking pro-SIAI people to hush up, you’re (perhaps unknowingly) seeking to cause a pro-SIAI bias.
Another thing that citation seems to imply is that reflecting on, rather than simply diverting our attention away from scary thoughts is essential to coming to a correct opinion on them.
One of the interesting morals from Roko’s contest is that if you care deeply about getting the most benefit per donated dollar you have to look very closely at who you’re giving it to.
Market forces work really well for lightbulb-sales businesses, but not so well for mom-and-pop shops, let alone charities. The motivations, preferences, and likely future actions of the people you’re giving money to become very important. Knowing if you can believe the person, in these contexts, becomes even more important.
As you note, I’ve studied marketing, sales, propaganda, cults, and charities. I know that there are some people who have no problem lying for their cause (especially if it’s for their god or to save the world).
I also know that there are some people who absolutely suck at lying. They try to lie, but the truth just seeps out of them.
That’s why I give Roko’s blurted comments more weight than whatever I’d hear from SIAI people who were chosen by you—no offence. I’ll still talk with you guys, but I don’t think a reasonably sane person can trust the sales guy beyond a point.
As far as your question goes, my primary desire is a public, consistent moderation policy for LessWrong. If you’re going to call this a community blog devoted to rationality, then please behave in sane ways. (If no one owns the blog—if it belongs to the community—then why is there dictatorial post deletion?)
I’d also like an apology from EY with regard to the chilling effects his actions have caused.
But back to what you replied to:
What would SIAI be willing to lie to donors about?
Do you have any answers to this?
To answer your question, despite David Gerard’s advice:
I would not lie to donors about the likely impact of their donations, the evidence concerning SIAI’s ability or inability to pull off projects, how we compare to other organizations aimed at existential risk reduction, etc. (I don’t have all the answers, but I aim for accuracy and revise my beliefs and my statements as evidence comes in; I’ve actively tried to gather info on whether we or FHI reduce risk more per dollar, and I often recommend to donors that they do their own legwork with that charity comparison to improve knowledge and incentives). If a maniacal donor with a gun came searching for a Jew I had hidden in my house, or if I somehow had a “how to destroy the world” recipe and someone asked me how to use it, I suppose lying would be more tempting.
While I cannot speak for others, I suspect that Michael Vassar, Eliezer, Jasen, and others feel similarly, especially about the “not lying to one’s cooperative partners” point.
I suppose I should add “unless the actual answer is not a trolley problem” to my advice on not answering this sort of hypothetical ;-)
(my usual answer to hypotheticals is “we have no plans along those lines”, because usually we really don’t. We’re also really good at not having opinions on other organisations, e.g. Wikileaks, which we’re getting asked about A LOT because their name starts with “wiki”. A blog post on the subject is imminent. Edit: up now.)
I notice that your list is future facing.
Lies are usually about the past.
It’s very easy to not lie when talking about the future. It is much easier to “just this once” lie about the past. You can do both, for instance, by explaining that you believe a project will succeed, even while withholding information that would convince a donor otherwise.
An example of this would be errors or misconduct in completing past projects.
Lack of relevant qualifications for people SIAI plans to employ on a project.
Or administrative errors and misconduct.
Or public relations / donor outreach misconduct.
To put the question another, less abstract way, have you ever lied to a SIAI donor? Do you know of anyone affiliated with SIAI who has lied a donor?
Hypothetically, If I said I had evidence in the affirmative to the second question, how surprising would that be to you? How much money would you bet that such evidence doesn’t exist?
You’re trying very hard to get everyone to think that SIAI has lied to donors or done something equally dishonest. I agree that this is an appropriate question to discuss, but you are pursuing the matter so aggressively that I just have to ask: do you know something we don’t? Do you think that you/other donors have been lied to on a particular occasion, and if so, when?
When I asked Anna about the coordination between SIAI and FHI, something like “Do you talk enough with each other that you wouldn’t both spend resources writing the same research paper?”, she was told me about the one time that they had in fact both presented a paper on the same topic at a conference, and that they do now coordinate more to prevent that sort of thing.
I have found that Anna and others at SIAI are honest and forthcoming.
Your comment here killed the hostage.
Well, uh, yeah. The horse has bolted. It’s entirely unclear what choosing to keep one’s head in the sand gains anyone.
Although this is a reasonable question to want the answer to, it’s obvious even to me that answering at all would be silly and no sensible person who had the answer would.
Investigating the logic or lack thereof behind the (apparently ongoing) memory-holing is, however, incredibly on-topic and relevant for LW.
Total agreement here. In Eliezer’s words:
A fellow called George Orwell.
Ahh, thankyou.
I presume you’re not a native English speaker then—pretty much any moderately intelligent native English speaker has been forced to familiarity with 1984 at school. (When governments in the UK are being particularly authoritarian, there is often a call to send MPs copies of 1984 with a note “This is not a manual.”) Where are you from? Also, you really should read the book, then lots of the commentary on it :-) It’s one of the greatest works of science fiction and political fiction in English.
I can tell you all about equal pigs and newspeak but ‘memory-holing’ has not seemed to make as much of a cultural footprint—probably because as a phrase it is rather awkward fit. I wholeheartedly approve of Orwell in principle but actually reading either of his famous books sounds too much like highschool homework. :)
Animal Farm is probably passable (though it’s so short). 1984 on the other hand is maybe my favorite book of all time. I don’t think I’ve had a stronger emotional reaction to another book. It makes Shakespeare’s tragedies look like comedies. I’d imagine you’d have similar feelings about it based on what I’ve read of your comments here.
That’s some high praise there.
So I take it there isn’t a romantic ‘happily ever after’ ending? :P
Actually, there is… ;)
Both are short and enjoyable- I strongly recommend checking them out from a library or picking up a copy.
Read them. They’re actually really good books. His less-famous ones are not as brilliant, but are good too.
(We were taught 1984 in school, I promptly read to the end with eyes wide. I promptly borrowed Animal Farm of my own accord.)
His less-famous novels aren’t as good. On the other hand, some of his essays are among the clearest, most intelligent thinking I’ve ever come across, and would probably be of a lot of interest to LessWrong readers...
Oh yeah. Politics and the English Language is a classic on a par with the great two novels. I first read that in 1992 and wanted to print copies to distribute everywhere (we didn’t have internet then).
Yeah, that’s one of those I was thinking of. Also things like the piece about the PEN ‘anti-censorship’ event that wasn’t, and his analysis of James Burnham’s Managerialist writing...
I’m terribly curious now—did the use of any of the phrases Orwell singles out in the article actually drop significantly after the article was published? Wikipedia will not say...
Well, reading it in the 1990s and having a burnt-out ex-Communist for a housemate at the time, I fear I recognised far too many of the cliches therein as current in those circles ;-)
A lot are still current in those less rational/more angry elements of the left who still think the Labour Party represents socialism and use phrases like that to justify themselves...
Because this is LessWrong—you can give a sane response and not only does it clear the air, people understand and appreciate it.
Cable news debating isn’t needed here.
Sure we might still wonder if they’re being perfectly honest, but saying something more sane on the topic than silence seems like a net-positive from their perspective.
By way of a reminder, the question under discussion was:
LessWrongers are not magically free of bias. Nor are they inherently moral people that wouldn’t stoop to using misleading rhetorical techniques, though here they are more likely to be called on it.
In any case, an answer here is available to the public internet for all to see.
I respectfully disagree, and have my hopes set on Carl (or some other level-headed person in a position to know) giving a satisfying answer.
This is LessWrong after all—we can follow complicated arguments, and at least hearing how SIAI is actually thinking about such things would (probably) reduce my paranoia.
Yeah, but this is on the Internet for everyone to see. The potential for political abuse is ridiculous and can infect even LessWrong readers. Politics is the mind-killer, but pretending it doesn’t affect almost everyone else strikes me as not smart.
The concept of ethical injunctions is known in SIAI circles I think. Enduring personal harm for your cause and doing unethical things for your cause are therefore different. Consider Eliezer’s speculation about whether a rationalist “confessor” should ever lie in this post, too. And these personal struggles with whether to ever lie about SIAI’s work.
That “confessor” link is terrific
If banning Roko’s post would reasonably cause discussion of those ideas to move away from LessWrong, then by EY’s own reasoning (the link you gave) it seems like a retarded move.
Right?
If the idea is actually dangerous, it’s way less dangerous to people who aren’t familiar with pretty esoteric Lesswrongian ideas. They’re prerequisites to being vulnerable to it. So getting conversation about the idea away from Lesswrong isn’t an obviously retarded idea.
Lying for good causes has a time honored history. Protecting fugitive slaves or holocaust victims immediately comes to mind. Just because it is more often practical to be honest than not doesn’t mean that dishonesty isn’t sometimes unambiguously the better option.
I agree that there’s a lot in history, but the examples you cited have something that doesn’t match here—historically, you lie to people you don’t plan on cooperating with later.
If you lie to an oppressive government, it’s okay because it’ll either get overthrown or you’ll never want to cooperate with it (so great is your reason for lying).
Lying to your donor pool is very, very different than lying to the Nazis about hiding jews.
You’re throwing around accusations of lying pretty lightly.
Am I missing something? Desrtopa responded to questions of lying to the donor pool with the equivalent of “We do it for the greater good”
Desrtopa isn’t affiliated with SIAI. You seem to be deliberately designing confusing comments, a la Glenn Beck’s “I’m just asking questions” motif.
Is calling someone here Glenn Beck equivalent to Godwination?
wfg’s post strikes me as almost entirely reasonable (except the last question, which is pointless to ask) and your response as excessively defensive.
Also, you’re saying this to someone who says he’s a past donor and has not yet ruled out being a future donor. This is someone who could reasonably expect his questions to be taken seriously.
(I have some experience of involvement in a charity that suffers a relentless barrage of blitheringly stupid questions from idiots, and my volunteer role is media handling—mostly I come up with good and effective soundbites. So I appreciate and empathise with your frustration, but I think I can state with some experience behind me that your response is actually terrible.)
Okay. Given your and the folks who downvoted my comment’s perceptions, I’ll revise my opinion on the matter. I’ll also put that under “analogies not to use”; I was probably insufficiently familiar with the pop culture.
The thing I meant to say was just… Roko made a post, Nick suggested it gave bad impressions, Roko deleted it. wfg spent hours commenting again and again about how he had been asked to delete it, perhaps by someone “high up within SIAI”, and how future censorship might be imminent, how the fact that Roko had had a bascially unrelated conversation suggested that we might be lying to donors (a suggestion that he didn’t make explicitly, but rather left to innuendo), etc. I feel tired of this conversation and want to go back to research and writing, but I’m kind of concerned that it’ll leave a bad taste in readers mouths not because of any evidence that’s actually being advanced, but because innuendo and juxtapositions, taken out of context, leave impressions of badness.
I wish I knew how to have a simple, high-content, low-politics conversation on the subject. Especially one that was self-contained and didn’t leave me feeling as though I couldn’t bow out after awhile and return to other projects.
The essential problem is that with the (spectacular) deletion of the Forbidden Post, LessWrong turned into the sort of place where posts get disappeared. Those are not good places to be on the Internet. They are places where honesty is devalued and statements of fact must be reviewed for their political nature.
So it can happen here—because it did happen. It’s no longer in the class “things that are unthinkable”. This is itself a major credibility hit for LW.
And when a Roko post disappears—well, it was one of his posts that was disappeared before.
With this being the situation, assumptions of bad faith are going to happen. (And “stupidity” is actually the assumption of good faith.)
Your problem now is to restore trust in LW’s intellectual integrity, because SIAI broke it good and hard. Note that this is breaking an expectation, which is much worse than breaking a rule—if you break a rule you can say “we broke this rule for this reason”, but if you break expectations, people feel the ground moving under their feet, and get very upset.
There are lots of suggestions in this thread as to what people think might restore their trust in LW’s intellectual integrity, SIAI needs to go through them and work out precisely what expectations they broke and how to come clean on this.
I suspect you could at this point do with an upside to all this. Fortunately, there’s an excellent one: no-one would bother making all this fuss if they didn’t really care about LW. People here really care about LW and will do whatever they can to help you make it better.
(And the downside is that this is separate from caring about SIAI, but oh well ;-) )
(and yes, this sort of discussion around WP/WMF has been perennial since it started.)
I’ve seen several variations of this expressed about this topic, and it’s interesting to me, because this sort of view is somewhat foreign to me. I wouldn’t say I’m pro-censorship, but as an attorney trained in U.S. law, I think I’ve very much internalized the idea that the most serious sorts of censorship actions are those taken by the government (i.e., this is what the First Amendment free speech right is about, and that makes sense because of the power of the government), and that there are various levels of seriousness/danger beyond that, with say, big corporate censorship also being somewhat serious because of corporate power, and censorship by the owner of a single blog (even a community one) not being very serious at all, because a blogowner is not very powerful compared to the government or a major corporation, and shutting down one outlet of communication on the Internet is comparatively not a big deal because it’s a big internet where there are lots of other places to express one’s views. If a siteowner exercises his or her right to delete something on a website, it’s just not the sort of harm that I weigh very heavily.
What I’m totally unsure of is where the average LW reader falls on the scale between you and me, and therefore, despite the talk about the Roko incident being such a public relations disaster and a “spectacular” deletion, I just don’t know how true that is and I’m curious what the answer would be. People who feel like me may just not feel the need to weigh in on the controversy, whereas people who are very strongly anti-censorship in this particular context do.
That’s not really the crux of the issue (for me, at least, and probably not for others). As David Gerard put it, the banning of Roko’s post was a blow to people’s expectations, which was why it was so shocking. In other words, it was like discovering that LW wasn’t what everyone thought it was (and not in a good way).
Note: I personally wouldn’t classify the incident as a “disaster,” but was still very alarming.
Like Airedale, I don’t have that impression—my impression is that 1) Censorship by website’s owner doesn’t have the moral problems associated with censorship by governments (or corporations), and 2) in online communities, dictatorship can work quite well, as long as the dictator isn’t a complete dick.
I’ve seen quite functional communities where the moderators would delete posts without warning if they were too stupid, offensive, repetitive or immoral (such as bragging about vandalizing wikipedia).
So personally, I don’t see a need for “restoring trust”. Of course, as your post attests, my experience doesn’t seem to generalize to other posters.
Great post. It confuses me why this isn’t at 10+ karma
+5 is fine!
Y’know, one of the actual problems with LW is that I read it in my Internet as Television time, but there’s a REALLY PROMINENT SCORE COUNTER at the top left. This does not help in not treating it as a winnable video game.
(That said, could the people mass-downvoting waitingforgodel please stop? It’s tiresome. Please try to go by comment, not poster.)
So true!
(Except it’s at the top right. At least, the one I’m thinking of.)
The other left.
(Yes, I actually just confused left and right. STOP POSTING.)
Probably because its buried in the middle of an enormous discussion that very few people have read and will read.
Lol. right, that’d do it
This is about politics. The censorship of an idea related to a future dictator implementing some policy is obviously about politics.
You tell people to take friendly AI serious. You tell people that we need friendly AI to marshal our future galactic civilisation. People take it serious. Now the only organisation working on this is the SIAI. Therefore the SIAI is currently in direct causal control of our collective future. So why do you wonder people care about censorship and transparency? People already care about what the U.S. is doing and demand transparency. Which is ludicrous in comparison to the power of a ruling superhuman artificial intelligence that implements what the SIAI came up with as the seed for its friendliness.
If you really think that the SIAI has any importance and could possible achieve to influence or implement the safeguards for some AGI project, then everything the SIAI does is obviously very important to everyone concerned (everyone indeed).
What? No way! The organisation seems very unlikely to produce machine intelligence to me—due to all the other vastly-better funded players.
I wish you used a classification algorithm that more naturally identified the tension between “wanting low-politics conversation” and comparing someone to Glenn Beck as a means of criticism.
Sorry. This was probably simply a terrible mistake born of unusual ignorance of pop culture and current politics. I meant to invoke “using questions as a means to plant accusations” and honestly didn’t understand that he was radically unpopular. I’ve never watched anything by him.
Well, it’s not that Beck is unpopular; it’s that he’s very popular with people of a particular political ideology.
In fairness, though, he is sort of the canonical example for “I’m just asking questions, here!”. (And I wasn’t one of those voting you down on this.)
I think referring to the phenomenon itself is enough to make one’s point on the issue, and it’s not necessary to identify a person who does it a lot.
-3 after less than 15 minutes suggests so!
Make that “they do it for the greater good”
Sorry about mistakingly implying s/he was affiliated. I’ll be more diligent with my google stalking in the future.
edit: In my defense, SIAI affiliation has been very common when looking up very “pro” people from this thread
Thanks. I appreciate that.
How? That is, what tool allowed you to restore the now deleted comments? Browser cache or something more impressive?
To be more specific, when I saw that comment I assumed Roko was about to delete it and opened up a second browser window.
I caught your comment with the script, because I’ve been half sure that EY would delete this thread all day...
Ahh, gotcha.
I like the script by the way… ruby! That is my weapon of choice these days. What is the nokogiri library like? I do a fair bit of work with html automation but haven’t used that particular package.
It’s pretty nice, just a faster version of _why’s Hpricot… or a ruby version of jQuery if you’re in to that :)
What tools do you use for html automation?
hpricot, celerity, eventmachine, unadorned regex from time to time, a custom built http client and server that I built because the alternatives at the time didn’t do what I needed them to do.
That’s awesome, thanks. I’m embarrassed to say I hadn’t heard of celerity—I’m excited to try it as a turbo-charger for watir tests :)
On past occasions I admired you for the sort of extreme skepticism that Harry’s heading toward in Ch 63 of MoR (not trusting anyone’s motives and hence not toeing the LW party line).
Glad to see that you’re also a rubiest! I know it’s a long shot… but does that make you a haskeller as well?
toeing the line, not towing
Thanks. Direct feedback is always appreciated. No need for you to tiptow.
Wow. That’s my new thing learned for the day.
Why thankyou. I think. :P It sounds like I’ll enjoy catching up on the recent MoR chapters. I lost interest there for a bit when the cringe density got a tad too high but I expect it will be well worth reading when I get back into it.
Afraid not. Sure, it’s on a list of things to learn when I have time to waste but it is below learning more LISP. I do suspect I would enjoy Haskell. I tend to use similar techniques where they don’t interfere too much with being practical.
Paul Graham’s On Lisp is free and an incredible LISP reference. He builds a mind-bending prolog compiler/lisp macro toward the end. Well worth the read.
Paul Graham’s ANSI Common Lisp is good preliminary reading, and also recommended.
Paul Graham’s essays are virulent memes, and are not recommended :p
But fun! :)
Most likely either browser cache or a left-open browser tab containing the comments, being that the formatting of the line “FormallyknownasRoko | 07 December 2010 07:39:36PM* | 1 point[-]” suggests it was just copied and pasted.
Pretty much
A bit of both. I don’t maintain a mirror of lesswrong or anything, but I do use a script to make checking for such things easier.
I’d be interested to know what you were hoping for in the way of “more impressive” though :)
note that script is pretty rough—some false positives, would’t count a “[redacted]” edit as deletion (though it would cache the content).
more to avoid rescanning the page while working, etc
Depending on what you’re planning to research, lack of access to university facilities could also be a major obstacle. If you have a reputation for credible research, you might be able to collaborate with people within the university system, but I suspect that making the original break in would be pretty difficult.
For those curious: we do agree, but he went to quite a bit more effort in showing that than I did (and is similarly more convincing).
No, the rationale for deletion was not based on the possibility that his exact, FAI-based scenario could actually happen.
What was the grandparent?
Hm? Did my comment get deleted? I still see it.
I noticed you removed the content of the comment from the record on your user page. I would have preferred you not do this; those who are sufficiently curious and know about the trick of viewing the user page ought to have this option.
Only if you disagree with correctness of moderator’s decision.
Disagreement may be only partial. One could agree to the extent of thinking that viewing of the comment ought to be restricted to a more narrowly-filtered subset of readers.
Yes, this is a possible option, depending on the scope of moderator’s decision. Banning comments from a discussion, even if they a backed up and publicly available elsewhere, is still an effective tool in shaping the conversation.
There was nothing particularly important or interesting in it, just a question I had been mildly curious about. I didn’t think there was anything dangerous about it either, but, as I said elsewhere, I’m willing to take Eliezer’s word for it if he thinks it is, so I blanked it. Let it go.
I know why you did it. My intention is to register disagreement with your decision. I claim it would have sufficed to simply let Eliezer delete the comment, without you yourself taking additional action to further delete it, as it were.
I could do without this condescending phrase, which unnecessarily (and unjustifiably) imputes agitation to me.
Sorry, you’re right. I didn’t mean to imply condescension or agitation toward you; it was written in a state of mild frustration, but definitely not at or about your post in particular.
Weird. I see:
How does Eliezer’s delete option work exactly? It stays visible to the author? Now I’m curious.
Yes, I’ve been told that it was deleted but that I still see it since I’m logged in.
In that case I won’t repeat what I said in it, partly because it’ll just be deleted again but mainly because I actually do trust Eliezer’s judgment on this. (I didn’t realize that I was saying more than I was supposed to.) All I’ll say about it is that it did not actually contain the question that Eliezer’s reply suggests he thought it was asking, but it’s really not important enough to belabor the point.
yep, this one is showing as deleted
What? Did something here get deleted?
No, I think you’re nitpicking to dodge the question, and looking for a more convenient world.
I think at this point it’s clear that you really can’t be expected to give a straight answer. Well done, you win!
Have you tried?
I read your comment, understood the error you made, and it was about not seeing the picture clearly enough. If you describe the situation in terms of the components I listed, I expect you’ll see what went wrong. If you don’t oblige, I’ll probably describe the solution tomorrow.
Edit in response to severe downvoting: Seriously? It’s not allowed to entertain exercises about a conversational situation? (Besides, I was merely explaining an exercise given in another comment.) Believe, argument can be a puzzle to understand, and not a fight. If clumsy attempts to understand are discouraged, how am I supposed to develop my mastery?
If you’re genuinely unaware of the status-related implications of the way you phrased this comment, and/or of the fact that some people rate those kinds of implications negatively, let me know and I’ll try to unpack them.
If you’re simply objecting to them via rhetorical question, I’ve got nothing useful to add.
If it matters, I haven’t downvoted anyone on this thread, though I reserve the right to do so later.
I understand that status-grabbing phrasing can explain why downvotes were in fact made, but object that they should be made for that reason here, on Less Wrong. If I turn out to be wrong, then sure. There could be other reasons beside that.
Likely this, but it’s not completely clear to me what you mean.
Not as an affiliation signal, since the question is about properties of my comments, not of the people who judge them. But since you are not one of the downvoters, this says that you have less access to the reasons behind their actions than if you were one of them.
I am not one of the downvoters you are complaining about but the distinction is a temporal one, not one of differing judgement. I have since had the chance to add my downvote. That suggests my reasoning may have a slightly higher correlation at least. :)
Something I have observed is that people can often get away with status grabbing ploys but they will be held to a much higher standard while they are doing so. People will extend more grace to you when you aren’t insulting them, bizarrely enough.
I often observe that the one state of mind that leads me to sloppy thinking is that of contempt. Contempt is also the signal you were laying on thickly in your comments here and thinking displayed therein was commensurably shoddy. Not in the sense that they were internally inconsistent but in as much as they didn’t relate at all well with the comments that you were presuming to reply to. (Whether the ‘contempt’ causality is, in fact, at play is not important—it is the results that get the votes.)
I wouldn’t normally make such critiques but rhetorically or not you asked for one and this is a sincere reply.
Thank you. Contempt was not intended (or felt), I’ll try keeping this possible impression in mind to figure out where I should tune down the way I talk to communicate emotion more accurately.
Yes, it’s fascinating for me how severely reasoning can be distorted by strong emotions, but generally I feel (social) emotions more rarely than other people. When that happens, I identify motivated thoughts by dozens, and have impaired ability to think clearly. I wish there was a reproducible way of inducing such emotional experience to experiment more with those states of mind.
I don’t believe that it’s the cause. I’m generally bad at guessing what people mean, I often need being told explicitly. I don’t believe it’s the case with David Gerard’s comments in this thread though (do you disagree?). I believe it was more the case with waitingforgodel’s comments today.
I appreciate such critiques, so at least my nonexistent disapproval of them shouldn’t be a reason for making them more rarely.
Much less so with David. David also expressed himself more clearly—or perhaps instead in a more compatible idiom.
While such things are never going to be perfectly tailored for the desired effect MDMA invokes a related state. :)
I meant that if you were asking the question as a way of expressing your objections I had nothing useful to add.
Yes. Of course, if the question isn’t about the people who judge the comments, then access to those people’s motivations isn’t terribly relevant to the question.
The reasons they had for making their decisions can (should) be about my comment, not about them.
To be fair, I think the parent of the downvoted comment also has status implications:
It’s a serious accusation hurled at the wrong type of guy IMO—Vladimir probably takes the objectivity award on this forum. I think his response was justified and objective, as usual.
When someone says “look, here is this thing you did that led to these clear problems in reality” and the person they’re talking to answers “ah, but what is reality?” then the first person may reasonably consider that dodging the question.
While it’s not geared specifically towards individuals trying to do research, the (Virtual) Employment Open Thread has relevant advice for making money with little work.
If you had a paper that was good enough to get published if you were a professor then the SIAI could probably find a professor to co-author with you.
Google Scholar has greatly reduced the benefit of having access to a college library.
That depends on the field. Some fields are so riddled with paywalls that Google Scholar is all but useless; others like computer science, are much more progressive.
I think this thread perfectly illustrates one disadvantage of doing research in an unstructured environment. It is so easy to become distracted from the original question by irrelevant, but bright and shiny distractions. Having a good academic adviser cracking the whip helps to keep you on track.
855 comments so far, with no sign of slowing down!
This is a good summary.
Eh what? He did and that’s what the whole scandal was about. If you mean that he did not succesfully delete the thread from the whole internet, then yes.
Also see my other comment.
Yeah, I thought about that as well. Trying to suppress it made it much more popular and gave it a lot of credibility. If they decided to act in such a way deliberately, that be fascinating. But that sounds like one crazy conspiracy theory to me.
I don’t think it gave it a lot of credibility. Everyone I can think of who isn’t an AI researcher or LW regular who’s read it has immediately thought “that’s ridiculous. You’re seriously concerned about this as a likely consequence? Have you even heard of the Old Testament, or Harlan Ellison? Do you think your AI will avoid reading either?” Note, not the idea itself, but that SIAI took the idea so seriously it suppressed it and keeps trying to. This does not make SIAI look more credible, but less because it looks strange.
These are the people running a site about refining the art of rationality; that makes discussion of this apparent spectacular multi-level failure directly on-topic. It’s also become a defining moment in the history of LessWrong and will be in every history of the site forever. Perhaps there’s some Xanatos retcon by which this can be made to work.
I just have a hard time to believe that they could be so wrong, people who write essays like this. That’s why I allow for the possibility that they are right and that I simply do not understand the issue. Can you rule out that possibility? And if that was the case, what would it mean to spread it even further? You see, that’s my problem.
Indeed. On the other hand, humans frequently use intelligence to do much stupider things than they could have done without that degree of intelligence. Previous brilliance means that future strange ideas should be taken seriously, but not that the future ideas must be even more brilliant because they look so stupid. Ray Kurzweil is an excellent example—an undoubted genius of real achievements, but also now undoubtedly completely off the rails and well into pseudoscience. (Alkaline water!)
Ray on alkaline water:
http://glowing-health.com/alkaline-water/ray-kurzweil-alkaine-water.html
See, RationalWiki is a silly wiki full of rude people. But one thing we know a lot about, is woo. That reads like a parody of woo.
Scary.
I don’t think that’s credible. Eliezer has focused much of his intelligence on avoiding “brilliant stupidity”, orders of magnitude more so than any Kurzweil-esque example.
So the thing to do in this situation is to ask them: “excuse me wtf are you doin?” And this has been done.
So far there’s been no explanation, nor even acknowledgement of how profoundly stupid this looks. This does nothing to make them look smarter.
Of course, as I noted, a truly amazing Xanatos retcon is indeed not impossible.
There is no problem.
If you observe an action (A) that you judge so absurd that it casts doubt on the agent’s (G) rationality, then your confidence (C1) in G’s rationality should decrease. If C1 was previously high, then your confidence (C2) in your judgment of A’s absurdity should decrease.
So if someone you strongly trust to be rational does something you strongly suspect to be absurd, the end result ought to be that your trust and your suspicions are both weakened. Then you can ask yourself whether, after that modification, you still trust G’s rationality enough to believe that there exist good reasons for A.
The only reason it feels like a problem is that human brains aren’t good at this. It sometimes helps to write it all down on paper, but mostly it’s just something to practice until it gets easier.
In the meantime, what I would recommend is giving some careful thought to why you trust G, and why you think A is absurd, independent of each other. That is: what’s your evidence? Are C1 and C2 at all calibrated to observed events?
If you conclude at the end of it that they one or the other is unjustified, your problem dissolves and you know which way to jump. No problem.
If you conclude that they are both justified, then your best bet is probably to assume the existence of either evidence or arguments that you’re unaware of (more or less as you’re doing now)… not because “you can’t rule out the possibility” but because it seems more likely than the alternatives. Again, no problem.
And the fact that other people don’t end up in the same place simply reflects the fact that their prior confidence was different, presumably because their experiences were different and they don’t have perfect trust in everyone’s perfect Bayesianness. Again, no problem… you simply disagree.
Working out where you stand can be a useful exercise. In my own experience, I find it significantly diminishes my impulse to argue the point past where anything new is being said, which generally makes me happier.
This comment is also relevant.
Another thing: rationality is best expressed as a percentage, not a binary. I might look at the virtues and say “wow, I bet this guy only makes mistakes 10% of the time! That’s fantastic!”- but then when I see something that looks like a mistake, I’m not afraid to call it that. I just expect to see fewer of them.
What issue? The forbidden one? You are not even supposed to be thinking about that! For pennance, go and say 30 “Hail Yudkowskys”!
You could make a similar comment about cryonics. “Everyone I can think of who isn’t a cryonics project member or LW regular who’s read [hypothetical cryonics proposal] has immediately thought “that’s ridiculous. You’re seriously considering this possibility?”. “People think it’s ridiculous” is not always a good argument against it.
Consider that whoever made the decision probably made it according to consequentialist ethics; the consequences of people taking the idea seriously would be worse than the consequences of censorship. As many consequentialist decisions tend to, it failed to take into account the full consequences of breaking with deontological ethics (“no censorship” is a pretty strong injunction). But LessWrong is maybe the one place on the internet you could expect not to suffer for breaking from deontological ethics.
Again, strange from a deontologist’s perspective. If you’re a deontologist, okay, your objection to the practice has been noted.
The perfect Bayesian consequentialist, however, would look at the decision, estimate the chances of the decision-maker being irrational (their credibility), and promptly revise their probability estimate of ‘bad idea is actually dangerous’ upwards, enough to approve of censorship. Nothing strange there. You appear to be downgrading SIAI’s credibility because it takes an idea seriously that you don’t—I don’t think you have enough evidence to conclude that they are reasoning imperfectly.
I’m speaking of convincing people who don’t already agree with them. SIAI and LW look silly now in ways they didn’t before.
There may be, as you posit, a good and convincing explanation for the apparently really stupid behaviour. However, to convince said outsiders (who are the ones with the currencies of money and attention), the explanation has to actually be made to said outsiders in an examinable step-by-step fashion. Otherwise they’re well within rights of reasonable discussion not to be convinced. There’s a lot of cranks vying for attention and money, and an organisation has to clearly show itself as better than that to avoid losing.
By the time a person can grasp the chain of inference, and by the time they are consequentialist and Aumann-agreement-savvy enough for it to work on them, they probably wouldn’t be considered outsiders. I don’t know if there’s a way around that. It is unfortunate.
To generalise your answer: “the inferential distance is too great to show people why we’re actually right.” This does indeed suck, but is indeed not reasonably avoidable.
The approach I would personally try is furiously seeding memes that make the ideas that will help close the inferential distance more plausible. See selling ideas in this excellent post.
For what it’s worth, I gather from various comments he’s made in earlier posts that EY sees the whole enterprise of LW as precisely this “furiously seeding memes” strategy.
Or at least that this is how he saw it when he started; I realize that time has passed and people change their minds.
That is, I think he believes/ed that understanding this particular issue depends on understanding FAI theory depends on understanding cognition (or at least on dissolving common misunderstandings about cognition) and rationality, and that this site (and the book he’s working on) are the best way he knows of to spread the memes that lead to the first step on that chain.
I don’t claim here that he’s right to see it that way, merely that I think he does. That is, I think he’s trying to implement the approach you’re suggesting, given his understanding of the problem.
Well, yes. (I noted it as my approach, but I can’t see another one to approach it with.) Which is why throwing LW’s intellectual integrity under the trolley like this is itself remarkable.
Well, there’s integrity, and then there’s reputation, and they’re different.
For example, my own on-three-minutes-thought proposed approach is similar to Kaminsky’s, though less urgent. (As is, I think, appropriate… more people are working on hacking internet security than on, um, whatever endeavor it is that would lead one to independently discover dangerous ideas about AI. To put it mildly.)
I think that approach has integrity, but it won’t address the issues of reputation: adopting that approach for a threat that most people consider absurd won’t make me seem any less absurd to those people.
However, discussion of the chain of reasoning is on-topic for LessWrong (discussing a spectacularly failed local chain of reasoning and how and why it failed), and continued removal of bits of the discussion does constitute throwing LessWrong’s integrity in front of the trolley.
There are two things going on here, and you’re missing the other, important one. When a Bayesian consequentialist sees someone break a rule, they perform two operations- reduce the credibility of the person breaking the rule by the damage done, and increase the probability that the rule-breaking was justified by the credibility of the rule-breaker. It’s generally a good idea to do the credibility-reduction first.
Keep in mind that credibility is constructed out of actions (and, to a lesser extent, words), and that people make mistakes. This sounds like captainitis, not wisdom.
Aside:
Why would it matter?
You have three options, since you have two adjustments to do and you can use old or new values for each (but only three because you can’t use new values for both).* Adjusting credibility first (i.e. using the old value of the rule’s importance to determine the new credibility, then the new value of credibility to determine the new value of the credibility’s importance) is the defensive play, and it’s generally a good idea to behave defensively.
For example, let’s say your neighbor Tim (credibility .5) tells you that there are aliens out to get him (prior probability 1e-10, say). If you adjust both using the old values, you get that Tim’s credibility has dropped massively, but your belief that aliens are out to get Tim has risen massively. If you adjust the action first (where the ‘rule’ is “don’t believe in aliens having practical effects”), your belief that aliens are out to get Tim rises massively- and then your estimate of Tim’s credibility drops only slightly. If you adjust Tim’s credibility first, you find that his credibility has dropped massively, and thus when you update the probability that aliens are out to get Tim it only bumps up slightly.
*You could iterate this a bunch of times, but that seems silly.
Er, any update that doesn’t use the old values for both is just wrong. If you use new values you’re double-counting the evidence.
I suppose that could be the case- I’m trying to unpack what exactly I’m thinking of when I think of ‘credibility.’ I can see strong arguments for either approach, depending on what ‘credibility’ is. Originally I was thinking of something along the lines of “prior probability a statement they make will be correct” but as soon as you know the content of the statement, that’s not really relevant- and so now I’m imagining something along the lines of “how much I weight unlikely statements made by them,” or more likely for a real person, “how much effort i put into checking their statements.”
And so for the first one, it doesn’t make sense to update the credibility- if someone previously trustworthy tells you something bizarre, you weight it highly. But for the second one, it does make sense to update the credibility first- if someone previously trustworthy tells you something bizarre, you should immediately become more skeptical of the that statement and subsequent ones.
But no more skeptical than is warranted by your prior probability.
Let’s say that if aliens exist, a reliable Tim has a 99% probability of saying they do. If they don’t, he has a 1% probability of saying they do.
An unreliable Tim has a 50⁄50 shot in either situation.
My prior was 50⁄50 reliable/unreliable, 1,000,000⁄1 don’t exist, exist so prior weights:
reliable, exist: 1 unreliable, exist: 1 reliable, don’t exist: 1,000,000 unreliable, don’t exist, 1,000,000
Updates after he says they do:
reliable, exist: .99 unreliable, exist: .5 reliable, don’t exist: 10,000 unreliable, don’t exist: 500,000
So we now believe approximately 50 to1 that he’s unreliable, and 510,000 to 1.49 or 342,000 to 1 that they don’t exist.
This is what you get if you decide each of the new based on the old.
Thanks for working that out- that made clearer to me what I think I was confused about before. What I was imagining by “update credibility based on their statement” was configuring your credibility estimate to the statement in question- but rather than ‘updating’ that’s just doing a lookup to figure out what Tim’s credibility is for this class of statements.
Looking at shokwave’s comment again with a clearer mind:
When you estimate the chances that the decision-maker is irrational, I feel you need to include the fact that you disagree with them now (my original position of playing defensively), instead of just looking at your past.
Why? Because it reduces the chances you get stuck in a trap- if you agree with Tim on propositions 1-10 and disagree on proposition 11, you might say “well, Tim might know something I don’t, I’ll change my position to agree with his.” Then, when you disagree on proposition 12, you look back at your history and see that you agree with Tim on everything else, so maybe he knows something you don’t. Now, even though you changed your position on proposition 11, you probably did decrease Tim’s credibility- maybe you have stored “we agreed on 10 (or 10.5 or whatever) of 11 propositions.”
So, when we ask “does SIAI censor rationally?” it seems like we should take the current incident into account before we decide whether or not to take their word on their censorship. It’s also rather helpful to ask that narrower question, instead of “is SIAI rational?”, because general rationality does not translate to competence in narrow situations.
This is a subtle part of Bayesian updating. The question “does SIAI censor rationally?” is different to “was SIAI’s decision to censor this case made rationally?” (it is different because in the second case we have some weak evidence that it was not—ie, that we as rationalists would not have made the decision they did). We used our prior for “SIAI acts rationally” to determine or derive the probability of “SIAI censors rationally” (as you astutely pointed out, general rationality is not perfectly transitive), and then used “SIAI censors rationally” as our prior for the calculation of “did SIAI censor rationally in this case”.
After our calculation, “did SIAI censor rationally in this case” is necessarily going to be lower in probability than our prior “SIAI censors rationally.” Then, we can re-assess “SIAI censors rationally” in light of the fact that one of the cases of rational censorship has a higher level of uncertainty (now, our resolved disagreement is weaker evidence that SIAI does not censor rationally). That will revise “SIAI censors rationally” downwards—but not down to the level of “did SIAI censor rationally in this case”.
To use your Tim’s propositions example, you would want your estimation of proposition 12 to depend on not only how much you disagreed with him on prop 11, but also how much you agreed with him on props 1-10.
Perfect-Bayesian-Aumann-agreeing isn’t binary about agreement; it would continue to increase the value of “stuff Tim knows that you don’t” until it’s easier to reduce the value of “Tim is a perfect Bayesian reasoner about aliens”—in other words, at about prop 13-14 the hypothesis “Tim is stupid with respect to aliens existing” would occur to you, and at prop 20 “Tim is stupid WRT aliens” and “Tim knows something I don’t WRT aliens” would be equally likely.
It was left up for ages before the censorship. The Streisand effect is well known. Yes, this is a crazy kind of marketing stunt—but also one that shows Yu’El’s compassion for the tender and unprotected minds of his flock—his power over the other participants—and one that adds to the community folklore.
See, that doesn’t make sense to me. It sounds more like an initiation rite or something… not a thought experiment about quantum billionaires...
I can’t picture EY picking up the phone and saying “delete that comment! wouldn’t you willingly be tortured to decrease existential risk?”
… but maybe that’s a fact about my imagination, and not about the world :p
Context was discussing hypothetical sacrifices one would make for utilitarian humanitarian gain, not just from one but from several different conversations.
Care to share a more concrete context?
That is the context in as concrete a way as is possible—discussing what people would really be prepared to sacrifice, versus making signallingly-useful statements. I responded that I wasn’t even prepared to say that I would make {sacrifice=rape, being tortured, forgoing many years of good life, being humiliated etc}.
Okay, you can leave it abstract. Here’s what I was hoping to have explained: why were you discussing what people would really be prepared to sacrifice?
… and not just the surface level of “just for fun,” but also considering how these “just for fun” games get started, and what they do to enforce cohesion in a group.
Big +1. Every cause wants to be a cult. Every individual (or, realistically, as many as possible) must know how to resist this for a group with big goals not to go off the rails.
The context was the distinction between signalling-related speech acts and real values.
Did you see Carl Shulman’s explanation?
They actually had multiple conversations about hypothetical sacrifices they would make for utilitarian humanitarian gain? That’s… adorable!
Nothing to do with “the ugly”.
I am doing something similar, except working as a freelance software developer. My mental model is that in both the traditional academic path and the freelance path, you are effectively spending a lot of your time working for money. In academia, the “dirty work” is stuff like teaching, making PowerPoint presentations (ugh), keeping your supervisor happy, jumping through random formatting hoops to get papers published, and then going to conferences to present the papers. For me, the decisive factor is that software development is actually quite fun, while academic money work is brain-numbing.
How hard is it to live off the dole in Finland? Also, non-academic research positions in think tanks and the like (including, of course, SIAI).
Not very hard in principle, but I gather it tends to be rather stressful, with stuff like payments not arriving when they’re supposed to happening every now and then. Also, I couldn’t avoid the feeling of being a leech, justified or not.
Non-academic think tanks are a possibility, but for Singularity-related matters I can’t think of others than the SIAI, and their resources are limited.
Many people would steal food to save lives of the starving, and that’s illegal.
Working within the national support system to increase the chance of saving everybody/everything? If you would do the first, you should probably do the second. But you need to weigh the plausibility of the get-rich-and-fund-institute option, including the positive contributions of the others you could potentially hire.
I wonder how far some people would go for the cause. For Kaj, clearly, leeching of an already wasteful state is too far.
I was once criticized by a senior singinst member for not being prepared to be tortured or raped for the cause. I mean not actually, but, you know, in theory. Precommiting to being prepared to make a sacrifice that big. shrugs
Forget entirely ‘the cause’ nonsense. How far would you go just to avoid not personally getting killed? How much torture per chance that your personal contribution at the margin will prevent your near term death?
Could we move this discussion somewhere, where we don’t have to constantly worry about it getting deleted.
I’m not aware that LW moderators have ever deleted content merely for being critical of or potentially bad PR for SIAI, and I don’t think they’re naive enough to believe deletion would help. (Roko’s infamous post was considered harmful for other reasons.)
“Harmful for other reasons” still has a chilling effect on free speech… and given that those reasons were vague but had something to do with torture, it’s not unreasonable to worry about deletion of replies to the above question.
The reasons weren’t vague.
Of course this is just your assertion against mine since we’re not going to actually discuss the reasons here.
There doesn’t seem to be anything censor relevant in my question and for my part I tend to let big brother worry about his own paranoia and just go about my business. In any case while the question is an interesting one to me it doesn’t seem important enough to create a discussion somewhere else. At least not until I make a post. Putting aside presumptions of extreme altruism just how much contribution to FAI development is rational? To what extent does said rational contribution rely on newcomblike reasoning? How much would a CDT agent contribute on the expectation that his personal contribution will make the difference and save his life?
On second thoughts maybe the discussion does seem to interest me sufficiently. If you are particularly interested in answering me feel free to copy and paste my questions elsewhere and leave a back-link. ;)
I think you/we’re fine—just alternate between two tabs when replying, and paste it to the rationalwiki if it gets deleted.
Don’t let EY chill your free speech—this is supposed to be a community blog devoted to rationality… not a SIAI blog where comments are deleted whenever convenient.
Besides, it’s looking like after the Roko thing they’ve decided to cut back on such silliness.
You are compartmentalizing. What you should be asking yourself is whether the decision is correct (has better expected consequences than the available alternatives), not whether it conflicts with freedom of speech. That the decision conflicts with freedom of speech doesn’t necessarily mean that it’s incorrect, and if the correct decision conflicts with freedom of speech, or has you kill a thousand children (estimation of its correctness must of course take this consequence into account), it’s still correct and should be taken.
(There is only one proper criterion to anyone’s actions, goodness of consequences, and if any normally useful heuristic stays in the way, it has to be put down, not because one is opposed to that heuristic, but because in a given situation, it doesn’t yield the correct decision. )
(This is a note about a problem in your argument, not an argument for correctness of EY’s decision. My argument for correctness of EY’s decision is here and here.)
This is possible but by no means assured. It is also possible that he simply didn’t choose to write a full evaluation of consequences in this particular comment.
Upvoted. This just helped me get unstuck on a problem I’ve been procrastinating on.
Sounds like a good argument for WikiLeaks dilemma (which is of course confused by the possibility the government is lying their asses off about potential harm)
The question with WikiLeaks is about long-term consequences. As I understand it, the (sane) arguments in favor can be summarized as stating that expected long-term good outweighs expected short-term harm. It’s difficult (for me) to estimate whether it’s so.
I suspect it’s also difficult for Julian (or pretty much anybody) to estimate these things; I guess intelligent people will just have to make best guesses about this type of stuff. In this specific case a rationalist would be very cautious of “having an agenda”, as there is significant opportunity to do harm either way.
Very much agree btw
Shouldn’t AI researchers precommit to not build AI capable of this kind of acausal self-creation? This will lower chances of disaster both causally and acausally.
And please, define how do you tell moral heuristics and moral values apart. E.g. which is “don’t change moral values of humans by wireheading”?
We’re basically talking about a logical illusion… an AI Ontological Argument… with all the flaws of an ontological argument (such as bearing no proof)… that was foolishly censored leading to a lot of bad press, hurt feelings, lost donations, and general existential risk increase.
From, as you call it, a purely correctness optimizing perspective, It’s long term bad having silly, irrational stuff like this associated with LW. I think that EY should apologize, and we should get an explicit moderation policy for LW, but in the mean time I’ll just undo any existential risk savings hoped to be gained from censorship.
In other words, this is less about Free Speech, as it is about Dumb Censors :p
Whether it’s irrational is one of the questions we are discussing in this thread, so it’s bad conduct to use your answer as an element of an argument. I of course agree that it appears silly and irrational and absurd, and that associating that with LW and SIAI is in itself a bad idea, but I don’t believe it’s actually irrational, and I don’t believe you’ve seriously considered that question.
In other words, you don’t understand the argument, and are not moved by it, and so your estimation of improbability of the outrageous prediction stays the same. The only proper way to argue past this point is to discuss the subject matter, all else would be sophistry that equally applies to predictions of Astrology.
Following is another analysis.
Consider a die that was tossed 20 times, and each time it fell even side up. It’s not surprising because it’s a low-probability event: you wouldn’t be surprised if you observed most other combinations equally improbable under the hypothesis that the die is fair. You are surprised because a pattern you see suggests that there is an explanation for your observations that you’ve missed. You notice your own confusion.
In this case, you look at the event of censoring a post (topic), and you’re surprised, you don’t understand why that happened. And then your brain pattern matches all sorts of hypotheses that are not just improbable, but probably meaningless cached phrases, like “It’s convenient”, or “To oppose freedom of speech”, or “To manifest dictatorial power”.
Instead of leaving the choice of a hypothesis to the stupid intuitive processes, you should notice your own confusion, and recognize that you don’t know the answer. Acknowledging that you don’t know the answer is better than suggesting an obviously incorrect theory, if much more probability is concentrated outside that theory, where you can’t suggest a hypothesis.
Since we’re playing the condescension game, following is another analysis:
You read a (well written) slogan, and assumed that the writer must be irrational. You didn’t read the thread he linked you to, you focused on your first impression and held to it.
I’m not. Seriously. “Whenever convenient” is a very weak theory, and thus using it is a more serious flaw, but I missed that on first reading and addressed a different problem.
Please unpack the references. I don’t understand.
Sorry, it looks like we’re suffering from a bit of cultural crosstalk. Slogans, much like ontological arguments, are designed to make something of an illusion in the mind—a lever to change the change the way you look at the world. “Whenever convenient” isn’t there as a statement of belief, so much as a prod to get you thinking...
“How much to I trust that EY knows what he’s doing?”
You may as well argue with Nike: “Well, I can hardly do everything...” (re: Just Do It)
That said I am a rationalist… I just don’t see any harm in communicating to the best of my ability.
I linked you to this thread, where I did display some biases, but also decent evidence for not having the ones you’re describing… which I take to be roughly what you’d expect of a smart person off the street.
I can’t place this argument at all in relation to the thread above it. Looks like a collection of unrelated notes to me. Honest. (I’m open to any restatement; don’t see what to add to the notes themselves as I understand them.)
The whole post you’re replying to comes from your request to “Please unpack the references”.
Here’s the bit with references, for easy reference:
The first part of the post you’re replying to’s “Sorry, it looks… best of my ability” maps to “You read a.. irrational” in the quote above, and this tries to explain the problem as I understand it: that you were responding to a slogans words not it’s meaning. Explained it’s meaning. Explained how “Whenever convenient” was a pointer to the “Do I trust EY?” thought. Gave a backup example via the Nike slogan.
The last paragraph in the post you’re replying to tried to unpack the “you focused… held to it” from the above quote
I see. So the “writer” in the quote is you. I didn’t address your statement per se, more a general disposition of the people who state ridiculous things as explanation for the banning incident, but your comment did make the same impression on me. If you correctly disagree that it applies to your intended meaning, good, you didn’t make that error, and I don’t understand what did cause you to make that statement, but I’m not convinced by your explanation so far. You’d need to unpack “Distrusting EY” to make it clear that it doesn’t fall in the same category of ridiculous hypotheses.
The Nike slogan is “Just Do It”, if it helps.
Thanks. It doesn’t change the argument, but I’ll still delete that obnoxious paragraph.
I believe EY takes this issue very seriously.
Ahh. Are you aware of any other deletions?
Here...
I’d like to ask you the following. How would you, as an editor (moderator), handle dangerous information that are more harmful the more people know about it? Just imagine a detailed description of how to code an AGI or create bio weapons. Would you stay away from censoring such information in favor of free speech?
The subject matter here has a somewhat different nature that rather fits a more people—more probable pattern. The question is if it is better to discuss it as to possible resolve it or to censor it and thereby impede it. The problem is that this very question can not be discussed without deciding to not censor it. That doesn’t mean that people can not work on it, but rather just a few people in private. It is very likely that those people who already know about it are the most likely to solve the issue anyway. The general public would probably only add noise and make it much more likely to happen by simply knowing about it.
Step 1. Write down the clearest non-dangerous articulation of the boundaries of the dangerous idea that I could.
If necessary, make this two articulations: one that is easy to understand (in the sense of answering “is what I’m about to say a problem?”) even if it’s way overinclusive, and one that is not too overinclusive even if it requires effort to understand. Think of this as a cheap test with lots of false positives, and a more expensive follow-up test.
Add to this the most compelling explanation I can come up with of why violating those boundaries is dangerous that doesn’t itself violate those boundaries.
Step 2. Create a secondary forum, not public-access (e.g., a dangerous-idea mailing list), for the discussion of the dangerous idea. Add all the people I think belong there. If that’s more than just me, run my boundary articulation(s) past the group and edit as appropriate.
Step 3. Create a mechanism whereby people can request to be added to dangerous-idea. (e.g., sending dangerous-idea-request).
Step 4. Publish the boundary articulations, a request that people avoid any posts or comments that violate those boundaries, an overview of what steps are being taken (if any) by those in the know, and a pointer to dangerous-idea-request for anyone who feels they really ought to be included in discussion of it (with no promise of actually adding them).
Step 5. In forums where I have editorial control, censor contributions that violate those boundaries, with a pointer to the published bit in step 4.
==
That said, if it genuinely is the sort of thing where a suppression strategy can work, I would also breathe a huge sigh of relief for having dodged a bullet, because in most cases it just doesn’t.
A real-life example that people might accept the danger of would be the 2008 DNS flaw discovered by Dan Kaminsky—he discovered something really scary for the Internet and promptly assembled a DNS Cabal to handle it.
And, of course, it leaked before a fix was in place. But the delay did, they think, mitigate damage.
Note that the solution had to be in place very quickly indeed, because Kaminsky assumed that if he could find it, others could. Always assume you aren’t the only person in the whole world smart enough to find the flaw.
Yes, several times other poster’s have brought up the subject and had their comments deleted.
I hadn’t seen a lot of stubs of deleted comments around before the recent episode, but you say people’s comments had gotten deleted several times.
So, have you seen comments being deleted in a special way that doesn’t leave a stub?
Comments only leave a stub if they have replies that aren’t deleted.
Interesting. Do you have links? I rather publicly vowed to undo any assumed existential risk savings EY thought were to be had via censorship.
That one stayed up, and although I haven’t been the most vigilant in checking for deletions, I had (perhaps naively) assumed they stopped after that :-/
Hard to say. Probably a lot if I could precommit to it in advance, so that once it had began I couldn’t change my mind.
There are many complicating factors, though.
Am I the only one who can honestly say that it would depend on the day?
There’s a TED talk I once watched about how republicans reason on five moral channels and democrats only reason on two.
They were (roughly):
harm/care
fairness/reciprocity
in-group/out-group
authority
purity/scarcity/correctness
According to the talk, Democrats reason with primarily the first two and Republicans with all of them.
I took this to mean that Republicans were allowed to do moral calculus that Democrats could not… for instance, if I can only reason with the firs two, then punching a baby is always wrong (it causes harm, and isn’t fair)… If, on the other hand, I’m allowed to reason with all five, it might be okay to punch a baby because my Leader said to do it, or because the baby isn’t from my home town, or because my religion says to.
Republicans therefore have it much easier in rationalizing self-serving motives.
(As an aside, it’s interesting to note that Democrats must have started with more than just the two when they were young. “Mommy said not to” is a very good reason to do something when you’re young. It seems that they must have grown out of it).
After watching the TED talk, I was reflecting on how it seems that smart people (myself sadly included) let relatively minor moral problems stop them from doing great things… and on how if I were just a little more Republican (in the five channel moral reasoning sense) I might be able to be significantly more successful.
The result is a WFG that cycles in and out of 2-channel/5-channel reasoning.
On my 2-channel days, I’d have a very hard time hurting another person to save myself. If I saw them, and could feel that human connection, I doubt I could do much more than I myself would be willing to endure to save another’s life (perhaps two hours assuming hand-over-a-candle level of pain—permanent disfigurement would be harder to justify, but if it was relatively minor).
On my 5-channel days, I’m (surprisingly not so embarrassed to say) I’d probably go arbitrarily high… after all, what’s their life compared to mine?
Probably a bit more than you were looking to hear.
What’s your answer?
First let me say that as a Republican/libertarian I don’t entirely agree with Haidt’s analysis.
In any case, the above is not quiet how I understand Haidt’s analysis. My understanding is that Democracts have no way to categorically say that punching (or even killing) a baby is wrong. While they can say it’s wrong because as you said it causes harm and isn’t fair, they can always override that judgement by coming up with a reason why not punching and/or killing the baby would also cause harm. (See the philosophy of Peter Singer for an example).
Republicans on the other hand can invoke sanctity of life.
Sure, agreed. The way I presented it only showed very simplistic reasoning.
Let’s just say that, if you imagine a Democrat that desperately wants to do x but can’t justify it morally (punch a baby, start a somewhat shady business, not return a lost wallet full of cash), one way to resolve this conflict is to add Republican channels to his reasoning.
It doesn’t always work (sanctity of life, etc), but I think for a large number of situations where we Democrats-at-heart get cold feet it works like a champ :)
So I’ve noticed. See the discussion following this comment for an example.
On the other hand other times Democrats take positions that Republicans horrific, e.g., euthanasia, abortion, Peter Singer’s position on infanticide.
Peter Singer’s media-touted “position on infanticide” is an excellent example of why even philosophers might shy away from talking about hypotheticals in public. You appear to have just become Desrtopa’s nighmare.
My problem with Singer is that his “hypotheticals” don’t appear all that hypothetical.
What specifically are you referring to? (I haven’t been following Desporta’s posts.)
It’s evident you really need to read the post. He can’t get people to answer hypotheticals in almost any circumstances and thought this was a defect in the people. Approximately everyone responded pointing out that in the real world, the main use of hypotheticals is to use them against people politically. This would be precisely what happened with the factoid about Singer.
Thanks for the link—very interesting reading :)
Here I was thinking it was, well, nearly the opposite of that! :)
Given the current economic situation in Europe, I’m not sure that’s a good long term strategy.
Also, I suspect spending to long on the dole may cause you to develop habits that’ll make it harder to work a paying job.
Reposting comments deleted by the authors or by moderators will be considered hostile behavior and interfering with the normal and intended behavior of the site and its software, and you will be asked to leave the Less Wrong site.
-- Eliezer Yudkowsky, Less Wrong Moderator.
This decree is ambiguous enough to be seen as threatening people not to repost their banned comments (made in good faith but including too much forbidden material by accident) even after removing all objectionable content. I think this should be clarified.
Consider that clarification made; no such threat is intended.
Does this decree have a retrospective effect? And what about the private message system?
Does this only apply to comments? The reason that I ask by replying to this old comment is because I noticed that you can’t delete posts. If you delete a post it is not listed anymore and the name of the author disappears but it is still available, can be linked to using the original link and can be found via the search function.
If you want to delete a post you first have to edit it to remove its content manually.
Then I guess I’ll be asked to leave the lesswrong site.
The 0.0001% bit was a reference to my earlier precommitment
In other words, you have allegedly precommited to existential terrorism, killing the Future with small probability if your demands are not met.
WFG gave this reply which was downvoted under the default voting threshold:
I’m also reposting this just in case wfg tries to delete it or modify it later, since I think it’s important for everyone to see. Ordinarily I’d consider that a violation of netiquette, but under these here exact circumstances...
Wow, that manages to signal a willingness to use unnecessarily risky tactics, malignancy AND marginal incompetence.
While I do understand that right wing people are naturally the kind of people who bring about increased existential risks, I think my own occasional emails to left wing blogers aren’t’ that shabby (since that makes me a dirty commie enabler). In fact I email all sorts of blogers with questions and citations to papers they might be interested in.
Muhahahaha.
How does one even estimate something like 0.0001% increase of existential risk out of something like sending an email to a blogger? The error bars on the thing are vast. All you are doing is putting up a giant sign with negative characteristics that will make people want to cooperate with you less.
Agree except for the ‘terrorism’ and ‘allegedly’ part.
I just emailed a right-wing blogger some stuff that probably isn’t good for the future. Not sure what the increase was, hopefully around 0.0001%.
I’ll write it up in more detail and post a top-level discussion thread after work.
-wfg
(Shrugs.)
Your decision. The Singularity Institute does not negotiate with terrorists.
WFG, please quit with the ‘increase existential risk’ idea. Allowing Eliezer to claim moral high ground here makes the whole situation surreal.
A (slightly more) sane response would be to direct your altruistic punishment towards the SIAI specifically. They are, after all, the group who is doing harm (to you according to your values). Opposing them makes sense (given your premises.)
I don’t think my addition gives EY the high ground.
What are the points you wanted to bring up with him?
I’m all ears.
If you can think of something equally bad that targets SIAI specifically, (or anyone reading this can), email it to badforsiai.wfg@xoxy.net
Note that comments like these are still not being deleted, by the way. LW censors Langford Basilisks, not arbitrarily great levels of harmful stupidity or hostility toward the hosts—those are left to ordinary downvoting.
Are you aware of the damage your censoring is doing?
That blocking these things you think are bad (and most people do not) is causing tangible PR problems, chilling effects, SIAI/LW appearing psycho, and repeated discussion of a ancient thread?
If you weren’t you, but were instead witnessing a friend do something so utterly stupid, wouldn’t you tell them to stop?
May I at this point point out that I agree that the post in question should not appear in public. Therefore, it is a question of the author’s right to retract material, not of censorship.
That does not actually follow. Censoring what other people say is still censoring even if you happened to have said related things previously.
Yes, this other aspect.
In this case, the comment censored was not posted by you. Therefore you’re not the author.
FYI the actual author didn’t even know it was censored.
Good idea. We should’ve started using this standard reference when the censorship complaints began, but at least henceforth.
Yes. THIS IS NOT CENSORSHIP. Just in case anyone missed it.
You are evidently confused about what the word means. The systematic deletion of any content that relates to an idea that the person with power does not wish to be spoken is censorship in the same way that threatening to (probabilistically) destroy humanity is terrorism. As in, blatantly obviously—it’s just what the words happen to mean.
Going around saying ‘this isn’t censorship’ while doing it would trigger all sorts of ‘crazy cult’ warning bells.
Yes, the acts in question can easily be denoted by the terms “blackmail” and “censorship.” And your final sentence is certainly true as well.
To avoid being called a cult, to avoid being a cult, and to avoid doing bad things generally, we should stop the definition debate and focus on whether people’s behavior has been appropriate. If connotation conundrums keep you quarreling about terms, pick variables (e.g. “what EY did”=E and “what WFG precommitted to doing, and in fact did”=G) and keep talking.
YES IT IS. In case anyone missed it. It isn’t Roko’s post we’re talking about right now
There is still a moral sense in which if, after careful thought, I decided that that material should not have been posted, then any posts which resulted solely from my post are in a sense a violation of my desire to not have posted it. Especially if said posts operate under the illusion that my original post was censored rather than retracted.
But in reality such ideas tend to propagate like the imp of the perverse: a gnawing desire to know what the “censored” material is, even if everyone who knows what it is has subsequently decided that they wished they didn’t! E.g both me and Nesov have been persuaded (once fully filled in) that this is really nasty stuff and shouldn’t be let out. (correct me if I am wrong).
This “imp of the perverse” property is actually part of the reason why the original post is harmful. In a sense, this is an idea-virus which makes people who don’t yet have it want to have it, but as soon as they have been exposed to it, they (belatedly) realize they really didn’t want to know about it or spread it.
Sigh.
The only people who seem to be filled in are you and Yudkowsky. I think Nesov just argues against it based on some very weak belief. As far as I can tell, I got all the material in question. The only possible reason I can see for why one wouldn’t want to spread it is that its negative potential does outweigh its very-very-low-probability (and that only if you accept a long chain of previous beliefs). It doesn’t. It also isn’t some genuine and brilliant idea that all this mystery mongering makes it seem to be. Everyone I sent it just laughed about it. But maybe you can fill me in?
If the idea is dangerous in the first place (which is very unlikely), it is only dangerous to people who understand it, because understanding it makes you vulnerable. The better you understand it and the more you think about it, the more vulnerable you become. In hindsight, I would prefer to never have read about the idea in question.
I don’t think this is a big issue, considering the tiny probability that the scenario will ever occur, but I am glad that discussing it continues to be discouraged and would appreciate it if people stopped needlessly resurrecting it over and over again.
This strikes me as tautological and/or confusing definitions. I’m happy to agree that the idea is dangerous to people who think it is dangerous, but I don’t think it’s dangerous and I think I understand it. To make an analogy, I understand the concept of hell but don’t think it’s dangerous, and so the concept of hell does not bother me. Does the fact that I do not have the born-again Christian’s fear of hell mean that they understand it and I don’t? I don’t see why it should.
I can’t figure out a way to explain this further without repropagating the idea, which I will not do. It is likely that there are one or more pieces of the idea which you are not familiar with or do not understand, and I envy your epistemological position.
Yes, but the concept of hell is easier to understand. From what I have read in the discussions, I have no idea how the Basilisk is supposed to work, while it’s quite easy to understand how hell is supposed to work.
Upvoted, agree strongly.
Upvoted, agree strongly.
If you people are this worried about reality, why don’t you work to support creating a Paperclip maximizer? It would have a lot of fun doing what it wants to do and everyone else would quickly die. Nobody ever after would have to fear what could possible happen to them at some point.
If you people want to try to turn the universe into a better place, at whatever cost, then why do you worry or wish to not know about potential obstacles? Both is irrational.
The forbidden topic seems to be a dangerous Ugh field for a lot of people here. You have to decide what you want and then follow through on it. Any self-inflicted pain just adds to the overall negative.
You do not understand what you are talking about.
The basilisk idea has no positive value. All it does is cause those who understand it to bear a very low probability of suffering incredible disutility at some point in the future. Explaining this idea to someone does them about as much good as slashing their tires.
I understand that but do not see that the description applies to the idea in question, insofar as it is in my opinion no more probable than fiction and that any likelihood is being outweighed by opposing ideas. There are however other well-founded ideas, free speech and transparency, that are being ignored. I also believe that people would benefit from talking about it and possible overcome and ignore it subsequently.
But I’m tired of discussing this topic and will do you the favor to shut up about it. But remember that I haven’t been the one who started this thread. It was Roko and whoever asked to delete Roko’s comment.
Look, you have three people all of whom think it is a bad idea to spread this. All are smart. Two initially thought it was OK to spread it.
Furthermore, I would add that I wish I had never learned about any of these ideas. In fact, I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity; I wish very strongly that my mind had never come across the tools to inflict such large amounts of potential self-harm with such small durations of inattention, uncautiousness and/or stupidity, even if it is all premultiplied by a small probability. (not a very small one, mind you. More like 1⁄500 type numbers here)
If this is not enough warning to make you stop wanting to know more, then you deserve what you get.
I wish you’d talk to someone other than Yudkowsky about this. You don’t need anyone to harm you, you already seem to harm yourself. You indulge yourself in self-inflicted psychological stress. As Seneca said, “there are more things that terrify us than there are that oppress us, and we suffer more often in opinion than in reality”. You worry and pay interest for debt that will likely never be made.
I read about quite a few smart people who hold idiot beliefs, I only consider this to be marginal evidence.
You’d rather be some ignorant pleasure maximizing device? For me truth is the most cherished good.
BS.
More so than not opening yourself up to a small risk of severe consequences? E.g. if you found a diary that clearly belonged to some organized crime boss, would you open it up and read it? I see this situation as analogous.
Really thought you were going to go with Tom Riddle on this one. Perfect line break for it :)
You are a truth seeker? Really? I think that makes you pretty rare and unusual!
There’s a lot of truth out there. Is there any pattern to which truths you are interested in?
Yes, I’d choose to eat from the tree of the knowledge of good and evil and tell God to fuck off.
So, as a gift: 63,174,774 + 6,761,374,774 = 6,824,549,548.
Or—if you don’t like that particular truth—care to say which truths you do like?
I can’t tell you, I cherry-pick what I want to know when it is hinted at. But generally most of all I want to know about truths that other agents don’t want me to know about.
There are thousands of truths I know that I don’t want you to know about. (Or, to be more precise, that I want you to not know about.) Are you really most interested in those, out of all the truths I know?
I think I’d be disturbed by that if I thought it were true.
I’m not sure that’s a very good heuristic—are you sure that truly describes the truths you care most about? It seems analogous to the fact that people are more motivated by a cause if they learn some people opposes it, which is silly.
Heh—OK. Thanks for the reply. Yes, that is not that bad a heuristic! Maybe someday you can figure this out in more detail. It is surely good to know what you want.
I love this reply. I don’t think it’s necessarily the best reply, and I don’t really even think it’s a polite reply, but it’s certainly one of the funniest ones I’ve seen here.
I see a lot more than three people here, most of whom are smart, and most of them think that Langford basilisks are fictional, and even if they aren’t, censoring them is the wrong thing to do. You can’t quarantine the internet, and so putting up warning signs makes more people fall into the pit.
I saw the original idea and the discussion around it, but I was (fortunately) under stress at the time and initially dismissed it as so implausible as to be unworthy of serious consideration. Given the reactions to it by Eliezer, Alicorn, and Roko, who seem very intelligent and know more about this topic than I do, I’m not so sure. I do know enough to say that, if the idea is something that should be taken seriously, it’s really serious. I can tell you that I am quite happy that the original posts are no longer present, because if they were I am moderately confident that I would want to go back and see if I could make more sense out of the matter, and if Eliezer, Alicorn, and Roko are right about this, making sense out of the matter would be seriously detrimental to my health.
Thankfully, either it’s a threat but I don’t understand it fully, in which case I’m safe, or it’s not a threat, in which case I’m also safe. But I am sufficiently concerned about the possibility that it’s a threat that I don’t understand fully but might be able to realize independently given enough thought that I’m consciously avoiding extended thought about this matter. I will respond to posts that directly relate to this one but am otherwise done with this topic—rest assured that, if you missed this one, you’re really quite all right for it!
This line of argument really bothers me. What does it mean for E, A, and R to seem very intelligent? As far as I can tell, the necessary conclusion is “I will believe a controversial statement of theirs without considering it.” When you word it like that, the standards are a lot higher than “seem very intelligent”, or at least narrower- you need to know their track record on decisions like this.
(The controversial statement is “you don’t want to know about X,” not X itself, by the way.)
I am willing to accept the idea that (intelligent) specialists in a field may know more about their field than nonspecialists and are therefore more qualified to evaluate matters related to their field than I.
Good point, though I would point out that you need E, A, and R to be specialists when it comes to how people react to X, not just X, and I would say there’s evidence that’s not true.
I agree, but I know what conclusion I would draw from the belief in question if I actually believed it, so the issue of their knowledge of how people react is largely immaterial to me in particular. I was mostly posting to provide a data point in favor of keeping the material off LW, not to attempt to dissolve the issue completely or anything.
You don’t need any specific kind of proof, you already have some state of knowledge about correctness of such statements. There is no “standard of evidence” for forming a state of knowledge, it just may be that without the evidence that meets that “standard” you don’t expect to reach some level of certainty, or some level of stability of your state of knowledge (i.e. low expectation of changing your mind).
Whatever man, go ahead and make your excuses, you have been warned.
I have not only been warned, but I have stared the basilisk in the eyes, and I’m still here typing about it. In fact, I have only cared enough to do so because it was banned, and I wanted the information on how dangerous it was to judge the wisdom of the censorship.
On a more general note, being terrified of very unlikely terrible events is a known human failure mode. Perhaps it would be more effective at improving human rationality to expose people to ideas like this with the sole purpose of overcoming that sort of terror?
I’ll just second that I also read it a while back (though after it was censored) and thought that it was quite interesting but wrong on multiple levels. Not ‘probably wrong’ but wrong like an invalid logic proof is wrong (though of course I am not 100% certain of anything). My main concern about the censorship is that not talking about what was wrong with the argument will allow the proliferation of the reasoning errors that left people thinking the conclusion was plausible. There is a kind of self-fulfilling prophesy involved in not recognizing these errors which is particularly worrying.
Consider this invalid proof that 1 = 2:
You could refute this by pointing out that step (5) involved division by (x—y) = (y—y) = 0, and you can’t divide by 0.
But imagine if someone claimed that the proof is invalid because “you can’t represent numbers with letters like ‘x’ and ‘y’”. You would think that they don’t understand what is actually wrong with it, or why someone might mistakenly believe it. This is basically my reaction to everyone I have seen oppose the censorship because of some argument they present that the idea is wrong and no one would believe it.
I’m actually not sure if I understand your point. Either it is a round-about way of making it or I’m totally dense and the idea really is dangerous (or some third option).
It’s not that the idea is wrong and no one would believe it, it’s that the idea is wrong and when presented with with the explanation for why it’s wrong no one should believe it. In addition, it’s kind of important that people understand why it’s wrong. I’m sympathetic to people with different minds that might have adverse reactions to things I don’t but the solution to that is to warn them off, not censor the topics entirely.
Yes, the idea really is dangerous.
And for those who understand the idea, but not why it is wrong, nor the explanation of why it is wrong?
This is a politically reinforced heuristic that does not work for this problem.
Transparency is very important regarding people and organisations in powerful and unique positions. The way they act and what they claim in public is weak evidence in support of their honesty. To claim that they have to censor certain information in the name of the greater public good, and to fortify the decision based on their public reputation, does bear no evidence about their true objectives. The only way to solve this issue is by means of transparency.
Surely transparency might have negative consequences, but it mustn’t and can outweigh the potential risks from just believing that certain people are telling the truth and do not engage in deception to follow through on their true objectives.
There is also nothing that Yudkowsky has ever achieved that would sufficiently prove his superior intellect that would in turn justify people to just believe him about some extraordinary claim.
When I say something is a misapplied politically reinforced heuristic, you only reinforce my point by making fully general political arguments that it is always right.
Censorship is not the most evil thing in the universe. The consequences of transparency are allowed to be worse than censorship. Deal with it.
I already had Anna Salamon telling me something about politics. You sound as incomprehensible to me. Sorry, not meant as an attack.
I stated several times in the past that I am completely in favor of censorship, I have no idea why you are telling me this.
Our rules and intuitions about free speech and censorship are based on the types of censorship we usually see in practice. Ordinarily, if someone is trying to censor a piece of information, then that information falls into one of two categories: either it’s information that would weaken them politically, by making others less likely to support them and more likely to support their opponents, or it’s information that would enable people to do something that they don’t want done.
People often try to censor information that makes people less likely to support them, and more likely to support their opponents. For example, many governments try to censor embarrassing facts (“the Purple Party takes bribes and kicks puppies!”), the fact that opposition exists (“the Pink Party will stop the puppy-kicking!”) and its strength (“you can join the Pink Party, there are 10^4 of us already!”), and organization of opposition (“the Pink Party rally is tomorrow!”). This is most obvious with political parties, but it happens anywhere people feel like there are “sides”—with religions (censorship of “blasphemy”) and with public policies (censoring climate change studies, reports from the Iraq and Afghan wars). Allowing censorship in this category is bad because it enables corruption, and leaves less-worthy groups in charge.
The second common instance of censorship is encouragement and instructions for doing things that certain people don’t want done. Examples include cryptography, how to break DRM, pornography, and bomb-making recipes. Banning these is bad if the capability is suppressed for a bad reason (cryptography enables dissent), if it’s entangled with other things (general-purpose chemistry applies to explosives), or if it requires infrastructure that can also be used for the first type of censorship (porn filters have been caught blocking politicians’ campaign sites).
These two cases cover 99.99% of the things we call “censorship”, and within these two categories, censorship is definitely bad, and usually worth opposing. It is normally safe to assume that if something is being censored, it is for one of these two reasons. There are gray areas—slander (when the speaker knows he’s lying and has malicious intent), and bomb-making recipes (when they’re advertised as such and not general-purpose chemistry), for example—but the law has the exceptions mapped out pretty accurately. (Slander gets you sued, bomb-making recipes get you surveilled.) This makes a solid foundation for the principle that censorship should be opposed.
However, that principle and the analysis supporting it apply only to censorship that falls within these two domains. When things fall outside these categories, we usually don’t call them censorship; for example, there is a widespread conspiracy among email and web site administrators to suppress ads for Viagra, but we don’t call that censorship, even though it meets every aspect of the definition except motive. If you happen to find a weird instance of censorship which doesn’t fall into either category, then you have to start over and derive an answer to whether censorship in that particular case is good or bad, from scratch, without resorting to generalities about censorship-in-general. Some of the arguments may still apply—for example, building a censorship-technology infrastructure is bad even if it’s only meant to be used on spam—but not all of them, and not with the same force.
If the usual arguments against censorship don’t apply, and we’re trying to figure out whether to censor it, the next two things to test are whether it’s true, and whether an informed reader would want to see it. If both of these conditions hold, then it should not be censored. However, if either condition fails to hold, then it’s okay to censor.
Either the forbidden post is false, in which case it does not deserve protection because it’s false, or it’s true, in which case it should be censored because no informed person should want to see it. In either case, people spreading it are doing a bad thing.
Even if this is right the censorship extends to perhaps true conversations about why the post is false. Moreover, I don’t see what truth has to do with it. There are plenty of false claims made on this site that nonetheless should be public because understanding why they’re false and how someone might come to think that they are true are worthwhile endeavors.
The question here is rather straight forward: does the harm of the censorship outweigh the harm of letting people talk about the post. I can understand how you might initially think those who disagree with you are just responding to knee-jerk anti-censorship instincts that aren’t necessarily valid here. But from where I stand the arguments made by those who disagree with you do not fit this pattern. I think XiXi has been clear in the past about why the transparency concern does apply to SIAI. We’ve also seen arguments for why censorship in this particular case is a bad idea.
There are clearly more than two options here. There seem to be two points under contention:
It is/is not (1/2) reasonable to agree with the forbidden post.
It is/is not (3/4) desirable to know the contents of the forbidden post.
You seem to be restricting us to either 2+3 or 1+4. It seems that 1+3 is plausible (should we keep children from ever knowing about death because it’ll upset them?), and 2+4 seems like a good argument for restriction of knowledge (the idea is costly until you work through it, and the benefits gained from reaching the other side are lower than the costs).
But I personally suspect 2+3 is the best description, and that doesn’t explain why people trying to spread it are doing a bad thing. Should we delete posts on Pascal’s Wager because someone might believe it?
Excluded middle, of course: incorrect criterion. (Was this intended as a test?) It would not deserve protection if it were useless (like spam), not “if it were false.”
The reason I consider sufficient to keep it off LessWrong is that it actually hurt actual people. That’s pretty convincing to me. I wouldn’t expunge it from the Internet (though I might put a warning label on it), but from LW? Appropriate. Reposting it here? Rude.
Unfortunately, that’s also an argument as to why it needs serious thought applied to it, because if the results of decompartmentalised thinking can lead there, humans need to be able to handle them. As Vaniver pointed out, there are previous historical texts that have had similar effects. Rationalists need to be able to cope with such things, as they have learnt to cope with previous conceptual basilisks. So it’s legitimate LessWrong material at the same time as being inappropriate for here. Tricky one.
(To the ends of that “compartmentalisation” link, by the way, I’m interested in past examples of basilisks and other motifs of harmful sensation in idea form. Yes, I have the deleted Wikipedia article.)
Note that I personally found the idea itself silly at best.
The assertion that if a statement is not true, fails to alter political support, fails to provide instruction, and an informed reader wants to see that statement, it is therefore a bad thing to spread that statement and a OK thing to censor, is, um, far from uncontroversial.
To begin with, most fiction falls into this category. For that matter, so does most nonfiction, though at least in that case the authors generally don’t intend for it to be non-true.
No, you reversed a sign bit: it is okay to censor if an informed reader wouldn’t want to see it (and the rest of those conditions).
No, I don’t think so. You said “if either condition fails to hold, then it’s okay to censor.” If it isn’t true, and an informed reader wants to see it, then one of the two conditions failed to hold, and therefore it’s OK to censor.
No?
Oops, you’re right—one more condition is required. The condition I gave is only sufficient to show that it fails to fall into a protected class, not that it falls in the class of things that should be censored; there are things which fall in neither class (which aren’t normally censored because that requires someone with a motive to censor it, which usually puts it into one of the protected classes). To make it worthy of censorship, there must additionally be a reason outside the list of excluded reasons to censor it.
Your comment that I am replying too is often way more salient than things you have said in the past that I may or may not have observed.
I just have trouble understanding what you are saying. That might very well be my fault. I do not intent any hostile attack against you or the SIAI. I’m just curious, not worried at all. I do not demand anything. I’d like to learn more about you people, what you believe and how you arrived at your beliefs.
There is this particular case of the forbidden topic and I am throwing everything I got at it to see if the beliefs about it are consistent and hold water. That doesn’t mean that I am against censorship or that I believe it is wrong. I believe it is right but too unlikely (...). I believe that Yudkowsky and the SIAI are probably honest (although my gut feeling is to be very skeptic) but that there are good arguments for more transparency regarding the SIAI (if you believe it is as important as being portrayed). I believe that Yudkowsky is wrong about his risk estimation regarding the idea.
I just don’t understand your criticism of my past comments and that included telling me something about how I use politics (I don’t get it) and that I should accept that censorship sometimes is necessary (which I haven’t argued against).
You are just going to piss off the management.
IMO, it isn’t that interesting.
Yudkowsky apparently agrees that squashing it was handled badly.
Anyway, now Roko is out of self-imposed exile, I figure it is about time to let it drop.
The problem with that is that Eliezer and those who agree with him, including me, cannot speak freely about our reasoning on the issue, because we don’t want to spread the idea, so we don’t want to describe it and point to details about it as we describe our reasoning. If you imagine yourself in our position, believing the idea is dangerous, you could tell that you wouldn’t want to spread the idea in the process of explaining its danger either.
Under more normal circumstances, where the ideas we disagree about are not thought by anyone to be dangerous, we can have effective discussion by laying out our true reasons for our beliefs, and considering counter arguments that refer to the details of our arguments. Being cut off from our normal effective methods of discussion is stressful, at least for me.
I have been trying to persuade people who don’t know the details of the idea or don’t agree that it is dangerous that we do in fact have good reasons for believing it to be dangerous, or at least that this is likely enough that they should let it go. This is a slow process, as I think of ways to express my thoughts without revealing details of the dangerous idea, or explaining them to people who know but don’t understand those details. And this ends up involving talking to people who, because they don’t think the idea is dangerous and don’t take it seriously, express themselves faster and less carefully, and who have conflicting goals like learning or spreading the idea, or opposing censorship in general, or having judged for themselves the merits of censorship (from others just like them) in this case. This is also stressful.
I engage in this stressful topic, because I think it is important, both that people do not get hurt from learning about this idea, and that SIAI/Eliezer do not get dragged through mud for doing the right thing.
Sorry, but I am not here to help you get the full understanding you need to judge if the beliefs are consistent and hold water. As I have been saying, this is not a normal discussion. And seriously, you would be better of dropping it and finding something else to worry about. And if you think it is important, you can remember to track if SIAI/Eliezer/supporters like me engage in a pattern of making excuses to ban certain topics to protect some hidden agenda. But then please remember all the critical discussion that don’t get banned.
Note that this shouldn’t be possible other than through arguments from authority.
(I’ve just now formed a better intuitive picture of the reasons for danger of the idea, and saw some of the comments previously made unnecessarily revealing, where the additional detail didn’t actually serve the purpose of convincing people I communicated with, who lacked some of the prerequisites for being able to use that detail to understand the argument for danger, but would potentially gain (better) understanding of the idea. It does still sound silly to me, but maybe the lack of inferential stability of this conclusion should actually be felt this way—I expect that the idea will stop being dangerous in the following decades due to better understanding of decision theory.)
Does this theory of yours require that Eliezer Yudkowsky plus several other old-time Less Wrongians are holding the Idiot Ball and being really stupid about something that you can just see as obvious?
Now might be a good time to notice that you are confused.
Something to keep in mind when you reply to comments here is that you are the default leader of this community and its highest status member. This means comments that would be reasonably glib or slightly snarky from other posters can come off as threatening and condescending when made by you. They’re not really threatening but they can instill in their targets strong fight-or-flight responses. Perhaps this is because in the ancestral environment status challenges from group leaders were far more threatening to our ancestor’s livelihood than challenges from other group members. When you’re kicking out trolls it’s a sight to see, but when you’re rhetorically challenging honest interlocutors it’s probably counter-productive. I had to step away from the computer because I could tell that even if I was wrong the feelings this comment provoked weren’t going to let me admit it (and you weren’t even actually mean, just snobby).
As to your question, I don’t think my understanding of the idea requires anyone to be an idiot. In fact from what you’ve said I doubt we’re that far a part on the matter of how threatening the idea is. There may be implications I haven’t thought through that you have and there maybe general responses to implications I’ve thought of that you haven’t. I often have trouble telling how much intelligence I needed to get somewhere but I think I’ve applied a fair amount in this case. Where I think we probably diverge significantly is in our estimation of the cost of the censorship which I think is more than high enough to outweigh the risk of making Roko’s idea public. It is at least plausible that you are underestimating this cost due to biases resulting from you social position in this group and your organizational affiliation.
I’ll note that, as wedrifid suggested, your position also seems to assume that quite a few Less Wrongians are being really stupid and can’t see the obvious. Perhaps those who have expressed disagreement with your decision aren’t quite as old-time as those who have. And perhaps this is because we have not internalized important concepts or accessed important evidence required to see the danger in Roko’s idea. But it is also noteworthy that the people who have expressed disagreement have mostly been outside the Yudkowsky/SIAI cluster relative to those who have agreed with you. This suggests that they might be less susceptible to the biases that may be affecting your estimation of the cost of the censorship.
I am a bit confused as I’m not totally sure the explanations I’ve thought of or seen posted for your actions sufficiently explain them- but that’s just the kind of uncertainty one always expects in disagreements. Are you not confused? If I didn’t think there was a downside to the censorship I would let it go. But I think the downside is huge, in particular I think the censorship makes it much harder to get more people to take Friendliness seriously as a scholarly field by people beyond the SIAI circle. I’m not sure you’re humble enough to care about that (that isn’t meant as a character attack btw). It makes the field look like a joke and makes its leading scholar look ridiculous. I’m not sure you have the political talents to recognize that. It also slightly increases the chances of someone not recognizing this failure mode (the one in Roko’s post) when it counts. I think you might be so sure (or so focused on the possibility that) you’re going to be the one flipping the switch in that situation that you aren’t worried enough about that.
Repeating “But I say so!” with increasing emphasis until it works. Been taking debating lessons from Robin?
It seems to me that the natural effect of a group leader persistently arguing from his own authority is Evaporative Cooling of Group Beliefs. This is of course conducive to confirmation bias and corresponding epistemological skewing for the leader; things which seem undesirable for somebody in Eliezer’s position. I really wish that Eliezer was receptive to taking this consideration seriously.
The thing is he usually does. That is one thing that has in the past set Eliezer apart from Robin and impressed me about Eliezer. Now it is almost as though he has embraced the evaporative cooling concept as an opportunity instead of a risk and gone and bought himself a blowtorch to force the issue!
Huh, so there was a change? Curious. Certainly looking over some of Eliezer’s past writings there are some that I identify with a great deal.
Far be it from me to be anything but an optimist. I’m going with ‘exceptions’. :)
Maybe, given the credibility he has accumulated on all these other topics, you should be willing to trust him on the one issue on which he is asserting this authority and on which it is clear that if he is right, it would be bad to discuss his reasoning.
The well known (and empirically verified) weakness in experts of the human variety is that they tend to be systematically overconfident when it comes to judgements that fall outside their area of exceptional performance—particularly when the topic is one just outside the fringes.
When it comes to blogging about theoretical issues of rationality Eliezer is undeniably brilliant. Yet his credibility specifically when it comes to responding to risks is rather less outstanding. In my observation he reacts emotionally and starts making rookie mistakes of rational thought and action. To the point when I’ve very nearly responded ‘Go read the sequences!’ before remembering that he was the flipping author and so should already know better.
Also important is the fact that elements of the decision are about people, not game theory. Eliezer hopefully doesn’t claim to be an expert when it comes to predicting or eliciting optimal reactions in others.
We were talking about his credibility in judging whether this idea is a risk, and that is within his area of expertise.
Was it not clear that I do not assign particular credence to Eliezer when it comes to judging risks? I thought I expressed that with considerable emphasis.
I’m aware that you disagree with my conclusions—and perhaps even my premises—but I can assure you that I’m speaking directly to the topic.
I do not consider this strong evidence as there are many highly intelligent and productive people who hold crazy beliefs:
Francisco J. Ayala who “…has been called the “Renaissance Man of Evolutionary Biology” is a geneticist ordained as a Dominican priest. “His “discoveries have opened up new approaches to the prevention and treatment of diseases that affect hundreds of millions of individuals worldwide…”
Francis Collins (geneticist, Human Genome Project) noted for his landmark discoveries of disease genes and his leadership of the Human Genome Project (HGP) and described by the Endocrine Society as “one of the most accomplished scientists of our time” is a evangelical Christian.
Peter Duesberg (a professor of molecular and cell biology at the University of California, Berkeley) claimed that AIDS is not caused by HIV, which made him so unpopular that his colleagues and others have — until recently — been ignoring his potentially breakthrough work on the causes of cancer.
Georges Lemaître (a Belgian Roman Catholic priest) proposed what became known as the Big Bang theory of the origin of the Universe.
Kurt Gödel (logician, mathematician and philosopher) who suffered from paranoia and believed in ghosts. “Gödel, by contrast, had a tendency toward paranoia. He believed in ghosts; he had a morbid dread of being poisoned by refrigerator gases; he refused to go out when certain distinguished mathematicians were in town, apparently out of concern that they might try to kill him.”
Mark Chu-Carroll (PhD Computer Scientist, works for Google as a Software Engineer) “If you’re religious like me, you might believe that there is some deity that created the Universe.” He is running one of my favorite blogs, Good Math, Bad Math, and writes a lot on debunking creationism and other crackpottery.
Nassim Taleb (the author of the 2007 book (completed 2010) The Black Swan) does believe: Can’t track reality with science and equations. Religion is not about belief. We were wiser before the Enlightenment, because we knew how to take knowledge from incomplete information, and now we live in a world of epistemic arrogance. Religious people have a way of dealing with ignorance, by saying “God knows”.
Kevin Kelly (editor) is a devout Christian. Writes pro science and technology essays.
William D. Phillips (Nobel Prize in Physics 1997) is a Methodist.
I could continue this list with people like Ted Kaczynski or Roger Penrose. I just wanted show that intelligence and rational conduct do not rule out the possibility of being wrong about some belief.
Taleb quote doesn’t qualify. (I won’t comment on others.)
I should have made more clearly that it is not my intention to indicate that I believe that those people, or crazy ideas in general, are wrong. But there are a lot of smart people out there who’ll advocate opposing ideas. Using their reputation of being highly intelligent to follow through on their ideas is in my opinion not a very good idea in itself. I could just believe Freeman Dyson that existing simulation models of climate contain too much error to reliably predict future trends. I could believe Peter Duesberg that HIV does not cause aids, after all he is a brilliant molecular biologist. But I just do not think that any amount of reputation is enough evidence to believe extraordinary claims uttered by such people. And in the case of Yudkowsky, there doesn’t even exist much reputation and no great achievements at all that would justify some strong belief in his infallibility. What there exists in Yudkowsky’s case seems to be strong emotional commitment. I just can’t tell if he is honest. If he really believes that he’s working on a policy for some future superhuman intelligence that will rule the universe, then I’m going to be very careful. Not because it is wrong, but because such beliefs imply huge payoffs. Not that I believe he is the disguised Dr. Evil, but can we be sure enough to just trust him with it? Censorship of certain ideas does bear more evidence against him as it does in favor of his honesty.
How extensively have you searched for experts who made correct predictions outside their fields of expertise? What would you expect to see if you just searched for experts making predictions outside their field of expertise and then determined if that prediction were correct? What if you limited your search to experts who had expressed the attitude Eliezer expressed in Outside the Laboratory?
“Rule out”? Seriously? What kind of evidence is it?
You extracted the “rule out” phrase from the sentence:
From within the common phrase ‘do not rule out the possibility’ no less!
You then make a reference to ‘0 and 1s not probabilities’ with exaggerated incredulity.
To put it mildly this struck me as logically rude and in general poor form. XiXiDu deserves more courtesy.
None of this affects my point that ruling out the possibility is the wrong, (in fact impossible), standard.
Not exaggerated. XiXiDu’s post did seem to be saying: here are these examples of experts being wrong so it is possible that an expert is wrong in this case, without saying anything useful about how probable it is for this particular expert to be wrong on this particular issue.
You have made an argument accusing me of logical rudeness that, quite frankly, does not stand up to scrutiny.
-
Better evidence than I’ve ever seen in support of the censored idea. I have these well-founded principles, free speech and transparency, and weigh them against the evidence I have in favor of censoring the idea. That evidence is merely 1.) Yudkowsky’s past achievements, 2.) his output and 3.) intelligence. That intelligent people have been and are wrong about certain ideas while still being productive and right about many other ideas is evidence to weaken #3. That people lie and deceive to get what they want is evidence against #1 and #2 and in favor of transparency and free speech, which are both already more likely to have a positive impact than the forbidden topic is to have a negative impact.
And what are you trying to tell me with this link? I haven’t seen anyone stating numeric probability estimations regarding the forbidden topic. And I won’t state one either, I’ll just say that it is subjectively improbable enough to ignore it because there are possible too many very-very-low-probability events to take into account (for every being that will harm me if I don’t do X there is another being that will harm me if I do X, which cancel out each other). But if you’d like to pull some number out of thin air, go ahead. I won’t because I don’t have enough data to even calculate the probability of AI going FOOM versus a slow development.
You have failed to address my criticisms of you points, that you are seeking out only examples that support your desired conclusion, and that you are ignoring details that would allow you to construct a narrower, more relevant reference class for your outside view argument.
I was telling you the “ruling out the possibility” is the wrong, (in fact impossible), standard.
Only now I understand your criticism. I do not seek out examples to support my conclusion but to weaken your argument that one should trust Yudkowsky because of his previous output. I’m aware that Yudkowsky can very well be right about the idea but do in fact believe that the risk is worth taking. Have I done extensive research on how often people in similar situations have been wrong? Nope. No excuses here, but do you think there are comparable cases of predictions that proved to be reliable? And how much research have you done in this case and about the idea in general?
I don’t, I actually stated a few times that I do not think that the idea is wrong.
Seeking out just examples that weaken my argument, when I never predicted that no such examples would exist, is the problem I am talking about.
What made you think that supporting your conclusion and weakening my argument are different things?
My reason to weaken your argument is not that I want to be right but that I want feedback about my doubts. I said that 1.) people can be wrong, regardless of their previous reputation, 2.) that people can lie about their objectives and deceive by how they act in public (especially when the stakes are high), 3.) that Yudkowsky’s previous output and achievements are not remarkable enough to trust him about some extraordinary claim. You haven’t responded on why you tell people to believe Yudkowsky, in this case, regardless of my objections.
I’m sorry if I made it appear as if I hold some particular belief. My epistemic state simply doesn’t allow me to arrive at your conclusion. To highlight this I argued in favor of what it would mean to not accept your argument, namely to stand to previously well-established concepts like free speech and transparency. Yes, you could say that there is no difference here, except that I do not care about who is right but what is the right thing to do.
Still, it’s incorrect to argue from existence of examples. You have to argue from likelihood. You’d expect more correctness from a person with reputation for being right than from a person with reputation for being wrong.
People can also go crazy, regardless of their previous reputation, but it’s improbable, and not an adequate argument for their craziness.
And you need to know what fact you are trying to convince people about, not just search for soldier-arguments pointing in the preferred direction. If you believe that the fact is that a person is crazy, you too have to recognize that “people can be crazy” is inadequate argument for this fact you wish to communicate, and that you shouldn’t name this argument in good faith.
(Craziness is introduced as a less-likely condition than wrongness to stress the structure of my argument, not to suggest that wrongness is as unlikely.)
I notice that Yudkowsky wasn’t always self-professed human-friendly. Consider this:
http://hanson.gmu.edu/vc.html#yudkowsky
Wow. That is scary. Do you have an estimated date on that bizarre declaration? Pre 2004 I assume?
He’s changed his mind since. That makes it far, far less scary.
(Parenthetical about how changing your mind, admitting you were wrong, oops, etc, is a good thing).
(Hence reference to Eliezier2004 sequence.)
He has changed his mind about one technical point in meta-ethics. He now realizes that super-human intelligence does not automatically lead to super-human morality. He is now (IMHO) less wrong. But he retains a host of other (mis)conceptions about meta-ethics which make his intentions abhorrent to people with different (mis)conceptions. And he retains the arrogance that would make him dangerous to those he disagrees with, if he were powerful.
″… far, far less scary”? You are engaging in wishful thinking no less foolish than that for which Eliezer has now repented.
I’m not at all sure that I agree with Eliezer about most meta-ethics, and definitely disagree on some fairly important issues. But, that doesn’t make his views necessarily abhorrent. If Eliezer triggers a positive Singularity (positive in the sense that it reflects what he wants out of a Singularity, complete with CEV), I suspect that that will be a universe which I won’t mind living in. People can disagree about very basic issues and still not hate each others’ intentions. They can even disagree about long-term goals and not hate it if the other person’s goals are implemented.
Have you ever have one of those arguments with your SO in which:
It is conceded that your intentions were good.
It is conceded that the results seem good.
The SO is still pissed because of the lack of consultation and/or presence of extrapolation?
I usually escape those confrontations by promising to consult and/or not extrapolate the next time. In your scenario, Eliezer won’t have that option.
When people point out that Eliezer’s math is broken because his undiscounted future utilities leads to unbounded utility, his response is something like “Find better math—discounted utility is morally wrong”.
When Eliezer suggests that there is no path to a positive singularity which allows for prior consultation with the bulk of mankind, my response is something like “Look harder. Find a path that allows people to feel that they have given their informed consent to both the project and the timetable—anything else is morally wrong.”
ETA: In fact, I would like to see it as a constraint on the meaning of the word “Friendly” that it must not only provide friendly consequences, but also, it must be brought into existence in a friendly way. I suspect that this is one of those problems in which the added constraint actually makes the solution easier to find.
Could you link to where Eliezer says that future utilities should not be discounted? I find that surprising, since uncertainty causes an effect roughly equivalent to discounting.
I would also like to point out that achieving public consensus about whether to launch an AI would take months or years, and that during that time, not only is there a high risk of unfriendly AIs, it is also guaranteed that millions of people will die. Making people feel like they were involved in the decision is emphatically not worth the cost
He makes the case in this posting. It is a pretty good posting, by the way, in which he also points out some kinds of discounting which he believes are justified. This posting does not purport to be a knock-down argument against discounting future utility—it merely states Eliezer’s reasons for remaining unconvinced that you should discount (and hence for remaining in disagreement with most economic thinkers).
ETA: One economic thinker who disagrees with Eliezer is Robin Hanson. His response to Eliezer’s posting is also well worth reading.
Examples of Eliezer conducting utilitarian reasoning about the future without discounting are legion.
Tim Tyler makes the same assertion about the effects of uncertainty. He backs the assertion with metaphor, but I have yet to see a worked example of the math. Can you provide one?
Of course, one obvious related phenomenon—it is even mentioned with respect in Eliezer’s posting—is that the value of a promise must be discounted with time due to the increasing risk of non-performance: my promise to scratch your back tomorrow is more valuable to you than my promise to scratch next week—simply because there is a risk that you or I will die in the interim, rendering the promise worthless. But I don’t see how other forms of increased uncertainty about the future should have the same (exponential decay) response curve.
So, start now.
Most tree-pruning heuristics naturally cause an effect like temporal discounting. Resource limits mean that you can’t calculate the whole future tree—so you have to prune. Pruning normally means applying some kind of evaluation function early—to decide which branches to prune. The more you evaluate early, the more you are effectively valuing the near-present.
That is not maths—but hopefully it has a bit more detail than previously.
It doesn’t really address the question. In the A* algorithm the heuristic estimates of the objective function are supposed to be upper bounds on utility, not lower bounds. Furthermore, they are supposed to actually estimate the result of the complete computation—not to represent a partial computation exactly.
Reality check: a tree of possible futures is pruned at points before the future is completely calculated. Of course it would be nice to apply an evaluation function which represents the results of considering all possible future branches from that point on. However, getting one of those that produces results in a reasonable time would be a major miracle.
If you look at things like chess algorithms, they do some things to get a more accurate utility valuation when pruning—such as check for quiescence. However, they basically just employ a standard evaluation at that point—or sometimes a faster, cheaper approximation. If is sufficiently bad, the tree gets pruned.
We are living in the same reality. But the heuristic evaluation function still needs to be an estimate of the complete computation, rather than being something else entirely. If you want to estimate your own accumulation of pleasure over a lifetime, you cannot get an estimate of that by simply calculating the accumulation of pleasure over a shorter period—otherwise no one would undertake the pain of schooling motivated by the anticipated pleasure of high future income.
The question which divides us is whether an extra 10 utils now is better or worse than an additional 11 utils 20 years from now. You claim that it is worse. Period. I claim that it may well be better, depending on the discount rate.
Correct me if I’m missing an important nuance, but isn’t this just about whether one’s utils are timeless?
I’m not sure I understand the question. What does it mean for a util to be ‘timeless’?
ETA: The question of the interaction of utility and time is a confusing one. In “Against Discount Rates”, Eliezer writes:
I think that Eliezer has expressed the issue in almost, but not quite, the right way. The right question is whether a decision maker in 2007 should be 5% more interested in doing something about the 2008 issue than about the 2009 issue. I believe that she should be. If only because she expects that she will have an entire year in the future to worry about the 2009 family without the need to even consider 2008 again. 2008′s water will be already under the bridge.
I’m sure someone else can explain this better than me, but: As I understand it, a util understood timelessly (rather than like money, which there are valid reasons to discount because it can be invested, lost, revalued, etc. over time) builds into how it’s counted all preferences, including preferences that interact with time. If you get 10 utils, you get 10 utils, full stop. These aren’t delivered to your door in a plain brown wrapper such that you can put them in an interest-bearing account. They’re improvements in the four-dimensional state of the entire universe over all time, that you value at 10 utils. If you get 11 utils, you get 11 utils, and it doesn’t really matter when you get them. Sure, if you get them 20 years from now, then they don’t cover specific events over the next 20 years that could stand improvement. But it’s still worth eleven utils, not ten. If you value things that happen in the next 20 years more highly than things that happen later, then utils according to your utility function will reflect that, that’s all.
That (timeless utils) is a perfectly sensible convention about what utility ought to mean. But, having adopted that convention, we are left with (at least) two questions:
Do I (in 2011) derive a few percent more utility from an African family having clean water in 2012 than I do from an equivalent family having clean water in 2013?
If I do derive more utility from the first alternative, am I making a moral error in having a utility function that acts that way?
I would answer yes to the first question. As I understand it, Eliezer would answer yes to the second question and would answer no to the first, were he in my shoes. I would claim that Eliezer is making a moral error in both judgments.
Do you (in the years 2011, 2012, 2013, 2014) derive different relative utilities for these conditions? If so, it seems you have a problem.
I’m sorry. I don’t know what is meant by utility derived in 2014 from an event in 2012. I understand that the whole point of my assigning utilities in 2014 is to guide myself in making decisions in 2014. But no decision I make in 2014 can have an effect on events in 2012. So, from a decision-theoretic viewpoint, it doesn’t matter how I evaluate the utilities of past events. They are additive constants (same in all decision branches) in any computation of utility, and hence are irrelevant.
Or did you mean to ask about different relative utilities in the years before 2012? Yes, I understand that if I don’t use exponential discounting, then I risk inconsistencies.
And that is a fact about 2007 decision maker, not 2008 family’s value as compared to 2009 family.
If, in 2007, you present me with a choice of clean water for a family for all of and only 2008 vs 2009, and you further assure me that these families will otherwise survive in hardship, and that their suffering in one year won’t materially affect their next year, and that I won’t have this opportunity again come this time next year, and that flow-on or snowball effects which benefit from an early start are not a factor here—then I would be indifferent to the choice.
If I would not be; if there is something intrinsic about earlier times that makes them more valuable, and not just a heuristic of preferring them for snowballing or flow-on reasons, then that is what Eliezer is saying seems wrong.
I would classify that as instrumental discounting. I don’t think anyone would argue with that—except maybe a superintelligence who has already exhausted the whole game tree—and for whom an extra year buys nothing.
Given that you also believe that distributing your charitable giving over many charities is ‘risk management’, I suppose that should not surprise me.
FWIW, I genuinely don’t understand your perspective. The extent to which you discount the future depends on your chances of enjoying it—but also on factors like your ability to predict it—and your ability to influence it—the latter are functions of your abilities, of what you are trying to predict and of the current circumstances.
You really, really do not normally want to put those sorts of things into an agent’s utility function. You really, really do want to calculate them dynamically, depending on the agent’s current circumstances, prediction ability levels, actuator power levels, previous experience, etc.
Attempts to put that sort of thing into the utility function would normally tend to produce an inflexible agent, who has more difficulties in adapting and improving. Trying to incorporate all the dynamic learning needed to deal with the issue into the utility function might be possible in principle—but that represents a really bad idea.
Hopefully you can see my reasoning on this issue. I can’t see your reasoning, though. I can barely even imagine what it might possibly be.
Maybe you are thinking that all events have roughly the same level of unpredictability in the future, and there is roughly the same level of difficulty in influencing them, so the whole issue can be dealt with by one (or a small number of) temporal discounting “fudge factors”—and that evoution built us that way because it was too stupid to do any better.
You apparently denied that resource limitation results in temporal discounting. Maybe that is the problem (if so, see my other reply here). However, now you seem to have acknowledged that an extra year of time to worry in helps with developing plans. What I can see doesn’t seem to make very much sense.
I really, really am not advocating that we put instrumental considerations into our utility functions. The reason you think I am advocating this is that you have this fixed idea that the only justification for discounting is instrumental. So every time I offer a heuristic analogy explaining the motivation for fundamental discounting, you interpret it as a flawed argument for using discounting as a heuristic for instrumental reasons.
Since it appears that this will go on forever, and I don’t discount the future enough to make the sum of this projected infinite stream of disutility seem small, I really ought to give up. But somehow, my residual uncertainty about the future makes me think that you may eventually take Cromwell’s advice.
To clarify: I do not think the only justification for discounting is instrumental. My position is more like: agents can have whatever utility functions they like (including ones with temporal discounting) without having to justify them to anyone.
However, I do think there are some problems associated with temporal discounting. Temporal discounting sacrifices the future for the sake of the present. Sometimes the future can look after itself—but sacrificing the future is also something which can be taken too far.
Axelrod suggested that when the shadow of the future grows too short, more defections happen. If people don’t sufficiently value the future, reciprocal altruism breaks down. Things get especially bad when politicians fail to value the future. We should strive to arrange things so that the future doesn’t get discounted too much.
Instrumental temporal discounting doesn’t belong in ultimate utility functions. So, we should figure out what temporal discounting is instrumental and exclude it.
If we are building a potentially-immortal machine intelligence with a low chance of dying and which doesn’t age, those are more causes of temporal discounting which could be discarded as well.
What does that leave? Not very much, IMO. The machine will still have some finite chance of being hit by a large celestial body for a while. It might die—but its chances of dying vary over time; its degree of temporal discounting should vary in response—once again, you don’t wire this in, you let the agent figure it out dynamically.
The point is that resource limitation makes these estimates bad estimates—and you can’t do better by replacing them with better estimates because of … resource limitation!
To see how resource limitation leads to temporal discounting, consider computer chess. Powerful computers play reasonable games—but heavily resource limited ones fall for sacrifice plays, and fail to make successful sacrifice gambits. They often behave as though they are valuing short-term gain over long term results.
A peek under the hood quickly reveals why. They only bother looking at a tiny section of the game tree near to the current position! More powerful programs can afford to exhaustively search that space—and then move on to positions further out. Also the limited programs employ “cheap” evaluation functions that fail to fully compensate for their short-term foresight—since they must be able to be executed rapidly. The result is short-sighted chess programs.
That resource limitation leads to temporal discounting is a fairly simple and general principle which applies to all kinds of agents.
Why do you keep trying to argue against discounting using an example where discounting is inappropriate by definition? The objective in chess is to win. It doesn’t matter whether you win in 5 moves or 50 moves. There is no discounting. Looking at this example tells us nothing about whether we should discount future increments of utility in creating a utility function.
Instead, you need to look at questions like this: An agent plays go in a coffee shop. He has the choice of playing slowly, in which case the games each take an hour and he wins 70% of them. Or, he can play quickly, in which case the games each take 20 minutes, but he only wins 60% of them. As soon as one game finishes, another begins. The agent plans to keep playing go forever. He gains 1 util each time he wins and loses 1 util each time he loses.
The main decision he faces is whether he maximizes utility by playing slowly or quickly. Of course, he has infinite expected utility however he plays. You can redefine the objective to be maximizing utility flow per hour and still get a ‘rational’ solution. But this trick isn’t enough for the following extended problem:
The local professional offers go lessons. Lessons require a week of time away from the coffee-shop and a 50 util payment. But each week of lessons turns 1% of your losses into victories. Now the question is: Is it worth it to take lessons? How many weeks of lessons are optimal? The difficulty here is that we need to compare the values of a one-shot (50 utils plus a week not playing go) with the value of an eternal continuous flow (the extra fraction of games per hour which are victories rather than losses). But that is an infinite utility payoff from the lessons, and only a finite cost, right? Obviously, the right decision is to take a week of lessons. And then another week after that. And so on. Forever.
Discounting of future utility flows is the standard and obvious way of avoiding this kind of problem and paradox. But now let us see whether we can alter this example to capture your ‘instrumental discounting due to an uncertain future’:
First, the obvious one. Our hero expects to die someday, but doesn’t know when. He estimates a 5% chance of death every year. If he is lucky, he could live for another century. Or he could keel over tomorrow. And when he dies, the flow of utility from playing go ceases. It is very well known that this kind of uncertainty about the future is mathematically equivalent to discounted utility in a certain future. But you seemed to be suggesting something more like the following:
Our hero is no longer certain what his winning percentage will be in the future. He knows that he experiences microstrokes roughly every 6 months, and that each incident takes 5% of his wins and changes them to losses. On the other hand, he also knows that roughly every year he experiences a conceptual breakthrough. And that each such breakthrough takes 10% of his losses and turns them into victories.
Does this kind of uncertainty about the future justify discounting on ‘instrumental grounds’? My intuition says ’No, not in this case, but there are similar cases in which discounting would work.” I haven’t actually done the math, though, so I remain open to instruction.
Temporal discounting is about valuing something happening today more than the same thing happening tomorrow.
Chess computers do, in fact discount. That is why they do prefer to mate you in twenty moves rather than a hundred.
The values of a chess computer do not just tell it to win. In fact, they are complex—e.g. Deep Blue had an evaluation function that was split into 8,000 parts.
Operation consists of maximising the utility function, after foresight and tree pruning. Events that take place in branches after tree pruning has truncated them typically don’t get valued at all—since they are not forseen. Resource-limited chess computers can find themselves preferring to promote a pawn sooner rather than later. They do so since they fail to see the benefit of sequences leading to promotion later.
So: we apparently agree that resource limitation leads to indifference towards the future (due to not bothering to predict it) - but I classify this as a kind of temporal discounting (since rewards in the future get ignored), wheras you apparently don’t.
Hmm. It seems as though this has turned out to be a rather esoteric technical question about exactly which set of phenomena the term “temporal discounting” can be used to refer to.
Earlier we were talking about whether agents focussed their attention on tomorrow—rather than next year. Putting aside the issue of whether that is classified as being “temporal discounting”—or not—I think the extent to which agents focus on the near-future is partly a consequence of resource limitation. Give the agents greater abilities and more resources and they become more future-oriented.
No, I have not agreed to that. I disagree with almost every part of it.
In particular, I think that the question of whether (and how much) one cares about the future is completely prior to questions about deciding how to act so as to maximize the things one cares about. In fact, I thought you were emphatically making exactly this point on another branch.
But that is fundamental ‘indifference’ (which I thought we had agreed cannot flow from instrumental considerations). I suppose you must be talking about some kind of instrumental or ‘derived’ indifference. But I still disagree. One does not derive indifference from not bothering to predict—one instead derives not bothering to predict from being indifferent.
Furthermore, I don’t respond to expected computronium shortages by truncating my computations. Instead, I switch to an algorithm which produces less accurate computations at lower computronium costs.
And finally, regarding classification, you seem to suggest that you view truncation of the future as just one form of discounting, whereas I choose not to. And that this makes our disagreement a quibble over semantics. To which I can only reply: Please go away Tim.
I think you would reduce how far you look forward if you were interested in using your resources intelligently and efficiently.
If you only have a million cycles per second, you can’t realistically go 150 ply deep into your go game—no matter how much you care about the results after 150 moves. You compromise—limiting both depth and breadth. The reduction in depth inevitably means that you don’t look so far into the future.
A lot of our communication difficulty arises from using different models to guide our intuitions. You keep imagining game-tree evaluation in a game with perfect information (like chess or go). Yes, I understand your point that in this kind of problem, resource shortages are the only cause of uncertainty—that given infinite resources, there is no uncertainty.
I keep imagining problems in which probability is built in, like the coffee-shop-go-player which I sketched recently. In the basic problem, there is no difficulty in computing expected utilities deeper into the future—you solve analytically and then plug in whatever value for t that you want. Even in the more difficult case (with the microstrokes) you can probably come up with an analytic solution. My models just don’t have the property that uncertainty about the future arises from difficulty of computation.
Right. The real world surely contains problems of both sorts. If you have a problem which is dominated by chaos based on quantum events then more resources won’t help. Whereas with many other types of problems more resources do help.
I recognise the existence of problems where more resources don’t help—I figure you probably recognise that there are problems where more resources do help—e.g. the ones we want intelligent machines to help us with.
Perhaps the real world does. But decision theory doesn’t. The conventional assumption is that a rational agent is logically omniscient. And generalizing decision theory by relaxing that assumption looks like it will be a very difficult problem.
The most charitable interpretation I can make of your argument here is that human agents, being resource limited, imagine that they discount the future. That discounting is a heuristic introduced by evolution to compensate for those resource limitations. I also charitably assume that you are under the misapprehension that if I only understood the argument, I would agree with it. Because if you really realized that I have already heard you, you would stop repeating yourself.
That you will begin listening to my claim that not all discounting is instrumental is more than I can hope for, since you seem to think that my claim is refuted each time you provide an example of what you imagine to be a kind of discounting that can be interpreted as instrumental.
I repeat, Tim. Please go elsewhere.
I am pretty sure that I just told you that I do not think that all discounting is instrumental. Here’s what I said:
Agents can have many kinds of utility function! That is partly a consequence of there being so many different ways for agents to go wrong.
Thx for the correction. It appears I need to strengthen my claim.
Not all discounting by rational, moral agents is instrumental.
Are we back in disagreement now? :)
No, we aren’t. In my book:
Being rational isn’t about your values, you can rationally pursue practially any goal. Epistemic rationality is a bit different—but I mosly ignore that as being unbiological.
Being moral isn’t really much of a constraint at all. Morality—and right and wrong—are normally with respect to a moral system—and unless a moral system is clearly specified, you can often argue all day about what is moral and what isn’t. Maybe some types of morality are more common than others—due to being favoured by the universe, or something like that—but any such context would need to be made plain in the discussion.
So, it seems (relatively) easy to make a temporal discounting agent that really values the present over the future—just stick a term for that in its ultimate values.
Are there any animals with ultimate temporal discounting? That is tricky, but it isn’t difficult to imagine natural selection hacking together animals that way. So: probably, yes.
Do I use ultimate temporal discounting? Not noticably—as far as I can tell. I care about the present more than the future, but my temporal discounting all looks instrumental to me. I don’t go in much for thinking about saving distant galaxies, though! I hope that further clarifies.
I should probably review around about now. Instead of that: IIRC, you want to wire temporal discounting into machines, so their preferences better match your own—whereas I tend to think that would be giving them your own nasty hangover.
If you are not valuing my responses, I recommend you stop replying to them—thereby ending the discussion.
Programs make good models. If you can program it, you have a model of it. We can actually program agents that make resource-limited decisions. Having an actual program that makes decisions is a pretty good way of modeling making resource-limited decisions.
Perhaps we have some kind of underlying disagreement about what it means for temporal discounting to be “instrumental”.
In your example of an agent with suffering from risk of death, my thinking is: this player might opt for a safer life—with reduced risk. Or they might choose to lead a more interesting but more risky life. Their degree of discounting may well adjust itself accordingly—and if so, I would take that as evidence that their discounting was not really part of their pure preferences, but rather was an instrumental and dynamic response to the observed risk of dying.
If—on the other hand—they adjusted the risk level of their lifestyle, and their level of temporal discounting remained unchanged, that would be cofirming evidence in favour of the hypothesis that their temporal discounting was an innate part of their ultimate preferences—and not instrumental.
This bothers me since, with reasonable assumptions, all rational agents engage in the same amount of catastrophe discounting.
That is, observed discount rate = instrumental discount rate + chance of death + other factors
We should expect everyone’s discount rate to change, by the same amount, unless they’re irrational.
Agents do not all face the same risks, though.
Sure, they may discount the same amount if they do face the same risks, but often they don’t—e.g. compare the motorcycle racer with the nun.
So: the discounting rate is not fixed at so-much per year, but rather is a function of the agent’s observed state and capabilities.
Of course. My point is that observing if the discount rate changes with the risk tells you if the agent is rational or irrational, not if the discount rate is all instrumental or partially terminal.
Stepping back for a moment, terminal values represent what the agent really wants, and instrumental values are things sought en-route.
The idea I was trying to express was: if what an agent really wants is not temporally discounted, then instrumental temporal discounting will produce a predictable temporal discounting curve—caused by aging, mortality risk, uncertainty, etc.
Deviations from that curve would indicate the presence of terminal temporal discounting.
Agreed.
I have no disagreement at all with your analysis here. This is not fundamental discounting. And if you have decision alternatives which affect the chances of dying, then it doesn’t even work to model it as if it were fundamental.
You recently mentioned the possibility of dying in the interim. There’s also the possibility of aging in the interim. Such factors can affect utility calculations.
For example: I would much rather have my grandmother’s inheritance now than years down the line, when she finally falls over one last time—because I am younger and fitter now.
Significant temporal discounting makes sense sometimes—for example, if there is a substantial chance of extinction per unit time. I do think a lot of discounting is instrumental, though—rather than being a reflection of ultimate values—due to things like the future being expensive to predict and hard to influence.
My brain spends more time thinking about tomorrow than about this time next year—because I am more confident about what is going on tomorrow, and am better placed to influence it by developing cached actions, etc. Next year will be important too—but there will be a day before to allow me to prepare for it closer to the time, when I am better placed to do so. The difference is not because I will be older then—or because I might die in the mean time. It is due to instrumental factors.
Of course one reason this is of interest is because we want to know what values to program into a superintelligence. That superintelligence will probably not age—and will stand a relatively low chance of extinction per unit time. I figure its ultimate utility function should have very little temporal discounting.
The problem with wiring discount functions into the agent’s ultimate utility function is that that is what you want it to preserve as it self improves. Much discounting is actually due to resource limitation issues. It makes sense for such discounting to be dynamically reduced as more resources become cheaply available. It doesn’t make much sense to wire-in short-sightedness.
I don’t mind tree-pruning algorithms attempting to normalise partial evaluations at different times—so they are more directly comparable to each other. The process should not get too expensive, though—the point of tree pruning is that it is an economy measure.
I suspect you want to replace “feel like they have given” with “give.”
Unless you are actually claiming that what is immoral is to make people fail to feel consulted, rather than to fail to consult them, which doesn’t sound like what you’re saying.
I think I will go with a simple tense change: “feel that they are giving”. Assent is far more important in the lead-up to the Singularity than during the aftermath.
Although I used the language “morally wrong”, my reason for that was mostly to make the rhetorical construction parallel. My preference for an open, inclusive process is a strong preference, but it is really more political/practical than moral/idealistic. One ought to allow the horses to approach the trough of political participation, if only to avoid being trampled, but one is not morally required to teach them how to drink.
Ah, I see. Sure, if you don’t mean morally wrong but rather politically impractical, then I withdraw my suggestion… I entirely misunderstood your point.
No, I did originally say (and mostly mean) “morally” rather than “politically”. And I should thank you for inducing me to climb down from that high horse.
I submit that I have many of the same misconceptions that Eliezer does; he changed his mind about one of the few places I disagree with him. That makes it far more of a change than it would be for you (one out of eight is a small portion, one out of a thousand is an invisible fraction).
Good point. And since ‘scary’ is very much a subjective judgment, that mean that I can’t validly criticize you for being foolish unless I have some way of arguing that yours and Eliezer’s positions in the realm of meta-ethics are misconceptions—something I don’t claim to be able to do.
So, if I wish my criticisms to be objective, I need to modify them. Eliezer’s expressed positions on meta-ethics (particularly his apparent acceptance of act-utilitarianism and his unwillingness to discount future utilities) together with some of his beliefs regarding the future (particularly his belief in the likelihood of a positive singularity and expansion of human population into the universe) make his ethical judgments completely unpredictable to many other people—unpredictable because the judgment may turn on subtle differences in the expect consequences of present day actions on people in the distant future. And, if one considers the moral judgments of another personal to be unpredictable, and that person is powerful, then one ought to consider that person scary. Eliezer is probably scary to many people.
True, but it has little bearing on whether Eliezer should be scary. That is, “Eliezer is scary to many people” is mostly a fact about many people, and mostly not a fact about Eliezer. The reverse of this (and what I base this distinction on) is that some politicians should be scary, and are not scary to many people.
I’m not sure the proposed modification helps: you seem to have expanded your criticism so far, in order to have them lead to the judgment you want to reach, that they cover too much.
I mean, sure, unpredictability is scarier (for a given level of power) than predictability. Agreed, But so what?
For example, my judgments will always be more unpredictable to people much stupider than I am than to people about as smart or smarter than I am. So the smarter I am, the scarier I am (again, given fixed power)… or, rather, the more people I am scary to… as long as I’m not actively devoting effort to alleviating those fears by, for example, publicly conforming to current fashions of thought. Agreed.
But what follows from that? That I should be less smart? That I should conform more? That I actually represent a danger to more people? I can’t see why I should believe any of those things.
You started out talking about what makes one dangerous; you have ended up talking about what makes people scared of one whether one is dangerous or not. They aren’t equivalent.
Well, I hope I haven’t done that.
Well, I certainly did that. I was trying to address the question more objectively, but it seems I failed. Let me try again from a more subjective, personal position.
If you and I share the same consequentialist values, but I know that you are more intelligent, I may well consider you unpredictable, but I won’t consider you dangerous. I will be confident that your judgments, in pursuit of our shared values, will be at least as good as my own. Your actions may surprise me, but I will usually be pleasantly surprised.
If you and I are of the same intelligence, but we have different consequentialist values (both being egoists, with disjoint egos, for example) then we can expect to disagree on many actions. Expecting the disagreement, we can defend ourselves, or even bargain our way to a Nash bargaining solution in which (to the extent that we can enforce our bargain) we can predict each others behavior to be that promoting compromise consequences.
If, in addition to different values, we also have different beliefs, then bargaining is still possible, though we cannot expect to reach a Pareto optimal bargain. But the more our beliefs diverge, regarding consequences that concern us, the less good our bargains can be. In the limit, when the things that matter to us are particularly difficult to predict, and when we each have no idea what the other agent is predicting, bargaining simply becomes ineffective.
Eliezer has expressed his acceptance of the moral significance of the utility functions of people in the far distant future. Since he believes that those people outnumber us folk in the present, that seems to suggest that he would be willing to sacrifice the current utility of us in favor of the future utility of them. (For example, the positive value of saving a starving child today does not outweigh the negative consequences on the multitudes of the future of delaying the Singularity by one day).
I, on the other hand, systematically discount the future. That, by itself, does not make Eliezer dangerous to me. We could strike a Nash bargain, after all. However, we inevitably also have different beliefs about consequences, and the divergence between our beliefs becomes greater the farther into the future we look. And consequences in the distant future are essentially all that matters to people like Eliezer—the present fades into insignificance by contrast. But, to people like me, the present and near future are essentially all that matter—the distant future discounts into insignificance.
So, Eliezer and I care about different things. Eliezer has some ability to predict my actions because he knows I care about short-term consequences and he knows something about how I predict short-term consequences. But I have little ability to predict Eliezer’s actions, because I know he cares primarily about long term consequences, and they are inherently much more unpredictable. I really have very little justification for modeling Eliezer (and any other act utilitarian who refuses to discount the future) as a rational agent.
I wish you would just pretend that they care about things a million times further into the future than you do.
The reason is that there are instrumental reasons to discount—the future disappears into a fog of uncertainty—and you can’t make decisions based on the value of things you can’t forsee.
The instrumental reasons fairly quickly dominate as you look further out—even when you don’t discount in your values. Reading your post, it seems as though you don’t “get” this, or don’t agree with it—or something.
Yes, the far-future is unpredictable—but in decision theory, that tends to make it a uniform grey—not an unpredictable black and white strobing pattern.
I don’t need to pretend. Modulo some mathematical details, it is the simple truth. And I don’t think there is anything irrational about having such preferences. It is just that, since I cannot tell whether or not what I do will make such people happy, I have no motive to pay any attention to their preferences.
Yet, it seems that the people who care about the future do not agree with you on that. Bostrom, Yudkowsky, Nesov, et al. frequently invoke assessments of far-future consequences (sometimes in distant galaxies) in justifying their recommendations.
We have crossed wires here. What I meant is that I wish you would stop protesting about infinite utilities—and how non-discounters are not really even rational agents—and just model them as ordinary agents who discount a lot less than you do.
Objections about infinity strike me as irrelevant and uninteresting.
Is that your true objection? I expect you can figure out what would make these people happy fairly easily enough most of the time—e.g. by asking them.
Indeed. That is partly poetry, though (big numbers make things seem important) - and partly because they think that the far future will be highly contingent on near future events.
The thing they are actually interested in influencing is mostly only a decade or so out. It does seem quite important—significant enough to reach back to us here anyway.
If what you are trying to understand is far enough away to be difficult to predict, and very important, then that might cause some oscillations. That is hardly a common situation, though.
Most of the time, organisms act as though want to become ancestors. To do that, the best thing they can do is focus on having some grandkids. Expanding their circle of care out a few generations usually makes precious little difference to their actions. The far future is unforseen, and usually can’t be directly influenced. It is usually not too relevant. Usually, you leave it to your kids to deal with.
That is a valid point. So, I am justified in treating them as rational agents to the extent that I can engage in trade with them. I just can’t enter into a long-term Nash bargain with them in which we jointly pledge to maximize some linear combination of our two utility functions in an unsupervised fashion. They can’t trust me to do what they want, and I can’t trust them to judge their own utility as bounded.
I think this is back to the point about infinities. The one I wish you would stop bringing up—and instead treat these folk as though they are discounting only a teeny, tiny bit.
Frankly, I generally find it hard to take these utilitarian types seriously in the first place. A “signalling” theory (holier-than-thou) explains the unusually high prevalance of utilitarianism among moral philosophers—and an “exploitation” theory explains its prevalance among those running charitable causes (utilitarianism-says-give-us-your-money). Those explanations do a good job of modelling the facts about utilitarianism—and are normally a lot more credible than the supplied justifications—IMHO.
Which suggests that we are failing to communicate. I am not surprised.
I do that! And I still discover that their utility functions are dominated by huge positive and negative utilities in the distant future, while mine are dominated by modest positive and negative utilities in the near future. They are still wrong even if they fudge it so that their math works.
I went from your “I can’t trust them to judge their own utility as bounded” to your earlier “infinity” point. Possibly I am not trying very hard here, though...
My main issue was you apparently thinking that you couldn’t predict their desires in order to find mutually beneficial trades. I’m not really sure if this business about not being able to agree to maximise some shared function is a big deal for you.
Mm. OK, so you are talking about scaring sufficiently intelligent rationalists, not scaring the general public. Fair enough.
What you say makes sense as far as it goes, assuming some mechanism for reliable judgments about people’s actual bases for their decisions. (For example, believing their self-reports.)
But it seems the question that should concern you is not whether Eliezer bases his decisions on predictable things, but rather whether Eliezer’s decisions are themselves predictable.
Put a different way: by your own account, the actual long-term consequences don’t correlate reliably with Eliezer’s expectations about them… that’s what it means for those consequences to be inherently unpredictable. And his decisions are based on his expectations, of course, not on the actual future consequences. So it seems to follow that once you know Eliezer’s beliefs about the future, whether those beliefs are right or wrong is irrelevant to you: that just affects what actually happens in the future, which you systematically discount anyway.
So if Eliezer is consistent in his beliefs about the future, and his decisions are consistently grounded in those beliefs, I’m not sure what makes him any less predictable to me than you are.
Of course, his expectations might not be consistent. Or they might be consistent but beyond your ability to predict. Or his decisions might be more arbitrary than you suggest here. For that matter, he might be lying outright. I’m not saying you should necessarily trust him, or anyone else.
But those same concerns apply to everybody, whatever their professed value structure. I would say the same things about myself.
But Eliezer’s beliefs about the future continue to change—as he gains new information and completes new deductions. And there is no way that he can practically keep me informed of his beliefs—neither he nor I would be willing to invest the time required for that communication. But Eliezer’s beliefs about the future impact his actions in the present, and those actions have consequences both in the near and distant future. From my point of view, therefore, his actions have essentially random effects on the only thing that matters to me—the near future.
Absolutely. But who isn’t that true of? At least Eliezer has extensively documented his putative beliefs at various points in time, which gives you some data points to extrapolate from.
I have no complaints regarding the amount of information about Eliezer’s beliefs that I have access to. My complaint is that Eliezer, and his fellow non-discounting act utilitarians, are morally driven by the huge differences in utility which they see as arising from events in the distant future—events which I consider morally irrelevant because I discount the future. No realistic amount of information about beliefs can alleviate this problem. The only fix is for them to start discounting. (I would have added “or for me to stop discounting” except that I still don’t know how to handle the infinities.)
Given that they predominantly care about things I don’t care about, and that I predominantly care about things they don’t worry about, we can only consider each other to be moral monsters.
You and I seem to be talking past each other now. It may be time to shut this conversation down.
Ethical egoists are surely used to this situation, though. The world is full of people who care about extremely different things from one another.
Yes. And if they both mostly care about modest-sized predictable things, then they can do some rational bargaining. Trouble arises when one or more of them has exquisitely fragile values—when they believe that switching a donation from one charity to another destroys galaxies.
I expect your decision algorithm will find a way to deal with people who won’t negotiate on some topics—or who behave in manner you have a hard time predicting. Some trouble for you, maybe—but probably not THE END OF THE WORLD.
Looking at the last 10 years, there seems to be some highly-predictable fund raising activity, and a lot of philosophising about the importance of machine morality.
I see some significant patterns there. It is not remotely like a stream of random events. So: what gives?
Sure, the question of whether a superintelligence will construct a superior morality to that which natural selection and cultural evolution have constructed on Earth is in some sense a narrow technical question. (The related question of whether the phrase “superior morality” even means anything is, also.)
But it’s a technical question that pertains pretty directly to the question of whose side one envisions oneself on.
That is, if one answers “yes,” it can make sense to ally with the Singularity rather than humanity (assuming that even means anything) as EY-1998 claims to, and still expect some unspecified good (or perhaps Good) result. Whereas if one answers “no,” or if one rejects the very idea that there’s such a thing as a superior morality, that justification for alliance goes away.
That said, I basically agree with you, though perhaps for different reasons than yours.
That is, even after embracing the idea that no other values, even those held by a superintelligence, can be superior to human values, one is still left with the same choice of alliances. Instead of “side with humanity vs. the Singularity,” the question involves a much narrower subset: “side with humanity vs. FAI-induced Singularity,” but from our perspective it’s a choice among infinities.
Of course, advocates of FAI-induced Singularity will find themselves saying that there is no conflict, really, because an FAI-induced Singularity will express by definition what’s actually important about humanity. (Though, of course, there’s no guarantee that individual humans won’t all be completely horrified by the prospect.)
Recall that after CEV extrapolates current humans’ volitions and construes a coherent superposition, the next step isn’t “do everything that superposition says”, but rather, “ask that superposition the one question ‘Given the world as it is right now, what program should we run next?’, run that program, and then shut down”. I suppose it’s possible that our CEV will produce an AI that immediately does something we find horrifying, but I think our future selves are nicer than that… or could be nicer than that, if extrapolated the right way, so I’d consider it a failure of Friendliness if we get a “do something we’d currently find horrifying for the greater good” AI if a different extrapolation strategy would have resulted in something like a “start with the most agreeable and urgent stuff, and other than that, protect us while we grow up and give us help where we need it” AI.
I really doubt that we’d need an AI to do anything immediately horrifying to the human species in order to allow it to grow up into an awesome fun posthuman civilization, so if CEV 1.0 Beta 1 appeared to be going in that direction, that would probably be considered a bug and fixed.
(shrug) Sure, if you’re right that the “most urgent and agreeable stuff” doesn’t happen to press a significant number of people’s emotional buttons, then it follows that not many people’s emotional buttons will be pressed.
But there’s a big difference between assuming that this will be the case, and considering it a bug if it isn’t.
Either I trust the process we build more than I trust my personal judgments, or I don’t.
If I don’t, then why go through this whole rigamarole in the first place? I should prefer to implement my personal judgments. (Of course, I may not have the power to do so, and prefer to join more powerful coalitions whose judgments are close-enough to mine. But in that case CEV becomes a mere political compromise among the powerful.)
If I do, then it’s not clear to me that “fixing the bug” is a good idea.
That is, OK, suppose we write a seed AI intended to work out humanity’s collective CEV, work out some next-step goals based on that CEV and an understanding of likely consequences, construct a program P to implement those goals, run P, and quit.
Suppose that I am personally horrified by the results of running P. Ought I choose to abort P? Or ought I say to myself “Oh, how interesting: my near-mode emotional reactions to the implications of what humanity really wants are extremely negative. Still, most everybody else seems OK with it. OK, fine: this is not going to be a pleasant transition period for me, but my best guess is still that it will ultimately be for the best.”
Is there some number of people such that if more than that many people are horrified by the results, we ought to choose to abort P?
Does the question even matter? The process as you’ve described it doesn’t include an abort mechanism; whichever choice we make P is executed.
Ought we include such an abort mechanism? It’s not at all clear to me that we should. I can get on a roller-coaster or choose not to get on it, but giving me a brake pedal on a roller coaster is kind of ridiculous.
It’s partly a chance vs necessity question.
It is partly a question about whether technological determinism is widespread.
Apparently he changed his mind about a bunch of things.
On what appears to be their current plan, the SIAI, don’t currently look very dangerous, IMHO.
Eray Ozkural recently complained: “I am also worried that backwards people and extremists will threaten us, and try to dissuade us from accomplishing our work, due to your scare tactics.”
I suppose that sort of thing is possible—but my guess is that they are mostly harmless.
Or so you hope.
Yes, I agree. I don’t really believe that he only learnt how to disguise his true goals. But I’m curious if you would be satisfied with his word alone if he would be able to run a fooming AI next week only if you gave your OK?
He has; this is made abundantly clear in the Metaethics sequence and particularly the “coming of age” sequence. That passage appears to be a reflection of the big embarrassing mistake he talked about, when he thought that he knew nothing about true morality (se “Could Anything Be Right?”) and that a superintelligence with a sufficiently “unconstrained” goal system (or what he’d currently refer to as “a rock”) would necessarily discover the ultimate true morality, so that whatever this superintelligence ended up doing would necessarily be the right thing, whether that turned out to consist of giving everyone a volcano lair full of catgirls/boys or wiping out humanity and reshaping the galaxy for its own purposes.
Needless to say, that is not his view anymore; there isn’t even any “Us or Them” to speak of anymore. Friendly AIs aren’t (necessarily) people, and certainly won’t be a distinct race of people with their own goals and ambitions.
Yes, I’m not suggesting that he is just signaling all that he wrote in the sequences to persuade people to trust him. I’m just saying that when you consider what people are doing for much less than shaping the whole universe to their liking, one might consider some sort of public or third-party examination before anyone is allowed to launch a fooming AI.
The hard part there is determining who’s qualified to perform that examination.
It will probably never come to it anyway. Not because the SIAI is not going to succeed but if it told anyone that it is even close to implementing something like CEV then the whole might of the world would crush it (if the world didn’t turn rational until then). Because to say that you are going to run a fooming AI will be interpreted as trying to take over all power and rule the universe. I suppose this is also the most likely reason for the SIAI to fail. The idea is out and once people notice that fooming AI isn’t just science fiction they will do everything to stop anyone from either implementing one at all or to run their own before anyone else does. And who’ll be the first competitor to take out in the race to take over the universe? The SIAI of course, just search Google. I guess it would have been a better idea to make this a stealth project from day one. But that train has left.
Anyway, if the SIAI does succeed one can only hope that Yudkowsky is not Dr. Evil in disguise. But even that would still be better than a paperclip maximizer. I assign more utility to a universe adjusted to Yudkowsky’s volition (or the SIAI) than paperclips (I suppose even if that means I’ll not “like” what happens to me then).
I don’t see who is going to enforce that. Probably nobody.
What we are fairly likely to see is open-source projects getting more limelight. It is hard to gather mindshare if your strategy is: trust the code to us. Relatively few programmers are likely to buy into such projects—unless you pay them to do so.
Yes on the question of humans vs Singularity.
(His word alone would not be enough to convince me he’s gotten the fooming AI friendly, though, so I would not give the OK for prudential reasons.)
So you take him at his word that he’s working in your best interest. You don’t think it is necessary to supervise the SIAI while working towards friendly AI. But once they finished their work, ready to go, you are in favor of some sort of examination before they can implement it. Is that correct?
I don’t think human selfishness vs. public interest is much of a problem with FAI; everyone’s interests with respect to FAI are well correlated, and making an FAI which specifically favors its creator doesn’t give enough extra benefit over an FAI which treats everyone equally to justify the risks (that the extra term will be discovered, or that the extra term introduces a bug). Not even for a purely selfish creator; FAI scenarios just doesn’t leave enough room for improvement to motivate implementing something else.
On the matter of inspecting AIs before launch, however, I’m conflicted. On one hand, the risk of bugs is very serious, and the only way to mitigate it is to have lots of qualified people look at it closely. On the other hand, if the knowledge that a powerful AI was close to completion became public, it would be subject to meddling by various entities that don’t understand what they’re doing. and it would also become a major target for espionage by groups of questionable motives and sanity who might create UFAIs. These risks are difficult to balance, but I think secrecy is the safer choice, and should be the default.
If your first paragraph turns out to be true, does that change anything with respect to the problem of human and political irrationality? My worry is that even if there is only one rational solution that everyone should favor, how likely is it that people understand and accept this? That might be no problem given the current perception. If the possibility of fooming AI will still be ignored at the point it will be possible to implement friendliness (CEV etc.), then there will be no opposition. So some quick quantum leaps towards AGI will likely allow the SIAI to follow through on it. But my worry is that if the general public or governments notice this possibility and take it serious, it will turn into a political mess never seen before. The world would have to be dramatically different for the big powers to agree on something like CEV. I still think this is the most likely failure mode in case the SIAI succeeds in defining friendliness before someone else runs a fooming AI. Politics.
I agree. But is that still possible? After all we’re writing about it in public. Although to my knowledge the SIAI never suggested that it would actually create a fooming AI, only come up with a way to guarantee its friendliness. But what you said in your second paragraph would suggest that the SIAI would also have to implement friendliness or otherwise people will take advantage of it or simply mess it up.
This?
http://www.acceleratingfuture.com/people-blog/?p=196
Probably it would be easier to run the examination during the SIAI’s work, rather than after. Certainly it would save more lives. So, supervise them, so that your examination is faster and more thorough. I am not in favour of pausing the project, once complete, to examine it if it’s possible to examine in in operation.
At the bottom—just after where he talks about his “transfer of allegiance”—it says:
©1998 by Eliezer S. Yudkowsky.
We can’t say he didn’t warn us ;-)
IMO, it is somewhat reminiscent of certain early Zuckerberg comments.
Eliezer1998 is almost as scary as Hanson2010 - and for similar reasons.
1998 you mean?
Yes. :)
What Zuckerberg comments are you referring to?
The IM ones where he says “trust me”.
Zuckerberg probably thought they were private, though. I added a link.
If you follow the link:
You shouldn’t seek to “weaken an argument”, you should seek what is the actual truth, and then maybe ways of communicating your understanding. (I believe that’s what you intended anyway, but think it’s better not to say it this way, as a protective measure against motivated cognition.)
I like your parenthetical, I often want to say something like this, and you’ve put it well.
Thank-you for pointing this out.
I took wedrifid’s point as being that whether EY is right or not, the bad effect described happens. This is part of the lose-lose nature of the original problem (what to do about a post that hurt people).
I don’t think this rhetoric is applicable. Several very intelligent posters have deemed the idea dangerous; a very intelligent you deems it safe. You argue they are wrong because it is ‘obviously safe’.
Eliezer is perfectly correct to point out that, on the whole of it, ‘obviously it is safe’ just does not seem like strong enough evidence when it’s up against a handful of intelligent posters who appear to have strong convictions.
Pardon? I don’t believe I’ve said any such thing here or elsewhere. I could of course be mistaken—I’ve said a lot of things and don’t recall them all perfectly. But it seems rather unlikely that I did make that claim because it isn’t what I believe.
This leads me to the conclusion that...
… This rhetoric isn’t applicable either. ;)
I should have known I wouldn’t get away with that, eh? I actually don’t know if you oppose the decision because you think the idea is safe, or because you think that censorship is wronger than the idea is dangerous, or whether you even oppose the decision at all and were merely pointing out appeals to authority. If you could fill me on the details, I could re-present the argument as it actually applies.
Thankyou, and yes I can see the point behind what you were actually trying to say. It just important to me that I am not misrepresented (even though you had no malicious intent).
There are obvious (well, at least theoretically deducible based on the kind of reasoning I tend to discuss or that used by harry!mor) reasons why it would be unwise to give a complete explanation of all my reasoning.
I will say that ‘censorship is wronger’ is definitely not the kind of thinking I would use. Indeed, I’ve given examples of things that I would definitely censor. Complete with LOTR satire if I recall. :)
This isn’t evidence about that hypothesis, it’s expected that most certainly nothing happens. Yet you write for rhetorical purposes as if it’s supposed to be evidence against the hypothesis. This constitutes either lying or confusion (I expect it’s unintentional lying, with phrases produced without conscious reflection about their meaning, so a little of both lying and confusion).
The sentence of Vaniver’s you quote seems like a straight forward case of responding to hyperbole with hyperbole in kind.
That won’t be as bad-intentioned, but still as wrong and deceptive.
The point we are trying to make is that we think the people who stared the basilisk in the eyes and metaphorically turned to stone are stronger evidence.
I get that. But I think it’s important to consider both positive and negative evidence- if someone’s testimony that they got turned to stone is important, so are the testimonies of people who didn’t get turned to stone.
The question to me is whether the basilisk turns people to stone or people turn themselves into stone. I prefer the second because it requires no magic powers on the part of the basilisk. It might be that some people turn to stone when they see goatse for the first time, but that tells you more about humans and how they respond to shock than about goatse.
Indeed, that makes it somewhat useful to know what sort of things shock other people. Calling this idea ‘dangerous’ instead of ’dangerous to EY” strikes me as mind projection.
I am considering both.
I generally find myself in support of people who advocate a policy of keeping people from seeing Goatse.
I’m not sure how to evaluate this statement. What do you mean by “keeping people from seeing Goatse”? Banning? Voluntarily choosing not to spread it? A filter like the one proposed in Australia that checks every request to the outside world?
Censoring posts that display Goatse on LessWrong.
Generally, censoring posts that display Goatse on non-Goatse websites.
I am much more sympathetic to “keeping goatse off of site X” than “keeping people from seeing goatse,” and so that’s a reasonable policy. If your site is about posting pictures of cute kittens, then goatse is not a picture of a cute kitten.
However, it seems to me that suspected Langford basilisks are part of the material of LessWrong. Imagine someone posted in the discussion “hey guys, I really want to be an atheist but I can’t stop worrying about whether or not the Rapture will happen, and if it does life will suck.” It seems to me that we would have a lot to say to them about how they could approach the situation more rationally.
And, if Langford basilisks exist, religion has found them. Someone got a nightmare because of Roko’s idea, but people fainted upon hearing Sinners in the Hands of an Angry God. Why are we not looking for the Perseus for this Medusa? If rationality is like an immune system, and we’re interested in refining our rationality, we ought to be looking for antibodies.
It seems to me that Eliezer’s response as moderator of LessWrong strongly implies that he does not believe this is the case. Your goal, then, would be to convince Eliezer that it ought to be part of the LessWrong syllabus, as it were. Cialdini’s Influence and other texts would probably advise you to work within his restrictions and conform to his desires as much as practical—on a site like LessWrong, though, I am not sure how applicable the advice would be, and in any case I don’t mean to be prescriptive about it.
Right. I see a few paths to do that that may work (and no, holding the future hostage is not one of them).
Is Goatse supposed to be a big deal? Someone showed it to me and I literally said “who cares?”
I totally agree. There are far more important internet requests that my (Australian) government should be trying to filter. Priorities people!
Yes.
I feel like reaction videos are biased towards people who have funny or dramatic reactions, but point taken.
I don’t understand this. (Play on conservation of expected evidence? In what way?)
Normal updating.
Original prior for basilik-danger.
Eliezer_Yudkowsky stares at basilisk, turns to stone (read: engages idea, decides to censor). Revise pr(basilisk-danger) upwards.
FormallyknownasRoko stares at basilisk, turns to stone (read: appears to truly wish e had never thought it). Revise pr(basilisk-danger) upwards.
Vladimir_Nesov stares at basilisk, turns to stone (read: engages idea, decides it is dangerous). Revise pr(basilisk-danger) upwards.
Vaniver stares at basilisk, is unharmed (read: engages idea, decides it is not dangerous). Revise pr(basilisk-danger) downwards.
Posterior is higher than original prior.
For the posterior to equal or lower than the prior, Vaniver would have to be more a rationalist than Eliezer, Roko, and you put together.
Okay, but more than four people have engaged with the idea. Should we take a poll?
The problem of course is that majorities often believe stupid things. That is why a free marketplace of ideas free from censorship is a really good thing! The obvious thing to do is exchange information until agreement but we can’t do that, at least not here.
Also, the people who think it should be censored all seem to disagree about how dangerous the idea really is, suggesting it isn’t clear how it is dangerous. It also seems plausible that some people have influenced the thinking of other people- for example it looks like Roko regretted posting after talking to Eliezer. While Roko’s regret is evidence that Eliezer is right, it isn’t the same as independent/blind confirmation that the idea is dangerous.
When you give all agents equal weight, sure. Without taking a poll of anything except my memory, Eliezer+Roko+VladNesov+Alicorn are against, DavidGerard+waitingforgodel+vaniver are for. Others are more sidelined than supporting a particular side.
Aumann agreement works in the case of hidden information—all you need are posteriors and common knowledge of the event alone.
Roko increased his estimation and Eliezer decreased his estimation—and the amounts they did so are balanced according to the strength of their private signals. Looking at two Aumann-agreed conclusions gives you the same evidence as looking that the pre-Aumann (differing) conclusions—the same way that 10, 10 gives you the same average as 5, 15.
I would prefer you not treat people avoiding a discussion as evidence that people don’t differentially evaluate the assertions made in that discussion.
Doing so creates a perverse incentive whereby chiming in to say “me too!” starts to feel like a valuable service, which would likely chase me off the site altogether. (Similar concerns apply to upvoting comments I agree with but don’t want to see more of.)
If you are seriously interested in data about how many people believe or disbelieve certain propositions, there exist techniques for gathering that data that are more reliable than speculating.
If you aren’t interested, you could just not bring it up.
I treat them as not having given me evidence either way. I honestly don’t know how I could treat them otherwise.
It is extremely hard to give no evidence by making a decision, even a decision to do nothing.
Okay. It is not that they give no evidence by remaining out of the discussion—it is that the evidence they give is spread equally over all possibilities. I don’t know enough about these people to say that discussion-abstainers are uniformly in support or in opposition to the idea. The best I can do is assume they are equally distributed between support and opposition, and not incorrectly constrain my anticipations.
You can do better than that along a number of different dimensions.
But even before getting there, it seems important to ask whether our unexpressed beliefs are relevant.
That is, if it turned out that instead of “equally distributed between support and opposition”, we are 70% on one side, or 90%, or 99%, or that there are third options with significant membership, would that information significantly affect your current confidence levels about what you believe?
If our unexpressed opinions aren’t relevant, you can just not talk about them at all, just like you don’t talk about millions of other things that you don’t know and don’t matter to you.
If they are relevant, one thing you could do is, y’know, research. That is, set up a poll clearly articulating the question and the answers that would affect your beliefs and let people vote for their preferred answers. That would be significantly better than assuming equal distribution.
Another thing you could do, if gathering data is unpalatable, is look at the differential characteristics of groups that express one opinion or another and try to estimate what percentage of the site shares which characteristics.
Yes. In the absence of actual evidence (which seems dangerous to gather in the case of this basilisk), I pretty much have to go by expressed opinions. To my mind, it was like trying to count the results of experiments that haven’t been performed yet.
I did not seek out more information because it was a throwaway line in an argument attempting to explain to people why it appears their voices are being ignored. I personally am on the side of censoring the idea, not having understood it at all when it first posted, and that may have bled into my posts (I should have exercised stronger control over that) but I am not arguing for censorship. I am arguing why, when someone says “it’s not dangerous!”, some people aren’t coming around to their perspective.
I don’t intend to argue for the censorship of the idea unless sorely pressed.
A few things:
** I’m confused. On the one hand, you say knowing the popularity of various positions is important to you in deciding your own beliefs about something potentially dangerous to you and others. On the other hand, you say it’s not worth seeking more information about and was just a throwaway line in an argument. I am having a hard time reconciling those two claims… you seem to be trying to have it both ways. I suspect I’ve misunderstood something important.
** I didn’t think you were arguing for censorship. Or against it. Actually, I have long since lost track of what most participants in this thread are arguing for, and in some cases I’m not sure they themselves know.
** I agree with you that the existence of knowledgeable people who think something is dangerous is evidence that it’s dangerous.
** Since it seems to matter: for my own part, I rate the expected dangerousness of “the basilisk” very low, and the social cost to the group of the dispute over “censoring” it significantly higher but still low.
** I cannot see why that should be of any evidentiary value whatsoever, to you or anyone else. Whether I’m right or wrong, my position is a pretty easy-to-reach one; it’s the one you arrive at in the absence of other salient beliefs (like, for example, the belief that EY/SIAI is a highly reliable estimator of potential harm done by “basilisks” in general, or the belief that the specific argument for the harmfulness of this basilisk is compelling). And most newcomers will lack those other beliefs. So I expect that quite a few people share my position—far more than 50% -- but I can’t see why you ought to find that fact compelling. That a belief is very widely shared among many many people like me who don’t know much about the topic isn’t much evidence for anything.
Sometimes that isn’t a bad state to be in. Not having an agenda to serve frees up the mind somewhat! :)
(nods) I’m a great believer in it. Especially in cases where a disagreement has picked up momentum, and recognizable factions have started forming… for example, if people start suggesting that those who side with the other team should leave the group. My confidence in my ability to evaluate an argument honestly goes up when I genuinely don’t know what team that argument is playing for.
I suspect I’ve obfuscated it, actually. The popularity of various positions is not intrinsically important to me—in fact, I give professions of believe about as little credit as I can get away with. This specific case is such that every form of evidence I find stronger (reasoning through the argument logically for flaws; statistical evidence about its danger) is not available. With a dearth of stronger evidence, I have to rely on weak evidence—but “the evidence is weak” is not an argument for privileging my own unsubstantiated position.
I don’t feel the need to collect weak evidence … I should, in this case. I was following a heuristic of not collecting weak evidence (waste of effort) without noticing that there was no stronger evidence.
Why are people’s beliefs of any value? Everyone has the ability to reason. All (non-perfect) reasoners fail in some way or another; if I look at many (controlling for biased reasoning) it gives me more of a chance to spot the biases—I have a control to compare it to.
This case is a special case; some people do have evidence. They’ve read the basilisk, applied their reasoning and logic, and deduced that it is / is not dangerous. These peoples’ beliefs are to be privileged over people who have not read the basilisk. I can’t access private signals like that—I don’t want to read a potential basilisk. So I make a guess at how strong their private signal is (this is why I care about their rationality) and use that as weak evidence for or against.
If seeking harder evidence wasn’t dangerous (and it usually isn’t) I would have done that instead.
The sentence I quoted sounded to me as though you were treating those of us who’ve remained “sidelined” as evidence of something. But if you were instead just bringing us up as an example of something that provides no evidence of anything, and if that was clear to everyone else, then I’m content.
I think I had a weird concept of what ‘sidelined’ meant in my head when I was writing that. Certainly, it seems out of place to me now.
I’m for. I believe Tim Tyler is for.
Human’s have this unfortunate feature of not being logically omniscient. In such cases where people don’t see all the logical implications of an argument we can treat those implications as hidden information. If this wasn’t the case then the censorship would be totally unnecessary as Roko’s argument didn’t actually include new information. We would have all turned to stone already.
There is no way for you to have accurately assessed this. Roko and Eliezer aren’t idealized Bayesian agents, it is extremely unlikely they performed a perfect Aumann agreement. If one is more persuasive than the other for reasons other than the evidence they share than their combined support for the proposition may not be worth the same as two people who independently came to support the proposition. Besides which, according to you, what information did they share exactly?
I had a private email conversation with Eliezer that did involve a process of logical discourse, and another with Carl.
Also, when I posted the material, I hadn’t thought it through. One I had thought it through, I realized that I had accidentally said more than I should have done.
David_Gerard, Jack, timtyler, waitingforgodel, and Vaniver do not currently outweigh Eliezer_Yudkowsky, FormallyknownasRoko, Vladimir_Nesov, and Alicorn, as of now, in my mind.
It does not need to be a perfect Aumann agreement; a merely good one will still reduce the chances of overcounting or undercounting either side’s evidence well below the acceptable limits.
They are approximations of Bayesian agents, and it is extremely likely they performed an approximate Aumann agreement.
To settle this particular question, however, I will pay money. I promise to donate 50 dollars to the Singularity Institute for Artificial Intelligence, independent of other plans to donate, if Eliezer confirms that he did revise his estimate down; or if he confirms that he did not revise his estimate down. Payable within two weeks of Eliezer’s comment.
I’m curious: if he confirms instead that the change in his estimate, if there was one, was small enough relative to his estimate that he can’t reliably detect it or detect its absence, although he infers that he updated using more or less the same reasoning you use above, will you donate or not?
I will donate.
I would donate even if he said that he revised his estimate upwards.
I would then seriously reconsider my evaluation of him, but as it stands the offer is for him to weigh in at all, not weigh in on my side.
edit: I misparsed your comment. That particular answer would dance very close to ‘no comment’, but unless it seemed constructed that way on purpose, I would still donate.
Yeah, that’s fair. One of the things I was curious about was, in fact, whether you would take that answer as a hedge, but “it depends” is a perfectly legitimate answer to that question.
How many of me would there have to be for that to work?
Also, why is rationalism the risk factor for this basilisk? Maybe the basilisk only turns to stone people with brown eyes (or the appropriate mental analog).
Only one; I meant ‘you’ in that line to refer to Vlad. It does raise the question “how many people disagree before I side with them instead of Eliezer/Roko/Vlad”. And the answer to that is … complicated. Each person’s rationality, modified by how much it was applied in this particular case, is the weight I give to their evidence; then the full calculation of evidence for and against should bring my prior to within epsilon but preferably below my original prior for me to decide the idea is safe.
Rationalism is the ability to think well and this is a dangerous idea. If it were a dangerous bacterium then immune system would be the risk factor.
Generally, if your immune system is fighting something, you’re already sick. Most pathogens are benign or don’t have the keys to your locks. This might be a similar situation- the idea is only troubling if your lock fits it- and it seems like then there would be rational methods to erode that fear (like the immune system mobs an infection).
The analogy definitely breaks down, doesn’t it? What I had in mind was Eliezer, Roko, and Vlad saying “I got sick from this infection” and you saying “I did not get sick from this infection”—I would look at how strong each person’s immune system is.
So if Eliezer, Roko, and Vlad all had weak immune systems and yours was quite robust, I would conclude that the bacterium in question is not particularly virulent. But if three robust immune systems all fell sick, and one robust immune system did not, I would be forced to decide between some hypotheses:
the first three are actually weak immune systems
the fourth was not properly exposed to the bacterium
the fourth has a condition that makes it immune
the bacterium is not virulent, the first three got unlucky
On the evidence I have, the middle two seem more likely than the first and last hypotheses.
I agree- my money is on #3 (but I’m not sure whether I would structure is as “fourth is immune” or “first three are vulnerable”- both are correct, but which is more natural word to use depends on the demographic response).
Er, are you describing rationalism (I note you say that and not “rationality”) as susceptible to autoimmune disorders? More so than in this post?
This equivocates the intended meaning of turning to stone in the original discussion you replied to. Fail. (But I understand what you meant now.)
Sorry, I should not have included censoring specifically. Change the “read:”s to ‘engages, reacts negatively’, ‘engages, does not react negatively’ and the argument still functions.
The argument does seem to function, but you shouldn’t have used the term in a sense conflicting with intended.
You would need a mechanism for actually encouraging them to “overcome” the terror, rather than reinforce it. Otherwise you might find that your subjects are less rational after this process than they were before.
Right- and current methodologies when it comes to that sort of therapy are better done in person than over the internet.
one wonders how something like that might have evolved, doesn’t one? What happened to all the humans who came with the mutation that made them want to find out whether the sabre-toothed tiger was friendly?
I don’t see how very unlikely events that people knew the probability of would have been part of the evolutionary environment at all.
In fact, I would posit that the bias is most likely due to having a very high floor for probability. In the evolutionary environment things with probability you knew to be <1% would be unlikely to ever be brought to your attention. So not having any good method for intuitively handling probabilities between 1% and zero would be expected.
In fact, I don’t think I have an innate handle on probability to any finer grain than ~10% increments. Anything more than that seems to require mathematical thought.
Probably less than 1% of cave-men died by actively seeking out the sabre-toothed tiger to see if it was friendly. But I digress.
But probably far more than 1% of cave-men who chose to seek out a sabre-tooth tiger to see if they were friendly died due to doing so.
The relevant question on an issue of personal safety isn’t “What % of the population die due to trying this?”
The relevant question is: “What % of the people who try this will die?”
In the first case, rollerskating downhill, while on fire, after having taken arsenic would seem safe (as I suspect no-one has ever done precisely that)
No, really, one doesn’t wonder. It’s pretty obvious. But if we’ve gotten to the point where “this bias paid off in the evolutionary environment!” is actually used as an argument, then we are off the rails of refining human rationality.
What’s wrong with using “this bias paid off in the evolutionary environment!” as an argument? I think people who paid more attention to this might make fewer mistakes, especially in domains where there isn’t a systematic, exploitable difference between EEA and now.
The evolutionary environment contained enetities capable of dishing out severe punishments, unertainty, etc.
If anything, I think that the heuristic that an idea “obviously” can’t be dangerous is the problem, not the heuristic that one should take care around possibilities of strong penalites.
It is a fine argument for explaining the widespread occcurrence of fear. However, today humans are in an environment where their primitive paranoia is frequently triggered by inappropriate stimulii.
Dan Gardener goes into this in some detail in his book: Risk: The Science and Politics of Fear
Video of Dan discussing the topic: Author Daniel Gardner says Americans are the healthiest and safest humans in the world, but are irrationally plagued by fear. He talks with Maggie Rodriguez about his book ‘The Science Of Fear.’
He says “we” are the healthiest and safest humans ever to live, but I’m very skeptical that this refers specifically to Americans rather than present day first world nation citizens in general.
Yes, we are, in fact, safer than in the EEA, in contemporary USA.
But still, there are some real places where danger is real, like the Bronx or scientology or organized crime or a walking across a freeway. So, don’t go rubbishing the heuristic of being frightened of potentially real danger.
I think it would only be legitimate to criticize fear itself on “outside view” grounds if we lived in a world with very little actual danger, which is not at all the case.
So, this may be a good way to approach the issue: loss to individual humans is, roughly speaking, finite. Thus, the correct approach to fear is to gauge risks by their chance of loss, and then discount if it’s not fatal.
So, we should be much less worried by a 1e-6 risk than a 1e-4 risk, and a 1e-4 risk than a 1e-2 risk. If you are more scared by a 1e-6 risk than a 1e-2 risk, you’re reasoning fallaciously.
Now, one might respond- “but wait! This 1e-6 risk is 1e5 times worse than the 1e-2 risk!”. But that seems to fall into the traps of visibility bias and privileging the hypothesis. If you’re considering a 1e-6 risk, have you worked out not just all the higher order risks, but also all of the lower order risks that might have higher order impact? And so when you have an idea like the one in question, which I would give a risk of 1e-20 for discussion’s sake, and you consider it without also bringing into your calculus essentially every other risk possible, you’re not doing it rigorously. And, of course, humans can’t do that computation.
Now, the kicker here is that we’re talking about fear. I might fear the loss of every person I know just as strongly as I fear the loss of every person that exists, but be willing to do more to prevent the loss of everyone that exists (because that loss is actually larger). Fear has psychological ramifications, not decision-theoretic ones. If this idea has 1e-20 chances of coming to pass, you can ignore it on a fear level, and if you aren’t, then I’m willing to consider that evidence you need help coping with fear.
I have a healthy respect for the adaptive aspects of fear. However, we do need an explanation for the scale and prevalence of irrational paranoia.
The picture of an ancestral water hole surrounded by predators helps us to understand the origins of the phenomenon. The ancestral environment was a dangerous and nasty place where people led short, brutish lives. There, living in constant fear made sense.
Someone’s been reading Terry Pratchett.
Considering the extraordinary appeal that forbidden knowledge has even for the average person, let alone the exceptionally intellectually curious, I don’t think this is a very effective way to warn a person off of seeking out the idea in question. Far from deserving what they get, such a person is behaving in a completely ordinary manner, to exceptionally severe consequence.
Personally, I don’t want to know about the idea (at least not if it’s impossible without causing myself significant psychological distress to no benefit,) but I’ve also put significant effort into training myself out of responses such as automatically clicking links to shock sites that say “Don’t click this link!”
Hmm. It is tricky to go back, I would imagine.
The material does come with some warnings, I believe. For instance, consider this one:
“Beware lest Friendliness eat your soul.”—Eliezer Yudkowsky
As I understand, you donate (and plan to in the future) to existential risk charities, and that is one of the consequences of you having come across that link. How does this compute into net negative, in your estimation, or are you answering a different question?
Sure I want to donate. But if you express it as a hypothetical choice between being a person who didn’t know about any of this and had no way of finding out, versus what I have now, I choose the former. Though since that is not an available choice, it is a somewhat academic question.
I can’t believe to hear this from a person who wrote about Ugh fields. I can’t believe to read a plead for ignorance on a blog devoted to refining rationality. Ignorance is bliss, is that the new motto now?
Well look, one has to do cost/benefit calculations, not just blindly surge forward in some kind of post-enlightenment fervor. To me, it seems like there is only one positive term in the equation:: the altrustic value of giving money to some existential risk charity.
All the other terms are negative, at least for me. And unless I actually overcome excuses, akrasia, etc to donate a lot, I think it’ll all have been a mutually detrimental waste of time.
There is only one final criterion, the human decision problem. It trumps any other rule, however good or useful.
(You appeal to particular heuristics, using the feeling of indignation as a rhetoric weapon.)
Not helping. I was referring to the the moral value of donations as an argument for choosing to know, as opposed to not knowing. You don’t seem to address that in your reply (did I miss something?).
Oh, I see. Well, I guess it depends upon how much I eventually donate and how much of an incremental difference that makes.
It would certainly be better to just donate, AND to also not know anything about anything dangerous. I’m not even sure that’s possible, though. For all we know, just knowing about any of this is enough to land you in a lot of trouble either in the causal future or elsewhere.
I gather doing so would irritate our site’s host and moderator.
I wasn’t “filled in”, and I don’t know whether my argument coincides with Eliezer’s. I also don’t understand why he won’t explain his argument, if it’s the same as mine, now that content is in the open (but it’s consistent with, that is responds to the same reasons as, continuing to remove comments pertaining to the topic of the post, which makes it less of a mystery).
But you think that it is not a good thing for this to propagate more?
As a decision on expected utility under logical uncertainty, but extremely low confidence, yes. I can argue that it most certainly won’t be a bad thing (which I even attempted in comments to the post itself, my bad), the expectation of it being a bad thing derives from remaining possibility of those arguments failing. As Carl said, “that estimate is unstable in the face of new info” (which refers to his own argument, not necessarily mine).
For everyone who wants to know what this discussion is all about, the forbidden idea, here is something that does not resemble it except for its stupefying conclusions:
There’s this guy who has the idea that it might be rational to rob banks to donate the money to charities. He tells a friend at a bank about it who freaks out and urges him to shut up about it. Unfortunately some people who also work at the local bank overheard the discussion and it gave them horrible nightmares. Since they think the idea makes sense they now believe that everyone will starve to death if they don’t rob banks and donate the money to charities that try to feed the world. The friend working at the bank now gets really upset and tells the dude with the idea about this. He argues that this shows how dangerous the idea is and that his colleagues and everyone else who’s told about the idea and who is working for a bank might just rob their own banks.
An inconvenient detail here, that makes the idea slightly less likely, is that it isn’t talking about the local bank in your town, or any bank on Earth at all, but one located in the system of Epsilon Eridani. And the friend and people with nightmares are not working for some bank but a charity concerned with space colonization. To conclude that the idea is dangerous, they don’t just have to accept its overall premise but also that there are banks over at Epsilon Eridani, that it is worth it to rob them, that one can build the necessary spaceships to reach that place and so on. In other words you have to be completely nuts and should seek help if you seriously believe that the idea is dangerous.
And what is the secondary problem?
The problem is not the idea itself but that there obviously are people crazy enough to take it serious and who might commit to crazy things due to their beliefs. The problem is that everyone is concerned with space colonization but that we don’t want to have it colonized by some freaks with pirate spaceships to rob alien banks because of some crazy idea.
P.S. Any inconsistency in the above story is intended to resemble the real idea.
I am familiar with the forbidden idea, and don’t think this analogy resembles it at all.
This analogy makes sense if you assume the conclusion that the argument for the post being a Basilisk is incorrect, but not as an argument for convincing people that it’s incorrect. To evaluate whether the argument is correct, you have to study the argument itself, there is no royal road (the conclusion can be studied in other ways, since particular proof can’t be demanded).
(See this summary of the structure of the argument.)
FWIW, loads of my comments were apparently deleted by administrators at the time.
I was away for a couple of months while the incident took place and when I returned I actually used your user page to reconstruct most of the missing conversation (with blanks filled from other user pages and an alternate source). Yours was particularly useful because of how prolific you were with quoting those to whom you were replying. I still have ten pages of your user comments stored on my harddrive somewhere. :)
Yes, some weren’t fully deleted, but—IIRC—others were. If I am remembering this right, the first deleted post (Roko’s) left comments behind in people’s profiles, but with the second deleted post the associated comments were rendered completely inaccessible to everyone. At the time, I figured that the management was getting better at nuking people’s posts.
After that—rather curiously—some of my subsequent “marked deleted” posts remained visible to me when logged in—so I wasn’t even aware of what had been “marked deleted” to everyone else for most of the time—unless I logged out of the site.
That could only apply to your original post, not subsequent stuff.
Right. Bottle. Genie.
I’d just like to insert a little tangent here: Roko’s post and the select comments are the only things that moderation had any effect on whatsoever since the launch of the site—if I remember correctly. I don’t think even the PUA wars had any interference from above. Of course, this is a community blog, but even this level of noninterference is very non-typical on the internet. Normally you’d join a forum and get a locked thread and a mod warning every so often.
Additionally, that on LW we get this much insight about the workings of SIAI-as-a-nonprofit and have this much space for discussion of any related topics is also an uncommon thing and should be considered a bonus.
Technically that’s not true, if I recall correctly all AI/singularity topics were banned for the first six months to prevent this from being just another Singularity community in the crucial early phases.
It was only two months; nevertheless you’re right about that.
Even I think you’re just being silly now. I really don’t see how this helps refine the art of human rationality.
It’s not about being ‘bad’, dammit.
Ask yourself what you want then ask yourself how to achieve it. Eliminate threats because they happen to be in your way, not out of spite.
I already did that, that’s why we’re here.… make that email horribleforsiai.wfg@xoxy.net
Releasing the stuff hinted at here would cause a minor scandal—perhaps rolling dice whenever a post is banned would be a happy medium.
Eh, if the stuff hinted at really exist, you should release it anyway. I expect the stuff is not really that bad and you’ll hurt SIAI more with innuendo than with the stuff.
If you feel more comfortable labeling it ‘terrorism’… well… it’s your thinking to bias.
someone has to stand up against your abuse
No, the label is accurate. Right smack bang inside the concept of terrorism. And I am saying that as someone who agrees that Eliezer is behaving like a socially inept git.
Why? Feeling obliged to fight against people just gives them power over you.
Dude, don’t be an idiot. Really.
There is a big mismatch here between “sending an email to a blogger” and “increase existential risk by one in a million”. All of the strategies for achieving existential risk increases that large are either major felonies, or require abusing a political office as leverage. When you first made the threat, I got angry at you on the assumption that you realized this. But if all you’re threatening to do is send emails, well, I guess that’s your right.
LW is a place where you’ll get useful help on weeding out mistakes in your plan to blow up the world, it looks like.
For Epistemic Rationality!
That reminds me of the joke about the engineer in the French revolution.
Are you joking? Do you have any idea what a retarded law can do to existential risks?
P(law will pass|it is retarded && its sole advocate publicly described it as retarded) << 10^-6
Huh? Try:
P(law will pass | (sane existential risk estimators would think it’s retarded) && (lots of right wingers heard great reasons) && (a lobby or two was convinced it was in their interest))
But (hopefully!) nothing like that could be caused with a single email. Baby steps.
Don’t vote this down under the default viewing threshold, please!
Oh, and I’m reposting it here just in case WFG tries to delete it later:
Ordinarily I’d consider that a violation of netiquette, but under these exact circumstances...
Don’t let Eliezer to provoke you like that. Obviously just reposting comments would be a waste of time and would just get more of the same. The legitimate purposes of your script are:
Ensure that you don’t miss out on content.
Allow you to inform other interested people of said content (outside of the LessWrong system).
Make it possible for you to make observations along the lines of “there has been a comment censored here”.
I’ll note that the policy didn’t restrict itself to comments reposted here.
Admittedly, Internet anonymity being what it is, it’s an unenforceable policy outside of LessWrong unless the policy-violator chooses to cop to it.
Could someone upvote the parent please? This is really quite important.
What not ignoring that bit would look like? Your answer seems to cover the problem fully.
It doesn’t matter how fast or slow you move, it matters which points you currently expect agreement or disagreement on. If you expect disagreement on a certain point, you can’t use it as a settled point in conversation, with other elements of the conversation depending on it, unless it’s a hypothetical. Otherwise you are shouting, not arguing.
Perhaps. I bet you use the same “fast moving explanation” with your friends to speed up communication.
Either way, please explain why you think it’s not irrational.
Isn’t a normative argument. If indeed I follow certain behavior, doesn’t make it less of an error. There could be valid reasons for the behavior not being an error, but me following it isn’t one of them.
The hope with pointing out that you do it, would be that you’d remember why you do it… it’s friggin effective. (also, my other reply makes the reasoning for this more explicit)
I try not to.
For achievement of what purpose? There certainly is a place for communication of one’s position, but not where the position is already known, and is being used as an argument.
Time Savings.
Imagine that Alice and Bob go into a room and Alice explains an interesting idea to Bob. They leave the room together after time t.
If a method exists, whereby Alice and Bob can instead leave the room in t-x time, with Bob knowing the same information, then this method of communicating can be said to yield a time savings of x compared to the previous method.
It’s not an answer to my question about the purpose, it’s a restatement of another aspect of it efficiency.
(Maybe you mean “stopping an argument is sometimes in order”? I agree, but that an argument didn’t proceed still doesn’t warrant you to assume as settled in the conversation things that your interlocutor doesn’t agree with.)
I should start waiting before replying to you :p
Why not? My view is you take a normal conversation/discussion, and then proceed as fast as you can toward resolution (being careful that it’s the same resolution).
I don’t understand this. What’s “resolution”? You can’t assume a bottom line and then rationalize your way towards it faster. You are not justified in agreeing with me faster than it takes you to understand and accept the necessary arguments. Sometimes you’ll even find an error along the way that will convince me that my original position was wrong.
Not implied by grandparent.
Likely not, this phrase was given as an example of hypothesis that I can see and that’s probably not right. I can’t find a reasonable hypothesis yet.
Comment on downvoting of the parent: In this comment, I explained why I wrote what I wrote in the grandparent comment. How should I interpret the downvotes? Discouragement of explaining one’s actions? (Probably not!) A signal that I shouldn’t have made that statement in the grandparent the way I have? (But then the grandparent should’ve been downvoted, and this comment possibly upvoted for clarification.) Punishment for inferred lying about my movites? (If that’s how you read me, please say so, I never lie about things like this, I can make errors though and I’d like to understand them.)
It’s a difficult question that a discussion in comments won’t do justice (too much effort required from you also, not just me). Read the posts if you will (see “Decision theory” section).
Also keep in mind this comment: we are talking about what justifies attaining a very weak belief, and this isn’t supposed to feel like agreement with a position, in fact it should feel like confident disagreement. Most of the force of the absurd decision is created by moral value of the outcome, not by its probability.
(I obviously recognize that this decision aligns with what you’d expect from a motivation to cover one’s incorrect reasons. Sometimes that happens by accident.)
I understand your argument re: very weak belief… but it seems silly.
How is this different than positing a very small chance that a future dictator will nuke the planet unless I mail a $10 donation to green peace?
Do you have any reason to believe that it’s more likely that a future dictator, or anyone else, will nuke the planet if you don’t send a donation to Greenpeace than if you do?
I agree that you are not justified in seeing a difference, unless you understand the theory of acausal control to some extent and agree with it. But when you are considering a person who agrees with that theory, and makes a decision based on it, agreement with the theory fully explains that decision, this is a much better explanation than most of the stuff people are circulating here. At that point, disagreement about the decision must be resolved by arguing about the theory, but that’s not easy.
You appear to be arguing that a bad decision is somehow a less bad decision if the reasoning used to get to it was consistent (“carefully, correctly wrong”).
No, because the decision is tested against reality. Being internally consistent may be a reason for doing something that it is obvious to others is just going to be counterproductive—as in the present case—but it doesn’t grant a forgiveness pass from reality.
That is: in practical effects, sincere stupidity and insincere stupidity are both stupidity.
You even say this above (“There is only one proper criterion to anyone’s actions, goodness of consequences”), making your post here even stranger.
(In fact, sincere stupidity can be more damaging, as in my experience it’s much harder to get the person to change their behaviour or the reasoning that led to it—they tend to cling to it and justify it when the bad effects are pointed out to them, with more justifications in response to more detail on the consequences of the error.)
Think of it as a trolley problem. Leaving the post is a bad option, the consequences of removing it are then the question: which is actually worse and results in the idea propagating further? If you can prove in detail that a decision theory considers removing it will make it propagate less, you’ve just found where the decision theory fails.
Removing the forbidden post propagated it further, and made both the post itself and the circumstances of its removal objects of fascination. It has also diminished the perceived integrity of LessWrong, as we can no longer be sure posts are not being quietly removed as well as loudly; this also diminished the reputation of SIAI. It is difficult to see either of these as working to suppress the bad idea.
More importantly it removed lesswrong as a place where FAI and decision theory can be discussed in any depth beyond superficial advocacy.
The problem is more than the notion that secret knowledge is bad—it’s that secret knowledge increasingly isn’t possible, and increasingly isn’t knowledge.
If it’s science, you almost can’t do it on your own and you almost can’t do it as a secret. If it’s engineering, your DRM or other constraints will last precisely as long as no-one is interested in breaking them. If it’s politics, your conspiracy will last as long as you aren’t found out and can insulate yourself from the effects … that one works a bit better, actually.
The forbidden topic can be tackled with math.
I don’t believe this is true to any significant extent. Why do you believe that? What kind of questions are not actually discussed that could’ve been discussed otherwise?
You are serious?
What qualifies as a ‘Friendly’ AI?
If someone is about to execute an AI running ‘CEV’ should I push a fat man on top of him and save five people from torture? What about an acausal fat man? :)
(How) can acausal trade be used to solve the cooperation problem inherent in funding FAI development? If I recall this topic was one that was explicitly deleted. Torture was mostly just a superficial detail.
… just from a few seconds brainstorming. These are the kinds of questions that can not be discussed without, at the very least, significant bias due to the threat of personal abuse and censorship if you are not careful. I am extremely wary of even trivial inconveniences.
Yes.
This doesn’t seem like an interesting question, where it intersects the forbidden topic. We don’t understand decision theory well enough to begin usefully discussing this. Most directions of discussion about this useless question are not in fact forbidden and the discussion goes on.
We don’t formally understand even the usual game theory, let alone acausal trade. It’s far too early to discuss its applications.
It wasn’t Vladimir_Nesov’s interest that you feigned curiosity in and nor is it your place to decide what things others are interested in discussing. They are topics that are at least as relevant as such things as ‘Sleeping Beauty’ that people have merrily prattled on about for decades.
That you support a censorship of certain ideas by no means requires you to exhaustively challenge every possible downside to said censorship. Even if the decision were wise and necessary there is allowed to be disappointing consequences. That’s just how things are sometimes.
The zeal here is troubling.
What do you mean by “decide”? Whether they are interested in that isn’t influenced by my decisions, and I can well think about whether they are, or whether they should be (i.e. whether there is any good to be derived from that interest).
I opened this thread by asking,
You answered this question, and then I said what I think about that kind of questions. It wasn’t obvious to me that you didn’t think of some other kind of questions that I find important, so I asked first, not just rhetorically.
What you implied in this comment seems very serious, and it was not my impression that something serious was taking place as a result of the banning incident, so of course I asked. My evaluation of whether the topics excluded (that you’ve named) are important is directly relevant to the reason your comment drew my attention.
On downvoting of parent comment: I’m actually surprised this comment got downvoted. It’s not as long inferential depth as this one that got downvoted worse, and it looks to me quite correct. Help me improve, say what’s wrong.
The other way around. I don’t “support censorship”, instead I don’t see that there are downsides worth mentioning (besides the PR hit), and as a result I disagree that censorship is important. Of course this indicates that I generally disagree with arguments for the harm of the censorship (that I so far understood), and so I argue with them (just as with any other arguments I disagree with that are on topic I’m interested in).
No zeal, just expressing my state of belief, and not willing to yield for reasons other than agreement (which is true in general, the censorship topic or not).
No, yielding and the lack thereof is not the indicator of zeal of which I speak. It is the sending out of your soldiers so universally that they reach even into the territory of other’s preferences. That critical line between advocation of policy and the presumption that others must justify their very thoughts (what topics interests them and how their thoughts are affected by the threat of public shaming and censorship) is crossed.
The lack of boundaries is a telling sign according to my model of social dynamics.
It was not my intention to discuss whether something is interesting to others. If it wasn’t clear, I do state so here explicitly. You were probably misled by the first part of this comment, where I objected to your statement that I shouldn’t speculate about what others are interested in. I don’t see why not, so I objected, but I didn’t mean to imply that I did speculate about that in the relevant comment. What I did state is that I myself don’t believe that conversational topic important, and motivation for that remark is discussed in the second part of the same comment.
Besides, asserting that the topic is not interesting to others is false as a point of simple fact, and that would be the problem, not the pattern of its alignment with other assertions. Are there any other statements that you believe I endorse (“in support of censorship”) and that you believe are mistaken?
On severe downvoting of the parent: What are that comment’s flaws? Tell me, I’ll try to correct them. (Must be obvious to warrant a −4.)
(Should I lump everything in one comment, or is the present way better? I find it more clear if different concerns are extracted as separate sub-threads.)
It’s not just more clear, it allows for better credit assignment in cases where both good and bad points are made.
Steven beat me to it—this way works well. Bear in mind though that I wasn’t planning to engage in this subject too deeply. Simply because it furthers no goal that I am committed to and is interesting only in as much as it can spawn loosely related tangents.
That some topics are excluded is tautological, so it’s important what kind of topics were. Thus, stating “nor is it your place to decide what things others are interested in discussing” seems to be equivalent with stating “censorship (of any kind) is bad!”, which is not very helpful in the discussion of whether it’s in fact bad. What’s the difference you intended?
You do see the irony there I hope...
Would you have censored the information? If not, do you think it would be a good idea to discuss the subject matter on an external (public) forum? Would you be interested to discuss it?
No, for several reasons. I have made no secret of the fact that I don’t think Eliezer processes perceived risks rationally and I think this applies in this instance.
This is not a claim that censorship is always a bad idea—there are other obvious cases where it would be vital. Information is power, after all.
Only if there is something interesting to say on the subject. Or any interesting conversations to be had on the various related subjects that the political bias would interfere with. But the mere fact that Eliezer forbids it doesn’t make it more interesting to me. In fact, the parts of Roko’s posts that were most interesting to me were not even the same parts that Eliezer threw a tantrum over. As far as I know Roko has been bullied out of engaging in such conversation even elsewhere and he would have been the person most worth talking to about that kind of counterfactual.
Bear in mind that the topic has moved from the realm of abstract philosophy to politics. If you make any mistakes, demonstrate any ignorance or even say things that can be conceivably twisted to appear as such then expect that will be used against you here to undermine your credibility on the subject. People like Nesov and and jimrandom care, and care aggressively.
Post away, if I have something to add then I’ll jump in. But warily.
I am not sure if I understand the issue and if it is as serious as some people obviously perceive it to be. Because if I indeed understand it, then it isn’t as dangerous to talk about it in public as it is portrayed to be. But that would mean that there is something wrong with otherwise smart people, which is unlikely? So should I conclude that it is more likely that I simply do not understand it?
What irritates me is that people like Nesov are saying that “we don’t formally understand even the usual game theory, let alone acausal trade”. Yet they care aggressively to censor the topic. I’ve been told before that it is due to people getting nightmares from it. If that is the reason then I do not think censorship is justified at all.
I wouldn’t rule out the possibility that you do not fully understand it and they are still being silly. ;)
How about the possibility that you do not understand it and that they are not silly? Do you think it could be serious enough to have nightmares about it and to censor it as far as possible, but that you simply don’t get it? How likely is that possibility?
Why would you even ask me that? Clearly I have considered the possibility (given that I am not a three year old) and equally clearly me answering you would not make much sense. :)
But the questioning of trusting people’s nightmares is an interesting one. I tend to be of the mind that if someone has that much of an anxiety problem prompted by a simple abstract thought then it is best to see that they receive the appropriate medication and therapy. After that has been taken care of I may consider their advice.
I wasn’t quite sure. I don’t know how to conclude that they are silly and you are not. I’m not just talking about Nesov but also Yudkowsky. You concluded that they are all wrong about their risk estimations and act silly. Yudkowsky explicitly stated that he does know more. But you conclude that they don’t know more, that they are silly.
Yes, I commented before saying that it is not the right move to truncate your child’s bed so that monsters won’t fit under it but rather explain that it is very unlikely for monsters to hide under the bed.
You can’t. Given the information you have available it would be a mistake for you to make such a conclusion. Particularly given that I have not even presented arguments or reasoning on the core of the subject, what, with the censorship and all. :)
Indeed. Which means that not taking his word for it constitutes disrespect.
Once the child grows up a bit you can go on to explain to them that even though there are monsters out in the world being hysterical doesn’t help either in detecting monsters or fighting them. :)
As I noted, it’s a trolley problem: you have the bad alternative of doing nothing, and then there’s the alternative of doing something that may be better and may be worse. This case observably came out worse, and that should have been trivially predictable by anyone who’d been on the net a few years.
So the thinking involved in the decision, and the ongoing attempts at suppression, admits of investigation.
But yes, it could all be a plot to get as many people as possible thinking really hard about the “forbidden” idea, with this being such an important goal as to be worth throwing LW’s intellectual integrity in front of the trolley for.
Caring “to censor the topic” doesn’t make sense, it’s already censored, and already in the open, and I’m not making any actions regarding the censorship. You’d need to be more accurate in what exactly you believe, instead of reasoning in terms of vague affect.
Regarding lack of formal understanding, see this comment: the decision to not discuss the topic, if at all possible, follows from a very weak belief, not from certainty. Lack of formal understanding expresses lack of certainty, but not lack of very weak beliefs.
If an organisation, that is working on a binding procedure for a all-powerful dictator to implement it on the scale of the observable universe, tried to censor information, that could directly affect me for the rest of time in the worst possible manner, I got a very weak belief that their causal control is much more dangerous than the acausal control between me and their future dictator.
So you don’t care if I post it everywhere and send it to everyone I can?
For what it’s worth, I’ve given up on participating in these arguments. My position hasn’t changed, but arguing it was counterproductive, and extremely frustrating, which led to me saying some stupid things.
No, I don’t (or, alternatively, you possibly could unpack this in a non-obvious way to make it hold).
I suppose it just so happens it was the topic I engaged yesterday, and similar “care aggressively” characteristic can probably be seen in any other discussion I engage.
I don’t dispute that, and this was part of what prompted the warning to XiXi. When a subject is political and your opponents are known to use aggressive argumentative styles it is important to take a lot of care with your words—give nothing that could potentially be used against you.
The situation is analogous to recent discussion of refraining to respond to the trolley problem. If there is the possibility that people may use your words against you in the future STFU unless you know exactly what you are doing!
No irony. You don’t construct complex machinery out of very weak beliefs, but caution requires taking very weak beliefs into account.
The irony is present and complex machinery is a red herring.
Well then, I don’t see the irony, show it to me.
Here, I’m talking about factual explanation, not normative estimation. The actions are explained by holding a certain belief, better than by alternative hypotheses. Whether they were correct is a separate question.
You’d need to explain this step in more detail. I was discussing a communication protocol, where does “testing against reality” enter that topic?
Ah, I thought you were talking about whether the decision solved the problem, not whether the failed decision was justifiable in terms of the theory.
I do think that if a decision theory leads to quite as spectacular a failure in practice as this one did, then the decision theory is strongly suspect.
As such, whether the decision was justifiable is less interesting except in terms of revealing the thinking processes of the person doing the justification (clinginess to pet decision theory, etc).
“Belief in the decision being a failure is an argument against adequacy of the decision theory”, is simply a dual restatement of “Belief in the adequacy of the decision theory is an argument for the decision being correct”.
This statement appears confusing to me: you appear to be saying that if I believe strongly enough in the forbidden post having been successfully suppressed, then censoring it will not have in fact caused it to propagated widely, nor will it have become an object of fascination and caused a reputational hit to LessWrong and hence SIAI. This, of course, makes no sense.
I do not understand how this matches with the effects observable in reality, where these things do in fact appear to have happened. Could you please explain how one tests this result of the decision theory, if not by matching it against what actually happened? That being what I’m using to decide whether the decision worked or not.
Keep in mind that I’m talking about an actual decision and its actual results here. That’s the important bit.
If you believe that “decision is a failure” is evidence that the decision theory is not adequate, you believe that “decision is a success” is evidence that the decision theory is adequate.
Since a decision theory’s adequacy is determined by how successful its decisions are, you appear to be saying “if a decision theory makes a bad decision, it is a bad decision theory” which is tautologically true.
Correct me if I’m wrong, but Vladimir_Nesov is not interested in whether the the decision theory is good or bad, so restating an axiom of decision theory evaluation is irrelevant.
The decision was made by a certain decision theory. The factual question “was the decision-maker holding to this decision theory in making this decision?” is entirely unrelated to the question “should the decision-maker hold to this decision theory given that it makes bad decisions?”. To suggest otherwise blurs the prescriptive/descriptive divide, which is what Vladimir_Nesov is referring to when he says
I believe that if the decision theory clearly led to an incorrect result (which it clearly did in this case, despite Vladimir Nesov’s energetic equivocation), then it is important to examine the limits of the decision theory.
As I understand it, the purpose of bothering to advocate TDT is that it beats CDT in the hypothetical case of dealing with Omega (who does not exist), and is therefore more robust, then this failure in a non-hypothetical situation suggests a flaw in its robustness, and it should be regarded as less reliable than it may have been regarded previously.
Assuming the decision was made by robust TDT.
The decision you refer to here… I’m assuming it is this still the Eliezer->Roko decision? (This discussion is not the most clearly presented.) If so for your purposes you can safely consider ‘TDT/CDT’ irrelevant. While acausal (TDTish) reasoning is at play in establishing a couple of the important premises, they are not relevant to the reasoning that you actually seem to be criticising.
ie. The problems you refer to here are not the fault of TDT or of abstract reasoning at all—just plain old human screw ups with hasty reactions.
That’s the one, that being the one specific thing I’ve been talking about all the way through.
Vladimir Nesov cited acausal decision theories as the reasoning here and here—if not TDT, then a similar local decision theory. If that is not the case, I’m sure he’ll be along shortly to clarify.
(I stress “local” to note that they suffer a lack of outside review or even notice. A lack of these things tends not to work out well in engineering or science either.)
Good, that had been my impression.
Independently of anything that Vladmir may have written it is my observation that the ‘TDT-like’ stuff was mostly relevant to the question “is it dangerous for people to think X?” Once that has been established the rest of the decision making, what to do after already having reached that conclusion, was for most part just standard unadorned human thinking. From what I have seen (including your references to reputation self sabotage by SIAI) you were more troubled by the the latter parts than the former.
Even if you do care about the more esoteric question “is it dangerous for people to think X?” I note that ‘garbage in, garbage out’ applies here as it does elsewhere.
(I just don’t like to see TDT unfairly maligned. Tarnished by association as it were.)
See section 7 of the TDT paper (you’ll probably have to read from the beginning to familiarize yourself with concepts). It doesn’t take Omega to demonstrate that CDT errs, it takes mere ability to predict dispositions of agents to any small extent to get out of CDT’s domain, and humans do that all the time. From the paper:
I wouldn’t use this situation as evidence for any outside conclusions. Right or wrong, the belief that it’s right to suppress discussion of the topic entails also believing that it’s wrong to participate in that discussion or to introduce certain kinds of evidence. So while you may believe that it was wrong to censor, you should also expect a high probability of unknown unknowns that would mess up your reasoning if you tried to take inferential steps from that conclusion to somewhere else.
I haven’t been saying I believed it was wrong to censor (although I do think that it’s a bad idea in general). I have been saying I believe it was stupid and counterproductive to censor, and that this is not only clearly evident from the results, but should have been trivially predictable (certainly to anyone who’d been on the Internet for a few years) before the action was taken. And if the LW-homebrewed, lacking in outside review, Timeless Decision Theory was used to reach this bad decision, then TDT was disastrously inadequate (not just slightly inadequate) for application to a non-hypothetical situation and it lessens the expectation that TDT will be adequate for future non-hypothetical situations. And that this should also be obvious.
Yes, the attempt to censor was botched and I regret the botchery. In retrospect I should have not commented or explained anything, just PM’d Roko and asked him to take down the post without explaining himself.
This is actually quite comforting to know. Thank you.
(I still wonder WHAT ON EARTH WERE YOU THINKING at the time, but you’ll answer as and when you think it’s a good idea to, and that’s fine.)
(I was down the pub with ciphergoth just now and this topic came up … I said the Very Bad Idea sounded silly as an idea, he said it wasn’t as silly as it sounded to me with my knowledge. I can accept that. Then we tried to make sense of the idea of CEV as a practical and useful thing. I fear if I want a CEV process applicable by humans I’m going to have to invent it. Oh well.)
And I would have taken it down. My bad for not asking first most importantly.
It is evidence for said conclusions. Do you mean, perhaps, that it isn’t evidence that is strong enough to draw confident conclusions on its own?
To follow from the reasoning the embedded conclusion must be ‘you should expect a higher probability’. The extent to which David should expect higher probability of unknown unknowns is dependent on the deference David gives to the judgement of the conscientious non-participants when it comes to the particular kind of risk assessment and decision making—ie. probably less than Jim does.
(With those two corrections in place the argument is reasonable.)
I agree, and in this comment I remarked that we were assuming this statement all along, albeit in a dual presentation.
If you’re interested, we can also move forward as I did over here by simply assuming EY is right, and then seeing if banning the post was net positive
It’s not “moving forward”, it’s moving to a separate question. That question might be worth considering, but isn’t generally related to the original one.
Why would the assumption that EY was right be necessary to consider that question?
I agree that it was net negative, specifically because the idea is still circulating, probably with more attention drawn to it than would happen otherwise. Which is why I started commenting on my hypothesis about the reasons for EY’s actions, in an attempt to alleviate the damage, after I myself figured it out. But that it was in fact net negative doesn’t directly argue that given the information at hand when the decision was made, it had net negative expectation, and so that the decision was incorrect (which is why it’s a separate question, not a step forward on the original one).
I like the precision of your thought.
All this time I thought we were discussing if blocking future censorship by EY was a rational thing to do—but it’s not what we were discussing at all.
You really are in it for the details—if we could find a way of estimating around hard problems to solve the above question, that’s only vaguely interesting to you—you want to know the answers to these questions.
At least that’s what I’m hearing.
It sounds like the above was your way of saying you’re in favor of blocking future EY censorship, which gratifies me.
I’m going to do the following things in the hope of gratifying you:
Writing up a post on less wrong for developing political muscles. I’ve noticed several other posters seem less than savvy about social dynamics, so perhaps a crash course is in order. (I know that there are certainly several in the archives, I guarantee I’ll bring several new insights [with references] to the table).
Reread all your comments, and come back at these issues tomorrow night with a more exact approach. Please accept my apology for what I assume seemed a bizarre discussion, and thanks for thinking like that.
Night!
I didn’t address that question at all, and in fact I’m not in favor of blocking anything. I came closest to that topic in this comment.
More than enough information about human behavior was available at the time. Negative consequences of the kind observed were not remotely hard to predict.
Yes, quite likely. I didn’t argue with this point, though I myself don’t understand human behavior enough for that expectation to be obvious. I only argued that the actual outcome isn’t a strong reason to conclude that it was expected.
I’m slowly moving through the sequences, I’ll comment back here if/when I finish the posts as well.
In the mean time, I’ve been told that if you can’t explain something simply then you don’t really understand it… wanna take a fast and loose whack?
edit: did you drastically edit your comment?
I believe this is utter nonsense, play on the meaning of the word “explain”. If explaining is to imply understanding by the recipient, then clearly fast explaining of great many things is not possible, otherwise education wouldn’t be necessary. Creating an illusion of understanding, or equivalently a shallow understanding might be manageable of course, the easier the less educated and rational the victim.
Interesting. I’ve found that intuitive explanations for relatively complex things are generally easier than a long, exact explanation.
Basically fast explanations use hardware accelerated paths to understanding (social reasoning, toy problems that can be played with, analogies), and then leave it to the listener to bootstrap themselves. If you listen to the way that researchers talk, it’s basically analogies and toy problems, with occasional black-board sessions if they’re mathy.
It’s hard to understand matrix inversion by such a route, which I think you’re saying is roughly what’s required to understand why you believe this censorship to be rational.
But, for the record, it ain’t no illusionary understanding when I talk fast with a professor or fellow grad student.
Certainly easier, but don’t give comparable depth of understanding or justify comparable certainty in statements about the subject matter. Also, the dichotomy is false, since detailed explanations are ideally accompanied by intuitive explanations to improve understanding.
What we were talking about instead is when you have only a fast informal explanation, without the detail.
It’s because they already have the rigor down. See this post by Terence Tao.
Yes, I do that, sorry. What I consider improvement over the original.
Your answer actually includes “you should try reading the sequences.”
The reference to the sequences is not the one intended (clarified by explicitly referring to “Decision theory” section in the grandparent comment).
I think for many kinds of research, working in groups drastically increases the efficacy of individual effort, due to specialization, etc.
Are you trying to get in on AI research?
Right now, my main interest is mostly in a) academic paper versions of the things SIAI has been talking about informally b) theoretical cognitive science stuff which may or may not be related to AI.
The Roko one.
I wonder whether it was Roko or big brother who deleted the nearby comments. One of them included the keywords ‘SIAI’ and ‘torture’ so neither would surprise me.
What about becoming a blogger at a place like ScienceBlogs?
Alternatively, if you’re willing to live very ascetically, what about emailing/asking philanthropists/millionaires with ideas, and asking them to fund them? (perhaps with a probationary period if necessary). What about emailing professors?
Theoretically, if you had VERY supportive+tolerant parents/friends (rare, but they exist on the Internet), you could simply ask to live with them, and to do research in their house as well.
Well obviously I can’t say that. We should just end the thread.
No you shouldn’t restore this, I’ve been asked to remove it as it could potentially be damaging.
No, just that people should be prepared to make significant sacrifices on consequentialist/utilitarian grounds. I sort of agreed at the time.
Now I think that it is somehow self-contradictory to make utilitarian sacrifices for a set of people who widely condemn utilitarian arguments, i.e. humanity at large.
Another idea is the “Bostrom Solution”, i.e. be so brilliant that you can find a rich guy to just pay for you to have your own institute at Oxford University.
Then there’s the “Reverse Bostrom Solution”: realize that you aren’t Bostrom-level brilliant, but that you could accrue enough money to pay for an institute for somebody else who is even smarter and would work on what you would have worked on.
Ensuring that is part of being a rationalist; if EY, Roko, and Vlad (apparently Alicorn as well?) were bad at error-checking and Vaniver was good at it, that would be sufficient to say that Vaniver is a better rationalist than E R V (A?) put together.
Certainly. However, error-checking oneself is notoriously less effective than having outsiders do so.
“For the computer security community, the moral is obvious: if you are designing a system whose functions include providing evidence, it had better be able to withstand hostile review.” - Ross Anderson, RISKS Digest vol 18 no 25
Until a clever new thing has had decent outside review, it just doesn’t count as knowledge yet.
That Eliezer wrote the Sequences and appears to think according to their rules and is aware of Löb’s Theorem is strong evidence that he is good at error-checking himself.
That’s pretty much a circular argument. How’s the third-party verifiable evidence look?
I dunno. Do the Sequences smell like bullshit to you?
edit: this is needlessly antagonistic. Sorry.
Mostly not—but then I am a human full of cognitive biases. Has anyone else in the field paid them any attention? Do they have any third-party notice at all? We’re talking here about somewhere north of a million words of closely-reasoned philosophy with direct relevance to that field’s big questions, for example. It’s quite plausible that it could be good and have no notice, because there’s not that much attention to go around; but if you want me to assume it’s as good as it would be with decent third-party tyre-kicking, I think I can reasonably ask for more than “the guy that wrote it and the people working at the institute he founded agree, and hey, do they look good to you?” That’s really not much of an argument in favour.
Put it this way: I’d be foolish to accept cryptography with that little outside testing as good, here you’re talking about operating system software for the human mind. It needs more than “the guy who wrote it and the people who work for him think it’s good” for me to assume that.
Fair enough. It is slightly more than Vaniver has going in their favour, to return to my attempt to balance their rationality against each other.
Upvoted to zero because of the edit.
I haven’t read fluffy (I have named it fluffy), but I’d guess it’s an equivalent of a virus in a monoculture: every mode of thought has its blind spots, and so to trick respectable people on LW, you only need an idea that sits in the right blind spots. No need for general properties like “only infectious to stupid people.”
Alicorn throws a bit of a wrench in this, as I don’t think she shares as many blind spots with the others you mention, but it’s still entirely possible. This also explains the apparent resistance of outsiders, without need for Eliezer to be lying when he says he thinks fluffy was wrong.
Could also be that outsiders are resistant because they have blind spots where the idea is infectious, and respectable people on LW are respected because they do not have the blind spots—and so are infected.
I think these two views are actually the same, stated as inverses of each other. The term blind spot is problematic.
I think the term blind spot is accurate, unless (and I doubt it) Eliezer was lying when he later said fluffy was wrong. What fits the bill isn’t a correct scary idea, but merely a scary idea that fits into what the reader already thinks.
Maybe fluffy is a correct scary idea, and your allocation of blind spots (or discouraging of the use of the term) is correct, but secondhand evidence points towards fluffy being incorrect but scary to some people.
I’m curious about why you think this.
Honestly? Doesn’t like to argue about quantum mechanics. That I’ve seen :D Your posts seem to be about noticing where things fit into narratives, or introspection, or things other than esoteric decision theory speculations. If I had to come up with an idea that would trick Eliezer and Vladimir N into thinking it was dangerous, it would probably be barely plausible decision theory with a dash of many worlds.
I was also surprised by your reaction to the the argument. In my case this was due to the opinions you’ve expressed on normative ethics.
How are my ethical beliefs related?
Answered by PM
First bit of advice: grow up. Be interested in research because you’re curious, not because you’re special and/or status-seeking. Until the word that comes to mind is “improve” rather than “save” you will be looking at the wrong questions for the wrong reasons.
Oh, absolutely. If you want to be part of academia, you have to be part of academia.
I would replace “might” here with “would.”
This comment strikes me as rather confrontational, and also as offering advice based on a misguided understanding of my motives.
I have very little clue of what you’re trying to say.
Of course. Whenever someone says they want to do something impossibly hard, the proper response is to dismiss them. Either they agree with you, and you made the right call, or they disagree with you, and you’ve cemented their resolve.
But JoshuaZ is right that I think the wording is ridiculous. “Save the world” is nothing but applause lights. If it were “see a positive singularity” we would have at least gone from 0th order to 1st order. If it were “make life extension available sooner” we’ve progressed from 1st order to 2nd order. If it were “make lab-grown organs available sooner” we’ve progressed from 2nd order to 3rd order. If one comes to me with the last desire, I’d tell them to move to Wake Forest and become friends with Anthony Atala. If someone comes to me with the first desire, I pat them on the head.
Even Norman Borlaug didn’t “save the world.” He might have personally extended lifespans by billions of human-years, but that’s not close, even on a log scale. And if you want to be a second Norman Borlaug, trying to save the world seems like a poor way to approach that goal because you focus on the wrong questions. Borlaug wanted to improve the world- to make hungry people full by creating wheat varieties that were disease-resistant. He had the altruistic impulse, but he was facing a problem worked out to third order. The altruistic impulse is a great thing, but if you don’t have a third order problem yet keep looking.
And when asking for career advice, it’s relevant information where you are in that process. If you’re at 2nd order, your interests will already be narrow enough that most advice will be inapplicable to your situation. The 2nd order person in the particular track I’ve outlined already knows they will strongly benefit from getting a medical degree in America, England, or China (there may be a few other countries on this list; this isn’t my specialty) or that they should earn a bunch of money while getting familiar with efforts already underway in those countries. If you’re at 3rd order, you’re already at the point where there are a handful of appropriate opportunities, and you’re better off looking for those opportunities specifically than you are getting generic advice.
If you’re at 0th order? Then you need to spend time cultivating your interests. If you go to Silicon Valley and your only interest is “I want to get rich!” you won’t get very far. While that may be the underlying interest of everyone there, the fact that it’s so common means that it conveys very little information. The way for an individual to tackle a 0th order problem is to find a 3rd order problem, uncover some 4th order problems while investigating that problem, and solve those.
EDIT: I fleshed this out further is this post.
You’re reading way too much into that single line. I wanted to express the sentiment of “I want to be as effective as possible in doing good”, and there was a recent post covering that topic which happened to be named “how to save the world”, so I linked to it. If that post hadn’t been there, I might have said something like “I want to do something meaningful with my life”. I was also assuming “saving the world” and other similar expressions to be standard LW jargon for “doing as much good as possible”.
As for my actual goals… Ideally I’d like to help avert a negative singularity, though since I don’t have very high hopes of that actually being possible, I also give the goals of “just have fun” and “help people in the short term” considerable weight, and am undecided as to how much effort I’ll in the end spend explicitly on singularity matters. But to the degree that I do end up trying to help the singularity, the three main approaches I’ve been playing with are
Just make money and donate that to SIAI.
Help influence academia to become more aware of these issues.
Become well-known enough (via e.g. writing, politics) among normal people that I can help spread singularity-related ideas and hopefully get more people to take them seriously.
These are obviously not mutually exclusive, and indeed, one of the reasons I’m playing around with the idea of “freelance academia” is that it allows me to do some of the academic stuff without the commitment that e.g. getting a PhD would involve (as I’m not yet sure whether the academic approach is the one that I’d find the most rewarding). All three also have to varying extent an intrinsic appeal, beyond just the singularity aspect: I wouldn’t mind having a bit more money, intellectual work is rewarding by itself, and so is writing and having a lot of people care about your opinions.
As for the details of the academic career path, the “help avoid a negative singularity” aspect of that currently mainly involves helping write up the ideas about the singularity into concise, well-sourced papers that people can be pointed to. (Here is one example of such work—an improved, full-length version of that paper is in the works.) Beyond that, maybe with time I can come up with original insights of my own to contribute to the field, as well as build a reputation and give those singularity-related ideas more merit by producing well-regarded papers in non-singularity-related fields that I happen to be interested in.
I believe that Vaniver is making a distinction between “improve” and “save” in saying that any given individual is unlikely to have a large-scale enough impact to be described as “saving” the world, but that many people can improve the world. This point may have some validity, although Norman Borlaugh may be a relevant counterexample to show that it isn’t completely impossible.
That would be only relevant if Kaj had said “I expect to save the world” instead of “Ideally, I’d like to save the world”. I read the latter as specifying something like “all existential risks are averted and the world gets much more awesome” as an optimization target, not as something that he wants to (let alone expects to be able to) do completely and singlehandedly. And as an optimization target, it makes good sense. Why aim for imperfection? The target is the measure of utility, not a proposed action or plan on its own. (Possibly relevant: Trying to Try.)
(One thing I see about that paragraph that could be legitimately disputed is the jump from specifying the optimization target to “One way to do that involves contributing academic research, which raises the question of what’s the most effective way of doing that” without establishing that academic research is itself the best way (or at least a good way) for a very smart person to optimize the aforementioned goal. That itself would be an interesting discussion, but I think in this post it is taken as an assumption. (See also this comment.))
There is no “singlehandedly”, individual decisions control actions of many people.
Indeed it is.
Agreed that it is too confrontational.
I think this varies from field to field. Some fields are more biased than others. Math for example is not very biased in that regard. The hard sciences are not as biased also but in order for people to do good work in them they need resources that they are going to have trouble getting outside academia or industry. I suspect (but have no strong evidence) that the “softer” a subject the more likely the journalists are to discriminate based on whether or not a submitter is in academia.
This sounds similar to adoption of pre-prints like ArXiv; in fields where most papers are accepted for publication (I think it’s 90% in physics) people use it; in fields where most papers aren’t accepted (I think it’s 20-30% in some humanities fields) people find them worthless (since a preprint hasn’t passed the gatekeeper, and that’s actually a significant cull).
How does this contribute to the thread?
Channeling John Paul Jones, “I have not yet begun to type.”