This post is highly critical of SIAI — both of its philosophy and its organizational choices. It is also now the #1 most highly voted post in the entire history of LessWrong — higher than any posts by Eliezer or myself.
I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.
Either I promoted this and then forgot I’d done so, or someone else promoted it—of course I was planning to promote it, but I thought I’d planned to do so on Tuesday after the SIAIers currently running a Minicamp had a chance to respond, since I expected most RSS subscribers to the Promoted feed to read comments only once (this is the same reason I wait a while before promoting e.g. monthly quotes posts). On the other hand, I certainly did upvote it the moment I saw it.
I agree (as a comparative outsider) that the polite response to Holden is excellent. Many (most?) communities—both online communities and real-world organisations, especially long-standing ones—are not good at it for lots of reasons, and I think the measured response of evaluating and promoting Holden’s post is exactly what LessWrong members would hope LessWrong could do, and they showed it succeeded.
I agree that this is good evidence that LessWrong isn’t just an Eliezer-cult. (The true test would be if Elizier and another long-standing poster were dismissive to the post, and then other people persuaded them otherwise. In fact, maybe people should roleplay that or something, just to avoid getting stuck in an argument-from-authority trap, but that’s a silly idea. Either way, the fact that other people spoke positively, and Elizier and other long-standing posters did too, is a good thing.)
However, I’m not sure it’s as uniquely a victory for the rationality of LessWrong as it sounds. In responose to srdiamond, Luke quoted tenlier saying “[Holden’s] critique mostly consists of points that are pretty persistently bubbling beneath the surface around here, and get brought up quite a bit. Don’t most people regard this as a great summary of their current views, rather than persuasive in any way?” To me, that suggests that Holden did a really excellent job expressing these views clearly and persuasively. However, it suggests that previous people had tried to express something similar, but it hadn’t been expressed well enough to be widely accepted, and people reading had failed to sufficiently apply the dictum of “fix your opponents’ arguments for them”. I’m not sure if that’s true (it’s certainly not automatically true), but I suspect it might be. What do people think?
If there’s any truth to it, it suggests one good answer to the recent post http://lesswrong.com/lw/btc/how_can_we_get_more_and_better_lw_contrarians (whether that was desirable in general or not) would be, as a rationalist exercise for someone familiar with/to the community and good at writing rationally, to take a survey of contrarian views on the topic that people on the community may have had but not been able to express, and don’t worry about showmanship like pretending to believe it yourself, but just say “I think what some people think is [well-expressed argument]. Do you agree that’s fair? If so, do I and other people think they have a point?” Whether or not that argument is right it’s still good to engage with it if many people are thinking it.
I suppose it’s that I naively expect, when opening the list of top LW posts ever, to see ones containing the most impressive or clever insights into rationality.
Not that I don’t think Holden’s post deserves a high score for other reasons. While I am not terribly impressed with his AI-related arguments, the post is of the very highest standards of conduct, of how to have a disagreement that is polite and far beyond what is usually named “constructive”.
My own primary inference from the popularity of this post is that there’s a lot of uncertainty/disagreement within the community about the idea that creating an AGI without an explicit (and properly tuned) moral structure constitutes significant existential risk, but that the social dynamics of the community cause most of that uncertainty/disagreement to go unvoiced most of the time.
Of course, there’s lots of other stuff going on as well that has little to do with AGI or existential risk, and a lot to do with the social dynamics of the community itself.
Well perhaps the normal practice is cult-like and dissent-suppressing and this is an atypical break. Kind of like the fat person who starts eating salad instead of nachos while he watches football. And congratulates himself on his healthy eating even though he is still having donuts for breakfast and hamburgers and french fries for lunch.
Seems to me the test for suppression of dissent is not when a high-status person criticizes. The real test is when someone with medium or low status speaks out.
And my impression is that lesswrong does have problems along these lines. Not as bad as other discussion groups, but still.
But LW isn’t reflective of SI, most of the people that voted on this article have no affiliation with SI. So the high number of upvotes is less reflective of SI welcoming criticism than LW being dissatisfied with the organization of SI.
Furthermore, this post’s criticism of Eliezer’s research less strong than its criticism of SI’s organization . SI has always been somewhat open to criticism of its organizational structure and many of the current leadership of SI has criticized the organizational structure at some point. But who criticize Eliezer’s research do not manage to rise in SI’s research division and generally aren’t well received even on LW (Roko).
Lastly, laughing at somebody when they call your organization a cult is not a convincing argument, they’re more likely to think of your organization as a cult (at least they will think you are arrogant).
How’s about you also have a critical discussion of ‘where can be we wrong and how do we make sure we are actually competent’ and ‘can we figure out what the AI will actually do, using our tools?’ instead of ‘how do we communicate our awesomeness better’ and ‘are we communicating our awesomeness right’ ?
This post is something that can’t be suppressed without losing big time, and you not suppressing it is only a strong evidence that you are not completely stupid (which is great).
I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.
Holden does not disagree with most of the basic beliefs that SI endorses. Which I think is rather sad and why I don’t view him as a real critic. And he has been very polite.
Here is the impolite version:
If an actual AI researcher would have written a similar post, someone who actually tried to build practical systems and had some economic success, not one of those AGI dreamers. If such a person would write a similar post and actually write in a way that they feel, rather than being incredible polite, things would look very different.
The trust is that you are incredible naive when it comes to technological progress. That recursive self-improvement is nothing more than a row of English words, a barely convincing fantasy. That expected utility maximization is practically unworkable, even for a superhuman intelligence. And that the lesswrong.com sequences are not original or important but merely succeed at drowning out all the craziness they include by a huge amount of unrelated clutter and an appeal to the rationality of the author.
What you call an “informed” critic is someone who shares most of your incredible crazy and completely unfounded beliefs.
Worst of all, you are completely unconvincing and do not even notice it because there are so many other people who are strongly and emotionally attached to the particular science fiction scenarios that you envision.
If such a person would write a similar post and actually write in a way that they feel, rather than being incredible polite, things would look very different.
I’m assuming you think they’d come in, scoff at our arrogance for a few pages, and then waltz off. Disregarding how many employed machine learning engineers also do side work on general intelligence projects, you’d probably get the same response from automobile engineer, someone with a track record and field expertise, talking to the Wright Brothers. Thinking about new things and new ideas doesn’t automatically make you wrong.
That recursive self-improvement is nothing more than a row of English words, a barely convincing fantasy.
Really? Because that’s a pretty strong claim. If I knew how the human brain worked well enough to build one in software, I could certainly build something smarter. You could increase the number of slots in working memory. Tweak the part of the brain that handles intuitive math to correctly deal with orders of magnitude. Improve recall to eidetic levels. Tweak the brain’s handling of probabilities to be closer to the Bayesian ideal. Even those small changes would likely produce a mind smarter than any human being who has ever lived. That, plus the potential for exponential subjective speedup, is already dangerous. And that’s assuming that the mind that results would see zero new insights that I’ve missed, which is pretty unlikely. Even if the curve bottoms out fairly quickly, after only a generation or two that’s STILL really dangerous.
Worst of all, you are completely unconvincing and do not even notice it because there are so many other people who are strongly and emotionally attached to the particular science fiction scenarios that you envision.
Really makes you wonder how all those people got convinced in the first place.
If I knew how the human brain worked well enough to build one in software, I could certainly build something smarter.
This is totally unsupported. To quote Lady Catherine de Bourgh, “If I had ever learned [to play the piano], I should have become a great proficient.”
You have no idea whether the “small changes” you propose are technically feasible, or whether these “tweaks” would in fact mean a complete redesign. For all we know, if you knew how the human brain worked well enough to build one in software, you would appreciate why these changes are impossible without destroying the rest of the system’s functionality.
After all, it would appear that (say) eidetic recall would provide a fitness advantage. Given that humans lack it, there may well be good reasons why.
“totally unsupported” seems extreme. (Though I enjoyed the P&P shoutout. I was recently in a stage adaptation of the book, so it is pleasantly primed.)
What the claim amounts to is the belief that: a) there exist good design ideas for brains that human evolution didn’t implement, and b) a human capable of building a working brain at all is capable of coming up with some of them.
A seems pretty likely to me… at least, the alternative (our currently evolved brains are the best possible design) seems so implausible as to scarcely be worth considering.
B is harder to say anything clear about, but given our experience with other evolved systems, it doesn’t strike me as absurd. We’re pretty good at improving the stuff we were born with.
Of course, you’re right that this is evidence and not proof. It’s possible that we just can’t do any better than human brains for thinking, just like it was possible (but turned out not to be true) that we couldn’t do any better than human legs for covering long distances efficiently.
I don’t doubt that it’s possible to come up with something that thinks better than the human brain, just as we have come up with something that travels better than the human leg. But to cover long distances efficiently, people didn’t start by replicating a human leg, and then tweaking it. They came up with a radically different design—e.g. the wheel.
I don’t see the evidence that knowing how to build a human brain is the key step in knowing how to build something better. For instance, suppose you could replicate neuron function in software, and then scan a brain map (Robin Hanson’s “em” concept). That wouldn’t allow you to make any of the improvements to memory, maths, etc, that Dolores suggests. Perhaps you could make it run faster—although depending on hardware constraints, it might run slower. If you wanted to build something better, you might need to start from scratch. Or, things could go the other way—we might be able to build “minds” far better than the human brain, yet never be able to replicate a human one.
But it’s not just that evidence is lacking—Dolores is claiming certainty in the lack of evidence. I really do think the Austen quote was appropriate.
To clarify, I did not mean having the data to build a neuron-by-neuron model of the brain. I meant actually understanding the underlying algorithms those slabs of neural tissue are implementing. Think less understanding the exact structure of a bird’s wing, and more understanding the concept of lift.
I think, with that level of understanding, the odds that a smart engineer (even if it’s not me) couldn’t find something to improve seem low.
I agree that I might not need to be able to build a human brain in software to be able to build something better, as with cars and legs.
And I agree that I might be able to build a brain in software without understanding how to do it, e.g., by copying an existing one as with ems.
That said, if I understand the principles underlying a brain well enough to build one in software (rather than just copying it), it still seems reasonable to believe that I can also build something better.
I agree that that the tone on both sides is intentionally respectful, and that people here delude themselves if they imagine they aren’t up for a bit of mockery from high status folks who don’t have the patience to be really engage.
I agree that we don’t really know what to expect from the first program that can meaningfully improve itself (including, I suppose, its self-improvement procedure) at a faster pace than human experts working on improving it. It might not be that impressive. But it seems likely to me that it will be a big deal, if ever we get there.
But you’re being vague otherwise. Name a crazy or unfounded belief.
But you’re being vague otherwise. Name a crazy or unfounded belief.
Holden asked me something similar today via mail. Here is what I replied:
You wrote in ‘Other objections to SI’s views’:
Unlike the three objections I focus on, these other issues have been discussed a fair amount, and if these other issues were the only objections to SI’s arguments I would find SI’s case to be strong (i.e., I would find its scenario likely enough to warrant investment in).
It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI’s from mind design space at random.
Further, the whole case for AI risk is based on the idea that there will be a huge jump in capability at some point. Which I think is at best good science fiction, like faster-than-light propulsion, or antimatter weapons (when in doubt that it is possible in principle).
The basic fact that an AGI will most likely need something like advanced nanotechnology to pose a risk, which is itself an existential risk, hints at a conjunction fallacy. We do not need AGI to then use nanotechnology to wipe us out, nanotechnology is already enough if it is possible at all.
Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.
I have written thousands of words about all this and never got any convincing reply. So if you have any specific arguments, let me know.
They say what what I write is unconvincing. But given the amount of vagueness they use to protect their beliefs, my specific criticisms basically amount to a reductio ad absurdum. I don’t even need to criticize them, they would have to support their extraordinary beliefs first or make them more specific. Yet I am able to come up with a lot of arguments that speak against the possibility they envision, without any effort and no knowledge of the relevant fields like complexity theory.
Here is a comment I received lately:
…in defining an AGI we are actually looking for a general optimization/compression/learning algorithm which when fed itself as an input, outputs a new algorithm that is better by some multiple. Surely this is at least an NP-Complete if not more problem. It may improve for a little bit and then hit a wall where the search space becomes intractable. It may use heuristics and approximations and what not but each improvement will be very hard won and expensive in terms of energy and matter. But no matter how much it tried, the cold hard reality is that you cannot compute an EXPonential Time algorithm in polynomial time unless (P=EXPTIME :S). A no self-recursive exponential intelligence theorem would fit in with all the other limitations (speed, information density, Turing, Gödel, uncertainties etc) the universe imposes.
If you were to turn IBM Watson gradually into a seed AI, at which point would it become an existential risk and why? They can’t answer that at all. It is pure fantasy.
The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI’s from mind design space at random.
I don’t have the energy to get into an extended debate, but the claim that this is “the basic idea” or that this would be “the strongest argument” is completely false. A far stronger basic idea is the simple fact that nobody has yet figured out a theory of ethics that would work properly, which means that even that AGIs that were specifically designed to be ethical are most likely to lead to bad outcomes. And that’s presuming that we even knew how to program them exactly.
I did skim through the last paper. I am going to review it thoroughly at some point.
On first sight one of the problems is the whole assumption of AI drives. On the one hand you claim that an AI is going to follow its code, is its code (as if anyone would doubt causality). On the other hand you talk about the emergence of drives like unbounded self-protection. And if someone says that unbounded self-protection does not need to be part of an AGI, you simply claim that your definition of AGI will have those drives. Which allows you to arrive at your desired conclusion of AGI being an existential risk.
Another problem is the idea that an AGI will be a goal executor (I can’t help but interpret that to be your position) when I believe that the very nature of artificial general intelligence implies the correct interpretation of “Understand What I Mean” and that “Do What I Mean” is the outcome of virtually any research. Only if you were to pull an AGI at random from mind design space could you possible arrive at “Understand What I Mean” without “Do What I Mean”.
To see why look at any software product or complex machine. Those products are continuously improved. Where “improved” means that they become better at “Understand What I Mean” and “Do What I Mean”.
There is no good reason to believe that at some point that development will suddenly turn into “Understand What I Mean” and “Go Batshit Crazy And Do What I Do Not Mean”.
There are other problems with the paper. I hope I will find some time to write a review soon.
One problem for me with reviewing such papers is that I doubt a lot of underlying assumptions like that there exists a single principle of general intelligence. As I see it there will never be any sudden jump in capability. I also think that intelligence and complex goals are fundamentally interwoven. An AGI will have to be hardcoded, or learn, to care about a manifold of things. No simple algorithm, given limited computational resources, will give rise to the drives that are necessary to undergo strong self-improvement (if that is possible at all).
It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI’s from mind design space at random.
An AI’s mind doesn’t have to be pulled from design space at random to be disastrous. The primary issue that the SIAI has to grapple with (based on my understanding,) is that deliberately designing an AI that does what we would want it to do, rather than fulfilling proxy criteria in ways that we would not like at all, is really difficult. Even getting one to recognize “humans” as a category in a way that would be acceptable to us is a major challenge.
Although it’s worth pointing out that this is also an obstacle to AGI, since presumably an AI that did not understand what a human was would be pretty unintelligent. So I think it’s unfair to claim this as a “friendliness” issue.
Note that I do think there are some important friendliness-related problems, but, assuming I understand your objection, this is not one of them.
An AI could be an extremely powerful optimizer without having a category for “humans” that mapped to our own. “Human,” the way we conceive of it, is a leaky surface generalization.
A strong paperclip maximizer would understand humans as well as it had to to contend with us in its attempts to paperclip the universe, but it wouldn’t care about us. And a strong optimizer programmed to maximize the values of “humans” would also probably understand us, but if we don’t program into its values an actual category that maps to our conception of humans, it could perfectly well end up applying that understanding to, for example, tiling the universe with crash test dummies.
How do you intend to build a powerful optimizer without having a method of representing (or of building a representation of) the concept of “human” (where “human” can be replaced with any complex concept, even probably paperclips)?
I agree that value specification is a hard problem. But I don’t think the complexity of “human” is the reason for this, although it does rule out certain simple approaches like hard-coding values.
(Also, since your link seems to indicate you believe otherwise, I am fairly familiar with the content in the sequences. Apologies if this statement represents an improper inference.)
How do you intend to build a powerful optimizer without having a method of representing (or of building a representation of) the concept of “human” (where “human” can be replaced with any complex concept, even probably paperclips)?
If a machine can learn, empirically, exactly what humans are, on the most fundamental levels, but doesn’t have any values associated with them, why should it need a concept of “human?” We don’t have a category that distinguishes igneous rocks that are circular and flat on one side, but we can still recognize them and describe them precisely.
Humans are an unnatural category. Whether a fetus, an individual in a persistent vegetative state, an amputee, a corpse, an em or a skin cell culture fall into the category of “human” depends on value-sensitive boundaries. It’s not necessarily because humans are so complex that we can’t categorize them in an appropriate manner for an AI (or at least, not just because humans are complex,) it’s because we don’t have an appropriate formulation of the values that would allow a computer to draw the boundaries of the category in a way we’d want it to.
(I wasn’t sure how familiar you were with the sequences, but in any case I figured it can’t hurt to add links for anyone who might be following along who’s not familiar.)
I’ve read most of that now, and have subscribed to your newsletter.
Reasonable people can disagree in estimating the difficulty of AI and the visibility/pace of AI progress (is it like hunting for a single breakthrough and then FOOM? etc).
I find all of your “it feels ridiculous” arguments by analogy to existing things interesting but unpersuasive.
Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.
Says the wooly mammoth, circa 100,000 BC.
Sounding silly and low status and science-fictiony doesn’t actually make it unlikely to happen in the real world.
Especially when not many people want to read a science fiction book where humanity gets quickly and completely wiped out by a superior force. Even works where humans slowly die off due to their own problems (e.g. On the Beach) are uncommon.
Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible
Do you acknowledge that :
We will some day make an AI that is at least as smart as humans?
Humans do try to improve their intelligence (rationality/memory training being a weak example, cyborg research being a better example, and im pretty sure we will soon design physical augmentations to improve our intelligence)
If you acknowledge 1 and 2, then that implies there can (and probably will) be an AI that tries to improve itself
I think you missed the “quickly and vastly” part as well as the “and then invent all kinds of technological magic to wipe us out”. Note I still think XiXiDu is wrong to be as confident as he is (assuming “there will never” implies >90% certainty), but if you are going to engage with him then you should engage with his actual arguments.
And that the lesswrong.com sequences are not original or important but merely succeed at drowning out all the craziness they include by a huge amount of unrelated clutter and an appeal to the rationality of the author.
Name three examples? (Of ‘craziness’ specifically… I agree that there are frequent, and probably unecessary, “appeals to the rationality of the author”.)
Name three examples? (Of ‘craziness’ specifically… I agree that there are frequent, and probably unecessary, “appeals to the rationality of the author”.)
XiXiDu may be too modest; he has some great examples on his blog.
One wonders when or if XiXiDu will ever get over the Roko incident. Yes, it was a weird and possibly disproportionate response, but it was also years ago.
Why yes, I do also believe that political figures are held to ridiculous conversational standards as well. It’s a miracle they deign to talk to anyone.
Name three examples? (Of ‘craziness’ specifically… I agree that there are frequent, and probably unecessary, “appeals to the rationality of the author”.)
So, Swimmer 963, are those quotes crazy enough for you? (I hope you don’t ask a question and neglect to comment on the answer.) What you do think? Anomalous?
Contrary to the impression the comments might convey, the majority don’t come from the Roko incident. But as to that incident, the passage of time doesn’t necessarily erase the marks of character. Romney is rightfully being held, feet to fire, for a group battering of another student while they attended high school—because such sadism is a trait of character and can’t be explained otherwise. How would one explain Yudkowsky’s paranoia, lack of perspective, and scapegoating—other than by positing a narcissistic personality structure?
Many LWers can’t draw conclusions because they eschew the only tools for that purpose: psychology and excellent fiction. And the second is more important than the first.
How would one explain Yudkowsky’s paranoia, lack of perspective, and scapegoating—other than by positing a narcissistic personality structure?
I had in fact read a lot of those quotes before–although some of them come as a surprise, so thank you for the link. They do show paranoia and lack of perspective, and yeah, some signs of narcissism, and I would be certainly mortified if I personally ever made comments like that in public…
The Sequences as a whole do come across as having been written by an arrogant person, and that’s kind of irritating, and I have to consciously override my irritation in order to enjoy the parts that I find useful, which is quite a lot. It’s a simplification to say that the Sequences are just clutter, and it’s extreme to call them ‘craziness’, too.
(Since meeting Eliezer in person, it’s actually hard for me to believe that those comments were written by the same person, who was being serious about them… My chief interaction with him was playing a game in which I tried to make a list of my values, and he hit me with a banana every time I got writer’s block because I was trying to be too specific, and made the Super Mario Brothers’ theme song when I succeeded. It’s hard making the connection that “this is the same person who seems to take himself way too seriously in his blog comments.” But that’s unrelated and doesn’t prove anything in either direction.)
My main point is that criticizing someone who believes in a particular concept doesn’t irrefutably damn that concept. You can use it as weak evidence, but not proof. Eliezer, as far as I know, isn’t the only person who has thought extensively about Friendly AI and found it a useful concept to keep.
Take metaethics, a solved problem: what are the odds that someone who still thought metaethics was a Deep Mystery could write an AI algorithm that could come up with a correct metaethics? I tried that, you know, and in retrospect it didn’t work.
Yudkowsky makes the megalomanic claim that he’s solved the questions of metaethics. His solution: morality is the function that the brain of a fully informed subject computes to determine what’s right. Laughable; pathologically arrogant.
Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.
The most extreme presumptuousness about morality; insufferable moralism. Morality, as you were perhaps on the cusp of recognizing in one of your posts, Swimmer963, is a personalized tool, not a cosmic command line. See my “Why do what you “ought”?—A habit theory of explicit morality.”
The preceding remark, I’ll grant, isn’t exactly crazy—just super obnoxious and creepy.
Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn’t need a social process of science
right?
This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination. That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language—an insight that should be almost intuitive to a rationalist. But noncrazy people may believe the foregoing; what they don’t believe is that they can at the present time replace the institution of science with the reasoning of smart people. That’s the absolutely bonkers claim Yudkowsky makes.
I didn’t say they were. I said that just because the speaker for a particular idea comes across as crazy doesn’t mean the idea itself is crazy. That applies whether all of Eliezer’s “crazy statements” are about AI, or whether none of them are.
Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.
The most extreme presumptuousness about morality; insufferable moralism.
Funny, I actually agree with the top phrase. It’s written in an unfortunately preachy, minister-scaring-the-congregation-by-saying-they’ll-go-to-Hell style, which is guaranteed to make just about anyone get defensive and/or go “ick!” But if you accept the (very common) moral standard that if you can save a life, it’s better to do it than not to do it, then the logic is inevitable that if you have the choice of saving one lives or two lives, by your own metric it’s morally preferable to save two lives. If you don’t accept the moral standard that it’s better to save one life than zero lives, then that phrase should be just as insufferable.
Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn’t need a social process of science right?
I decided to be charitable, and went and looked up the post that this was in: it’s here. As far as I can tell, Eliezer doesn’t say anything that could be interpreted as “science exists because people are stupid, and I’m not stupid, therefore I don’t need science”. He claims that scientific procedures compensates for people being unwilling to let go of their pet theories and change their minds, and although I have no idea if this goal was in the minds of the people who came up with the scientific method, it doesn’t seem to be false that it accomplishes this goal.
Newton definitely wrote down his version of scientific method to explain why people shouldn’t take his law of gravity and just add, “because of Aristotelian causes,” or “because of Cartesian mechanisms.”
This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination.
Ok. I disagree with a large bit of the sequences on science and the nature of science. I’ve wrote a fair number of comments saying so. So I hope you will listen when I say that you are taking a strawman version of what Eliezer wrote on these issues, and it almost borders on something that I could only see someone thinking if they were trying to interpret Eliezer’s words in the most negative fashion possible.
His solution: morality is the function that the brain of a fully informed subject computes to determine what’s right. Laughable; pathologically arrogant.
You either didn’t read that sequence carefully, or are intentionally misrepresenting it.
He thinks the social institution of science is superfluous, were everyone as smart as he.
Didn’t read that sequence carefully either.
That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language—an insight that should be almost intuitive to a rationalist.
You didn’t read that sequence at all, and probably don’t actually know what simplicity means in an information-theoretic sense.
That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language—an insight that should be almost intuitive to a rationalist.
You didn’t read that sequence at all, and probably don’t actually know what simplicity means in an information-theoretic sense.
To be fair, that sequence doesn’t really answer questions about choice-of-language; it took reading some of Solomonoff’s papers for me to figure out what the solution to that problem is.
That’s true; I admit I didn’t read the sequence. I had a hard time struggling through the single summating essay. What I wrote was his conclusion. As Hanson wrote in the first comment to the essay I did read, Yudkowsky really should summarize the whole business in a few lines. Yudkowsky didn’t get around to that, as far as I know.
The summation essay contained more than 7,000 words for the conclusion I quoted. Maybe the rest of the series contradicts what is patent in the essay I read.
I simply don’t get the attraction of the sequences. An extraordinarily high ratio of filler to content; Yudkowsky seems to think that every thought along the way to his personal enlightenment is worth the public’s time.
Asking that a critic read those sequences in their entirety is asking for a huge sacrifice; little is offered to show it’s even close in being worth the misery of reading inept writing or the time.
You know, the sequences aren’t actually poorly written. I’ve read them all, as have most of the people here. They are a bit rambly in places, but they’re entertaining and interesting. If you’re having trouble with them, the problem might be on your end.
In any case, if you had read them, you’d know, for instance, that when Yudkowsky talks about simplicity, he is not talking about the simplicity of a given English sentence. He’s talking about the combined complexity of a given Turing machine and the program needed to describe your hypothesis on that Turing machine.
89 people (8.2%) have never looked at the Sequences; a further 234 (32.5%) have only given them a quick glance. 170 people have read about 25% of the sequences, 169 (15.5%) about 50%, 167 (15.3%) about 75%, and 253 people (23.2%) said they’ve read almost all of them. This last number is actually lower than the 302 people who have been here since the Overcoming Bias days when the Sequences were still being written (27.7% of us).
In addition, there are places in the Sequences where Eliezer just states things as though he’s dispensing wisdom from on high, without bothering to state any evidence or reasoning. His writing is still entertaining, of course, but still less than persuasive.
You know, the sequences aren’t actually poorly written. I’ve read them all, as have most of the people here. They are a bit rambly in places, but they’re entertaining and interesting. If you’re having trouble with them, the problem might be on your end.
The problem is partly on my end, for sure; obviously, I find rambling intolerable in Internet writing, and I find it in great abundance in the sequences. You’re more tolerant of rambling, and you’re entertained by Yudkowsky’s. I also think he demonstrates mediocre literary skills when it comes to performances like varying his sentence structure. I don’t know what you think of that. My guess is you don’t much care; maybe it’s a generational thing.
I’m intrigued by what enjoyment readers here get from Yudkowsky’s sequences. Why do you all find interesting what I find amateurish and inept? Do we have vastly different tastes or standards, or both? Maybe it is the very prolixity that makes the writing appealing in founding a movement with religious overtones. Reading Yudkowsky is an experience comparable to reading the Bible.
As a side issue, I’m dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.
Of course I understand simplicity doesn’t pertain to simplicity in English! (Or in any natural language.) I don’t think you understand the language-relativity issue.
If you were willing to point me to two or three of your favorite Internet writers, whom you consider reliably enjoyable and interesting and so forth, I might find that valuable for its own sake, and might also be better able to answer your question in mutually intelligible terms.
As a side issue, I’m dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.
Having to have original ideas is a very high standard. I doubt a single one of my posts contains a truly original idea, and I don’t try–I try to figure out which ideas are useful to me, and then present why, in a format that I hope will be useful to others. Eliezer creates a lot of new catchy terms for pre-existing ideas, like “affective death spiral” for “halo effect.” I like that.
His posts are also quite short, often witty, and generally presented in an easier-to-digest format than the journal articles I might otherwise have to read to encounter the not-new ideas. You apparently don’t find his writing easy to digest or amusing in the same way I do.
Affective death spiral is not the same thing as the Halo effect, though the halo effect (/ horns effect) might be part of the mechanism of affective death spiral.
Agreed… I think the Halo effect is a sub-component of an affective death spiral, and “affective death spiral” is a term unique to LW [correct me if I’m wrong!], while ‘Halo effect’ isn’t.
I don’t know any specific examples of secondhand ideas coming off as original (indeed, he often cites experiments from the H&B literature), but there’s another possible source for the confusion. Sometimes Yudkowsky and somebody else come up with ideas independently, and those aren’t cited because Yudkowsky didn’t know they existed at the time. Drescher and Quine are two philosophers who have been mentioned as having some of the same ideas as Yudkowsky, and I can confirm the former from experience.
I’m intrigued by what enjoyment readers here get from Yudkowsky’s sequences. Why do you all find interesting what I find amateurish and inept?
I find his fictional interludes quite entertaining, because they are generally quite lively, and display a decent amount of world-building—which is one aspect of science fiction and fantasy that I particularly enjoy. I also enjoy the snark he employs when trashing opposing ideas, especially when such ideas are quite absurd. Of course, the snark doesn’t make his writing more persuasive—just more entertaining.
he demonstrates mediocre literary skills when it comes to performances like varying his sentence structure
I know I’m exposing my ignorance here, but I’m not sure what this means; can you elaborate ?
Asking that a critic read those sequences in their entirety is asking for a huge sacrifice; little is offered to show it’s even close in being worth the misery of reading inept writing or the time.
Indeed, the sequences are long. I’m not sure about the others here, but I’ve never asked anybody to “read the sequences.”
But I don’t even know how to describe the arrogance required to believe that you can dismiss somebody’s work as “crazy,” “stupid,” “megalomanic,” “laughably, pathologically arrogant,” “bonkers,” and “insufferable” without having even read enough of what you’re criticizing the get an accurate understanding of it.
ETA: Edited in response to fubarobfusco, who brought up a good point.
That’s a fully general argument against criticizing anything without having read all of it, though. And there are some things you can fairly dismiss without having read all of. For instance, I don’t have to read every page on the Time Cube site to dismiss it as crazy, stupid, pathologically arrogant, and so on.
The reason EY wrote an entire sequence on metaethics is precisely because without the rest of the preparation people such as you who lack all that context immediately veer off course and start believing that he’s asserting the existence (or non-existence) of “objective” morality, or that morality is about humans because humans are best or any other standard philosophical confusion that people automatically come up with whenever they think about ethics.
Of course this is merely a communication issue. I’d love to see a more skilled writer present EY’s metaethical theory in a shorter form that still correctly conveys the idea, but it seems to be very difficult (especially since even half the people who do read the sequence still come away thinking it’s moral relativism or something).
I read your post on habit theory, and I liked it, but I don’t think it’s an answer to the question “What should I do?”
It’s interesting to say that if you’re an artist, you might get more practical use out of virtue theory, and if you’re a politician, you might get more practical use out of consequentialism. I’m not sure who it is that faces more daily temptations to break the rules than the rest of us; bankers, I suppose, and maybe certain kinds of computer security experts.
Anyway, saying that morality is a tool doesn’t get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer’s answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you ‘should’ do, and that’s all there is to it. This is not exactly a brilliant answer, but it may nevertheless be the best one out there. I’ve never yet heard a moral theory that made more sense than that, and believe me, I’ve looked.
It’s reasonable to insist that people put their conclusions in easily digestible bullet points to convince you to read the rest of what they’ve written...but if, noting that there are no such bullet points, you make the decision not to read the body text—you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he’s not stupid, and he’s not crazy. If you want to offer a meaningful critique of his ideas, you’ll have to read them first.
but if, noting that there are no such bullet points, you make the decision not to read the body text—you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he’s not stupid, and he’s not crazy.
This is sound general advice, but at least one observation makes this situation exceptional: Yudkowsky’s conclusions about ethics are never summarized in terms that contradict my take. I don’t think your rendition, for example, contradicts mine. I’m certainly not surprised to hear his position described the way you describe it:
Anyway, saying that morality is a tool doesn’t get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer’s answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you ‘should’ do, and that’s all there is to it.
Now, I don’t think the decision of whether to be an artist, politician, or banker is a moral decision. It isn’t one you make primarily because of what’s ethically right or wrong. To the extent you do (and in the restricted sense that you do), your prior moral habits are your only guide.
But we’re looking at whether Yudkowsky’s position is intellectually respectable, not whether objective morality—which he’s committed to but I deny—exists. To say we look at what our brain says when we’re fully informed says essentially that we seek a reflective equilibrium in solving moral problems. So far so good. But it goes further in saying brains compute some specific function that determines generally when individuals reach that equilibrium. Leaving aside that this is implausible speculation, requiring that the terms of moral judgments be hardwired—and hardwired identically for each individual—it also simply fails to answer Moore’s open question, although Yudkowsky claims he has that answer. There’s nothing prima facie compelling ethically about what our brains happen to tell us is moral; no reason we should necessarily follow our brains’ hardwiring. I could consistently choose to consider my brain’s hardwired moralisms maladaptive or even despicable holdovers from the evolutionary past that I choose to override as much as I can.
Robin Hanson actually asked the right question. If what the brain computes is moral, what does it correspond to that makes it moral? Unless you think the brain is computing a fact about the world, you can’t coherently regard its computation as “accurate.” But if not, what makes it special and not just a reflex?
I do feel a bit guilty about criticizing Yudkowsky without reading all of him. But he seems to express his ideas at excessive and obfuscating length, and if there were more to them, I feel somewhat confident I’d come across his answers. It isn’t as though I haven’t skimmed many of these essays. And his answers would certainly deserve some reflection in his summation essay.
There’s no question Yudkowsky is no idiot. But he has some ideas that I think are stupid—like his “metaethics”—and he expresses them in a somewhat “crazy” manner, exuding grandiose self-confidence. Being surrounded and discussing mostly with people who agree with him is probably part of the cause.
As someone who has read Eliezer’s metaethics sequence, let me say that what you think his position is, is only somewhat related to what it actually is; and also, that he has answered those of your objections that are relevant.
It’s fine that you don’t want to read 30+ fairly long blog posts, especially if you dislike the writing style. But then, don’t try to criticize what you’re ignorant about. And no, openly admitting that you haven’t read the arguments you’re criticizing, and claiming that you feel guilty about it, doesn’t magically make it more acceptable. Or honest.
One doesn’t need to have read the whole Bible to criticize it. But the Bible is a fairly short work, so an even more extreme example might be better: one doesn’t need to have read the entire Talmud to criticize it.
It’s fine that you don’t want to read 30+ fairly long blog posts, especially if you dislike the writing style. But then, don’t try to criticize what you’re ignorant about. And no, openly admitting that you haven’t read the arguments you’re criticizing, and claiming that you feel guilty about it, doesn’t magically make it more acceptable. Or honest.
It’s hardly “dishonest” to criticize a position based on a 7,000-word summary statement while admitting you haven’t read the whole corpus! You’re playing with words to make a moralistic debating point: dishonesty involves deceit, and everyone has been informed of the basis for my opinions.
Consider the double standard involved. Yudkowsky lambasts “philosophers” and their “confusions”—their supposedly misguided concerns with the issues other philosophers have commented on to the detriment of inquiry. Has Yudkowsky read even a single book by each of the philosophers he dismisses?
In a normal forum, participants supply the arguments supposedly missed by critics who are only partially informed. Here there are vague allusions to what the Apostle Yudkowsky (prophet of the Singularity God) “answered” without any substance. An objective reader will conclude that the Prophet stands naked; the prolixity is probably intended to discourage criticism.
I think the argument you make in this comment isn’t a bad one, but the unnecessary and unwarranted “Apostle Yudkowsky (prophet of the Singularity God)” stuff amounts to indirectly insulting the people you’re talking with and, makes them far less likely to realize that you’re actually also saying something sensible. If you want to get your points across, as opposed to just enjoying a feeling of smug moral superiority while getting downvoted into oblivion, I strongly recommend leaving that stuff out.
Thanks for the advice, but my purpose—given that I’m an amoralist—isn’t to enjoy a sense of moral superiority. Rather, to test a forum toward which I’ve felt ambivalent for several years, mainly for my benefit but also for that of any objective observers.
Strong rhetoric is often necessary in an unreceptive forum because it announces that the writer considers his criticisms fundamental. If I state the criticisms neutrally, something I’ve often tried, they are received as minor—like the present post. They may even be voted up, but they have little impact. Strong language is appropriate in expressing severe criticisms.
How should a rationalist forum respond to harsh criticism? It isn’t rational to fall prey to the primate tendency to in-group thinking by neglecting to adjust for any sense of personal insult when the group leader is lambasted. Judging by reactions, the tendency to in-group thought is stronger here than in many forums that don’t claim the mantle of rationalism. This is partly because the members are more intelligent than in most other forums, and intelligence affords more adept self-deception. This is why it is particularly important for intelligent people to be rationalists but only if they honestly strive to apply rational principles to their own thinking. Instead, rationality here serves to excuse participants’ own irrationality. Participants simply accept their own tendencies to reject posts as worthless because they contain matter they find insulting. Evolutionary psychology, for instance, here serves to produce rationalizations rather than rationality. (Overcoming Bias is a still more extreme advocacy of this perversion of rationalism, although the tendency isn’t expressed in formal comment policies.)
“Karma” means nothing to me except as it affects discourse; I despise even the term, which stinks of Eastern mysticism. I’m told that the karma system of incentives, which any rationalist should understand vitally affects the character of discussion, was transplanted from reddit. How is a failure to attend to the vital mechanics of discussion and incentives rational? Laziness? How could policies so essential be accorded the back seat?
Participants, I’m told, don’t question the karma system because it works. A rationalist doesn’t think that way. He says, “If a system of incentives introduced without forethought and subject to sound criticisms (where even its name is an insult to rationality) produces the discourse that we want, then something must be wrong with what we want!” What’s wanted is the absence of any tests of ideology by fundamental dissent.
I think the argument you make in this comment isn’t a bad one, but the unnecessary and unwarranted “Apostle Yudkowsky (prophet of the Singularity God)” stuff amounts to indirectly insulting the people you’re talking with and, makes them far less likely to realize that you’re actually also saying something sensible. If you want to get your points across, as opposed to just enjoying a feeling of smug moral superiority while getting downvoted into oblivion, I strongly recommend leaving that stuff out.
Consider the double standard involved. Yudkowsky lambasts “philosophers” and their “confusions”—their supposedly misguided concerns with the issues other philosophers have commented on to the detriment of inquiry. Has Yudkowsky read even a single book by each of the philosophers he dismisses?
Some of them are simply not great writers. Hegel for example is just awful- the few coherent ideas in Hegel are more usefully described by other later writers. There’s also a strange aspect to this in that you are complaining about Eliezer not having read books while simultaneously defending your criticism of Eliezer’s metaethics positions without having read all his posts. Incidentally, if one wants to criticize Eliezer’s level of knowledge of philosophy, a better point is not so much the philosophers that he criticizes without reading, but rather his lack of knowledge of relevant philosophers that Eliezer seems unaware of, many of whom would agree with some of his points. Quine and Lakatos are the most obvious ones.
Here there are vague allusions to what the Apostle Yudkowsky (prophet of the Singularity God) “answered” without any substance. An objective reader will conclude that the Prophet stands naked; the prolixity is probably intended to discourage criticism.
I strongly suspect that your comments would be responded to more positively if they didn’t frequently end with this sort of extreme rhetoric that has more emotional content than rational dialogue. It is particularly a problem because on theLW interface, the up/down buttons are at the end of everything one has read, so what the last sentences say may have a disproportionate impact on whether people upvote or downvote and what they focus on in their replies.
Frankly, you have some valid points, but they are getting lost in the rhetoric. We know that you think that LW pattern matches to religion. Everyone gets the point. You don’t need to repeat that every single time you make a criticism.
I could consistently choose to consider my brain’s hardwired moralisms maladaptive or even despicable holdovers from the evolutionary past that I choose to override as much as I can.
And you would be making the decision to override with… what, your spleen?
But Yudkowsky says “built around the assumption that you’re too stupid… to just use …”
If Solomonoff induction can’t easily be used in place of science, why does the first sentence imply the process is simple: you just use it?
You’ve clarified what Yudkowsky does not mean. But what does he mean? And why is it so hard to find out? This is the way mystical sects retain their aura while actually saying little.
“You’re too stupid and self-deceiving to just use Solomonoff induction” ~ “If you were less stupid and self deceiving you’d be able to just use Solomonoff induction” + “but since you are in fact stupid and self-deceiving, instead you have to use the less elegant approximation Science”
Actually, yes, because of the misleading signals in the inept writing. But thank you for clarifying.
Conclusion: The argument in written in a crazy fashion, but it really is merely stupid. There is no possible measure of simplicity that isn’t language relative. How could there be?
You seem to be confusing “language relative” with “non-mathematical.” Kolmogorov Complexity is “language-relative,” if I’m understanding you right; specifically, it’s relative (if I’m using the terminology right?) to a Turing Machine. This was not relevant to Eliezer’s point, so it was not addressed.
(Incidentally, this is a perfect example of you “hold{ing} views contrary to scientific consensus in specialized fields where {you} lack expert knowledge based on pure ratiocination,” since Kolmogorov Complexity is “one of the fundamental concepts of theoretical computer science”, you seemingly lack expert knowledge since you don’t recognize these terms, and your argument seems to be based on pure ratiocination.)
When I read that line for the first time, I understood it. Between our two cases, the writing was the same, but the reader was different. Thus, the writing cannot be the sole cause of our different outcomes.
Well, if a substantial fraction of readers read something differently or can’t parse it, it does potentially reflect a problem with the writing even if some of the readers, or even most readers, do read it correctly.
Absolutely. I intended to convey that if you don’t understand something, that the writing is misleading and inept is not the only possible reason. srdiamond is speaking with such confidence that I felt safe tabling further subtleties for now.
I can’t tell which way your sarcasm was supposed to cut.
The obvious interpretation is that you think rationality is somehow hindered by paying attention to form rather than substance, and the “exemplary rationality” was intended to be mocking.
But your comment being referenced was an argument that form has something very relevant to say about substance, so it could also be that you were actually praising gwern for practicing what you preach.
I read your three-part series. Your posts did not substantiate the claim “good thinking requires good writing.” Your second post slightly increased my belief in the converse claim, “good thinkers are better-than-average writers,” but because the only evidence you provided was a handful of historical examples, it’s not very strong evidence. And given how large the population of good thinkers, good writers, bad thinkers, and bad writers is relative to your sample, evidence for “good thinking implies good writing” is barely worth registering as evidence for “good writing implies good thinking.”
Romney is rightfully being held, feet to fire, for a group battering of another student while they attended high school—because such sadism is a trait of character and can’t be explained otherwise.
I was going to upvote your comment until I got to this point. Aside from the general mindkilling, this looks like the fundamental attribution error, and moreover, we all know that people do in fact mature and change. Bringing up external politics is not helpul in a field where there’s already concern that AI issues may be becoming a mindkilling subject themselves on LW. Bringing up such a questionable one is even less useful.
That’s LW “rationality” training for you—”fundamental error of attribution” out of context—favored because it requires little knowledge and training in psychology. Such thinking would preclude any investigation of character. (And there are so many taboos! How do you all tolerate the lockstep communication required here?)
Paul Meehl, who famously studied clinical versus statistical prediction empirically, noted that even professionals, when confronted by instance of aberrant behavior, are apt to call it within normal range when it clearly isn’t. Knowledge of the “fundamental error of attribution” alone is the little bit of knowledge that’s worse than total ignorance.
Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.
You can’t dismiss politics just because it kills some people’s minds, when so much of the available information and examples come from politics. (There are other reasons, but that’s the main one here.) Someone who can’t be rational about politics simply isn’t a good rationalist. You can’t be a rationalist about the unimportant things and rationalist about the important ones—yet call yourself a rationalist overall.
I’m sure I wouldn’t have done what Romney did, and not so sure about whether I would have done what Yudkowsky did. Romney wanted to hurt people for the fun of it. Yudkowsky was trying to keep people from being hurt, regardless of whether his choice was a good one.
It seems almost unfair to criticize something as a problem of LW rationality when in your second paragraph you note that professionals do the same thing.
Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.
I’m not sure. A while ago, I was involved in a situation where someone wanted to put personal information of an individual up on the internet knowing that that person had an internet stalker who had a history of being a real life stalker for others. The only reason I didn’t react pretty close to how Eliezer reacted in the quoted incident is that I knew that the individual in question was not going to listen to me and would if anything have done the opposite of what I wanted. In that sort of context, Eliezer’s behavior doesn’t seem to be that extreme. Eliezer’s remarks involve slightly more caps than I think I would use in such a circumstance, but the language isn’t that different.
This does connect to another issue though- the scale in question of making heated comments on the internet as opposed to traumatic bullying, are different. The questions I ask myself for what it would take to do something similar to what Eliezer did are very different than the same questions for the Romney incident.
Your basic statement does it seem have some validity. One could argue that the Romney matter reflects the circumstances where he was at the time, and what was considered socially acceptable as forms of interaction or establishing dominance hierarchies. Through most of human history, that sort of behavior would probably be considered fairly tame. But this is a weak argument- even if it was due to the circumstances that Romney was in at the time, there’s no question that those were his formative years, and thus could plausibly have had a permanent impact on his moral outlook.
You can’t dismiss politics just because it kills some people’s minds, when so much of the available information and examples come from politics.
The problem is that even as relevant examples come from politics, those are precisely the examples that people are least likely to agree actually demonstrate the intended point in question. For example, in this case, many people who aren’t on the left will downplay the Romney bullying. Given that I’m someone who dislikes Romney (both in terms of personality and in terms of policy) and am not convinced that this is at all fair, using such a controversial example seems unwise. Even if one needs to use political examples, one can use examples from 10 or 15 or 30 years ago that are well known but have had their tribalness diminish in time. For example, in this context one could use a variety of examples connected to Richard Nixon.
Someone who can’t be rational about politics simply isn’t a good rationalist. You can’t be a rationalist about the unimportant things and rationalist about the important ones—yet call yourself a rationalist overall.
Well, we can acknowledge that we’re better at being rational in some areas than we are in others. Frankly, I wouldn’t mind and for reasons essentially similar to your remark would endorse some amount of reduction of the no-politics rule here. Where that becomes a problem is when one tries to connect politics to other potentially controversial issues.
When Mitt Romney was in high school, he and some friends bullied a kid who looked (and later turned out to be) homosexual. At one point, Romney and some others grabbed the guy, held him down, and cut off a bunch of his hair with scissors.
The point, such as it is, would better have been left implied. Now, it’s subject to explicit scrutiny, and it must be found wanting. Consider what would have happened had Yudkowsky not shown exceptional receptivity to this post: he would have blatantly proven his critics right. The knowledge and reputation of the poster is unimpeachable.
The more significant fact is that these criticisms were largely unknown to the community. As Will Newsome implied, this is because the critical posts—lacking the high-status credential of this poster—remained in discussion and were almost ignored.
The majority’s intolerance for dissent is manifested mostly in its refusal to acknowledge it. Dissent is cabined to Discussion. It only gets noticed when the dissenter becomes frustrated and violates group norms. Then it gets voted down, but it still gets noticed and commented on. This is a malfunctioning reinforcement system, but maybe its the best possible. Still, it’s irrational to deny all in-group bias in LukeProg’s cheerleading fashion—in an instance where the absence of evidence (here, of bias) truly does not offer anything substantial in the way of evidence of lack of bias, to elicit LukeProg’s smug laughter.
After all, even the lead poster held off until now in voicing his opinion.
[Holden’s] critique mostly consists of points that are pretty persistently bubbling beneath the surface around here, and get brought up quite a bit. Don’t most people regard this as a great summary of their current views, rather than persuasive in any way? In fact, the only effect I suspect this had on most people’s thinking was to increase their willingness to listen to Karnofsky in the future if he should change his mind.
I had a post moved from main to Discussion just today: before it had accumulated any negative votes, so I think you’re probably misinformed about editorial practices.But I don’t want to use my posts as evidence; the charge of bias would be hard to surmount. What’s plainly evident is that posters are reluctant to post to the Main area except by promotion.
You’re evidence is unpersuasive because you don’t weigh it against the evidence to the contrary. One good example to the contrary more than counter-balances it, since the point isn’t that no dissent is tolerated, not even that some dissent isn’t welcomed, but only that there are some irrational boundaries.
One is the quasi-ban on politics. Here is a comment that garnered almost 800 responses and was voted up 37. Why wasn’t it promoted? I bitterly disagree with the poster; so I’m not biased by my views. But the point is that it is a decidedly different view, one generating great interest, but the subject would not be to the liking of the editors.
Of course, it lacked the elaborateness—dare I say, the prolixity—of a typical top-level post. But this “scholarly” requirement is part of the process of soft censorship. The post—despite my severe disagreement with it—is a more significant intellectual contribution than many of the top-level posts, such as some of the second-hand scholarship.
[And I have to add: observe that the present discussion is already being downvoted at my first comment. I predict the same for this post in record timeWhat does that mean?]
Here is a comment that garnered almost 800 responses and was voted up 37. Why wasn’t it promoted?
Can comments be promoted? Perhaps the commenter should have been encouraged to turn his comment into a top-level post, but a moderator can’t just change a comment into a promoted post with the same username. Also it would have split the discussion, so people might have been reluctant to encourage that.
As for people tending to post more in Discussion than Main, I read somewhere that Discussion has more readers. I for one read Discussion almost exclusively.
I downvoted you because you’re wrong. For one, comments can’t be promoted to main, only posts, and for two, plenty of opposition has garnerned a great deal of upvotes, as shown by the numerous links lukeprog provided.
For example, where do you get ‘almost 800 responses’ from? That comment (not post) only has 32 comments below it.
Yes, I was wrong. But my point was correct. The 781 comments applied to the Main Post So:
The topic was popular, like I said.
The post could have been promoted!
But ask yourself, would you have been so harsh on a factual error had you agreed with the message? This is the way bias works, after all, by double standard more than outright discrimination. You could say I should have been more careful. But then, when you’ve learned not to expect a hearing, you’re not so willing to jump the hoops. But it’s your loss, if you’re a rationalist and if you’re losing input because dissenters find it’s not worth their time.
As to LukeProg providing example demonstrating welcoming dissent: you couldn’t have considered my counter-balancing evidence when you downvoted before taking the time even to explore the post to which the cited comment belongs.
To LukeProg: have I made my point about the limits of dissent at LW?
I’ve addressed factual inaccuracies in another comment. But as for discussing karma effects—that wasn’t extraneous whining but was at the heart of the discussion. If you downvote discussion of karma—like you did—simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?
LukeProg: What do you say about the grounds on which downvotes are issued for dissenting matter. Isn’t it clear that this is a bias LW doesn’t want to talk about; perhaps altogether doesn’t want to discuss its own biases?
If you downvote discussion of karma—like you did—simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?
I don’t do that; I only downvote when it’s combined with incorrect facts. Which I’m tempted to do for this statement: “like you did—simply for mentioning it”, since you’re inferring my motivations, and once again incorrect.
Look, Rain, this is an Internet ongoing discussion. Nobody says everything precisely right. The point is that you would hardly be so severe on someone unless you disagreed strongly. You couldn’t be, because nobody would satisfy your accuracy demands. The kind of nitpicking you engage in your post would ordinarily lead you to be downvoted—and you should be, although I won’t commit the rudeness of so doing in a discussion.
The point wasn’t that you downvote when the only thing wrong with the comment is discussion of karma. It was that you treat discussion of karma as an unconditional wrong. So you exploited weaknesses in my phrasing to ignore what I think was obviously the point—that marking down for the bare mention of karma (even if it doesn’t produce a downvote in each case) is an irrational policy, when karma is at the heart of the discussion. There’s no rational basis for throwing it in as an extra negative when the facts aren’t right.
You’re looking for trivial points to pick to downvote and to ignore the main point, which was your counting mention of karma a negative, without regard to the subject, is an irrational policy. If we were on reversed sides, your nitpicking and evasion would itself be marked down. As matters stand, you don’t even realize you’re acting in a biased fashion, and readers either don’t know or don’t care.
Is that rational? Shouldn’t a rationalist community be more concerned with criticizing irrationalities in its own process?
Having been a subject of both a relatively large upvote and a relatively large downvote in the last couple of weeks, I still think that the worst thing one can do is to complain about censorship or karma. The posts and comments on any forum aren’t judged on their “objective merits” (because there is no such thing), but on its suitability for the forum in question. If you have been downvoted, your post deserves it by definition. You can politely inquire about the reasons, but people are not required to explain themselves. As for rationality, I question whether it is rational to post on a forum if you are not having fun there. Take it easy.
The posts and comments on any forum aren’t judged on their “objective merits” (because there is no such thing), but on its suitability for the forum in question. If you have been downvoted, your post deserves it by definition.
First, you’re correct that it’s irrational to post to a forum you don’t enjoy. I’ll work on decreasing my akrasia.
But it’s hard not to comment on a non sequitur like the above. (Although probably futile because one who’s really not into a persuasion effort won’t do it well.) That posts are properly evaluated by suitability to the forum does not imply that a downvoted post deserves the downvote by definition! That’s a maladaptive view of the sort I’m amazed is so seldom criticized on this forum. Your view precludes (by definition yet) criticism of the evaluators’ biases, which do not advance the forum’s purpose. You would eschew not only absolute merits but also any objective consideration of the forum’s function.
A forum devoted to rationality, to be effective and honest, must assess and address the irrationalities in its own functioning. (This isn’t always “fun.”) To define a post that should be upvoted as one that is upvoted constitutes an enormous obstacle to rational function.
The point is that you would hardly be so severe on someone unless you disagreed strongly.
I disagree; a downvote is not ‘severe’.
The kind of nitpicking you engage in your post would ordinarily lead you to be downvoted
I disagree; meta-discussions often result in many upvotes.
It was that you treat discussion of karma as an unconditional wrong.
I do not, and have stated as much.
There’s no rational basis for throwing it in as an extra negative when the facts aren’t right.
If there is no point in downvoting incorrect facts, then I wonder what the downvote button is for.
You’re looking for trivial points to pick to downvote and to ignore the main point,
I disagree; your main point is that you are being unfairly downvoted, along with other posts critical of SI being downvoted unfairly, which I state again is untrue, afactual, incorrect, a false statement, a lie, a slander, etc.
Is that rational? Shouldn’t a rationalist community be more concerned with criticizing irrationalities in its own process?
Questioning the rationality of meta-meta-voting patterns achieves yet another downvote from me. Sorry.
I don’t follow your reasoning, here. Having read this particular thread, it does seem as though you are, in fact, going out of your way to criticize and downvote srdiamond. Yes, he has, in fact, made a few mistakes. Given, however, that the point of this post in general is about dissenting from the mainstream opinions of the LW crowd, and given the usual complaints about lack of dissent, I find your criticism of srdiamond strange, to say the least. I have, accordingly, upvoted a number of his comments.
As expected, my previous comment was downvoted almost immediately.
This would, for reference, be an example of the reason why some people believe LW is a cult that suppresses dissent. After all, it’s significantly easier to say that you disagree with something than it is to explain in detail why you disagree; just as it’s far easier to state agreement than to provide an insightful statement in agreement. Nonetheless, community norms dictate that unsubstantiated disagreements get modded down, while unsubstantiated agreements get modded up. Naturally, there’s more of the easy disagreement then the hard disagreement… that’s natural, since this is the Internet, and anyone can just post things here.
In any event, though, the end result is the same; people claim to want more dissent, but what they really mean is that they want to see more exceptionally clever and well-reasoned dissent. Any dissent that doesn’t seem at least half as clever as the argument it criticizes seems comparatively superfluous and trivial, and is marginalized at best. And, of course, any dissent that is demonstrably flawed in any way is aggressively attacked. That really is what people mean by suppression of dissent. It doesn’t really mean ‘downvoting arguments which are clever, but with which you personally disagree’… community norms here are a little better then that, and genuinely good arguments tend to get their due. In this case, it means, ‘downvoting arguments which aren’t very good, and with which you personally disagree, when you would at the same time upvote arguments that also aren’t very good, but with which you agree’. Given the nature of the community norms, someone who expresses dissent regularly, but without taking the effort to make each point in an insightful and terribly clever way, would tend to be downvoted repeatedly, and thus discouraged from making more dissent in the future… or, indeed, from posting here at all.
I don’t know if there’s a good solution to the problem. I would be inclined to suggest that, like with Reddit, people not downvote without leaving an explanation as to why. For instance, in addition to upvoting some of srdiamond’s earlier comments, I have also downvoted some of Rain’s, because a number of Rain’s comments in this thread fit the pattern of ‘poor arguments that support the community norms’, in the same sense that srdiamond’s fit the pattern of ‘poor arguments that violate the community norms’; my entire point here is that, in order to cultivate more intelligent dissent, there should be more of the latter and less of the former.
So, in other words, you automatically downvote anyone who explicitly mentions that they realize they are violating community norms by posting whatever it is they are posting, but feels that the content of their post is worth the probable downvotes? That IS fairly explicitly suppressing dissent, and I have downvoted you for doing so.
I don’t think it is suppression of dissent per se. It is more annoying behavior- it implies caring a lot about the karma system, and it is often not even the case when people say that they will actually get downvoted. If it is worth the probable downvote, then they can, you know, just take the downvote. If they want to point out that a view is unpopular they can just say that explicitly. It is also annoying to people like me, who are vocal about a number of issues that could be controversial here (e.g. criticizing Bayesianism, cryonics,, and whether intelligence explosions would be likely) and get voted up. More often than not, when someone claims they are getting downvoted for having unpopular opinions, they are getting downvoted in practice for having bad arguments or for being uncivil.
There are of course exceptions to this rule, and it is disturbing to note that the exceptions seem to be coming more common (see for example, this exchange where two comments are made with about the same quality of argument and about the same degree of uncivility- (“I’m starting to hate that you’ve become a fixture here.” v. “idiot”—but one of the comments is at +10 and the other is at −7.) Even presuming that there’s a real disagreement in quality or correctness of the arguments made, this suggests that uncivil remarks are tolerated more when people agree with the rest of the claim being made. That’s problematic. And this exchange was part of what prompted me to earlier suggest that we should be concerned if AGI risk might be becoming a mindkiller here. But even given that, issues like this seem not at all common.
Overall, if one needs to make a claim about one is going to be downvoted, one might even be correct, but it will often not be for the reasons one thinks it is.
More often than not, when someone claims they are getting downvoted for having unpopular opinions, they are getting downvoted in practice for having bad arguments or for being uncivil.
I don’t think it’s so much ‘caring a lot about the karma system’ per se, so much as the more general case of ‘caring about the approval and/or disapproval of one’s peers’. The former is fairly abstract, but the latter is a fairly deep ancestral motivation.
Like I said before, it’s clearly not much in the way of suppression. That said, given that, barring rare incidents of actual moderation, it is the only ‘suppression’ that occurs here, and since there is a view among various circles that there there is, in fact, suppression of dissent, and since people on the site frequently wonder why there are not more dissenting viewpoints here, and look for ways to find more… it is important to look at the issue in great depth, since it’s clearly an issue which is more significant than it seems on the surface.
[P]eople on the site frequently wonder why there are not more dissenting viewpoints here, and look for ways to find more… it is important to look at the issue in great depth, since it’s clearly an issue which is more significant than it seems on the surface.
Exactly right. But a group that claims to be dedicated to rationality loses all credibility when participants not only abstain from considering this question but adamantly resist it. The only upvote you received for your post—which makes this vital point—is mine.
This thread examines HoldenKarnofsky’s charge that SIAI isn’t exemplarily rational. As part of that examination, the broader LW environment on which it relies is germane. That much has been granted by most posters. But when the conversation reaches the touchstone of how the community expresses its approval and disapproval, the comments are declared illegitimate and downvoted (or if the comments are polite and hyper-correct, at least not upvoted).
The group harbors taboos. The following subjects are subject to them: the very possibility of nonevolved AI; karma and the group’s own process generally (an indespensable discussion ); and politics. (I’ve already posted a cite showing how the proscription on politics works, using an example the editors’ unwillingness to promote the post despite receiving almost 800 comments).
These defects in the rational process of LW help sustain Kardofsky’s argument that SIAI is not to be recommended based on the exemplary rationality of its staff and leadership. They are also the leadership of LW, and they have failed by refusing to lead the forum toward understanding the biases in its own process. They have fostered bias by creating the taboo on politics, as though you can rationally understand the world while dogmatically refusing even to consider a big part of it—because it “kills” your mind.
P.S. Thank you for the upvotes where you perceived bias.
...AGI risk might be becoming a mindkiller here...
Nah. If there is a mindkiller then it is the reputation system. Some of the hostility is the result of the overblown ego and attitude of some of its proponents and their general style of discussion. They created an insurmountable fortress that shields them from any criticism:
Troll: If you are so smart and rational, why don’t you fund yourself? Why isn’t your organisation sustainable?
SI/LW: Rationality is only aimed at expected winning.
Troll: But you don’t seem to be winning yet. Have you considered the possibility that your methods are suboptimal? Have you set yourself any goals, that you expect to be better at than less rational folks, to test your rationality?
SI/LW: Rationality is a caeteris paribus predictor of success.
Troll: Okay, but given that you spend a lot of time on refining your rationality, you must believe that it is worth it somehow? What makes you think so then?
SI/LW: We are trying to create a friendly artificial intelligence implement it and run the AI, at which point, if all goes well, we Win. We believe that rationality is very important to achieve that goal.
Troll: I see. But there surely must be some sub-goals that you anticipate to be able to solve and thereby test if your rationality skills are worth the effort?
SI/LW: Many of the problems related to navigating the Singularity have not yet been stated with mathematical precision, and the need for a precise statement of the problem is part of the problem.
Troll: Has there been any success in formalizing one of the problems that you need to solve?
SI/LW: There are some unpublished results that we have had no time to put into a coherent form yet.
Troll: It seems that there is no way for me to judge if it is worth it to read up on your writings on rationality.
SI/LW: If you want to more reliably achieve life success, I recommend inheriting a billion dollars or, failing that, being born+raised to have an excellent work ethic and low akrasia.
Troll: Awesome, I’ll do that next time. But for now, why would I bet on you or even trust that you know what you are talking about?
SI/LW: We spent a lot of time on debiasing techniques and thought long and hard about the relevant issues.
Troll: That seems to be insufficient evidence given the nature of your claims and that you are asking for money.
SI/LW: We make predictions. We make statements of confidence of events that merely sound startling. You are asking for evidence we couldn’t possibly be expected to be able to provide, even given that we are right.
Troll: But what do you anticipate to see if your ideas are right, is there any possibility to update on evidence?
SI/LW: No, once the evidence is available it will be too late.
Troll: But then why would I trust you instead of those experts who tell me that you are wrong?
SI/LW: You will soon learn that your smart friends and experts are not remotely close to the rationality standards of SI/LW, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don’t.
Troll: But you have never achieved anything when it comes to AI, why would I trust your reasoning on the topic?
SI/LW: That is magical thinking about prestige. Prestige is not a good indicator of quality.
Troll: You won’t convince me without providing further evidence.
SI/LW: That is a fully general counterargument you can use to discount any conclusion.
First, none of this dissent has been suppressed in any real sense. It’s still available to be read and discussed by those who desire reading and discussing such things. The current moderation policy has currently only kicked in when things have gotten largely out of hand—which is not the case here, yet.
Second, net karma isn’t a fine enough tool to express amount of detail you want it to express. The net comment on your previous comment is currently −2; congrats, you’ve managed to irritate less than a tenth of one percent of LW (presuming the real karma is something like −2/+0 or −3/+1)!
Third, the solution you propose hasn’t been implemented anywhere that I know of. Reddit’s suggested community norm (which does not apply to every subreddit) suggests considering posting constructive criticism only when one thinks it will actually help the poster improve. That’s not really the case much of the time, at least on the subreddits I frequent, and it’s certainly not the case often here.
Fourth, the solution you propose would, if implemented, decrease the signal-to-noise ratio of LW further.
Fifth, reddit’s suggested community norm also says “[Don’t c]omplain about downvotes on your posts”. Therefore, I wonder how much you really think reddit is doing the community voting norm thing correctly.
First; downvoted comments are available to be read, yes; but the default settings hide comments with 2 or more net downvotes. This is enough to be reasonably considered ‘suppression’. It’s not all that much suppression, true, but it is suppression… and it is enough to discourage dissent. Actual moderation of comments is a separate issue entirely, and not one which I will address here.
Second; when I posted my reply, and as of this moment, my original comment was at −3. I agree; net karma isn’t actually a huge deal, except that it is, as has been observed, the most prevalent means by which dissent is suppressed. In my case, at least, ‘this will probably get downvoted’ feels like a reason to not post something. Not much of a reason, true, but enough of one that I can identify the feeling of reluctance.
Third; on the subreddits I follow (admittedly a shallow sampling), I have frequently seen comments explaining downvotes, sometimes in response to a request specifically for such feedback, but just as often not. I suspect that this has a lot to do with the “Down-voting? Please leave an explanation in the comments.” message that appears when mousing over the downvote icon. I am aware that this is not universal across Reddit, but on the subreddits I follow, it seems to work reasonably well.
Fourth; I agree that this is a possible result. Like I said before, I’m not sure if there is a good solution to this problem, but I do feel that it’d result in a better state then that which currently exists, if people would more explicitly explain why they downvote when they choose to do so. That said, given that downvoted comments are hidden from default view anyway, and that those who choose to do so can easily ignore such comments, I don’t think it’d have all that much effect on the signal/noise ratio.
Fifth; on the subreddits I follow, it seems as though there is less in the way of complaints about downvotes, and more honest inquiries as to why a comment has been downvoted; such questions seem to usually receive honest responses. This may be anomalous within Reddit as a whole; as I said before, my own experience with Reddit is a shallow sampling.
I don’t know if there’s a good solution to the problem. I would be inclined to suggest that, like with Reddit, people not downvote without leaving an explanation as to why. For instance, in addition to upvoting some of srdiamond’s earlier comments, I have also downvoted some of Rain’s, because a number of Rain’s comments in this thread fit the pattern of ‘poor arguments that support the community norms’, in the same sense that srdiamond’s fit the pattern of ‘poor arguments that violate the community norms’; my entire point here is that, in order to cultivate more intelligent dissent, there should be more of the latter and less of the former.
Perhaps the solution is not to worry so much about my bad contrarian arguments being downvoted as to assure that bad “establishment” arguments are downvoted—as in Rain’s case, they aren’t. Regurgitation of arguments others have repeatedly stated should also be downvoted, no matter how good the arguments.
The reason to think an emphasis on more criticism of Rain rather than less criticism of me is that after I err, it’s a difficult argument to establish that my error wasn’t serious enough to avoid downvote. But when Rain negligently or intentionally misses the entire point, there’s less question that he isn’t benefiting the discussion. It’s easier to convict of fallacy than to defend based on the fallacy being relatively trivial. There’s a problem in that the two determinations are somewhat inter-related, but it doesn’t eliminate the contrast.
Increasing the number of downvotes would deflate the significance of any single downvote and would probably foster more dissent. This balance may be subject to easy institutional control. Posters are allotted downvotes based on their karma, while the karma requirements for upvotes are easily satisfied, if they exist. This amounts to encouraging upvotes relative to downvotes, with the result that many bad posts are voted up and some decent posts suffer the disproportionate wrath of extreme partisans. (Note that Rain, a donor, is a partisan of SIAI.)
The editors should experiment with increasing the downvote allowance. I favor equal availability of downvotes and upvotes as optimal (but this should be thought through more carefully).
Consider what would have happened had Yudkowsky not shown exceptional receptivity to this post: he would have blatantly proven his critics right.
After turning this statement around in my head for a while I’m less certain than I was that I understand its thrust. But assuming you mean those critics pertinent to lukeprog’s post, i.e. those claiming LW embodies a cult of personality centered around Eliezer—well, no. Eliezer’s reaction is in fact almost completely orthogonal to that question.
If you receive informed criticism regarding a project you’re heavily involved in, and you react angrily to it, that shows nothing more or less than that you handle criticism poorly. If the community around you locks ranks against your critics, either following your example or (especially) preemptively, then you have evidence of a cult of personality.
That’s not what happened here, though. Eliezer was fairly gracious, as was the rest of the community. Now, that is not by itself behavior typically associated with personality cults, but before we start patting ourselves on the back it’s worth remembering that certain details of timing and form could still point back in the other direction. I’m pretty sick of the cult question myself, but if you’re bound and determined to apply this exchange to it, that’s the place you should be looking.
I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.
After I recently read that the lead poster was a major financial contributor to SIAI, I’d have to call LukeProg’s argument disingenuous if not mendacious.
Rain (who noted that he is a donor to SIAI in a comment) and HoldenKarnofsky (who wrote the post) are two different people, as indicated by their different usernames.
Well, different usernames isn’t usually sufficient evidence that there are two different people, but in this case there’s little doubt about their separability.
I don’t understand. Holden is not a major financial contributor to SIAI. And even if he was: which argument are you talking about, and why is it disingenuous?
If Holden were a major contributor, your argument that the LW editors demonstrated their tolerance for dissent by encouraging the criticisms he made would be bogus. Suppressing the comments of a major donor would be suicidal, and claiming not doing so demonstrates any motive but avoiding suicide would be disingenuous at the least.
If he’s not a donor, my apologies. In any event, you obviously don’t know that he’s a donor if he is, so my conclusion is wrong. I thought Yudkowsky said he was.
I’m confused. Holden doesn’t believe SI is a good organisation to recommend giving money to, he’s listed all those objections to SI in his post, and you somehow assumed he’s been donating money to it?
This post is highly critical of SIAI — both of its philosophy and its organizational choices. It is also now the #1 most highly voted post in the entire history of LessWrong — higher than any posts by Eliezer or myself.
I shall now laugh harder than ever when people try to say with a straight face that LessWrong is an Eliezer-cult that suppresses dissent.
Either I promoted this and then forgot I’d done so, or someone else promoted it—of course I was planning to promote it, but I thought I’d planned to do so on Tuesday after the SIAIers currently running a Minicamp had a chance to respond, since I expected most RSS subscribers to the Promoted feed to read comments only once (this is the same reason I wait a while before promoting e.g. monthly quotes posts). On the other hand, I certainly did upvote it the moment I saw it.
Original comment now edited; I wasn’t aware anyone besides you might be promoting posts.
I agree (as a comparative outsider) that the polite response to Holden is excellent. Many (most?) communities—both online communities and real-world organisations, especially long-standing ones—are not good at it for lots of reasons, and I think the measured response of evaluating and promoting Holden’s post is exactly what LessWrong members would hope LessWrong could do, and they showed it succeeded.
I agree that this is good evidence that LessWrong isn’t just an Eliezer-cult. (The true test would be if Elizier and another long-standing poster were dismissive to the post, and then other people persuaded them otherwise. In fact, maybe people should roleplay that or something, just to avoid getting stuck in an argument-from-authority trap, but that’s a silly idea. Either way, the fact that other people spoke positively, and Elizier and other long-standing posters did too, is a good thing.)
However, I’m not sure it’s as uniquely a victory for the rationality of LessWrong as it sounds. In responose to srdiamond, Luke quoted tenlier saying “[Holden’s] critique mostly consists of points that are pretty persistently bubbling beneath the surface around here, and get brought up quite a bit. Don’t most people regard this as a great summary of their current views, rather than persuasive in any way?” To me, that suggests that Holden did a really excellent job expressing these views clearly and persuasively. However, it suggests that previous people had tried to express something similar, but it hadn’t been expressed well enough to be widely accepted, and people reading had failed to sufficiently apply the dictum of “fix your opponents’ arguments for them”. I’m not sure if that’s true (it’s certainly not automatically true), but I suspect it might be. What do people think?
If there’s any truth to it, it suggests one good answer to the recent post http://lesswrong.com/lw/btc/how_can_we_get_more_and_better_lw_contrarians (whether that was desirable in general or not) would be, as a rationalist exercise for someone familiar with/to the community and good at writing rationally, to take a survey of contrarian views on the topic that people on the community may have had but not been able to express, and don’t worry about showmanship like pretending to believe it yourself, but just say “I think what some people think is [well-expressed argument]. Do you agree that’s fair? If so, do I and other people think they have a point?” Whether or not that argument is right it’s still good to engage with it if many people are thinking it.
Third highest now. Eliezer just barely gets into the top 20.
1st.
At this point even I am starting to be confused.
Can you articulate the nature of your confusion?
I suppose it’s that I naively expect, when opening the list of top LW posts ever, to see ones containing the most impressive or clever insights into rationality.
Not that I don’t think Holden’s post deserves a high score for other reasons. While I am not terribly impressed with his AI-related arguments, the post is of the very highest standards of conduct, of how to have a disagreement that is polite and far beyond what is usually named “constructive”.
(nods) Makes sense.
My own primary inference from the popularity of this post is that there’s a lot of uncertainty/disagreement within the community about the idea that creating an AGI without an explicit (and properly tuned) moral structure constitutes significant existential risk, but that the social dynamics of the community cause most of that uncertainty/disagreement to go unvoiced most of the time.
Of course, there’s lots of other stuff going on as well that has little to do with AGI or existential risk, and a lot to do with the social dynamics of the community itself.
Maybe. I upvoted it because it will have (and has had) the effect of improving SI’s chances.
Some people who upvoted the post may think it is one of the best-written and most important examples of instrumental rationality on this site.
I wish I could upvote this ten times.
Well perhaps the normal practice is cult-like and dissent-suppressing and this is an atypical break. Kind of like the fat person who starts eating salad instead of nachos while he watches football. And congratulates himself on his healthy eating even though he is still having donuts for breakfast and hamburgers and french fries for lunch.
Seems to me the test for suppression of dissent is not when a high-status person criticizes. The real test is when someone with medium or low status speaks out.
And my impression is that lesswrong does have problems along these lines. Not as bad as other discussion groups, but still.
Looks like 3rd now. As impressed as I am with the post, at this point I’m a little surprised.
But LW isn’t reflective of SI, most of the people that voted on this article have no affiliation with SI. So the high number of upvotes is less reflective of SI welcoming criticism than LW being dissatisfied with the organization of SI.
Furthermore, this post’s criticism of Eliezer’s research less strong than its criticism of SI’s organization . SI has always been somewhat open to criticism of its organizational structure and many of the current leadership of SI has criticized the organizational structure at some point. But who criticize Eliezer’s research do not manage to rise in SI’s research division and generally aren’t well received even on LW (Roko).
Lastly, laughing at somebody when they call your organization a cult is not a convincing argument, they’re more likely to think of your organization as a cult (at least they will think you are arrogant).
How’s about you also have a critical discussion of ‘where can be we wrong and how do we make sure we are actually competent’ and ‘can we figure out what the AI will actually do, using our tools?’ instead of ‘how do we communicate our awesomeness better’ and ‘are we communicating our awesomeness right’ ?
This post is something that can’t be suppressed without losing big time, and you not suppressing it is only a strong evidence that you are not completely stupid (which is great).
Holden does not disagree with most of the basic beliefs that SI endorses. Which I think is rather sad and why I don’t view him as a real critic. And he has been very polite.
Here is the impolite version:
If an actual AI researcher would have written a similar post, someone who actually tried to build practical systems and had some economic success, not one of those AGI dreamers. If such a person would write a similar post and actually write in a way that they feel, rather than being incredible polite, things would look very different.
The trust is that you are incredible naive when it comes to technological progress. That recursive self-improvement is nothing more than a row of English words, a barely convincing fantasy. That expected utility maximization is practically unworkable, even for a superhuman intelligence. And that the lesswrong.com sequences are not original or important but merely succeed at drowning out all the craziness they include by a huge amount of unrelated clutter and an appeal to the rationality of the author.
What you call an “informed” critic is someone who shares most of your incredible crazy and completely unfounded beliefs.
Worst of all, you are completely unconvincing and do not even notice it because there are so many other people who are strongly and emotionally attached to the particular science fiction scenarios that you envision.
I’m assuming you think they’d come in, scoff at our arrogance for a few pages, and then waltz off. Disregarding how many employed machine learning engineers also do side work on general intelligence projects, you’d probably get the same response from automobile engineer, someone with a track record and field expertise, talking to the Wright Brothers. Thinking about new things and new ideas doesn’t automatically make you wrong.
Really? Because that’s a pretty strong claim. If I knew how the human brain worked well enough to build one in software, I could certainly build something smarter. You could increase the number of slots in working memory. Tweak the part of the brain that handles intuitive math to correctly deal with orders of magnitude. Improve recall to eidetic levels. Tweak the brain’s handling of probabilities to be closer to the Bayesian ideal. Even those small changes would likely produce a mind smarter than any human being who has ever lived. That, plus the potential for exponential subjective speedup, is already dangerous. And that’s assuming that the mind that results would see zero new insights that I’ve missed, which is pretty unlikely. Even if the curve bottoms out fairly quickly, after only a generation or two that’s STILL really dangerous.
Really makes you wonder how all those people got convinced in the first place.
This is totally unsupported. To quote Lady Catherine de Bourgh, “If I had ever learned [to play the piano], I should have become a great proficient.”
You have no idea whether the “small changes” you propose are technically feasible, or whether these “tweaks” would in fact mean a complete redesign. For all we know, if you knew how the human brain worked well enough to build one in software, you would appreciate why these changes are impossible without destroying the rest of the system’s functionality.
After all, it would appear that (say) eidetic recall would provide a fitness advantage. Given that humans lack it, there may well be good reasons why.
“totally unsupported” seems extreme. (Though I enjoyed the P&P shoutout. I was recently in a stage adaptation of the book, so it is pleasantly primed.)
What the claim amounts to is the belief that:
a) there exist good design ideas for brains that human evolution didn’t implement, and
b) a human capable of building a working brain at all is capable of coming up with some of them.
A seems pretty likely to me… at least, the alternative (our currently evolved brains are the best possible design) seems so implausible as to scarcely be worth considering.
B is harder to say anything clear about, but given our experience with other evolved systems, it doesn’t strike me as absurd. We’re pretty good at improving the stuff we were born with.
Of course, you’re right that this is evidence and not proof. It’s possible that we just can’t do any better than human brains for thinking, just like it was possible (but turned out not to be true) that we couldn’t do any better than human legs for covering long distances efficiently.
But it’s not negligible evidence.
I don’t doubt that it’s possible to come up with something that thinks better than the human brain, just as we have come up with something that travels better than the human leg. But to cover long distances efficiently, people didn’t start by replicating a human leg, and then tweaking it. They came up with a radically different design—e.g. the wheel.
I don’t see the evidence that knowing how to build a human brain is the key step in knowing how to build something better. For instance, suppose you could replicate neuron function in software, and then scan a brain map (Robin Hanson’s “em” concept). That wouldn’t allow you to make any of the improvements to memory, maths, etc, that Dolores suggests. Perhaps you could make it run faster—although depending on hardware constraints, it might run slower. If you wanted to build something better, you might need to start from scratch. Or, things could go the other way—we might be able to build “minds” far better than the human brain, yet never be able to replicate a human one.
But it’s not just that evidence is lacking—Dolores is claiming certainty in the lack of evidence. I really do think the Austen quote was appropriate.
To clarify, I did not mean having the data to build a neuron-by-neuron model of the brain. I meant actually understanding the underlying algorithms those slabs of neural tissue are implementing. Think less understanding the exact structure of a bird’s wing, and more understanding the concept of lift.
I think, with that level of understanding, the odds that a smart engineer (even if it’s not me) couldn’t find something to improve seem low.
I agree that I might not need to be able to build a human brain in software to be able to build something better, as with cars and legs.
And I agree that I might be able to build a brain in software without understanding how to do it, e.g., by copying an existing one as with ems.
That said, if I understand the principles underlying a brain well enough to build one in software (rather than just copying it), it still seems reasonable to believe that I can also build something better.
I agree that that the tone on both sides is intentionally respectful, and that people here delude themselves if they imagine they aren’t up for a bit of mockery from high status folks who don’t have the patience to be really engage.
I agree that we don’t really know what to expect from the first program that can meaningfully improve itself (including, I suppose, its self-improvement procedure) at a faster pace than human experts working on improving it. It might not be that impressive. But it seems likely to me that it will be a big deal, if ever we get there.
But you’re being vague otherwise. Name a crazy or unfounded belief.
Holden asked me something similar today via mail. Here is what I replied:
You wrote in ‘Other objections to SI’s views’:
It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI’s from mind design space at random.
Further, the whole case for AI risk is based on the idea that there will be a huge jump in capability at some point. Which I think is at best good science fiction, like faster-than-light propulsion, or antimatter weapons (when in doubt that it is possible in principle).
The basic fact that an AGI will most likely need something like advanced nanotechnology to pose a risk, which is itself an existential risk, hints at a conjunction fallacy. We do not need AGI to then use nanotechnology to wipe us out, nanotechnology is already enough if it is possible at all.
Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.
I have written thousands of words about all this and never got any convincing reply. So if you have any specific arguments, let me know.
They say what what I write is unconvincing. But given the amount of vagueness they use to protect their beliefs, my specific criticisms basically amount to a reductio ad absurdum. I don’t even need to criticize them, they would have to support their extraordinary beliefs first or make them more specific. Yet I am able to come up with a lot of arguments that speak against the possibility they envision, without any effort and no knowledge of the relevant fields like complexity theory.
Here is a comment I received lately:
If you were to turn IBM Watson gradually into a seed AI, at which point would it become an existential risk and why? They can’t answer that at all. It is pure fantasy.
END OF EMAIL
For more see the following posts:
Is an Intelligence Explosion a Disjunctive or Conjunctive Event?
Risks from AI and Charitable Giving
Why I am skeptical of risks from AI
Implicit constraints of practical goals (including the follow-up comments that I posted.)
Some old posts:
Should I believe what the SIAI claims?
What I would like the SIAI to publish
SIAI’s Short-Term Research Program
See also:
Interview series on risks from AI
We are SIAI. Argument is futile.
If you believe I don’t understand the basics, see:
A Primer On Risks From AI
Also:
Open Problems in Ethics and Rationality
Objections to Coherent Extrapolated Volition
There is a lot more, especially in the form of comments where I talk about specifics.
I don’t have the energy to get into an extended debate, but the claim that this is “the basic idea” or that this would be “the strongest argument” is completely false. A far stronger basic idea is the simple fact that nobody has yet figured out a theory of ethics that would work properly, which means that even that AGIs that were specifically designed to be ethical are most likely to lead to bad outcomes. And that’s presuming that we even knew how to program them exactly.
This isn’t even something that you’d need to read a hundred blog posts for, it’s well discussed in both The Singularity and Machine Ethics and Artificial Intelligence as a Positive and Negative Factor in Global Risk. Complex Value Systems are Required to Realize Valuable Futures, too.
I did skim through the last paper. I am going to review it thoroughly at some point.
On first sight one of the problems is the whole assumption of AI drives. On the one hand you claim that an AI is going to follow its code, is its code (as if anyone would doubt causality). On the other hand you talk about the emergence of drives like unbounded self-protection. And if someone says that unbounded self-protection does not need to be part of an AGI, you simply claim that your definition of AGI will have those drives. Which allows you to arrive at your desired conclusion of AGI being an existential risk.
Another problem is the idea that an AGI will be a goal executor (I can’t help but interpret that to be your position) when I believe that the very nature of artificial general intelligence implies the correct interpretation of “Understand What I Mean” and that “Do What I Mean” is the outcome of virtually any research. Only if you were to pull an AGI at random from mind design space could you possible arrive at “Understand What I Mean” without “Do What I Mean”.
To see why look at any software product or complex machine. Those products are continuously improved. Where “improved” means that they become better at “Understand What I Mean” and “Do What I Mean”.
There is no good reason to believe that at some point that development will suddenly turn into “Understand What I Mean” and “Go Batshit Crazy And Do What I Do Not Mean”.
There are other problems with the paper. I hope I will find some time to write a review soon.
One problem for me with reviewing such papers is that I doubt a lot of underlying assumptions like that there exists a single principle of general intelligence. As I see it there will never be any sudden jump in capability. I also think that intelligence and complex goals are fundamentally interwoven. An AGI will have to be hardcoded, or learn, to care about a manifold of things. No simple algorithm, given limited computational resources, will give rise to the drives that are necessary to undergo strong self-improvement (if that is possible at all).
Not saying I particularly disagree with your other premises, but saying something can’t be true because it sounds implausible is not a valid argument.
An AI’s mind doesn’t have to be pulled from design space at random to be disastrous. The primary issue that the SIAI has to grapple with (based on my understanding,) is that deliberately designing an AI that does what we would want it to do, rather than fulfilling proxy criteria in ways that we would not like at all, is really difficult. Even getting one to recognize “humans” as a category in a way that would be acceptable to us is a major challenge.
Although it’s worth pointing out that this is also an obstacle to AGI, since presumably an AI that did not understand what a human was would be pretty unintelligent. So I think it’s unfair to claim this as a “friendliness” issue.
Note that I do think there are some important friendliness-related problems, but, assuming I understand your objection, this is not one of them.
An AI could be an extremely powerful optimizer without having a category for “humans” that mapped to our own. “Human,” the way we conceive of it, is a leaky surface generalization.
A strong paperclip maximizer would understand humans as well as it had to to contend with us in its attempts to paperclip the universe, but it wouldn’t care about us. And a strong optimizer programmed to maximize the values of “humans” would also probably understand us, but if we don’t program into its values an actual category that maps to our conception of humans, it could perfectly well end up applying that understanding to, for example, tiling the universe with crash test dummies.
How do you intend to build a powerful optimizer without having a method of representing (or of building a representation of) the concept of “human” (where “human” can be replaced with any complex concept, even probably paperclips)?
I agree that value specification is a hard problem. But I don’t think the complexity of “human” is the reason for this, although it does rule out certain simple approaches like hard-coding values.
(Also, since your link seems to indicate you believe otherwise, I am fairly familiar with the content in the sequences. Apologies if this statement represents an improper inference.)
If a machine can learn, empirically, exactly what humans are, on the most fundamental levels, but doesn’t have any values associated with them, why should it need a concept of “human?” We don’t have a category that distinguishes igneous rocks that are circular and flat on one side, but we can still recognize them and describe them precisely.
Humans are an unnatural category. Whether a fetus, an individual in a persistent vegetative state, an amputee, a corpse, an em or a skin cell culture fall into the category of “human” depends on value-sensitive boundaries. It’s not necessarily because humans are so complex that we can’t categorize them in an appropriate manner for an AI (or at least, not just because humans are complex,) it’s because we don’t have an appropriate formulation of the values that would allow a computer to draw the boundaries of the category in a way we’d want it to.
(I wasn’t sure how familiar you were with the sequences, but in any case I figured it can’t hurt to add links for anyone who might be following along who’s not familiar.)
I’ve read most of that now, and have subscribed to your newsletter.
Reasonable people can disagree in estimating the difficulty of AI and the visibility/pace of AI progress (is it like hunting for a single breakthrough and then FOOM? etc).
I find all of your “it feels ridiculous” arguments by analogy to existing things interesting but unpersuasive.
Says the wooly mammoth, circa 100,000 BC.
Sounding silly and low status and science-fictiony doesn’t actually make it unlikely to happen in the real world.
Especially when not many people want to read a science fiction book where humanity gets quickly and completely wiped out by a superior force. Even works where humans slowly die off due to their own problems (e.g. On the Beach) are uncommon.
Do you acknowledge that :
We will some day make an AI that is at least as smart as humans?
Humans do try to improve their intelligence (rationality/memory training being a weak example, cyborg research being a better example, and im pretty sure we will soon design physical augmentations to improve our intelligence)
If you acknowledge 1 and 2, then that implies there can (and probably will) be an AI that tries to improve itself
I think you missed the “quickly and vastly” part as well as the “and then invent all kinds of technological magic to wipe us out”. Note I still think XiXiDu is wrong to be as confident as he is (assuming “there will never” implies >90% certainty), but if you are going to engage with him then you should engage with his actual arguments.
Name three examples? (Of ‘craziness’ specifically… I agree that there are frequent, and probably unecessary, “appeals to the rationality of the author”.)
XiXiDu may be too modest; he has some great examples on his blog.
One wonders when or if XiXiDu will ever get over the Roko incident. Yes, it was a weird and possibly disproportionate response, but it was also years ago.
Do we have any evidence that Eliezer’s attitude or approach to that sort of thing has changed since then?
Sure. His moderation activities over the last year or so have been far more… sunglasses… moderate.
So said Newt Gingrich.
Why yes, I do also believe that political figures are held to ridiculous conversational standards as well. It’s a miracle they deign to talk to anyone.
So, Swimmer 963, are those quotes crazy enough for you? (I hope you don’t ask a question and neglect to comment on the answer.) What you do think? Anomalous?
Contrary to the impression the comments might convey, the majority don’t come from the Roko incident. But as to that incident, the passage of time doesn’t necessarily erase the marks of character. Romney is rightfully being held, feet to fire, for a group battering of another student while they attended high school—because such sadism is a trait of character and can’t be explained otherwise. How would one explain Yudkowsky’s paranoia, lack of perspective, and scapegoating—other than by positing a narcissistic personality structure?
Many LWers can’t draw conclusions because they eschew the only tools for that purpose: psychology and excellent fiction. And the second is more important than the first.
I had in fact read a lot of those quotes before–although some of them come as a surprise, so thank you for the link. They do show paranoia and lack of perspective, and yeah, some signs of narcissism, and I would be certainly mortified if I personally ever made comments like that in public…
The Sequences as a whole do come across as having been written by an arrogant person, and that’s kind of irritating, and I have to consciously override my irritation in order to enjoy the parts that I find useful, which is quite a lot. It’s a simplification to say that the Sequences are just clutter, and it’s extreme to call them ‘craziness’, too.
(Since meeting Eliezer in person, it’s actually hard for me to believe that those comments were written by the same person, who was being serious about them… My chief interaction with him was playing a game in which I tried to make a list of my values, and he hit me with a banana every time I got writer’s block because I was trying to be too specific, and made the Super Mario Brothers’ theme song when I succeeded. It’s hard making the connection that “this is the same person who seems to take himself way too seriously in his blog comments.” But that’s unrelated and doesn’t prove anything in either direction.)
My main point is that criticizing someone who believes in a particular concept doesn’t irrefutably damn that concept. You can use it as weak evidence, but not proof. Eliezer, as far as I know, isn’t the only person who has thought extensively about Friendly AI and found it a useful concept to keep.
The quotes aren’t all about AI. A few:
Yudkowsky makes the megalomanic claim that he’s solved the questions of metaethics. His solution: morality is the function that the brain of a fully informed subject computes to determine what’s right. Laughable; pathologically arrogant.
The most extreme presumptuousness about morality; insufferable moralism. Morality, as you were perhaps on the cusp of recognizing in one of your posts, Swimmer963, is a personalized tool, not a cosmic command line. See my “Why do what you “ought”?—A habit theory of explicit morality.”
The preceding remark, I’ll grant, isn’t exactly crazy—just super obnoxious and creepy.
This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination. That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language—an insight that should be almost intuitive to a rationalist. But noncrazy people may believe the foregoing; what they don’t believe is that they can at the present time replace the institution of science with the reasoning of smart people. That’s the absolutely bonkers claim Yudkowsky makes.
>
I didn’t say they were. I said that just because the speaker for a particular idea comes across as crazy doesn’t mean the idea itself is crazy. That applies whether all of Eliezer’s “crazy statements” are about AI, or whether none of them are.
Funny, I actually agree with the top phrase. It’s written in an unfortunately preachy, minister-scaring-the-congregation-by-saying-they’ll-go-to-Hell style, which is guaranteed to make just about anyone get defensive and/or go “ick!” But if you accept the (very common) moral standard that if you can save a life, it’s better to do it than not to do it, then the logic is inevitable that if you have the choice of saving one lives or two lives, by your own metric it’s morally preferable to save two lives. If you don’t accept the moral standard that it’s better to save one life than zero lives, then that phrase should be just as insufferable.
I decided to be charitable, and went and looked up the post that this was in: it’s here. As far as I can tell, Eliezer doesn’t say anything that could be interpreted as “science exists because people are stupid, and I’m not stupid, therefore I don’t need science”. He claims that scientific procedures compensates for people being unwilling to let go of their pet theories and change their minds, and although I have no idea if this goal was in the minds of the people who came up with the scientific method, it doesn’t seem to be false that it accomplishes this goal.
Newton definitely wrote down his version of scientific method to explain why people shouldn’t take his law of gravity and just add, “because of Aristotelian causes,” or “because of Cartesian mechanisms.”
Ok. I disagree with a large bit of the sequences on science and the nature of science. I’ve wrote a fair number of comments saying so. So I hope you will listen when I say that you are taking a strawman version of what Eliezer wrote on these issues, and it almost borders on something that I could only see someone thinking if they were trying to interpret Eliezer’s words in the most negative fashion possible.
You either didn’t read that sequence carefully, or are intentionally misrepresenting it.
Didn’t read that sequence carefully either.
You didn’t read that sequence at all, and probably don’t actually know what simplicity means in an information-theoretic sense.
To be fair, that sequence doesn’t really answer questions about choice-of-language; it took reading some of Solomonoff’s papers for me to figure out what the solution to that problem is.
There are a variety of proposed solutions. None of them seem perfect.
I’m referring to encoding in several different languages, which makes it progressively more implausible that choice of language matters.
I agree that’s not a perfect solution, but it’s good enough for me.
That’s true; I admit I didn’t read the sequence. I had a hard time struggling through the single summating essay. What I wrote was his conclusion. As Hanson wrote in the first comment to the essay I did read, Yudkowsky really should summarize the whole business in a few lines. Yudkowsky didn’t get around to that, as far as I know.
The summation essay contained more than 7,000 words for the conclusion I quoted. Maybe the rest of the series contradicts what is patent in the essay I read.
I simply don’t get the attraction of the sequences. An extraordinarily high ratio of filler to content; Yudkowsky seems to think that every thought along the way to his personal enlightenment is worth the public’s time.
Asking that a critic read those sequences in their entirety is asking for a huge sacrifice; little is offered to show it’s even close in being worth the misery of reading inept writing or the time.
You know, the sequences aren’t actually poorly written. I’ve read them all, as have most of the people here. They are a bit rambly in places, but they’re entertaining and interesting. If you’re having trouble with them, the problem might be on your end.
In any case, if you had read them, you’d know, for instance, that when Yudkowsky talks about simplicity, he is not talking about the simplicity of a given English sentence. He’s talking about the combined complexity of a given Turing machine and the program needed to describe your hypothesis on that Turing machine.
http://lesswrong.com/lw/8p4/2011_survey_results/
23% for ‘almost all’
39% have read > three-quarters
54% have read > half
My mistake. I’ll remember that in the future.
In addition, there are places in the Sequences where Eliezer just states things as though he’s dispensing wisdom from on high, without bothering to state any evidence or reasoning. His writing is still entertaining, of course, but still less than persuasive.
I also found this to be true.
I’m pretty sure the 2011 survey puts this claim to the test, but I don’t have the time to look it up.
The problem is partly on my end, for sure; obviously, I find rambling intolerable in Internet writing, and I find it in great abundance in the sequences. You’re more tolerant of rambling, and you’re entertained by Yudkowsky’s. I also think he demonstrates mediocre literary skills when it comes to performances like varying his sentence structure. I don’t know what you think of that. My guess is you don’t much care; maybe it’s a generational thing.
I’m intrigued by what enjoyment readers here get from Yudkowsky’s sequences. Why do you all find interesting what I find amateurish and inept? Do we have vastly different tastes or standards, or both? Maybe it is the very prolixity that makes the writing appealing in founding a movement with religious overtones. Reading Yudkowsky is an experience comparable to reading the Bible.
As a side issue, I’m dismayed upon finding that ideas I had thought original to Yudkowsky were secondhand.
Of course I understand simplicity doesn’t pertain to simplicity in English! (Or in any natural language.) I don’t think you understand the language-relativity issue.
If you were willing to point me to two or three of your favorite Internet writers, whom you consider reliably enjoyable and interesting and so forth, I might find that valuable for its own sake, and might also be better able to answer your question in mutually intelligible terms.
Having to have original ideas is a very high standard. I doubt a single one of my posts contains a truly original idea, and I don’t try–I try to figure out which ideas are useful to me, and then present why, in a format that I hope will be useful to others. Eliezer creates a lot of new catchy terms for pre-existing ideas, like “affective death spiral” for “halo effect.” I like that.
His posts are also quite short, often witty, and generally presented in an easier-to-digest format than the journal articles I might otherwise have to read to encounter the not-new ideas. You apparently don’t find his writing easy to digest or amusing in the same way I do.
Affective death spiral is not the same thing as the Halo effect, though the halo effect (/ horns effect) might be part of the mechanism of affective death spiral.
Agreed… I think the Halo effect is a sub-component of an affective death spiral, and “affective death spiral” is a term unique to LW [correct me if I’m wrong!], while ‘Halo effect’ isn’t.
Are there specific examples? It seems to me that in most cases when he has a pre-existing idea he gives relevant sources.
I don’t know any specific examples of secondhand ideas coming off as original (indeed, he often cites experiments from the H&B literature), but there’s another possible source for the confusion. Sometimes Yudkowsky and somebody else come up with ideas independently, and those aren’t cited because Yudkowsky didn’t know they existed at the time. Drescher and Quine are two philosophers who have been mentioned as having some of the same ideas as Yudkowsky, and I can confirm the former from experience.
I find his fictional interludes quite entertaining, because they are generally quite lively, and display a decent amount of world-building—which is one aspect of science fiction and fantasy that I particularly enjoy. I also enjoy the snark he employs when trashing opposing ideas, especially when such ideas are quite absurd. Of course, the snark doesn’t make his writing more persuasive—just more entertaining.
I know I’m exposing my ignorance here, but I’m not sure what this means; can you elaborate ?
Indeed, the sequences are long. I’m not sure about the others here, but I’ve never asked anybody to “read the sequences.”
But I don’t even know how to describe the arrogance required to believe that you can dismiss somebody’s work as “crazy,” “stupid,” “megalomanic,” “laughably, pathologically arrogant,” “bonkers,” and “insufferable” without having even read enough of what you’re criticizing the get an accurate understanding of it.
ETA: Edited in response to fubarobfusco, who brought up a good point.
That’s a fully general argument against criticizing anything without having read all of it, though. And there are some things you can fairly dismiss without having read all of. For instance, I don’t have to read every page on the Time Cube site to dismiss it as crazy, stupid, pathologically arrogant, and so on.
The reason EY wrote an entire sequence on metaethics is precisely because without the rest of the preparation people such as you who lack all that context immediately veer off course and start believing that he’s asserting the existence (or non-existence) of “objective” morality, or that morality is about humans because humans are best or any other standard philosophical confusion that people automatically come up with whenever they think about ethics.
Of course this is merely a communication issue. I’d love to see a more skilled writer present EY’s metaethical theory in a shorter form that still correctly conveys the idea, but it seems to be very difficult (especially since even half the people who do read the sequence still come away thinking it’s moral relativism or something).
I read your post on habit theory, and I liked it, but I don’t think it’s an answer to the question “What should I do?”
It’s interesting to say that if you’re an artist, you might get more practical use out of virtue theory, and if you’re a politician, you might get more practical use out of consequentialism. I’m not sure who it is that faces more daily temptations to break the rules than the rest of us; bankers, I suppose, and maybe certain kinds of computer security experts.
Anyway, saying that morality is a tool doesn’t get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer’s answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you ‘should’ do, and that’s all there is to it. This is not exactly a brilliant answer, but it may nevertheless be the best one out there. I’ve never yet heard a moral theory that made more sense than that, and believe me, I’ve looked.
It’s reasonable to insist that people put their conclusions in easily digestible bullet points to convince you to read the rest of what they’ve written...but if, noting that there are no such bullet points, you make the decision not to read the body text—you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he’s not stupid, and he’s not crazy. If you want to offer a meaningful critique of his ideas, you’ll have to read them first.
This is sound general advice, but at least one observation makes this situation exceptional: Yudkowsky’s conclusions about ethics are never summarized in terms that contradict my take. I don’t think your rendition, for example, contradicts mine. I’m certainly not surprised to hear his position described the way you describe it:
Now, I don’t think the decision of whether to be an artist, politician, or banker is a moral decision. It isn’t one you make primarily because of what’s ethically right or wrong. To the extent you do (and in the restricted sense that you do), your prior moral habits are your only guide.
But we’re looking at whether Yudkowsky’s position is intellectually respectable, not whether objective morality—which he’s committed to but I deny—exists. To say we look at what our brain says when we’re fully informed says essentially that we seek a reflective equilibrium in solving moral problems. So far so good. But it goes further in saying brains compute some specific function that determines generally when individuals reach that equilibrium. Leaving aside that this is implausible speculation, requiring that the terms of moral judgments be hardwired—and hardwired identically for each individual—it also simply fails to answer Moore’s open question, although Yudkowsky claims he has that answer. There’s nothing prima facie compelling ethically about what our brains happen to tell us is moral; no reason we should necessarily follow our brains’ hardwiring. I could consistently choose to consider my brain’s hardwired moralisms maladaptive or even despicable holdovers from the evolutionary past that I choose to override as much as I can.
Robin Hanson actually asked the right question. If what the brain computes is moral, what does it correspond to that makes it moral? Unless you think the brain is computing a fact about the world, you can’t coherently regard its computation as “accurate.” But if not, what makes it special and not just a reflex?
I do feel a bit guilty about criticizing Yudkowsky without reading all of him. But he seems to express his ideas at excessive and obfuscating length, and if there were more to them, I feel somewhat confident I’d come across his answers. It isn’t as though I haven’t skimmed many of these essays. And his answers would certainly deserve some reflection in his summation essay.
There’s no question Yudkowsky is no idiot. But he has some ideas that I think are stupid—like his “metaethics”—and he expresses them in a somewhat “crazy” manner, exuding grandiose self-confidence. Being surrounded and discussing mostly with people who agree with him is probably part of the cause.
As someone who has read Eliezer’s metaethics sequence, let me say that what you think his position is, is only somewhat related to what it actually is; and also, that he has answered those of your objections that are relevant.
It’s fine that you don’t want to read 30+ fairly long blog posts, especially if you dislike the writing style. But then, don’t try to criticize what you’re ignorant about. And no, openly admitting that you haven’t read the arguments you’re criticizing, and claiming that you feel guilty about it, doesn’t magically make it more acceptable. Or honest.
One doesn’t need to have read the whole Bible to criticize it. But the Bible is a fairly short work, so an even more extreme example might be better: one doesn’t need to have read the entire Talmud to criticize it.
It’s hardly “dishonest” to criticize a position based on a 7,000-word summary statement while admitting you haven’t read the whole corpus! You’re playing with words to make a moralistic debating point: dishonesty involves deceit, and everyone has been informed of the basis for my opinions.
Consider the double standard involved. Yudkowsky lambasts “philosophers” and their “confusions”—their supposedly misguided concerns with the issues other philosophers have commented on to the detriment of inquiry. Has Yudkowsky read even a single book by each of the philosophers he dismisses?
In a normal forum, participants supply the arguments supposedly missed by critics who are only partially informed. Here there are vague allusions to what the Apostle Yudkowsky (prophet of the Singularity God) “answered” without any substance. An objective reader will conclude that the Prophet stands naked; the prolixity is probably intended to discourage criticism.
I think the argument you make in this comment isn’t a bad one, but the unnecessary and unwarranted “Apostle Yudkowsky (prophet of the Singularity God)” stuff amounts to indirectly insulting the people you’re talking with and, makes them far less likely to realize that you’re actually also saying something sensible. If you want to get your points across, as opposed to just enjoying a feeling of smug moral superiority while getting downvoted into oblivion, I strongly recommend leaving that stuff out.
Thanks for the advice, but my purpose—given that I’m an amoralist—isn’t to enjoy a sense of moral superiority. Rather, to test a forum toward which I’ve felt ambivalent for several years, mainly for my benefit but also for that of any objective observers.
Strong rhetoric is often necessary in an unreceptive forum because it announces that the writer considers his criticisms fundamental. If I state the criticisms neutrally, something I’ve often tried, they are received as minor—like the present post. They may even be voted up, but they have little impact. Strong language is appropriate in expressing severe criticisms.
How should a rationalist forum respond to harsh criticism? It isn’t rational to fall prey to the primate tendency to in-group thinking by neglecting to adjust for any sense of personal insult when the group leader is lambasted. Judging by reactions, the tendency to in-group thought is stronger here than in many forums that don’t claim the mantle of rationalism. This is partly because the members are more intelligent than in most other forums, and intelligence affords more adept self-deception. This is why it is particularly important for intelligent people to be rationalists but only if they honestly strive to apply rational principles to their own thinking. Instead, rationality here serves to excuse participants’ own irrationality. Participants simply accept their own tendencies to reject posts as worthless because they contain matter they find insulting. Evolutionary psychology, for instance, here serves to produce rationalizations rather than rationality. (Overcoming Bias is a still more extreme advocacy of this perversion of rationalism, although the tendency isn’t expressed in formal comment policies.)
“Karma” means nothing to me except as it affects discourse; I despise even the term, which stinks of Eastern mysticism. I’m told that the karma system of incentives, which any rationalist should understand vitally affects the character of discussion, was transplanted from reddit. How is a failure to attend to the vital mechanics of discussion and incentives rational? Laziness? How could policies so essential be accorded the back seat?
Participants, I’m told, don’t question the karma system because it works. A rationalist doesn’t think that way. He says, “If a system of incentives introduced without forethought and subject to sound criticisms (where even its name is an insult to rationality) produces the discourse that we want, then something must be wrong with what we want!” What’s wanted is the absence of any tests of ideology by fundamental dissent.
Some of them are simply not great writers. Hegel for example is just awful- the few coherent ideas in Hegel are more usefully described by other later writers. There’s also a strange aspect to this in that you are complaining about Eliezer not having read books while simultaneously defending your criticism of Eliezer’s metaethics positions without having read all his posts. Incidentally, if one wants to criticize Eliezer’s level of knowledge of philosophy, a better point is not so much the philosophers that he criticizes without reading, but rather his lack of knowledge of relevant philosophers that Eliezer seems unaware of, many of whom would agree with some of his points. Quine and Lakatos are the most obvious ones.
I strongly suspect that your comments would be responded to more positively if they didn’t frequently end with this sort of extreme rhetoric that has more emotional content than rational dialogue. It is particularly a problem because on theLW interface, the up/down buttons are at the end of everything one has read, so what the last sentences say may have a disproportionate impact on whether people upvote or downvote and what they focus on in their replies.
Frankly, you have some valid points, but they are getting lost in the rhetoric. We know that you think that LW pattern matches to religion. Everyone gets the point. You don’t need to repeat that every single time you make a criticism.
And you would be making the decision to override with… what, your spleen?
Another part of my brain—besides the part computing the morality function Yudkowsky posits.
Surely you can’t believe Yudkowsky simply means whatever our brain decides is “moral”—and that he offers that as a solution to anything?
I’m not saying he’s right, just that your proposed alternative isn’t even wrong.
I’m not saying he’s right, I’m saying your proposed alternative isn’t even wrong.
This is obviously false. Yudkowsky does not claim to be able to do Solomonoff induction in his head.
In general, when Yudkowsky addresses humanity’s faults, he is including himself.
Point taken.
But Yudkowsky says “built around the assumption that you’re too stupid… to just use …”
If Solomonoff induction can’t easily be used in place of science, why does the first sentence imply the process is simple: you just use it?
You’ve clarified what Yudkowsky does not mean. But what does he mean? And why is it so hard to find out? This is the way mystical sects retain their aura while actually saying little.
“You’re too stupid and self-deceiving to just use Solomonoff induction” ~ “If you were less stupid and self deceiving you’d be able to just use Solomonoff induction” + “but since you are in fact stupid and self-deceiving, instead you have to use the less elegant approximation Science”
That was hard to find out?
Actually, yes, because of the misleading signals in the inept writing. But thank you for clarifying.
Conclusion: The argument in written in a crazy fashion, but it really is merely stupid. There is no possible measure of simplicity that isn’t language relative. How could there be?
You seem to be confusing “language relative” with “non-mathematical.” Kolmogorov Complexity is “language-relative,” if I’m understanding you right; specifically, it’s relative (if I’m using the terminology right?) to a Turing Machine. This was not relevant to Eliezer’s point, so it was not addressed.
(Incidentally, this is a perfect example of you “hold{ing} views contrary to scientific consensus in specialized fields where {you} lack expert knowledge based on pure ratiocination,” since Kolmogorov Complexity is “one of the fundamental concepts of theoretical computer science”, you seemingly lack expert knowledge since you don’t recognize these terms, and your argument seems to be based on pure ratiocination.)
When I read that line for the first time, I understood it. Between our two cases, the writing was the same, but the reader was different. Thus, the writing cannot be the sole cause of our different outcomes.
Well, if a substantial fraction of readers read something differently or can’t parse it, it does potentially reflect a problem with the writing even if some of the readers, or even most readers, do read it correctly.
Absolutely. I intended to convey that if you don’t understand something, that the writing is misleading and inept is not the only possible reason. srdiamond is speaking with such confidence that I felt safe tabling further subtleties for now.
The philosophizing of inept, verbose writers like Yudkowsky can be safely dismissed based solely on their incompetence as writers. For a succinct defense of this contention, see my “Can bad writers be good thinkers? Part 1 of THE UNITY OF LANGUAGE AND THOUGHT” OR see the 3-part “Writing & Thought series” — all together, fewer than 3,000 words.
I believe what you wrote because you used so much bolding.
Way to deflect attention from substance to form. Exemplary rationality!
I can’t tell which way your sarcasm was supposed to cut.
The obvious interpretation is that you think rationality is somehow hindered by paying attention to form rather than substance, and the “exemplary rationality” was intended to be mocking.
But your comment being referenced was an argument that form has something very relevant to say about substance, so it could also be that you were actually praising gwern for practicing what you preach.
I choose to interpret it as praise, and receive a warm fuzzy feeling.
I read your three-part series. Your posts did not substantiate the claim “good thinking requires good writing.” Your second post slightly increased my belief in the converse claim, “good thinkers are better-than-average writers,” but because the only evidence you provided was a handful of historical examples, it’s not very strong evidence. And given how large the population of good thinkers, good writers, bad thinkers, and bad writers is relative to your sample, evidence for “good thinking implies good writing” is barely worth registering as evidence for “good writing implies good thinking.”
I was going to upvote your comment until I got to this point. Aside from the general mindkilling, this looks like the fundamental attribution error, and moreover, we all know that people do in fact mature and change. Bringing up external politics is not helpul in a field where there’s already concern that AI issues may be becoming a mindkilling subject themselves on LW. Bringing up such a questionable one is even less useful.
That’s LW “rationality” training for you—”fundamental error of attribution” out of context—favored because it requires little knowledge and training in psychology. Such thinking would preclude any investigation of character. (And there are so many taboos! How do you all tolerate the lockstep communication required here?)
Paul Meehl, who famously studied clinical versus statistical prediction empirically, noted that even professionals, when confronted by instance of aberrant behavior, are apt to call it within normal range when it clearly isn’t. Knowledge of the “fundamental error of attribution” alone is the little bit of knowledge that’s worse than total ignorance.
Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.
You can’t dismiss politics just because it kills some people’s minds, when so much of the available information and examples come from politics. (There are other reasons, but that’s the main one here.) Someone who can’t be rational about politics simply isn’t a good rationalist. You can’t be a rationalist about the unimportant things and rationalist about the important ones—yet call yourself a rationalist overall.
I’m sure I wouldn’t have done what Romney did, and not so sure about whether I would have done what Yudkowsky did. Romney wanted to hurt people for the fun of it. Yudkowsky was trying to keep people from being hurt, regardless of whether his choice was a good one.
That’s a reasonable answer.
It seems almost unfair to criticize something as a problem of LW rationality when in your second paragraph you note that professionals do the same thing.
I’m not sure. A while ago, I was involved in a situation where someone wanted to put personal information of an individual up on the internet knowing that that person had an internet stalker who had a history of being a real life stalker for others. The only reason I didn’t react pretty close to how Eliezer reacted in the quoted incident is that I knew that the individual in question was not going to listen to me and would if anything have done the opposite of what I wanted. In that sort of context, Eliezer’s behavior doesn’t seem to be that extreme. Eliezer’s remarks involve slightly more caps than I think I would use in such a circumstance, but the language isn’t that different.
This does connect to another issue though- the scale in question of making heated comments on the internet as opposed to traumatic bullying, are different. The questions I ask myself for what it would take to do something similar to what Eliezer did are very different than the same questions for the Romney incident.
Your basic statement does it seem have some validity. One could argue that the Romney matter reflects the circumstances where he was at the time, and what was considered socially acceptable as forms of interaction or establishing dominance hierarchies. Through most of human history, that sort of behavior would probably be considered fairly tame. But this is a weak argument- even if it was due to the circumstances that Romney was in at the time, there’s no question that those were his formative years, and thus could plausibly have had a permanent impact on his moral outlook.
The problem is that even as relevant examples come from politics, those are precisely the examples that people are least likely to agree actually demonstrate the intended point in question. For example, in this case, many people who aren’t on the left will downplay the Romney bullying. Given that I’m someone who dislikes Romney (both in terms of personality and in terms of policy) and am not convinced that this is at all fair, using such a controversial example seems unwise. Even if one needs to use political examples, one can use examples from 10 or 15 or 30 years ago that are well known but have had their tribalness diminish in time. For example, in this context one could use a variety of examples connected to Richard Nixon.
Well, we can acknowledge that we’re better at being rational in some areas than we are in others. Frankly, I wouldn’t mind and for reasons essentially similar to your remark would endorse some amount of reduction of the no-politics rule here. Where that becomes a problem is when one tries to connect politics to other potentially controversial issues.
What’s that about? (PM me if it’s still taboo.)
When Mitt Romney was in high school, he and some friends bullied a kid who looked (and later turned out to be) homosexual. At one point, Romney and some others grabbed the guy, held him down, and cut off a bunch of his hair with scissors.
Why do you continue to participate? Almost all of the cool stuff that high status people agree is plausible is available elsewhere.
The point, such as it is, would better have been left implied. Now, it’s subject to explicit scrutiny, and it must be found wanting. Consider what would have happened had Yudkowsky not shown exceptional receptivity to this post: he would have blatantly proven his critics right. The knowledge and reputation of the poster is unimpeachable.
The more significant fact is that these criticisms were largely unknown to the community. As Will Newsome implied, this is because the critical posts—lacking the high-status credential of this poster—remained in discussion and were almost ignored.
The majority’s intolerance for dissent is manifested mostly in its refusal to acknowledge it. Dissent is cabined to Discussion. It only gets noticed when the dissenter becomes frustrated and violates group norms. Then it gets voted down, but it still gets noticed and commented on. This is a malfunctioning reinforcement system, but maybe its the best possible. Still, it’s irrational to deny all in-group bias in LukeProg’s cheerleading fashion—in an instance where the absence of evidence (here, of bias) truly does not offer anything substantial in the way of evidence of lack of bias, to elicit LukeProg’s smug laughter.
After all, even the lead poster held off until now in voicing his opinion.
LWer tenlier disagrees, saying:
Also, you said:
Luckily, evidence on the matter is easy to find. As counter-evidence I present: Self-improvement or shiny distraction, SIAI an examination, Why we can’t take expected value estimates literally, Extreme rationality: it’s not that great, Less Wrong Rationality and Mainstream Philosophy, and the very post you are commenting on. Many of these are among the most upvoted posts ever.
Moreover, the editors rarely move posts from Main to Discussion. The posters themselves decide whether to post in Main or Discussion.
Also Should I believe what the SIAI claims? and the many XiXiDu posts linked therein, like What I would like the SIAI to publish.
I had a post moved from main to Discussion just today: before it had accumulated any negative votes, so I think you’re probably misinformed about editorial practices.But I don’t want to use my posts as evidence; the charge of bias would be hard to surmount. What’s plainly evident is that posters are reluctant to post to the Main area except by promotion.
You’re evidence is unpersuasive because you don’t weigh it against the evidence to the contrary. One good example to the contrary more than counter-balances it, since the point isn’t that no dissent is tolerated, not even that some dissent isn’t welcomed, but only that there are some irrational boundaries.
One is the quasi-ban on politics. Here is a comment that garnered almost 800 responses and was voted up 37. Why wasn’t it promoted? I bitterly disagree with the poster; so I’m not biased by my views. But the point is that it is a decidedly different view, one generating great interest, but the subject would not be to the liking of the editors.
Of course, it lacked the elaborateness—dare I say, the prolixity—of a typical top-level post. But this “scholarly” requirement is part of the process of soft censorship. The post—despite my severe disagreement with it—is a more significant intellectual contribution than many of the top-level posts, such as some of the second-hand scholarship.
[And I have to add: observe that the present discussion is already being downvoted at my first comment. I predict the same for this post in record time What does that mean?]
Can comments be promoted? Perhaps the commenter should have been encouraged to turn his comment into a top-level post, but a moderator can’t just change a comment into a promoted post with the same username. Also it would have split the discussion, so people might have been reluctant to encourage that.
As for people tending to post more in Discussion than Main, I read somewhere that Discussion has more readers. I for one read Discussion almost exclusively.
It would advance this discussion if someone would explain the down votes. I await LukeProg’s explanation of the present example of soft censorship.
I downvoted you because you’re wrong. For one, comments can’t be promoted to main, only posts, and for two, plenty of opposition has garnerned a great deal of upvotes, as shown by the numerous links lukeprog provided.
For example, where do you get ‘almost 800 responses’ from? That comment (not post) only has 32 comments below it.
Yes, I was wrong. But my point was correct. The 781 comments applied to the Main Post So:
The topic was popular, like I said.
The post could have been promoted!
But ask yourself, would you have been so harsh on a factual error had you agreed with the message? This is the way bias works, after all, by double standard more than outright discrimination. You could say I should have been more careful. But then, when you’ve learned not to expect a hearing, you’re not so willing to jump the hoops. But it’s your loss, if you’re a rationalist and if you’re losing input because dissenters find it’s not worth their time.
As to LukeProg providing example demonstrating welcoming dissent: you couldn’t have considered my counter-balancing evidence when you downvoted before taking the time even to explore the post to which the cited comment belongs.
To LukeProg: have I made my point about the limits of dissent at LW?
Posts which contain factual inaccuracies along with meta-discussion of karma effects are often downvoted.
I’ve addressed factual inaccuracies in another comment. But as for discussing karma effects—that wasn’t extraneous whining but was at the heart of the discussion. If you downvote discussion of karma—like you did—simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?
LukeProg: What do you say about the grounds on which downvotes are issued for dissenting matter. Isn’t it clear that this is a bias LW doesn’t want to talk about; perhaps altogether doesn’t want to discuss its own biases?
I don’t do that; I only downvote when it’s combined with incorrect facts. Which I’m tempted to do for this statement: “like you did—simply for mentioning it”, since you’re inferring my motivations, and once again incorrect.
Look, Rain, this is an Internet ongoing discussion. Nobody says everything precisely right. The point is that you would hardly be so severe on someone unless you disagreed strongly. You couldn’t be, because nobody would satisfy your accuracy demands. The kind of nitpicking you engage in your post would ordinarily lead you to be downvoted—and you should be, although I won’t commit the rudeness of so doing in a discussion.
The point wasn’t that you downvote when the only thing wrong with the comment is discussion of karma. It was that you treat discussion of karma as an unconditional wrong. So you exploited weaknesses in my phrasing to ignore what I think was obviously the point—that marking down for the bare mention of karma (even if it doesn’t produce a downvote in each case) is an irrational policy, when karma is at the heart of the discussion. There’s no rational basis for throwing it in as an extra negative when the facts aren’t right.
You’re looking for trivial points to pick to downvote and to ignore the main point, which was your counting mention of karma a negative, without regard to the subject, is an irrational policy. If we were on reversed sides, your nitpicking and evasion would itself be marked down. As matters stand, you don’t even realize you’re acting in a biased fashion, and readers either don’t know or don’t care.
Is that rational? Shouldn’t a rationalist community be more concerned with criticizing irrationalities in its own process?
Having been a subject of both a relatively large upvote and a relatively large downvote in the last couple of weeks, I still think that the worst thing one can do is to complain about censorship or karma. The posts and comments on any forum aren’t judged on their “objective merits” (because there is no such thing), but on its suitability for the forum in question. If you have been downvoted, your post deserves it by definition. You can politely inquire about the reasons, but people are not required to explain themselves. As for rationality, I question whether it is rational to post on a forum if you are not having fun there. Take it easy.
First, you’re correct that it’s irrational to post to a forum you don’t enjoy. I’ll work on decreasing my akrasia.
But it’s hard not to comment on a non sequitur like the above. (Although probably futile because one who’s really not into a persuasion effort won’t do it well.) That posts are properly evaluated by suitability to the forum does not imply that a downvoted post deserves the downvote by definition! That’s a maladaptive view of the sort I’m amazed is so seldom criticized on this forum. Your view precludes (by definition yet) criticism of the evaluators’ biases, which do not advance the forum’s purpose. You would eschew not only absolute merits but also any objective consideration of the forum’s function.
A forum devoted to rationality, to be effective and honest, must assess and address the irrationalities in its own functioning. (This isn’t always “fun.”) To define a post that should be upvoted as one that is upvoted constitutes an enormous obstacle to rational function.
I disagree; a downvote is not ‘severe’.
I disagree; meta-discussions often result in many upvotes.
I do not, and have stated as much.
If there is no point in downvoting incorrect facts, then I wonder what the downvote button is for.
I disagree; your main point is that you are being unfairly downvoted, along with other posts critical of SI being downvoted unfairly, which I state again is untrue, afactual, incorrect, a false statement, a lie, a slander, etc.
Questioning the rationality of meta-meta-voting patterns achieves yet another downvote from me. Sorry.
I don’t follow your reasoning, here. Having read this particular thread, it does seem as though you are, in fact, going out of your way to criticize and downvote srdiamond. Yes, he has, in fact, made a few mistakes. Given, however, that the point of this post in general is about dissenting from the mainstream opinions of the LW crowd, and given the usual complaints about lack of dissent, I find your criticism of srdiamond strange, to say the least. I have, accordingly, upvoted a number of his comments.
As expected, my previous comment was downvoted almost immediately.
This would, for reference, be an example of the reason why some people believe LW is a cult that suppresses dissent. After all, it’s significantly easier to say that you disagree with something than it is to explain in detail why you disagree; just as it’s far easier to state agreement than to provide an insightful statement in agreement. Nonetheless, community norms dictate that unsubstantiated disagreements get modded down, while unsubstantiated agreements get modded up. Naturally, there’s more of the easy disagreement then the hard disagreement… that’s natural, since this is the Internet, and anyone can just post things here.
In any event, though, the end result is the same; people claim to want more dissent, but what they really mean is that they want to see more exceptionally clever and well-reasoned dissent. Any dissent that doesn’t seem at least half as clever as the argument it criticizes seems comparatively superfluous and trivial, and is marginalized at best. And, of course, any dissent that is demonstrably flawed in any way is aggressively attacked. That really is what people mean by suppression of dissent. It doesn’t really mean ‘downvoting arguments which are clever, but with which you personally disagree’… community norms here are a little better then that, and genuinely good arguments tend to get their due. In this case, it means, ‘downvoting arguments which aren’t very good, and with which you personally disagree, when you would at the same time upvote arguments that also aren’t very good, but with which you agree’. Given the nature of the community norms, someone who expresses dissent regularly, but without taking the effort to make each point in an insightful and terribly clever way, would tend to be downvoted repeatedly, and thus discouraged from making more dissent in the future… or, indeed, from posting here at all.
I don’t know if there’s a good solution to the problem. I would be inclined to suggest that, like with Reddit, people not downvote without leaving an explanation as to why. For instance, in addition to upvoting some of srdiamond’s earlier comments, I have also downvoted some of Rain’s, because a number of Rain’s comments in this thread fit the pattern of ‘poor arguments that support the community norms’, in the same sense that srdiamond’s fit the pattern of ‘poor arguments that violate the community norms’; my entire point here is that, in order to cultivate more intelligent dissent, there should be more of the latter and less of the former.
I downvote any post that says “I expect I’ll get downvoted for this, but...” or “the fact that I was downvoted proves I’m right!”
I’m fond of downvoting “I dare you to downvote this!”
So, in other words, you automatically downvote anyone who explicitly mentions that they realize they are violating community norms by posting whatever it is they are posting, but feels that the content of their post is worth the probable downvotes? That IS fairly explicitly suppressing dissent, and I have downvoted you for doing so.
I don’t think it is suppression of dissent per se. It is more annoying behavior- it implies caring a lot about the karma system, and it is often not even the case when people say that they will actually get downvoted. If it is worth the probable downvote, then they can, you know, just take the downvote. If they want to point out that a view is unpopular they can just say that explicitly. It is also annoying to people like me, who are vocal about a number of issues that could be controversial here (e.g. criticizing Bayesianism, cryonics,, and whether intelligence explosions would be likely) and get voted up. More often than not, when someone claims they are getting downvoted for having unpopular opinions, they are getting downvoted in practice for having bad arguments or for being uncivil.
There are of course exceptions to this rule, and it is disturbing to note that the exceptions seem to be coming more common (see for example, this exchange where two comments are made with about the same quality of argument and about the same degree of uncivility- (“I’m starting to hate that you’ve become a fixture here.” v. “idiot”—but one of the comments is at +10 and the other is at −7.) Even presuming that there’s a real disagreement in quality or correctness of the arguments made, this suggests that uncivil remarks are tolerated more when people agree with the rest of the claim being made. That’s problematic. And this exchange was part of what prompted me to earlier suggest that we should be concerned if AGI risk might be becoming a mindkiller here. But even given that, issues like this seem not at all common.
Overall, if one needs to make a claim about one is going to be downvoted, one might even be correct, but it will often not be for the reasons one thinks it is.
Bears repeating.
I don’t think it’s so much ‘caring a lot about the karma system’ per se, so much as the more general case of ‘caring about the approval and/or disapproval of one’s peers’. The former is fairly abstract, but the latter is a fairly deep ancestral motivation.
Like I said before, it’s clearly not much in the way of suppression. That said, given that, barring rare incidents of actual moderation, it is the only ‘suppression’ that occurs here, and since there is a view among various circles that there there is, in fact, suppression of dissent, and since people on the site frequently wonder why there are not more dissenting viewpoints here, and look for ways to find more… it is important to look at the issue in great depth, since it’s clearly an issue which is more significant than it seems on the surface.
Exactly right. But a group that claims to be dedicated to rationality loses all credibility when participants not only abstain from considering this question but adamantly resist it. The only upvote you received for your post—which makes this vital point—is mine.
This thread examines HoldenKarnofsky’s charge that SIAI isn’t exemplarily rational. As part of that examination, the broader LW environment on which it relies is germane. That much has been granted by most posters. But when the conversation reaches the touchstone of how the community expresses its approval and disapproval, the comments are declared illegitimate and downvoted (or if the comments are polite and hyper-correct, at least not upvoted).
The group harbors taboos. The following subjects are subject to them: the very possibility of nonevolved AI; karma and the group’s own process generally (an indespensable discussion ); and politics. (I’ve already posted a cite showing how the proscription on politics works, using an example the editors’ unwillingness to promote the post despite receiving almost 800 comments).
These defects in the rational process of LW help sustain Kardofsky’s argument that SIAI is not to be recommended based on the exemplary rationality of its staff and leadership. They are also the leadership of LW, and they have failed by refusing to lead the forum toward understanding the biases in its own process. They have fostered bias by creating the taboo on politics, as though you can rationally understand the world while dogmatically refusing even to consider a big part of it—because it “kills” your mind.
P.S. Thank you for the upvotes where you perceived bias.
Nah. If there is a mindkiller then it is the reputation system. Some of the hostility is the result of the overblown ego and attitude of some of its proponents and their general style of discussion. They created an insurmountable fortress that shields them from any criticism:
Troll: If you are so smart and rational, why don’t you fund yourself? Why isn’t your organisation sustainable?
SI/LW: Rationality is only aimed at expected winning.
Troll: But you don’t seem to be winning yet. Have you considered the possibility that your methods are suboptimal? Have you set yourself any goals, that you expect to be better at than less rational folks, to test your rationality?
SI/LW: Rationality is a caeteris paribus predictor of success.
Troll: Okay, but given that you spend a lot of time on refining your rationality, you must believe that it is worth it somehow? What makes you think so then?
SI/LW: We are trying to create a friendly artificial intelligence implement it and run the AI, at which point, if all goes well, we Win. We believe that rationality is very important to achieve that goal.
Troll: I see. But there surely must be some sub-goals that you anticipate to be able to solve and thereby test if your rationality skills are worth the effort?
SI/LW: Many of the problems related to navigating the Singularity have not yet been stated with mathematical precision, and the need for a precise statement of the problem is part of the problem.
Troll: Has there been any success in formalizing one of the problems that you need to solve?
SI/LW: There are some unpublished results that we have had no time to put into a coherent form yet.
Troll: It seems that there is no way for me to judge if it is worth it to read up on your writings on rationality.
SI/LW: If you want to more reliably achieve life success, I recommend inheriting a billion dollars or, failing that, being born+raised to have an excellent work ethic and low akrasia.
Troll: Awesome, I’ll do that next time. But for now, why would I bet on you or even trust that you know what you are talking about?
SI/LW: We spent a lot of time on debiasing techniques and thought long and hard about the relevant issues.
Troll: That seems to be insufficient evidence given the nature of your claims and that you are asking for money.
SI/LW: We make predictions. We make statements of confidence of events that merely sound startling. You are asking for evidence we couldn’t possibly be expected to be able to provide, even given that we are right.
Troll: But what do you anticipate to see if your ideas are right, is there any possibility to update on evidence?
SI/LW: No, once the evidence is available it will be too late.
Troll: But then why would I trust you instead of those experts who tell me that you are wrong?
SI/LW: You will soon learn that your smart friends and experts are not remotely close to the rationality standards of SI/LW, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don’t.
Troll: But you have never achieved anything when it comes to AI, why would I trust your reasoning on the topic?
SI/LW: That is magical thinking about prestige. Prestige is not a good indicator of quality.
Troll: You won’t convince me without providing further evidence.
SI/LW: That is a fully general counterargument you can use to discount any conclusion.
The last exchange was hilarious. This is parody, right?
Downvoted for downvoting downvoting of downvoting of downvoting.
If you do the same to this comment, we can enter a stable loop!
First, none of this dissent has been suppressed in any real sense. It’s still available to be read and discussed by those who desire reading and discussing such things. The current moderation policy has currently only kicked in when things have gotten largely out of hand—which is not the case here, yet.
Second, net karma isn’t a fine enough tool to express amount of detail you want it to express. The net comment on your previous comment is currently −2; congrats, you’ve managed to irritate less than a tenth of one percent of LW (presuming the real karma is something like −2/+0 or −3/+1)!
Third, the solution you propose hasn’t been implemented anywhere that I know of. Reddit’s suggested community norm (which does not apply to every subreddit) suggests considering posting constructive criticism only when one thinks it will actually help the poster improve. That’s not really the case much of the time, at least on the subreddits I frequent, and it’s certainly not the case often here.
Fourth, the solution you propose would, if implemented, decrease the signal-to-noise ratio of LW further.
Fifth, reddit’s suggested community norm also says “[Don’t c]omplain about downvotes on your posts”. Therefore, I wonder how much you really think reddit is doing the community voting norm thing correctly.
First; downvoted comments are available to be read, yes; but the default settings hide comments with 2 or more net downvotes. This is enough to be reasonably considered ‘suppression’. It’s not all that much suppression, true, but it is suppression… and it is enough to discourage dissent. Actual moderation of comments is a separate issue entirely, and not one which I will address here.
Second; when I posted my reply, and as of this moment, my original comment was at −3. I agree; net karma isn’t actually a huge deal, except that it is, as has been observed, the most prevalent means by which dissent is suppressed. In my case, at least, ‘this will probably get downvoted’ feels like a reason to not post something. Not much of a reason, true, but enough of one that I can identify the feeling of reluctance.
Third; on the subreddits I follow (admittedly a shallow sampling), I have frequently seen comments explaining downvotes, sometimes in response to a request specifically for such feedback, but just as often not. I suspect that this has a lot to do with the “Down-voting? Please leave an explanation in the comments.” message that appears when mousing over the downvote icon. I am aware that this is not universal across Reddit, but on the subreddits I follow, it seems to work reasonably well.
Fourth; I agree that this is a possible result. Like I said before, I’m not sure if there is a good solution to this problem, but I do feel that it’d result in a better state then that which currently exists, if people would more explicitly explain why they downvote when they choose to do so. That said, given that downvoted comments are hidden from default view anyway, and that those who choose to do so can easily ignore such comments, I don’t think it’d have all that much effect on the signal/noise ratio.
Fifth; on the subreddits I follow, it seems as though there is less in the way of complaints about downvotes, and more honest inquiries as to why a comment has been downvoted; such questions seem to usually receive honest responses. This may be anomalous within Reddit as a whole; as I said before, my own experience with Reddit is a shallow sampling.
Perhaps the solution is not to worry so much about my bad contrarian arguments being downvoted as to assure that bad “establishment” arguments are downvoted—as in Rain’s case, they aren’t. Regurgitation of arguments others have repeatedly stated should also be downvoted, no matter how good the arguments.
The reason to think an emphasis on more criticism of Rain rather than less criticism of me is that after I err, it’s a difficult argument to establish that my error wasn’t serious enough to avoid downvote. But when Rain negligently or intentionally misses the entire point, there’s less question that he isn’t benefiting the discussion. It’s easier to convict of fallacy than to defend based on the fallacy being relatively trivial. There’s a problem in that the two determinations are somewhat inter-related, but it doesn’t eliminate the contrast.
Increasing the number of downvotes would deflate the significance of any single downvote and would probably foster more dissent. This balance may be subject to easy institutional control. Posters are allotted downvotes based on their karma, while the karma requirements for upvotes are easily satisfied, if they exist. This amounts to encouraging upvotes relative to downvotes, with the result that many bad posts are voted up and some decent posts suffer the disproportionate wrath of extreme partisans. (Note that Rain, a donor, is a partisan of SIAI.)
The editors should experiment with increasing the downvote allowance. I favor equal availability of downvotes and upvotes as optimal (but this should be thought through more carefully).
After turning this statement around in my head for a while I’m less certain than I was that I understand its thrust. But assuming you mean those critics pertinent to lukeprog’s post, i.e. those claiming LW embodies a cult of personality centered around Eliezer—well, no. Eliezer’s reaction is in fact almost completely orthogonal to that question.
If you receive informed criticism regarding a project you’re heavily involved in, and you react angrily to it, that shows nothing more or less than that you handle criticism poorly. If the community around you locks ranks against your critics, either following your example or (especially) preemptively, then you have evidence of a cult of personality.
That’s not what happened here, though. Eliezer was fairly gracious, as was the rest of the community. Now, that is not by itself behavior typically associated with personality cults, but before we start patting ourselves on the back it’s worth remembering that certain details of timing and form could still point back in the other direction. I’m pretty sick of the cult question myself, but if you’re bound and determined to apply this exchange to it, that’s the place you should be looking.
Conservation of expected evidence may be relevant here.
After I recently read that the lead poster was a major financial contributor to SIAI, I’d have to call LukeProg’s argument disingenuous if not mendacious.
Rain (who noted that he is a donor to SIAI in a comment) and HoldenKarnofsky (who wrote the post) are two different people, as indicated by their different usernames.
Well, different usernames isn’t usually sufficient evidence that there are two different people, but in this case there’s little doubt about their separability.
I don’t understand. Holden is not a major financial contributor to SIAI. And even if he was: which argument are you talking about, and why is it disingenuous?
If Holden were a major contributor, your argument that the LW editors demonstrated their tolerance for dissent by encouraging the criticisms he made would be bogus. Suppressing the comments of a major donor would be suicidal, and claiming not doing so demonstrates any motive but avoiding suicide would be disingenuous at the least.
If he’s not a donor, my apologies. In any event, you obviously don’t know that he’s a donor if he is, so my conclusion is wrong. I thought Yudkowsky said he was.
I’m confused. Holden doesn’t believe SI is a good organisation to recommend giving money to, he’s listed all those objections to SI in his post, and you somehow assumed he’s been donating money to it?
That don’t make sense.