In light of how important it is (especially in terms of all the decision theory posts!) to know about Judea Pearl’s causality ideas, I thought I might share something that helped me get up to speed on it: this lecture on Reasoning with Cause and Effect.
For me, it has the right combination of detail and brevity. Other books and papers on Pearlean causality were either too pedantic or too vague about the details, but I learned a lot from the slides, which come with good notes. Anyone know what proceedings papers it refers to though?
Does anyone feel the occasional temptation to become religious?
I do, sometimes, and push it away each time. I doubt that I could, really, fall for that temptation—even if I tried to, the illogicality of the whole thing would very likely prevent me from really believing in even a part of it very seriously. And as more and more time passes, religious structures begin to seem more and more ridiculous and contrived. Not that I’d have believed that them starting to feel more contrived would even have been possible.
And yet… occasionally I remember the time back in my teens, when I had some sort of a faith. I remember the feeling of ultimate safety it brought with it—the knowledge that no matter what happens, everything will turn out well in the end. It might be a good thing that I spend time worrying over existential risks, and spend time thinking about what I could do about them, but it sure doesn’t exactly improve my mental health. The thought of returning to the mindset of a believer appeals on an emotional level, in the same way stressed adults might longingly remember the carefree days of childhood. But while you can’t become a child again, becoming a believer is at least theoretically possible. And sometimes I do play around with the idea of what it’d be like, to adopt a belief again.
Uh, here is a confession. Twice in the last 6-7 years, at moments of extreme psychological distress, I talked to the God of my Catholic youth. Once I went to an empty church after a series of coincidences (running into two people from my Catholic grade school separately) that I thought was a sign from God. Really, embarrassing, right? It was like my mind segmented and the rationalist was put aside and the devout Catholic school boy was put in charge. The last time this happened was about four years ago.
I don’t know if this could still happen today (my atheism is probably more entrenched now). There have been plenty of periods of distress where this didn’t happen, so I don’t know what triggered it in particular. I think my brain must have really needed a God figure at that moment and didn’t know how to deal with the pain without one so it hacked itself and turned off the rationalist defenses. Or something, it seems so screwed up looking back on it.
Interestingly, I told my theistic-non religious girlfriend about this who in turn told her Christian best friend. Talking to me on the phone for the first time, the friend something along the lines of “I know you say you’re an atheist but B(my girlfriend) told me that you sometimes pray when you’re upset so I know you’re really a good person/God loves you.” In other words, I’m least wrong when I’m thinking least clearly.
On the consolidation of dust specks and the preservation of utilitarian conclusions:
Suppose that you were going to live for at least 3^^^3 seconds. (If you claim that you cannot usefully imagine a lifespan of 3^^^3 seconds or greater, I must insist that you concede that you also cannot usefully imagine a group of 3^^^3 persons. After all, persons are a good deal more complicated than seconds, and you have experienced more seconds than people.)
Suppose that while you are contemplating how to spend your 3^^^3-plus seconds, you are presented with a binary choice: you may spend the next 50 years of this period of time being tortured, or you may spend the next 3^^^3 seconds with a speck of dust in your eye that you cannot get rid of until that time period is up. (Should you succeed in uploading or similar over the course of the next 3^^^3 seconds, the sensation of the speck in the eye will accompany you in the absence of a physical eye until you have waited it out). Assume that after the conclusion of the torture (should you select it), you will be in fine physical health to go on with the rest of your lengthy life, although no guarantees are made for your sanity. Assume that the speck of dust does not impede your vision, and that you will not claw out your eye trying to be rid of it at any time; likewise, no guarantees are made for your sanity.
If this choice was actually presented to someone, my guess would be that he would first choose the speck, and then after an extremely long time (i.e. much, much longer than 50 years, giving him a sense of proportion) he would undergo a preference reversal and ask for the torture.
What if the TORTURE occurs during a random time in the next 3^^^3 seconds, not right at the beginning? Also, I think we definitely require a limit on sanity damage because otherwise the scenario is being tortured for 50 years and then spending the next 3^^^3 seconds being insane which Vastly outweighs the ordinary scenario of being tortured for 50 years.
I think I did specify that no one would die who would otherwise be immortal; eternal insanity or 3^^^3 years of insanity ought to be implicitly included, I’d think.
This phrase makes the difference for me- the 3^^^3 other people in the original argument weren’t mad- or at least, no more than would have been mad anyway.
Additionally, in your scenario, we have to consider discount rates- it’s certainly conceivable that someone might choose the dust specks over torture now, but be willing to forgo the dust specks in return for torture in 3^^^3 seconds time.
Does it seem likely to you that out of 3^^^3 people chosen with no particular safeguards, not one of them will find a dust speck in the eye to be maddening? It could be the last straw in a string of misfortunes; it could set off some causal chain that will lead to other maddening events, etc.
3^^^3 is such a huge number, some must find it maddening, but the proportion will be a lot lower than the odds that 50 years of torture breaks you mentally.
That’s interesting: I have much, much less hesitation in saying TORTURE to this one. With the original, I can grudgingly concede that I suppose I possibly ought to choose TORTURE, but I still can’t ever quite convince myself to feel that it’s a good answer. This one, I think I can.
Suppose you are given a button that you can press at any time during the 50 years of torture, that will stop the torture (and erase your memory of it if you wish), but you’ll have to live with the dust speck from then on.
I predict that you’ll press the button after actually being tortured for a couple of hours, maybe days, but at most weeks. Even professional spies/soldiers/terrorists who have trained to resist torture end up betraying their cause, so I find it hard to believe that you can hold out for 50 years.
But if you really prefer TORTURE now, that brings up an interesting question: whose preferences are more important, the current you, or the hypothetical future you? It could be argued that the future you is in a better position to decide, since she knows what it actually feels like to be tortured for a significant period of time, whereas you don’t.
But I don’t consider that a knock-down argument, so what do you think? Suppose you can also commit to not pressing the button (say by disabling your arm/hand muscles for 50 years), would you do so?
(This is related to a recent comment by Rolf Andreassen, which I think applies better to this scenario.)
Yes, I think you’re almost certainly right about the button, which thought does indeed put a dent in my lesser hesitation in choosing TORTURE. I think I would definitely not commit to not pressing the button if I were able to “try out” the SPECKS scenario for some short period of time first (say, a week). That way I could make a comparison. Absent that condition… I don’t know. I can’t imagine having the guts to commit to not pressing it.
The fact is that while I certainly don’t see a momentary dust speck as torture, I can easily imagine beginning to see it as torture after a day, never mind 3^^^3 seconds, which is rather longer than 50 years. But I can’t be certain of that, nor of to what extent I would get used to it, nor of how it would compare with much worse kinds of torture. (But then again, there’s a spanner in the works: knowing that you won’t have any lasting physical damage… perhaps that would lend some kind of strength of mind to a torturee?)
Sean Carroll and Carl Zimmer are leaving Bloggingheads, mostly because it’s started playing nice with creationists. Click their names to read their full explanations.
Thanks for the link. I saw the original Behe podcast and was surprised when it suddenly disappeared with a brief note from an admin. I haven’t been able to follow up on this matter much since it hasn’t generated much buzz, but I’ll look forward to reading these links—if people are boycotting Bloggingheads for giving platforms to people whom they don’t like.… well, that’s an interesting precedent to set which should be scrutinized carefully.
Hm. (Reads.) Well… if BHTV has me on, an “anti-accomodationist” as I’ve recently heard my diavlogs called (what a lovely term!), then I don’t think it’s unfair for them to give Behe a slot. It is extremely foolish that they paired him up with a linguist. (Reads.) Looks like Robert Wright agrees with that last part.
You’re one of the last people I would have expected to be concerned with ‘fairness’ when one of the sides is blatant crackpottery. I suppose you wouldn’t have a problem if BHTV invited an astrologer, as long as they paired him up with an astronomer?
What harm is done by bringing on an astrologer? At worst it fail to amuse.
But it’s obvious you’re not talking about the diavlog’s impact on you… you’re concerned with the poor, unwashed masses who might actually be left to form their own opinions from the available information. Well, that’s very nice of you, but I don’t think it’s unreasonable to believe that it’s safe to expose people to views which might be labeled as “crackpottery” by some.
High-status serious people want to associate with high-status others. Allowing crackpots on the same venue dramatically decreases its attractiveness for quality participants.
You can’t really give an elaborate justification for why in this particular case it’s OK, because signals are shallow.
The “unwashed masses”, as you call them, are already getting plenty of exposition to crackpottery, much more than they get to real science, so that a few crackpots on BHTV are barely a drop in the ocean. That’s not what concerns me, no.
What concerns me is that BHTV has a reputation as a respectable website thanks to the participation of respectable academics and experts. It’s reasonable to assume that such a respectable website wouldn’t invite crackpots to promote their brand of crackpottery; in fact, that’s an assumption I made myself until I read Sean’s and Carl’s posts. Inviting crackpots therefore gives the impression that these people should be taken seriously, even if we think they’re wrong.
In any event, your own motives are suspect, to say the least. Characterizing creationist nuts as “people whom [Sean and Carl] don’t like”, as if creationism was merely a distasteful political opinion, or something, makes you sound like a crackpot yourself, or worse, a postmodernist.
So BHTV can’t both enjoy the participation of respectable academics and also host the occasional crackpot? There exists no such universe where the two could possibly coincide? Is there some implicit assumption here that there’s a fixed amount of BHTV episodes, each of which will feature either crackpots or respectable academics? Even if this were so, wouldn’t the reasonable response be to skip over the crackpots rather than avoiding the entire medium? The only justifiable rationale I can see for skipping over BHTV because of this is if you just watched diavlogs at random and having crackpots degraded the signal:noise ratio of the site. But I doubt that you, I, Sean, Carl, or your average Bloggingheads viewer navigates the site in this manner.
Even though I profoundly degree with Behe’s epistemology (and theology), which should go without saying in these parts, I found the debate interesting (I think irreducible complexity is a neat topic), certainly moreso than I’ve enjoyed other diavlogs. Can anyone honestly say that Behe’s presence is less valuable than any other podcast on the website? I doubt it, and thus it strikes me as disingenuous that the unique response his presence generates can be explained away purely through outrage at the notion that somewhere, someone’s time may be wasted.
And I didn’t include the “people whom others don’t like” line as a defense of creationism per se, but as a broader point about silencing views found in contempt. To rip off Will Wilkinson, I’d probably venture to assert that unrepentant Marxists are just as high on the crackpottery scale as Creationists, but I highly doubt we’d see people abandon the site in protest in BHTV hosted some of them. “Respectability” in this context is a tricky term to use, since “respectability” tends to be conferred by social fashions just as much as actual correspondence to whatever virtues we’ve deemed to be worthy of respect. On a more base level, I suspect that many participants in this community have been dismissed as “crackpots” in some context or other before, and are skeptical of the neutrality and intellectual virtues of those who tend to yield the power of the censor. This isn’t a philosophical defense of subjectivism or postmodernism, but an institutional defense of the rough reasons why we don’t just go ahead and burn Behe at the stake.
Should BHTV invite Perez Hilton to debate the fearsome Man Bat? Michael Behe is as credible an author as Pamela Anderson, although not quite as illuminating. I used to think that the worst kind of ignorance was when you knew you were wrong and refused to accept it. Now I think the worst kind is when you know you’re capable of knowing when you’re wrong but refuse to let yourself. Michael Behe wants to be ignorant of his own ignorance. Let him do so in the peace and quiet of his own sad little world.
You’re shifting the goalposts some. I’m not defending the original decision to invite Behe. I’m questioning the notion that inviting Behe is such an egregious offense against BHTV’s “respectability” that it should be boycotted. I wouldn’t boycott BHTV if 90% of the diavlogs were replaced by midget porn, if it meant that I would get the occasional episode of Free Will.
I think Behe’s critics should just admit that what’s really motivating the reaction is the notion that Creationists not only should not be given forums to speak, but those who do grant Creationists forums to speak should be actively identified and boycotted in a way which is reserved for an arguably arbitrarily-defined set of social undesirables. This isn’t an indefensible position, but people have to admit to holding this belief (or some similar belief which is constructed in a more-charitable manner) before a meaningful debate can be enjoined.
[Edit]
Reading over the comments section of the CV posts, it looks like a lot of people are quick to point to Megan McArdle as the political crackpot equivalent of Behe. Should her presence be boycotted too as detrimental to the site? Where should the line be drawn? Where do you actually think the line would be drawn, if not along questionable ideological lines? Why have a line at all?
I think Behe’s critics should just admit that what’s really motivating the reaction is the notion that Creationists not only should not be given forums to speak, but those who do grant Creationists forums to speak should be actively identified and boycotted in a way which is reserved for an arguably arbitrarily-defined set of social undesirables.
What critic will not admit that? It’s hardly a fringe opinion in the scientific community that Creationists should not be given forums to speak on the thoroughly unscientific topic of Creationism, and that those who do so and call it science are being absurdly and unnecessarily tolerant. Creationism has never been more or less than an attack on science. It’s extremely toxic, and while I would never try to “silence” anyone, I don’t think it deserves more publicity. I grew up being taught that dreck in a fundamentalist Christian school and I’m more familiar with Behe than I’d care to be. Frankly, he’s an idiot, and his life purpose seems to be toward making more idiots. He doesn’t need anyone’s help.
As for McArdle, I don’t really care. Politics is not a hard science, and while she’s something of a crackpot, she’s not that way because somebody proved her map doesn’t follow the territory. It’s the difference between someone who thinks the earth is flat and someone who thinks it’s run by the Illuminati. The former is just wrong, the latter is just crazy. I don’t mind crazy, because crazy isn’t nearly as dangerous as wrong.
What I objected to about the creationists was that they were not worthy opponents with whom I disagree; they’re just crackpots. Go to a biology conference, read a biology journal, spend time in a biology department; nobody is arguing about the possibility that an ill-specified supernatural “designer” is interfering at whim with the course of evolution. It’s not a serious idea. It may be out there in the public sphere as an idea that garners attention—but, as we all know, that holds true for all sorts of non-serious ideas.
It seems like science snobbery. BHTV has loads of political commentary—and other non-science. It just isn’t remotely like a peer-reviewed science journal.
Since the topic of atheism, morality, and the like often come up here, I would like to point people to the free online book Secular Wholeness by David Cortesi. He approaches the topic of religion by trying to determine what benefits it can provide (community, challenges to improve oneself, easier ethical decisions, etc.), then tries to describe how to achieve these same benefits without resorting to religion. It’s not very heavy on detail but seems very well sourced and with some good pointers on why people choose religions and what they get out of them.
ETA: This could have been a reply to the thread on Scientology, had I seen it before posting.
It’s obviously a counter-conspiracy against my conspiracy to get people to see rationality as something that exists outside of this site. And also to make you look at girls.
Scott Aaronson announced Worldview Manager, “a program that attempts to help users uncover hidden inconsistencies in their personal beliefs”.
You can experiment with it here. The initial topics are Complexity Theory, Strong AI, Axiom of Choice, Quantum Computing, Libertarianism, Quantum Mechanics.
I find it hard to believe that you could really think the most likely explanation of the flaws you perceive are that Aaronson and the students that implemented this purposely introduced flaws and are trying to sabotage the work. So why do you utter such nonsense?
And did it not occur to you that disagreeing that children should have the vote could be resolved by being neutral on everybody having the vote, which is what I did after realizing that there are plausible interpretations under which I would disagree and plausible interpretations under which I would agree.
Whether you consider this as sabotage or not depends on what you think the goal of the site’s authors was. It certainly wasn’t to help find inconsistencies
in people’s thinking, given the obvious effort that went into constructing questions that had multiple conflicting interpretations.
there are plausible interpretations under which I would disagree and plausible interpretations under which I would agree.
I just tried the one for AI and I think its not quite accurate. One of the biggest issues is that I think some of the terms need to be precisely defined and they are not. The other issue I found was that the analysis of my beliefs was not completely accurate because it did not take into account all the answers properly.
I got this conflict between my acceptance of the draft in the unlikely event it would be useful, and my belief that all acts I think the Government should be allowed to do are currently allowed. It doesn’t seem to know of the existence of this supreme court ruling
Interesting link. I played with it for a while. It kept misunderstanding the nuances of my responses, telling me I was wrong when I wasn’t then refusing to listen to my replies. So I stopped playing with it. Two in one day. What are chances?
If the mind is physically independent of the material body but the physical world is closed, empirical observation of the material body cannot be sufficient to determine the existence of a mind.
Versus:
If the program is [abstractly] independent of the [particular] material computer but the physical world is closed, empirical observation of the material computer cannot be sufficient to determine the existence of a running program.
It’s the [...] that hurts. “It is possible for one’s mind to exist outside of one’s material body.” does not imply “the mind is physically independent of the material body”. It’s physically dependent and abstractly independent.
I did have some difficulty resolving all tensions, but I was able to do so. I found that there were often alternate interpretations of a statement that would resolve a tension but were still plausible interpretations. For example, one that I remember was interpreting some of the questions about “physical body” more generally as “physical substrate”. Sometimes the tension page didn’t offer the question that needed reinterpretation, in which case I deferred the tension until I saw a tension that contained the statement to be reinterpreted.
It definitely does need a lot of work, but I can imagine a tool like this having profound effects on people when all the bugs are worked out and it is applied to mind killers and beliefs/habits where cognitive biases figure prominently.
One major thing that needs to be improved if they intend normal people to use it for normal issues like politics, abortion, etc., is to make the tension page much friendlier. Most LWers have probably studied logic, and can pretty easily interpret the tension explanation, but most people have no clue about logic and won’t understand the implicit implications that aren’t explained (like that contrapositive of “A → B” is valid).
I wouldn’t call it a bad implementation for the occasional wrong reported tension, especially if it’s not completely clear for why it reports such a wrong tension. To me, the purpose of the service is not to provide 100% coherent and consistent questionnaire. The idea is that it points to conceptions that might contradict themselves. Whether or not they in fact do contradict should be up to a closer investigation. But merely pointing the user to these possible contradictions should prove to be useful, because it’s so difficult find these inconsistencies by oneself.
It seems clear to me that it will generate some false positives. It will also come up chains of logic that aren’t obviously true or false (because it’s impossible to create statements that are completely free of differing interpretations). Of course, the better the implementation in whole (both the logic system and the sets of statements) the less it will generate these false positives and other inconsistencies, but I do think that it’s impossible to remove them all. Instead, the service should perhaps be considered more like a probing machine.
To claim that it’s a bad implementation sounds to me like it’s not a useful implementation at all. Sure, it’ll probably have a relatively many glitches and bugs, but the above comment doesn’t give any particular evidence that the implementation as such doesn’t work correctly. It seems almost equally likely that such possible inconsistencies are an inherent part of this kind of implementation.
If the implementation would constantly point to tensions that are obviously not real tensions (or useful observations in general), then I’d be more inclined to call it a bad implementation. After all, such claim will discourage people from trying out the service and I don’t see reason for such claim in the example cousin_it gave.
The other common complaint seems to be the lack of precise definitions. Again, I see this more like a feature than a bug. When taking the questionnaire, you take whatever definition you have for the concept and with the service you can find out if your definition leads to inconsistent beliefs.
I wouldn’t call it a bad implementation for occasional wrong reported tension. To me, the purpose of the service is not to provide 100% coherent and consistent questionnaire. The idea is that it points to conceptions that might contradict themselves. Whether or not they in fact do contradict should be up to a closer investigation. But merely pointing the user to these possible contradictions should prove to be useful, because it’s so difficult find these inconsistencies by oneself.
It seems clear to me that it will generate some false positives. It will also come up chains of logic that aren’t obviously true or false (because it’s impossible to create statements that are completely free of differing interpretations). Of course, the better the implementation in whole (both the logic system and the sets of statements) the less it will generate these false positives and other inconsistencies, but I do think that it’s impossible to remove them all. Instead, the service should perhaps be considered more like a probing machine.
To claim that it’s a bad implementation sounds to me like it’s not a useful implementation at all. Sure, it’ll probably have a relatively many glitches and bugs, but the above comment doesn’t give any particular evidence that the implementation as such doesn’t work correctly. It seems almost equally likely that such possible inconsistencies are an inherent part of this kind of implementation.
If the implementation would constantly point to tensions that are obviously not real tensions (or useful observations in general), then I’d be more inclined to call it a bad implementation. After all, such claim will discourage people from trying out the service and I don’t see reason for such claim in the example cousin_it gave.
The other common complaint seems to be the lack of precise definitions. Again, I see this more like a feature than a bug. When taking the questionnaire, you take whatever definition you have for the concept and with the service you can find out if your definition leads to inconsistent beliefs.
Is there an easy way to access the first comment of someone without looking at their comment page and uploading “next” zillions of times?
You’re probably wondering why I would want to do that. I have been motivated occasionally to read someone’s comments from beginning to end, and today I found myself wondering what my first comment was about.
My mind frequently returns to and develops the idea that sometime in the future, a friendly artificial intelligence is going to read Less Wrong. It uploads all the threads and is simultaneously able to (a) read all the comments as they were written chronologically, and (b) keep track of each person’s comments as they chronologically developed and evolved. I probably fantasize about this because I like to analyze patterns and wish I could do that.
The AI would notice all sorts of patterns that I only scantly glimpse...
patterns regarding time of day and frequency of postings; building posteriors about where people live (Europe or the U.S.) and whether they’re a night owl or an early riser or travel across the Atlantic frequently
patterns of speech and the mutual social evolution of these patterns among LWers, yet still being able to tell if people communicate with one another outside LW or if they’re really multiple accounts for the same person
of course, this FAI notices that byrnema has most excellent qualities, etc
Earlier today I was assigning probabilities to whether or not someone on this site was someone I used to date. If they aren’t then I really should introduce them...
Readers of Less Wrong may be interested in this New Scientist article by Noel Sharkey, titled Why AI is a dangerous dream, in which he attacks Kurzweil’s and Moravec’s “fairy tale” predictions and questions whether intelligence is computational (“[the mind] could be a physical system that cannot be recreated by a computer”).
[edit] I thought this would go without saying, but I suspect the downvotes speak otherwise, so: I strongly disagree with the content of this article. I still consider it interesting because it is useful to be aware of differing and potentially popular perspectives on these subjects (and Sharkey is something of a “populist” scientist). I think the opinions it espouses are staggeringly ill-conceived, however.
I strongly disagree. First, on the grounds that LW readers have strong reason to believe this:
(“[the mind] could be a physical system that cannot be recreated by a computer”).
to be false, and so treat it similarly to a proof that 2=1.
But instead of just being a grouch this time, I decided to save you guys the effort and read it myself to see if there’s anything worth reading.
There isn’t. It’s just repetitions of skepticism you’ve already heard, based on Sharkey’s rejection of “the assumption that intelligence is computational” (as opposed to what, and which is different and uncreatable why?), which “It might be, and equally it might not be”.
Other than that, it’s a puff piece interview without much content.
Agreed, I don’t see why the mind isn’t a type of “computer”, and why living organisms aren’t “machines”. If there was something truly different and special about being organic, then we could just build an organic AI. I don’t get the distinction being made.
(Phase 2)
technological artifacts that have no possibility of empathy, compassion or understanding.
Oh: sounds like dualism of some kind if it is impossible for a machine to have empathy, compassion or understanding. Meaning beings with these qualities are more than physical machines, somehow.
(Phase 3)
Reading through some of the comments to the article, it sounds like the objection isn’t that intelligence is necessarily non-physical, but that “computation” doesn’t encompass all possible physical activity. I guess the idea is that if reality is continuous, then there could be some kind of complexity gap between discrete computation and an organic process.
Phases 1-3 is the sequential steps I’ve taken to try to understand this point of view. A view can’t be rejected until its understood...I’m sure people here have considered the AI-is-impossible view before, but I hadn’t.
What is the physical materialist view on whether reality is discrete? (I would guess it’s agnostic.) What is the AI view on whether computations must be discrete? (I would guess AI researchers wouldn’t eschew a continuous computation as as a non-computational thing if it were possible?)
I agree it’s important to apply the principle of charity, but people have to apply the principle of effort too. If Sharkey’s point is about some crucial threshold that continuous systems possess, he should say so. The term “computational” is already taken, so he needs to find another term.
And he can’t be excused on the grounds that “it’s a short interview”, considering that he repeated the same point several times and seemed to find enough space to spell out what (he thinks) his view implies.
“[the mind] could be a physical system that cannot be recreated by a computer”
Let me quote an argument in favor of this, despite the apparently near universal consensus here that it is wrong.
There is a school of thought that says, OK, let’s suppose the mind is a computation, but it is an unsolved problem in philosophy how to determine whether a given physical system implements a given computation. In fact there is even an argument that a clock implements every computation, and it has yet to be conclusively refuted.
If the connection between physical systems and computation is intrinsically uncertain, then we can never say with certainty that two physical systems implement the same computation. In particular, we can never know that a given computer program implements the same computation as a given brain.
Therefore we cannot, in principle, recreate a mind on a computer; at least, not reliably. We can guess that it seems pretty close, but we can never know.
If LessWrongers have solved the problem of determining what counts as instantiating a computation, I’d like to hear more.
If LessWrongers have solved the problem of determining what counts as instantiating a computation, I’d like to hear more.
Sure thing. I solved the problem here and here in response to Paul Almond’s essays on the issue. So did Gary Drescher, who said essentially the same thing in pages 51 through 59 of Good and Real. (I assume you have a copy of it; if not, don’t privately message me and ask me how to pirate it. That’s just wrong, dude. On so many levels.)
I left this comment there:
I thought this might be something like Eliezer’s arguments against developing a GAI until it could be made provably Friendly AI, instead I just got an argument exactly like the ones in 1903 that said heavier than air flight by men was impossible—go back and read some of them, some of the arguments were almost identical. Some of the arguments are currently true, but some of them amount to “I can’t do it, and no one else has done it, therefore there must be some fundamental reason it can’t be done”.
Just found this note in Shalizi’s notebooks which casts an interesting shadow on the Solomonoff prior:
The technical results say that a classification rule is simple if it has a short description, measured in bits. (That is, we are in minimum description length land, or very close to it.) The shorter the description, the tighter the bound on the generalization error. I am happy to agree that this is a reasonable (if language-dependent) way of defining “simplicity” for classifier rules. However, so far as I can tell, this really isn’t what makes the proofs work.
What actually makes the proofs go is the fact that there are not very many binary strings of length m for small m. Finding a classifier with a short description means that you have searched over a small space. It’s the restriction of the search space which really does all the work. It’s not the simplicity of the rules which matters, but the fact that simple rules are scarce in the space of all possible rules. If confines oneself to elaborate, baroque rules, but sticks to a particular style of baroque elaboration, one obtains the same effect.
Here’s my understanding of the dialog, which (as I read it) is not particularly critical of the Solomonoff prior, if that is what cousin_it meant by “casts a shadow”.
(Background knowledge) Shalizi understands “Occam’s Razor” to be something like “In order to reach the truth, among the theories compatible with the evidence, chose the simplest”.
There is a claim that he wishes to refute. The claim is that a certain result is an explanation or proof of Occam’s Razor. The result says that if one finds a simple classification rule which works well in-sample, then it is highly probable that it will continue to work well out-of-sample.
This is a failure of relevance. Occam’s Razor, as Shalizi understands it, is a way of obtaining TRUTH, but the proof only concludes something about GENERALIZATION PERFORMANCE. To illustrate the difference, he points to an example where, in order to increase generalization performance, one might decrease truth.
Shalizi contrasts the algorithmic information theory proof with Kevin T. Kelly’s Ockham Efficiency Theorem, which seems to Shalizi more productive. In particular, Kevin T. Kelly’s formalization does talk about truth rather than generalization performance.
Finally, Shalizi provides an alternative ending to the algorithmic information theory proof. If instead of choosing the simplest classification rule, one chose the simplest rule within a sparse random subset of rules (even a non-computable random subset), then you could still conclude a bound on generalization performance. By providing an alternative ending, he has constructed an alternative proof. Presumably this alternative proof does NOT seem like it is a formalization of Occam’s Razor. Therefore, the interpretation of the original proof as demonstrating some version of Occam’s Razor must also be mistaken.
To sum up, Shalizi is arguing that a certain rhetorical/motivational/interpretational notion which often occurs near a specific proof is wrong. I don’t think he’s concluding anything at all about the Solomonoff prior.
Looks like I almost missed a very interesting discussion. Hope I’m not too late in joining it.
As far as I can tell, you still need the Solomonoff prior for decision making. Kevin T. Kelly’s Ockham Efficiency Theorem says that by using Ockham’s Razor you minimize reversals of opinion prior to finding the true theory, but that seems irrelevant when you have to bet on something, especially since even after you’ve found the truth using Ockham’s Razor, you don’t know that you’ve found it.
Also, I think there’s a flaw in Shalizi’s argument:
Suppose one of these rules correctly classifies all the data.
But if you’re working with a sparse random subset of the rules, why would any of them correctly classify all the data? In algorithmic information theory, the set of rules is universal so one of them is guaranteed to fit the data (assuming the input is computable, which may not be a good assumption but that’s a separate issue).
All good points if the universe has important aspects that are computable, which seems uncontroversial to me. Thanks for the link, I’d lost it sometime ago, that thread is pretty epic.
If confines oneself to elaborate, baroque rules, but sticks to a particular style of baroque elaboration, one obtains the same effect.
The point of Kolmogorov complexity is that there is some limit to how baroque your rules can ever become, or how baroque you can make a fundamentally simple rule by choosing a really weird prior.
In algorithmic information theory, the problem of choosing a prior is equivalent to choosing a particular universal Turing machine. If you pick a really weird Turing machine, you will end up assigning low prior probability to things that “normal” people would consider simple—like a sine wave, for example. But because of universal computation, there’s a limit to how low a probability you can assign. Your weird machine is still universal, so somehow or other it’s got to be able to produce a sine wave, after whatever translation-prefix machinations you have to perform to get it to act like a normal programming language.
Another way of viewing this is just to eschew the notion of generalization and state that the goal of learning is compression. If you do this, you end up doing all the same kinds of work, with many fewer philosophical headaches. Now the great deep problem of picking a prior boils down to the rather more quotidian one of picking a data format to use to transmit/encode data.
But because of universal computation, there’s a limit to how low a probability you can assign.
I don’t understand this claim. Surely we can make the probability of a specific program as low as we want by restricting the programming language in ad hoc ways, while letting it stay Turing complete.
I agree there is some philosophical greyness here. But let me try again.
Let’s say we are adversaries. I am going to choose a data set which I claim is simple, and you are going to try to prove me wrong by picking a weird Turing machine which assigns the data a low probability. I generate my data by taking T samples from a sine wave. You pick some strange Turing machine which is designed to be unable to produce sine waves. But regardless of the choice you make, I can always just crank up T to a high enough value so that the compression rate of the data set is arbitrarily close to 100%, proving its simplicity.
It’s not the simplicity of the rules which matters, but the fact that simple rules are scarce in the space of all possible rules. If confines oneself to elaborate, baroque rules, but sticks to a particular style of baroque elaboration, one obtains the same effect.
But a particular style of baroque elaboration is one that has a short description.
Not necessarily. (Or did I misread your comment?) The particular style can have an arbitrarily long/complex description, and learning will still work as long as the class of described rules is small enough. This observation seems to imply that algorithmic simplicity doesn’t play the central role I’d imagined it to play. This is precisely the point where I’d like to hear LW’s informed replies.
Not necessarily. The particular style can have an arbitrarily long description, and learning will still work as long as the class of described rules is small.
Fixed overheads are ignored in the definition of Kolmogorov complexity.
A “particular style of baroque elaboration” is simply a program through whose eyes certain rules look short, which look elaborate and baroque through the eyes of another program.
The PAC-learning scenario doesn’t have any parameter that goes to infinity, so I’m not sure why you dismiss “fixed” overheads :-)
Once you’ve chosen them, they’re fixed, and don’t run off to infinity.
This definition-only-up-to-a-constant is one of the weaknesses of minimum description length. (The other is its uncomputability. Shalizi somewhere else remarks that in discussions of algorithmic complexity, it is traditional to solemnly take out Kolmogorov complexity, exhibit its theoretical properties, remark on its uncomputability, and put it away again before turning to practical matters.)
This is precisely the point where I’d like to hear LW’s informed replies.
FWIW, this is mine, informed or otherwise. Anyone else have light to shed?
No, let me try nailing this jelly to the wall once again. The definition-only-up-to-a-constant is a weakness of MDL, but this weakness isn’t relevant to my question at all! Even if we had some globally unique variant of MDL derived from some nice mathematical idea, learning theory still doesn’t use description lengths, and would be perfectly happy with rules that have long descriptions as long as we delineate a small set of those rules. To my mind this casts doubt on the importance of MDL.
Consider this alternative characterization. Someone wants to fit a polynomial to some data. They pre-selected a sparse set of polynomials, which are in general ridiculously complex. Against all odds, they get a good fit to the training data. This theorem says that, because they haven’t examined lots and lots of polynomials, they definitely haven’t fallen into the trap of overfitting. Therefore, the good fit to the training data can be expected to generalize to the real data.
Shalizi is saying that this story is fine as far as it goes—it’s just not Occam’s Razor.
Good characterization. It’s worth noting that learning theory never gives any kind of guarantee that you will actually find a function that provides a good fit to the training data, it just tells you that if you do, and the function comes from a low-complexity set, it will probably give good generalization.
learning theory still doesn’t use description lengths, and would be perfectly happy with rules that have long descriptions as long as we delineate a small set of those rules
Any delineation of a small set of rules leads immediately to a short description length for the rules. You just need to encode the index of the rule in the set, costing log(N) bits for a set of size N.
Note that MDL is not the same as algorithmic information theory (definition-up-to-a-constant comes up in AIT, not MDL), though they’re of course related.
You just need to encode the index of the rule in the set, costing log(N) bits for a set of size N.
I see everyone’s assuming that some things, by their nature, always go to infinity (e.g. number of samples) while others stay constant (e.g. rule set). This is a nice convention, but not always realistic—and it certainly wasn’t mentioned in the original formulation of the problem of learning, where everything is finite. If you really want things to grow, why don’t you then allow the set itself to be specified by increasingly convoluted algorithms as N goes to infinity? Like, exponential in N? :-) Learning can still work—in theory, it’ll work just as well as a simple rule set—but you’ll have a hard time explaining that with MDL.
(If there’s some theoretical justification why this kind of outrageous bloatware won’t be as successful as simple algorithms, I’d really like to hear it...)
Johnicolas, thanks. This link and your other comments in those threads are very close to what I’m looking for, though this realization took me some time.
To my mind this casts doubt on the importance of MDL.
I think its uncomputability already does that. When you make a computable version by limiting attention to some framework of descriptive capabilities smaller than universal computation, different choices of that framework will give you different measures of simplicity. What is simple in one framework may seem elaborate and baroque in another. Or as some military strategist once put it:
“To the foot-soldier, the strategy of a general may seem obscure, shrouded in shadows and fog, but to the general himself, his way is as plain as if he were marching his army down a broad, straight highway.”
Any Santa Fe or Albuquerque lesswrong-ers out there, who might want to chat for an hour? I’ll be in Santa Fe for a conference from 9⁄13 to 9⁄17, and am flying in and out of Albuquerque, and will have some free time Sunday 9⁄13.
After some more thought, I realized that making a game with fancy graphics and complex gameplay would probably not be a good idea for a first project to try.
A better idea would be a simple text-based game you play in your browser, probably running on either PHP or Python.
This might not have as much fun appeal as a traditional video game, since it would probably look like a university exam, but it could still be an effective way for us to measure our rationality.
So far I thought of the following ideas:
1) Multiple Choice questions, with only one right answer. Not just a quiz of trivia you memorized from Less Wrong, but applications of the techniques taught here. Also, the standard tests that reveal if you’ve learned how to not fall for the standard biases.
2) Questions that require you to do some math, and enter your answer, which can be scored on how close it is to the correct answer.
3) A “spot the logical fallacy” game, or a “spot the bias” game. The player is presented with a few paragraphs of text. It could be a news article, a section of a short story, a conversation between two people, or maybe something else. The purpose is to spot the first sentence that contains a logical fallacy, or a bias. Once you select which sentence contains the error, you have to say what type of error it is. Maybe typing in the name of the fallacy, or maybe selecting it from a list of the fallacies that have pages about them on the LW Wiki.
We could also implement questions that start with an introduction that you can read before the timer starts ticking, then you proceed to the timed part, where you are scored based on how quickly you answer. Maybe some questions could have a time limit of a few seconds, to force an immediate response.
This LW game could be integrated with the rest of the LW website, where your score is linked to your username. Though this feature is entirely optional. The wiki already has a separate user account not linked directly to your main LW user account, so the game could do the same. The game would keep track of which questions you got right, and how long it took you to answer. This would give speedreaders an advantage, but maybe this isn’t a problem, because speedreading is a generally useful skill to learn.
The game should keep track of which questions you’ve already answered, so it doesn’t ask you the same question again. Or maybe for the questions where you need to do math, it could ask you again with randomized numbers. Maybe there could be a calculator built into the question webpage, with functions for calculating probabilities, etc.
There should also be a way to compare your scores with other players, or see how you rank.
To generate questions for the game, we could use a comments thread in the main LW site, or we could use the LW wiki, or we could set up a special submission form in the game itself.
another random idea: If you give an answer that you think is right, but the game says is wrong, there should be a way to enter your explanation for why your answer is right. Someone would then check these messages, and if your explanation is valid, the game could award you double points.
(Another random idea I had was to make a text adventure game, where you participate in conversations, and sometimes need to interrupt a conversation to point out a logical fallacy, to prevent the conversation from going off-track and preventing you from getting the information you needed from the conversation. But that would be more of a challenge to implement, and generate content for.)
3) A “spot the logical fallacy” game, or a “spot the bias” game. The player is presented with a few paragraphs of text. It could be a news article, a section of a short story, a conversation between two people, or maybe something else. The purpose is to spot the first sentence that contains a logical fallacy, or a bias. Once you select which sentence contains the error, you have to say what type of error it is. Maybe typing in the name of the fallacy, or maybe selecting it from a list of the fallacies that have pages about them on the LW Wiki.
I think the problem is that we lack a good game mechanic. Come up with a mechanic, and the goals can follow, but it’s hard to go from a goal like ‘calibrate yourself to avoid overconfidence’ to a fun game. We need to think about how to borrow games like Zendo and repurpose them.
That LogicTutor site you linked to provides a good, basic introduction to a few concepts and fallacies. However, the practice problems just ask you to identify which fallacy is in the sentence they give you. They’re missing the other half of the game, which is spotting the fallacy in a block of text that’s deliberately designed to hide the fallacy. I’ll keep looking in case someone has already made a game that contains this part.
One way to make the game more fun would be to have interesting text to find the fallacy in. Eliezer’s short stories are a good example of this. Though for the purpose of the game, we would need just short segments of stories, which contain one clear example of a fallacy. Preferably one that’s well-hidden, but obvious once you see it. Also, to keep players on their toes, we could include segments that don’t actually contain a fallacy, and players would have the option of saying that there is no fallacy.
And as I mentioned before, another idea is to flesh out the stories even more, so that it could be expanded into a mystery game, or an adventure game, or an escape-the-room game, where in order to continue you need to talk to people, and some of these people will give inaccurate information, because they didn’t notice a flaw in their own reasoning, and you will need to point out the flaw in their reasoning before they will give you the accurate information.
You would also have to choose your replies during the conversation, and have to choose a reply that doesn’t introduce a new fallacy and send the conversation off in the wrong direction. Many of the possible responses would be to question why the person believes specific things that they just said. Maybe there could also be a feature where you could interrupt the person in the middle of what they’re saying, to point out the problem. Optionally, score the player based on how long they took, and how many wrong paths they went down before finding the correct path.
And if this still doesn’t make the game fun, then there are ways to make people want to play games even if they aren’t fun.
One way is to make the not-so-fun game into a small but necessary part of a bigger game, that is fun.
Another way is to make it into a Flash game, and submit it to a site that tracks your achievements in the games, and gives you an overall score over all the games you’ve played. Examples of sites like this are Kongregate.com, and Newgrounds.com. If you’re lucky, the game might even get to spend a few days on the site’s front page, or get a limited-time challenge, for extra bonus points.
If the game could be made into a series, that would be even better.
So far I’ve only seen one “educational” game get promoted to the front page of Kongragate.com, with its own limited-time-only challenge. That was Globetrotter XL. The game says the name of a city, and you have to click on where that city is, on a map of the world with no border lines or labels. You’re scored for how close you click to the actual city. Unfortunately, I ended up not completing this game’s limited-time challenge, because the game was so unforgiving, and my knowledge of geography was so poor.
Anyway, while it would be a really nice bonus if the spot-the-fallacy game became popular outside LW, the main purpose of the game is to give current LW members an objective way to test their rationality, even if the game isn’t especially fun.
There have been several LW posts now talking about how desperately we need a way to measure our rationality, and so far I haven’t seen any serious proposals that anyone is actually working on. Or maybe there’s a project already started, and I just didn’t notice it because I got so far behind on reading the LW posts.
This game/quiz/whatever is a proposal for a way to go from having no objective way to measure our rationality, to at least having something.
And as I mentioned before, another idea is to flesh out the stories even more, so that it could be expanded into a mystery game, or an adventure game, or an escape-the-room game, where in order to continue you need to talk to people, and some of these people will give inaccurate information, because they didn’t notice a flaw in their own reasoning, and you will need to point out the flaw in their reasoning before they will give you the accurate information.
Many of the possible responses would be to question why the person believes specific things that they just said.
(Starting to think maybe we could use a wiki page, even if only for links and ideas. This game discussion is now spread out over something like 5 LW articles...)
Out of pure curiosity, what’s your probability distribution of Scientology (or some other such group) being useful? Not the Xenu part, but is it possible that they’ve discovered some techniques to make people happier, more successful, etc.?
We already have some limited evidence that conventionally religious people are happier, and conventional religions are quite weak.
We already have some limited evidence that conventionally religious people are happier
But see Will Wilkinson on this too (arguing that this only really holds in the US, and speculating that it’s really about “a good individual fit with prevailing cultural values” rather than religion per se).
That’s a good counter-argument, but the linked post doesn’t actually measure religion-happiness correlation within those other countries (which is the relevant factor), and it’s very plausible that European monopolistic religions are far less effective than American freely competing religions for creating happiness.
The Snoep paper Will linked to measured the correlation for the US, Denmark and the Netherlands (and found no significant correlation in the latter two).
The monopolist religion point is of course a good one. It would be interesting to see what the correlation looked like in relatively secular, yet non-monopolistic countries. (Not really sure what countries would qualify though.)
I’m going to completely ignore “statistical significance”, as scientific papers are well known to have no idea how to do statistics properly with multiple hypotheses, and can be assumed to be doing it wrong until proven otherwise.
If null hypothesis were false, the chance of all almost signs pointing in the same direction would be very low. As far as I can tell what the paper finds out is that religion is less effective in Denmark and Netherlands than in US, but it increases happiness, and it’s extremely unlikely to be a false positive result due to chance.
I know a scientist who’s spent significant money on Landmark Education (nee “est”). He’s happy with what he got out of it, but doesn’t feel the need for any more of it now.
To answer both, there’s no consequence. So I choose not to invent a completely arbitrary prior.
I do enjoy fantasizing about possible measurable consequences of particular types of simulations. Perhaps if I’m interesting enough, I’ll be copied into other simulations; perhaps we can discover some artifact of variably approximate simulation when no important observer is near, etc.
A simulation hypothesis such as “our universe is a simulation” is not falsifiable even given perfect knowledge of the universe at some point in time; maybe the universe has a definite beginning and end and it’s simulated perfectly the whole way through. Therefore, I’ll use the following definition of the simulation hypothesis: “The best description of the universe as we are capable of observing it describes our observations as happening entirely within a simulation crafted by optimizing processes.”
Let’s assume for the sake of convenience that “the” priors for the laws of physics are P, and let’s call the distribution of universes that optimizing processes would simulate P’. The only necessary difference between P and P’ is that P’ is biased toward universes that are easy and/or useful to simulate. How easy a universe is to simulate in general can probably be estimated by how easy a universe is to simulate in itself. We have quantum mechanics but quantum computers have been late in coming, suggesting that our universe would be difficult to simulate. Now, as for utility, evolution optimizes for things that themselves optimize for reproduction, but it also produces optimization for pretty much random things. We can ignore the random things, and ask how useful our universe is for reproduction. I’m guessing that the universe, as it seems to involve lots of pointless computation, is not good for that.
So, given the above, I’d estimate the probability as being… oh, how does 20% sound?
Now, of course, the other thing to look for in a simulated universe is simulation artifacts: things that seem to not follow the laws of physics, and behaviors that are only approximations to how things should behave. Suffice to say, we haven’t seen any of those.
We have quantum mechanics but quantum computers have been late in coming, suggesting that our universe would be difficult to simulate.
Quantum computers are computers which use quantum superposition for parallel processing, and are not required for simulating quantum mechanics. And our “classical” computers do in fact take advantage of quantum mechanics, as classical physics does not allow for solid state transistors.
I consider any evidence that a truly random/spontaneous process occurs is evidence that the universe isn’t closed, because something is happening without an internal mechanism to arbitrate it. And here we have the 2008 Nobel prize in physics, “for the discovery of the mechanism of spontaneous broken symmetry in subatomic physics”.
QM could be interpreted in a deterministic way, but this is not a common view. I would like to learn more about it from you and others here on LW.
“Spontaneous” means that something happens without precursor; without any apparent cause. It is orthogonal in meaning to “determined”.
When you write that spontaneous symmetry breaking is deterministic, perhaps you mean that its description is analytic—wholly described by a set of deterministic mathematical equations?
Spontaneous symmetry breaking is part of stat mech. It has practically nothing to do with QM. Stat mech can be interpreted probabilistically, but it is not at all controversial to apply it to deterministic systems.
“Spontaneous” means that something happens without precursor; without any apparent cause. It is orthogonal in meaning to “determined”.
Maybe that’s a reasonable definition, but you contrasted “spontaneous” with “closed,” which is not orthogonal to “determined.”
My point was that true randomness of any kind would be evidence that a system is not closed. This might be a novel observation (I haven’t heard it before) but I think it is a logical one. It is relevant to reductionism (we wouldn’t want supernatural processes swooping down to make choices for our free processes) and whether we are in a simulation.
Spontaneous symmetry breaking is part of stat mech. [...] Stat mech can be interpreted probabilistically, but it is not at all controversial to apply it to deterministic systems.
When applied to deterministic systems, the spontaneous symmetry breaking isn’t really spontaneous, just apparently so. The idea is that the direction of breaking is determined by the initial conditions, but we may not have enough information about the initial conditions to predict it.
Spontaneous symmetry breaking is part of stat mech. It has practically nothing to do with QM.
It sounds like you want like to argue with whoever is responsible for, “spontaneous symmetry breaking in subatomic physics”. I didn’t mention QM apart from that.
All of your examples count as random events with a collapse postulate, but not with many worlds, and hidden-variables have been formulated both ways.
Based on your past comments, I assume you already know that. Still, since your examples don’t suffice to distinguish interpretations of QM, they also don’t suffice to distinguish a universe with randomness from one without. Or are you just pointing out that we should assign higher probability to randomness than we would have if we hadn’t observed anything that looked like collapse?
I’m not much interested in creating bogus/useless “probability estimates”. The simulation hypothesis I rate, as I do religion, as “false, barring further evidence”. Evidence that the simulation hypothesis is true could be a “physically impossible” inconsistency, like in Heinlein’s story “Them”. If I became convinced that this was a simulation, I’d become a complete hedonist, why bother with anything else when you are completely under the thumb of whatever’s running the simulation.
Stuff with which you interact is part of the rules of the game applied to you. The more generally applicable of these rules you call “physical laws”. Those are the rules that can’t be helped. If you are in the domain of a singleton, then its preference is one of the inescapable laws.
You can analyze the raw content of the laws applied to you, just as you can analyze sensory input, and see patterns such as individual agents making decisions that affect your condition. Maybe such patterns are there, maybe they are not, but the judgment of what to do under the given rules must depend on what exactly those patterns are, not just a fact of “their existence”.
That’s interesting: Vladimir_Nesov chastised me for exactly the same thing a month ago. While I probably do reify physical laws, I’m sure he was just applying a metaphor.
My point in both cases is more that the concept of “existence” is very low on meaningfulness, that you shouldn’t act on mere “existence” of “nonexistence” of something, you must instead understand what that something is.
So, I was thinking about how people conclude stuff. We tend to think of ourselves as having about two levels of conclusion: the “rational” level, which is the level we identify with, considering its conclusions to be our conclusions, and the “emotional” level, which is the one that determines our behavior. (Akrasia is disagreement between the two levels.)
Now, there doesn’t seem to be any obvious rule for what becomes a rational level conclusion. If you go outside and wonder at nature, have you proven that God exists? For some people, it sure feels like it. If you listen to a speech on television and are enchanted by its rhetoric, have you proven that the person making it is a good person? For some people, it sure feels like it. If you open your eyes and see a table, have you proven that there’s a table there? For just about everybody, it sure feels like it. If you perform a mathematical calculation and determine that the expected value of utility is maximized by performing a certain action, have you proven that that is the best action to take? For some people, it sure feels like it.
My first guess as to what’s going on was this: The neural mechanisms converting patterns into beliefs are essentially arbitrary. People learn to ignore those mechanisms that turn out to be wrong; those who end up having to ignore unusually many of these mechanisms end up unintelligent, and those who ignore few but still pay attention to too many wrong ones end up insane. Some people ignore few of their mechanisms but still manage to ignore most of the wrong ones, and these people end up sane and intelligent, but it’s not really possible to determine who these people are, since everyone thinks it’s the people who agree with them.
The plenty of in-group references that are scattered in are almost as good. An excerpt from the backstory:
A group of pro-AI researchers known as the Singularity Foundation (that would later join with other groups to form Firewall in the wake of the Fall) developed the first true seed AIs years before the Fall. Having been heavily involved in the creation of AI and AGIs for many years previously, thanks in large part to their open source AI framework software, the Singularity Foundation’s goal was to generate “friendly AI” by carefully designing AI goal systems.
That group sounds kinda familiar. I wonder if they might be inspired by some real-life organization?
ETA: Oh, and it gets better. Later on, they mention within the same sentence the “Lifeboat Institute” and the “Singularity Foundation” as two organizations that existed before the Singularity. Hmmmmmmmm...
Almost certainly not. After all, that’s not the goal. Edit: After 72 pages I retract my previous two statements.
Bought, downloaded, read intro. Basically it’s Shadowrun IN SPACE! and mixed with Paranoia. A good analogy would be that this tabletop RPG is to Transhumanism what I, Robot the movie is to the book.
In the default story (also known as “campaign setting”), every player character is a “sentinel,” an agent-on-call (or potential recruit) for a shadowy network known as “Firewall.” Firewall is dedicated to counteracting “existential risks”—threats to the existence of transhumanity. These risks can and do include biowar plagues, nanotech swarm outbreaks, nuclear proliferation, terrorists with WMDs, net-breaking computer attacks, rogue AIs, alien encounters, and so on. Firewall isn’t content to simply counteract these threats as they arise, of course, so characters may also be sent on information gathering missions or to put in place pre-emptive or failsafe measures. Characters may be tasked to investigate seemingly innocuous people and places (who turn out not to be), make deals with shady criminal networks (who turn out not to be trustworthy), or travel through a Pandora’s Gate wormhole to analyze the relics of some alien ruin (and see if the threat that killed them is still real). Sentinels are recruited from every faction of transhumanity; those who aren’t ideologically loyal to the cause are hired as mercenaries. These campaigns tend to mix a bit of mystery and investigation with erce bouts of action and combat, also stirring in a nice dose of awe and horror.
In reading the Singularity Institute’s research goals, and the ruminations of Yudkowski, Wei Dai, Nesov et al. in postings here, the approach to developing friendly AI which stands out the most, and from my perspective seems to just always have been the case, seems to be exclusively logic based in the vein of John McCarthy.
I am wondering how the decision was made to focus research for SIAI on the pure logic side, rather than, for example building a synthetic conscious which uses the brain as a model?
To be sure, nearly all AI approaches overlap at some point. However it seems a bit like reinventing the wheel from an almost purely theoretical perspective, when you have a very good model to work from.
Let’s say you’ve got a prototype you want to improve. How do you tell if a proposed change would make it smarter, break it, introduce a subtle cognitive bias, or make the AI want to kill you?
In order to set on limits on the kinds of things an AI will do, you need to understand how it works. You can’t be experimenting on a structure you partially understand, AND be certain that the experiments won’t be fatal.
This is easier when you’ve got a clearly defined structure to the AI, and know how the parts interact, and why.
In order to set on limits on the kinds of things an AI will do, you need to understand how it works.
How is that impossible with a replicated brain architecture? We can’t make one if we don’t know how it works.
This is easier when you’ve got a clearly defined structure to the AI, and know how the parts interact, and why.
Of course. However, how you plan to structure AI what I am asking about. There are many theories about how to structure the AI—so why did the SIAI choose to only focus on a theoretical mathematical logic based approach rather than taking the most advanced, if still flawed, logic device known to man and replicating and improving that?
In order to set on limits on the kinds of things an AI will do, you need to understand how it works.
How is that impossible with a replicated brain architecture? We can’t make one if we don’t know how it works.
If you have the right tools, you can make a brain without understanding it. Reproductive system can make brains. Whole brain emulation doesn’t require understanding of brain architecture, only the dynamics of its lowest-level components.
You “know” how pi works, and how to set up a program that computes it, but you don’t know what its quandrillionth digit is.
Whole brain emulation doesn’t require understanding of brain architecture, only the dynamics of its lowest-level components.
I fear the same philosophical reasoning may be applied to model neural architecture as is currently being used for econometric forecasting. Even the most complex economic models cannot account for significant exogenous variables.
For the record I think we can get to WBE, however I think a premature launch would be terrible. Based on the lack of research into developmental AI (much work notably done by a friend—Dr. Frank Guerin at Aberdeen college) I think there is a long way to go.
Granting that a brain model or WBE, would be as accurate as the biological version, why then would that not be the most efficient method? The problems with testing and implementation are the same as any other AI, if not easier because of familiarity, however it is grounded on specific biological benchmarks which at that point would be immediately identifiable.
I could go on with my particular thoughts as to why biological simulation is in my estimation a better approach, however I am more interested in why the organization (people who have been thinking longer and with more effort than myself) decided otherwise. It would seem that their collective reasoning would give a sufficiently clear and precise answer such that there would be no ambiguity.
Granting that a brain model or WBE, would be as accurate as the biological version, why then would that not be the most efficient method?
You have to instill the right preference, and just having a working improved brain doesn’t give this capability. You are trying to make an overwhelmingly powerful ally; just making something overwhelmingly powerful is a suicide. As CannibalSmith said, brains are not particularly Friendly. Read the paper.
You have to instill the right preference, and just having a working improved brain doesn’t give this capability.
Of course—we have to BUILD IT RIGHT. I couldn’t agree more. The cognitive model does not suggest a mere carbon copy of any particular brain at random, as you know it is not so limited in focus. The fantastic part about the method is at the point in which it is possible to do correctly (not simply an apparent approximation), the tools will likely be available (in the process of it being structured) to correct a large portion of what we identify as fatal cognitive errors. Any errors that are missed it stands to reason would be also missed given the same amount of time with any other developmental structure.
I am familiar with the global risk paper you linked, AI: A modern approach which addresses the issue of cognitive modeling as well as Drescher’s Good and Real and the problems associated with an FAI.
The same existential risks and potential for human disasters are inherent in all AI systems—regardless of the structure, by virtue of it’s “power.” I think one of the draws to this type of development is the fantastic responsibility which comes with it’s development, recognizing and accounting for the catastrophic results that are possible.
That said, I have yet to read a decision theoretic explication as to which structure is an optimized method of development, weighing all known limiting factors. I think AI: A modern approach comes closest to doing this but falls short in that it specifically narrows it’s focus without a thorough comparison of methods. So again, I ask, by what construct has it been determined that a logical symbolic programming approach is optimized?
In other words they are doing it where the light’s better, rather than where they dropped the keys. Given the track record of correctness proofs in comp sci, I don’t think provably Friendly AI is even possible, hopefully I’m wrong there, but all they are doing is further crippling their likelihood of achieving AI before some military or business does.
People control how companies operate—even though they don’t understand all the components involved (in particular the brains).
Any idea that you have to understand all the components of a system in order to exert a high level of control over it thus appears to have dubious foundations.
There are alternative approaches to producing predictable systems which basically involve lots of unit tests.
Testing is in vogue in the software engineering world. Few systems are simple enough to prove much about their behaviour. So: to make sure they behave as they are intended, they are intensively tested. It seems likely that machine intelligence will be no different.
This would be true of any AI. Thus the AI box problem.
It is unclear however, how a formal logic approach overcomes this problem and a replication approach does not. They both will need testing, but as you said the methodology should be reconsidered. The easiest way to change testing methodology for logic would be to improve on current logic methodology which has yielded arguably fantastic results—all done by faulty human brains.
Normally testing is done in an offline “testing” mode—using a test harness or sandbox arrangement. Tests themselves are consequently harmless.
Of course it is possible for the world to present eventualities that are not modelled by the test suite—but that’s usually no big deal.
I don’t think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product.
I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM
will be widely ignored. This isn’t exactly the first time that people have claimed that the world is going to end.
Forget about whether your sandbox is a realistic enough test. There are even questions about how much safety you’re getting from a sandbox. So, we follow your advice, and put the AI in a box in order to test it. And then it escapes anyway, during the test.
The idea that society is smart enough to build machine intelligence, but not smart enough to build a box to test it in does not seem credible to me:
Humans build boxes to put other humans in—and have a high success rate of keeping them inside when they put their minds to it. The few rogue agents that do escape are typically hunted down and imprisoned again. Basically the builders of the box are much stronger and more powerful than what it will contain. Machine intelligence testing seems unlikely to be significantly different from that situation.
The cited “box” scenario discusses the case of weak gatekeepers and powerful escapees. That scenario isn’t very relevant in this case—since we will have smart machines on both sides when restraining intelligent machines in order to test them.
Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.
DOOM is actually big business. Check out all the disaster movies out there. DOOM sells. What could be more important than… THE END OF THE WORLD? What greater cause could there be than… SAVING THE WORLD? So, people buy the DOOM merchandise, contribute the DOOM dollars, and warn their friends about the impending DOOM—thus perpetuating the DOOM virus. That is part of why there have been so many DOOM prophets—DOOM pays.
I would disagree. The overwhelming majority of the average human’s life is spent peacefully. It is actually fairly remarkable how rarely we have significant conflict, especially considering the relatively overcrowded places that humans live. Not to mention that it is only a small proportion of the human population that engages other humans destructively (not by proxy).
The overwhelming majority of the average human’s life is also spent in conditions where they are on relatively even grounds with everyone else. But once you start looking at what happens when people end up in situations where they are clearly more powerful than others? And can treat those others the way they like and without fear of retribution? Ugly.
I disagree. While there are some spectacular examples of what you describe, and they are indeed ugly, by and large there is wide distribution of hierarchical disparity even in daily life which is more often than not mutually beneficent.
As an emperor I optimize my empire by ensuring that my subjects are philosophically and physically satisfied do I not? I think there is plenty of evidence to support this philosophy as the most sustainable (and positive) of hierarchical models; after all some of the most successful businesses are laterally organized.
A certain philosophy being the most sustainable and positive isn’t automatically the same as being the one people tend to adopt. Plus the answer to your question depends on what you’re trying to optimize.
Also, it sounds like you’re still talking about a situation where people don’t actually have ultimate power. If we’re discussing a potential hard takeoff scenario, then considerations such as “which models have been the most successful for businesses before” don’t really apply. Any entity genuinely undergoing a hard takeoff is one that isn’t afterwards bound by what’s successful for humans, any more than we are bound by the practices that work the best for ants.
A certain philosophy being the most sustainable and positive isn’t automatically the same as being the one people tend to adopt
I think there is more than ample evidence to suggest that those are significantly less likely to be adopted—however wouldn’t a group of people who know that and can correct for it be the best test case of implementing an optimized strategy?
Also, it sounds like you’re still talking about a situation where people don’t actually have ultimate power.
I hold the view that it is unnecessary to hold ultimate power over FAI. I certainly wouldn’t bind it to what has worked for humans thus far. Don’t fear the AI, find a way to assimilate.
“But wait – who’s to say that progress will remain “only” exponential? Might not progress exceed this rate, following an inverse polynomial curve (like gravity) or even an inverse exponential curve? I, for one, don’t see why it shouldn’t. If we consider specifically the means whereby the Singularity is most widely expected to occur, namely the development of computers with the capacity for recursive improvement of their own workings,4 I can see no argument why the rate at which such a computer would improve itself should not follow an inverse exponential curve, i.e. one in which the time taken to achieve a given degree of improvement takes time X, the time taken to repeat that degree of improvement is X/2, then X/4 and so on.”
He can’t see the argument? What sort of argument is that?!?
I will spell it out again: to build a fast computer you don’t just need to be able to perform computations quickly. You also need to be able to design and perform real-world experiments and tests. There is no Moore’s law for robotics—and while sensor and motor capabilities are improving, they are not doing so with the same doubling time as exists for computers. Consequently, progress in these fields (and thus overall progress) is correspondingly slower.
I have a couple of intuitions about the structure of human preferences over a large universe.
The first intuition is that your preferences over one part of the universe (or universe-history) should be independent of what happens in another part of the universe, if the “distance” between the two parts is great enough. In other words, if you prefer A happening to B happening in one part of the universe, this preference shouldn’t be reversed no matter what you learn about a distant part of the universe. (“Distance” might be spatial, temporal, or possibly involving other notions like distance between Everett branches.)
The above implies that your preferences can be approximated by partitioning the universe into regions, valuing the regions independently and then summing up the individual values. Errors may occur at borders of such regions, but they can be made arbitrarily small by making the regions large enough.
The second intuition is that your preferences are such that when approximated by such a valuing method, each independent region is valued the same way. In other words, if U(A)>U(B) is true of one region, then the same is true of another region.
There may be better ways of stating or formalizing what I mean (improvements are welcome), but I hope it gets the point across. I’m interested in what others think. If these are reasonable assumptions, perhaps some interesting implications can be derived from them.
P.S. The first intuition directly rules out average utilitarianism. In average utilitarianism, my life might not be considered worth living, just because a bunch of people on the opposite side of the universe happens to have more fun.
Several times recently I asked for simple clarifications about a comment that replied to something I wrote, and had my question ignored. (See here, here, and here.) And I don’t know why. Did I violate some rule of etiquette, or what? How can I rephrase my questions to get a better response rate?
ETA: Here are the questions, in case people don’t want to search through the comments to find them:
But I’m not sure what you mean by “metaethics, a solved problem”. Can you give a link?
What prior work are you referring to, that hasn’t been broadly disseminated?
What is the remaining Problem that you’re referring to?
In my case, what tends to happen is that I either don’t notice the question, or I notice that the question requires a bunch of work to respond to and then either get to it some time later or let it slide off entirely.
I think in cases like these, you’re more likely to get a response by adding another post as a reply to the person with just a single unanswered question (start with the one you care about most), so the person will see they have a new response in their inbox and realize they never answered an earlier question. If you post each of those 3 questions as a response to the person, in context, I’d be very surprised if you didn’t get a response to at least 2 of the 3, as long as you include little to nothing else in each post so it’s obvious what you’re asking for and they can’t respond to something else in the post.
I’ve noticed that longish posts with multiple questions often get just one question answered and all the others ignored, intentionally or unintentionally. And posts that are longish with questions interspersed with non-questions tend to get responded too as if the non-questions were the substantive part, with the questions often ignored.
(The other extremely common reason for not getting a response is identifying a flaw or asking a question that shows problems with the person’s position, in which case most people seem to just ignore the post rather than admit they were wrong or can’t answer a critique. I don’t think that’s the case here at all though.)
This seems like good advice. I did think about repeating the unanswered question, but was worried that I’d come off as obnoxious if the commenter was avoiding it deliberately for some reason. Given the multiple confirmations that that’s probably not the case, I think I’ll do so more often in the future. Thanks.
Not answering a question is Internet’s way of walking away from a conversation. You don’t usually say “excuse me, I’m late to a meeting”, as interaction is asynchronous. In the current Internet culture, saying “I don’t want to bother answering” sounds rude, and so the best solution signaling-wise is to just not answer.
The first two have responses as of the time that you posted this. (If there are remaining questions in a post they are far less likely to be answered after the first couple of replies.)
The final example suffers somewhat from ‘nobody knows what science doesn’t know’. There are probably not too many people who can think of an example of a problem that UDT1 can not handle. For my part I probably wouldn’t answer just because I don’t like the name UDT1 and the language used to describe it irritates me.
I’m not sure why Eleizer didn’t answer but I probably wouldn’t bother wasting thoughts wondering. Want an answer? Make a top level post about it. Include enough of a useful description of the theory and the problems it has already solved to make you not look bad. In particular, include links to said problems and resolutions. If you really want an answer then include an assertion that UDT1 has solved all the significant decision Problems that have been discussed on LessWrong.
Why should I make that much effort to get a simple answer to a simple question? Eliezer obviously had something specific in mind when he wrote “Problem”. Why didn’t he just write a couple of sentences saying what it was when I indicated that I didn’t get the reference?
Same with the other two questions. I wasn’t asking difficult questions, just simple clarifications.
Why should I make that much effort to get a simple answer to a simple question?
I’m not suggesting that you ought to have to. I don’t think you violated any particular etiquette with your requests for clarification. If going meta and questioning whether the lack of reply is justifiable is your preferred use for the effort then by all means do that instead.
I have no idea why Eliezer didn’t answer you. Maybe he was busy. Maybe he was self absorbed. But I do make this observation in general: If someone presents a position along the lines of “it is right to believe there is are Problems’s with X” then I usually don’t expect them to answer me if I press them for an example. This is particularly the case if there are, in fact, no obvious examples. Even if an example could be given, successfully justifying themselves in response to what can be construed as a challenge does not necessarily benefit them. If you present your question in your own frame, however, the dynamics are entirely different.
Concerning Newcomb’s Problem I understand that the dominant position among the regular posters of this site is that you should one-box. This is a position I question.
Suppose Charlie takes on the role of Omega and presents you with Newcomb’s Problem. So far as it is pertinent to the problem Charlie is identical to Omega with the notable exception that his prediction is only %55 likely to be accurate. Should you one-box or two-box in this case?
If you one-box then the expected utility is (.55 1,000,000) $550,000 and if you two-box then it is (.45 1,001,000) $450,450 so it seems you should still one-box even when the prediction is not particularly accurate. Thoughts?
Good question. And with Charlie known to be operating exactly as defined then yes, I would one box. I wouldn’t call him Charlie however as that leads to confusion. The significant problem with dealing with someone who is taking the role of Omega is in my ability to form a prediction about them that is sufficient to justify the ‘cooperate’ response. Once I have that prediction the rest, as you have shown, is just simple math.
I don’t think Newcomb’s Problem can easily be stated as a real (as opposed to a simply logical) problem. Any instance of Newcomb’s problem that you can feasibly construct in the real world it is not a strict one shot problem. I would suggest that optimizing a rational agent for the strictly logical one shot problem one is optimizing for a reality that we don’t exist in.
Even if I am wrong about Newcomb’s problem effectively being an iterated type of problem treating it as if it is seems to solve the dilemma.
Consider this line of reasoning. Omega wants to make the correct prediction. I want Omega to put the million dollars in the box. If I one-box I will either reward Omega for putting the money in the box or punish Omega for not putting the money in the box. Since Omega has a very high success rate I can deduce that Omega puts a high value on making the correct prediction I will therefore put a correspondingly high value on the instrumental value of spending the thousand dollars to influence Omega’s decision. But here’s the thing, this reasoning occurs before Omega even presents you with the problem. It is worked out by Omega running your decision algorithm based on Omega’s scan of your brain. It is effectively the first iteration.
You are then presented with the choice for what is effectively the second time and you deduce that any real Omega (as opposed to some platonic ideal of Omega) does something like the sequence described above in order to generate it’s prediction.
In Charlie’s case you may reason that Charlie either doesn’t care or isn’t able to produce a very accurate prediction and so reason he probably isn’t running your decision algorithm so spending the thousand dollars to try to influence Charlie’s decision has very low instrumental value.
In effect you are not just betting on the probability that the prediction is accurate you are also betting on whether your decision algorithm is affecting the outcome.
I’m not sure how to calculate this but to take a stab at it:
Edit: Removed a misguided attempt at a calculation.
I don’t think Newcomb’s Problem can easily be stated as a real (as opposed to a simply logical) problem.
It can be stated as real in any and every universe that happens to have an omniscient benefactor who is known to be truthful and prone to presenting such scenarios. It’s not real in any other situation. The benefit for optimising a decision making strategy to handle such things as the Newcomb problem is that it is a boundary case. If our decision making breaks down entirely at extreme cases then we can not trust it to be correct.
12 healthy male volunteers were chosen to study what is “just right” amount of beer for driving car. These men consumed doses of beer at 2 bottles, 4 bottles, 8 bottles, and 16 bottles per day for two weeks for each dose amount, with beer being the only alcohol in their diet. Surely the 2 bottles would win, but it definitely ain’t the “just right” amount.
Am I missing something in the sciencedaily news, or did they really end up to that conclusion, of 200mg from that test?
From the abstract: “Twelve healthy male volunteers (aged 53–65 yr) were assigned to consume an intake of successively 200, 400, 800, and 1600 mg/d DHA, as the only {omega}-3 fatty acid, for 2 wk each dose.”
I don’t know what inconsistency you noticed between the news article and the abstract, but it seems the abstract itself describes a study that is missing the control group that gets a dosage of 0.
The following sounds like a control measurement was taken:
“Blood and urine samples were collected before and after each dose of DHA and at 8 wk after arrest of supplementation.”
Also note, that the abstract doesn’t say that 200mg is ideal as the science daily description does it says:
“It is concluded that low consumption of DHA could be an effective and nonpharmacological way to protect healthy men from platelet-related cardiovascular events.”
Taking measurements before and after the treatment is good, but that is not the same as having a separate control group, which could filter out effects of timing, taking the dose with food or water, etc.
Also note, that the abstract doesn’t say that 200mg is ideal as the science daily description does it says:
“It is concluded that low consumption of DHA could be an effective and nonpharmacological way to protect healthy men from platelet-related cardiovascular events.”
The abstract also claims “Therefore, supplementation with only 200 mg/d DHA for 2 wk induced an antioxidant effect.” It is likely that there was a more complete conclusion in the full article.
But the abstract does not make any “just right” claims, unlike the summary on science daily. Which is what you where complaining about.
The abstract reads—we did an incremental test, and even at the lowest dosage we found an effect. This suggest that low dosages could be effective. I don’t see anything wrong with that reasoning.
The science daily summary is simply misrepresenting it. So, the original commenter isn’t missing something in the science news, it is science daily who made the error.
The news article was not based on the abstract. It was based on the journal article (which is available with a subscription) that the abstract summarized. It is not reasonable to expect that every point in the news article be supported by the abstract.
So, perhaps the news article was based on press release that was based on the journal article. My point was that it was not produced solely from the abstract.
While the article is more reliable than the abstract, the abstract is more reliable than the press release and the news coverage, because there is better policing of its claims. And the abstract is more policed than the article, so though it may be less reliable because of compression, it is not biased towards sensationalism.
So, perhaps the news article was based on press release that was based on the journal article. My point was that it was not produced solely from the abstract.
I don’t see why this is your point? In the very least it doesn’t present counter evidence to my claim that the abstract contains information not present in the news article which mitigates or negates the concerns of the original comment.
In the very least it doesn’t present counter evidence to my claim that the abstract contains information not present in the news article which mitigates or negates the concerns of the original comment.
So what? That point was in response to your other claim about what the abstract did not contain.
It’s just that with two distinctly different conclusions from the results mentioned from two different sources: the article authors (in the abstract) and Gerald Weissmann, M.D., Editor-in-Chief (in the news article), I place a much lower confidence in later being a reasonable reading of the research paper.
But of course we could quite safely argue about readings and interpretations indefinitely. I’d point you to Derrida and Hermeneutics if you want to go that route.
In any case, I’ll update my estimates on the likelihood of the research paper having an errant conclusion based on Weismann’s quote, and I suggest you do the same based on the evidence in the abstract—and then I suspect we have little more to discuss on the subject.
In light of how important it is (especially in terms of all the decision theory posts!) to know about Judea Pearl’s causality ideas, I thought I might share something that helped me get up to speed on it: this lecture on Reasoning with Cause and Effect.
For me, it has the right combination of detail and brevity. Other books and papers on Pearlean causality were either too pedantic or too vague about the details, but I learned a lot from the slides, which come with good notes. Anyone know what proceedings papers it refers to though?
Hm, on first browse this does seem like a good intro.
Does anyone feel the occasional temptation to become religious?
I do, sometimes, and push it away each time. I doubt that I could, really, fall for that temptation—even if I tried to, the illogicality of the whole thing would very likely prevent me from really believing in even a part of it very seriously. And as more and more time passes, religious structures begin to seem more and more ridiculous and contrived. Not that I’d have believed that them starting to feel more contrived would even have been possible.
And yet… occasionally I remember the time back in my teens, when I had some sort of a faith. I remember the feeling of ultimate safety it brought with it—the knowledge that no matter what happens, everything will turn out well in the end. It might be a good thing that I spend time worrying over existential risks, and spend time thinking about what I could do about them, but it sure doesn’t exactly improve my mental health. The thought of returning to the mindset of a believer appeals on an emotional level, in the same way stressed adults might longingly remember the carefree days of childhood. But while you can’t become a child again, becoming a believer is at least theoretically possible. And sometimes I do play around with the idea of what it’d be like, to adopt a belief again.
Uh, here is a confession. Twice in the last 6-7 years, at moments of extreme psychological distress, I talked to the God of my Catholic youth. Once I went to an empty church after a series of coincidences (running into two people from my Catholic grade school separately) that I thought was a sign from God. Really, embarrassing, right? It was like my mind segmented and the rationalist was put aside and the devout Catholic school boy was put in charge. The last time this happened was about four years ago.
I don’t know if this could still happen today (my atheism is probably more entrenched now). There have been plenty of periods of distress where this didn’t happen, so I don’t know what triggered it in particular. I think my brain must have really needed a God figure at that moment and didn’t know how to deal with the pain without one so it hacked itself and turned off the rationalist defenses. Or something, it seems so screwed up looking back on it.
Interestingly, I told my theistic-non religious girlfriend about this who in turn told her Christian best friend. Talking to me on the phone for the first time, the friend something along the lines of “I know you say you’re an atheist but B(my girlfriend) told me that you sometimes pray when you’re upset so I know you’re really a good person/God loves you.” In other words, I’m least wrong when I’m thinking least clearly.
On the consolidation of dust specks and the preservation of utilitarian conclusions:
Suppose that you were going to live for at least 3^^^3 seconds. (If you claim that you cannot usefully imagine a lifespan of 3^^^3 seconds or greater, I must insist that you concede that you also cannot usefully imagine a group of 3^^^3 persons. After all, persons are a good deal more complicated than seconds, and you have experienced more seconds than people.)
Suppose that while you are contemplating how to spend your 3^^^3-plus seconds, you are presented with a binary choice: you may spend the next 50 years of this period of time being tortured, or you may spend the next 3^^^3 seconds with a speck of dust in your eye that you cannot get rid of until that time period is up. (Should you succeed in uploading or similar over the course of the next 3^^^3 seconds, the sensation of the speck in the eye will accompany you in the absence of a physical eye until you have waited it out). Assume that after the conclusion of the torture (should you select it), you will be in fine physical health to go on with the rest of your lengthy life, although no guarantees are made for your sanity. Assume that the speck of dust does not impede your vision, and that you will not claw out your eye trying to be rid of it at any time; likewise, no guarantees are made for your sanity.
What selection would you make?
If this choice was actually presented to someone, my guess would be that he would first choose the speck, and then after an extremely long time (i.e. much, much longer than 50 years, giving him a sense of proportion) he would undergo a preference reversal and ask for the torture.
What if the TORTURE occurs during a random time in the next 3^^^3 seconds, not right at the beginning? Also, I think we definitely require a limit on sanity damage because otherwise the scenario is being tortured for 50 years and then spending the next 3^^^3 seconds being insane which Vastly outweighs the ordinary scenario of being tortured for 50 years.
In the original scenario, where just some random person got tortured, no constraints were specified about eir sanity or lifespan post-torture.
I think I did specify that no one would die who would otherwise be immortal; eternal insanity or 3^^^3 years of insanity ought to be implicitly included, I’d think.
This phrase makes the difference for me- the 3^^^3 other people in the original argument weren’t mad- or at least, no more than would have been mad anyway.
Additionally, in your scenario, we have to consider discount rates- it’s certainly conceivable that someone might choose the dust specks over torture now, but be willing to forgo the dust specks in return for torture in 3^^^3 seconds time.
Does it seem likely to you that out of 3^^^3 people chosen with no particular safeguards, not one of them will find a dust speck in the eye to be maddening? It could be the last straw in a string of misfortunes; it could set off some causal chain that will lead to other maddening events, etc.
3^^^3 is such a huge number, some must find it maddening, but the proportion will be a lot lower than the odds that 50 years of torture breaks you mentally.
That’s interesting: I have much, much less hesitation in saying TORTURE to this one. With the original, I can grudgingly concede that I suppose I possibly ought to choose TORTURE, but I still can’t ever quite convince myself to feel that it’s a good answer. This one, I think I can.
Suppose you are given a button that you can press at any time during the 50 years of torture, that will stop the torture (and erase your memory of it if you wish), but you’ll have to live with the dust speck from then on.
I predict that you’ll press the button after actually being tortured for a couple of hours, maybe days, but at most weeks. Even professional spies/soldiers/terrorists who have trained to resist torture end up betraying their cause, so I find it hard to believe that you can hold out for 50 years.
But if you really prefer TORTURE now, that brings up an interesting question: whose preferences are more important, the current you, or the hypothetical future you? It could be argued that the future you is in a better position to decide, since she knows what it actually feels like to be tortured for a significant period of time, whereas you don’t.
But I don’t consider that a knock-down argument, so what do you think? Suppose you can also commit to not pressing the button (say by disabling your arm/hand muscles for 50 years), would you do so?
(This is related to a recent comment by Rolf Andreassen, which I think applies better to this scenario.)
Yes, I think you’re almost certainly right about the button, which thought does indeed put a dent in my lesser hesitation in choosing TORTURE. I think I would definitely not commit to not pressing the button if I were able to “try out” the SPECKS scenario for some short period of time first (say, a week). That way I could make a comparison. Absent that condition… I don’t know. I can’t imagine having the guts to commit to not pressing it.
The fact is that while I certainly don’t see a momentary dust speck as torture, I can easily imagine beginning to see it as torture after a day, never mind 3^^^3 seconds, which is rather longer than 50 years. But I can’t be certain of that, nor of to what extent I would get used to it, nor of how it would compare with much worse kinds of torture. (But then again, there’s a spanner in the works: knowing that you won’t have any lasting physical damage… perhaps that would lend some kind of strength of mind to a torturee?)
Sean Carroll and Carl Zimmer are leaving Bloggingheads, mostly because it’s started playing nice with creationists. Click their names to read their full explanations.
Thanks for the link. I saw the original Behe podcast and was surprised when it suddenly disappeared with a brief note from an admin. I haven’t been able to follow up on this matter much since it hasn’t generated much buzz, but I’ll look forward to reading these links—if people are boycotting Bloggingheads for giving platforms to people whom they don’t like.… well, that’s an interesting precedent to set which should be scrutinized carefully.
Hm. (Reads.) Well… if BHTV has me on, an “anti-accomodationist” as I’ve recently heard my diavlogs called (what a lovely term!), then I don’t think it’s unfair for them to give Behe a slot. It is extremely foolish that they paired him up with a linguist. (Reads.) Looks like Robert Wright agrees with that last part.
You’re one of the last people I would have expected to be concerned with ‘fairness’ when one of the sides is blatant crackpottery. I suppose you wouldn’t have a problem if BHTV invited an astrologer, as long as they paired him up with an astronomer?
What harm is done by bringing on an astrologer? At worst it fail to amuse.
But it’s obvious you’re not talking about the diavlog’s impact on you… you’re concerned with the poor, unwashed masses who might actually be left to form their own opinions from the available information. Well, that’s very nice of you, but I don’t think it’s unreasonable to believe that it’s safe to expose people to views which might be labeled as “crackpottery” by some.
High-status serious people want to associate with high-status others. Allowing crackpots on the same venue dramatically decreases its attractiveness for quality participants.
You can’t really give an elaborate justification for why in this particular case it’s OK, because signals are shallow.
The “unwashed masses”, as you call them, are already getting plenty of exposition to crackpottery, much more than they get to real science, so that a few crackpots on BHTV are barely a drop in the ocean. That’s not what concerns me, no.
What concerns me is that BHTV has a reputation as a respectable website thanks to the participation of respectable academics and experts. It’s reasonable to assume that such a respectable website wouldn’t invite crackpots to promote their brand of crackpottery; in fact, that’s an assumption I made myself until I read Sean’s and Carl’s posts. Inviting crackpots therefore gives the impression that these people should be taken seriously, even if we think they’re wrong.
In any event, your own motives are suspect, to say the least. Characterizing creationist nuts as “people whom [Sean and Carl] don’t like”, as if creationism was merely a distasteful political opinion, or something, makes you sound like a crackpot yourself, or worse, a postmodernist.
So BHTV can’t both enjoy the participation of respectable academics and also host the occasional crackpot? There exists no such universe where the two could possibly coincide? Is there some implicit assumption here that there’s a fixed amount of BHTV episodes, each of which will feature either crackpots or respectable academics? Even if this were so, wouldn’t the reasonable response be to skip over the crackpots rather than avoiding the entire medium? The only justifiable rationale I can see for skipping over BHTV because of this is if you just watched diavlogs at random and having crackpots degraded the signal:noise ratio of the site. But I doubt that you, I, Sean, Carl, or your average Bloggingheads viewer navigates the site in this manner.
Even though I profoundly degree with Behe’s epistemology (and theology), which should go without saying in these parts, I found the debate interesting (I think irreducible complexity is a neat topic), certainly moreso than I’ve enjoyed other diavlogs. Can anyone honestly say that Behe’s presence is less valuable than any other podcast on the website? I doubt it, and thus it strikes me as disingenuous that the unique response his presence generates can be explained away purely through outrage at the notion that somewhere, someone’s time may be wasted.
And I didn’t include the “people whom others don’t like” line as a defense of creationism per se, but as a broader point about silencing views found in contempt. To rip off Will Wilkinson, I’d probably venture to assert that unrepentant Marxists are just as high on the crackpottery scale as Creationists, but I highly doubt we’d see people abandon the site in protest in BHTV hosted some of them. “Respectability” in this context is a tricky term to use, since “respectability” tends to be conferred by social fashions just as much as actual correspondence to whatever virtues we’ve deemed to be worthy of respect. On a more base level, I suspect that many participants in this community have been dismissed as “crackpots” in some context or other before, and are skeptical of the neutrality and intellectual virtues of those who tend to yield the power of the censor. This isn’t a philosophical defense of subjectivism or postmodernism, but an institutional defense of the rough reasons why we don’t just go ahead and burn Behe at the stake.
Should BHTV invite Perez Hilton to debate the fearsome Man Bat? Michael Behe is as credible an author as Pamela Anderson, although not quite as illuminating. I used to think that the worst kind of ignorance was when you knew you were wrong and refused to accept it. Now I think the worst kind is when you know you’re capable of knowing when you’re wrong but refuse to let yourself. Michael Behe wants to be ignorant of his own ignorance. Let him do so in the peace and quiet of his own sad little world.
You’re shifting the goalposts some. I’m not defending the original decision to invite Behe. I’m questioning the notion that inviting Behe is such an egregious offense against BHTV’s “respectability” that it should be boycotted. I wouldn’t boycott BHTV if 90% of the diavlogs were replaced by midget porn, if it meant that I would get the occasional episode of Free Will.
I think Behe’s critics should just admit that what’s really motivating the reaction is the notion that Creationists not only should not be given forums to speak, but those who do grant Creationists forums to speak should be actively identified and boycotted in a way which is reserved for an arguably arbitrarily-defined set of social undesirables. This isn’t an indefensible position, but people have to admit to holding this belief (or some similar belief which is constructed in a more-charitable manner) before a meaningful debate can be enjoined.
[Edit]
Reading over the comments section of the CV posts, it looks like a lot of people are quick to point to Megan McArdle as the political crackpot equivalent of Behe. Should her presence be boycotted too as detrimental to the site? Where should the line be drawn? Where do you actually think the line would be drawn, if not along questionable ideological lines? Why have a line at all?
What critic will not admit that? It’s hardly a fringe opinion in the scientific community that Creationists should not be given forums to speak on the thoroughly unscientific topic of Creationism, and that those who do so and call it science are being absurdly and unnecessarily tolerant. Creationism has never been more or less than an attack on science. It’s extremely toxic, and while I would never try to “silence” anyone, I don’t think it deserves more publicity. I grew up being taught that dreck in a fundamentalist Christian school and I’m more familiar with Behe than I’d care to be. Frankly, he’s an idiot, and his life purpose seems to be toward making more idiots. He doesn’t need anyone’s help.
As for McArdle, I don’t really care. Politics is not a hard science, and while she’s something of a crackpot, she’s not that way because somebody proved her map doesn’t follow the territory. It’s the difference between someone who thinks the earth is flat and someone who thinks it’s run by the Illuminati. The former is just wrong, the latter is just crazy. I don’t mind crazy, because crazy isn’t nearly as dangerous as wrong.
I haven’t seen BhTV endorse Creationism as science in any official capacity.
We been known to shoot us some subjectivists ’round these here parts, y’hear?
Sean Carrol concurs
It seems like science snobbery. BHTV has loads of political commentary—and other non-science. It just isn’t remotely like a peer-reviewed science journal.
PZ Myers says no to BHTV: http://scienceblogs.com/pharyngula/2009/09/phil_plait_ditches_blogginghea.php
Phil Plait pulls the plug on BHTV: http://blogs.discovermagazine.com/badastronomy/2009/09/04/bloggingheads-capo-non-grata/
Robert Wright discusses the whole issue on BHTV:
http://bloggingheads.tv/diavlogs/22300
He says 48 minutes in that the Templeton foundation “were no longer” and that they only supported the site for four months.
However, their ads are still on the sponsored videos beneath:
http://bloggingheads.tv/percontations
...so the Templeton foundation link apparently remains. Maybe there will be no more percontations, though.
Wright gives much the same arguments about ID as I give—in the section on “Viable Intelligent Design Hypotheses”—on:
http://originoflife.net/intelligent_design/
Since the topic of atheism, morality, and the like often come up here, I would like to point people to the free online book Secular Wholeness by David Cortesi. He approaches the topic of religion by trying to determine what benefits it can provide (community, challenges to improve oneself, easier ethical decisions, etc.), then tries to describe how to achieve these same benefits without resorting to religion. It’s not very heavy on detail but seems very well sourced and with some good pointers on why people choose religions and what they get out of them.
ETA: This could have been a reply to the thread on Scientology, had I seen it before posting.
It’s been awhile since this came up. But hay, look, arguments about sex and race and stuff, as applicable to atheism as to rationality in general.
Boo to whoever voted this down in the first place. Great link.
It’s obviously a counter-conspiracy against my conspiracy to get people to see rationality as something that exists outside of this site. And also to make you look at girls.
Scott Aaronson announced Worldview Manager, “a program that attempts to help users uncover hidden inconsistencies in their personal beliefs”.
You can experiment with it here. The initial topics are Complexity Theory, Strong AI, Axiom of Choice, Quantum Computing, Libertarianism, Quantum Mechanics.
Mostly agree is a higher degree of agreement than Agree ?
To Somewhat agree that everyone should have the vote and Disagree that children should have the vote is inconsistent ?
Obviously this is the work of the Skrull “Scott Aaronson”, whose thinking is not so clear.
Also, almost every question is so broken as to make answering it completely futile. So much so that it’s hard to believe it was an accident.
I find it hard to believe that you could really think the most likely explanation of the flaws you perceive are that Aaronson and the students that implemented this purposely introduced flaws and are trying to sabotage the work. So why do you utter such nonsense?
And did it not occur to you that disagreeing that children should have the vote could be resolved by being neutral on everybody having the vote, which is what I did after realizing that there are plausible interpretations under which I would disagree and plausible interpretations under which I would agree.
Whether you consider this as sabotage or not depends on what you think the goal of the site’s authors was. It certainly wasn’t to help find inconsistencies in people’s thinking, given the obvious effort that went into constructing questions that had multiple conflicting interpretations.
Quite.
I just tried the one for AI and I think its not quite accurate. One of the biggest issues is that I think some of the terms need to be precisely defined and they are not. The other issue I found was that the analysis of my beliefs was not completely accurate because it did not take into account all the answers properly.
Its an interesting idea but needs work.
I didn’t find the lack of precise definitions a problem.
I got this conflict between my acceptance of the draft in the unlikely event it would be useful, and my belief that all acts I think the Government should be allowed to do are currently allowed. It doesn’t seem to know of the existence of this supreme court ruling
Interesting link. I played with it for a while. It kept misunderstanding the nuances of my responses, telling me I was wrong when I wasn’t then refusing to listen to my replies. So I stopped playing with it. Two in one day. What are chances?
Good idea, bad implementation. Right now it thinks I have this “tension”, but I’m pretty sure it’s not a tension.
Versus:
It’s the [...] that hurts. “It is possible for one’s mind to exist outside of one’s material body.” does not imply “the mind is physically independent of the material body”. It’s physically dependent and abstractly independent.
I did have some difficulty resolving all tensions, but I was able to do so. I found that there were often alternate interpretations of a statement that would resolve a tension but were still plausible interpretations. For example, one that I remember was interpreting some of the questions about “physical body” more generally as “physical substrate”. Sometimes the tension page didn’t offer the question that needed reinterpretation, in which case I deferred the tension until I saw a tension that contained the statement to be reinterpreted.
It definitely does need a lot of work, but I can imagine a tool like this having profound effects on people when all the bugs are worked out and it is applied to mind killers and beliefs/habits where cognitive biases figure prominently.
One major thing that needs to be improved if they intend normal people to use it for normal issues like politics, abortion, etc., is to make the tension page much friendlier. Most LWers have probably studied logic, and can pretty easily interpret the tension explanation, but most people have no clue about logic and won’t understand the implicit implications that aren’t explained (like that contrapositive of “A → B” is valid).
I wouldn’t call it a bad implementation for the occasional wrong reported tension, especially if it’s not completely clear for why it reports such a wrong tension. To me, the purpose of the service is not to provide 100% coherent and consistent questionnaire. The idea is that it points to conceptions that might contradict themselves. Whether or not they in fact do contradict should be up to a closer investigation. But merely pointing the user to these possible contradictions should prove to be useful, because it’s so difficult find these inconsistencies by oneself.
It seems clear to me that it will generate some false positives. It will also come up chains of logic that aren’t obviously true or false (because it’s impossible to create statements that are completely free of differing interpretations). Of course, the better the implementation in whole (both the logic system and the sets of statements) the less it will generate these false positives and other inconsistencies, but I do think that it’s impossible to remove them all. Instead, the service should perhaps be considered more like a probing machine.
To claim that it’s a bad implementation sounds to me like it’s not a useful implementation at all. Sure, it’ll probably have a relatively many glitches and bugs, but the above comment doesn’t give any particular evidence that the implementation as such doesn’t work correctly. It seems almost equally likely that such possible inconsistencies are an inherent part of this kind of implementation.
If the implementation would constantly point to tensions that are obviously not real tensions (or useful observations in general), then I’d be more inclined to call it a bad implementation. After all, such claim will discourage people from trying out the service and I don’t see reason for such claim in the example cousin_it gave.
The other common complaint seems to be the lack of precise definitions. Again, I see this more like a feature than a bug. When taking the questionnaire, you take whatever definition you have for the concept and with the service you can find out if your definition leads to inconsistent beliefs.
You’re calling it a “bad implementation”, because you think you’ve found a tension that in reality is not a tension?
I wouldn’t call it a bad implementation for occasional wrong reported tension. To me, the purpose of the service is not to provide 100% coherent and consistent questionnaire. The idea is that it points to conceptions that might contradict themselves. Whether or not they in fact do contradict should be up to a closer investigation. But merely pointing the user to these possible contradictions should prove to be useful, because it’s so difficult find these inconsistencies by oneself.
It seems clear to me that it will generate some false positives. It will also come up chains of logic that aren’t obviously true or false (because it’s impossible to create statements that are completely free of differing interpretations). Of course, the better the implementation in whole (both the logic system and the sets of statements) the less it will generate these false positives and other inconsistencies, but I do think that it’s impossible to remove them all. Instead, the service should perhaps be considered more like a probing machine.
To claim that it’s a bad implementation sounds to me like it’s not a useful implementation at all. Sure, it’ll probably have a relatively many glitches and bugs, but the above comment doesn’t give any particular evidence that the implementation as such doesn’t work correctly. It seems almost equally likely that such possible inconsistencies are an inherent part of this kind of implementation.
If the implementation would constantly point to tensions that are obviously not real tensions (or useful observations in general), then I’d be more inclined to call it a bad implementation. After all, such claim will discourage people from trying out the service and I don’t see reason for such claim in the example cousin_it gave.
The other common complaint seems to be the lack of precise definitions. Again, I see this more like a feature than a bug. When taking the questionnaire, you take whatever definition you have for the concept and with the service you can find out if your definition leads to inconsistent beliefs.
Is there an easy way to access the first comment of someone without looking at their comment page and uploading “next” zillions of times?
You’re probably wondering why I would want to do that. I have been motivated occasionally to read someone’s comments from beginning to end, and today I found myself wondering what my first comment was about.
My mind frequently returns to and develops the idea that sometime in the future, a friendly artificial intelligence is going to read Less Wrong. It uploads all the threads and is simultaneously able to (a) read all the comments as they were written chronologically, and (b) keep track of each person’s comments as they chronologically developed and evolved. I probably fantasize about this because I like to analyze patterns and wish I could do that.
The AI would notice all sorts of patterns that I only scantly glimpse...
patterns regarding time of day and frequency of postings; building posteriors about where people live (Europe or the U.S.) and whether they’re a night owl or an early riser or travel across the Atlantic frequently
patterns of speech and the mutual social evolution of these patterns among LWers, yet still being able to tell if people communicate with one another outside LW or if they’re really multiple accounts for the same person
of course, this FAI notices that byrnema has most excellent qualities, etc
What else would the FAI learn?
Does anyone else have this daydream?
Earlier today I was assigning probabilities to whether or not someone on this site was someone I used to date. If they aren’t then I really should introduce them...
Readers of Less Wrong may be interested in this New Scientist article by Noel Sharkey, titled Why AI is a dangerous dream, in which he attacks Kurzweil’s and Moravec’s “fairy tale” predictions and questions whether intelligence is computational (“[the mind] could be a physical system that cannot be recreated by a computer”).
[edit] I thought this would go without saying, but I suspect the downvotes speak otherwise, so: I strongly disagree with the content of this article. I still consider it interesting because it is useful to be aware of differing and potentially popular perspectives on these subjects (and Sharkey is something of a “populist” scientist). I think the opinions it espouses are staggeringly ill-conceived, however.
I strongly disagree. First, on the grounds that LW readers have strong reason to believe this:
to be false, and so treat it similarly to a proof that 2=1.
But instead of just being a grouch this time, I decided to save you guys the effort and read it myself to see if there’s anything worth reading.
There isn’t. It’s just repetitions of skepticism you’ve already heard, based on Sharkey’s rejection of “the assumption that intelligence is computational” (as opposed to what, and which is different and uncreatable why?), which “It might be, and equally it might not be”.
Other than that, it’s a puff piece interview without much content.
(Phase 1)
Agreed, I don’t see why the mind isn’t a type of “computer”, and why living organisms aren’t “machines”. If there was something truly different and special about being organic, then we could just build an organic AI. I don’t get the distinction being made.
(Phase 2)
Oh: sounds like dualism of some kind if it is impossible for a machine to have empathy, compassion or understanding. Meaning beings with these qualities are more than physical machines, somehow.
(Phase 3)
Reading through some of the comments to the article, it sounds like the objection isn’t that intelligence is necessarily non-physical, but that “computation” doesn’t encompass all possible physical activity. I guess the idea is that if reality is continuous, then there could be some kind of complexity gap between discrete computation and an organic process.
Phases 1-3 is the sequential steps I’ve taken to try to understand this point of view. A view can’t be rejected until its understood...I’m sure people here have considered the AI-is-impossible view before, but I hadn’t.
What is the physical materialist view on whether reality is discrete? (I would guess it’s agnostic.) What is the AI view on whether computations must be discrete? (I would guess AI researchers wouldn’t eschew a continuous computation as as a non-computational thing if it were possible?)
I agree it’s important to apply the principle of charity, but people have to apply the principle of effort too. If Sharkey’s point is about some crucial threshold that continuous systems possess, he should say so. The term “computational” is already taken, so he needs to find another term.
And he can’t be excused on the grounds that “it’s a short interview”, considering that he repeated the same point several times and seemed to find enough space to spell out what (he thinks) his view implies.
“[the mind] could be a physical system that cannot be recreated by a computer”
Let me quote an argument in favor of this, despite the apparently near universal consensus here that it is wrong.
There is a school of thought that says, OK, let’s suppose the mind is a computation, but it is an unsolved problem in philosophy how to determine whether a given physical system implements a given computation. In fact there is even an argument that a clock implements every computation, and it has yet to be conclusively refuted.
If the connection between physical systems and computation is intrinsically uncertain, then we can never say with certainty that two physical systems implement the same computation. In particular, we can never know that a given computer program implements the same computation as a given brain.
Therefore we cannot, in principle, recreate a mind on a computer; at least, not reliably. We can guess that it seems pretty close, but we can never know.
If LessWrongers have solved the problem of determining what counts as instantiating a computation, I’d like to hear more.
Sure thing. I solved the problem here and here in response to Paul Almond’s essays on the issue. So did Gary Drescher, who said essentially the same thing in pages 51 through 59 of Good and Real. (I assume you have a copy of it; if not, don’t privately message me and ask me how to pirate it. That’s just wrong, dude. On so many levels.)
This was linked on Hacker News http://news.ycombinator.com/item?id=797871
I left this comment there: I thought this might be something like Eliezer’s arguments against developing a GAI until it could be made provably Friendly AI, instead I just got an argument exactly like the ones in 1903 that said heavier than air flight by men was impossible—go back and read some of them, some of the arguments were almost identical. Some of the arguments are currently true, but some of them amount to “I can’t do it, and no one else has done it, therefore there must be some fundamental reason it can’t be done”.
Just found this note in Shalizi’s notebooks which casts an interesting shadow on the Solomonoff prior:
For context, see the Wikipedia page on PAC learning or this lecture by Scott Aaronson.
Here’s my understanding of the dialog, which (as I read it) is not particularly critical of the Solomonoff prior, if that is what cousin_it meant by “casts a shadow”.
(Background knowledge) Shalizi understands “Occam’s Razor” to be something like “In order to reach the truth, among the theories compatible with the evidence, chose the simplest”.
There is a claim that he wishes to refute. The claim is that a certain result is an explanation or proof of Occam’s Razor. The result says that if one finds a simple classification rule which works well in-sample, then it is highly probable that it will continue to work well out-of-sample.
This is a failure of relevance. Occam’s Razor, as Shalizi understands it, is a way of obtaining TRUTH, but the proof only concludes something about GENERALIZATION PERFORMANCE. To illustrate the difference, he points to an example where, in order to increase generalization performance, one might decrease truth.
Shalizi contrasts the algorithmic information theory proof with Kevin T. Kelly’s Ockham Efficiency Theorem, which seems to Shalizi more productive. In particular, Kevin T. Kelly’s formalization does talk about truth rather than generalization performance.
Finally, Shalizi provides an alternative ending to the algorithmic information theory proof. If instead of choosing the simplest classification rule, one chose the simplest rule within a sparse random subset of rules (even a non-computable random subset), then you could still conclude a bound on generalization performance. By providing an alternative ending, he has constructed an alternative proof. Presumably this alternative proof does NOT seem like it is a formalization of Occam’s Razor. Therefore, the interpretation of the original proof as demonstrating some version of Occam’s Razor must also be mistaken.
To sum up, Shalizi is arguing that a certain rhetorical/motivational/interpretational notion which often occurs near a specific proof is wrong. I don’t think he’s concluding anything at all about the Solomonoff prior.
Looks like I almost missed a very interesting discussion. Hope I’m not too late in joining it.
As far as I can tell, you still need the Solomonoff prior for decision making. Kevin T. Kelly’s Ockham Efficiency Theorem says that by using Ockham’s Razor you minimize reversals of opinion prior to finding the true theory, but that seems irrelevant when you have to bet on something, especially since even after you’ve found the truth using Ockham’s Razor, you don’t know that you’ve found it.
Also, I think there’s a flaw in Shalizi’s argument:
But if you’re working with a sparse random subset of the rules, why would any of them correctly classify all the data? In algorithmic information theory, the set of rules is universal so one of them is guaranteed to fit the data (assuming the input is computable, which may not be a good assumption but that’s a separate issue).
All good points if the universe has important aspects that are computable, which seems uncontroversial to me. Thanks for the link, I’d lost it sometime ago, that thread is pretty epic.
The point of Kolmogorov complexity is that there is some limit to how baroque your rules can ever become, or how baroque you can make a fundamentally simple rule by choosing a really weird prior.
In algorithmic information theory, the problem of choosing a prior is equivalent to choosing a particular universal Turing machine. If you pick a really weird Turing machine, you will end up assigning low prior probability to things that “normal” people would consider simple—like a sine wave, for example. But because of universal computation, there’s a limit to how low a probability you can assign. Your weird machine is still universal, so somehow or other it’s got to be able to produce a sine wave, after whatever translation-prefix machinations you have to perform to get it to act like a normal programming language.
Another way of viewing this is just to eschew the notion of generalization and state that the goal of learning is compression. If you do this, you end up doing all the same kinds of work, with many fewer philosophical headaches. Now the great deep problem of picking a prior boils down to the rather more quotidian one of picking a data format to use to transmit/encode data.
I don’t understand this claim. Surely we can make the probability of a specific program as low as we want by restricting the programming language in ad hoc ways, while letting it stay Turing complete.
I agree there is some philosophical greyness here. But let me try again.
Let’s say we are adversaries. I am going to choose a data set which I claim is simple, and you are going to try to prove me wrong by picking a weird Turing machine which assigns the data a low probability. I generate my data by taking T samples from a sine wave. You pick some strange Turing machine which is designed to be unable to produce sine waves. But regardless of the choice you make, I can always just crank up T to a high enough value so that the compression rate of the data set is arbitrarily close to 100%, proving its simplicity.
cousin_it quoting Shalizi:
But a particular style of baroque elaboration is one that has a short description.
Not necessarily. (Or did I misread your comment?) The particular style can have an arbitrarily long/complex description, and learning will still work as long as the class of described rules is small enough. This observation seems to imply that algorithmic simplicity doesn’t play the central role I’d imagined it to play. This is precisely the point where I’d like to hear LW’s informed replies.
Fixed overheads are ignored in the definition of Kolmogorov complexity.
A “particular style of baroque elaboration” is simply a program through whose eyes certain rules look short, which look elaborate and baroque through the eyes of another program.
The PAC-learning scenario doesn’t have any parameter that goes to infinity, so I’m not sure why you dismiss “fixed” overheads :-)
Once you’ve chosen them, they’re fixed, and don’t run off to infinity.
This definition-only-up-to-a-constant is one of the weaknesses of minimum description length. (The other is its uncomputability. Shalizi somewhere else remarks that in discussions of algorithmic complexity, it is traditional to solemnly take out Kolmogorov complexity, exhibit its theoretical properties, remark on its uncomputability, and put it away again before turning to practical matters.)
FWIW, this is mine, informed or otherwise. Anyone else have light to shed?
No, let me try nailing this jelly to the wall once again. The definition-only-up-to-a-constant is a weakness of MDL, but this weakness isn’t relevant to my question at all! Even if we had some globally unique variant of MDL derived from some nice mathematical idea, learning theory still doesn’t use description lengths, and would be perfectly happy with rules that have long descriptions as long as we delineate a small set of those rules. To my mind this casts doubt on the importance of MDL.
Consider this alternative characterization. Someone wants to fit a polynomial to some data. They pre-selected a sparse set of polynomials, which are in general ridiculously complex. Against all odds, they get a good fit to the training data. This theorem says that, because they haven’t examined lots and lots of polynomials, they definitely haven’t fallen into the trap of overfitting. Therefore, the good fit to the training data can be expected to generalize to the real data.
Shalizi is saying that this story is fine as far as it goes—it’s just not Occam’s Razor.
Good characterization. It’s worth noting that learning theory never gives any kind of guarantee that you will actually find a function that provides a good fit to the training data, it just tells you that if you do, and the function comes from a low-complexity set, it will probably give good generalization.
Any delineation of a small set of rules leads immediately to a short description length for the rules. You just need to encode the index of the rule in the set, costing log(N) bits for a set of size N.
Note that MDL is not the same as algorithmic information theory (definition-up-to-a-constant comes up in AIT, not MDL), though they’re of course related.
I see everyone’s assuming that some things, by their nature, always go to infinity (e.g. number of samples) while others stay constant (e.g. rule set). This is a nice convention, but not always realistic—and it certainly wasn’t mentioned in the original formulation of the problem of learning, where everything is finite. If you really want things to grow, why don’t you then allow the set itself to be specified by increasingly convoluted algorithms as N goes to infinity? Like, exponential in N? :-) Learning can still work—in theory, it’ll work just as well as a simple rule set—but you’ll have a hard time explaining that with MDL.
(If there’s some theoretical justification why this kind of outrageous bloatware won’t be as successful as simple algorithms, I’d really like to hear it...)
You might find what you’re looking for in Kevin T. Kelly’s work:
http://www.andrew.cmu.edu/user/kk3n/ockham/Ockham.htm
Dr. Shalizi mentioned it as an alternative formalization of Occam’s Razor.
Johnicolas, thanks. This link and your other comments in those threads are very close to what I’m looking for, though this realization took me some time.
I think its uncomputability already does that. When you make a computable version by limiting attention to some framework of descriptive capabilities smaller than universal computation, different choices of that framework will give you different measures of simplicity. What is simple in one framework may seem elaborate and baroque in another. Or as some military strategist once put it:
“To the foot-soldier, the strategy of a general may seem obscure, shrouded in shadows and fog, but to the general himself, his way is as plain as if he were marching his army down a broad, straight highway.”
Any Santa Fe or Albuquerque lesswrong-ers out there, who might want to chat for an hour? I’ll be in Santa Fe for a conference from 9⁄13 to 9⁄17, and am flying in and out of Albuquerque, and will have some free time Sunday 9⁄13.
I’m going to use this open thread to once again suggest the idea of a Less Wrong video game.
( Here’s a link to the post I made last month about it )
After some more thought, I realized that making a game with fancy graphics and complex gameplay would probably not be a good idea for a first project to try.
A better idea would be a simple text-based game you play in your browser, probably running on either PHP or Python.
This might not have as much fun appeal as a traditional video game, since it would probably look like a university exam, but it could still be an effective way for us to measure our rationality.
So far I thought of the following ideas:
1) Multiple Choice questions, with only one right answer. Not just a quiz of trivia you memorized from Less Wrong, but applications of the techniques taught here. Also, the standard tests that reveal if you’ve learned how to not fall for the standard biases.
2) Questions that require you to do some math, and enter your answer, which can be scored on how close it is to the correct answer.
3) A “spot the logical fallacy” game, or a “spot the bias” game. The player is presented with a few paragraphs of text. It could be a news article, a section of a short story, a conversation between two people, or maybe something else. The purpose is to spot the first sentence that contains a logical fallacy, or a bias. Once you select which sentence contains the error, you have to say what type of error it is. Maybe typing in the name of the fallacy, or maybe selecting it from a list of the fallacies that have pages about them on the LW Wiki.
We could also implement questions that start with an introduction that you can read before the timer starts ticking, then you proceed to the timed part, where you are scored based on how quickly you answer. Maybe some questions could have a time limit of a few seconds, to force an immediate response.
This LW game could be integrated with the rest of the LW website, where your score is linked to your username. Though this feature is entirely optional. The wiki already has a separate user account not linked directly to your main LW user account, so the game could do the same. The game would keep track of which questions you got right, and how long it took you to answer. This would give speedreaders an advantage, but maybe this isn’t a problem, because speedreading is a generally useful skill to learn.
The game should keep track of which questions you’ve already answered, so it doesn’t ask you the same question again. Or maybe for the questions where you need to do math, it could ask you again with randomized numbers. Maybe there could be a calculator built into the question webpage, with functions for calculating probabilities, etc.
There should also be a way to compare your scores with other players, or see how you rank.
To generate questions for the game, we could use a comments thread in the main LW site, or we could use the LW wiki, or we could set up a special submission form in the game itself.
another random idea: If you give an answer that you think is right, but the game says is wrong, there should be a way to enter your explanation for why your answer is right. Someone would then check these messages, and if your explanation is valid, the game could award you double points.
(Another random idea I had was to make a text adventure game, where you participate in conversations, and sometimes need to interrupt a conversation to point out a logical fallacy, to prevent the conversation from going off-track and preventing you from getting the information you needed from the conversation. But that would be more of a challenge to implement, and generate content for.)
This could be used to make a game based off of Dungeons and Discourse.
When you attack, you have to select an argument without a flaw, or it gets blocked. When the opponent attacks, if you find a flaw, it deals no damage.
This has already been done many times as part of critical thinking courses; people don’t use free sites like http://www.wwnorton.com/college/phil/logic3/ because they’re boring and hard.
I think the problem is that we lack a good game mechanic. Come up with a mechanic, and the goals can follow, but it’s hard to go from a goal like ‘calibrate yourself to avoid overconfidence’ to a fun game. We need to think about how to borrow games like Zendo and repurpose them.
Thanks for the link.
That LogicTutor site you linked to provides a good, basic introduction to a few concepts and fallacies. However, the practice problems just ask you to identify which fallacy is in the sentence they give you. They’re missing the other half of the game, which is spotting the fallacy in a block of text that’s deliberately designed to hide the fallacy. I’ll keep looking in case someone has already made a game that contains this part.
One way to make the game more fun would be to have interesting text to find the fallacy in. Eliezer’s short stories are a good example of this. Though for the purpose of the game, we would need just short segments of stories, which contain one clear example of a fallacy. Preferably one that’s well-hidden, but obvious once you see it. Also, to keep players on their toes, we could include segments that don’t actually contain a fallacy, and players would have the option of saying that there is no fallacy.
And as I mentioned before, another idea is to flesh out the stories even more, so that it could be expanded into a mystery game, or an adventure game, or an escape-the-room game, where in order to continue you need to talk to people, and some of these people will give inaccurate information, because they didn’t notice a flaw in their own reasoning, and you will need to point out the flaw in their reasoning before they will give you the accurate information.
You would also have to choose your replies during the conversation, and have to choose a reply that doesn’t introduce a new fallacy and send the conversation off in the wrong direction. Many of the possible responses would be to question why the person believes specific things that they just said. Maybe there could also be a feature where you could interrupt the person in the middle of what they’re saying, to point out the problem. Optionally, score the player based on how long they took, and how many wrong paths they went down before finding the correct path.
And if this still doesn’t make the game fun, then there are ways to make people want to play games even if they aren’t fun.
One way is to make the not-so-fun game into a small but necessary part of a bigger game, that is fun.
Another way is to make it into a Flash game, and submit it to a site that tracks your achievements in the games, and gives you an overall score over all the games you’ve played. Examples of sites like this are Kongregate.com, and Newgrounds.com. If you’re lucky, the game might even get to spend a few days on the site’s front page, or get a limited-time challenge, for extra bonus points.
If the game could be made into a series, that would be even better.
So far I’ve only seen one “educational” game get promoted to the front page of Kongragate.com, with its own limited-time-only challenge. That was Globetrotter XL. The game says the name of a city, and you have to click on where that city is, on a map of the world with no border lines or labels. You’re scored for how close you click to the actual city. Unfortunately, I ended up not completing this game’s limited-time challenge, because the game was so unforgiving, and my knowledge of geography was so poor.
Anyway, while it would be a really nice bonus if the spot-the-fallacy game became popular outside LW, the main purpose of the game is to give current LW members an objective way to test their rationality, even if the game isn’t especially fun.
There have been several LW posts now talking about how desperately we need a way to measure our rationality, and so far I haven’t seen any serious proposals that anyone is actually working on. Or maybe there’s a project already started, and I just didn’t notice it because I got so far behind on reading the LW posts.
This game/quiz/whatever is a proposal for a way to go from having no objective way to measure our rationality, to at least having something.
Well that would be an interesting game mechanic.
How about a courtroom drama?
I object!
(Alternate Higurashi ending: /me claws throat open to get at insects on me)
Incidentally, another link right up your alley: http://projects.csail.mit.edu/worldview/about
(Starting to think maybe we could use a wiki page, even if only for links and ideas. This game discussion is now spread out over something like 5 LW articles...)
5 articles? I only know of 2 articles where this topic was discussed. Can you post a link to the other 3, please?
Anyway, I went ahead and created a wiki page for discussing the game idea, and posted links to the two threads I know of:
http://wiki.lesswrong.com/wiki/The_Less_Wrong_Video_Game
It certainly feels like it’s been spread out over more than 2 open threads!
Out of pure curiosity, what’s your probability distribution of Scientology (or some other such group) being useful? Not the Xenu part, but is it possible that they’ve discovered some techniques to make people happier, more successful, etc.?
We already have some limited evidence that conventionally religious people are happier, and conventional religions are quite weak.
But see Will Wilkinson on this too (arguing that this only really holds in the US, and speculating that it’s really about “a good individual fit with prevailing cultural values” rather than religion per se).
That’s a good counter-argument, but the linked post doesn’t actually measure religion-happiness correlation within those other countries (which is the relevant factor), and it’s very plausible that European monopolistic religions are far less effective than American freely competing religions for creating happiness.
The Snoep paper Will linked to measured the correlation for the US, Denmark and the Netherlands (and found no significant correlation in the latter two).
The monopolist religion point is of course a good one. It would be interesting to see what the correlation looked like in relatively secular, yet non-monopolistic countries. (Not really sure what countries would qualify though.)
I’m going to completely ignore “statistical significance”, as scientific papers are well known to have no idea how to do statistics properly with multiple hypotheses, and can be assumed to be doing it wrong until proven otherwise.
If null hypothesis were false, the chance of all almost signs pointing in the same direction would be very low. As far as I can tell what the paper finds out is that religion is less effective in Denmark and Netherlands than in US, but it increases happiness, and it’s extremely unlikely to be a false positive result due to chance.
I know a scientist who’s spent significant money on Landmark Education (nee “est”). He’s happy with what he got out of it, but doesn’t feel the need for any more of it now.
I’m curious about how Less Wrong readers would answer these questions:
What is your probability estimate for some form of the simulation hypothesis being true?
If you received evidence that changed your estimate to be much higher (or lower), what would you do differently in your life?
To answer both, there’s no consequence. So I choose not to invent a completely arbitrary prior.
I do enjoy fantasizing about possible measurable consequences of particular types of simulations. Perhaps if I’m interesting enough, I’ll be copied into other simulations; perhaps we can discover some artifact of variably approximate simulation when no important observer is near, etc.
A simulation hypothesis such as “our universe is a simulation” is not falsifiable even given perfect knowledge of the universe at some point in time; maybe the universe has a definite beginning and end and it’s simulated perfectly the whole way through. Therefore, I’ll use the following definition of the simulation hypothesis: “The best description of the universe as we are capable of observing it describes our observations as happening entirely within a simulation crafted by optimizing processes.”
Let’s assume for the sake of convenience that “the” priors for the laws of physics are P, and let’s call the distribution of universes that optimizing processes would simulate P’. The only necessary difference between P and P’ is that P’ is biased toward universes that are easy and/or useful to simulate. How easy a universe is to simulate in general can probably be estimated by how easy a universe is to simulate in itself. We have quantum mechanics but quantum computers have been late in coming, suggesting that our universe would be difficult to simulate. Now, as for utility, evolution optimizes for things that themselves optimize for reproduction, but it also produces optimization for pretty much random things. We can ignore the random things, and ask how useful our universe is for reproduction. I’m guessing that the universe, as it seems to involve lots of pointless computation, is not good for that.
So, given the above, I’d estimate the probability as being… oh, how does 20% sound?
Now, of course, the other thing to look for in a simulated universe is simulation artifacts: things that seem to not follow the laws of physics, and behaviors that are only approximations to how things should behave. Suffice to say, we haven’t seen any of those.
Quantum computers are computers which use quantum superposition for parallel processing, and are not required for simulating quantum mechanics. And our “classical” computers do in fact take advantage of quantum mechanics, as classical physics does not allow for solid state transistors.
I don’t understand what point you’re trying to make here, but classical physics allows for mechanical computers.
It seems that quantum computers are required for simulating quantum mechanics in sub-exponential time, though.
When discussing asymptotic algorithmic complexity, you should specify the varying parameter of problem complexity.
The usual default parameter is number of bits it takes to write down the problem. It could also be number of particles. Either one works in this case.
What quantum algorithm for simulating quantum mechanics takes sub-exponential time with respect to the number of particles?
I didn’t have a particular algorithm in mind when I said that, but since you ask I went and found this one.
I consider any evidence that a truly random/spontaneous process occurs is evidence that the universe isn’t closed, because something is happening without an internal mechanism to arbitrate it. And here we have the 2008 Nobel prize in physics, “for the discovery of the mechanism of spontaneous broken symmetry in subatomic physics”.
I do not think that phrase means what you think it means.
I thought it meant what you linked to, and after checking I’m pretty sure that was what the prize was about.
So what do you think about the possibility of a physical mechanism being able to make a free choice?
Perhaps some better examples:
spontaneous creation of particles in a vacuum
spontaneous particle decay
QM is deterministic. Spontaneous symmetry breaking also occurs in stat mech, which applies to deterministic classical systems.
QM could be interpreted in a deterministic way, but this is not a common view. I would like to learn more about it from you and others here on LW.
“Spontaneous” means that something happens without precursor; without any apparent cause. It is orthogonal in meaning to “determined”.
When you write that spontaneous symmetry breaking is deterministic, perhaps you mean that its description is analytic—wholly described by a set of deterministic mathematical equations?
Spontaneous symmetry breaking is part of stat mech. It has practically nothing to do with QM. Stat mech can be interpreted probabilistically, but it is not at all controversial to apply it to deterministic systems.
Maybe that’s a reasonable definition, but you contrasted “spontaneous” with “closed,” which is not orthogonal to “determined.”
My point was that true randomness of any kind would be evidence that a system is not closed. This might be a novel observation (I haven’t heard it before) but I think it is a logical one. It is relevant to reductionism (we wouldn’t want supernatural processes swooping down to make choices for our free processes) and whether we are in a simulation.
When applied to deterministic systems, the spontaneous symmetry breaking isn’t really spontaneous, just apparently so. The idea is that the direction of breaking is determined by the initial conditions, but we may not have enough information about the initial conditions to predict it.
It sounds like you want like to argue with whoever is responsible for, “spontaneous symmetry breaking in subatomic physics”. I didn’t mention QM apart from that.
All of your examples count as random events with a collapse postulate, but not with many worlds, and hidden-variables have been formulated both ways.
Based on your past comments, I assume you already know that. Still, since your examples don’t suffice to distinguish interpretations of QM, they also don’t suffice to distinguish a universe with randomness from one without. Or are you just pointing out that we should assign higher probability to randomness than we would have if we hadn’t observed anything that looked like collapse?
I’m not much interested in creating bogus/useless “probability estimates”. The simulation hypothesis I rate, as I do religion, as “false, barring further evidence”. Evidence that the simulation hypothesis is true could be a “physically impossible” inconsistency, like in Heinlein’s story “Them”. If I became convinced that this was a simulation, I’d become a complete hedonist, why bother with anything else when you are completely under the thumb of whatever’s running the simulation.
You are completely under the thumb of the physical laws.
Sorry, but that is a reification of “physical laws”; physical laws aren’t a thing, they are simply our description of “how things work”.
Stuff with which you interact is part of the rules of the game applied to you. The more generally applicable of these rules you call “physical laws”. Those are the rules that can’t be helped. If you are in the domain of a singleton, then its preference is one of the inescapable laws.
You can analyze the raw content of the laws applied to you, just as you can analyze sensory input, and see patterns such as individual agents making decisions that affect your condition. Maybe such patterns are there, maybe they are not, but the judgment of what to do under the given rules must depend on what exactly those patterns are, not just a fact of “their existence”.
That’s interesting: Vladimir_Nesov chastised me for exactly the same thing a month ago. While I probably do reify physical laws, I’m sure he was just applying a metaphor.
My point in both cases is more that the concept of “existence” is very low on meaningfulness, that you shouldn’t act on mere “existence” of “nonexistence” of something, you must instead understand what that something is.
So, I was thinking about how people conclude stuff. We tend to think of ourselves as having about two levels of conclusion: the “rational” level, which is the level we identify with, considering its conclusions to be our conclusions, and the “emotional” level, which is the one that determines our behavior. (Akrasia is disagreement between the two levels.)
Now, there doesn’t seem to be any obvious rule for what becomes a rational level conclusion. If you go outside and wonder at nature, have you proven that God exists? For some people, it sure feels like it. If you listen to a speech on television and are enchanted by its rhetoric, have you proven that the person making it is a good person? For some people, it sure feels like it. If you open your eyes and see a table, have you proven that there’s a table there? For just about everybody, it sure feels like it. If you perform a mathematical calculation and determine that the expected value of utility is maximized by performing a certain action, have you proven that that is the best action to take? For some people, it sure feels like it.
My first guess as to what’s going on was this: The neural mechanisms converting patterns into beliefs are essentially arbitrary. People learn to ignore those mechanisms that turn out to be wrong; those who end up having to ignore unusually many of these mechanisms end up unintelligent, and those who ignore few but still pay attention to too many wrong ones end up insane. Some people ignore few of their mechanisms but still manage to ignore most of the wrong ones, and these people end up sane and intelligent, but it’s not really possible to determine who these people are, since everyone thinks it’s the people who agree with them.
There is an audio interview with EY on “Make A Public Commitment”.
Eclipse Phase
Does it deserve a top level post?
I see it tries to push the right buttons, but does it contain any novel insights?
The plenty of in-group references that are scattered in are almost as good. An excerpt from the backstory:
That group sounds kinda familiar. I wonder if they might be inspired by some real-life organization?
ETA: Oh, and it gets better. Later on, they mention within the same sentence the “Lifeboat Institute” and the “Singularity Foundation” as two organizations that existed before the Singularity. Hmmmmmmmm...
Almost certainly not. After all, that’s not the goal. Edit: After 72 pages I retract my previous two statements.
Bought, downloaded, read intro. Basically it’s Shadowrun IN SPACE! and mixed with Paranoia. A good analogy would be that this tabletop RPG is to Transhumanism what I, Robot the movie is to the book.
Not really related to rationality, so no top-post if you ask me, but it’s a very good setting and an excellent product.
In reading the Singularity Institute’s research goals, and the ruminations of Yudkowski, Wei Dai, Nesov et al. in postings here, the approach to developing friendly AI which stands out the most, and from my perspective seems to just always have been the case, seems to be exclusively logic based in the vein of John McCarthy.
I am wondering how the decision was made to focus research for SIAI on the pure logic side, rather than, for example building a synthetic conscious which uses the brain as a model?
To be sure, nearly all AI approaches overlap at some point. However it seems a bit like reinventing the wheel from an almost purely theoretical perspective, when you have a very good model to work from.
Such a design would be harder to reason about.
Let’s say you’ve got a prototype you want to improve. How do you tell if a proposed change would make it smarter, break it, introduce a subtle cognitive bias, or make the AI want to kill you?
In order to set on limits on the kinds of things an AI will do, you need to understand how it works. You can’t be experimenting on a structure you partially understand, AND be certain that the experiments won’t be fatal.
This is easier when you’ve got a clearly defined structure to the AI, and know how the parts interact, and why.
How is that impossible with a replicated brain architecture? We can’t make one if we don’t know how it works.
Of course. However, how you plan to structure AI what I am asking about. There are many theories about how to structure the AI—so why did the SIAI choose to only focus on a theoretical mathematical logic based approach rather than taking the most advanced, if still flawed, logic device known to man and replicating and improving that?
If you have the right tools, you can make a brain without understanding it. Reproductive system can make brains. Whole brain emulation doesn’t require understanding of brain architecture, only the dynamics of its lowest-level components.
You “know” how pi works, and how to set up a program that computes it, but you don’t know what its quandrillionth digit is.
I fear the same philosophical reasoning may be applied to model neural architecture as is currently being used for econometric forecasting. Even the most complex economic models cannot account for significant exogenous variables.
For the record I think we can get to WBE, however I think a premature launch would be terrible. Based on the lack of research into developmental AI (much work notably done by a friend—Dr. Frank Guerin at Aberdeen college) I think there is a long way to go.
Granting that a brain model or WBE, would be as accurate as the biological version, why then would that not be the most efficient method? The problems with testing and implementation are the same as any other AI, if not easier because of familiarity, however it is grounded on specific biological benchmarks which at that point would be immediately identifiable.
I could go on with my particular thoughts as to why biological simulation is in my estimation a better approach, however I am more interested in why the organization (people who have been thinking longer and with more effort than myself) decided otherwise. It would seem that their collective reasoning would give a sufficiently clear and precise answer such that there would be no ambiguity.
You have to instill the right preference, and just having a working improved brain doesn’t give this capability. You are trying to make an overwhelmingly powerful ally; just making something overwhelmingly powerful is a suicide. As CannibalSmith said, brains are not particularly Friendly. Read the paper.
Of course—we have to BUILD IT RIGHT. I couldn’t agree more. The cognitive model does not suggest a mere carbon copy of any particular brain at random, as you know it is not so limited in focus. The fantastic part about the method is at the point in which it is possible to do correctly (not simply an apparent approximation), the tools will likely be available (in the process of it being structured) to correct a large portion of what we identify as fatal cognitive errors. Any errors that are missed it stands to reason would be also missed given the same amount of time with any other developmental structure.
I am familiar with the global risk paper you linked, AI: A modern approach which addresses the issue of cognitive modeling as well as Drescher’s Good and Real and the problems associated with an FAI.
The same existential risks and potential for human disasters are inherent in all AI systems—regardless of the structure, by virtue of it’s “power.” I think one of the draws to this type of development is the fantastic responsibility which comes with it’s development, recognizing and accounting for the catastrophic results that are possible.
That said, I have yet to read a decision theoretic explication as to which structure is an optimized method of development, weighing all known limiting factors. I think AI: A modern approach comes closest to doing this but falls short in that it specifically narrows it’s focus without a thorough comparison of methods. So again, I ask, by what construct has it been determined that a logical symbolic programming approach is optimized?
In other words they are doing it where the light’s better, rather than where they dropped the keys. Given the track record of correctness proofs in comp sci, I don’t think provably Friendly AI is even possible, hopefully I’m wrong there, but all they are doing is further crippling their likelihood of achieving AI before some military or business does.
People control how companies operate—even though they don’t understand all the components involved (in particular the brains).
Any idea that you have to understand all the components of a system in order to exert a high level of control over it thus appears to have dubious foundations.
There are alternative approaches to producing predictable systems which basically involve lots of unit tests.
Testing is in vogue in the software engineering world. Few systems are simple enough to prove much about their behaviour. So: to make sure they behave as they are intended, they are intensively tested. It seems likely that machine intelligence will be no different.
When a failed test destroys the world, applicability of the normally very useful testing methodology should be reconsidered.
This would be true of any AI. Thus the AI box problem.
It is unclear however, how a formal logic approach overcomes this problem and a replication approach does not. They both will need testing, but as you said the methodology should be reconsidered. The easiest way to change testing methodology for logic would be to improve on current logic methodology which has yielded arguably fantastic results—all done by faulty human brains.
Normally testing is done in an offline “testing” mode—using a test harness or sandbox arrangement. Tests themselves are consequently harmless.
Of course it is possible for the world to present eventualities that are not modelled by the test suite—but that’s usually no big deal.
I don’t think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product.
I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM will be widely ignored. This isn’t exactly the first time that people have claimed that the world is going to end.
Forget about whether your sandbox is a realistic enough test. There are even questions about how much safety you’re getting from a sandbox. So, we follow your advice, and put the AI in a box in order to test it. And then it escapes anyway, during the test.
That doesn’t seem like a reliable plan.
The idea that society is smart enough to build machine intelligence, but not smart enough to build a box to test it in does not seem credible to me:
Humans build boxes to put other humans in—and have a high success rate of keeping them inside when they put their minds to it. The few rogue agents that do escape are typically hunted down and imprisoned again. Basically the builders of the box are much stronger and more powerful than what it will contain. Machine intelligence testing seems unlikely to be significantly different from that situation.
The cited “box” scenario discusses the case of weak gatekeepers and powerful escapees. That scenario isn’t very relevant in this case—since we will have smart machines on both sides when restraining intelligent machines in order to test them.
Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.
DOOM is actually big business. Check out all the disaster movies out there. DOOM sells. What could be more important than… THE END OF THE WORLD? What greater cause could there be than… SAVING THE WORLD? So, people buy the DOOM merchandise, contribute the DOOM dollars, and warn their friends about the impending DOOM—thus perpetuating the DOOM virus. That is part of why there have been so many DOOM prophets—DOOM pays.
The more I think about this, the more it seems incorrect.
Human brains are not particularly friendly.
I would disagree. The overwhelming majority of the average human’s life is spent peacefully. It is actually fairly remarkable how rarely we have significant conflict, especially considering the relatively overcrowded places that humans live. Not to mention that it is only a small proportion of the human population that engages other humans destructively (not by proxy).
The overwhelming majority of the average human’s life is also spent in conditions where they are on relatively even grounds with everyone else. But once you start looking at what happens when people end up in situations where they are clearly more powerful than others? And can treat those others the way they like and without fear of retribution? Ugly.
I disagree. While there are some spectacular examples of what you describe, and they are indeed ugly, by and large there is wide distribution of hierarchical disparity even in daily life which is more often than not mutually beneficent.
As an emperor I optimize my empire by ensuring that my subjects are philosophically and physically satisfied do I not? I think there is plenty of evidence to support this philosophy as the most sustainable (and positive) of hierarchical models; after all some of the most successful businesses are laterally organized.
A certain philosophy being the most sustainable and positive isn’t automatically the same as being the one people tend to adopt. Plus the answer to your question depends on what you’re trying to optimize.
Also, it sounds like you’re still talking about a situation where people don’t actually have ultimate power. If we’re discussing a potential hard takeoff scenario, then considerations such as “which models have been the most successful for businesses before” don’t really apply. Any entity genuinely undergoing a hard takeoff is one that isn’t afterwards bound by what’s successful for humans, any more than we are bound by the practices that work the best for ants.
I think there is more than ample evidence to suggest that those are significantly less likely to be adopted—however wouldn’t a group of people who know that and can correct for it be the best test case of implementing an optimized strategy?
I hold the view that it is unnecessary to hold ultimate power over FAI. I certainly wouldn’t bind it to what has worked for humans thus far. Don’t fear the AI, find a way to assimilate.
I have at least one other legacy identity here, dating from the old OB days, “mitchell_porter2”. Is there some way to merge us?
To quote from http://www.sens.org/files/sens/FHTI07-deGrey.pdf:
“But wait – who’s to say that progress will remain “only” exponential? Might not progress exceed this rate, following an inverse polynomial curve (like gravity) or even an inverse exponential curve? I, for one, don’t see why it shouldn’t. If we consider specifically the means whereby the Singularity is most widely expected to occur, namely the development of computers with the capacity for recursive improvement of their own workings,4 I can see no argument why the rate at which such a computer would improve itself should not follow an inverse exponential curve, i.e. one in which the time taken to achieve a given degree of improvement takes time X, the time taken to repeat that degree of improvement is X/2, then X/4 and so on.”
He can’t see the argument? What sort of argument is that?!?
I will spell it out again: to build a fast computer you don’t just need to be able to perform computations quickly. You also need to be able to design and perform real-world experiments and tests. There is no Moore’s law for robotics—and while sensor and motor capabilities are improving, they are not doing so with the same doubling time as exists for computers. Consequently, progress in these fields (and thus overall progress) is correspondingly slower.
Came across this: What We Can Learn About Pricing From Menu Engineers. Probably nothing new to you/us. Summary: Judgements of acceptable prices are strongly influenced by other seen prices.
I have a couple of intuitions about the structure of human preferences over a large universe.
The first intuition is that your preferences over one part of the universe (or universe-history) should be independent of what happens in another part of the universe, if the “distance” between the two parts is great enough. In other words, if you prefer A happening to B happening in one part of the universe, this preference shouldn’t be reversed no matter what you learn about a distant part of the universe. (“Distance” might be spatial, temporal, or possibly involving other notions like distance between Everett branches.)
The above implies that your preferences can be approximated by partitioning the universe into regions, valuing the regions independently and then summing up the individual values. Errors may occur at borders of such regions, but they can be made arbitrarily small by making the regions large enough.
The second intuition is that your preferences are such that when approximated by such a valuing method, each independent region is valued the same way. In other words, if U(A)>U(B) is true of one region, then the same is true of another region.
There may be better ways of stating or formalizing what I mean (improvements are welcome), but I hope it gets the point across. I’m interested in what others think. If these are reasonable assumptions, perhaps some interesting implications can be derived from them.
P.S. The first intuition directly rules out average utilitarianism. In average utilitarianism, my life might not be considered worth living, just because a bunch of people on the opposite side of the universe happens to have more fun.
Several times recently I asked for simple clarifications about a comment that replied to something I wrote, and had my question ignored. (See here, here, and here.) And I don’t know why. Did I violate some rule of etiquette, or what? How can I rephrase my questions to get a better response rate?
ETA: Here are the questions, in case people don’t want to search through the comments to find them:
But I’m not sure what you mean by “metaethics, a solved problem”. Can you give a link?
What prior work are you referring to, that hasn’t been broadly disseminated?
What is the remaining Problem that you’re referring to?
In my case, what tends to happen is that I either don’t notice the question, or I notice that the question requires a bunch of work to respond to and then either get to it some time later or let it slide off entirely.
“or I notice that the question requires a bunch of work to respond to and then either get to it some time later or let it slide off entirely”
Or don’t recall it.
I think in cases like these, you’re more likely to get a response by adding another post as a reply to the person with just a single unanswered question (start with the one you care about most), so the person will see they have a new response in their inbox and realize they never answered an earlier question. If you post each of those 3 questions as a response to the person, in context, I’d be very surprised if you didn’t get a response to at least 2 of the 3, as long as you include little to nothing else in each post so it’s obvious what you’re asking for and they can’t respond to something else in the post.
I’ve noticed that longish posts with multiple questions often get just one question answered and all the others ignored, intentionally or unintentionally. And posts that are longish with questions interspersed with non-questions tend to get responded too as if the non-questions were the substantive part, with the questions often ignored.
(The other extremely common reason for not getting a response is identifying a flaw or asking a question that shows problems with the person’s position, in which case most people seem to just ignore the post rather than admit they were wrong or can’t answer a critique. I don’t think that’s the case here at all though.)
This seems like good advice. I did think about repeating the unanswered question, but was worried that I’d come off as obnoxious if the commenter was avoiding it deliberately for some reason. Given the multiple confirmations that that’s probably not the case, I think I’ll do so more often in the future. Thanks.
Not answering a question is Internet’s way of walking away from a conversation. You don’t usually say “excuse me, I’m late to a meeting”, as interaction is asynchronous. In the current Internet culture, saying “I don’t want to bother answering” sounds rude, and so the best solution signaling-wise is to just not answer.
In my experience (for myself and interacting with others), it’s not uncommon at all for someone to
miss that there was a question at all
see a question, but put off thinking about it and then forget to get back to it
reply to some other part and forget about answering the question
figure replying to the question is uninteresting ‘because the thread is dead’
etc.
The first two have responses as of the time that you posted this. (If there are remaining questions in a post they are far less likely to be answered after the first couple of replies.)
The final example suffers somewhat from ‘nobody knows what science doesn’t know’. There are probably not too many people who can think of an example of a problem that UDT1 can not handle. For my part I probably wouldn’t answer just because I don’t like the name UDT1 and the language used to describe it irritates me.
I’m not sure why Eleizer didn’t answer but I probably wouldn’t bother wasting thoughts wondering. Want an answer? Make a top level post about it. Include enough of a useful description of the theory and the problems it has already solved to make you not look bad. In particular, include links to said problems and resolutions. If you really want an answer then include an assertion that UDT1 has solved all the significant decision Problems that have been discussed on LessWrong.
Please expand on that.
Why should I make that much effort to get a simple answer to a simple question? Eliezer obviously had something specific in mind when he wrote “Problem”. Why didn’t he just write a couple of sentences saying what it was when I indicated that I didn’t get the reference?
Same with the other two questions. I wasn’t asking difficult questions, just simple clarifications.
I’m not suggesting that you ought to have to. I don’t think you violated any particular etiquette with your requests for clarification. If going meta and questioning whether the lack of reply is justifiable is your preferred use for the effort then by all means do that instead.
I have no idea why Eliezer didn’t answer you. Maybe he was busy. Maybe he was self absorbed. But I do make this observation in general: If someone presents a position along the lines of “it is right to believe there is are Problems’s with X” then I usually don’t expect them to answer me if I press them for an example. This is particularly the case if there are, in fact, no obvious examples. Even if an example could be given, successfully justifying themselves in response to what can be construed as a challenge does not necessarily benefit them. If you present your question in your own frame, however, the dynamics are entirely different.
Has Eliezer written anything good about the evolution of morality? It should probably go on a wiki page titled “Evolution of morality”.
ETA: While I’m at it, how about reasons people are religious?
Concerning Newcomb’s Problem I understand that the dominant position among the regular posters of this site is that you should one-box. This is a position I question.
Suppose Charlie takes on the role of Omega and presents you with Newcomb’s Problem. So far as it is pertinent to the problem Charlie is identical to Omega with the notable exception that his prediction is only %55 likely to be accurate. Should you one-box or two-box in this case?
If you one-box then the expected utility is (.55 1,000,000) $550,000 and if you two-box then it is (.45 1,001,000) $450,450 so it seems you should still one-box even when the prediction is not particularly accurate. Thoughts?
Good question. And with Charlie known to be operating exactly as defined then yes, I would one box. I wouldn’t call him Charlie however as that leads to confusion. The significant problem with dealing with someone who is taking the role of Omega is in my ability to form a prediction about them that is sufficient to justify the ‘cooperate’ response. Once I have that prediction the rest, as you have shown, is just simple math.
I don’t think Newcomb’s Problem can easily be stated as a real (as opposed to a simply logical) problem. Any instance of Newcomb’s problem that you can feasibly construct in the real world it is not a strict one shot problem. I would suggest that optimizing a rational agent for the strictly logical one shot problem one is optimizing for a reality that we don’t exist in.
Even if I am wrong about Newcomb’s problem effectively being an iterated type of problem treating it as if it is seems to solve the dilemma.
Consider this line of reasoning. Omega wants to make the correct prediction. I want Omega to put the million dollars in the box. If I one-box I will either reward Omega for putting the money in the box or punish Omega for not putting the money in the box. Since Omega has a very high success rate I can deduce that Omega puts a high value on making the correct prediction I will therefore put a correspondingly high value on the instrumental value of spending the thousand dollars to influence Omega’s decision. But here’s the thing, this reasoning occurs before Omega even presents you with the problem. It is worked out by Omega running your decision algorithm based on Omega’s scan of your brain. It is effectively the first iteration.
You are then presented with the choice for what is effectively the second time and you deduce that any real Omega (as opposed to some platonic ideal of Omega) does something like the sequence described above in order to generate it’s prediction.
In Charlie’s case you may reason that Charlie either doesn’t care or isn’t able to produce a very accurate prediction and so reason he probably isn’t running your decision algorithm so spending the thousand dollars to try to influence Charlie’s decision has very low instrumental value.
In effect you are not just betting on the probability that the prediction is accurate you are also betting on whether your decision algorithm is affecting the outcome.
I’m not sure how to calculate this but to take a stab at it:
Edit: Removed a misguided attempt at a calculation.
It can be stated as real in any and every universe that happens to have an omniscient benefactor who is known to be truthful and prone to presenting such scenarios. It’s not real in any other situation. The benefit for optimising a decision making strategy to handle such things as the Newcomb problem is that it is a boundary case. If our decision making breaks down entirely at extreme cases then we can not trust it to be correct.
Does this http://www.sciencedaily.com/releases/2009/08/090831130751.htm suffer same problem as:
12 healthy male volunteers were chosen to study what is “just right” amount of beer for driving car. These men consumed doses of beer at 2 bottles, 4 bottles, 8 bottles, and 16 bottles per day for two weeks for each dose amount, with beer being the only alcohol in their diet. Surely the 2 bottles would win, but it definitely ain’t the “just right” amount.
Am I missing something in the sciencedaily news, or did they really end up to that conclusion, of 200mg from that test?
Well, the article abstract isn’t consistent with the description you linked to. One of the dangers of paraphrasing science.
From the abstract: “Twelve healthy male volunteers (aged 53–65 yr) were assigned to consume an intake of successively 200, 400, 800, and 1600 mg/d DHA, as the only {omega}-3 fatty acid, for 2 wk each dose.”
I don’t know what inconsistency you noticed between the news article and the abstract, but it seems the abstract itself describes a study that is missing the control group that gets a dosage of 0.
The following sounds like a control measurement was taken:
“Blood and urine samples were collected before and after each dose of DHA and at 8 wk after arrest of supplementation.”
Also note, that the abstract doesn’t say that 200mg is ideal as the science daily description does it says:
“It is concluded that low consumption of DHA could be an effective and nonpharmacological way to protect healthy men from platelet-related cardiovascular events.”
Taking measurements before and after the treatment is good, but that is not the same as having a separate control group, which could filter out effects of timing, taking the dose with food or water, etc.
The abstract also claims “Therefore, supplementation with only 200 mg/d DHA for 2 wk induced an antioxidant effect.” It is likely that there was a more complete conclusion in the full article.
But the abstract does not make any “just right” claims, unlike the summary on science daily. Which is what you where complaining about.
The abstract reads—we did an incremental test, and even at the lowest dosage we found an effect. This suggest that low dosages could be effective. I don’t see anything wrong with that reasoning.
The science daily summary is simply misrepresenting it. So, the original commenter isn’t missing something in the science news, it is science daily who made the error.
The news article was not based on the abstract. It was based on the journal article (which is available with a subscription) that the abstract summarized. It is not reasonable to expect that every point in the news article be supported by the abstract.
extremely implausible, as a general rule.
ETA:
So, perhaps the news article was based on press release that was based on the journal article. My point was that it was not produced solely from the abstract.
While the article is more reliable than the abstract, the abstract is more reliable than the press release and the news coverage, because there is better policing of its claims. And the abstract is more policed than the article, so though it may be less reliable because of compression, it is not biased towards sensationalism.
I don’t see why this is your point? In the very least it doesn’t present counter evidence to my claim that the abstract contains information not present in the news article which mitigates or negates the concerns of the original comment.
So what? That point was in response to your other claim about what the abstract did not contain.
It’s just that with two distinctly different conclusions from the results mentioned from two different sources: the article authors (in the abstract) and Gerald Weissmann, M.D., Editor-in-Chief (in the news article), I place a much lower confidence in later being a reasonable reading of the research paper.
But of course we could quite safely argue about readings and interpretations indefinitely. I’d point you to Derrida and Hermeneutics if you want to go that route.
In any case, I’ll update my estimates on the likelihood of the research paper having an errant conclusion based on Weismann’s quote, and I suggest you do the same based on the evidence in the abstract—and then I suspect we have little more to discuss on the subject.
If noone was using computers today, how would you convince people to use them?