Polled. I check Discussion almost every day on a desktop, but almost never comment (though I plan to do so more often). I occasionally click through links but I prefer LW posts because of the community aspect. Ideas reliably get a meaningful response from intelligent readers with a common background, and the author of the post is always part of the discussion. For the same reason, I don’t think any other rationalist discussion site has the potential of LW. Although I wasn’t here back before the Diaspora, I do think a revival is in order—centralization may have its drawbacks, but the existence of this website, even now, gives us a baseline both for what has been explored in the past and what topics are currently open/unresolved.
dogiv
I don’t really understand when this would be useful. Is it for an oracle that you don’t trust, or that otherwise has no incentive to explain things to you? Because in that case, nothing here constrains the explanation (or the questions) to be related to reality. It could just choose to explain something that is easy for the human to understand, and then the questions will all be answered correctly. If you do trust the oracle, the second AI is unnecessary—the first one could just ask questions of the human to confirm understanding, like any student-teacher dynamic. What am I missing here?
Thank you, this is clearer than it was before, and it does seem like a potentially useful technique. I see a couple of limitations:
First, it still seems that the whole plan rests on having a good selection of questions, and the mechanism for choosing them is unclear. If they are chosen by some structured method that thoroughly covers the AI’s representation of the prior, the questions asked of the human are unlikely to capture the most important aspects of the update from new evidence. Most of the differences between the prior and the posterior could be insignificant from a human perspective, and so even if the human “understands” the posterior a broad sense they will not be likely to have the answers to all of these. Even if they can figure out those answers correctly, it does not necessarily test whether they are aware of the differences that are most important.
Second, the requirement for the two AIs to have a common prior, and differ only by some known quantum of new evidence, seems like it might restrict the applications considerably. In simple cases you might handle this by “rolling back” a copy of the first AI to a time when it had not yet processed the new evidence, and making that the starting point for the second AI. But if the processing of the evidence occurred before some other update that you want included in the prior, then you would need some way of working backward to a state that never previously existed.
I agree there’s something to the exploration-exploitation view of people becoming more closed-minded. But don’t be too quick to write it off as “people don’t think carefully anymore”, or simple tribalism. Some important questions really do get settled by all those late-night college debates, though often the answer is “I don’t think it’s possible to know this” or “It’s not worth the years of effort it would take to understand at a more-than-amateur level.”
People are recognizing their limitations and zeroing in on the areas where they can get the highest return on investment for their thoughts. That’s a difficult thing to do when you’re younger, because you don’t have much to compare yourself to. If you’ve never met a physicist more knowledgeable than your 9th-grade science teacher, you might well think you can make big contributions to the theory of relativity in the space of a few weeks’ discussion with your friends.
Similarly, when it comes to politics, the idea of considering every idea with an open mind can fall victim to the pressures of reality—some ideas are superficially appealing but actually harmful; some are nice in theory but are so far from what could reasonably be implemented that their return on investment is low. And because politics is so adversarial, many ideas that are promoted as novel and non-partisan are actually trying to sneak in a not-so-novel agenda through the back door.
The attempt to analytically model the recalcitrance of Bayesian inference is an interesting idea, but I’m afraid it leaves out some of the key points. Reasoning is not just repeated applications of Bayes’ theorem. If it were, everyone would be equally smart except for processing speed and data availability. Rather, the key element is in coming up with good approximations for P(D|H) when data and memory are severely limited. This skill relies on much more than a fast processor, including things like simple but accurate models of the rest of the world, or knowing the correct algorithms to combine various truths into logical conclusions.
Some of it does fall into the category of having the correct prior beliefs, but they are hardly “accidentally gifted”—learning the correct priors, either from experience with data or through optimization “in a box” is a critical aspect of becoming intellectually capable. So the recalcitrance of prediction, though it clearly does eventually go to infinity in the absence of new data, is not obviously high. I would add also that for your argument against the intelligence explosion to hold, the recalcitrance of prediction would have to be not just “predictably high” but would need to increase at least linearly with intelligence in the range of interest—a very different claim, and one for which you have given little support.
I do think it’s likely that strictly limiting access to data would slow down an intelligence explosion. Bostrom argues that a “hardware overhang” could be exploited for a fast takeoff, but historically, advanced AI projects like AlphaGo or Watson have used state-of-the-art hardware during development, and this seems probable in the future as well. Data overhang, on the other hand, would be nearly impossible to avoid if the budding intelligence is given access to the internet, of which it can process only a small fraction in any reasonable amount of time.
Hi Jared, Your question about vegetarianism is an interesting one, and I’ll give a couple of responses because I’m not sure exactly what direction you’re coming from.
I think there’s a strong rationalist argument in favor of limiting consumption of meat, especially red meat, on both health and environmental grounds. These issues get more mixed when you look at moderate consumption of chicken or fish. Fish especially is the best available source of healthy fats, so leaving it out entirely is a big trade-off, and the environmental impact of fishing varies a great deal by species, wild vs. farmed, and even the fishing method. Veganism gives relatively small environmental gains over vegetarianism, and is generally considered a loss in terms of health.
When you look at animal suffering, things get a lot more speculative. Clearly you can’t treat a chicken’s suffering the same as a human’s, but how many chickens does it take to be equivalent to a human? At what point is a chicken’s life not worth living? This quickly bogs down in questions of the repugnant conclusion, a standard paradox in utilitarianism. Although I have seen no thorough analysis of the topic, my sense is that 1) Scaling of moral value is probably more-than-linear with brain mass (that is, you are worth more than the ~300 chickens it would take to equal your gray matter) but I can’t be much more precise than that 2) Most of the world’s neurons are in wild inverterbrates: http://reflectivedisequilibrium.blogspot.com/2013/09/how-is-brain-mass-distributed-among.html which argues against focusing specially on domesticated vertebrates 3) Effort expended to reduce animal suffering is largely self-contained—that is, if you choose not to eat a chicken, you probably reduce the number of factory-farmed chickens by about one, with no longer-term effects. Effort to help humans, on the other hand, often has a difficult-to-estimate multiplier from follow-on effects. See here for more on this argument: http://globalprioritiesproject.org/2014/06/human-and-animal-interventions/
The upshot is that when you make any significant investment in animal welfare, including vegetarianism and especially veganism, you should consider the opportunity costs. If it makes your life more difficult and reduces the amount of good you can do in other ways, it may not be worth it.
Personally, I used to be a pescetarian and would consider doing so again, depending on the people around me. Trying to do it in my current circumstances would cause more hassle than I think it’s worth (having to ask people for separate meals, not participating in group activities, etc). If you know a lot of other vegetarians, there may be no social cost or even some social benefit. But don’t assume that’s the case for everyone.
It seems like the key problem described here is that coalitions of rational people, when they form around scientific propositions, cause the group to become non-scientific out of desire to support the coalition. The example that springs to my mind is climate change, where there is social pressure for scientific-minded people (or even those who just approve of science) to back the rather specific policy of reducing greenhouse gas emissions rather than to probe other aspects of the problem or potential solutions and adaptations.
I wonder if we might solve problems like this by substituting some rational principle that is not subject to re-evaluation. Ultimate goals (CEV, or the like) would fit the bill in principle, but in practice, even if enough people could agree on them, I suspect they are too vague and remote to form a coalition around. The EA movement may be closer to succeeding, where the key idea is not an ultimate goal but rather the general technique of quantitatively evaluating opportunities to achieve altruistic objectives in general. Still, it’s difficult to extend a coalition like that to a broader population, since most people can’t easily identify with it.
Perhaps the middle ground is to start with a goal that is controversial enough to distinguish coalition members from outsiders, but too vague to form a strong coalition around—say, aggregative consequentialism or something. Then find a clear practical implication of the goal that has the necessary emotional impact. As long as the secondary goal follows easily enough from the first goal that it won’t need to be re-evaluated later on, the coalition can hold together and make progress toward the original goal without much danger of becoming irrational. Can’t think of a good example for the sub-goal, though.
I would argue that the closest real-world analogue is computer hacking. It is a rare ability, but it can bestow a large amount of power on an individual who puts in enough effort and skill. Like magic, it requires almost no help from anyone else. The infrastructure has to be there, but since the infrastructure isn’t designed to allow hacking, having the infrastructure doesn’t make the ability available to everyone who can pay (like, say, airplanes). If you look at the more fantasy-style sci-fi, science is often treated like magic—one smart scientist can do all sorts of cool stuff on their own. But it’s never plausible. With hacking, that romanticization isn’t nearly as far from reality.
I haven’t seen any feminists addressing that particular argument (most are concerned with cultural issues rather than genetic ones) but my initial sense is something like this: a successful feminist society would have 1) education and birth control easily available to all women, and 2) a roughly equal division of the burden of child-rearing between men and women. These changes will remove most of the current incentives that seem likely to cause a lower birth rate among feminists than non-feminists. Of course, it could remain true that feminists tend to be more educated, more independent, less traditional, etc—traits that might correlate with reduced desire for children. However, I suspect we already have that issue (for both men and women) entirely separately from feminism. Some highly-educated countries try to increase fertility with tax incentives and ad campaigns (Denmark, for instance) but I’m not sure how successful it is. In the end the only good solution to such Malthusian problems may be genetic engineering.
The idea that friendly superintelligence would be massively useful is implicit (and often explicit) in nearly every argument in favor of AI safety efforts, certainly including EY and Bostrom. But you seem to be making the much stronger claim that we should therefore altruistically expend effort to accelerate its development. I am not convinced.
Your argument rests on the proposition that current research on AI is so specific that its contribution toward human-level AI is very small, so small that the modest efforts of EAs (compared to all the massive corporations working on narrow AI) will speed things up significantly. In support of that, you mainly discuss vision—and I will agree with you that vision is not necessary for general AI, though some form of sensory input might be. However, another major focus of corporate AI research is natural language processing, which is much more closely tied to general intelligence. It is not clear whether we could call any system generally intelligent without it.
If you accept that mainstream AI research is making some progress toward human-level AI, even though it’s not the main intention, then it quickly becomes clear that EA efforts would have greater marginal benefit in working on AI safety, something that mainstream research largely rejects outright.
Interesting piece. It seems like coming up with a good human-checkable way to evaluate parsing is pretty fundamental to the problem. You may have noticed already, but Ozora is the only one that didn’t figure out “easily” goes with “parse”.
This may be partially what has happened with “science” but in reverse. Liberals used science to defend some of their policies, conservatives started attacking it, and now it has become an applause light for liberals—for example, the “March for Science” I keep hearing about on Facebook. I am concerned about this trend because the increasing politicization of science will likely result in both reduced quality of science (due to bias) and decreased public acceptance of even those scientific results that are not biased.
Agreed. There are plenty of liberal views that reject certain scientific evidence for ideological reasons—I’ll refrain from examples to avoid getting too political, but it’s not a one-sided issue.
It does seem like a past tendency to overbuild things is the main cause. Why are the pyramids still standing five thousand years later? Because the only way they knew to build a giant building back then was to make it essentially a squat mound of solid stone. If you wanted to build a pyramid the same size today you could probably do it for 1/1000 of the cost but it would be hollow and it wouldn’t last even 500 years.
Even when cars were new they couldn’t be overbuilt the way buildings were in prehistory because they still had to be able to move themselves around. Washing machines are somewhere in between, I guess. But I don’t think rich people demand less durability. If anything, rich people have more capital to spend up front on a quality product and more luxury to research which one is a good long-term investment.
- 22 Mar 2017 23:18 UTC; 2 points) 's comment on Globally better means locally worse by (
I’ve been trying to understand the differences between TDT, UDT, and FDT, but they are not clearly laid out in any one place. The blog post that went along with the FDT paper sheds a little bit of light on it—it says that FDT is a generalization of UDT intended to capture the shared aspects of several different versions of UDT while leaving out the philosophical assumptions that typically go along with it.
That post also describes the key difference between TDT and UDT by saying that TDT “makes the mistake of conditioning on observations” which I think is a reference to Gary Drescher’s objection that in some cases TDT would make you decide as if you can choose the output of a pre-defined mathematical operation that is not part of your decision algorithm. I am still working on understanding Wei Dai’s UDT solution to that problem, but presumably FDT solves it in the same way.
I agree with that… personally I have tried several times to start a private journal, and every time I basically end up failing to write down any important thoughts because I am inhibited by the mental image of how someone else might interpret what I write—even though in fact no one will read it. Subconsciously it seems much more “defensible” to write nothing at all, and therefore effectively leave my thoughts unexamined, than to commit to having thought something that might be socially unacceptable.
Encrypting/obscuring it does help a little bit, but doesn’t eliminate the problem, so it’s not just that.
I just tried this out for a project I’m doing at work, and I’m finding it very useful—it forces me to think about possible failure modes explicitly and then come up with specific solutions for them, which I guess I normally avoid doing.
Does anybody think this will actually help with existential risk? I suspect the goal of “keeping up” or preventing irrelevance after the onset of AGI is pretty much a lost cause. But maybe if it makes people smarter it will help us solve the control problem in time.
I find it interesting that you apply this argument to pigs as well. The view that most farm-raised pigs have lives worth living implies that you should eat as much pork as you want (or even more) since your purchases would result in more pigs being raised for food. If this applied to pigs even on factory farms, I’m not sure why you would assume the opposite about chickens.