Ah—I hadn’t read that post yet, so I missed the reference. Thanks!
roystgnr
A problem with self-reference which I find nearly as amusing but which is much more terse:
“This sentence is false, and Santa Claus does not exist.”
Unfortunately, all these data points have been already filtered. Learning that some non-cultural-nerds aren’t scared away by the nerdisms here is good, and thank you, but I wish I saw way to coax a more meaningful statistic out of them. “100% of non-nerds who still post on Less Wrong haven’t been discouraged away from posting on Less Wrong”, fine, but for each one of you are there 0.01 or 100 others who didn’t stick around?
I’m reminded of a comment here about “borrowing offense”: you can’t try to single-handedly anticipate and eliminate everything that might hypothetically bother someone else, because without actual bothered people to talk to you have no way to tell where to start or stop. But what are the odds that someone reading this page is so anti-nerdiness that they’re considering leaving the site but not so anti-nerdiness that they’ve left already?
A priori it seems like a good guess. In any academic discipline, at the thesis level and above, originality is necessary for your work to be perceived as having any value, But most complicated questions with one true answer have a thousand false answers. Once people have beaten you to the true answer, it’s going to be easier to be original if you go for one of the false ones, so long as you’re in a discipline where verifying truth or falsehood is harder than obscuring it.
I suspect a good bellwether for identifying such disciplines is the way they treat repetition of existing truths. A lemma in a math paper might be “obvious”, a paper with too many obvious lemmas might be “too verbose”, but there’s never any suggestion that the lemmas are “trite” or that the reviewers would prefer to read their converses instead.
In the early modern period various thinkers were asking questions that would ultimately lead to the foundations of modern science
Indeed, much of what we think of as “modern science” used to be called “natural philosophy”. Even “logic” used to be considered the realm of philosophers rather than mathematicians. Philosophy may be maligned in part due to a linguistic selection bias where, as soon as we start to really understand a subject, we stop calling it “philosophy”.
Apostrophe rules are scatterbrained, but I’m not sure how high they be on a list of grating flaws or kludges in English. I might not even have made them a separate list entry; they’re a subset of homophones, and many other homophones are more detrimental to clear communication. Non-phonetic spelling rules make it harder for everyone to learn to read. Irregular conjugations add unnecessary and illogical hoops to jump through before anyone can even speak without unintentionally signaling low intelligence. Hell, I used to think of I/l/1 as a programmer’s problem until I discovered that toddlers stumble on the same unnecessary ambiguity; at least programmers get to choose their own fonts.
It says “make money for Apple”, which is a roundabout way of saying “make money for Apple’s shareholders”, who are the humans that most directly make up “Apple”. Apple’s employees are like Apple’s customers—they have market power that can strongly influence Apple’s behavior, but they don’t directly affect Apple’s goals. If Joe wants a corporation to give more money to charity, but the corporation incorporated with the primary goal of making a profit, that’s not the decision of an employee (or even of a director; see “duty of loyalty”); that’s the decision of the owners.
There’s definitely a massive inertia in such decisions, but for good reason. If you bought a chunk of Apple to help pay for your retirement, you’ve got a ethically solid interest in not wanting Apple management to change it’s mind after the fact about where its profits should go.
If you want to look for places where corporate goals (or group goals in government or other contexts) really do differ from the goals of the humans who created and/or nominally control them, I’d suggest starting with the “Iron Law of Bureaucracy”.
What if I know that my uncertain parameter C from a model equation C*exp(T/T0) is in a certain range… but then I wonder whether I should instead be evaluating the exact same equation as exp(T/T0 + D) (with D = ln(C), C = exp(D))? Do I use the uniform distribution on C, which will correspond to a skewed distribution on D, or do I use the uniform distribution on D which will correspond to a skewed distribution on C?
Non-Euclidean geometries? IIRC the questions of “what can you still/now prove with this one postulate removed” were studied for centuries before hyperbolic or elliptic geometries were really understood.
Or maybe I’m misremembering. That always did seem odd to me. I guess hyperbolic geometries can’t be isometrically embedded in R^3, which makes them hard to intuitively comprehend. But the educated classes have known the Earth was a sphere for millennia; surely somebody noticed that this was an example of an otherwise well-behaved geometry where straight lines always intersect.
How do we describe this shutdown command? “Shut down anything you have control over” sounds like the sort of event we’re trying to avoid.
How about 6.: arguing for the lurkers’ benefit? That argument was voluminous, repetitious, and random sampling of the low-voted comments turned out to only be useful for verifying that their scores weren’t capricious, but a few of the upvoted comments were worth my time to find and read; if a few dozen other people felt the same then they might have also been worth the authors’ time to write.
I’ve got a truly marvelous proof of this theorem, but it would take forever to write it all out.
A box that does nothing except predict the next bit in a sequence seems pretty innocuous, in the unlikely event that its creators managed to get its programming so awesomely correct on the first try that they didn’t bother to give it any self-improvement goals at all.
But even in that case there are probably still gotchas. Once you start providing the box with sequences that correspond to data about the real-world results of the previous and current predictions, then even a seemingly const optimization problem statement like “find the most accurate approximation of the probability distribution function for the next data set” becomes a form of a real-world goal. Stochastic approximation accuracy is typically positively correlated with the variance of the true solution, for instance, and it’s clear that the output variance of the world’s future would be greatly reduced if only there weren’t all those random humans mucking it up...
I never said the box was trying to minimize the variance of the true solution for it’s own sake, just that it was trying to find an efficient accurate approximation to the true solution. That this efficiency typically increases as the variance of the true solution decreases means that the possibility of increasing efficiency by manipulating the true solution follows. Surely, no matter how goal-agnostic your oracle is, you’re going to try to make it as accurate as possible for a given computational cost, right?
That’s just the first failure mode that popped into my mind, and I think it’s a good one for any real computing device, but let’s try to come up with an example that even applies to oracles with infinite computational capability (and that explains how that manipulation occurs in either case). Here’s a slightly more technical but still grossly oversimplified discussion:
Suppose you give me the sequence of real world data y1, y2, y3, y4… and I come up with a superintelligent way to predict y5, so I tell you y5 := x5. You tell me the true y5 later, I use this new data to predict y6 := x6.
But wait! No matter how good my rule xn = f(y1...y{n-1}) was, it’s now giving me the wrong answers! Even if y4 was a function of {y1,y2,y3}, the very fact that you’re using my prediction x5 to affect the future of the real world means that y5 is now a function of {y1, y2, y3, y4, x5}. Eventually I’m going to notice this, and now I’m going to have to come up with a new, implicit rule for xn = f(y1...y{n-1},xn).
So now we’re not just trying to evaluate an f, we’re trying to find fixed points for an f—where in this context “a fixed point” is math lingo for “a self-fulfilling prophecy”. And depending on what predictions are called for, that’s a very different problem. “What would the stock market be likely to do tomorrow in a world with no oracles?” may give you a much more stable answer than “What is the stock market likely to do tomorrow after everybody hears the announcement of what a super-intelligent AI thinks the stock market is likely to do tomorrow?” “Who would be likely to kill someone tomorrow in a world with no oracles?” will probably result in a much shorter list than “Who is likely to kill someone tomorrow, after the police receives this answer from the oracle and sends SWAT to break down their doors?” “What is the probability of WW3 within ten years have been without an oracle?” may have a significantly more pleasant answer than “What would the probability of WW3 within ten years be, given that anyone whom the oracle convinces of a high probability has motivation to react with arms races and/or pre-emptive strikes?”
- 20 Apr 2011 13:36 UTC; 14 points) 's comment on Is it possible to build a safe oracle AI? by (
- 9 Mar 2012 17:14 UTC; 9 points) 's comment on A taxonomy of Oracle AIs by (
- 1 Dec 2011 20:49 UTC; 9 points) 's comment on Intuitive Explanation of Solomonoff Induction by (
- 9 Mar 2012 9:15 UTC; 6 points) 's comment on A taxonomy of Oracle AIs by (
- 20 Aug 2019 14:44 UTC; 5 points) 's comment on Self-supervised learning & manipulative predictions by (
- 3 Dec 2011 16:03 UTC; 1 point) 's comment on Why safe Oracle AI is easier than safe general AI, in a nutshell by (
- 25 Mar 2019 20:57 UTC; 1 point) 's comment on Predictors as Agents by (
- 1 Dec 2011 21:34 UTC; 0 points) 's comment on Intuitive Explanation of Solomonoff Induction by (
The only way you’re likely to be making profits at the expense of others is if you pick stocks in industries with significant externalized costs… which I admit may be hard to avoid, especially with an index fund, depending on the true impact of CO2 emissions and on the status of corresponding Pigovian taxes that apply to your investment targets.
Otherwise, for the most part you’re making profits at the profit of others. Life isn’t a zero sum game. Capital markets want your dollars because investment can help create and improve products that are worth more than their inputs plus the investment, and “worth more” is in the opinion of the people who buy those products, who after all wouldn’t have bought them if they didn’t value them more than the money they spent. Likewise for employees as well as consumers—even “sweat shops” with dismal conditions by first world standards have to supply locally-high wages to attract workers, and historically this ends up raising wages for the country as a whole.
Your profits are (infinitesimally) reducing the profits made by other investors. Microeconomics says that if you increase the supply of something, including capital, the price paid to the suppliers typically goes down. But no wealth goes away in this scenario. Other investors’ loss is exceeded by employees’, consumers’ and your gain, and since consumers and employees are on average less wealthy than investors the net gain is even greater in utility than in dollars.
I’d like to know what “software torture” means once the metaphor has been stripped. As it is, that phrase doesn’t tell me what they’re doing, but does tell me what I’m supposed to think about it, which combination is worrisome. Fuzz testing would probably be considered “torture” by anyone who hadn’t heard of it and who didn’t realize that criminals looking for software exploits were doing it too.
This runs into strongly believed theoretical comp sci limits such as the likelyhood that P != NP.
Does it? There are certainly situations (breaking encryption) where the problem statement looks something like “I’d like my program to be able to get the single exact solution to this problem in polynomial time”, but for optimization we’re often perfectly happy with “I’d like my program to be able to get close to the exact solution in polynomial time”, or even just “I’d like my program to be able to get a much better solution than people’s previous intuitive guesses”.
If we assume that our time-discounting function happens to be perfectly adjusted to match our rate of economic growth now, is it wise to assume that eventually the latter will change drastically but the former will remain fixed?
A list I made a few years ago:
This morning I harnessed the power of lightning to make breakfast, made some synthetic milk for my baby daughter, watched some moving pictures that were sent over the ether, and then piloted my metal horseless carriage to work. I had some productive discussions about how to improve the ways in which we send people to space and back. (Although making predictions by using machines to solve trillions of nonlinear equations is very cheap, the results need to be more reliable.) Once I finish sending my musings to you folks via this world-girdling network of glass and copper wires, I plan to drive home at a hundred kilometers per hour over a few concrete ribbons (often suspended dozens of feet off the ground), instantly reheat some food with microwave radiation, and maybe enjoy a glass of grape juice that’s been partially consumed and replaced by yeast excretions.
Some of these ideas are better or worse than others, they’re all relatively normal today, and some of them have been pretty normal for decades or even millennia. But could you imagine what a crackpot you would sound like explaining each concept for the first time to someone unfamiliar with it?
To evaluate calibration.