When I first read this, I imagined a favorite politician (I won’t mention who) giving this mock speech.
To my embarrassment, I found myself nodding in completely genuine enthusiasm. This guy clearly knows what he’s talking about!
(This in turn made me consider just how much of this politician’s speeches was similarly composed. I came to the conclusion that quite a significant amount of it was)
...Nobody ever told me cognitive bias would be this annoying!
Eliezer, I accept your point about the underlying laws of probability. However, your example is extremely flawed.
Of course what the researcher operates by should affect our interpretation of the evidence; it is, in itself, another piece of evidence! Specifically in this case, publishing your research only when you reach a certain conclusion implies that any similar researches that did not reach this threshold did not get published, and are thus not available to our evidence pool. This is filtered evidence.
So without knowing how many similar researches were conducted, the conclusion from the one research that did get published can’t be seen as very strong. Do I need to draw the Bayesian analysis that shows why?
Thank you. This is exactly what I needed right now.
Eliezer, I hope you will take it as a form of high praise rather than insult that I stopped reading your article halfway through, typed this short comment, and am now going back to do some much-needed work.
(Hopefully I’ll get back to reading the rest later.)
I actually had a long-time close friend who turned out to be something of a compulsive liar.
These days I place almost zero weight on him having stated anything as fact… But a strange thing happened: Some of these things entered my head as cached thoughts and I didn’t remember their source at all—just that they were “true”. This has managed to cause me some embarrassment in the past.
These days I often take care to mention what my source is for claims I make, so it’s no longer a problem… But the general principle remains that most people don’t always know where their “knowledge” came from.
TheOtherDave, that is a very constructive approach :)
I am already prone to requiring policy specifics from politicians and being dissatisfied with vague points. But one thing I (and many others) do have is a tendency to note, when hearing a few specifics in a sea of “general direction” applause cues, is that my own preference for solutions is compatible with the speech; and from compatibility, I get hope that they would implement it—despite a lack of evidence that they’re even aware of such a solution, much less want to implement it. So this is something to be cautious of and to note mid-speech.
I could go further and try to strike from mental record anything that isn’t specifics, making a point-by-point list of substantive statements. An easy way to do this is ask “is anyone really considering doing otherwise? No? Then it doesn’t count. Yes? Then why are they?” This method might not always be wise—motivations and beliefs are also important in trying to predict a politician’s future choices they did not yet address, and the speech can pronounce those. However it would be a good mental exercise when trying to evaluate positions on a specific policy question.
Lastly, try to separate emotional jargon from actual policy. If your politician says we “need to be prepared for the 21st century”, recognize the fuzzy excitement that this statement gives you and squash it—it’s caused by the phrase “21st century” being linked in your mind with progress and technology. Wait until that politician says they’re going to specifically invest in technological literacy of 8th graders before you give it any significance, and treat it as suspect until then. (This is very similar to the first thing I suggested, except it focuses on recognizing an immediately triggered emotion in response to a phrase, rather than your own mind building scenarios which then in turn excite you).
I’ll try to remember all that for the next speech I hear :P
I don’t think this is at all the core issue.
Eliezer’s original post stated that beliefs need to come from mind-reality entangling processes.
If math is a part of “reality”, then Eliezer’s point stands and empirical reasoning makes perfect sense.
If math is not a part of “reality”, then we would expect it to influence nothing at all, including our beliefs. Or even suppose that knowledge came from somewhere and could influence belief but still did not otherwise correlate with reality: Then it would be irrelevant. This, of course, is not the case—as anyone who’s ever used any mass-manufactured device as well as bridges and roads, should realize. Math DOES have utility in real life. And I daresay that if it suddenly stopped helping us reliably predict the load-bearing limit of bridges, we’d treat is as suspect and false.
The ACTUAL core issue remains that a belief that cannot be reversed is useless.
I’ve had forms of this said to me; it basically means “I’m losing the debate because you personally are smart, not because I’m wrong. Whichever authority I listen to in order to reinforce my existing beliefs would surely crush all your arguments. So stop assailing me with logic...”
It’s Dark Side because it surrenders personal understanding to authority, and treats it as a default epistemological position.
When I was studying under Amotz Zahavi (originator of the handicap principle theory, which is what you’re actually discussing), he used to make the exact same points. In fact, he used to say that “no communication is reliable unless it has a cost”.
Having this outlook on life in the past 5 years made a lot of things seem very different—small questions like why some people don’t use seatbelts and brag about it, or why men on dates leave big tips; but also bigger questions like advertizing, how hierarchical relationships really work, etc.
Also explained a lot about possible origins of (some) altruistic behaviors; Zahavi’s favorite examples were from the research he and his wife conducted, wherein they observed small groups of social birds (forgot the species, sorry) where altrusitic behavior is common. And it turns out, it’s the dominant birds who behave altrusitically, rather than exploit their weaker brethren—but doing so as a show of strength. My own favorite example is when a lower-status male caught a worm and tried feeding it to the alpha male. The latter proceeded to beat him up, take the morsel, and force-feed it back to the weaker male.
Best course I ever took :)
I’m afraid not.
Actually, as someone with background in Biology I can tell you that this is not a problem you want to approach atoms-up. It’s been tried, and our computational capabilities fell woefully short of succeeding.
I should explain what “woefully short” means, so that the answer won’t be “but can’t the AI apply more computational power than us?”. Yes, presumably it can. But the scales are immense. To explain it, I will need an analogy.
Not that long ago, I had the notion that chess could be fully solved; that is, that you could simply describe every legal position and every position possible to reach from it, without duplicates, so you could use that decision tree to play a perfect game. After all, I reasoned, it’s been done with checkers; surely it’s just a matter of getting our computational power just a little bit better, right?
First I found a clever way to minimize the amount of bits necessary to describe a board position. I think I hit 34 bytes per position or so, and I guess further optimization was possible. Then, I set out to calculate how many legal board positions there are.
I stopped trying to be accurate about it when it turned out that the answer was in the vicinity of 10^68, give or take a couple orders of magnitude. That’s about a billionth billionth of the TOTAL NUMBER OF ATOMS IN THE ENTIRE UNIVERSE. You would literally need more than our entire galaxy made into a huge database just to store the information, not to mention accessing it and computing on it.
So, not anytime soon.
Now, the problem with protein folding is, it’s even more complex than chess. At the atomic level, it’s incredibly more complex than chess. Our luck is, you don’t need to fully solve it; just like today’s computers can beat human chess players without spanning the whole planet. But they do it with heuristics, approximations, sometimes machine learning (though that just gives them more heuristics and approximations). We may one day be able to fold proteins, but we will do so by making assumptions and approximations, generating useful rules of thumb, not by modeling each atom.
I’ve up-voted several lists containing statements with which I disagree (some vehemently so), but which were thought provoking or otherwise helpful. So, even if this is just anecdotal evidence, the process you described seems to be happening.
I don’t know that there’s literally zero correlation between the two. If you can deduce the laws of physics the aliens are running on, you could have stronger and weaker hypotheses regarding the physics creating their mental processes and, if your simulation was powerful enough, maybe deduce some proto-hypotheses about their psychology. You could further examine these hypotheses in light of what you know they know, and what they choose to show you.
I do agree though that you will need more direct data (actually ask them questions) to home in on the correct hypothesis in good enough precision (necessary for the manipulation Eliezer mentions). Mind you, if you get Internet access, it’s Game Over. That amount of information should be enough to derive any parameters relevant to any practical actions you may want to take.
This strikes me as particularly galling because I have in fact repeated this claim to someone new to the field. I think I prefaced it with “studies have conclusively shown...”. Of course, it was unreasonable of me to think that what is being touted by so many as well-researched was not, in fact, so.
Mind, it seems to me that defects do follow both patterns: Introducing defects earlier and/or fixing them later should come at a higher dollar cost, that just makes sense. However, it could be the same type of “makes sense” that made Aristotle conclude that heavy objects fall faster than light objects—getting actual data would be much better than reasoning alone, especially is it would tell us just how much costlier, if at all, these differences are—it would be an actual precise tool rather than a crude (and uncertain) rule of thumb.
I do have one nagging worry about this example: These days a lot of projects collect a lot of metrics. It seems dubious to me that no one has tried to replicate these results.
...Represents a choice where the internal simulation of any outcome is intolerable, providing a strong negative feedback for trying to choose it.
From a rational perspective there really isn’t that much of a difference between the two stated options—I don’t think any of us would morally condemn her for one and not the other. And abstaining from decision is almost immediately out the door in terms of desired outcome. So Eliezer’s point pretty much stands.
Of course, it wouldn’t really be fair to expect someone facing Sophie’s Choice to stay fully rational… Some experiences are just too jarring.
I think this is actually the core of the issue—not certainty of your probability, per se, but rather how it is derived. I think I may have finally solved this!
See if you can follow me on this… If Pascal Muggers were completely independent instances of each other—that is, every person attempting a Pascal’s Mugging has their own unique story and motivation for initiating it, without it correlating to you or the other muggers, then you have no additional information to go on. You shut up and multiply, and if the utility calculation comes out right, you pay the mugger. Sure, you’re almost certainly throwing money away, but the off-chance more than offsets this by definition. Note that the probability calculation itself is complicated and not linear: Claiming higher numbers increases the probability that they are lying. However it’s still possible they would come up with a number high enough to override this function.
At which point we previously said: “Aha! So this is a losing strategy! The Mugger ought not be able to arbitrarily manipulate me in this manner!” Or: “So what’s stopping the mugger from upping the number arbitrarily, or mugging me multiple times?” …To which I answer, “check the assumptions we started with”.
Note that the assumption was that the Mugger is not influenced by me, nor by other muggings. The mugger’s reasons for making the claim are their own. So “not trying to manipulate me knowing my algorithm” was an explicit assumption here.
What if we get rid of the assumption? Why, then now an increasingly higher utility claim (or recurring muggings) don’t just raise the probability that the mugger is wrong/lying for their own inscrutable reasons. It additionally raises the probability that they are lying to manipulate me, knowing (or guessing) my algorithm.
Basically, I add in the question “why did the mugger choose the number 3^^^3 and not 1967? This makes it more likely that they are trying to overwhelm my algorithm, (mistakenly) thinking that it can thus be overwhelmed”. If the mugger chooses 4^^^4 instead, this further (and proportionally?) increases said suspicion. And so on.
I propose that the combined weight of these probabilities rises faster than the claimed utility. If that is the case, then for all claimed utilities x higher than N, where N is a number that prompts a negative expected utility result, x would likewise produce a negative expected utility result.
Presumably, an AI with good enough grasp of motives and manipulation, this would not pose a problem for very long. We can specifically test for this behavior, checking the AI’s analysis for increasingly higher claims and seeing whether the expected utility function really has a downward slope under these conditions.
I can try to further mathematize this (is this even a real word?). Is this necessary? The answer seems superficially satisfactory. Have I actually solved it? I don’t really have a lot of time to keep grappling with it (been thinking about this on and off for the past few months), so I would welcome criticism even more than usual.
I came to this thread by way of someone discussing a specific comment in an outside forum.
I’m finding it difficult to think of an admission criterion to the conspiracy that would not ultimately result in even larger damage than discussing matters openly in the first place.
To clarify: It’s only a matter of time before the conspiracy leaks, and when it does, the public would take its secrecy as further damning evidence.
Perhaps the one thing you could do is keep the two completely separate on paper (and both public). Guilt by association would still be easy to invoke once the overlapping of forum participants is discovered, but that is much weaker than actually keeping a secret society discussing such issues.
With a budget closer to 5,000EUR, access to firearms, and enough willingness to use Dark Arts, I could probably keep it up for a decade or more. Possibly even pass on knowledge to selected disciples who would likewise guard these technological secrets, even as they rule the ignorant peasants.
If, on the other hand, our purpose is as originally stated—prove to the scholars of the time that I have knowledge worthy of them becoming my disciples so I can impart as much knowledge to them as possible—I probably won’t need much more than the superstimuli I described and a couple of afternoons. Something decidedly useful could cement this, and still on budget: A map of the world, a geographically appropriate taxonomy book, and a wristwatch (doubles as nautical navigational aide) would be enough. And I don’t think I’ve even reached 100EUR yet, all told :)
What we can achieve with today’s technology is so marvelous, it’s amazing how ordinary it seems to us. One day I turned on the faucet at my house and just marveled at the incredible and unlikely wonder of having fresh drinking water at practically limitless capacity being instantly transported to my residence, at my whim.
This isn’t just magic. It’s better than magic.
Ooh, I like that one. Call it the “sweet spot” theory of intelligent design—things of high enough complexity must be designed, but only if they are under a certain complexity, at which point they must be eternal. (And apparently also personal and omnibenevolent, for some reason).
At any rate, this would all be nice and dandy were it not completely arbitrary… Though if we had an agreed upon measure for complexity and could measure enough relevant objects, we might possibly actually be able to devise a test of sorts for this.
Well, at least for the lower bound. Seeing as we can’t actually show that something is eternal, the upper bound can always be pushed upwards a-la the invisible dragon’s permeability to flour.
[Late edit: I have since retracted this solution as wrong, see comments below; left here for completeness. The ACTUAL solution that really works I’ve written in a different comment :) ]
I do believe I’ve solved this. Don’t know if anyone is still reading or not after all this time, but here goes.
Eliezer speaks of the symmetry of Pascal’s wager; I’m going to use something very similar here to solve the issue. The number of things that could happen next—say, in the next nanosecond—is infinite, or at the very least incalculable. A lot of mundane things could happen, or a lot of unforeseen things could happen. It could happen that a car would go through my living room and kill me. Or it could happen that the laws of energy conservation were violated and the whole world would turn into bleu cheese. Each of these possibilities could, in theory, have a probability assigned to it, given our priors.
But! We only have enough computing power to calculate a finite number of outcomes at any given moment. That means that we CANNOT go around assigning probabilities by calculation. Rather, we’re going to need some heuristic to deal with all the probabilities we do NOT calculate.
Suppose our AI is very good at predicting things. It manages to assign SOME probability to what will happen next about 99% of the time (Note: My solution works equally well for anything from 0% to 100% minus epsilon—and I shouldn’t have to explain why a Bayesian AI should never be 100% certain that it got an answer right). That means that 1% of the time, something REALLY surprises it; it just did not assign any probability at all. Now, because the number of things that could be in that category is infinite, they cancel out. Sure, we could all turn to cheese if it says “abracadabra”. Or we could turn to cheese UNLESS it says so. The utility functions will always end in 0 for the uncalculated mass of probabilities.
That means that the AI always works under the assumptions that “or something I didn’t see coming will happen; but I must be neutral regarding such an outcome until I know more about it”.
Now. Say the AI manages to consider 1 million possibilities per prediction it makes (how it still gets 1% of them wrong is beyond me but again, the exact number doesn’t matter for my solution). So any outcome that has NOT been calculated could, in fact, be considered to have a probability of 1%/ 1 million—not because there are only a million possibilities the AI hasn’t considered, but because that is how many it could TRY to consider.
This number is your cutoff. Before you multiply a probability with a utility function, you subtract this number from the probability, first. So now if someone comes up to you and says it’ll kill 3^^^^3 people and you decide to actually spend the cycles to consider how likely that is, and you get 1/googol, that number is LESS than the background noise of everything you don’t have time to calculate. You round it down to zero, not because it is arbitrarily small enough, but because anything you have not considered for calculation must be considered to have higher probability—and like in Pascal’s wager, those options’ utility is infinite and can counter any number that Pascal’s Mugger can throw at me. You subtract, not an arbitrary number, but rather a number depending on how long the AI is thinking about the problem; how many possibilities it takes into account.
Does this solve the problem? I think it does.
(By the way: ChrisA’s way also works against this problem, except that coding your AI so that it may disregard value and morality if certain conditions are met seems like a pretty risky proposition).
I’d like to mention that I had an entire family branch hacked off in the Holocaust, in fact have a great uncle still walking around with a number tattooed on his forearm, and have heard dozens of eye witness accounts of horrors I could scarce imagine. And I’m still not okay with Holocaust Denial laws, which do exist where I live.
In part, this is just my aversion to abandoning the Schelling point you mention; but lately, this is becoming more of an actual concern: My country is starting to legislate some more prohibitions on free speech, all of them targeting one side of the political spectrum, and one of the main arguments touted for such laws is “well, we’re already banning Holocaust denial, nothing wrong came of that, right?”.
The slope can be slippery indeed...