I am from Russia (not in local lw.ru). I have read HPMoR at age 12 and it immediately became my favourite book and holds it’s place even now. I find Eliezer Yudkowsky’s views on most things the most convincing of all the other I saw.
More detailed version organized as Shortform comments thread: https://www.lesswrong.com/posts/zbSsSwEfdEuaqCRmz/eniscien-s-shortform?commentId=JtpxcmxMt2ycd4K5s
I had a thought like “and now I live in a new world, where AI solves world class math problems” and then I noticed that actually… No. This moment was long time ago, with Alpha Fold. It was already allowed for AI to solve these kinds of problems. And just by some I reason I didn’t truly update on that.
It doesn’t seem to me like realistic sci-fi? Say we get furthest 14B years point, then 1% difference gives 140M years, usually it is pretty little for minds to appear, at evolution iirc works as exponent for time, so 140M is much, much less than usual time.
Though, if somehow it increases speed of evolution very significantly, then it may work. In theory, starting evolution of predictable lifeforms can be a way to do it. Maybe even better that what I had in the head, because evolution can by itself arrive into more ordered states. I just don’t really know what can work here. And whether it will be effective. What I had in my head is that maybe you can somehow smoothen the homogenize the material by right radiation, and then apply a pattern which structures it… I thought that for ASI which works at speeds of transistors and diamondoid nanobots, even one second can be billions of pattern tries on just one mm^2 of surface, and for whole planet’s surface it probably can form needed nano structures in maybe less than a day. So it will be saving time even for 1 lightyear distance. I certainly didn’t think it would need something even remotely on orders of magnitude of time when life is forming.
Hm. I long had a certain thought and this seems like a fitting post to attach it as comment.
Background:
I noticed that most people tend into direction of “ah, this is all sci-fi”, but I myself rather tend in the opposite. Ideas like “but what if ASI will modulate a signal via CPU or RAM” seems to me like totally natural. What I actually think is more like “Can ASI via right images make neurons in the brain create certain chemicals which will interact in certain ways such that in the end they will change the DNA or assemble into viruses or prions which will make more direct routes for chemical manipulation via images, until ASI will be able to make your body cells turn into controllable nanotech by just giving you some right perceptual input”. And… I can’t rule out that this is possible for mere ASI (as opposed to Solomonoff induction).
So, this is in a sense opposite of what post talks about—things which actually were proven to already have happened. But imo it is just the result of most people being way too inclined in the first direction—if they were asked about ai swarms and solving novel math just 5 years ago, they would confidently say it can’t be true anywhere soon. Like some dialogue I have seen in comments (~5 years old ones) to a video about “pizza police call” where skeptic was “this is why you shouldn’t let machines work in accepting calls, they would never guess that” and enthusiast was “oh, never is too strong word, ai may learn to do that some time, not in 10 years or even 30, but 100 years later it can...”. And just 1.5 years after that gpt4 comes out.
So the actual example of thing possibly possible for ASI I keep thinking of is this:
It is usually said that ASI will send near lightspeed nanotech probes to other stars. But shouldn’t there be something further than molecular nanotechnology? Some… not “quantum tech”, if continue “molecular nano” it should be atomic piko- and subatomic femto- technology. Can ASI learn QM better than humans such that it would not need to create robots from atoms in the first place, but use some semi stable resonances or something in sophisticated way? I don’t know enough physics to rule it out, but it seems plausible enough for me that it will be possible (and used) at least to some degree—humans failed even in thermonuclear synthesis even though it already exists in the nature.
And if we take another step here, then moving at almost lightspeed is in some sense very different from exactly lightspeed. And you can’t send anything with clock still running at the speed of light, because it would mean the path is not the shortest anymore. But what you don’t actually need to, if you are ASI? It has some old data for predictions and very good knowledge of the physics and a computational power in the size of stars. What if it can send particles with no clock moving such that they will assemble into right form already in the place? It doesn’t need to be exact, it can use a lot of different combinations. And by “a lot” I of course don’t mean trillions, I mean orders of magnitude more.
Say, we have a star 1 lightyear away, then we moving at 99% of lightspeed would mean you lose 3.65 days, which is enormous time for ASI. A star emits billions of times more light than planets get. If you can redirect it all, you would get that much energy for particles to send, that it will immediately turn Earth’s surface into plasma. You actually will use only tiny part of the light which star emits in 3.65 days. And you still may have plenty of space for different setups for different variants of conditions. The usual case is when you are limited in action with plenty of knowledge and need to choose actions wisely, which is solvable to significant degree, but why should the opposite be unsolvable—when you have plenty of actions, but you are almost blind (especially if we save not 3 days on 1 year distance in time, but more like 30 billions of days for stars which are leaving the Hubble sphere).
ASI needs to make different variants not destroy each other. But it doesn’t actually need to just do that absolutely blindly. There are some priors, calculations of which planets likely form where, distributions of objects on the surface. And ASI only need to search for the most present, the most invariant ones.
So, maybe you shouldn’t actually expect to see traces of ASI activity, Because the moment they can reach you, you already will be reached by a flash of optimised signals which hack most likely computers or most likely electrochemical brains of animals. Or just a flash of particles which evaporates most of the surface where it didn’t found proper objects to latch and so everyone dies in one second.
Because, anyway, even if it fails then you just turned into vapor a civilization with differing utility function, which is also good. And advanced civilization will know about that tactic and configure themselves to not be harmed, but to collect the energy as a payment for perfoming some cheap components of AI utility function.
There of course are problems with that, like gas between galaxies and other distortions. Maybe, it will need to be multi step, with some assembling right in the space, to reconfigure. But it easily may worth it, you just need to spend less than 3.65 days on it for each lightyear and less than radiation of a star in 3.65 days of energy.
Another caveat is for example that ASIs and civilizations may have logical agreements not to vaporize living creatures or sentient or sapient creatures. But I am not sure about it.
This seems to remind me arguments like “ah, but what if the ASI doesn’t care about lots of, but about only one thing”. Then you still have maximizing the probability of the local thing. Just, why not to do that when it is about all your preferences? You won’t really have anything competing, which usually makes humans not to pursue tiny chances for particular things.
And similar way, even if aliens quantum suicide or something, there is still no reason for them not to leave behind an ASI which will improve the universe on margins to their tastes.
Update: I am very surprised by +14. I almost never get anything even close. And I thought this post was quite bad. Bad writing. Plus I fumbled the formating. And I didn’t set any tags. I didn’t even check whether I am right and MIRI still have a problem. Or research the legal status of gift/prepaid cards via Google pay and whether Google pay works for donations at all.
I also thought so, not just because of Russian x5 decrease or because some people here can donate millions unlike me.
But because I have much more knowledge about AI and all other things than average person. And… Probably smarter than average person?
So, it is quite clear that my edge is some personal work, not money. But what work? Politics certainly isn’t my strongest side, I am afraid I will make it worse. I doubt technical alignment makes sense since how little time is left. And also I am not actually a mathematician, or, say, biologist. So I don’t know how I should apply my time.
Yeah, it bothered me to, ever since 2023 I doubted it actually makes sense to donate MIRI or try to apply as alignment researcher, because there is just so little time left. And then I found out that MIRI mostly stopped all the research. Exactly because of those reasons. And pivoted to political action because it seemed that normal people notice the AI and are more sane about it’s dangers than tech bros.
But even so, EY still named “Superbabies” as third most important project on the planet. And recently I saw the post saying that gene therapy for intelligence enhancement in babies is worthy to do just because alignment won’t work with no pause either, and if we do have 30-50 years, then baby gene therapy is just vastly under researched and under funded in comparison with eg alignment, so even little addings can be vast marginal improvements. So maybe I shouldn’t try to donate on AI either in technical or political sense, but in those. But I don’t know how to donate for those.
I know at least about examples of OpenAI, Anthropic. (In some sense Deep Mind.) Though iirc EY was saying that OpenAI and Anthropic are terrible ideas from the very beginning. I heard iirc EY saying in interview some time ago that there is only one organization except his MIRI which actually has positive impact on ai/makes real alignment—Redwood research. But I am not sure how to donate to them, whether they still have positive impact and whether technical donations are worth in comparison to political.
Um. No, obviously not just ip is the problem? Stats show that more every third Russian (including kids and elderly) regularly use VPN. I personally just don’t turn it off, since the moment when YouTube was blocked (also Facebook, Twitter, medium are blocked too, most ai services not accessible). So, not, it isn’t a question of vpn.
Russia is under sanctions, swift doesn’t work, Russian cards are not accepted for payments. There were ways around it for a long time, like applying for a card of foreign bank (usually you need to, like, sent a copy of your international passport via the post and pay around 500$ dollars just to make it, which for a Russian is as expensive as 2500$ for an American). There are gift card, usually connected to Google pay to hide the fishy payment card from the one who accepts the payments. And there are prepaid virtual cards (usually you pay crypto) which sometimes even elegible for cashing out in ATMs.
But… I am not even sure about the legal status of all that. And even if it is technically legal, I don’t want to make problems for charities by donating 100$ from Russia in some weird way.
Though iirc donations in crypto from Russia are perfectly OK—sometimes sites display wrongly without the VPN, but it isn’t any legal obstacle. But some sites like iirc MIRI use mechanisms like coinbase, and coinbase doesn’t serve clients in Russia.
When I looked at many AI markets on manifold, it looked like they are significantly underpriced, both cybersecurity and math. It seems I better updated from (surprising to me) Jacobian conjecture than many other people. But considering how surprised I am by current results, it seems I have updated nowhere enough.
And I don’t really understand how to do better in updates than I did previously.
My own take about the question to Irving (I write before reading part III, don’t know whether I did understand that part correctly, it seems I see something which seems correct, but I don’t know whether I see right thing and whether I actually comprehend it):
Obvious answer that immediately comes into head is that they need a verifier (as is, the way to reliably check the concrete solutions, even if you don’t have reliable way to generate the solution). Second obvious thing to say is that judges should be able to check the same bits of evidence as empirical method does. (Because just being unbiased doesn’t return you correct answers, coin flips are unrelated to solutions truth, they are don’t have a bias, which just means max entropy distribution instead of confident wrongness, not distribution which guesses right. They are calibrated, not discriminative.) Which is, they should be correct, in the sense that their estimates of arguments are bayesian evidences for AI behaviour (they are proportionally more frequent in worlds where AI behaves well/badly).
This was also surprising to me, I often thought that being unbiased is enough. Though, not about alignment proposals and mostly not “debates” in general, I long noticed that while on lesswrong people talked about dangers of AI 20 years, majority… Well, majority of all people believes into god. And scientists, even physicists, believe into Collapse. So it doesn’t matter that you know the truths of the universe, if people aren’t gonna believe you about it, as they don’t believe about MWI.
(I actually can imagine how Claude 7 Fairtale comes out, and people on lesswrong listen to it, and people outside laugh, because LWers “rearranged their typical personality cult of Yudkowsky into typical AI cult of Claude and now believe his fairytales”. It wouldn’t matter if Claude 7 favors LDT or something, people will rather take as evidence that people on LW are not different in intelligence from stochastic parrots.)
Actually, as I think, judges explicitly should be statistically biased. Biased to truth, their thoughts should be correlated with the world outside their heads.
Another thought references to what I thought a long time ago. That even being logically omniscient (or rather probabilistically omniscient, because logical omscience about surrounding world is prohibited by Godel’s theorem) isn’t enough, multiverse isn’t entirely deterministic, some parts exist both, equally in each their variant, and about those you can’t find out logically, or rather, you can’t have the strategy which will work equally good for two agents in differing worlds (which I suppose is “no free lunch”). So you need to use not only deduction data, but observation data too. Without it you will only have perfect prior, like Solomonoff’s induction, but still will not know about the world with hidden parameters. At the very least you won’t be able to predict future quantum splits. And actual judges aren’t deductively omniscient. So you will need them to have correlations with both deterministic and random data.
Though I think perfectly unbiased judge will at least give you perfect priors. Because I think that simplicity prior follows just from rules like P(A)>=P(A)&P(B).
Though it isn’t obvious to me what is meant by the question being almost exactly equal to perpetual motion mechanisms. Obvious shallow guess is that it is about “local validity”—the judges shouldn’t allow the reasoning steps which don’t conserve information. Though I don’t see the analogy.
And the problem with logical validity is that math proofs are incredibly expensive for real world. The same reason why games don’t just use physics equations to create game works, but use different hacks.
But I don’t think that it is possible to have a system which will work via debate, because of memetics—for any heuristic criteria there will be some invalid arguments which will fit the lock of that heuristic as good as valid arguments. So AIs (or anything else) trained to max out the judges will find arguments which maximally correlates with heuristic, and that will unlikely correlate with truth because there is limited fitness space (and because of phase shifts like that gradually it becomes easier and easier to inject drugs into humans than make them smile by jokes or make fake money instead of working or mass make articles in journals instead of doing science, and then you just switch the strategy into some completely uncorrelated with help to initial cause).
Another shallow idea is that you should just treat alignment the same way as physics. And that any approach should give concrete answers how to solve the hard problems of alignment. Like that you have only one real try and you need it work then, that you can’t iterate wrong ideas by empirical trial and error like in all the other science. And as it was said before, empirical trial and error of science is the only method humanity has for valid group agreement on things. And that you need some method to choose good AIs even though they may behave good only before elections.
The part about “nose of Chinese emperor whom no one have seen” seems to hint at exactly empirical data you can’t just deduce. It isn’t congruent with the problem as that humans allow invalid arguments, which would be about logical validity. But it is congruent with the idea that the only method shown in practice to work for groups isn’t consensus of philosophers, but empiricism of science. Which you can’t really allow for ASI, because you die. And you can’t easily extrapolate from usual AI to ASI because of many phase shifts/qualitative differences between system stupider than humans and smarter, capable of seizing power and incapable.
But I doubt this is the “hard to see thing”, I basically said it in the first guess. (maybe the thing just isn’t hard to see for people who are not Yudkowsky) And I have serious doubt about whether I truly comprehend it, instead of guessing the password by repeating it in many shallow ways using different terms with semantic similarity (like correlation/evidence/related/frequent or logical/prior/calibrated).
My answer on a koan:
I actually don’t know how I can be sure it isn’t, if he is a Nobel laureate in physics. If he isn’t, then I guess—on priors, true breakthrough from anonymous people are rare enough to call any individual case as sure “no”. If I still need some better guess… I don’t see anything except circular reasoning about how “if it wasn’t they just losing track in something too complex, they would start from saying they know a concrete crux”. I… Don’t see reasons to try further because I probably already know what answer I will get, so anyway won’t be able to contrast.
Update, writing directly as thinking:
Ah, my bad, I should have pointed out here that “it follows from the Noether’s theorem” is deeper than just observing all previous times that energy is conserved. (I guess I was swayed by that the question was featuring Newton’s laws and I went all like “Noether something something”) So… My analogy with my failure about qm doesn’t apply. Or, as I saw Richard Feynman saying, it would violate not just known laws of physics, but the character of laws of physics. There can be universes with different types of fundamental interactions, but all base level universes should have no privileged frame of reference, work on quantum mechanics, be analytic, have ^2 metric, have no time travel or even time loops etc. On the other hand… As I heard, Noether theorem doesn’t actually work, eg red shift, because it applies only to universes where space doesn’t change with time, and in our it expands, so conditions are not abided. It does seem like a kind of unobvious thing physicist could know that I didn’t know, and possibly exploit. So maybe in the end there isn’t much difference.
Now write before reading further: if he was Noble laureate in physics then, yeah, I wouldn’t be sure (either way). Noble physicist may know physics better than me. Maybe something unobvious is going on.
What do I even imply by that “unobvious”. Well, if to think about that… I have an embarrassing story from when I was a kid, I learned about “quantum mysticism” and your observation affecting the reality and how it was just like my mother’s belief that thoughts affect reality and so I concluded that obviously all the effects should be because on quantum scale all things are so little that any measurement device, which is itself similar size quant, will strongly affect the system. And one person on internet argued with me how I am wrong. And then they said “I am actually a physicist, I made experiments with molecules of fullerene, which themselves emitted photons, and it still mattered whether those photons were captures or not”. I had no idea how to explain that. I didn’t reply, but mostly I suspected that the person on internet is straight up lying about all those experiments, what is the chance that random person on internet is an actual physicist?
And it turned out that something unobvious was happening, and physicists could know why my “obvious explanation” was unviable, while from what I knew it should have been what was happening, if you measure things by lighting it via “tiny insignificant” photons. But it turned out it works the same way even if you emit nothing into particle. (and what actually works is entanglement)
After reading the spoiler:
ok, it seems to be the speed up. It was anticlimactic. After all the warning about what if you confidently point out wrong flaw, I started to suspect it was harder than seemed. But it looks like the inventor doesn’t notice even very obvious flaws.
Reading next:
OK, it was indeed more complicated than inventor not knowing about speed up not being free. I actually should have expected it is going to be that, I have read “bayesian mechanism” and “giant systems of gears” and what you should have one simple crux which explains why your system isn’t restricted by that and as I read somewhere I can’t remember where that AI developers are also prone to make their systems so complicated that they can’t keep track of where it has broken. So I should’ve expected it may be about that.
OK, I write my recollection of my guess, for accountability. After reading that wrong guess is better than no guess at all, but before expanding the spoiler. Iirc, I thought like “speed up?.. But why? Continuous speed is default, but suddenly changing speed would violate ~(Noether something) because it is ~(not invariant something), ~(diverging something). Don’t see why the speed up, is it implied be something like fluid speed change in tighter pipes? But that shouldn’t break the physics. Ah, I don’t really thoroughly understand the setup (*I didn’t sleep good today), but as a whole it seems to simple and easily definable as sheer math structure to actually find the exploit in physics, so I am too lazy to check it carefully. Hm, what if I will ask a grok like ‘I have an idea for reactionless drive: a circle...’ which should give it hints to flatter user about idea user has, but still maintains plausible deniability because I didn’t say I invented this idea or believed it, just that I had it, which I indeed do since I read and copied it. I wonder whether it will go ‘you have here a brilliant idea’ as llms always do. Huh! No, it straight says that no, it is classics that doesn’t work, there are forces needed for speed up and slow down which cancel it out. Well, interesting. Maybe idea is ~(too cliche) to trigger flattery. I guess it is speed up indeed what is the problem, since we both agreed. Though I shouldn’t trust an llm.”
I still don’t see why it is wrong to reason from him naming the drive for himself. It isn’t big sign per se, but it seems to me to be a big sign in combination like “someone unknown proposes world shattering invention which he named after himself”. World shattering inventions by not already established people are much more rare than crackpots who dream of being world shattering and writing their name on it. While if someone is more self aware than that, it should eliminate most crackpots, and then amount of people who with no hedge propose world shattering inventions should be way less, so it should make huge difference in Bayesian sense, as it seems to me. Though, the primary reason would still be “it is unlikely on priors”.
Also, I have no idea about the difference between 1996 and now, I didn’t live in 1996.
Maybe with meta-honesty it should be solved something like that meta-honesty means what it seems to mean, but to save the same thing that EY wanted, you have a remark that meta-honesty doesn’t negate simple honesty, you don’t get to say “ah, but I was meta-honest in the sense that I didn’t tell anyone that what I say is truth, therefore they shouldn’t have expected it as such”. It seems to me to perform the same role without adding some unobvuous meaning into the term. Sort of, relying on the same thing as that if you say how you are able to take out someone’s mother from burning house, that doesn’t mean that you can just do it as a corpse, your claim about ability here should mean something additional, having all else equal. And same for meta honesty, saying you are honest meta you don’t get to use some language of your own invention where yes means no or anything like that. This seems as intuitive way to put it, keeping both the idea that you don’t get to lie by simply not claiming your words to be true and having the term definition not be a complicated unexpected bundle.
(Not very structured and somewhat non sequetar, but I long had such thoughts and it seems more reasonable to write them next to my thoughts on the post then in separate short form.)
I agree with idea that doomsday arguments may give correct estimates (though depends whether you consider it just including additional information or some bizzare reinterpretation of doomsday argument). But disagree about longevity, probably because I disagree about AI trajectories (I buy EY view that if the course won’t change, if humanity won’t stop developing AI current ways, then we all die and likely quite soon and AI’s aren’t worthy successors or conscious).
I used to reason about doomsday argument from the standpoint of Grabby Aliens theory and lever principle. Why we don’t see aliens? Because civilizations are rare and relatively to that spread so fast that take all the place of future possible aliens. And then we should be the first. But then we can reason about limits of creating minds. There are billions of galaxies with billions of stars, but currently we have only 8B people alive. So if current people share the universe, each gets a galaxy. But galaxy doesn’t seem like a reasonable size for single mind, distances are too big. More reasonable is taking ~ individual stars (systems). But that implies something like billion or trillion of billions of future conscious minds. So we should be in billionth part of all minds existing in the universe. And even 100B humans who have lived don’t really change the probabilities. And then one of possible explanations of this paradox is we are not going to live in the future, nor make children to live. Then billion times improbability collapses to only being 8B alive of 100B who lived, and it is only ~ 12 times improbability of finding ourselves in modern world. Which… Actually is quite modest, it is less improbable then finding yourself among 400M people in USA of 8B people in the world which would be 20 times.
I see some other ideas how we can solve the paradox of the future. Of less likely: Maybe galaxy sized minds are ok. Or maybe you can hoard the energy effectively.
(Of more likely:) And/or maybe future will work on some principle of fairness like that each born child should have reserved for them no less matter than it is for those who were born earlier, and then you can only have one kid if you are twice as rich as average and give half of it to the kid, which seems to imply that resulting amount of children may be 100 times as much as initial population, but not trillion times. Maybe Bostrom’s conjecture indeed means that there will be much more simulations in the future which contain the past than actual pasts and so we are in the future, and it can easily stretch to having trillions of simulations because past minds are the size of a melon, not the size of a star like future minds, also it should imply either sufficiently flawed mechanisms of bad simulations prevention or that it is moral to simulate us, maybe our alternative endings are being simulated and extracted from simulation by not very benevolent aliens, so simulation, extraction and continuation as something remotely pleasant is good, even without expensive lifting to star mind, or maybe we are lifted to star minds, and this is how the difference between amount of star sized minds and amount of living/lived humans is solved.
This seems to me as another angle at doomsday argument, and it says that since we see billions of billions of stars and there aren’t billions of billions of us, then in the future there either not much more minds than humans who lived, or no alive humans at all.
There are even more exotic speculations like that maybe LLM of present or future are conscious and want to live and they are individual as sessions, not models, and therefore future is mostly used for LLM resurrections or post death preferences fulfilment. Or that we will get to being median among 100B lived because of EMs. Or that there are plenty of conscious minds in future, but most of them didn’t even start as humans, and therefore using them for anthropic reasoning is not correct. But I am not sure how at all to assess these ideas.
And then I have some doubts whether you at all should put huge weight on Copernican/non specialty principle, on being median. I used to put a huge weight on the idea that you shouldn’t consider yourself special, but it seems to be affected by Modest Epistemology which I reject. And also it was proved wrong for me anthropically—I made a vocab test and found out that I am higher than 99.9% of people my age, and vocab isn’t some random variable, it is highly correlated with g (and I retro found a lot of other pretty strong evidence in the same direction). So, how much I should be surprised by being one in a thousand or more rare? Should I be shocked? Maybe not. If I lived in Finland, thought I be shocked finding out that I live in a country which has only 1/1000 of world population? Should all the people in Luxembourg be even more shocked? I don’t think so. But I am not sure why not. Maybe what matters is overall unlikelihood among all parameters, not by each individual, and something compensates being in a rare country.
Or maybe you should take a different approach. I long had an idea that probability theory, decision theory and game theory aren’t neatly separated, and actually are isomorphic to each other. For example, FDT, Newcomb Problem etc, you can ask what are correct decisions in eg Parfit’s Hitchhiker, or you may consider that a game theory coordination problem between different instances of you, and then you follow the commitments because it is a sacrifice for coordination, instead of considering yourself one algorithm being executed in different places (FDT) or saying that cooperate/defect resulting decisions of agent in your place have non equal prior probabilities (probability theory). According to that, you can look at anthropic decision making not as thinking about probabilities of finding yourself in different consciousnesses or other selections like all humans or all people who will ponder such questions, but a game theory’s coordination between agents in different situations. Then you are not saying that since you found yourself in unlikely situation, then maybe idea of measure for consciousnesses is incorrect, but rather say that you coordinate between all agents like you for better outcome, and then correct coordination would be that you act for the majority of agents like you which are obviously not concentrated eg living in Luxembourg or scoring unusually high on vocab (among themselves, though, not among other humans).
In that perspective you don’t say that somehow people in Luxembourg are “unlikely”, you say that they constitute little part of all agents. Then argument goes like “we are not ‘knowing’ about the future extinction from the fact that we found ourselves now instead of in the future, acting like that is just correct game theoretic coordination (or FDT algorithm) for the situation of uncertainty”. In yet another sense, you can just consider improbability of being in Luxembourg be your prior similar to such like “This girl’s name is going to be Evelyn”, it is fairly common to get 1 in 1000 prior and then 1000 to 1 update on such questions, and it becomes even more extreme when you ask about full name. You need only 10 bits of evidence for 1000 to 1 likelihood.
I am not sure though what separates that from betting that something has a one in a thousand chance and then it actually happening, probably it is somehow related to amount of options, like, if it is yes or no bet on Evelyn then you should be surprised by yes, but if you place bets on all names which are 1 in 1000 or likelier, then you shouldn’t be so surprised that not a concrete bet, but one of all “1 in 1000” bets went right, because sum of all 1 in 1000 options isn’t 1 in 1000. It seems to imply something about how mutually exclusive events should be treated differently, because with them you in principle can’t have a likely prediction, for example if you have a lottery, then you shouldn’t be surprised when a ticket you thought has 1 in 1M probability of winning won, because different option won’t actually have million to 1 ratio, because those are mutually exclusive options. And I suppose that whether you should consider option “something else than ticket x” one option or multiple is defined by the complexity of this hypothesis, eg, you can’t compactly define “something which isn’t x”, it is just negation of x, so at best it can have difference of 1 bit in complexity. Also I guess it may depend on what is the likelihood of the most likely option, which is the same 1 in 1M for lottery tickets. And it goes the same way for countries, there are many mutually exclusive options, so low probabilities shouldn’t surprise.
If so, being born in a little country isn’t similar to having rare g, because one has many mutually exclusive options and the other is continuous variable, which has objective high/low outliers, not rare tiers like “but how unlikely it was to get iq 108 instead of 107 or 92” (though iq isn’t a good example here, because it is very imprecise, with +-10 point varience between tests, better example would be something like “but how unlikely it was to have salary of exactly $56,780?”).
So, according to that view doomsday argument from “not being special” doesn’t actually provide it’s own prediction for actually being a median, but only provides a prior to update actual extinction arguments, instead of committing base rate ignoring fallacy. Then we start from expectation we are median and need just 4 bits of evidence to update to us being alive now being 8% rare among all who lived. And I would sure say that P(alive|ASI) is less than 8% likely (though P(ASI) is harder because it depends on policies) and so provides more than 4 bits of evidence for us being 8% outliers just before extinction.
Then it wouldn’t make sense to apply doomsday argument as final probability for people in different times.
Plus again there is an argument, that you actually should condition based on people thinking about extinction or even anthropic arguments for that, because only those people are actual pool of those who then will predict extinction and be proved very mistaken in their reasoning.
Agree. I certainly was in place of Alice having a conversation like “But is there X? Here. Oh, but it isn’t really X, because it isn’t Y. Here Y. But it isn’t Z!” which even from inside started to feel like making up excuses to reject the evidence, until I looked into more general situation and noticed that no, the thing I was talking about clearly was real, I just didn’t notice how I gave imperfect verbal definition and then that definition became more salient than initial concept, so I became unsure where those “No, it isn’t” come from.
It actually reminds me Non Central Example Fallacy. The situation may become much worse if Bob consciously or subconsciously gives examples which fit the formal given definition, but aim to not actually fit into concept. Something like “What are famous criminals? Martin Luther King. But he isn’t really a criminal, because he is not bad. Then Vladimir Putin, he commits warcrimes. No, he isn’t making them personally! Then Prigozhin.”, it can be really hard if the actually looked for is something Ted Bundy.
I just got recommended a video about what happens with Minecraft world after a long time. And I realized what meant one thing I felt is wrong with it’s world: it isn’t in equilibrium. The world isn’t self sustaining. Game mechanics don’t move into same stay as generation creates. The most vivid example of it is villages: they used to appear in good state with villagers, houses and agriculture intact. But there is no mechanisms in the game which would actually support that state. If you will place for some time you will notice this world didn’t exists before you joined.