I am from Russia (not in local lw.ru). I have read HPMoR at age 12 and it immediately became my favourite book and holds it’s place even now. I find Eliezer Yudkowsky’s views on most things the most convincing of all the other I saw.
More detailed version organized as Shortform comments thread: https://www.lesswrong.com/posts/zbSsSwEfdEuaqCRmz/eniscien-s-shortform?commentId=JtpxcmxMt2ycd4K5s
It seems to miss the context of OP. Why isn’t math full of universes which very fast dissolve into max entropy thermal baths which exist indefinitely, able to directly generate Boltzmann brains randomly, but unable to have evolution.
I think that a hint for solution may be that many sources of evidence we live in Big Universe imply that if any non tiny amount of universes ended up in Boltzmann generating states, then we will find ourselves as Boltzmann brains in a universe which allows them. So it should not just disprove all physical theories that end up in Boltzmann brains. There should be some factor which makes Boltzmann brains less likely than normal consciousnesses in all of universes, not just in result of physics of our universe.
Also I don’t quite see how you would observe not dissolving into chaos. But there is still the fact that when we look into our past memories, our minds, they look much, much more regular and complex that would you expect is needful for having consciousness (and ability to reason). Boltzmann brain generation seem to imply that it will not produce human brains first, it will produce first near perfectly optimal mind designs with consciousness written in the minimum possible amount of atoms.
This all is new argument for me. I expected universe to exactly be in maximal entropy indefinitely long. And this arguments refutes this possibility.
I am not sure what then. Obvious idea is that maybe even in maximum entropy it is more likely to have a Boltzmann seed which will causally produce a lower entropy bubble than have Boltzmann brain generated directly. But I doubt that possibility of such seeds isn’t exact opposite of “maximum entropy universe”.
I have a mixed feeling about reading that post. The most mixed of all post on lesswrong. In some very high percentile among all I have read or watched.
(edited: I thought that I maybe need to clarify that I don’t have ocd, to avoid confusions)
First, it reads easily and with an interest. So I was going to upvote.
Then there is very confident assurance that you can’t stop having some feeling. Which is simply not true, according to my experience. You can. And even do it immediately.
(And, after talking to LLM about, I suspect that I need to add: not, I am not ‘suppressing’ feelings, in the sense that feeling is not stuffed somewhere outside out attention—it is gone, and physiological signs like tension are gone too and signs like heartbeat calming down at a standard pace, and those feelings don’t try to resurface and even stronger than before, if there is a problem, then it is opposite of this—that they may become too weak or disappear)
You can control your feelings the same way you can control your thoughts or your imagination. Brain reinforces connections between things which happen together, and if you can associate a feeling to a color of the room, then you can do the same with eg imagining an apple, or just with intention/attempt to feel it, and for feeling calm it is the same as for any other feeling.
(!!! But it is mind modification, and a very quick one, and you need to be careful with mind modification in general, and it is not just performative disclaimer, you can accidentaly edit out your joy and become unhappy, or your fear and become reckless on emotional level, or motivation and become passive and not enough motivated to try to stop being passive, with the last being generally a huge special danger of mind modification—you do it from the inside, you modificate your mind with itself, and you are the only person who has direct access to your mind, so if anything goes wrong you may both have broken a tool and a tool to fix the tool, and also your mind is not a neatly organized system with manuals, it is a heap of undocumented spaghetti code created by blind idiot gods of evolution and then neural net training, so everything is going to have weird unexpected dependencies and side effects. So, be careful. Though, unfortunately I can’t recommend something which will explain in more detail on dangers, visceral examples, what you can and cannot do)
I wanted to downvote because of that confident assertion. Or even strong downvote, because actually belief that you can’t control emotion, which I had my whole life until ~3 years ago, was quite harmful for me in comparison with alternative.
But then I read that description of not engaging with thoughts, and it was quite enlightening that it is the exact mistake I was making with many of my thoughts—I was engaging (actually, seems applicable not only to thoughts). I already noticed before that I was making a mistake when on having some thought which is not true I was starting to list all the reasons it is not true, that I was only hebbian enforcing it. But Pavlovian untraining makes it much clearer for me.
Or actually… In fact I see a broader picture. It is not just about reinforcing thoughts. It is about that some thoughts don’t deserve to be engaged with at all. Or actually even more broad. That there exists some property of mind, I was not considering before. Not deciding what to believe is true or which consequences follow from which, but what of those beliefs should fill limited space of your working memory. Which is different from simply deciding which trains of thoughts to explore or even thought control. I don’t know how to convey the subtle difference, but it is in some sense difference between believing something implicitly and believing it explicitly.
So, not in the sense of triage, but in the sense of salience, you can decide how much weight you put for some thoughts.
It seems very insightful for me. Before I would’ve classified any try to not engage with a thought as “trying to avoid thinking about something”, which as heard many times is very bad. But on reflection, prioritizing for what to give your limited mind space is not at all the same as avoiding unpleasant thoughts.
That seems like something incredibly insightful and useful, especially for me personally, so I decided to upvote strongly on the basis of that effect (even if that wasn’t the topic or probably even intention of the post).
(Before writing that comment I want to ask you to please use reactions which exist on lesswrong when you downvote, because otherwise I will have no idea what I did wrong, except “writing a comment on lesswrong”, and what I ought to do differently the next time.)
This is somewhat a side line, but… When reading this post’s point about the difference between lawful and unlawful violence, I could generally follow the logical chain. And somehow it still was surprisingly very unintuitive for me that any such difference exists. It was only the next day when I realized: I live in Russia.
There is a popular proverb in Russia, which translates roughly as “The severity of the law is compensated by no necessity to follow it” (one of Russian spellings: “Суровость закона компенсируется необязательностью его исполнения”).
When I said once here, on lesswrong, IIRC (unfortunately, can’t find the comment) that to say, that war against Ukraine is a war, was made a felony in Russia—I got a very confused reaction from other users. Because—if that was true, wouldn’t that mean that everyone was made a criminal? To me, it was obvious that of course that it is the point! I remember casually hearing in books such phrases as:
“But it is physiologically impossible for human to not do that!” “Yes, and when you will, we will be able to do anything to you, but now it won’t be arbitrary, it will be The Rule Of Law.”
One of the favourite phrases of Putin is “everything should be done in the frames of law”. It is said that in the USA there are too many laws, so you probably broke some of them. In Russia there are such laws that anyone certainly broke any of them. Even long before the war, there were at least three. Felony of fraud is phrased such, that generally any commerce falls under that law. Exactly that law was the one, Navalny was found breaking, when he started to make his investigations. And there also is a felony of extremism—“incitement of hatred or enmity, as well as the humiliation of the dignity of a person or group of persons on the basis of [...], as well as membership in any social group”.
It is said that the legal language is a specification language. But that is in case of countries with actual rule of law. In hybrid regimes like Russia it is exact opposite of that, some “language of non specificity”. In the countries with rule of law the purpose of making laws is so that people don’t break them. But in many other countries the purpose of making laws is so that people break them. Everyone should be a criminal. There shouldn’t be a situation such that person could do something to avoid the “lawful violence”. Laws in hybrid regimes are written so that breaking them will be not avoided, because not avoidable.
Considering that, I now have much better idea, why I have so much of libertarian sympathies. Though, surprisingly it is not because I dealt with too many regulations—I dealt with no regulations. And with wantonly violence. So no regulations and no violence would be a strict improvement for me.
This seemed to me like an interesting observation. And that makes me question, how trying to prevent extinction by law will work in the countries, where there is no actual rule of law. They are probably excluded from the main topic as NOT “countries with functioning police”? Though, there still even before war was a kind of consensus among Russian dissidents that on “while Putin poisons people with polonium, how you dare try to stay peaceful word sayers—you should go cook Molotov cocktails!” there is an answer that “Putin’s regime is a kind of people who is better in a game of violence, not us, if we will move game here, not we will be winners, and ordinary people will never support us if we will start it”. So (at least before the war) Russia was a country where even if you couldn’t do anything to guarantee you being exempt from violence, you still were able to not get attention of government to mostly expect you are safe.
And on the other hand, even if there is no any respect for Law… after failure of Roskomnadzor to block VPNs by tech, government simply ordered IT companies to start blocking their own clients who use VPNs. And they started. So, maybe it will not work worse in Russia, if you will be able to convince Putin that he will not get his “150 years by transplanting other people organs” if the AGI will be created. But it is only Russia, I don’t know about other countries.
I have some experience with talking with llms in English (Grok is just ludicrously worse in all dimensions, when it talks in Russian), so I think I will be able to call out at least some llm flavors (like “it is not [that]; it is [this]!”).
Though, I think I will ask to highlight the edits and then manually apply only them, it seems like a good ratio between efforts applied and having control over resulting text.
I also suspect that in my case it is more than just not being English native. Because before I learned English I for a few years was reading English text with Google translate, and so got completely desensitized to noticing even the most egregious mistakes in text (I especially remember “intelligence is not a superpower” being regularly translated like “spying is not a great state”, which I still got used to automatically interpret in my head).
I am the most uncertain—should I try to pick a shorter piece of my original comment and repost it with revised spelling? Or restrict myself to revising original collapsed comment, and writing something completely new?
Thank you for your feedback. It wasn’t at all in my hypothesis space. English is not my native language, so it is not obvious for me what is easy to read in it (though missing “I” before “have” certainly wasn’t intent!). Maybe I should try to use an LLM for corrections...
After changing mental stack, it seems obvious that for abstract task “writing my thoughts on lesswrong such that they get positive karma in expectation”, giving up after only three posts actually got any karma is just nonsensical. I need a sample of at least 15.
The reason I didn’t notice that before seems to be that humans don’t have fixed preference functions—it changes with mood and feelings. And didn’t notice how my pseudo utility function was hijacked by fleeting feelings in such fashion that it’s change became reflectively stable. But now I am not at all confident that I actually should include those feelings into my pseudo utility function. And if no, it boils down to whether I can effectively ignore them or not have them.
Though… I still feel surprised how pleasant as experience it may be to read posts on lesswrong and how unpleasant to post something of your own. Maybe, I thought they should be positively correlated, but in fact there are mechanisms that make them work as opposites?
And I am still surprised that my posts did get so negative karma. Iirc, I never got so much negative as to be literally hidden. I really, really don’t understand why. It seems in the past I wrote much worse comments, what I write now seems to be much more clear, goal oriented etc, but still it was rated as much worse than anything else I ever wrote. Including a post where I used the sequences as a reason to claim that my political preferences were right all along and I don’t need to change them from arguments.
OK, I can get a hint from karma system—if I get so much downvotes on any long post then I better leave commenting. Maybe I will try Twitter...
Hm. I must notice the is some mismatch between “You have not been a good user on lesswrong” on lesswrong and its lyrics right there. Starts just before first chorus.
But actually, many of the songs are quite good, many even from the first listening. And with more… Aligned lyrics. Before I had only surrogate by DEEP-EX-SENSE and ЛжеДмитрий IV. (Like. Literally. Surrogate is name of the song)
Hm. Fooming Shoggoths certainly self improve very fast, previous year I didn’t like generally any track, but this year I started to like listening to them very quick. I don’t have enough oral English aptitude to comprehend what they actually sing, but I still can become very happy when listening to Fooming Shoggoths.
I think that saying that good calibration is always achievable and good discrimination is hard is wrong.
I long felt like something is off in saying that “no matter what is a coin sequence—you can always just say 50% and be well calibrated even as max entropy distribution”. It seemed for me that if the coin is falling 19⁄20 heads and you say 50⁄50 it shouldn’t be correct, well calibrated, just worse at prediction discrimination than some other probability distributions. And it may be just rationalization for human instinct to see patterns and so treat 19⁄20 as “inherently non random”. Though, when I tried to articulate my intuition I got this:
(I badly remember exact sequence, but...) Consider would be a normal distribution expected from probability 50⁄50 - it would have most variants having near 1:1 ratio between heads and tails. And 19 of 20 is not like this. One of definitions of good calibration is that thing you predict with 50% probability should happen 50% of cases. But if you consider as cases results of those 20 flips, then you will see that it is not 10 to 10 or even close to that, it is 19 to 1.
So, 50⁄50 of randomness will have terrible calibration if coin tends to fall 19 of 20. So it only works if you have some kind of coin which drops HTHTHT. But then we still have generally the same problem—if we just take different sample again, instead of switching to internal sequence cases switch now to samples of every first flip and every second flips, we will notice that 50⁄50 prediction is terribly calibrated, because you predict 50% cases where it is actually 0 or 100% cases.
According to grok, it has zero new math. And still, i watched a video of Yudkowsky from 2009, and he said that, and then in video of (?2024) he said the same. And generally I never saw on lesswrong any refutation of the idea.
What I guess from all that is that probably standard idea of calibration isn’t something special at all. It is more equavalent to one specific kind of checking market price on pattern and possible predictable arbitrage. Generally, every time you don’t have a perfect prediction, somebody can pump out money of you, by finding some pattern in your predictions you didn’t account for.
And to check for absense of any predictable disrepansies of your probability patterns you need to spend some calculations—that is true for standard calibration, and for calibration among internal cases of sequence, and for calibration among samples of each second, and for any pattern, even if it isn’t considered as calibration.
Generally. there may be a sense in saying that calibration is something which can be achieved by running tests on disrepancies on your predictions, and discrimination is something you need empirical data for. But it still needs infinite calculations. Perfect calibration is something achievable only by Solomonoff induction priors and then Solomonoff induction updates after any bit of data.
And also all that means that there is no cheat code of being humbly uncertain, but well calibrated. The only way to not have disrepancies between your probabilities and reality is to predict accurately.
That also reminds me of my old idea that maybe every mental construct should matching something in physical world. Including probabilities. If so, they aren’t measures of your personal uncertainty—their are imperfect predictions of all the frequencies of results to same sensory input as yourself in whole universe. You are objectively in some concrete universe and so your predictions are wrong and exploitable. Though, there is a way to see it as coordination between all the agents having the same sensory input as you to get the best result for sum of them. Where they don’t “update that their anthropic prediction was wrong”, but aknowledge that there is no better way to have predictions among all agents—if you would have made that one better, some other agent with the same sensory input will get worse result.
No, really, can somebody who voted against say why they think my comment is so bad?
When writing I was unsure what is less of a problem: split text on 7 different comments or have one comment be too long? Now it seems that having one long comment was certainly a mistake (as well as lots of P.S.). I am not sure whether it is possible to fix it by reposting it as many comments.
Maybe I wasn’t clear enough, but changing a job is a good political expression gesture if you are proposing to vote for Blue. If you are saying something as extreme as that the world is ending, you should be ready to make signal as extreme as that—changing a job isn’t.
To be clear: by ‘signaling’ I don’t mean that people are calculating how much of hard to fake bayesian evidence changing a job provides, they don’t feel that changing a job anywhere as near worldshattering as belief in world end. Eh. It doesn’t feel that I convey it well. I think of something like person getting cancer diagnosis—people expect that if it is true, then Walter White is maximum level of calmness allowed (as to say—Breaking Bad into a drug dealer).
And now I know why I so rarely try to post (or even comment) on lesswrong. I almost was puzzled why. Now I am not. Just commented something and got karma −4 on it and I have no idea why (Ok, I have too much ideas why, I don’t know which one is right)
Suddenly I have much more appreciation for the terrible system of eg twitter where you can’t downvote things. Probably I should have written it there...
Oh, it is a long post and have a lot of thoughts I wanted to comment about it. Not sure how to organize it all.
But wait! The utter extinction of humanity—argue people who do not believe that premise—is a danger so extreme, that belief in it might possibly be used to argue for unlawful force! By the Fallacy of Appeal to Consequences, then, that belief can’t be true; thus we know as a matter of politics that it is impossible for superintelligence to extinguish humanity. Either it must be impossible for any cognitive system to exist that is advanced beyond a human brain; or the many never-challenged problems of controlling machine superintelligence must all prove to be easy. We cannot deduce which of these two facts is true, but their disjunction must be true and also knowable, because if it weren’t knowable, somebody might be able to argue for violence. Never in human history has any proposition proven to be true if anyone could possibly use it to argue for violence. The laws of physics check whether that could be a possible outcome of any physical situation, and avoid it with perfect reliability.
That whole line of reasoning is deranged, of course.
The problem here is not even so much that it reads more as mockery than as arguments, but, as @Algon noticed about other fragment, that this probably is not at all how described people think. I wasn’t in such cognitive state as described here, so I don’t know, but, as my brain guesses with using automatic empathetic modeling, they probably think more more along the lines of “this belief is obviously not true, so all this unlawful force would be for nothing, so Yudkowsky shouldn’t be allowed to cause it for nothing. And you should avoid dangerous contacts with his arguments, because they are directed into convincing you about super danger of AI, and then you may do unlawful force. Because of a belief which is obviously wrong. And he may start something about epistemology and proper rules of updates, but I can forsee where it is going on—to try to make people believe in violence causing, ridicously implausible belief!” and then they may also go say something about how Yudkowsky shouldn’t tell his false stories about AI danger because they would just depress teenagers for nothing.
I am actually not sure, maybe EY has reasons to put it that way, I had very much cases when he actually had reasons for things his was doing (though in this case my guess would be “I am too tired to make it polished and respectful, so either like that, or I won’t write at all”).
Maybe also there is influence of the fact it is cross posted twitter article. Which actually isn’t marked in any way I have seen, and I found out it just accidentaly, and was confused what is it while reading, because it wasn’t said to be copied series of twitters posts and was too long for that, but was somewhat strange for LessWrong and more reminding twitter. I am actually not even sure now, should I comment here or on twitter? (on both??) Edited: no, really, does someone know some kind of standard policy for that or something?
There’s in fact a difference between calling for a law, and calling for individual outbursts of violence. (Receipt that I am not arguing with a strawman, and that some people purport to not understand any such distinction: Here). Libertarian philosophy aside, most normal ordinary people can tell the difference, and care. They correctly think that they are less personally endangered by someone calling for a law than by someone calling for street violence.
After reading the link, I actually feel being afraid that I in fact totally could also not notice the difference. If I try to figure out why by introspection, it seems that the problem may be in too much using logical mental mode with drawing consequences and too little the mode where you just stare at the thing until considerations pop out. Which actually seems to be a result of that I was trying to not make mistakes of the latter mode such like dismissing MWI because it “seems weird” and bite the bullets instead. But ended up drawing consequences from a mental model which wasn’t account for some intuitively obviously significant factors.
Oh, also I am glad that good writing is back here, unlike Top Tier Intelligence and Fleshling Story posts.
(I probably should have wrote or even posted in parallel to reading, because now I have a feeling I forgot at least one point I wanted to say)
P.S. Oh, and just post comment that it seems by avoiding social networks since I was a kid, because I heard that they are harmful for your mind, I managed to end up having no actual personal experience of what is bad in social networks. But after some examples in that post I actually have better reference point for what was spoken about in “Clickbait Subtly Destroys Intelligence”.
Actually… I looked up and it is “Is Clickbait Destroying Our General Intelligence?”. Which reminds me about another point: I suspect that “bomb data centers” meme causal story was not somebody lying, but somebody recalling by memory without a thought that such serious allegation maybe is worthy to actually look up it and not rely on unreliable memory. Or even not ever hearing about how unreliable memory is and thinking like “I vividly remember it, so it is certainly true”. And then refusing to back off because it would be Dangerous Admitting of Weakness.
P.P.S. Oh, totally forgot. Another moment when I have obvious alternate hypothesis to what these people think is about “you should commit some violence”. Just… I recently first heard about Handicap concept in signaling theory. And it immediately sparked the idea that it would have explained why people talk about all that desperate violence. Because if you people truly believe that the world is ending, then you would be ready to show something more than cheap words, make some great sacrifices which would only people in such a great desperation make. It shouldn’t even be something about AI actually. Just some great expressive sacrifice of truly desperate. Like, maybe start sawing off your own legs. Or stop eating for half a month. Though, okay, that one doesn’t prove how much you truly believe in world end, people who stop eat believe in much less extreme things.
Of course, all this “signaling theory” may be some complicated matter and there is a tiny chance my half second idea was not quite correct. Like, I was surprised that eg Robin Hanson didn’t come to explain why of course you need to start sawing off your legs, or, even more desperate—go throw your own money into heaps and burn them.
It seems to me very similar to some old lw post I have seen about being there some very easy, but very powerful technique (it had also said that it is equivalent of some iirc basketball technique/throw), which is “thinking before doing” or more precisely “visualizing what will likely happen before doing”. Because you didn’t mention it I suppose you didn’t read—unfortunately I failed to find it even using Grok, maybe someone other have read it?
I generally agree that it is very useful. Though I personally try not to ask any questions, but just visualize. Also humans do have natural ability to do that, those automatic anticipations, for example when I think about asking Grok for something often in my imagination automatically pops up a rough picture of its reply which follows some unpleasant pattern and so I drop the idea with disappointment. Unfortunately that happens almost fully outside of focus of attention and very quick and far from always—usually tied to you feeling pain in the past so you flinch from it and/or after many repeats.
I mostly try to develop those natural anticipations into more long, detailed and conscious, because it feels to me like they have some elegant aspect which my intentional tries in visualizing future lack.
I am surprised that nobody wrote about it—on lesswrong, but… Bayes theorem. I remember the story how it was lying in a drawer and found only after Bayes’ death.
Per Grok:
Thomas Bayes developed the core idea in the 1740s (published posthumously in 1763), framing inverse probability to update beliefs given evidence. Pierre-Simon Laplace independently rediscovered and significantly extended it starting in 1774, giving it much of its modern form and broad applications—without apparently knowing of Bayes’ work. This is a classic case of independent rediscovery, but with a key caveat: the gap was substantial (roughly 30+ years from Bayes’ work to Laplace’s publication).
It is certainly not just plaincrash, there is at least also The Erogamer.
If I needed to guess why, I would say that it is not about studying metaphysics per se, but about what people will be ready to give a try for reasoning about realityfluid metaphysics—same kind of people who will give a try writing about it inside of online erotica. Though I am actually even less sure why you can’t publish about realityfluid metaphysics in journals.