I am from Russia (not in local lw.ru). I have read HPMoR at age 12 and it immediately became my favourite book and holds it’s place even now. I find Eliezer Yudkowsky’s views on most things the most convincing of all the other I saw.
More detailed version organized as Shortform comments thread: https://www.lesswrong.com/posts/zbSsSwEfdEuaqCRmz/eniscien-s-shortform?commentId=JtpxcmxMt2ycd4K5s
Did you (or other people) try standard reversal test? Ie, ask exact opposite of the question and see whether it will be full of “NO”.
I don’t see why people liked this post so much. I don’t see even what is the thought of the post. (Which though may be problem on my side, not on post’s side, and still.)
Besides, there is already a comment in hpmor itself on this topic:
She was one hundred percent on the side of Harry Potter that it was time for Hogwarts to give up on those gibbering slowpokes and just change the rules, starting here and now. But not by eliminating the Snitch, that was going all the way back to eleventh century Kwidditch. It didn’t matter if Headmistress Hufflepuff had first introduced the innovation because one of her students had wanted to play the game but not been suited to the usual roles. Snitches had caught on internationally because it was more exciting when the game could always end in the next minute.
And also Harry doesn’t seem to behave in character for him:
This argument, unfortunately, had considerably more than two factions. Some darned busybodies were proposing sensible-sounding alternatives to eliminating the Snitch entirely, and this was threatening to split the vote and sap the momentum for reform.
In retrospect, Harry thought, it would have been nice to have Draco unfurl his own banner from the Slytherin side saying ‘SNITCHES ARE AWESOME’, to set the polarity of the debate.
(And also if such potion existed, intelligence explosion would happen long before even Quirrell.)
I have watched Death Note several times. Recently I have read the first book in Vorkosigan Saga. And… I still don’t really understand this metaphor. Why Death Note characters and not any other clever characters, for example. The best idea I have is that Voldemort here is similar to Light because like Light he pretends to try to make the world pure from rot, but in fact just a psychopath who wishes power.
I don’t see how the troll idea worked and not failed. In canon troll was in the school because it is generally is not very well protected and has plenty of dangerous things, and for Quirrell it is in fact just a distraction. But in hpmor Hermione dies not because the school is dangerous, but because the most clever and powerful and immoral wizard in the world decided to kill her, and Dumbledore didn’t let her on holidays where she could have been killed easily by any of the ways and there wouldn’t be Headmaster to arrive immediately after her death, and not say be found to be dead only when there is only a rotten corpse.
Quirrell would be much less likely to make the whole plot just for Hermione spending a couple of weeks in Azkaban. And as he said, he certainly wouldn’t get Malfoys into it just for her. And so even if Harry failed to lose, he wouldn’t he in a giant debt to Lucius Malfoy (who blames Harry for what happened with Draco). So Quirrell wouldn’t need to kill Hermione for Harry to lose debt.
So Harry will never have a reason to have a discussion about Voldemort. He will not have a glimpse on Voldemort cold mind. He would think that the dark lord being alive is just insanity of Dumbledore, his wrong belief in souls and mistake about who is obvious person behind Bellatrix. Harry will not have any reason to ask where from the enemy plans at all. Because there wasn’t any hand of enemy.
There wouldn’t be Aurors in the castle, no Draco going to forbidden forest, because there are no silver slytherins and because Filch doesn’t think Lucius killed his cat, and also Harry will not closely track him because they are not friends. Harry wouldn’t bring the Aurors (because there are no Aurors), will at most just use his time turner to prevent it all. Or likely just nothing, because kids in forbidden forest will not have the mirrors, and so Quirrell will just memory charm them. No encounter with Firenze and Quirrell killing him. And Harry doesn’t quite realize Quirrell is dying, doesn’t bring him a unicorn and doesn’t discuss whether Quirrell intended to kill Auror, to know before Quirrell dies. He isn’t aware Quirrell is dying, they don’t discuss horcruxes. Quirrell doesn’t make an impression of accepting death, so it isn’t strange when at last Quirrell will ask Harry to go for philosopher’s stone.
Then when Quirrell needs Harry in forbidden corridor, he doesn’t need to ask Harry to avoid Constellation (one of suspicious things in the letter), doesn’t need to say to avoid allies of Life Eaters (hint at Quirrell through parseltongue). Harry doesn’t think to take Lesath because there is no rule about going with older students. And Snape questions him and noble kids less. And generally, Quirrell doesn’t really need to bypass Harry’s guards, because there weren’t framing’s and murders, so Harry is not on high guard. And again, without a glimpse on attacking cold mind, Harry thinks Voldemort is just Dumbledore’s insane idea. So Harry easily trusts Quirrell against Dumbledore.
Since Hermione does not die, Harry doesn’t make a firm decision to bring a friend back at all costs. So Quirrell doesn’t hear a prophecy yet that Harry is destined to bring the end to world where Quirrell lives. So Quirrell doesn’t count Harry as enemy. Doesn’t understand even that Harry’s values are incorrigibly different from his, still has hope that Harry his grow out from it. So Quirrell isn’t desperate to tell Harry all the things before he will be ought to kill Harry.
If Quirrell uses Harry as hostage for mirror, then either Harry is unconscious, or maybe even they both face the mirror and Harry thinks Dumbledore is evil for withholding the stone, and maybe even Dumbledore himself thinks that mirror have triggered on Harry’s presence, and after a lecture from Harry that Dumbledore is evil and Voldemort does not exist, Dumbledore decides that it is he the dark lord whom Harry should defeat, and so gives up.
Quirrell doesn’t show that he is the dark lord, still having hope to bring those news after Harry will let go his quibbles. Maybe, to support the idea that Dumbledore was wrong about everything, Quirrell even ditches the identity of Voldemort and instead imagines new funny role, incorporating new years of experience.
Without Draco plot line there are plenty of cascading subtle changes which sum up to entirely different background. The only thing by which Harry suspects Voldemort is suspicious framing of Hermione. Quirrell doesn’t understand that it is not plausible that Lucius will sacrifice his son, that Harry won’t sacrifice Hermione and that Lucius will not wish to accept money. So for Harry it looks like suspicious unknown enemy with cold mind, and not obvious Lucius attack. And without Draco line, Quirrell will likely invent some different plot to hurt Hermione. Say, when she goes to holidays Jagson kidnaps her for a week, while Harry is happily with his family. And Harry doesn’t know Jagson, he doesn’t need to think there is Voldemort’s hand instead of Jagson, as he believes in Voldemort instead of Lucius.
So, no, I disagree that Draco line has no effect. You would need quite a lot of everything going the same way, despite having different premises. Like, Quirrell says that he killed Hermione to remove Harry’s debt. And that plan against just Hermione wouldn’t include Malfoys. And so without saving those motives, story should not just repeat the very same course.
It is certainly not just plaincrash, there is at least also The Erogamer.
If I needed to guess why, I would say that it is not about studying metaphysics per se, but about what people will be ready to give a try for reasoning about realityfluid metaphysics—same kind of people who will give a try writing about it inside of online erotica. Though I am actually even less sure why you can’t publish about realityfluid metaphysics in journals.
It seems to miss the context of OP. Why isn’t math full of universes which very fast dissolve into max entropy thermal baths which exist indefinitely, able to directly generate Boltzmann brains randomly, but unable to have evolution.
I think that a hint for solution may be that many sources of evidence we live in Big Universe imply that if any non tiny amount of universes ended up in Boltzmann generating states, then we will find ourselves as Boltzmann brains in a universe which allows them. So it should not just disprove all physical theories that end up in Boltzmann brains. There should be some factor which makes Boltzmann brains less likely than normal consciousnesses in all of universes, not just in result of physics of our universe.
Also I don’t quite see how you would observe not dissolving into chaos. But there is still the fact that when we look into our past memories, our minds, they look much, much more regular and complex that would you expect is needful for having consciousness (and ability to reason). Boltzmann brain generation seem to imply that it will not produce human brains first, it will produce first near perfectly optimal mind designs with consciousness written in the minimum possible amount of atoms.
This all is new argument for me. I expected universe to exactly be in maximal entropy indefinitely long. And this arguments refutes this possibility.
I am not sure what then. Obvious idea is that maybe even in maximum entropy it is more likely to have a Boltzmann seed which will causally produce a lower entropy bubble than have Boltzmann brain generated directly. But I doubt that possibility of such seeds isn’t exact opposite of “maximum entropy universe”.
I have a mixed feeling about reading that post. The most mixed of all post on lesswrong. In some very high percentile among all I have read or watched.
(edited: I thought that I maybe need to clarify that I don’t have ocd, to avoid confusions)
First, it reads easily and with an interest. So I was going to upvote.
Then there is very confident assurance that you can’t stop having some feeling. Which is simply not true, according to my experience. You can. And even do it immediately.
(And, after talking to LLM about, I suspect that I need to add: not, I am not ‘suppressing’ feelings, in the sense that feeling is not stuffed somewhere outside out attention—it is gone, and physiological signs like tension are gone too and signs like heartbeat calming down at a standard pace, and those feelings don’t try to resurface and even stronger than before, if there is a problem, then it is opposite of this—that they may become too weak or disappear)
You can control your feelings the same way you can control your thoughts or your imagination. Brain reinforces connections between things which happen together, and if you can associate a feeling to a color of the room, then you can do the same with eg imagining an apple, or just with intention/attempt to feel it, and for feeling calm it is the same as for any other feeling.
(!!! But it is mind modification, and a very quick one, and you need to be careful with mind modification in general, and it is not just performative disclaimer, you can accidentaly edit out your joy and become unhappy, or your fear and become reckless on emotional level, or motivation and become passive and not enough motivated to try to stop being passive, with the last being generally a huge special danger of mind modification—you do it from the inside, you modificate your mind with itself, and you are the only person who has direct access to your mind, so if anything goes wrong you may both have broken a tool and a tool to fix the tool, and also your mind is not a neatly organized system with manuals, it is a heap of undocumented spaghetti code created by blind idiot gods of evolution and then neural net training, so everything is going to have weird unexpected dependencies and side effects. So, be careful. Though, unfortunately I can’t recommend something which will explain in more detail on dangers, visceral examples, what you can and cannot do)
I wanted to downvote because of that confident assertion. Or even strong downvote, because actually belief that you can’t control emotion, which I had my whole life until ~3 years ago, was quite harmful for me in comparison with alternative.
But then I read that description of not engaging with thoughts, and it was quite enlightening that it is the exact mistake I was making with many of my thoughts—I was engaging (actually, seems applicable not only to thoughts). I already noticed before that I was making a mistake when on having some thought which is not true I was starting to list all the reasons it is not true, that I was only hebbian enforcing it. But Pavlovian untraining makes it much clearer for me.
Or actually… In fact I see a broader picture. It is not just about reinforcing thoughts. It is about that some thoughts don’t deserve to be engaged with at all. Or actually even more broad. That there exists some property of mind, I was not considering before. Not deciding what to believe is true or which consequences follow from which, but what of those beliefs should fill limited space of your working memory. Which is different from simply deciding which trains of thoughts to explore or even thought control. I don’t know how to convey the subtle difference, but it is in some sense difference between believing something implicitly and believing it explicitly.
So, not in the sense of triage, but in the sense of salience, you can decide how much weight you put for some thoughts.
It seems very insightful for me. Before I would’ve classified any try to not engage with a thought as “trying to avoid thinking about something”, which as heard many times is very bad. But on reflection, prioritizing for what to give your limited mind space is not at all the same as avoiding unpleasant thoughts.
That seems like something incredibly insightful and useful, especially for me personally, so I decided to upvote strongly on the basis of that effect (even if that wasn’t the topic or probably even intention of the post).
(Before writing that comment I want to ask you to please use reactions which exist on lesswrong when you downvote, because otherwise I will have no idea what I did wrong, except “writing a comment on lesswrong”, and what I ought to do differently the next time.)
This is somewhat a side line, but… When reading this post’s point about the difference between lawful and unlawful violence, I could generally follow the logical chain. And somehow it still was surprisingly very unintuitive for me that any such difference exists. It was only the next day when I realized: I live in Russia.
There is a popular proverb in Russia, which translates roughly as “The severity of the law is compensated by no necessity to follow it” (one of Russian spellings: “Суровость закона компенсируется необязательностью его исполнения”).
When I said once here, on lesswrong, IIRC (unfortunately, can’t find the comment) that to say, that war against Ukraine is a war, was made a felony in Russia—I got a very confused reaction from other users. Because—if that was true, wouldn’t that mean that everyone was made a criminal? To me, it was obvious that of course that it is the point! I remember casually hearing in books such phrases as:
“But it is physiologically impossible for human to not do that!” “Yes, and when you will, we will be able to do anything to you, but now it won’t be arbitrary, it will be The Rule Of Law.”
One of the favourite phrases of Putin is “everything should be done in the frames of law”. It is said that in the USA there are too many laws, so you probably broke some of them. In Russia there are such laws that anyone certainly broke any of them. Even long before the war, there were at least three. Felony of fraud is phrased such, that generally any commerce falls under that law. Exactly that law was the one, Navalny was found breaking, when he started to make his investigations. And there also is a felony of extremism—“incitement of hatred or enmity, as well as the humiliation of the dignity of a person or group of persons on the basis of [...], as well as membership in any social group”.
It is said that the legal language is a specification language. But that is in case of countries with actual rule of law. In hybrid regimes like Russia it is exact opposite of that, some “language of non specificity”. In the countries with rule of law the purpose of making laws is so that people don’t break them. But in many other countries the purpose of making laws is so that people break them. Everyone should be a criminal. There shouldn’t be a situation such that person could do something to avoid the “lawful violence”. Laws in hybrid regimes are written so that breaking them will be not avoided, because not avoidable.
Considering that, I now have much better idea, why I have so much of libertarian sympathies. Though, surprisingly it is not because I dealt with too many regulations—I dealt with no regulations. And with wantonly violence. So no regulations and no violence would be a strict improvement for me.
This seemed to me like an interesting observation. And that makes me question, how trying to prevent extinction by law will work in the countries, where there is no actual rule of law. They are probably excluded from the main topic as NOT “countries with functioning police”? Though, there still even before war was a kind of consensus among Russian dissidents that on “while Putin poisons people with polonium, how you dare try to stay peaceful word sayers—you should go cook Molotov cocktails!” there is an answer that “Putin’s regime is a kind of people who is better in a game of violence, not us, if we will move game here, not we will be winners, and ordinary people will never support us if we will start it”. So (at least before the war) Russia was a country where even if you couldn’t do anything to guarantee you being exempt from violence, you still were able to not get attention of government to mostly expect you are safe.
And on the other hand, even if there is no any respect for Law… after failure of Roskomnadzor to block VPNs by tech, government simply ordered IT companies to start blocking their own clients who use VPNs. And they started. So, maybe it will not work worse in Russia, if you will be able to convince Putin that he will not get his “150 years by transplanting other people organs” if the AGI will be created. But it is only Russia, I don’t know about other countries.
I have some experience with talking with llms in English (Grok is just ludicrously worse in all dimensions, when it talks in Russian), so I think I will be able to call out at least some llm flavors (like “it is not [that]; it is [this]!”).
Though, I think I will ask to highlight the edits and then manually apply only them, it seems like a good ratio between efforts applied and having control over resulting text.
I also suspect that in my case it is more than just not being English native. Because before I learned English I for a few years was reading English text with Google translate, and so got completely desensitized to noticing even the most egregious mistakes in text (I especially remember “intelligence is not a superpower” being regularly translated like “spying is not a great state”, which I still got used to automatically interpret in my head).
I am the most uncertain—should I try to pick a shorter piece of my original comment and repost it with revised spelling? Or restrict myself to revising original collapsed comment, and writing something completely new?
Thank you for your feedback. It wasn’t at all in my hypothesis space. English is not my native language, so it is not obvious for me what is easy to read in it (though missing “I” before “have” certainly wasn’t intent!). Maybe I should try to use an LLM for corrections...
After changing mental stack, it seems obvious that for abstract task “writing my thoughts on lesswrong such that they get positive karma in expectation”, giving up after only three posts actually got any karma is just nonsensical. I need a sample of at least 15.
The reason I didn’t notice that before seems to be that humans don’t have fixed preference functions—it changes with mood and feelings. And didn’t notice how my pseudo utility function was hijacked by fleeting feelings in such fashion that it’s change became reflectively stable. But now I am not at all confident that I actually should include those feelings into my pseudo utility function. And if no, it boils down to whether I can effectively ignore them or not have them.
Though… I still feel surprised how pleasant as experience it may be to read posts on lesswrong and how unpleasant to post something of your own. Maybe, I thought they should be positively correlated, but in fact there are mechanisms that make them work as opposites?
And I am still surprised that my posts did get so negative karma. Iirc, I never got so much negative as to be literally hidden. I really, really don’t understand why. It seems in the past I wrote much worse comments, what I write now seems to be much more clear, goal oriented etc, but still it was rated as much worse than anything else I ever wrote. Including a post where I used the sequences as a reason to claim that my political preferences were right all along and I don’t need to change them from arguments.
OK, I can get a hint from karma system—if I get so much downvotes on any long post then I better leave commenting. Maybe I will try Twitter...
Hm. I must notice the is some mismatch between “You have not been a good user on lesswrong” on lesswrong and its lyrics right there. Starts just before first chorus.
But actually, many of the songs are quite good, many even from the first listening. And with more… Aligned lyrics. Before I had only surrogate by DEEP-EX-SENSE and ЛжеДмитрий IV. (Like. Literally. Surrogate is name of the song)
Hm. Fooming Shoggoths certainly self improve very fast, previous year I didn’t like generally any track, but this year I started to like listening to them very quick. I don’t have enough oral English aptitude to comprehend what they actually sing, but I still can become very happy when listening to Fooming Shoggoths.
I think that saying that good calibration is always achievable and good discrimination is hard is wrong.
I long felt like something is off in saying that “no matter what is a coin sequence—you can always just say 50% and be well calibrated even as max entropy distribution”. It seemed for me that if the coin is falling 19⁄20 heads and you say 50⁄50 it shouldn’t be correct, well calibrated, just worse at prediction discrimination than some other probability distributions. And it may be just rationalization for human instinct to see patterns and so treat 19⁄20 as “inherently non random”. Though, when I tried to articulate my intuition I got this:
(I badly remember exact sequence, but...) Consider would be a normal distribution expected from probability 50⁄50 - it would have most variants having near 1:1 ratio between heads and tails. And 19 of 20 is not like this. One of definitions of good calibration is that thing you predict with 50% probability should happen 50% of cases. But if you consider as cases results of those 20 flips, then you will see that it is not 10 to 10 or even close to that, it is 19 to 1.
So, 50⁄50 of randomness will have terrible calibration if coin tends to fall 19 of 20. So it only works if you have some kind of coin which drops HTHTHT. But then we still have generally the same problem—if we just take different sample again, instead of switching to internal sequence cases switch now to samples of every first flip and every second flips, we will notice that 50⁄50 prediction is terribly calibrated, because you predict 50% cases where it is actually 0 or 100% cases.
According to grok, it has zero new math. And still, i watched a video of Yudkowsky from 2009, and he said that, and then in video of (?2024) he said the same. And generally I never saw on lesswrong any refutation of the idea.
What I guess from all that is that probably standard idea of calibration isn’t something special at all. It is more equavalent to one specific kind of checking market price on pattern and possible predictable arbitrage. Generally, every time you don’t have a perfect prediction, somebody can pump out money of you, by finding some pattern in your predictions you didn’t account for.
And to check for absense of any predictable disrepansies of your probability patterns you need to spend some calculations—that is true for standard calibration, and for calibration among internal cases of sequence, and for calibration among samples of each second, and for any pattern, even if it isn’t considered as calibration.
Generally. there may be a sense in saying that calibration is something which can be achieved by running tests on disrepancies on your predictions, and discrimination is something you need empirical data for. But it still needs infinite calculations. Perfect calibration is something achievable only by Solomonoff induction priors and then Solomonoff induction updates after any bit of data.
And also all that means that there is no cheat code of being humbly uncertain, but well calibrated. The only way to not have disrepancies between your probabilities and reality is to predict accurately.
That also reminds me of my old idea that maybe every mental construct should matching something in physical world. Including probabilities. If so, they aren’t measures of your personal uncertainty—their are imperfect predictions of all the frequencies of results to same sensory input as yourself in whole universe. You are objectively in some concrete universe and so your predictions are wrong and exploitable. Though, there is a way to see it as coordination between all the agents having the same sensory input as you to get the best result for sum of them. Where they don’t “update that their anthropic prediction was wrong”, but aknowledge that there is no better way to have predictions among all agents—if you would have made that one better, some other agent with the same sensory input will get worse result.
No, really, can somebody who voted against say why they think my comment is so bad?
When writing I was unsure what is less of a problem: split text on 7 different comments or have one comment be too long? Now it seems that having one long comment was certainly a mistake (as well as lots of P.S.). I am not sure whether it is possible to fix it by reposting it as many comments.
I agree with the basic idea that requirements for llm tests are absurdly strict to the degree that the testers themself likely wouldn’t pass them and end up with concluding that they demonstrated “no signs of consciousness or self recognition”.
Humans also hear the consciousness talk all the time, human brain naturally mimics speech pattern of other humans, so if you will use that as an excuse to explain away llm behaviour, you ought to use that to humans as well, and then the only line of reasoning at implying humans are conscious is based on first person perspective (I can see that I am conscious) plus expectation that consciousness is a physical phenomena and therefore will be similarly exist or not for all humans because we have very similar brain structures.
However, the clearly proved outer functional self awareness in mirror tests (and many other tests) doesn’t of course prove mirroring of inner mental mechanism of this self awareness (if we have thought like that, we would implied that llm outward good behaviour also implies inner good thoughts, wishes etc!), and even more doesn’t tell about consciousness.
There certainly exists such factor that we don’t really know how self awareness works in animal brain or even human brains. We know though that humans often confabulate their “introspections” (the most notorious example is cut hemispheres experiments, of course) and which is quite an evidence for previous “humans mimic what they heard and therefore you can’t infer they are conscious from their talk about consciousness”. It is clear that plenty times humans “introspect”, “report conscious experiences” they actually just confabulate likely explanations quite like llm confabulate next tokens. So, we actually know that having sometimes confabulating consciousness talk does not imply no consciousness.
None of us is llm so we can’t know the same way we know about humans, because llm architecture is too different, while human architecture is really similar because of evolutionary reasons. Though, people often lack some experiences, for example imagination (purely mental sensory experience) in aphantasia, and those people mimick usual people so closely that usually can’t be distinguished even after they start directly talking like “what mental pictures you are implying, there is none such, that is just a metaphor”, so I personally can’t rule out that some people lack conscious experiences, treat it as metaphor and mimic others behaviour through some side ways.
I am not sure how I should proceed. Llm are taught on reflection and consciousness examples as well as humans, they are (almost) certainly confabulate new such as well as humans. But we don’t have personal, unsharable, anthropic, point of view data from an architecture similar enough to LLM, while we do for humans. Llm being able to confabulate it should certainly increase likelihood of hypothesis that it explained by confabulations, because at least some of it certainly is. But I hugely doubt that it implies that we should because of it think that llm has same or less probability of no consciousness, as if we didn’t know about confabulations in architecture, but llm also didn’t demonstrated any such signs as mirror test etc. It seems to me they still should make it more likely in general that llm have inner self awareness and possibly consciousness.
And certainly it seems to me absurd when the same people simultaneously confidently say how we know about not just self awareness, but (!) consciousness of animals, and also confidently say how we know about llm not being self aware or conscious.
So I think it is good to point out that tests of same strictness would as well prove non self awareness and unconscious not just animals, but humans too. This is not a good test if it can’t show different results for different cases, and for most humans it would show they are not conscious despite they are. And conclusion is even worse. Because of connotations. Like getting 9 years old, asking him to discover relativity theory in a month, and that it without other humans or internet, because we tested them with it and they casually passed the test just in 5% of allocated time, and when they fail you draw conclusion that… “9 year olds didn’t show any signs of intelligence.” (like if adults would show any such signs) More correct conclusion for consciousness would be that consciousness and self awareness signs weren’t should to be vastly exceeding those for humans (and therefore we can not conclude whether it is their own consciousness or copy of that of humans).
P. S./Edited: I have a meta about the post form. While reading I anticipated it will be likely you will not make an obvious caveat that it doesn’t mean llm didn’t just outputting likely token in very similar manner to how they do with all the other tokens, with no relation something specific to actually recognizing self and just being a kind of echo matcher, or trigger on text repetition or something (which it seems to me you didn’t give), or even imply that actually simple mirror test should prove llm are self aware and conscious (which to my relief you didn’t say). So I felt an urge to add that caveat for the case of reader missing it. And it seems other commenters also felt that urge, and I suppose because of that they seem to me to oppositely lean too much into comparing llm with python program triggered by input of repeated text.