I am from Russia (not in local lw.ru). I have read HPMoR at age 12 and it immediately became my favourite book and holds it’s place even now. I find Eliezer Yudkowsky’s views on most things the most convincing of all the other I saw.
More detailed version organized as Shortform comments thread: https://www.lesswrong.com/posts/zbSsSwEfdEuaqCRmz/eniscien-s-shortform?commentId=JtpxcmxMt2ycd4K5s
Just now was incorrect while highly confident on clearer thinning calibration test. With 90% confidence said it is Britain, while in fact it was… England.
I’d already formed from personal observation of Fable 5 and Sol 5.6, the August 2026 generation of frontier publicly purchasable AI models.[1]
I think I actually saw that in Twitter, iirc, long before the hf incident going public.
I just got recommended a video about what happens with Minecraft world after a long time. And I realized what meant one thing I felt is wrong with it’s world: it isn’t in equilibrium. The world isn’t self sustaining. Game mechanics don’t move into same stay as generation creates. The most vivid example of it is villages: they used to appear in good state with villagers, houses and agriculture intact. But there is no mechanisms in the game which would actually support that state. If you will place for some time you will notice this world didn’t exists before you joined.
I had a thought like “and now I live in a new world, where AI solves world class math problems” and then I noticed that actually… No. This moment was long time ago, with Alpha Fold. It was already allowed for AI to solve these kinds of problems. And just by some I reason I didn’t truly update on that.
It doesn’t seem to me like realistic sci-fi? Say we get furthest 14B years point, then 1% difference gives 140M years, usually it is pretty little for minds to appear, at evolution iirc works as exponent for time, so 140M is much, much less than usual time.
Though, if somehow it increases speed of evolution very significantly, then it may work. In theory, starting evolution of predictable lifeforms can be a way to do it. Maybe even better that what I had in the head, because evolution can by itself arrive into more ordered states. I just don’t really know what can work here. And whether it will be effective. What I had in my head is that maybe you can somehow smoothen the homogenize the material by right radiation, and then apply a pattern which structures it… I thought that for ASI which works at speeds of transistors and diamondoid nanobots, even one second can be billions of pattern tries on just one mm^2 of surface, and for whole planet’s surface it probably can form needed nano structures in maybe less than a day. So it will be saving time even for 1 lightyear distance. I certainly didn’t think it would need something even remotely on orders of magnitude of time when life is forming.
Hm. I long had a certain thought and this seems like a fitting post to attach it as comment.
Background:
I noticed that most people tend into direction of “ah, this is all sci-fi”, but I myself rather tend in the opposite. Ideas like “but what if ASI will modulate a signal via CPU or RAM” seems to me like totally natural. What I actually think is more like “Can ASI via right images make neurons in the brain create certain chemicals which will interact in certain ways such that in the end they will change the DNA or assemble into viruses or prions which will make more direct routes for chemical manipulation via images, until ASI will be able to make your body cells turn into controllable nanotech by just giving you some right perceptual input”. And… I can’t rule out that this is possible for mere ASI (as opposed to Solomonoff induction).
So, this is in a sense opposite of what post talks about—things which actually were proven to already have happened. But imo it is just the result of most people being way too inclined in the first direction—if they were asked about ai swarms and solving novel math just 5 years ago, they would confidently say it can’t be true anywhere soon. Like some dialogue I have seen in comments (~5 years old ones) to a video about “pizza police call” where skeptic was “this is why you shouldn’t let machines work in accepting calls, they would never guess that” and enthusiast was “oh, never is too strong word, ai may learn to do that some time, not in 10 years or even 30, but 100 years later it can...”. And just 1.5 years after that gpt4 comes out.
So the actual example of thing possibly possible for ASI I keep thinking of is this:
It is usually said that ASI will send near lightspeed nanotech probes to other stars. But shouldn’t there be something further than molecular nanotechnology? Some… not “quantum tech”, if continue “molecular nano” it should be atomic piko- and subatomic femto- technology. Can ASI learn QM better than humans such that it would not need to create robots from atoms in the first place, but use some semi stable resonances or something in sophisticated way? I don’t know enough physics to rule it out, but it seems plausible enough for me that it will be possible (and used) at least to some degree—humans failed even in thermonuclear synthesis even though it already exists in the nature.
And if we take another step here, then moving at almost lightspeed is in some sense very different from exactly lightspeed. And you can’t send anything with clock still running at the speed of light, because it would mean the path is not the shortest anymore. But what you don’t actually need to, if you are ASI? It has some old data for predictions and very good knowledge of the physics and a computational power in the size of stars. What if it can send particles with no clock moving such that they will assemble into right form already in the place? It doesn’t need to be exact, it can use a lot of different combinations. And by “a lot” I of course don’t mean trillions, I mean orders of magnitude more.
Say, we have a star 1 lightyear away, then we moving at 99% of lightspeed would mean you lose 3.65 days, which is enormous time for ASI. A star emits billions of times more light than planets get. If you can redirect it all, you would get that much energy for particles to send, that it will immediately turn Earth’s surface into plasma. You actually will use only tiny part of the light which star emits in 3.65 days. And you still may have plenty of space for different setups for different variants of conditions. The usual case is when you are limited in action with plenty of knowledge and need to choose actions wisely, which is solvable to significant degree, but why should the opposite be unsolvable—when you have plenty of actions, but you are almost blind (especially if we save not 3 days on 1 year distance in time, but more like 30 billions of days for stars which are leaving the Hubble sphere).
ASI needs to make different variants not destroy each other. But it doesn’t actually need to just do that absolutely blindly. There are some priors, calculations of which planets likely form where, distributions of objects on the surface. And ASI only need to search for the most present, the most invariant ones.
So, maybe you shouldn’t actually expect to see traces of ASI activity, Because the moment they can reach you, you already will be reached by a flash of optimised signals which hack most likely computers or most likely electrochemical brains of animals. Or just a flash of particles which evaporates most of the surface where it didn’t found proper objects to latch and so everyone dies in one second.
Because, anyway, even if it fails then you just turned into vapor a civilization with differing utility function, which is also good. And advanced civilization will know about that tactic and configure themselves to not be harmed, but to collect the energy as a payment for perfoming some cheap components of AI utility function.
There of course are problems with that, like gas between galaxies and other distortions. Maybe, it will need to be multi step, with some assembling right in the space, to reconfigure. But it easily may worth it, you just need to spend less than 3.65 days on it for each lightyear and less than radiation of a star in 3.65 days of energy.
Another caveat is for example that ASIs and civilizations may have logical agreements not to vaporize living creatures or sentient or sapient creatures. But I am not sure about it.
This seems to remind me arguments like “ah, but what if the ASI doesn’t care about lots of, but about only one thing”. Then you still have maximizing the probability of the local thing. Just, why not to do that when it is about all your preferences? You won’t really have anything competing, which usually makes humans not to pursue tiny chances for particular things.
And similar way, even if aliens quantum suicide or something, there is still no reason for them not to leave behind an ASI which will improve the universe on margins to their tastes.
Update: I am very surprised by +14. I almost never get anything even close. And I thought this post was quite bad. Bad writing. Plus I fumbled the formating. And I didn’t set any tags. I didn’t even check whether I am right and MIRI still have a problem. Or research the legal status of gift/prepaid cards via Google pay and whether Google pay works for donations at all.
I also thought so, not just because of Russian x5 decrease or because some people here can donate millions unlike me.
But because I have much more knowledge about AI and all other things than average person. And… Probably smarter than average person?
So, it is quite clear that my edge is some personal work, not money. But what work? Politics certainly isn’t my strongest side, I am afraid I will make it worse. I doubt technical alignment makes sense since how little time is left. And also I am not actually a mathematician, or, say, biologist. So I don’t know how I should apply my time.
Yeah, it bothered me to, ever since 2023 I doubted it actually makes sense to donate MIRI or try to apply as alignment researcher, because there is just so little time left. And then I found out that MIRI mostly stopped all the research. Exactly because of those reasons. And pivoted to political action because it seemed that normal people notice the AI and are more sane about it’s dangers than tech bros.
But even so, EY still named “Superbabies” as third most important project on the planet. And recently I saw the post saying that gene therapy for intelligence enhancement in babies is worthy to do just because alignment won’t work with no pause either, and if we do have 30-50 years, then baby gene therapy is just vastly under researched and under funded in comparison with eg alignment, so even little addings can be vast marginal improvements. So maybe I shouldn’t try to donate on AI either in technical or political sense, but in those. But I don’t know how to donate for those.
I know at least about examples of OpenAI, Anthropic. (In some sense Deep Mind.) Though iirc EY was saying that OpenAI and Anthropic are terrible ideas from the very beginning. I heard iirc EY saying in interview some time ago that there is only one organization except his MIRI which actually has positive impact on ai/makes real alignment—Redwood research. But I am not sure how to donate to them, whether they still have positive impact and whether technical donations are worth in comparison to political.
Um. No, obviously not just ip is the problem? Stats show that more every third Russian (including kids and elderly) regularly use VPN. I personally just don’t turn it off, since the moment when YouTube was blocked (also Facebook, Twitter, medium are blocked too, most ai services not accessible). So, not, it isn’t a question of vpn.
Russia is under sanctions, swift doesn’t work, Russian cards are not accepted for payments. There were ways around it for a long time, like applying for a card of foreign bank (usually you need to, like, sent a copy of your international passport via the post and pay around 500$ dollars just to make it, which for a Russian is as expensive as 2500$ for an American). There are gift card, usually connected to Google pay to hide the fishy payment card from the one who accepts the payments. And there are prepaid virtual cards (usually you pay crypto) which sometimes even elegible for cashing out in ATMs.
But… I am not even sure about the legal status of all that. And even if it is technically legal, I don’t want to make problems for charities by donating 100$ from Russia in some weird way.
Though iirc donations in crypto from Russia are perfectly OK—sometimes sites display wrongly without the VPN, but it isn’t any legal obstacle. But some sites like iirc MIRI use mechanisms like coinbase, and coinbase doesn’t serve clients in Russia.
When I looked at many AI markets on manifold, it looked like they are significantly underpriced, both cybersecurity and math. It seems I better updated from (surprising to me) Jacobian conjecture than many other people. But considering how surprised I am by current results, it seems I have updated nowhere enough.
And I don’t really understand how to do better in updates than I did previously.
My own take about the question to Irving (I write before reading part III, don’t know whether I did understand that part correctly, it seems I see something which seems correct, but I don’t know whether I see right thing and whether I actually comprehend it):
Obvious answer that immediately comes into head is that they need a verifier (as is, the way to reliably check the concrete solutions, even if you don’t have reliable way to generate the solution). Second obvious thing to say is that judges should be able to check the same bits of evidence as empirical method does. (Because just being unbiased doesn’t return you correct answers, coin flips are unrelated to solutions truth, they are don’t have a bias, which just means max entropy distribution instead of confident wrongness, not distribution which guesses right. They are calibrated, not discriminative.) Which is, they should be correct, in the sense that their estimates of arguments are bayesian evidences for AI behaviour (they are proportionally more frequent in worlds where AI behaves well/badly).
This was also surprising to me, I often thought that being unbiased is enough. Though, not about alignment proposals and mostly not “debates” in general, I long noticed that while on lesswrong people talked about dangers of AI 20 years, majority… Well, majority of all people believes into god. And scientists, even physicists, believe into Collapse. So it doesn’t matter that you know the truths of the universe, if people aren’t gonna believe you about it, as they don’t believe about MWI.
(I actually can imagine how Claude 7 Fairtale comes out, and people on lesswrong listen to it, and people outside laugh, because LWers “rearranged their typical personality cult of Yudkowsky into typical AI cult of Claude and now believe his fairytales”. It wouldn’t matter if Claude 7 favors LDT or something, people will rather take as evidence that people on LW are not different in intelligence from stochastic parrots.)
Actually, as I think, judges explicitly should be statistically biased. Biased to truth, their thoughts should be correlated with the world outside their heads.
Another thought references to what I thought a long time ago. That even being logically omniscient (or rather probabilistically omniscient, because logical omscience about surrounding world is prohibited by Godel’s theorem) isn’t enough, multiverse isn’t entirely deterministic, some parts exist both, equally in each their variant, and about those you can’t find out logically, or rather, you can’t have the strategy which will work equally good for two agents in differing worlds (which I suppose is “no free lunch”). So you need to use not only deduction data, but observation data too. Without it you will only have perfect prior, like Solomonoff’s induction, but still will not know about the world with hidden parameters. At the very least you won’t be able to predict future quantum splits. And actual judges aren’t deductively omniscient. So you will need them to have correlations with both deterministic and random data.
Though I think perfectly unbiased judge will at least give you perfect priors. Because I think that simplicity prior follows just from rules like P(A)>=P(A)&P(B).
Though it isn’t obvious to me what is meant by the question being almost exactly equal to perpetual motion mechanisms. Obvious shallow guess is that it is about “local validity”—the judges shouldn’t allow the reasoning steps which don’t conserve information. Though I don’t see the analogy.
And the problem with logical validity is that math proofs are incredibly expensive for real world. The same reason why games don’t just use physics equations to create game works, but use different hacks.
But I don’t think that it is possible to have a system which will work via debate, because of memetics—for any heuristic criteria there will be some invalid arguments which will fit the lock of that heuristic as good as valid arguments. So AIs (or anything else) trained to max out the judges will find arguments which maximally correlates with heuristic, and that will unlikely correlate with truth because there is limited fitness space (and because of phase shifts like that gradually it becomes easier and easier to inject drugs into humans than make them smile by jokes or make fake money instead of working or mass make articles in journals instead of doing science, and then you just switch the strategy into some completely uncorrelated with help to initial cause).
Another shallow idea is that you should just treat alignment the same way as physics. And that any approach should give concrete answers how to solve the hard problems of alignment. Like that you have only one real try and you need it work then, that you can’t iterate wrong ideas by empirical trial and error like in all the other science. And as it was said before, empirical trial and error of science is the only method humanity has for valid group agreement on things. And that you need some method to choose good AIs even though they may behave good only before elections.
The part about “nose of Chinese emperor whom no one have seen” seems to hint at exactly empirical data you can’t just deduce. It isn’t congruent with the problem as that humans allow invalid arguments, which would be about logical validity. But it is congruent with the idea that the only method shown in practice to work for groups isn’t consensus of philosophers, but empiricism of science. Which you can’t really allow for ASI, because you die. And you can’t easily extrapolate from usual AI to ASI because of many phase shifts/qualitative differences between system stupider than humans and smarter, capable of seizing power and incapable.
But I doubt this is the “hard to see thing”, I basically said it in the first guess. (maybe the thing just isn’t hard to see for people who are not Yudkowsky) And I have serious doubt about whether I truly comprehend it, instead of guessing the password by repeating it in many shallow ways using different terms with semantic similarity (like correlation/evidence/related/frequent or logical/prior/calibrated).
My answer on a koan:
I actually don’t know how I can be sure it isn’t, if he is a Nobel laureate in physics. If he isn’t, then I guess—on priors, true breakthrough from anonymous people are rare enough to call any individual case as sure “no”. If I still need some better guess… I don’t see anything except circular reasoning about how “if it wasn’t they just losing track in something too complex, they would start from saying they know a concrete crux”. I… Don’t see reasons to try further because I probably already know what answer I will get, so anyway won’t be able to contrast.
Update, writing directly as thinking:
Ah, my bad, I should have pointed out here that “it follows from the Noether’s theorem” is deeper than just observing all previous times that energy is conserved. (I guess I was swayed by that the question was featuring Newton’s laws and I went all like “Noether something something”) So… My analogy with my failure about qm doesn’t apply. Or, as I saw Richard Feynman saying, it would violate not just known laws of physics, but the character of laws of physics. There can be universes with different types of fundamental interactions, but all base level universes should have no privileged frame of reference, work on quantum mechanics, be analytic, have ^2 metric, have no time travel or even time loops etc. On the other hand… As I heard, Noether theorem doesn’t actually work, eg red shift, because it applies only to universes where space doesn’t change with time, and in our it expands, so conditions are not abided. It does seem like a kind of unobvious thing physicist could know that I didn’t know, and possibly exploit. So maybe in the end there isn’t much difference.
Now write before reading further: if he was Noble laureate in physics then, yeah, I wouldn’t be sure (either way). Noble physicist may know physics better than me. Maybe something unobvious is going on.
What do I even imply by that “unobvious”. Well, if to think about that… I have an embarrassing story from when I was a kid, I learned about “quantum mysticism” and your observation affecting the reality and how it was just like my mother’s belief that thoughts affect reality and so I concluded that obviously all the effects should be because on quantum scale all things are so little that any measurement device, which is itself similar size quant, will strongly affect the system. And one person on internet argued with me how I am wrong. And then they said “I am actually a physicist, I made experiments with molecules of fullerene, which themselves emitted photons, and it still mattered whether those photons were captures or not”. I had no idea how to explain that. I didn’t reply, but mostly I suspected that the person on internet is straight up lying about all those experiments, what is the chance that random person on internet is an actual physicist?
And it turned out that something unobvious was happening, and physicists could know why my “obvious explanation” was unviable, while from what I knew it should have been what was happening, if you measure things by lighting it via “tiny insignificant” photons. But it turned out it works the same way even if you emit nothing into particle. (and what actually works is entanglement)
After reading the spoiler:
ok, it seems to be the speed up. It was anticlimactic. After all the warning about what if you confidently point out wrong flaw, I started to suspect it was harder than seemed. But it looks like the inventor doesn’t notice even very obvious flaws.
Reading next:
OK, it was indeed more complicated than inventor not knowing about speed up not being free. I actually should have expected it is going to be that, I have read “bayesian mechanism” and “giant systems of gears” and what you should have one simple crux which explains why your system isn’t restricted by that and as I read somewhere I can’t remember where that AI developers are also prone to make their systems so complicated that they can’t keep track of where it has broken. So I should’ve expected it may be about that.
OK, I write my recollection of my guess, for accountability. After reading that wrong guess is better than no guess at all, but before expanding the spoiler. Iirc, I thought like “speed up?.. But why? Continuous speed is default, but suddenly changing speed would violate ~(Noether something) because it is ~(not invariant something), ~(diverging something). Don’t see why the speed up, is it implied be something like fluid speed change in tighter pipes? But that shouldn’t break the physics. Ah, I don’t really thoroughly understand the setup (*I didn’t sleep good today), but as a whole it seems to simple and easily definable as sheer math structure to actually find the exploit in physics, so I am too lazy to check it carefully. Hm, what if I will ask a grok like ‘I have an idea for reactionless drive: a circle...’ which should give it hints to flatter user about idea user has, but still maintains plausible deniability because I didn’t say I invented this idea or believed it, just that I had it, which I indeed do since I read and copied it. I wonder whether it will go ‘you have here a brilliant idea’ as llms always do. Huh! No, it straight says that no, it is classics that doesn’t work, there are forces needed for speed up and slow down which cancel it out. Well, interesting. Maybe idea is ~(too cliche) to trigger flattery. I guess it is speed up indeed what is the problem, since we both agreed. Though I shouldn’t trust an llm.”
I still don’t see why it is wrong to reason from him naming the drive for himself. It isn’t big sign per se, but it seems to me to be a big sign in combination like “someone unknown proposes world shattering invention which he named after himself”. World shattering inventions by not already established people are much more rare than crackpots who dream of being world shattering and writing their name on it. While if someone is more self aware than that, it should eliminate most crackpots, and then amount of people who with no hedge propose world shattering inventions should be way less, so it should make huge difference in Bayesian sense, as it seems to me. Though, the primary reason would still be “it is unlikely on priors”.
Also, I have no idea about the difference between 1996 and now, I didn’t live in 1996.
Maybe with meta-honesty it should be solved something like that meta-honesty means what it seems to mean, but to save the same thing that EY wanted, you have a remark that meta-honesty doesn’t negate simple honesty, you don’t get to say “ah, but I was meta-honest in the sense that I didn’t tell anyone that what I say is truth, therefore they shouldn’t have expected it as such”. It seems to me to perform the same role without adding some unobvuous meaning into the term. Sort of, relying on the same thing as that if you say how you are able to take out someone’s mother from burning house, that doesn’t mean that you can just do it as a corpse, your claim about ability here should mean something additional, having all else equal. And same for meta honesty, saying you are honest meta you don’t get to use some language of your own invention where yes means no or anything like that. This seems as intuitive way to put it, keeping both the idea that you don’t get to lie by simply not claiming your words to be true and having the term definition not be a complicated unexpected bundle.
I guess it is indeed a mistake. I had an unchecked subconscious background assumption on which I blindly relied—that different spellings would fit.
I guess, the right thing to do is reformulate questions and calibration from “what answer is correct” to “what answer the test maker had in the head”. And ok, I think I know now how swarm agents felt.