I can imagine that an AI first destroys humanity, then poofs away before reaching for the stars, for sure. My point is that the window is quite narrow there.
I can also appreciate the argument that we are special and a space-faring civilization, artificial or otherwise, did not arise until now, or at least not in any visible to us ways, let alone a lightcone-consuming way. But that argument is not really new.
I might be missing your point...
Well, we do know that interstellar travel is possible, just currently very hard. It’s a technological issue, not a physical one. I can certainly imagine slow travel, a few thousands km/s, being possible with nuclear rocket propulsion or other tech. Getting a spaceship-size object traveling at an appreciable fraction of the speed of light is a different story, anything like that would create observational artifacts visible from far away, so those probably aren’t anywhere in the observed past light cone. But I am not sure how this is related to the evaluation of the odds of AIpocalypse.
All good questions! We might be the only recognizably technological civilization out there, any other “life” might occupy something we cannot even tell is life.
I guess my point is that, provided we are not the only ones like us to begin with, concluding that we are still the unique ones that passed the Great Filter makes us pretty unique, and the question becomes “why?”. If AI x-risk is the great filter, then we would see the artifacts of it out there, unless the AGI magically stops growing somewhere between the current level and the observationally visible one.
1) I agree, that is quite possible. Also, a Dyson sphere still radiates JUST AS MUCH OR NEARLY AS MUCH AS THE HOST STAR DOES, that’s just physics. Best you can do is to shift the spectrum to the microwave region, masking it with CMB. But even there one would see odd luminosity bumps from specific directions. Subject to the known physics, which we have no reason to think is violated anywhere.
2) That is a good point. Though we do know where the dust is, from the observations, since it occults many stars at once in a very specific way. Though maybe you are right, I have not looked into it in enough detail. So, if your point is that Kardashev II is so rare that we do not see observational signatures of non-grabby aliens just living their lives, then yes, I guess it is not impossible. I have not checked the literature in the field recently. I guess there might be something like SAI out there that consumed its creators and is sitting inside one or many Dysonspheres, not a bubble of computronium expanding at near-light speed.
That’s definitely possible! And it is a worry. But there is no reason to worry specifically about AI x-risk, since there is no indication that an SAI will have this oddly specific behavior.
I don’t disagree with any of it, my point is that the odds of the AI paperclipping the Earth and then randomly stopping without any observational consequences from far away are pretty low.
Fortunately what you are suggesting is already happening, kind of: the rise of NPs.
Morale comes from having the nice things in your life correlated with effort.
Well said. The flip side is learned helplessness https://dictionary.apa.org/learned-helplessness.
you had (and lost) me at (A → B) → A
Occurred to me that a perfect predictor would not need to go through the ritual of presenting boxes and asking to choose. It already knows the outcome, so it would just give $1000 to those who would two-box and $1M to those who would one-box. The Newcomb’s paradox thought experiment has been thereby dissolved. Thank you for your attention to the matter.
I use the top 4-5 models for fun and profit several hours a day, and my distinct impressions is that they do not CARE. They LARP as a human, but they have no drive and no values. These might emerge at some point, but I have not seen any progress in the last year or so. Then again, progress is discontinuous and hard to anticipate. We are lucky that is the case. In the famous Yud-Karn debate from over a decade ago so far Holden is right: we get smart tools, not true agents, despite all the buzzwords. They don’t care about what is true, or accurate, they hallucinate the moment you are not looking, or don’t close the feedback loop yourself. They could. But they do not. Seems like a small price to pay for non-extinction though.
I once conjectured that
Studying a subject gets progressively harder as you learn more and more, and the effort required is conjectured to be exponential or worse … the initial ‘honeymoon’ phase tends to peter out eventually.
In terms of AI this would mean that the model size/power consumption would be exponential in “intelligence” (whatever it might mean, probably some unsaturated benchmark score). Do the last 3 years confirm or refute this?
If confirmed, would it not give us some optimism that we are not all gonna die, because the “true” superintelligence we cannot ever hope to control would require so much resources, we would have to colonize the lightcone as non-superintelligent humans to get there?
Do you notice the sound your brain makes when you change your mind?
From my very much outside view, extending the rate limiting to 3 comments a week indefinitely would have solved most of the stated issues.
Sorry for the delayed reply… I don’t get notifications of replies, and the LW RSS has been broken for me for years now, so I only poke my head here occasionally.
Well that sounds… scary, at best. I hope you’ve come out of it okay.
50⁄100. But that rather exciting story is best not told in a public forum.
Though these distinctions are kinda confusing for me.
Well, lack of appearance of something otherwise expected would be negative, and appearance of something otherwise unexpected would be positive?
For example, a false pregnancy is a “positive somatization”. Or stigmata. Having trouble coming up with intentionally “good” examples, other than the visualizations helping you shoot a hoop better or something. Not sure if the new-agey “think yourself better” is actually a thing. Hence my question. “Send more blood to your hands” seems like a good example, actually. Not something one would normally think possible except by physical labor.
I really like this post! (I have liked most of your posts of the last decade and a bit. They also inspired me to learn hypnosis, which led to rather cataclysmic changes in my life.) I think therapists call this “somatization”, which can be both positive and negative, in the same sense the hypnotic (or psychotic) illusions are. You seem to mainly focus on the negative somatization (no swelling) and a bit on positive ones, though I suspect that positive somatization (both beneficial and detrimental) is just as controllable with the intent/expectation fusion. Maybe visualizing making the hoop really helps to steady your hand.
Still trying to reconcile “we are in a singularity or shortly will be, where the predictability horizon is shrinking fast” with “and yet we are nearly certain that the SAI will kill us all”. I guess the best analogy I saw is that of a rising river, where you know it will overflow the banks, but you don’t know which stream will break first.