no clue what package I’d have to install to get
pip- neitherpipnorpip3exist
On Ubuntu, at least, there’s a python3-pip package, separately from the python3 package? (Other distros may be similar.) It’s also supposed to be possible to install pip using Python itself.
It doesn’t help when Yudkowsky actively encourages this confusion! As he Tweeted today: “Anyways, Scott, this is just the usual division of labor in our caliphate: we’re both always right, but you cater to the crowd that wants to hear it from somebody too modest to admit that, and I cater to the crowd that wants somebody out of that closet.”
Just—the absolute gall of that motherfucker! I still need to finish my memoir about why I don’t trust him the way I used to, but it’s just so emotionally hard—like a lifelong devout Catholic denouncing the Pope. But what can you do when the Pope is actually wrong? My loyalty is to the truth, not to him.
The very concept of a “rationalist” is an egregious one! What is a rationalist, really? The motte: “one who studies the methods of rationality, systematic methods of thought that result in true beliefs and goal achievement”. The bailey: “a member of the social ingroup of Eliezer Yudkowsky and Scott Alexander fans, and their friends.”
Sorry, the function of bringing up Three Worlds Collide was to point out the apparent contradiction in the Yudkowskian canon. Forget the story; I agree that fiction didn’t happen and therefore isn’t evidence.
The actual issue is that it seems like worlds shaped by the goal systems of other evolved biological creatures probably don’t “contain almost nothing of worth”: the lives of octopuses mean much less to me than human lives, but more than tiny molecular paperclips. The theme of “animal-like organisms that feel pleasure and pain” is something that natural selection will tend to reinvent, and the idealized values of those organisms are not a random utility function. (Do you disagree? If so, you at least face a Sorites problem on how fast value drops off as you look at our evolutionary history. Do chimpanzees matter? If not, did Homo erectus?) But if other animals aren’t literally-as-valueless-as-paperclips, then some classes of AI architecture might not be, either.
Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.
Did anyone notice that this flatly contradicts Three Worlds Collide? The superhappies and babyeaters don’t inherit from human morals at all (let alone detailedly and reliably), but the humans still regard the aliens as moral patients, having meddling preferences for the babyeater children to not be eaten, rather than being as indifferent as they would be to heaps of pebbles being scattered.
(Yes, it was fiction, but one would imagine the naturalistic metaethics of aliens were meant to be taken at face value, even if there is no Alderson drive and the evolutionary psychology resulting in baby-eaters specifically was literary casuistry.)
So if the moral of this post isn’t quite right, how should it be revised?
“Any Future not shaped by a goal system with detailed reliable inheritance from human morals will be incomprehensibly alien and arbitrary-looking in a non-valued direction, even after taking into account how much you think you’re ‘cosmopolitan’; furthermore, AI goal systems are expected to look even more arbitrary than those of biological aliens, which would at least share the design signature of natural selection”?
How much is this modified moral weakened by potential analogies between natural selection of biological creatures, and gradient-descent or self-play AI training regimens?
The Categories Were Made for Man, not Man for the Categories, Scott Alexander
The correctness of that post has been disputed; for an extended rebuttal, see “Where to Draw the Boundaries?” and “Unnatural Categories Are Optimized for Deception”.
A key psychological advantage of the “modest alignment” agenda is that it’s not insanity-inducing. When I seriously contemplate the problem of selecting a utility function to determine the entire universe until the end of time, I want to die (which seems safer and more responsible).
But the problem of making language models “be honest” instead of just continuing the prompt? That’s more my speed; that, I can think about, and possibly even usefully contribute to, without wanting to die. (And if someone else in the future uses honest language models as one of many tools to help select a utility function to determine the entire universe until the end of time, that’s not my problem and not my fault.)
Another illustration: if you’re currently falling from a 90-story building, most of the expected utility is in worlds where there coincidentally happens to be a net to safely catch you before you hit the ground, or interventionist simulators decide to rescue you—even if virtually all of the probability is in worlds where you go splat and die. The decision theory looks right, but this is a lot less comforting than the interview made it sound.
even buying all the arguments of the race/IQ chaps hook, line, and sinker (which, to reiterate, neither I nor any rationalist I’ve ever spoken to or read actually does)
In public. Absent some kind of epic infosec fail (like a disgrunted acquaintance leaking private emails), you can’t know what beliefs people might have that they’re incentivized not to talk about. To this you might reply, “Absence of evidence is evidence of absence: without a reason to think that people are holding back, you could just as baselessly speculate that maybe rationalists are secretly Satanists or flat-earthers.”
Is there any such reason? I don’t know. There is an interesting Scott Alexander post, “Kolmogorov Complicity And The Parable Of Lightning”, about the pragmatic necessity for intellectuals to avoid contradicting powerful orthodoxies. But doesn’t that seem like a weirdly specific abstract topic for Scott to write about, unless he had some concrete contemporary example in mind? What do you think he was thinking of?
the differences are tiny and absolutely dwarfed by within-race variance anyway, so just wouldn’t make much difference to anything.
Sort of. Statistical group differences can be large enough to have social consequences in at least some contexts, while also being small enough such that the correct Bayesian update about an individual based on group membership is small (not zero, but small). For cognitive ability measures, the current black–white gap in the U.S. is about 0.85 standard deviations (Cohen’s d), and the white–Asian gap is at about d ≈ 0.3. Is that “tiny”? I don’t know; depends on how you want to use words. For comparison, the male–female height gap is about d ≈ 1.5. Physical sex differences like the fact that men are taller than women on average seem like the kind of background common knowledge that is sometimes decision-relevant (e.g., to product designers) even if tall women exist and you can notice someone’s height independently of their sex (um, almost; empirically, your brain does do a Bayesian group-membership base rate adjustment without your conscious awareness).
“Ed Teller Syndrome”, named after one of the worst [...] humans in history, afaict
One of the worst humans!—because of the H-bomb, presumably? Does intent matter at all? I feel like MAD-style theorizing (“If we have a slim chance of survival, it lies in the possibility to get rid of wars”) and fear of the Soviets getting there first (“Would I have given the same answer if my friend Tisza had not told me about the persecution of Lev Landau in the Soviet Union?”) are sufficiently sympathetic motivations to escape the worst humans shortlist.
Ugh, maybe this wasn’t such a good strategy. After downloading all the message pages and catting them in a loop, I did “successfully” end up with a 128 MiB HTML file …
But, first of all, it somehow ended up with a lot of escape characters (\n, \') displayed literally in the page. (Not sure how that happe—oh. The str(message.content) in my script should have been a message.content.decode().) Second of all, this enormous file is not actually smooth to read on my machine (it just crashes in Firefox, and Chromium is very laggy). On net, this may not actually be an improved experience over just clicking through the messages on sl4.org. (Third, Mediafire’s pop-up ad when you click the “Download” button feels kind of scummy, but it was the first convenient way that came to mind for sharing a large file.)
How much are you offering? I downloaded by the by-date archive page and wrote this Python code to download the individual message pages, with the thought of catting them together to get your “single file.” (Which is an ugly hack and not even technically valid HTML, but it worked when I’ve done it in the past.) But it’s only halfway through downloading and I need to turn the computer and its noisy fan off and go to sleep now; this should be easy to finish tomorrow.
import re
import requests
# http://sl4.org/archive/date.html
with open(“sl4_archive_date.html”, ‘rb’) as toc:
archive_page = str(toc.read())
results = re.findall(r”http://sl4.org/archive/\d+/\d+.html″, archive_page)
for i, result in enumerate(results):
message = requests.get(result)
with open(“messages/{:07}.html”.format(i), ‘w’) as f:
f.write(str(message.content))
a real preference to stay in the current state [...] financial markets[ are] the ur-example of efficiency and inexploitability, yet it turns out that a market does not have a utility function in general [...] hidden state creates situations where the agent “prefers” to stay in whatever state they’re in.
Wait, does this help solve the problem of fully updated deference?!
My summary: one of the reasons to expect AI to be dangerous by default is that smart agents are coherent, coherent agents are expected utility maximizers, and sufficiently powerful expected utility maximizers destroy everything in their path: even if you’re my best friend in the whole actually-existing world, if I’m sufficiently powerful, I’m still going to murder you and use your atoms to build my best friend across all possible worlds.
This would seem to sink alignment schemes of the form “Just make the AI uncertain about the correct utility function, and have it ask us questions—and let us modify it if necessary—when it’s not sure.” Becuase if it’s sufficiently powerful, it’ll just disassemble our brains to figure out how we would have responded to questions, without letting us modify it. (Which would be fine if the value-loading problem were already completely solved—disassemble us now and resurrect us after it’s done turning off all those negentropy-wasting stars—but you’d want to be very sure of that, first.)
But if the “coherent agents are expected utility maximizers” part isn’t true because state-dependent preferences are still coherent in the relevant sense, doesn’t deference/corrigibility potentially become a lot easier? In some sense, you just (“just”) need one of the subagents on the committee to veto all plans that prevent us from hitting the Off switch … right?
(Separately from the violence issue, it looks like “the community” has an initials collision: I initially read “RB” as “Rob Bensinger” rather than “Ruby Bloom.”)
How hard is it to make the Curation admin tooling include a text field for curation reason, that both gets posted as a comment, and included in the email? This really seems like a nice-to-have software feature request, rather than something that merits the grotesque violence of editing the OP!
Strongly recommended: rose-colored glasses are Redshift/f.lux for the whole world, not just screens.
I’m mildly disappointed with my Oura ring, although I still use it and don’t actively regret purchasing it; I like having “objective” information on my sleep (gathered from actual sensors on the ring, not just me looking at the clock before going to bed and having to guess how long it took for me to go down), but the distillation from objective sensor data to the information the app shows isn’t super high quality: the sleep-stages breakdown barely registers any REM at all (which I don’t think is biologically plausible), and sometimes it fails to register periods of sleep (e.g., if I woke up at 5 a.m., but am sure I dozed off again from about 6 to 7:30, the latter doesn’t register as part of my night’s sleep, although sometimes it gets picked up as “Rest”). It’s interesting that I seem to spend less time “objectively” asleep than I thought, while still seeming to function OK: the app claims I’m averaging 4 hours, 9 minutes of sleep in 5 hours, 21 minutes in bed per night, which, while surely an underestimate, is less than I would have expected after taking into account how often it seems I was clearly asleep but the app didn’t pick it up. I appreciate the developer API on principle, even if I don’t have any practical need for it. (I did write a program to Tweet about my sleep, but that was also just on principle.)
Programmers operating with partial insight, create a mind that performs a number of tasks very well, but can’t really handle self-modification let alone AI theory [...] This scenario seems less likely to my eyes, but it is not ruled out by any effect I can see.
Twelve and a half years later, does new evidence for the scaling hypothesis make this scenario more plausible? If we’re in the position of being able to create increasingly capable systems without really understanding how they work by throwing lots of compute at gradient descent, then won’t those systems themselves likely also be in the position of not understanding themselves enough to “close the loop” on recursive self-improvement?
Yeah, the exaggeration didn’t seem like a crux for anything important.
Okay, I can see that, but as a writing tip for the future, rhetoric in the vein of “We are the great hope of our civilization” looks heavily optimized for the feeling-good-about-group-identification thing, rather than merely noticing the startling fact of being somewhat influential. And the startling fact of being somewhat influential makes it much more critical not to fall into the trap of valuing the group’s brand, if the reputational pressures of needing to protect the brand make us worse at thinking.
My guess is that “Msr.” was intended as an abbreviation for monsieur (French for mister), in ignorance of the fact that the standard abbreviation is actually just M.