If this question becomes important, there are people in our community who are.. domain experts. We can ask
Yonatan Cale
Karma: 285
Hey,
TL;DR I know a researcher who’s going to start studying C. elegans worms in a way that seems interesting as far as I can tell. Should I do something about that?
I’m trying to understand if this is interesting for our community, specifically as a path to brain emulation, which I wonder if could be used to (A) prevent people from dying, and/or (B) creating a relatively-aligned AGI.
This is the most relevant post I found on LW/EA (so far).
I’m hoping someone with more domain expertise can say something like:
“OMG we should totally extra fund this researcher and send developers to help with the software and data science and everything!”
“This sounds pretty close to something useful but there are changes I’d really like to see in that research”
“Whole brain emulation is science fiction, we’ll obviously destroy the world or something before we can implement it”
“There is a debate on whether this is useful, the main positions are [link] and [link], also totally talk to [person]”
Any chance someone can give me direction?
Thx!
(My background is in software, not biology or neurology)
I heard “kiwi” is a company with a good reputation, but I didn’t try their head strap myself. I have their controller-straps which I really like
(I’m not sure but why would this be important? Sorry for the silly answer, feel free to reply in the anonymous form again)
I think a good baseline for comparison would be
Training large ML models (expensive)
Running trained ML models (much cheaper)
I think comparing to blockchain is wrong, because
it was explicitly designed to be resource intensive on purpose (this adds to the security of proof-of-work blockchains)
there is a financial incentive to use a specific (very high) amount of resources on blockchain mining (because what you get is literally a currency, and this currency has a certain value, so it’s worthwhile to spend any money lower than that value on the mining process)
None of these are true for ML/AI, where your incentive is more something like “do useful things”
+1 for the Abusive Relationships section.
I think there’s a lot of expected value in a project that raises awareness to “these are good reason to break up” and/or “here are common-but-very-bad reasons to stay in an abusive relationship”, perhaps with support for people who choose to break up. It’s a project I sometimes think of opening but I’m not sure where I’d start
Anonymous question (ask here) :
Given all the computation it would be carrying out, wouldn’t an AGI be extremely resource-intensive? Something relatively simple like bitcoin mining (simple when compared to the sort of intellectual/engineering feats that AGIs are supposed to be capable of) famously uses up more energy than some industrialized nations.
If you buy a VR (especially if it’s an Oculus Quest 2), here’s my getting started guide
Just saying I appreciate this post being so short <3
(and still informative)
Ok,
I’m willing to assume for sake of the conversation that the AGI can’t get internet-disconnected weapons.
Do you think that would be enough to stop it?
(“verified programmatically”: I’m not sure what you mean. That new software needs to be digitally signed with a key that is not connected to the internet?)
But don’t you think “reverse engineering human instincts” is a necessary part of the solution?
I don’t know, I don’t have a coherent idea for a solution. Here’s one of my best ideas (not so good).
Yudkowsky split up the solutions in his post, see point 24. The first sub-bullet there is about inferring human values.
Maybe someone else will have different opinions
TL;DR: Hacking
Doesn’t require trial and error in the sense you’re talking about. Totally doable. We’re good at it. Just takes time.
What good are humans without their (internet connected) electronics?
How harmless would an AGI be if it had access merely to our (internet connected) existing weapons systems, to send orders to troops, and to disrupt any supplies that rely on the internet?
What do you think?
Update: Anthropic’s own computers are connected to the internet. link. This was said publicly by the person in charge of Anthropic’s information security.
[extra dumb question warning!]
Why are all the AGI doom predictions around 10%-30% instead of ~99%?
Is it just the “most doom predictions so far were wrong” prior?
I’d be pretty happy to bet on this and then keep discussing it, wdyt? :)
Here are my suggested terms:
All major AI research labs that we know about (deep mind, openai, facebook research, china, perhaps a few more*)
Stop “research that would advance AGI” for 1 month, defined not as “practical research” but as “research that will be useful for AGI coming sooner”. So for example if they stopped only half of their “useful to AGI” research, but they did it for 3 months, you win. If they stopped training models but keep doing the stuff that is the 90% bottleneck (which some might call “theoretical”), I win
*You judge all these parameters yourself however you feel like
I’m just assuming you agree that the labs mentioned above are currently going towards AGI, at least for the purposes of this bet. If you believe something like “openai (and the other labs) didn’t change anything about their research but hey, they weren’t doing any relevant research in the first place”, then say so now
I might try to convince you to change your mind, or ask others to comment here, but you have the final say
Regarding “the catastrophe was unambiguously attributed to the AI”—I ask that you judge if it was unambiguously because AI, and that you don’t rely on public discourse, since the public can’t seem to unambiguously agree on anything (like even vaccines being useful).
I suggest we bet $20 or so mainly “for fun”
What do you think?
My answer for myself is that I started practicing: I started talking to some friends about this, hoping to get better at presenting the topic (which is currently something I’m kind of afraid to do) (I also have other important goals like getting an actual inside view model of what’s going on)
If you want something more generic, here’s one idea:
Do you mean something like “only get 100 paperclips, not more?”
If so—the AGI will never be sure it has 100 paperclips, so it can take lots of precautions to be very, very sure. Like turning all the world into paperclip counters or so
I don’t know, I’m replying here with my priors from software development.
TL;DR:
Do something that is
Mostly useful (software/ML/math/whatever are all great and there are others too, feel free to ask)
Where you have a good fit, so you’ll enjoy and be curious about your work, and not burn out from frustration or because someone told you “you must take this specific job”
Get mentorship so that you’ll learn quickly
And this will almost certainly be useful somehow.
Main things my prior is based on:
EA in general and AI Alignment specifically need lots of different “professions”. We probably don’t want everyone picking the number one profession and nobody doing anything else. We probably want each person doing whatever they’re a good fit for.
The amount we “need” is going up over time, not down, and I can imagine it going up much more, but can’t really imagine it going down (so in other words, I mostly assume whatever we need today, which is quite a lot, will also be needed in a few years. So there will be lots of good options to pick)
You suggested:
But if we were to work on it today, it would only have a sub-human level, and we could iterate like on a child
But as you yourself pointed out: “We are not sure that this would extrapolate well to higher levels of capability”
You suggested:
and we had “Reverse-enginered human social instincts”
As you said, “The brain’s face recognition algorithm is not perfect either. It has a tendency to create false positives”
And so perhaps the AI would make human pictures that create false positives. Or, as you said, “We are not sure that this would extrapolate well to higher levels of capability”
The classic example is humans creating condoms, which is a very unfriendly thing to do to Evolution, even though it raised us like children, sort of
Adding: “Intro to Brain-Like-AGI Safety” (I didn’t read it yet, seems interesting)
I personally like the idea of uploading ourselves (and asked about it here).
Note that even if we are uploaded—if someone creates an unaligned AGI that is MUCH SMARTER than us, it will still probably kill us.
“keeping up” in the sense of improving/changing/optimizing so quickly that we’d compete with software that is specifically designed (perhaps by itself) to do that—seems like a solution I wouldn’t be happy with. As much as I’m ok with posting my profile picture on Facebook, there are some degrees of self modification that I’m not ok with
Anonymous question (ask here) :
Why do so many Rationalists assign a negligible probability to unaligned AI wiping itself out before it wipes humanity out?
What if it becomes incredibly powerful before it becomes intelligent enough to not make existential mistakes? (The obvious analogy being: If we’re so certain that human wisdom can’t keep up with human power, why is AI any different? Or even: If we’re so certain that humans will wipe themselves out before they wipe out monkeys, why is AI any different?)
I’m imagining something like: In a bid to gain a decisive strategic advantage over humans and aligned AIs, an unaligned AI amasses an astonishing amount of power, then messes up somewhere (like AlphaGo making a weird, self-destructive move, or humans failing at coordination and nearly nuking each other), and ends up permanently destroying its code and backups and maybe even melting all GPUs and probably taking half the planet with it, but enough humans survive to continue/rebuild civilisation. And maybe it’s even the case that hundreds of years later, we’ve made AI again, and an unaligned AI messes up again, and the cycle repeats itself potentially many, many times because in practice it turns out humans always put up a good fight and it’s really hard to kill them all off without AI killing itself first.
Or this scenario considered doom? (Because we need superintelligent AI in order to spread to the stars?)
(Inspired by Paul’s reasoning here: “Most importantly, it seems like AI systems have huge structural advantages (like their high speed and low cost) that suggest they will have a transformative impact on the world (and obsolete human contributions to alignment retracted) well before they need to develop superhuman understanding of much of the world or tricks about how to think, and so even if they have a very different profile of abilities to humans they may still be subhuman in many important ways.” and similar to his thoughts here: “One way of looking at this is that Eliezer is appropriately open-minded about existential quantifiers applied to future AI systems thinking about how to cause trouble, but seems to treat existential quantifiers applied to future humans in a qualitatively rather than quantitatively different way (and as described throughout this list I think he overestimates the quantitative difference).”)