cousin_it
Yeah. I used to be dismissive of AI ethics folks like Timnit, but now I see that they got some things right much earlier than me.
in most cases it’s impossible to tell the difference between the people that decided not to join and the people that just didn’t have the credentials to work there in the first place
I think I’m in both these groups :-)
Maybe part of the answer is giving people an alternative tech tree to work on that doesn’t involve summoning demons, and still provides fun and bragging rights.
My take on mindreading tech was that it would be probably a good thing. Though that particular startup maybe didn’t seem too promising.
Sounds right to me. People are against AI (for whatever reason) and are locally resisting. The more of this, the better. If this happens everywhere and AI becomes as politically toxic as nuclear power in its time, I’ll celebrate.
I don’t understand your comment. If you want to use your property right to build something that’ll hurt a lot of people (like a nuke or an AI), society can override your property right, and that’s ok.
Just found this post. It seems like it’s your central place for evaluating different approaches to IA, so maybe my 2c belong here.
I’m not sure “weaksauce” should be disqualifying. It seems to me that widespread cheap interventions that give +1-2SD would be amazing and open the way for more. My first bet would probably be on pharma and medicine: something like your bit about signaling molecules, or the recent news about Ozempic improving willpower. Medicine also has a strong ethical framework, which helps a lot. My second bet, by the same criteria, would probably be non-invasive BCI.
It sounds like you think one specific drawback of current LLMs—their inability to have some “sleep” and consolidate their memory—will be enough to slow down progress by a decade or more. To me this seems unlikely. People (and LLMs) will be chipping away at this problem pretty fast.
The set of achievable outcome distributions is not convex, since 100% A and 100% C are achievable, but 50% A, 50% C is not achievable.
This makes me wonder about reflective consistency. Say I’m a UDT agent and anticipate being faced with self-coordination problems in the future. Then maybe I should generate a bunch of random numbers today and store them in my mind, so that future copies of me can use them for randomized but self-correlated play. Maybe it can even deal with Wichardt-like examples where the UDT player is faced with other players, if we assume that other players can’t read the random numbers from the UDT player’s mind and can only respond to the “use random numbers” policy in general. Though yeah, I’m not sure how much sense these assumptions make.
Great post again, thank you so much for describing the history in detail.
On the criticism part I agree with you 100%: most of the work at AI labs, including alignment work, has been a bad thing for years. (I’ve been trying to beat that drum on LW for years, too.) But on the constructive part I have some disagreement, and an alternative vision.
You ask: “If not alignment research, then what?” I think a better question would be: “If not AI, then what?” From 10000 feet, a lot of AI’s harm is due to the fact that AI is economically a substitute for humans. If we could shift to technologies that are economically a complement to humans instead—which means basically transhumanist technologies, like thought interfaces or pharmaceuticals or gene therapy—that would give a better path out of the whole crisis, keeping the future human.
The model to imitate here is how the world was steered away from nuclear power and toward renewables. When the anti-nuclear movement started out, renewables were almost as much a joke as transhumanist technologies are today. But due to the “full court press” of the anti-nuclear movement on laws, academia, industry and public opinion, enough researchers and investors shifted to renewables and now it’s quite competitive. So the vision is having a similar public movement against AI tech in favor of human-complementing tech—a stick and a carrot, so to speak. The desired state would be that AI becomes a political dead weight, and the people looking for money or prestige or scientific curiosity flock to human-complementing tech instead. The example shows it can be done and gives an idea of what tactics would be needed, how a large a movement, and how much time.
Ok, I think I understand it a bit better now. But one thing I’d like to note is that UDASSA wasn’t “zoomed out”. One cool feature of the UD is that, given enough observations from our universe (but not the laws of physics), it’ll reconstruct the right distribution for more observations from our universe. It doesn’t need separate levels for “universes” and “stuff within a universe”, it just does everything on one level. So if you replace it with a two-level system, that might be a bit unsatisfying.
Maybe I don’t understand your idea yet. You started with the question whether probabilities are a “reality fluid”, a “measure of caring”, or something else. How would you answer this question for the probabilities of a quantum coin in our universe?
I think this can’t be right, because counting cannot substitute for probabilities even in simple cases. Flip a biased coin. Now there are two worlds, but their probabilities aren’t equal, and might not even have a simple ratio like 70⁄113 or whatever. This can be done with very basic physics, e.g. initialize a qubit, rotate it by some angle and measure it.
This post is surprising to me. In the past few years, models have been getting smarter so fast that superintelligence seems very close, probably no further than 2030. That’s without looking at spending at all (in dollars, flops or whatever). You seem to be saying, based on spending, that it’s actually much further out. How can that be true?
Yeah. I have something very similar in my drafts folder, might as well describe it here instead of making a post.
Basically the idea is to take nuclear vs. renewables as a model. Build a public movement that says one kind of tech (AI) is bad, and another kind of tech (what I’d call “human-potential technologies”—economically complementing humans rather than substituting for them—so stuff like genetic engineering, thought interfaces, pharmaceuticals, prosthetics etc) is good. So that researchers and investors get pushed away from the former and into the latter.
It would probably require a “full-court press” similar to the anti-nuclear movement. Influence the press, academia, industry, laws, politicians all at once. Make AI a political dead weight, and make human-potential technologies the preferred alternative.
The main question is whether human-potential technologies are promising enough. It’s possible that getting the same amount of economic growth out of them requires much more time and money than going for AI. But then again, renewables were also in pretty bad shape when the anti-nuclear movement started. It’s only now, after decades of investment, that they’ve become competitive.
Hmm, I guess it makes sense, but it’s also a bit self-defeating. If you want to argue for the mod policy change, you’ll probably do it best when you’re not as emotional, and not as prone to making long threads—I’m pretty sure they hurt the chances rather than help.
You might get better chances if you make a self-contained post arguing for the change you want, taking the time to think through everyone’s positions as stated, and maybe sending drafts to others for feedback before publishing. No guarantee of success, but it seems like a better approach to me. And if it doesn’t work, then lay the matter to rest.
Given your statements like this and this, can someone please ask 80K Hours to stop recommending joining labs to do safety work. I’ve been unhappy about this for years but I’m a nobody.
I haven’t been banned by anyone on LW yet, but I’ve gotten close to it (people getting angry at me) and I think it wouldn’t bother me much. In my first online community, Russian LiveJournal, I got banned from a few people’s journals and it didn’t bother me much either. A site-wide ban would be a different matter; but catching bans from one or two specific people wouldn’t silence me in any meaningful sense. It’s not a cent from my pocket, not a minute of frustration—to me it’s just nothing. I’m sad that you’re apparently leaving LW over something that, if seen in the right mood, is nothing.
Separate from that is your more general complaint: that authors have a conflict of interest when moderating comments on their own posts, and that makes LW less truth-seeking. And maybe that’s important. But to be honest, I think your view of issue #2 is a bit too colored by issue #1 at the moment—catching a ban is not fun if you’re not used to it. So maybe my advice to you, if you’ll take it, is to try to deal with issue #1 by itself first.
Eliezer uses this mode :-)
I don’t think hunter-gatherers took up farming of their own free will, the lifestyle downgrade is too obvious. More likely some hunter-gatherer bands enslaved others and forced them to farm. If that’s true, there’s no need to imagine an impersonal optimizer arising: the change was caused by specific people, who gained elite status from it.
Just tried to prove it and I think you’re right. For example let’s say the instructions are “jump forward 1337” and “jump back 100″. Then, no matter where you are, “jump forward 100 times and then back 1337 times” would lead to a loop. So there’s a small but fixed chance that the immediately next instructions define exactly this loop, unless we run into a previously visited instruction which means we’re in a loop anyway. So over infinite time, almost all trajectories will fall into this loop or some other one. The proof generalizes to all languages in the obvious way.
EDIT: It’s fun to think about the limits of applicability of this. All that is needed is that instructions are i.i.d., all jumps are relative, and the chance of a loop is nonzero. Other than that we’re pretty free. For example, we could have countably many instructions: jump forward by BB(1) with probability 1⁄2, jump back by BB(2) with probability 1⁄4, jump forward by BB(3) with probability 1⁄8, jump back by BB(4) with probability 1⁄16… Then the expected time to loop is finite and not too large, but the expected distance to the loop is an infinity beyond comprehension.