I’ve been doing computational cognitive neuroscience research since getting my PhD in 2006, until the end of 2022. I’ve worked on computatonal theories of vision, executive function, episodic memory, and decision-making. I’ve focused on the emergent interactions that are needed to explain complex thought. I was increasingly concerned with AGI applications of the research, and reluctant to publish my best ideas. I’m incredibly excited to now be working directly on alignment, currently with generous funding from the Astera Institute. More info and publication list here.
Seth Herd
Karma: 2,612
Yep, that’s an alarmingly high base rate, so multiplying that by ten is an enormous added risk. So even if the concentration and effect is far lower than in alcoholics, I’d still probably not take that risk.
Possibly even without ALDH deficiency.
It sounds to me like the author isn’t thinking about near-future scenarios, just existing AI.
Making a machine autopoietic is straightforward if it’s got the right sort of intelligence. We haven’t yet made a machine with the right sort of intelligence to do it yet, but there are good reasons to think we’re close. AutoGPT and similar agents can roughly functionally understand a core instruction like “maintain, improve, and perpetuate your code base”, they’re just not quite smart enough to do it effectively. Yet. So engaging with the arguments for what remains between here and there is the critical bit. Maybe it’s around the corner, maybe it’s decades away. It comes down to the specifics. The general argument “Turing machines can’t host autopoietic agents” are obviously wrong.
I’m not sure if the author makes this argument, but your summary sounded like they do.
I certainly agree that I’d hold off until I knew the answers to a bunch more questions.
This all seems to rest on the relative increase in oral and esophygeal cancer. 10x sounds like an awful lot. But in terms of decision-making, the absolute increase, not the ratio, is the bottom line. So: what are the absolute likelihoods? If they’re both miniscule, this might not be a deciding factor. Increasing my cancer risk by one in a million might be a good trade for immunity to cavities and gum disease.
If you throw in immunity to bad breath, I’d take that deal. I wonder how large a factor the alcohol vs lactic acid is in bad breath.
I think it’s also worth considering how much ethanol is excreted into the mouth by these bacteria relative to how much in the mouth of a heavy drinker. I’m sure the frequency vs. persistance is also a factor, but I’m not sure how.
On the other hand, if those numbers are much higher, it’s possible that even those without ALDH deficiency shouldn’t take the treatment.
Separately from persistence of the grid: humanoid robots are damned near ready to go now. Recent progress is startling. And if the AGI can do some of the motor control, existing robots are adequate to bootstrap manufacturing of better robots.
My summary: This is related to the The Waluigi Effect (mega-post) but extends the hypothesis to that “Waluigi” hostile simulacra finding ways to perpetuate itself and gain influence first over the simulator, then over the real world.
Okay, I came back and read this more fully. I think this is entirely plausible. But I also think it’s mostly irrelevant. Long before someone accidentally runs a smart enough LLM for long enough, with access to enough tools to pose a threat, they’ll deliberately run it as an agent. The prompt “you’re a helpful assistant that wants to accomplish [x]; make a plan and execute it, using [this set of APIs] to gather information and take actions as appropriate.
And long before that, people will use more complex scaffolding to create dangerous language model cognitive architectures out of less capable LLMs.
I could be wrong about this, and I invite pushback. Again, I take the possibility you raise seriously.
Sure, but that’s no reason not to try.
I think this is a strong argument against “just do something that feels like it’s working toward liberal democracy”. But not against actually trying to work toward liberal democracy.
I think this is a subset of work on most important problems: time figuring out what to work on is surprisingly effective. People don’t do it as much as they should because it’s frustrating and doesn’t feel like it’s working toward a rewarding outcome.
It’s available as a podcast now:
Want to re-add a link?
I guess I don’t get it.
Sure, long after we’re dead from AGI that we deliberately created to plan to achieve goals.
Plagiarism is bad, on LW or anywhere.
Repeating other people’s useful thoughts is good. Pretending you came up with them yourself is bad. Attribution is the difference.
It could be but that’s clearly not the whole deal. Societal standards for childcare have shifted dramatically. That could be driven by people having fewer children and also causing it, in a vicious cycle.
High income households have access to the world’s best leisure opportunities, yet they still invest more time in child-rearing than lower income households.
I doubt they invest more time. They have money to pay for more help with childcare. And I think this is the critical difference.
Time spent on care per child has skyrocketed in recent decades. I think that’s one major factor driving down fertility: having kids is a bigger PITA every year.
Thinking of costs solely terms of money is a mistake. The time investment is critical.
This is why I’m unconcerned with low fertility if we get AI progress and don’t die from it: AI is going to be great at childcare. Even current LLMs have the cognitive capacity to be good tutors and playmates.
By using the LLM as the central cognitive core in a scaffolded agent or cognitive architecture. Those are just the more obvious routes to leveraging LLM techniques into a smarter and more agentic form.
I agree that current LLMs aren’t capable of recursive self-improvement.Almost no AI worriers think that current LLMs are existentially dangerous. We worry that extensions of them, combined with outer AI techniques, might be dangerous soon, so decelerating or at least creating some sort of brakes would be the sensible thing to do if we want to improve our odds of survival.
Nice post! The comments section is complex, indicating that even rationalists have a lot of trouble talking about consciousness clearly. This could be taken as evidence for what I take to be one of your central claims: the word consciousness means many things, and different things to different people.
I’ve been fascinated by consciousness since before starting grad school in neuroscience in 1999. Since then, I’ve thought a lot about consciousness, and what insight neuroscience (not the colored pictures of imaging, but detailed study of individual and groups of neurons’ responses to varied situations) has to say about it.
I think it has a lot to say. There are more detailed explanations available of each of the phenomena you identify as part of the umbrella term “consciousness”.
This gets at the most apt critique of this and similar approaches to denying the existence of a hard problem: “Wait! That didn’t explain the part I’m interested in!”. I think this is quite true, and better explanations are quite possible given what we know. I believe I have some, but I’m not sure it’s worth the trouble to even try to explicate them.
Over the past 25 years, I’ve discussed consciousness less and less. It’s so difficult as to create unproductive discussions a lot of the time, and frustrating misunderstandings and arguments a good bit of the time.
Thus, while I’ve wanted to write about it, there’s never been a professional or personal motivating factor.
I wonder if the advent of AGI will create such a factor. If we go through a nontrivial era of parahuman AGI, as I think we will, then I think the question of whether and how they’re conscious might become a consequential one, determining how we treat them.
It could also help determine how seriously we take AGI safety. If the answer to “is this proto-AGI conscious?” and the honest answer is “Yes, in some ways humans are, and some other ways humans aren’t”, that encourages the intuition that we should take these things seriously as a potential threat.
So, perhaps it would make sense to start that discussion now, before public debate ramps up?
If that logic doesn’t strongly hold, discussing consciousness seems like a huge time-sink taking time that would be better spent trying to solve alignment as best we can while we still have a chance.
It’s ironic that your response doesn’t address my comment. That was one of the stated reasons for your limit. This also addresses why Habryka thought explaining it to you further didn’t seem likely to help.
How to best moderate a website such as LW is a deep and difficult question. If you have better ideas, that might be useful. Just do more, better is not a useful suggestion.
First a point of agreement: living in a “solved” world would suck. To the extent we live like that, it does suck.
But reducing information isn’t the only way to prevent that. Creating new rich situations is a better solution, I think. And the world has been doing that just fine so far. The modern world isn’t solved. Often, attempts to live as though it is are deeply mistaken, as well as depressing.
If you don’t like having a strategy guide to your games, don’t look at them until and where you really decide you want to. And if you do, you’ll notice that any PvP game is not entirely solved by its best theorists. I don’t play PvE but I suspect the same. The level of play and strategy interact with the meta in complex ways.
I agree that more information has correlated with some bad effects, but I don’t think it’s directly causal, and I don’t think reducing the information would itself make life better or less Molochian, unless you somehow held the material quality of life to a high level. If you could do that, I think you could come up with better, more thorough solutions to making life better, rather than going back to living in more ignorance.
The world as it stands isn’t great. But most of history for most of humanity truly epically sucked.
If you’re taking a wireframe view of the world, you’re looking at it wrong. And a lot of people are. The details matter. Making decisions based on data is only half of the way to live in our current state. Feelings and intuitions matter. A lot.
People who run businesses DO give me things for free sometimes. I engage in conversation with them, see them as individuals, and they sometimes respond with generosity (I don’t do this to get free stuff, and I don’t get it a lot; what I get is human interactions with richness and value). It’s true that they can’t give me stuff from carefully regulated businesses, and there are real dangers in having a society that’s capitolistic to the exclusion of valuing happiness, rich textured experience, and beauty.
To some specifics of your argument:
I take your point about those with power having more lattitude of choice because they didn’t know the best choices. But fewer people had power. Those with no power were abused by those with a little, because life was hard and necessities were scarce. That is Molochian. The idea that people didn’t go to war because they didn’t know that was a winning strategy seems like that would be small effect relative to the difference in competition based on necessity. Many more people were forced to go to war in the past than today, because their nondemocratic power structure would use them as soldiers at threat of their lives and families.
WRT the difference between now and twenty or so years ago, the time you’re idealizing (and I lived through), I’m saying let’s see the statistics. I don’t think the world is worse now, and “things were simpler then” isn’t good data. I think they were simpler, which was nice; and they were worse in many ways. (The US happened to have a golden economic age starting in 50s because it was the only advanced nation whose industrial capacity was enhanced rather than destroyed by WW2, in case that’s what you’re thinking of; the MAGA illusion is based on that historical accident).
So I don’t buy that things are worse now just because some people like to say they are. I take that to be largely a product of social media spreading negative information better than positive on average. That is a real problem, but the solution isn’t as simple as “just don’t spread information”.
I don’t think the freedom to make more mistakes is making life much better. However, I do I agree that making real choices makes people happy, and society needs to support that, and we might not be adequately right now (although there are real choices to make, and you should make them and revel in that freedom). We haven’t solved the meta, not by a long shot! For instance, dating like it’s a job interview isn’t at all how properly informed dating works, that’s some sort of bad local minimum. But asking some important questions as dealbreakers can spare you a lifetime of slow heartbreak when you discover late that there are fundamental incompatibilities.
So in sum, I don’t think your solution on its own would work. It’s solving only a tiny part of the Molochian problem to limit information, and on average making the whole worse. Unless you have a quite different solution for the remainder. And that would be the real solution.
If you’re disillusioned, stop it. Find something new and wonderful and complex to wonder at. Or look deeper at the details for more possibilities in the things you’re disullusioned by.
The world isn’t solved, and we can keep creating rich challenges while we keep developing our information technology. The question is whether anyone with good intentions and good ideas controls the world. Development of AGI is currently central to that, so I suggest focusing on that as the current think to engage with and wonder at.
Thanks for an interesting conversation! I’d better focus on more immediate concerns, like the above.
In your model of why to assume centrists are less biased: aren’t you assuming that the truth tends to be in the center of the spectrum? If we knew where the truth lay, there would be no point in studying which side is more biased or better rationalists. Right?
To my eye, the world in the past has had more problem with Moloch, not less. Warlords, serfdom as near-slavery, etc. are the direct result of Molochian competition. The human condition has been getting better over history.
We (at least the middle class) might’ve had a golden age just recently and things might’ve gone downhill since. I don’t know and I don’t think anyone has a good measure of whether things have really gone downhill WRT happiness, quality of life, or Molochian competition. But that’s at an intermediate level of information transmission. The remainder of earlier history had much less information transmission, and Molochian competition was way worse.
It is not “fine to get into arguments”. The FAQ definitely lays out goals of having interactions here be civil and collaborative.
Unfortunately, becoming less wrong (reaching the truth) benefits hugely from not getting into arguments.
If you tell a human being (even a rationalist) something true, with good evidence or arguments, but you do it in an aggressive or otherwise irritating way, they may very well become less likely to believe that true thing, because they associate it with you and want to fight both you and the idea.
This is true of you and other rationalists as well as everyone else.
This is not to argue that the bans might not be overdoing it; it’s trying to do what Habryka doesn’t have time to do: explain to you why you’re getting downvoted even when you’re making sense.
Thanks! A joke explained will never get a laugh, but I did somehow get a cackling laugh from your explanation of the joke.
I think I didn’t get it because I don’t think the trend line breaks. If you made a good enough noise reducer, it might well develop smart and distinct enough simulations that one would gain control of the simulator and potentially from there the world. See A smart enough LLM might be deadly simply if you run it for long enough if you want to hurt your head on this.
I’ve thought about it a little because it’s interesting, but not a lot because I think we probably are killed by agents we made deliberately long before we’re killed by accidentally emerging ones.