Independent AI safety researcher
NicholasKees(Nicholas Kees Dupuis)
Karma: 1,144
I’m confused by the way people are engaging with this post. That well functioning and stable democracies need protections against a “tyranny of the majority” is not at all a new idea; this seems like basic common sense. The idea that the American civil war was precipitated by the South perceiving an end to their balance of power with the North also seems pretty well accepted. Furthermore, there are lots of other things that make democratic systems work well: e.g. a system of laws/conflict resolution or mechanisms for peaceful transfers of power.
It seems hard to me to be extremely confident in either direction. I’m personally quite sympathetic to the idea, but there is very little consensus on what consciousness is, or what a principled approach would look like to determining whether/to what extent a system is conscious.
Here is a recent paper that gives a pretty in-depth discussion: Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
What you write seems to be focused entirely on the behavior of a system, and while I know there people who agree with that focus, from what I can tell most consciousness researchers are interested in particular properties of the internal process that produces that behavior.
Thank you for this gorgeously written comment. You really capture the heart of all this so perfectly, and I completely agree with your sentiments.
That sounds cool! Though I think I’d be more interested using this to first visualize and understand current LW dynamics rather than immediately try to intervene on it by changing how comments are ranked.
I agree that this is important. Are you more concerned about cyborgs than other human-in-the-loop systems? To me the whole point is figuring out how to make systems where the human remains fully in control (unlike, e.g. delegating to agents), and so answering this “how to say whether a person retains control” question seems critical to doing that successfully.
Are you lost and adrift, looking at the looming danger from AI and wondering how you can help? Are you feeling overwhelmed by the size and complexity of the problem, not sure where to start or what to do next?
I can’t promise a lot, but if you reach out to me personally I commit to doing SOMETHING to help you help the world. Furthermore, if you are looking for specific things to do, I also have a long list of projects that need doing and questions that need answering.
I spent so many years of my life just upskilling, because I thought I needed to be an expert to help. The truth is, there are no experts, and no time to become one. Please don’t hesitate to reach out <3
Assuming that what evolution ‘wants’ is child-bearing heterosexual sex, then human sexuality has a large number of deviations from this in practice including homosexuality, asexuality, and various paraphilias.
I don’t think this is a safe assumption. Sex also serves a social bonding function beyond procreation, and there are many theories about the potential advantages of non-heterosexual sex from an evolutionary perspective.
A couple things you might find interesting:
-Men are 33% more likely to be gay for every older brother they have: https://pubmed.ncbi.nlm.nih.gov/11534970/
-Women are more likely to be bisexual than men, which may have been advantageous for raising children: https://pubmed.ncbi.nlm.nih.gov/23563096/
- Homosexuality is extremely common in the animal kingdom (in fact the majority of giraffe sex is homosexual): https://en.wikipedia.org/wiki/List_of_mammals_displaying_homosexual_behavior
First of all, thank you so much for this post! I found it generally very convincing, but there were a few things that felt missing, and I was wondering if you could expand on them.
However, I expect that neither mechanism will produce as much of a relative jump in AI capabilities, as cultural development produced in humans. Neither mechanism would suddenly unleash an optimizer multiple orders of magnitude faster than anything that came before, as was the case when humans transitioned from biological evolution to cultural development.
Why do you expect this? Surely the difference between passive and active learning, or the ability to view and manipulate one’s own source code (or that of a successor) would be pretty enormous? Also it feels like this implicitly assumes that relatively dumb algorithms like SGD or Predictive-processing/hebbian-learning will not be improved upon during such a feedback loop.
On the topic of alignment, it feels like many of the techniques you mention are not at all good candidates, because they focus on correcting bad behavior as it appears. It seems like we mainly have a problem if powerful superhuman capabilities arrive before we have robustly aligned a system to good values. Currently, none of those methods have (as far as I can tell) any chance of scaling up, in particular because at some point we won’t be able apply corrective pressures to a model that has decided to deceive us. Do we have any examples of a system where we apply corrective pressure early to instill some values, and then scale up performance without needing to continue to apply more corrective pressure?
A monarch is an unincentivized incentivizer. He actually has the god’s-eye-view and is outside of and above every system. He has permanently won all competitions and is not competing for anything, and therefore he is perfectly free of Moloch and of the incentives that would otherwise channel his incentives into predetermined paths. Aside from a few very theoretical proposals like my Shining Garden, monarchy is the only system that does this.
It seems to me that a monarch is far from outside every system, and is highly dependent on their key supporters (generals, bureaucrats, etc.) to stay in power. Their rule is not guaranteed to be permanent, and they are forever competing to avoid being replaced by their court. The Dictator’s Handbook makes this argument better, but it seems the main difference between an authoritarian ruler and a democratically elected one is not whether or not they have to satisfy the people they rule, but rather how many of those people they need to satisfy to stay in power.
Actually, autocrats are often terrible, not because of bad luck, but because their particular flavor of incentives also gives them a race to the bottom. I highly recommend the Dictator’s Handbook, they give a much fuller explanation there. Even selling our souls to a potential Stalin won’t rid us of the great almighty Moloch!
“Note: for now, to avoid overfitting on our very small dataset, we only use 1-dimensional factors. We expect to increase this dimensionality as our dataset size grows significantly.”
This was the reason given from the documentation.
It’s been a while since I read about this, but I think your slavery example might be a bit misleading. If I’m not mistaken, the movement to abolish slavery initially only gained serious steam in the United Kingdom. Adam Hochschild tells a story in Bury the Chains that makes the abolition of slavery look extremely contingent on the role activists played in shaping the UK political climate. A big piece of this story is how the UK used their might as a global superpower to help force an end to the transatlantic slave trade (as well as precedent setting).
I really loved the post! I wish more people took S-risks completely seriously before dismissing them, and you make some really great points.
In most of your examples, however, it seems the majority of the harm is in an inability to reason about the consequences of our actions, and if humans became smarter and better informed it seems like a lot of this would be ironed out.
I will say the hospice/euthanasia example really strikes a chord with me, but even there, isn’t it more a product of cowardice than a failure of our values?
Sometimes something can be infohazardous even if it’s not completely true. Even though the northwest passage didn’t really exist, it inspired many European expeditions to find it. There’s a lot of hype about AI right now, and I think the idea for a cool new capabilities idea (even if it turns out not to work well) can also do harm by inspiring people try similar things.
Wow, this post is fantastic! In particular I love the point you make about goal-directedness:
If a model is goal-directed with respect to some goal, it is because such goal-directed cognition was selected for.
Looking at our algorithms as selection processes that incentivize different types of cognition seems really important and underappreciated.
Natural language is more interpretable than the inner processes of large transformers.
There’s certainly something here, but it’s tricky because this implicitly assumes that the transformer is using natural language in the same way that a human is. I highly recommend these posts if you haven’t read them already:
How would (unaligned) superintelligent AI interact with extraterrestrial life?
Humans, at least, have the capacity for this kind of “cosmopolitanism about moral value.” Would the kind of AI that causes human extinction share this? It would be such a tragedy if the legacy of the human race is to leave behind a kind of life that goes forth and paves the universe, obliterating any and all other kinds of life in its path.
You might enjoy this post which approaches this topic of “closing the loop,” but with an active inference lens: https://www.lesswrong.com/posts/YEioD8YLgxih3ydxP/why-simulator-ais-want-to-be-active-inference-ais
That sounds right intuitively. One thing worth noting though is that most notes get very few ratings, and most users rate very few notes, so it might be trickier than it sounds. Also if I were them I might worry about some drastic changes in note rankings as a result of switching models. Currently, just as notes can become helpful by reaching a threshold of 0.4, they can lose this status by dropping below 0.39. They may also have to manually pick new thresholds, as well as maybe redesign the algorithm slightly (since it seems that a lot of this algorithm was built via trial and error, rather than clear principles).
Thanks for pointing that out. I’ve added some clarification.
I really enjoyed this post. No assumptions are made about the moral value of insects, but rather the author just points out just how little we ever thought about it in the first place. Given that, as a species, we already tend to ignore a lot of atrocities that form a part of our daily lives, if it WERE true, beyond a reasonable doubt, that washing our sheets killed thousands of sentient creatures, I still can’t imagine we’d put in a significant effort to find an alternative. (And it certainly wouldn’t be socially acceptable to have stinky sheets!) I think it would be healthy to cultivate genuine curiosity and caring about these things, rather than ridicule people who depart from social norms. If insects do deserve moral weight, I’d like to be the sort of person who, and a part of a community that, would notice and take that seriously.