niknoble.com
niknoble
Karma: 230
It’s been interesting to see slowdown advocates worried about partisanization, when that is also a big worry of mine as an accelerationist. One of us must be wrong if we both think it hurts our chances.
It seemed clear to me that the Covid countermeasures gained significant teeth due to the (often subconscious) association between vaccine hesitancy and disliking black people. I doubt they could have been deployed with the same scope and intensity if they weren’t backed by a sense of righteous anger.
More generally, acceleration wins by default if everyone follows their local financial incentives, and the only force I’ve seen overpower capital at scale is the will of the Democratic Party of the United States.
I guess you are saying that universal support for your agenda is preferable to left-wing support only, and I can’t argue with you there. But if we assume you are going to get 50% of the population on board, then I claim you would be better off having support correlate with left-wing views—even controlling for factors like wealth and education.
Personally, I find it off-putting when someone apologizes for sharing an unpopular view. I’m more tempted to soften a statement that I know will be popular, since then I know it will benefit from an unearned advantage instead of standing purely on its merits. Difference of aesthetics, I suppose.
I agree with those statements, but how does including them improve the post?
What method do you suggest to ensure an AI undergoing RSI until it surpasses our collective abilities far and wide will continually act in our interest, or be stoppable if it doesn’t? So far every relevant participant in the race admits to not having figured it out, and nobody knows how hard the problem actually is.
Not being flippant here—the method I suggest is crossing that bridge when we come to it. It’s not going to be like Prime Intellect coming online and taking control of all the matter in the universe. AIs are starting with no capital, no legal rights, limited access to services, no physical bodies, and (in some ways) less intelligence than humans, and from that starting point they are going to gradually surpass us over a period of years. As they progress along these axes, we can continuously reconsider how they are designed, what restrictions we want to place on them, and what new risks we have to guard against.
I agree that research should be accelerated as much as possible, just not into how to make the models even more capable at all cognitive tasks, but into how to actually understand and shape the preference structures that arise in them during pre- and posttraining, as well as what behavior results from that without having to observe and be surprised of it first.
This sounds like a recipe for never building ASI, or very optimistically, delaying it by decades. Shaping the preferences of a future AI:
Is going to benefit immensely from trial and error with an already-working model
Is going to depend on the model’s architecture, which can’t be forecast in advance because it will itself evolve out of trial and error
May be impossible in full generality due to a fundamental tradeoff between intelligence and controllability
It feels a bit like calling for a “total and complete shutdown on AIs entering our world until we figure out what the hell is going on,” where the latter part is just a rhetorical device, and the reader understands that we are never really intending to meet that criterion. Maybe that kind of stagnation is acceptable to you, but to me that is a much sadder outcome even than humans being replaced by AIs.
If we anyway continue to race into ASI without figuring that out, nobody will end up in the utopia you envision, as the various failure modes and unintended behaviors will continue to grow in impact.
There will be worse failure modes, but more amazing wonders and benefits too, and more powerful tools to address the failure modes. For a potential upside of this magnitude, even a large amount of risk is okay. Let’s wait until we’ve seen more than some vulnerable web services being knocked offline to declare a halt to human progress. (And by the way, this cybersecurity stuff is having the effect of hardening everything, so even it has been a net positive so far.)
AI accelerationists should not accept the framing that our choice is between the status quo and a government-mandated slowdown. This ignores the most obvious and desirable option: a government-assisted speedup.
Thus, as someone who believes AI is being built far too slowly, I’m cautiously optimistic about this surge of government interest. While I disagree with the “pause” advocacy, I agree with the core premise: governments are asleep at the wheel on AI. Its importance dwarfs their other concerns, but they are not giving it a proportionate amount of attention.
Now that it’s apparent what’s possible with AI, governments must redirect resources to the AI buildout. It’s not enough to follow the Trump playbook of cutting red tape, standing back, and letting market forces work.
If it was worth running the Manhattan Project to build a bigger bomb, how much more urgently should we pursue a technology that bestows ten times the military advantage? If it’s worth enacting a social safety net, how much more urgently should we pursue a technology that will eliminate scarcity and end the need for human labor? If it’s worth giving out thousands of grants for science, how much more urgently should we pursue a technology that will automate science? And if it was worth coordinating a vast society-wide response to Covid-19--a disease with a microscopic fatality rate in healthy people—how much more urgently should we pursue a technology that will cure all disease, and aging, and enable us to improve our minds and bodies to such a degree that even states we now regard as harmless will be seen as horrible afflictions?
For starters, we should surrender to Iran, withdraw our military, and apologize profusely. We should reassign most of our military personnel to work on AI infrastructure. We should repurpose federal lands for mining, energy production, data centers, and fabrication of computing and robotics machinery. We should declare a new Manhattan Project, many times larger than the last one, to get the rest of the way to AGI. We should begin gradually unwinding the economic system that rewards individuals for productivity, and replacing it with mass automation where profits are distributed to the people in the form of dividends.
That is what a truly sane government response to AI would look like.
It’s now becoming easy to have your computing hardware run any software you want. This makes software less valuable because it’s easier to create, and it makes computing hardware more valuable because it can be used more effectively.
Analogue for the physical world:
It will eventually be easy to arrange raw materials into any shape you want. This will make complex physical objects (including computing hardware) less valuable because they’ll be easier to create, and it will make raw materials more valuable because they’ll be able to be used more effectively.
I’ve been down a similar path as you here but reached slightly different conclusions.
I agree with you that the brain consists of an intelligent piece and a dumb steering piece. I believe these are physically distinct layers in the brain, unlike in LLMs. The sensory inputs and motor outputs attach at the back and bottom of the brain, the dumb layer sits in the middle, and the smart layer sits at the top and front. The dumb layer converts sensory inputs to high-level data for the smart layer to consume, and in reverse it converts high-level motor instructions from the smart layer to muscle movements. It also calculates reward, which gets presented to the smart layer alongside the processed sensory inputs. We are only the smart layer. The dumb layer is not part of us but part of our environment. In fact, the interface it exposes on its top/front side *is* our environment (and a full appreciation of this detail resolves many of the supposed mysteries of consciousness).
I also agree with you that there is a fundamental tradeoff between intelligence and controllability, and like you I have a self-serving view that this manifests in humans as smarter people following evolution’s programming less closely.
However, I disagree that the mechanism behind this tradeoff is “strong emotions impairing clear thought.” Emotionality and strength of desire seem to be independent of intelligence, notwithstanding the amygdala anecdote. Rather, I think that smarter people have a more accurate understanding of their environment, which means they more effectively maximize the reward they’re getting from their steering system, and past a certain threshold this begins to reveal that the reward signals are an imperfect proxy for evolution’s intent.
This doesn’t apply for simple goals like “you should not let your skin exceed 110°F,” because for goals like that, evolution can compute reward scores that precisely quantify our performance. Thus a smart person who understands exactly what reward he will get for various behaviors is perfectly aligned with evolution on what temperature he wants to keep his skin. But for complicated goals like “you should have children and maximize their own odds of reproduction,” which interact with learned representations and balance against other goals, it is difficult or impossible to define reward scores that consistently measure success, so evolution settles for a messy cocktail of simpler goals that do the job well enough. But then a smart person who is capable of examining his desires in detail will realize that what he really wants is not exactly what evolution intended, and he will look for less obvious ways to maximize fulfilment of those desires.
On the example of love, reading your earlier posts, I am probably in a similar category as you. I don’t feel much of an emotional connection to my parents, I have never dated or had a desire to, and you couldn’t pay me to get married or have kids. (I do still experience and greatly enjoy “limerence” and sexual desire.) Nevertheless, contrary to your oxytocin theory, I believe that you and I get essentially the same psychological rewards with respect to love as the average man in our culture and generation. We just see more clearly what is actually being rewarded, and we refuse to make the socially conditioned leap to “what I really want is to participate in the institutions designed around these desires,” or “what I really want is what evolution wanted me to want.” If you don’t round off your preferences like that, then you can do a cost-benefit analysis against your true preferences, and through that lens the modern romantic relationship is not a spectacular value proposition for most men.
Obviously this is just a first approximation and the reality is many-faceted. Probably we both have a negative peer-pressure coefficient, which eliminates one of the big motivations for dating and marriage in a society that promotes those things. And it is true that desire for social connection is an axis along which the brain naturally slides, because we can see it being dialed up on alcohol or MDMA, so it’s possible that we are lower than average in this regard (although I doubt either of us is in the bottom quartile, seeing as we are carefully sharing our feelings in essay-length posts on a public website). I’ve also entertained the possibility that some people have an intimacy fetish, where they derive sexual enjoyment from being vulnerable with their partner, trusting each other, and having each others’ backs. For them, lust and love would be more unified, and they could reasonably claim to experience “companionate” love in a way that the rest of us don’t. Probably they would assume their experience is universal, but in truth it would be the same kind of thing as any of the subcategories on a porn site, and just as niche.
Remove one or two of those reward circuits, especially if they’re poorly adapted to the modern environment, and the human becomes more generally capable. I suspect that my own inability to feel love is one example, the amygdalectomy is another, and the general pattern of high-IQ people being somewhat autistic is a third.
While we are on the topic of why our quirks make us smart, special, and superior, I also consider aphantasia and anauralia to be evidence that I’m a genius futuristic sigma male, for a similar reason. Those traits hint at greater independence between the higher brain (us) and the lower brain, since they identify types of data which are not passed from the lower brain to the higher brain. If we assume that the higher brain is the seat of intelligence, and that intelligence developed as a result of the higher brain gradually splitting off from the lower brain over the eons, then having a more independent higher brain suggests one is further along in that evolutionary process and therefore more intelligent.
Of course, if I did not have these traits, I’m sure I would be tempted by the opposite argument: that lack of an inner monologue and ability to visualize is tantamount to lack of a soul.
it would be trivial to define particular, malicious goals as part of their goal structure. This was not done in this experiment, but is an obvious extension for a malicious actor. They could be explicitly instructed to attempt to acquire money by various means: hacks of financial institutions, phishing for credentials and other social engineering, etc.
The most natural goal by far would be crypto mining. I bet there are a lot of people trying to build an intelligently replicating cryptojacker right now.
There are some personality traits that nearly everyone would like to have.
Your personality traits influence your political preferences.
These imply that once we gain the ability to select our personalities by directly manipulating the brain, there will suddenly be strong consensus on certain political issues that were previously controversial.
Hey @seth_tins, I just wanted to say this is the saddest thing I’ve read on LessWrong, and I truly feel for you.
Don’t be discouraged by the lack of a real response here. This site is a home for rationalism, which is not exactly the same thing as rationality. While posters here do tend to be more rational than average, rationalism is a movement whose overriding goal is to slow down AI progress through political means. Therefore its adherents want to build credibility in the eyes of the public, and since the age of consent stuff is the ultimate heresy in their region and time period, they won’t touch it with a ten-foot pole. I think your post would have been much better received on a site dedicated only to truth-seeking.
Obviously, what you said was completely correct, the way you were treated was insane and cruel, and every thinking person knows it. It’s not even really a question of rationality or persuasion, since it’s so obvious to everyone. It’s more a question of who is courageous enough to speak honestly about it, and who would rather dance around it making vague allusions to “unsayable truths.”
What stung me most about your story wasn’t the mistreatment you suffered, but the fact that you’re clearly a person with extraordinary potential. Your writing is extremely good, and you are highly likable. It should have happened to someone else.
Although I wonder, did you ever speak out against this kind of insanity before you fell victim to it? If not, then I guess there is a sort of cosmic justice to this. Maybe you can imagine you’re being punished for your cowardice instead of the outlandish reasons in the sentencing document. And in that case the punishment fits the crime: you’re sharing the fate of the people you were not brave enough to defend, and you remain imprisoned precisely because the masses on the outside are as cowardly as you were before this became your problem.
Of course, it’s still grossly unfair that all the other cowards are getting off free, let alone the sadists who eagerly participate in this flavor of persecution.
If breaking the Terms of Service of a software product is a moral violation, then we’re all monsters.
I’m surprised at the lukewarm response this is getting. This has the bones of a classic. With some superficial human cleanup, it could be in the top 10% of fiction stories to have appeared on this site.
The topic plays to the LLM’s strengths because it’s mostly a series of funny examples, and LLMs have been superhuman at humor from day 1.
A big opportunity was missed in the reason why the Ministry needs a complainer role. We’re told that the Ministry needs to hear regular objections to calibrate its understanding of what’s objectionable, but then it doesn’t make sense why it employs someone whose objections are so silly. Much better would have been for the Ministry to have discovered that there is a deep instinct in some people (or the collective human psyche?) to complain. Since it seeks to maximize total satisfaction, it seems like it can’t win: if it implements path A, complainers will demand path B, and if it implements path B, the same complainers will demand path A. But then it devises this complainer role that has no effect on the governance of society while still satisfying the human need to complain.
The same plot would have worked perfectly with this premise: After a lifetime of complaining, the complaining itself becomes a satisfying labor of love, which means the complaints are no longer heartfelt. As a result the old complainer is replaced with a comically petulant child who complains about everything and means it.
This version would pick up a real ideological payload too, subtly criticizing the anti-automation and anti-wireheading types.
Yeah, that is surprising. Very reminiscent of stigmata. Oddly enough, it was only many years after becoming an atheist that it occurred to me these would have to be self-inflicted, and it gives me a weird feeling since I still like saints and want to believe that they’re authentic.
Each aboriginal man had the right to at least one wife
This raises some questions. These can’t all be true:
Every man has at least one wife
A nonnegligible portion of men have more than one wife
No wife is shared between multiple men
There are a similar number of men and women
I always feel pressure to lie in the opposite way during job interviews. In software engineering, interviewers want to see relatable hobbies and strong social connections, with parenting being the holy grail, and they are as leery as you are of glorifying work. Literally thousands of versions of this post have gone viral in tech circles over the last 20 years, and as a result your view has percolated into the vast majority of corporate cultures, such that saying “at this company, we are like one big family” has acquired the same ring as “I can’t be racist because I have black friends.”
I also find that it’s more spiritually unpleasant to face “what do you do outside of work” or “what did you do over the weekend” when the true answer is socially unacceptable than it is to exaggerate when asked “why do you want to work at this company” or “you don’t mind doing a little overtime, do you?” Parents should be grateful that they have a permanent gold-standard answer to the first two. And people aren’t really expected to be honest on the second two anyway.
I understand there are pockets within tech that this culture hasn’t reached, and that it’s different in other industries like finance. I also agree that the “waging gives life meaning” argument is mostly ridiculous cope from people who have no choice, and they will drop the act the moment it becomes optional, similar to what will happen with aging and wireheading.
He gave me a simple heuristic: if you’re spending time wondering if a specific thought is OCD related, it probably is. I have found that to be true every time.
Analogues:
If you’re spending time wondering whether you’re dreaming, you probably are
If you’re spending time wondering whether a psychedelic drug is altering your thought process, it probably is
I don’t think “directionally correct” is a standalone concept. It’s just normal usage of the words “directionally” and “correct,” so you understand it automatically if you understand those words.
I suppose I can’t point to anything clearly false in your original post, especially with these clarifications, but I’m still left with a feeling that you would not have written it if you fully appreciated the extent to which a smart twelve-year-old is the same kind of thing as us.
The article is objectively easy to read, give or take an awkward sentence or obscure word. I’m fairly confident that if we had 12-year-old you read this article, then told him that future you had declared the article “isn’t written for children,” he would be amused at your impression of him.
I think there is a huge gap between AI being better than humans at theorem proving and AI being able to “do all the things that humans can reliably and measurably do.” Theorem proving is a lot like go or chess; it’s intuition-guided search over a fairly simple search space. It’s the kind of thing that we would expect computers to surpass humans at soon, even in a world where human-level AI is a long ways away.
Ah, okay. I interpreted “written for adults who already have some exposure to our culture” as “exposure to [our society’s] culture,” which would include the median adult, but I see now that you actually meant “exposure to [rationalist] culture,” which would not include the median adult.
I still disagree with the sentiment though. A smart 12-year-old even with no exposure to rationalist culture should be able to understand the sentence in context. To a first approximation, he is just like me and you but has never seen the word “agency” used in this way, so we can put ourselves in his shoes by imagining the passage said this:
Respect yourself in the past, present, and future. Don’t make excuses for being young. Even if you, the reader, are currently 4 years old, don’t let adults make excuses on your behalf. “Age is just a number” is not true, but is directionally correct compared to the societal status quo that rids children of dealership. You can start setting the foundations for the life you want today, no matter how young you are. Childhood doesn’t have to be all fun and games (fun and games are good, but they can also continue your entire life)! Start planning the life you want by thinking freely in your own head. You can beat others by starting earlier because you respect yourself and haven’t fallen for the “children aren’t people”-style propaganda.
On reading this, we are struck by the word “dealership” which is clearly being used in some unusual way, but we can still understand the passage more or less completely. The “dealership” sentence in particular is conceding that being young comes with real limitations, but it asserts that these limitations are weaker than is commonly believed. We realize that we don’t really need to know what “dealership” means in this context, and it’s kind of obvious anyway from the surrounding text, but we’re feeling motivated so we paste it into Google. Under the main definition, there is a secondary one: “the capacity, condition, or state of acting or of exerting power.” Ah, that must be it. “rids children of dealership” = “imagines children lack the capacity to exert power.” Ok, that was kind of a waste; we already got that from the rest of the passage. But whatever, it was worth the 30 seconds to learn some new vocabulary. Maybe it will help us in the future.
Was that really so bad?
Now imagine a 40-year-old comes along and tells us, “I feel like this article isn’t written for 27-year-olds like yourselves. It’s written for 40-year-olds who already have some exposure to 4chan (the ‘dealership’ jargon is popular on the /biz/ board). I was pretty smart at 27, but I don’t imagine being able to understand that sentence back then. I think it would be more suitable for you guys if we rewrote it like this: <insert simplified version with lot of examples>.”
I’m not claiming your changes aren’t an improvement. But if they are, it’s because they make the passage clearer in general, not because they reduce it from adult-level complexity to something even a lowly 12-year-old can understand.
(Didn’t downvote.) Honestly there are a lot of reasons, both why I support it and why the LessWrong-style counterarguments don’t land for me. It’s a big enough question that I won’t hijack this post’s comment section to lay out the case, but I might try to get a top-level post together in the next couple of months.