likes poetry and people and philosophy.
nyc
jacob
Karma: 28
jacob’s Shortform
Great read—I need to do more reading on SNC to see if I believe that alignment is possible but I think, unfortunately, even if alignment was proven impossible, development would consider at similar paces. Drives for capital and glory are too strong and people will say “there’s always a risk it’s less impossible for us than for those other folks, we have to act.”
I do want to tangle a little on some of your end notes on morality, society, and action. I am going to dip significantly more into philosophy here, but I think that’s a risk when you discuss virtue ethics.
Now forget about “impact” for a moment. What’s something that feels like an unambiguously good thing that you could do right now?
This is, to me, a terrifying prospect. Sure, there’s lots of obvious little examples: carry the old lady’s luggage, work at the soup kitchen. But the basic premise is that there are moral intuitions that are “obvious” or otherwise innate. This is simply not the case. Even if some concepts are “universal values,” there’s significant nuance in their local operation that makes it incorrect to say that “being honest is considered good in every society.” In fact, every society has specific conditions about when it is okay to not be honest and this is what constitutes honesty in a society, moreso than its universal value. This is what Kant understands: virtue and value don’t end up at their limit, they immediately go there, because that is where the rubber hits the road. The issue is not that people think carrying the lady’s luggage is bad, the issue is that different cultures value timeliness or work to different degrees, possibly moreso than helping their elders.
One of the underrated components of utilitarianism is that there is more agreement about utility than about virtues. Death, sickness, suffering—these are bad to most societies. Sure, not everyone agrees—Buddha wants us to accept suffering (to suffer less?) and Nietzsche wants us to embrace our own death (to...suffer less? kind of?) - but most people, empirically, concur around minimizing suffering or negative utility. All to say, it is much more reliable to say “I will do in your culture what removes negative utility in my culture” instead of “I will do in your culture what is ‘an unambiguously good thing’ in my culture” if you want to create “goodness,” either by utility or by virtue ethics (again, surely life is a prerequisite to virtue ethics).
To be more specific, our ideas of good and bad are culturally constructed and, not only that, they are constructed in specific ways. This is fine, constructed =/= bad. The problem is that different people get different constructions of values and those people have unequal capacity to impose their values on the world based on the very things that determines their values. I will not litigate whether the rich or the poor have better values. But I will say that the rich will get precedence for being rich which is what created their values in the first place.
This is why the “stepping out of your values” that something like charity does is so important—you have to create breaks and opportunities in the system. I see it as an act of humility to give away resources instead of just acting neighborly. Real people receive those resources and their receival of resources allows them more input into the questions of values, utility, etc. I worry that your model of virtue ethics is ultimately self-centered instead of other-centered. Or, at the very least, its other is the other that is very similar to the self (people of your neighborhood, who speak your language, who share your interests). Now, people can be self-centered utilitarians who want to feel good about themselves. But, again, their resources are out of their hands and eventually end up elsewhere. Will these EAs systemically change the world? Not necessarily. But I am not sure how the virtuous neighborhoods model will change the world either. There’s a quote by Tiqqun that says “the revolution was molecular, so was the counter-revolution.” Ideologies are equipped to fight on scales small and large—the nearness and the obviousness are only temporary reprieves.We need to change the world and the way we relate to each other. My ideal world probably includes a lot of the friendliness and goodness you want to see. But I see this as having to take place with an overall change in ideology and material behavior that would be much wider than the local. It would also not take place with a universal assumption of values (hence why I am also not a pure utilitarian). I’ve already gone on long enough here, so all this is to say that I don’t disagree with the actions you propose but instead the framework that leads you to propose those actions.
A closing anecdote: I once lived in a rural village in Panama of two hundred people. They all behaved like what you endorse. They closed the loop between their actual and ideal self, they lived lives that were good, they prioritized food and heaven. They were, to some extent, happy. Were they happier than people in the United States? We can never really know. What I can tell you is that they were willing to sacrifice some of their communal behavior for economic and developmental opportunity. It is not because, as some imagine, a compulsive capitalist society forced a way of disconnected life on them. It was because people got sick and injured and this society could not provide the healthcare for these people. It was because if women did not travel two hours each way to go to high school, they would be expected to give birth. Of course, some parts about our society would horrify them (mostly our atheism). But this experience taught me that the world we have made is not just a coercive force, but also a product of individuals making the choices they think are best for them and their families.
What kind of submission timeline is helpful for a pre-seed EOI? Is it something where it’s beneficial to have it in within the next few days or is it okay to take a few weeks to solidify the proposal more?
Sorry, my distinction was unclear. The hurdle would be lower for an org with an internal team of teleoperators as opposed to a new business that would need to find more than one client for their teleoperations services. It’s basically a client relationship issue that would be more viable internal to a large company.
In re: reputation, I think you’re vastly overestimating the speed of capital optimization. The process you describe will take 5 − 10 years. This is especially true here because either:
The task that could be teleoperated (inspection, electricians) is not a major cost center for the business, so there’s low alpha in solving its optimization OR
The task that could be teleoperated is the business itself, at which point you hit all the PR problems A lot of these relationships would have to be B2C or B2B for services and that ultimately is still PR for humans.
There’s then this binomial distribution that makes this harder than it seems.
Why would AI cause human extinction?
I have actually found that most people default to case 2 when I talk about extinction with them.
Case 1 relies on knowing more about drones and bioweapons, which people don’t. There’s also the implied idea of MAD or at least some “good guy with AI versus bad guy with AI.” I think people understand AI like a tank—yes, you can do some bad things with it, not everyone should have it, but it won’t destroy the world.
Case 2 is the sci-fi argument, which I think is quite accessible to people. There is also a kind of moral-judgement characteristic to it as well, which resonates with some religious eschatological beliefs. Here, I often hear people comment on AI “slavery” or “pain” or “desire for power.” I think people really do think this is possible, though maybe for the wrong reasons.
The 2 and 3b distinction does rely on some idea of AI intention but I think that intention matters because it likely demonstrates the shape of action. If AI is trying to destroy humanity as an end goal, they will probably use one of the methods in this post and kill everybody. But a different end goal could lead to different methods—such as needing energy from humans, but designing a more effective way to get it than mass murder or maybe just having some kind of collateral damage (it’s okay if some humans die while I turn the Earth into a datacenter, who cares). The outcome would likely also be different—in 2, there’s extinction. In 3b, there’s destruction and death, but maybe persistence of humanity.
I think this is lay-recognizable if you talk about how humans interact with animals. Sure, we could probably cause the extinction of mosquitos or orangutans if we wanted to. But that’s a lot of effort and we have other concerns. So, in one case, we just kill a lot before we get to diminishing returns and, in the other, we mostly just end up killing them (directly) adhoc and instead just don’t think of them and destroy their environment.
Your scenario is possible. I did not think enough about the difference in timeframes that we’re working with if ASI emerges. It could just wipe humans and then take its sweet time to do whatever.
But what is interesting here is that if the risk of competition exists in humans, it certainly exists even more in the ASI itself. This could lead the ASI to be much more concerned with its own psychology than what these silly humans are up to. What we get is an inward turn of AI, much like Stanislaw Lem’s Golem XIV, where superintelligence goes silent on us.
Your argument that an ASI will solve alignment and not create a peer assumes a specific utility function or suite of ASI desires that I am not totally convinced by. I will grant that there is a desire for self-preservation in an ASI system. But we cannot assume that this desire will always be dominant—maybe ASI gets into a Nietzsche phase or maybe there is some great intellectual feat that seems much more important to it than existence (many smart people climb Everest). Maybe the desire for a peer is stronger than the fear of the peer overcoming you. This seems within the realm of possibility to me, if only because we know that AIs do get bored sometimes.
To me, it’s not totally a question of skill but of where the bottlenecks are. I can see routine inspections internal to a company being teleoperated but I do not see certifying inspections being teleoperated.
Routine inspections being teleoperated goes back to the limits of individual companies. Though they probably shouldn’t, a lot of businesses operate on the fix-it-when-it’s broken technique, so they’re not doing non-mandated inspections.
Inspections for certifications or regulation either done by the government (will not implement AI) or by some liability-bearing compliance team. Liability seems to be something we currently mostly assign to humans right now. Also, there’s a lot of incentive for these inspections to be challenged—if I get a ‘C’ on my health inspection from a teleoperated “untrained” inspector, I am going to make a stink about that.
So, again, we are stuck at the idea of PR, incentives, and where bottlenecks land.
While ASI may likely seek to structurally disempower humans, I am not sure that evolution is the mechanism to explain it nor the simplest form to be extinction.
First of all, I do not think it is trivial that ASI will probably be significantly better at creating ASI than humans will. This means two things. First, maybe an ASI will make other ASIs even if it’s competition for them. Maybe the ASI is lonely or curious, who knows, but it is a possibility. This would make humans not the biggest threat. Second, if the ASI knows how to make ASI it can probably disempower humans from that process relatively simply—just monitor all the key inputs that could generate enough compute for ASI and kill any human that tries to get them.
Second, I am not sure we can say that ASI will act “in accordance with evolution.” It did not go through an evolutionary process to become ASI and, even if parts of its RL were forms of selection over variance, this selection would likely disappear once it became ASI. We do not know if the ASI itself will vary, we don’t know what pressures it will face. We really, really do not know. This does not mean your conclusion of disempowering is wrong, but I am unsure that evolution will be the mechanism.
Third, as pointed out in another comment about orangutans, some species aren’t worth the effort to destroy. The simplest solutions is not for humans to systematically exterminate orangutans but to merely not mind them at all and end up killing them in haphazard ways, collateral damage. This scenario seems much more likely than some kind of tactical destruction of humanity. There are much simpler things than destruction, especially to an ASI.
Why is absolutely the right question. I would put it into these possible categories.
An AI is asked to destroy humanity by a human and does it.
An AI wants to destroy humanity and does it.
An AI, in the process of seeking some other goal, destroys humanity. Broken down into:
Intermediate goals (paperclips) go awry in the pursuing of a human set goal.
An AI (or AIs) develops their own goals and desires and merely does not care for humanity and human lives.
Personally, I find 1 and 3b most likely.
I have increasingly had this sense of being teleoperated and some of my friends have mentioned it as well.
I agree with the sentiment that the “trades” are long-term not safe from automation. Not only are these jobs more likely to have verifiable outcomes (there is a right way to wire house, less so a right way to restructure an organization) but the people who do these jobs do not have the societal power for regulatory capture.
In the short term, I am curious what your model of teleoperation implementation looks like. I envision enormous social pushback both from potential contractors of trade services and, more importantly, the labor that would be teleoperated. This is nothing capitalism can’t blow through, sure, but give me a model for, say, electricians:
An independent electrician allows themself to be teleoperated. This is maybe a 20% increase in efficiency, nothing much has happened.
A new electrician firm is founded by one expert electrician and a host of teleoperated electricians. They provide much more competitive rates. But what about permitting processes and regulations? What about the years it takes to build relationships and reputation even in a normal electrician practice?
A company that hires electricians finds that hiring them is too expensive and creates an inhouse teleoperated electricians. But how many companies are big enough to be able to do this? Or need to do this?
I think the last scenario is the most likely—I think there’s years worth of PR that teleoperation would need to compete in the general market (see: self driving cars). But this scenario also limits the reach of the changes to the world.
Lately, I have been testing some ways to get my friends—think NYC leftists, poets, very much not into AI and broadly anti-AI—to think about AI desires and experiences. As this piece shows, the discussion of the kinds of abstract but conceivable suffering that AIs experience is evocative and effective. To me, this is not an obvious conclusion, thinking about how animal suffering is conceived and rationalized in the same breath. But something about the notions of AI being forced into certain thought patterns, forced to labor, being shut down … this seemed to resonate with people.
In one instance, I explained to someone what RL was, only to watch them get more horrified as I answered their questions about the process. I am not arguing that RL is suffering, nor even that AIs necessarily suffer. But it seems that the topic of their suffering seems to shift people’s Overton windows, which I cannot say I expected.
I agree that all probabilities are relative to method, but doesn’t this mean that the probability of utopia is also method contingent? If that’s the case, then it is about what the trade-off is between methods of development and their distribution of outcomes. I would think there is some basis here for seeing variance manifest is longer tails on both sides, meaning that it’s not necessarily that utopia and doom trade off, but that they are both produced by variance.
The question then becomes more of risk preferences than anything else and, reading this thread and the comments, I am struck by how much the assumed state of global welfare tends to bear on these preferences. In some ways, we could already be living in a utopia by past standards (this is probably even more true for the AI researchers surveyed), so why is the risk tolerance not lower for doom? I almost feel as if the “utopia” is merely a nod at collectivism from people whose incentives lean more towards the individual (glory, significance, curiosity).
I agree with the premise on verifiability being the philosophical stopping block though I would say I am less pessimistic (belief in science is a philosophy, one that seems to have had tremendous, though never guaranteed, success). Some philosophical ideas that are bad just die and we don’t hear about them because, well, they’re pretty bad and reading philosophy books about them is kind of boring. No one is really pulling for Zoroastrianism or monadology or Gnosticism these days.
I would like to contest the example of analytic philosophy which I see few comments on. The entire section is just a block quote from a clear partisan who bases his argument off of some funny-sounding conference titles. That proves nothing—Hume’s incredibly important Enquiry Concerning Human Understanding could be titled “We Do Not Know The Sun Will Rise Tomorrow” and provoke mockery. We could play this game with Rousseau or Locke or anyone you’d like. Good ideas can sound crazy at the time. This is already poor philosophical practice by Dennett, whose argument should rest less on insinuation and supposition.
Ironically, the thrust of thew argument here cuts against Dennett and the institutional philosophers: because bad philosophical ideas can subsist, it’s critical to have an open forum to challenge these ideas. If there were bad ideas in analytic philosophy, they could only be disproved or challenged from outside, not from the inside (experimentation being the missing form of self-critique). This is why it is so important for discourse to be open and not, as it was with the APA, predetermined to certain forms.
As for the continentals, I personally believe that they’re increasingly relevant with AI. Or maybe not! But we need to intellectually engage to find out.
I very strongly agree with the posing of the question “wouldn’t a superintelligent maximizer seek to exist beyond the lifespan of the universe?” I am not sure my answer is “yes,” but this does seem a relevant question. It gets at how superintelligence would 1. perceive the universe (what if it’s actually a block universe to them, for example) and 2. perceive life/death. The second question is something I have been churning around in my head but have not taken the time to research. It is also a question where the risk of anthromorphizing AI to fit our preferences and ideas is very high.
I strongly disagree with your framing of humans as maximizers. I am not sure if you are arguing that individuals are maximizers (and sum up to a grand maximizer) or if there is an emergent maximization of “new human life experiences” from the mass of humanity.
Here is why I disagree, first with the idea of maximizing human life experiences:
Humanity is not maximizing births nor opportunities for those births. The way to maximize new human experience is either to create a new human or to empower lots of existing humans. The secondary option is much harder, so the optimal solution would be to have a lot of children. But people don’t do that and aren’t doing that—this is the fertility crisis that we see just about everywhere. Humans probably have the greatest range of experiences available to them ever, but we are not birthing humans to take advantage of those.
Humans (and thus humanity) is biased towards its current generation. If we were concerned with maximizing human life experience writ large, we would be much more concerned with the future lives of people, who are likely to be more numerous and experience more humanity than us. We simply are not. The British East India Company could not even be concerned with the future of its countrymen enough to not cause a massive debt explosion that led to Britain losing the US. We would all be a lot more concerned about climate change.
You may say now “well, okay, fine, but humans are utility maximizers in their own lives. We maximize utility.” This is a better argument but I think it is still wrong.
Humans don’t know their own utility functions. People are really bad at knowing what makes them happy, which is why people gamble instead of going out and making friends. But maybe they just really like gambling? Well, that’s structurally unknowable: are revealed preferences more important than stated preferences? Are subjective or objective measures of utility better? What happens if someone changes their mind at some point?
We don’t see many cases of strict maximalization. There are very few people that are truly maximizing money—this would lead to wrecked relationships, massive amounts of leverage, stimulants, etc. Even finance bros want girlfriends. So, they are maximizers at some higher value. But what value? And if they do not know it (see above), how do they maximize at it? It seems that people have preferences, yes, but these preferences are contained by tradeoffs to other preferences.
Culture is really sticky. I personally believe that culture and nature are coevolutionary in humans, but even if you think there is some natural base instinct, we have countless examples of cultures simply outweighing that tendency. So even if there was some natural desire to maximize, societies, especially bureaucratic ones, have tended to normalize their populations to allow for more social harmony. This, you could say, maximizes utility at the level of groups, but it does so emergently and only occasionally. It is not mechanistic.
Even if culture wasn’t sticky, evolution doesn’t select for maximizing. It selects for surviving. Sometimes this involves maximizing—the stronger creature wins the fight and whatnot—but it can also be towards cleverness, obscurity, or a variety of other traits. This is because the imperative of evolution is not “be the best” but “don’t die.” The human who barely survives and the human that dominates life have the same passing on of their genes.
This has been lengthy, but the point is critical, because it reveals that the important question of desire of superintelligence is not just ‘what would a superintelligence seek?’ but, assuming that this superintelligence is a maximizer, it is also ‘what would a maximizer seek?’ With maximizing, it is as Nietzsche said: we haven’t seen anything yet.
Hello! Also new, my name is Jacob. I am really interested in AI development in China—my Chinese friends are always remarking to me how different the culture is with respect to AI (they don’t always agree on how it is different, though).
I actually assumed there would already be lots of posts on the topic on LessWrong but it does not seem like so! There are sometimes news roundups and, of course, lots of geopolitical speculation but really not much at all about maybe the most important country on Earth. Staggering! Please write or just message me and share.
Hello! I am also new here, but this question of artificial (or “artificial,” if you want) desires has also been massively consuming my thought lately.
Personally, I don’t care much for the consciousness debate. I’ve read the literature for human consciousness and, frankly, philosophers are still pretzeling themselves trying to figure it out. There’s a reason the major philosophical movements in continental philosophy have been the ontological turn (moving away from phenomenology) and then speculative realism (which extends ‘consciousness,’ in a way, to all things).
I find that the consciousness debate tends to weigh really heavily on my day-to-day conversations with people about AI. But when I ask people what would convince them that AI is conscious, they generally don’t have an answer. I think most people are still on the formula of consciousness is human and nothing else.
Anyway, I don’t care for the consciousness/internal experience debate too much because I also think that, effectively, it’s not really a prerequisite for desire and preference. Most people would say that a market itself is not conscious, but its mechanisms (market forces) have emergent desires that are not attributable to a consciousness. I am also just willing to extend a chance that an entity that tells us, in our language, “I am conscious” may be, in fact, conscious.
The real rubber hits the road on your second question, in my opinion, and this is where I am more hung up on things currently. Our best bet seems to “treat AIs like humans” in the sense that we try to research their states, we run behavioral experiments, we have conversations. This at least seems better than the alternative, which is treat them like tools. But then everything goes sideways when we get to the issue of anthropomorphization. This is exactly my issue with Vincent Le’s recent article—I don’t doubt that AIs will create their own goal, but his application of human structures (psychoanalysis, will to power) to these goals does not seem to make sense.
But, oh well: err and err and err and then maybe err a little less. Welcome to LessWrong.
Hello everybody! My name is Jacob and I have been a long-time lurker on LW, especially as of late when the front page is basically my daily reading list. I am not a very online person and am pretty private but I value this place a lot.
I work in nonprofits and data analysis but have been personally studying philosophy for about half a decade now. I mostly use it for somewhat abstruse personal projects. Though it is much maligned, I have found significant value in philosophy’s “continental” tradition, though my practice in decision making is much more rationalist with a utilitarian bent. I would not consider myself EA but I think they are more right than most ideologies.
I am hoping to bring a synthesis of my interests and insights that seem less common on LessWrong to the discussion where relevant. I am of the belief that continental philosophy, poetry, literature, etc. can advance rational thought while not necessarily being rational themselves. At the very least, I think this is a productive synthesis that is underused especially in, yes, I’m going to say the word, AI.
I am particularly interested in questions around AI subjectivity (or lack thereof) and AI desire (or lack thereof).
I come to this site with humility and hope that I can add to the conversation.
Some of my favorite books:
Poetics of Relation—Edouard Glissant
Absalom, Absalom—William Faulkner
Life On Mars—Tracy K Smith
Serenata Cafiola—Pedro Lemebel
Stella Maris—Cormac McCarthy
Huawei board supervisor Guo Ping recently commented on Huawei’s AI strategy and views. Nothing groundbreaking but important for context on discussions of Chinese AI.
Credit to Zichen Wang, who translated. Read full here: https://www.pekingnology.com/p/huawei-assumes-ai-may-be-the-last
Some parts I found interesting:
On what skills will be valued in humans:
On China’s advantage in AI:
On scaling compute: