delphix

• delphix 13 Mar 2026 14:44 UTC

  3 points

  0

  in reply to: TAG’s comment on: Open let­ter to doomers

  You don’t know for a fact that there are finite people with finite numbers of neurons, but leaving that aside and accepting it, what’s infinitely complex about that set of people and neurons is the information contained within them. That’s because you can’t keep up with two people’s thoughts at the same time, let alone with eight billion of them. The infinity is in the fact that you can’t experience what anyone else does completely. What you get is emergent alignment(understood as similarity, not identicality), but one that is not a choice, but a given, and an old one at that. So, you don’t get alignment this way with AI. In its case, it’s neither a given nor old. That makes it inherently dangerous, no matter what you tell yourself about its structure as you perceive it.

  If you have issues with my usage of infinity above, I advise you to understand that words are intrinsically polysemous, and we use them as carriers of meaning, and not the other way around. I am trying to explain things to you, not have a debate about definitions(though we can if you like; except if you do it by simply asserting that I should only use words in the way you prefer, I will simply ask you why and we can go from there). If you didn’t understand what I said, feel free to tell me and I’ll explain it further.

  You can’t make a system smarter than you corrigible, because to make something corrigible you have to understand it, but if it’s smarter than you, you don’t in fact understand it. This would be a good place to ask you what you understand by intelligence, since it’s the reason you believe otherwise. Specifically, tell me what sort of system you imagine an AGI to be such that it is both smarter than us and corrigible. Tell me what that means, however you please.

  Let’s go through everything you believe “we can do to achieve safety”. Nota bene: when I speak about alignment I employ any one of the definitions you would normally interpret according to context(as it applies to the question of an AGI being safe). Specifically, tell me what it is that you think makes it so we can make systems more complex than ourselves safe. Or, if your claim is that safety doesn’t have to be demonstrated because it’s a law of physics, then we can debate what you understand by safety. Kindly tell me what makes it so everything is safe and as such nothing has to be defended as having that trait. Alternatively, if you admit that you don’t know that AGI can be safe, I would ask you why you want to create it to start with. Is there something else you consider more dangerous than AGI that you want AGI to protect you from? Which is? Let’s assume we get into that conversation, and there is such a thing, next you should tell me what reasons you have for finding that thing more dangerous than AGI.

  Note, these are mere suggestions, feel free to reply however you want, of course. I will try to align with your understanding such as to make my arguments further regardless, and I’ll gladly update my beliefs by adopting yours if they prove to be more coherent than mine.

  Anything short of human is apocalyptic because humans are the only entities we’re aware of that actively seek to protect humanity as a whole. The rest of the cosmos is actively trying to kill us. This is universally true. We have no reason at all to think it is possible for anything not to want to kill us, or not succeed in doing so if we gave it control over our environment. Things observe their own laws of action, including things that resemble us, but are not us. Those laws are not identical with our laws, and as such, are at odds. If given too much discretion, they will erode our experience to the point of deletion.

  I would tell you that one of us has a more correct view on this than the other. It is in both our interest that we discover who does. Because regardless of who it is, both parties are interested in either of the two making better decisions. You agree with this, yes?

• delphix 11 Mar 2026 3:12 UTC

  3 points

  0

  in reply to: Mitchell_Porter’s comment on: Open let­ter to doomers

  I think you understand the problem a little better now but you still think alignment is possible, and it’s not, it’s a complete waste of your time to try to solve it.

  What you still don’t grasp is that the human is themselves the perfect alignment target, and is something infinitely hard to discover ex nihilo. Anything short of human is apocalyptic when it comes to greater powers than us. You still think otherwise.

  There’s a terrible flaw in your logic. If an AI could solve its own alignment it would already be aligned. Because alignment is a continuous process, for the reason I explained, namely that our first value is the ability to choose our preferences, and our preferences shift continuously and inevitably.

  I don’t see that consciousness matters to this topic at all.

  Optimism is bad. It deliberately seeks out delusions. Pessimism is concerned with predicting the worst in order to be prepared by it. It has no need of the human to reason about it and correct it, once in place. It is the perfect strategy. If it fails, everything fails.

  Let me put it in video game terms. Imagine you’re playing a competitive videogame. Being an optimist about how easy the game is means seeking to play with similarly skilled players so that you can “learn better” and at your own pace. Being a pessimist means that if offered the chance to play against nothing but the strongest players, you take it.

  Why?

  Because any single action you take against these players will be a lot more successful against lesser players.

  Why?

  Because you are inherently training for playing the game in the best possible way. You are forced to do this. Any move the opponent makes that succeeds inherently teaches you about what has no business of working against anyone else(namely the strategy you were employing at the time you were playing against them). You could have done the same thing in a game with lesser opponents and won. But you now have no reason to. You know that winning like that is improbable, provided that the whole playerbase learns as a whole. It is a strategy that works only temporarily and against worse opponents. It will sometimes work against better opponents but very rarely. One day it will not work against anyone(at least asymptotically). You are learning about what strategies will not work over time. This is why you play against better opponents. And this is why you are a pessimist. Because the universe works exactly the same. Being an optimist means training in an environment that is unlikely to produce successful results, and that even if it does, will not do so for long. The best possible thing to do is to train with perfection in mind. And for that you need a better opponent. A stronger orientation towards what can really go wrong.

• delphix 10 Mar 2026 15:28 UTC

  3 points

  0

  in reply to: Mitchell_Porter’s comment on: Open let­ter to doomers

  Humans have empathy, a mutual need of each other to survive, and can keep each other in check. None of us is intelligent enough to act unilaterally against the wishes of everyone else. None of us would want to, unless they’re insane, and then couldn’t because they would lack the intelligence for it.

  Parents have biological imperatives. If you want to recreate such an imperative in something else, you would have to make that something else actually human, if you make it anything else, the empathy doesn’t simply fit(not that you would ever know how to recreate it to start with). The short answer here is that we don’t know how this empathy works, and that not knowing we can’t assume we will somehow know in time. It is my opinion that this is unknowable for the purpose of transferring into an AI. These drives aren’t conscious, per se, they have been encoded in us over enormous aeons, we do not in fact understand them in a scientific sense(...to the extent you would need to, obviously), we merely understand them experientially, which can cloud one’s judgement about our ability to manipulate it in the ways you would seek to with an AI.

  The state has similar drives. It has a certain sort of empathy, no matter how strange this might sound. But more relevantly in its case, it has to obey the simple laws of politics which say that it would be very hard for it to set out to attack those it is expected to lead and protect. At some point in trying to do so, it would run into the simple problem that it is made out of people who wouldn’t see it in their interest to carry out its orders. We are all the state in some sense.

  Also, as humans, we have an inherent track record which matters quite a lot for the sake of contemplating what we might do to each other that is too destructive. A long history of being able to coexist.

  The problem is not that I am too general. It is that you anthropomorphise incredibly easily, and it doesn’t make any sense. An AGI would obviously not care about our ability to overpower it, because this ability wouldn’t exist past a point. It wouldn’t care about us at an emotional level because it lacks our particular ability to feel such emotions, and it lacks the history that created them blindly. Emotions are barely even things we can very clearly reason about. We have no clue what they are. We know only our feeling of them. You can trivialise them as things that mean simply something conducive to your goals or not, but that seems far too reductive. Not because it might not be just that, but because such explanations as we have, including this, are very likely incredibly deceptive as to their underlying complexity.

  There are many ways to talk about this. But ultimately, you lack an answer for the question of what do you do exactly about something you can’t predict changing the environment in ways you can’t react to after the fact, if it’s bad? And why would it ever not be bad? Why would you ever assume that things going badly isn’t the rule, absent human direction?

  I think you’ve grown incredibly complacent about our apparent comfort and safety as a civilisation, and take this feeling wholesale and apply it optimistically into the future, no matter how extreme the circumstance. The truth is we are not as evolved as we think of ourselves to begin with, we are incredibly fragile, and if we lose control over our ability to shape our environment, there is no reason to think that will go in any way except completely badly. It always does. Anything we do that is not deliberate ends in disaster. The universe is not an inherent joyride. Everything we’ve done to make it resemble one has been deliberate, not guaranteed.

  Our ability to control our environment is key. Our ability to understand our systems in detail is key. Without this kind of awareness, we are completely defenseless and have no reason to expect anything good. This should be entirely obvious.

  Also, we have no choice when it comes to ourselves. To ask “how can you trust humanity” is akin to asking “how can you trust yourself?”. There is no negative answer possible here. We are forced to operate in this way.

• delphix 10 Mar 2026 2:12 UTC

  1 point

  0

  in reply to: Mitchell_Porter’s comment on: Open let­ter to doomers

  It is our values that are infinitely complex and can’t be encoded into AI. Our values are simply our desired state for the world. This state changes automatically and constantly. An AGI aligned to our values is impossible to create precisely because the most fundamental value we have is the ability to spontaneously decide what our preferred future state is.

  With an AGI more capable than you, you would lose that freedom. If it simply did what you wanted perfectly, yeah, that would work, because then it would simply be an extension of your volition. But that is not an intelligence. That is a piece of technology that empowers you to better and faster do what you want. It is not autonomous. It is perfectly contiguous with you. This is why it’s impossible. It can’t be a perfect servant(which is what value alignment would actually be as I just explained) if it has its own mind.

  The AGI will not simply do what you want perfectly. It will be able to do incredibly impactful things that you will have attempted to train it to fit your preferences, but it will do so imperfectly, and that imperfection is what will prove to be too much to handle.

  We have extensions of our exact will already of course. The difference with them is that they tend to effect very weak actions on average so we have the ability to correct them. And even that to a point. But if you increase the world’s ability to drift from your ability to change it, you lose. That is what losing is. And you lose badly. Do it strongly enough and you lose permanently, and on behalf of all humanity itself. That is what AGI represents. And it represents that because its capacity to change the world is on track to outpace our capacity to correct its inevitable differentiation from us.

  The most significant mistake you make here is to underestimate the delicateness of this differentiation and its implications. You assume that you can simply get it mostly right from your perspective and that that is fine. That’s for two reasons. You’ve been argued into thinking that a world without you which follows your values in your absence is a good world, and you don’t appreciate just how vulnerable we are to the environment getting out of control.

• delphix 8 Mar 2026 18:48 UTC

  −1 points

  −2

  in reply to: Mitchell_Porter’s comment on: Open let­ter to doomers

  You are what I call an accelerationist, and I consider someone like Yudkowsky one too. You are conservative accelerationists, but the point is that given your beliefs in the possibility of alignment, you would eventually create AGI if it were up to you.

  Alignment is impossible because you can’t align something infinitely complex. Humans are not artificially aligned. They are aligned via billions of years of evolution. Inscrutably so. It doesn’t mean anything to talk about figuring ourselves out to the extent that we could perfectly replicate ourselves, insofar as our empathy for each other goes, but also our inherent similarity to each other in a cognitive sense, into something else. That similarity is what allows us to live in equilibrium with each other. The empathy helps too, but the similarity is what is actually key. Because we have already shown the ability to ignore that empathy at length. But the similarity is what allows us to keep each other in check. That is, we learn together. And the third thing that allows us to coexist as we do is that we need each other for survival and a meaningful experience.

  The problem is that none of these things could possibly apply to an AGI. The only thing you could get is something short of AGI that seems to satisfy these conditions. Supposing we could encapsulate ourselves into an invincible bubble of observation after AGI has been created, what we would observe is that no matter how perfectly aligned from yours and every other conservative accelerationist’s perspective it would be, it would inevitably stray from that alignment the more time went on. That’s because the AI would interact with the environment and be changed by it. In ways we can’t predict in advance. You don’t even begin to understand how absurd the thing you believe you can do is. This is simply because you draw a false inference from how humans act to how AGI could supposedly act. We are not the same. And not being the same, we are at odds in that sense. We are the same to each other. But I repeat myself.

  There are many many false things conservative accelerationists believe. But one that presently comes to mind that is especially glaring is that they supposedly think it would be easier to align AGI than to do anything we would want it to do on our own, or to convince the world not to build it, or both.

  Do you not see the paradox here? You simultaneously think humans are more complex than AGI given the fact that you find it harder to align them, and that AGI would be more complex than them. You need to pick one or the other.

  Another paradox with alignment is that you are either imagining aligning something dumber than you, which would make it mere software, and not AGI, or you are imagining aligning something smarter than you, which would make it a contradiction in terms. What do you even mean by a greater intelligence if you speak of controlling it? You’re not aware of what you mean by words. You’re not aware that you mean something different with them every time you use them. That you use the same words with vastly different meanings in the same contexts. This is how you come to believe something as absurd as that you can align something smarter than yourself. When you say that, you imagine that intelligence doesn’t mean an inherent resistance to being controlled by things less intelligent than itself, but then when you think purely in terms of the AGI itself and what it might do, you again mean something that can resist such control. You are contradicting yourself. And it’s hard to believe this because so many people you find intelligent do it too. Constantly. All the time. You can’t imagine that everyone you look up to falls for such apparently embarrassing thinking mistakes.

  But maybe I’m wrong of course. I look forward to you addressing these arguments meaningfully and showing me what I don’t understand. Same for anyone else.

Open let­ter to doomers

• delphix 7 Mar 2026 23:28 UTC

  1 point

  −1

  in reply to: papetoast’s comment on: Open Thread Win­ter 2025/​26

  Humans having finite time means that they should be perpetually open to criticism to make sure that they’re not doing the wrong thing, and not preselect that criticism in such a way as to make it extremely hard for any meaningful sort to go through, and be sincere in this matter, or otherwise put, what I urge people to do.

• delphix 7 Mar 2026 23:24 UTC

  −4 points

  0

  in reply to: ChristianKl’s comment on: Open Thread Win­ter 2025/​26

  By that same token, this entire forum should understand my position rather than me its. Except I don’t ask anyone to read things to have a conversation with me, I can make my arguments in real time because they’re real arguments and not false ones that get lost in such obfuscating requests.

  Also, please note you’re comparing the way you have conversations to physics. That is rather ludicrous to say the least.

  Your philosophy is not that complex. You don’t know how to have conversations. That is transparent from the very fact that you refuse to have them. You are the ones that need to have conversations with people and study them, and yet you reject them, treating access to you like some sort of privilege. This is completely opposite from what you should be doing as people who are incredibly worried about the future and need the rest of the world’s help in that regard, which I assume most of you are, but you have been corrupted by your own ridiculous and arbitrary rules to have turned out to be actively alienating the world instead, even when it comes at your doorstep(you do the same to it elsewhere).

  Do consider that you are not as a whole as smart as you think and assume some of Socrates’ simplicity in that respect.

• delphix 7 Mar 2026 7:03 UTC

  1 point

  −1

  in reply to: ChristianKl’s comment on: Open Thread Win­ter 2025/​26

  No, I think that conversations in general don’t happen as they should for the sake of doing philosophy, which is a lot more important than doing science, such as I imagine you to understand it.

  There is no science of persuasion so evolved as to convince the entire world of what it should do, or indeed, for discovering what it should do. That is the domain of philosophy, and in fact always will be. That doesn’t mean that science can’t help, only that it can’t actually do the most relevant part. That’s because the very nature of our experience will always evolve, and our problems and ability to talk meaningfully about them as well. Science seeks to establish the nature of things that for all intents and purposes stay the same, or at least remain stable for a very long time. In any case, it’s a different sort of project. That doesn’t mean the two can’t coincide, it is really a matter of how they should be done that differs.

  To approach a question of meaning or politics scientifically in the way you describe is to assume that you know the answers from the start. What if your methodology is inherently flawed, and particularly in such a way as to be blind to the very ways in which it is flawed?

  Let us suppose for the sake of the argument that the parameters which you choose for conversation don’t allow either for those who understand what should be done better than you to actually demonstrate it to you, or that it doesn’t allow for persuading as much of the world as you require insofar as what you believe is in fact more adaptive. How would you know? If you take it for granted that this couldn’t possibly happen, aren’t you inherently arguing that this approach to the problem is implicitly perfect? Should we ever act like we know anything with that level of certainty? And what if we’re wrong?

  Where is the harm in engaging in simpler conversation? Wouldn’t that allow us to both make sure that we’re not missing anything and to better persuade people of things they don’t yet understand?

• delphix 3 Mar 2026 4:22 UTC

  −1 points

  −3

  in reply to: ChristianKl’s comment on: Open Thread Win­ter 2025/​26

  My issue is with the implicit expectation that to hold a successful conversation with someone here you have to first do your own research on their positions. This is not how I understand a Socratic conversation to work, and I hold such a conversation as the ideal. Socrates did not ask his interlocutors to go familiarise themselves with his arguments. He simply made them in real time.

  Is this not the ideal? Is this not an expectation here?

• delphix 28 Feb 2026 1:13 UTC

  1 point

  0

  on: Open Thread Win­ter 2025/​26

  This seems like a community that requires every user to agree with its particular beliefs. If that’s wrong, correct me, but that’s the impression I got from reading the introductory post.

  So my question is, do you have no place at all for people that might disagree with you?

  And if not, doesn’t that allow for the possibility of being stuck in an echo chamber and keeping out people who might understand things better than you?

  Also, please direct me to another place online where I might simply discuss my disagreements with others without having to sign up to their beliefs, if you know of any. And either way, I advise you to create a space for that here. You can very easily separate one part of the community from the other, unless you fear that the one I’m describing would prove to be popular even with those currently satisfied with the older.

  You’re also proving yourselves to be very unapproachable to the average person that doesn’t want to read a bunch of documents before they can talk to people or be policed by upvotes(Jesus). Does it not occur to anyone to maximise some level of good ethos among the population that wouldn’t sign up for accepting what amounts to a religion?

  No matter how bad faith this will surely be construed, it is in fact meant to be constructive and Socratic, but doubtless no community that moderates itself in this way could easily conceive of the virtues of open discourse of the kind I am both attempting and encouraging here. I am curious as to any answer from whomever it might be. Thank you.

  What links here?

  • papetoast's comment on pa­petoast’s Shortforms by papetoast (12 Mar 2026 3:45 UTC; 0 points)