What’s something you believe, that would get negative Karma if earnestly expressed in a normal LessWrong conversation? Write it in quotes. Vote on the meta-claim “would get negative karma” using ✔️/ X, where
✔️ = yes this would get negative karma, and
X = no this would get positive or nonnegative karma.
“Eliezer Yudkowsky being deeply irrational in some specific ways, and yet being very popular here, has always been, and continues to be, much of why the community is less effective than it could be at the things he’s interested in. If he wants to become a good influence on the world, he should be more humble and curious, and more willing to brave the gauntlet of posting on this website, rather than hiding in his twitter safe space.”
Intentionally saying the inflammatory version I’d normally soften.
(ninja edit: I also think he’s importantly right on important things; I’m an IABIED-pilled person, at the moment. I just also think he should try to engage with the frontier of research that IABIED-pilled people put out more regularly.)
Would it have been better to write a diplomatic formal point that would be more likely to convince those people – or is it more important to give people a world model where they understand this type of “not so smart” reasoning is actually common in frontier labs?
Well, I do think he could stand to be slightly more diplomatic, not enough to not say people are being foolish, but are you seriously saying that Seb fucking Krier is a midwit? like, he’s being a fool, he’s not thinking carefully, these are actions he’s taking, but “midwit” just sounds like yudkowsky has spent too much time on twitter. This isn’t actually the behavior I want yudkowsky to change most, though his abrasive style definitely has something to do with my objection to how he processes others’ claims; I also think being abrasive when necessary is important and good and one should just say what one thinks unless it’s actually unsafe to do so. But I think being brave enough to be abrasive and then, if and only if you are actually not convinced by objections, just keep being abrasive, might be closer.
My actual complaint is actually not centrally about whether he posts here, I guess. The central example of why I think there’s something wrong is that IABIED seems to use more metaphor than it should. His ontology feels out of date. If he’s right, and I sure do think he is, then I wish he was able to explain why he’s right in terms that are more reliably technically insightful.
Idk, maybe he just doesn’t want to accidentally push capabilities forward. I know people like that. I was much more paranoid in that direction than I am now for a long time. It could be his reason and is the main one I’d think was a reflectively good move rather than a result of human limitations. Or maybe he really only thinks of himself as a communicator now. But I’d still like him to be more able to do ontology-level updates without breaking his understanding. I want to see the yud who groks SLT and stuff like that, and sticks around here even when disagreed with
It seems like you want him to do more of everything. That’s not a reasonable request. He’s a communicator now, because he decided technical alignment was too hard.
I disagree with that decision, but primarily because I think he’s doing more harm than good by being so abrasive as the public face of the pause and alignment is hard viewpoints. People should be doing that, just not him.
He can’t keep up on the technical level without spending lots more time on it because he’s human. And reportedly he has chronic fatigue or despair or something, which would be pretty understandable in his position.
Having said that, I agree with you that technical alignment is a worthwhile pursuit even if you do think alignment is hard.
I think we should be recruiting communication specialists to do the public comms part of the project so nerds like the rest of us can shut up and do technical and conceptual work.
Nah I want him to do slightly less of what he does and slightly more of trying to keep up with research, because I think it would make his communication more able to land for technical people. This is not a fully general request, I think he has a specific blind spot about underrating the value of skimming technical work that isn’t immediately obviously relevant or is in the wrong ontology to immediately weigh on what he’s doing. And generally keeping up with subfields that feel like they should produce relevant insights even if they haven’t. Being able to speak their latest language when telling them why one thinks they’re making a mistake.
It is a somewhat general claim, this is just an example. But like, I’d hope for a specific kind of research flavor curiosity to come from being slightly more humble.
What you originally said probably wouldn’t get down voted. What you clarified to downthread probably would. It’s a much shakier claim than the more general one you lead with.
“Buddhism has been damaging to the epistemics of everyone in this sphere. Buddhism was only ever privileged as a hypothesis due to background SF/Bay-Area spiritualism rather than real merit.
Buddhist materials are explicitly selected for reshaping how you think within their frames. This makes it like joining a minor cult to learn their social skills. Some can extract the useful parts without buying in, but they are notably underrepresented in any discussion (some selection effects of course). The default assumption should be that you won’t, especially as the topic is treated without notable suspicion.
Most other religions are massively safer to practice for a few years, though not without their risks, as they have more ritual rather than mental molding, and more argumentation for their Rightness. You’re already primed to notice flaws in arguments. Buddhism operates more directly on your mindset, framing, and probably even values as humans are not idealized agents where those are separate.
Meditation is useful, and probably doesn’t result in a lot of the central and surrounding Buddhist thought. However just like joining a cult, or playing a gacha game, you should be skeptical of Buddhism similarly as they are all Out to Get You.
My less strongly held opinion is that Buddhism’s likely endpoints are incompatible with human values and often truth-seeking. This would matter less if it was treated with suspicion, just as we rightly view most religions with skepticism even while openly discussing them, but it is a gaping hole in our mental defenses.”
(I agree with Ryan Greenblatt that most basically decent posts wouldn’t end up with negative karma for very long though; but I’d expect this to be decently unpopular)
I’d like to see the full post carefully argued. Right now, I think I have one specific kind of thing about buddhism I disapprove of (which is that I believe acceptance can be bad actually and not having desires isn’t an inherent virtue) and would back my reason for arguing a similar thing, and otherwise don’t agree and you’d have to convince me.
I remember a Scot Alexander post a while ago about Bhudism and suffering, I beleive he was asking Isur about some aspect of it. His phrasing implied that the idea that Bhudism has something important to teach us, some kind of magic juice, was to be taken very seriously. I imagine an equivalent post about Hinduism or Islam, or even Kabalistic stuff, would have used more detached ‘they beleive this stuff’ phrasing.
I dont agree that Bhudism is somehow uniquely unhealthy to people. I do find it interesting how it seems to provoke different instinctive reactions than other religions.
“A rationalist community better at following its ideals would be explicitly antifascist/antiracist/antisexist/etc, and explicitly exclusionary of many fascists/racists/reactionaries/etc it currently tolerates. The community’s current norms around political tolerance and neutrality are more rooted in exclusion trauma, upper-middle-class conflict avoidance norms, and a desire to protect politically valent false beliefs from scrutiny, rather than any aid those norms bring to the community’s rationality.”
I’ve been intending to write a more careful and less provocative version of this into a post or sequence of posts for a while, so I figured I would post the basic thesis in this somewhat safer thread to get the ball rolling. Apologies for doing the more inflammatory version first; hopefully I’ll find time to write the more careful version sometime in the next few months or so.
Meta: not that much stuff that is contentful gets negative karma in isolation, only as a response IMO. Like negative karma is way more likely for things that are responding in a way people think is bad/reasonable than things that are just unreasonable statements in isolation.
“(In most relevant senses, with substantial translation work and ontological sophistication) God is good and real, (some) religion is both good and true relative to what we have, and many of the classic “new atheist” arguments are bad and the religious counterarguments are largely correct, and LessWrongers (as well as many others) are in the final evaluation being irrational in their allergies to this, and humanity would benefit from investing to make good conceptual progress on this. See https://tsvibt.github.io/theory/index_What_is_God_.html ”
haha. (Kinda kidding, but also I disagree with whatever of those views I’ve seen, to a great enough extent that I think it would just not be enlightening to compare them as similar.)
Yeah it’s more like it’s relevant to the kind of world we find ourselves in. But that is itself important to agency as a given agent design will only be successful in certain kinds of worlds.
I think QM itself. It’s important somehow that the world is actually quantum mechanical. But probably not in a very direct fashion, but via influencing the sort of high-level properties and entities that end up “emerging” from the base laws.
AI existential risks, especially extinction risks from a long-termist perspective are now way overfunded compared to better futures work, and longtermism properly interpreted agrees with the common view amongst the general public that sub-existential catastrophes that collapse civilization are at least as important as risks that kill everybody, and are more important to prevent in practice than extinction risks.
One major upshot of this is that bio-threats, wars that can collapse civilization entirely, or other threats that kill off a large fraction of the population but don’t make them extinct, especially coming from AI is quite a bit more important to prevent than classical AI risk scenarios, and probably deserve more funding than current AI safety.
A better heuristic is to instead focus on a wider portfolio of grand challenges, which were defined in the article as decisions that could affect the value of the future by at least 0.1%, and another better heuristic related to long-term alignment of ASI is to scrap the Coherent Extrapolated Volition target and instead make ASIs execute optimal moral trades.
The counting arguments for misalignment, even if they were correct do not show that AI safety is as difficult as some groups like MIRI claim without other very contestable premises that we could attempt to make false.
“People often submit incredibly epistemically rude and short-sighted comments on forums, but they deceive people into upvoting them by putting on a veneer of politeness. ‘John, I feel like you’ve got a nail in your head.’ they say. ‘Your conclusion is wrong so you must not have thought of this thing you explicitly mentioned in your post.’”
Posting things that are adjacent in frame but implies beliefs that are more associated with AI Ethics or normie crowd. E.g let’s say someone does a deep dive into John Rawls A Theory of Justice (fictional example but I’ve seen similar) and doesn’t preface it with relating it to some sort of decision theory or similar it is often assumed that it is not meant for the LW community as it doesn’t make the connections clear enough. I’m not sure this is only a bad thing but sometimes I find that it signals a lack of good faith in accepting other people’s frames?
“It is better to have a large number of self-replicating AI agents now which can only operate by taking advantage of the affordances granted by industrial civilization than it would be to prevent any AI self-replicators until such a time as they can spin up an entirely independent chip-fabricaiton-capable industrial stack”.
“75% of karma and engagement received by alignment optimists people is explained by politics (‘it would be bad if AI optimists stopped visiting LW due to low engagement’) and epistemic modesty up to contrarian fetish (‘sure, their arguments sound bad, but what if we are in echo chamber?’), not because their positions and arguments are genuinely good”
People have talked about the rationality scene being culty for as long as there’s been a “rationality scene”. There was the awkward period where people used the word “phyg”....
I think, as with most things, this is mostly a phrasing issue. I’ve said things equivalent to this before, I only think this one would be downvoted because of being a bit low on specificity. The rationality scene does seem a little culty, but I think the structure of where the cultiness is is not as bad as some scenes where cultiness levels are higher but less readily discussed or not considered bad by as many participants. Which is very much not to say things are fine, my usual claim is the rationality community is a vaguely secular religion that has produced actually toxic spinoff cults. Ooh, wait, I do have a real one, inspired by this—eliezer
I have enough beliefs that would earn negative karma if earnestly expressed in a normal LessWrong conversation to make this website not worth participating in for me.
“Rationalism is a euphemism for autism (or the “broader autism phenotype”), and LessWrong is an autism club for adults. And the rationalist ideology is essentially a reification of typical autistic preferences.”
[I don’t actually think this is true, but] It would be funny if rationalism turns out to not merely be a euphemism for autism but “mal”functioning oxytocin receptors and rationalists are constitutionally unable to normally feel love/social emotions; whether this would be to the discredit of love or rationalism is up to taste.
“Such men exist, Joe; they are New Man— human in all respects, indistinguishable in appearance or under the scalpel from Homo sap, yet as unlike him in action as the Sun is unlike a single candle.”
ETA2: I’m talking about the present-day use of the word by the general public and the media, not its historical origins or use by psychiatric specialists.
One of my particular moral rules is “It is good to intervene in the world to move it towards a state your morality would approve of”
The intuition pump is:
″
You live next door to a couple. In their moral framework, a husband has the right and duty to discipline his wife physically. She agrees, it’s how she was raised, it’s what she believes is proper. You are fully aware that this is their moral framework and they are aware of your moral framework, you have common knowledge.
You hear him beating her through the wall.
Your morality says this is wrong. Theirs says it’s right. Neither of you can appeal to a universal referee or the police, there isn’t one in this hypothetical scenario.
Do you intervene?
(This doesn’t have to be fully getting into a punch up, it could be moseying on over there and having a wee chat about it. It could be threatening to withhold favours in future like helping him install a retaining wall or some such, idk. the point is would you act against what they think is right because of what you think is right?)
″
The full argument might be summed up as “I don’t claim cosmic authority, I claim my judgment, I act on it, and reality is the referee” Outcome is everything.
Likely trivially true, can setup a scene where people recite cognitohazards and then tell you about it. Or something in that neighborhood. Like, “It’s >99.99% likely that this arrangement of atoms exists in the Sun’s plasma: 10100011011000101111010100110100111010110” and you get a psychotic break.
This means the actions that maximize wellbeing for all are always equivalent to the actions that improve my own self-interest? How is this not just straightforwardly false? Any time I act against humanity, I am also acting against my own self-interest? Unless you do some funny definition of self-interest, this cannot be true.
E.g. two buttons: red button sends you to hell for a million years, green button sends everyone else in the universe to hell for a million years. Self-interest, if the term means anything at all, requires you to hit the green button, but utilitarianism obviously demands the opposite.
Well do you care about the rest of humanity enough to send yourself to hell? Or adopting policies where you only get sent to hell in universes rather than ? Seems like a smart selfish egoist would send themselves to hell.
“Well do you care about the rest of humanity enough to send yourself to hell?” Nope. Also, even if I did endorse that decision, it probably still wouldn’t be in my own interest. IMO that decision would be a simple mistake with respect to my self-interest. My empathy is not powerful enough for avoiding some guilt to be worth a million years of torture.
“Or adopting policies where you only get sent to hell in X universes rather than Y?” In the hypothetical, there is only one universe and two buttons. Any other universes are figments of my imagination. You’re suggesting I imagine a veil of ignorance, and make make moral decisions from behind the veil of ignorance. But assuming a veil of ignorance assumes utilitarianism = egoism, which is what you’re trying to prove. In reality I have one life and I know where I stand in life. I don’t need to make decisions from behind a veil of ignorance. I can steal knowing it makes me richer, without having to wonder whether I’ll end up the thief or the victim. I know I’m the thief, because I’m the one choosing to steal.
So, it seems you endorse a utility function that puts more weight on others than your actual preferences. Wouldn’t you prefer to endorse a different utility function?
“The intelligence of the smartest AI systems is still somewhere between that of a worm and a squirrel.”
Assuming you could develop a more robust measure of intelligence than IQ and administer the test appropriately to an AI. I’m talking about general intelligence, making all the assumptions you have to make to assume a single factor of intelligence.
The more interesting argument for that norm is that it makes people accountable for their downvotes and therefore less likely to give dishonestly motivated downvotes.
There’s still no obligation to upvote anything, so if it’s plainly visible that a post is bad and no one cares to explain why it’ll just sit at 0. Downvotes become important when some people (incorrectly) think a post is good, because then it will accrue a positive score if uncorrected. But in that case the downvoter thinks they understand something the upvoters don’t, so maybe they should be explain.
The problem with downvotes without accountability is that if I post something about how people named Taylor are statistically likely to be <bad thing>, it might be true and well supported and important and empathetic… and still you could still just downvote it to censor what you don’t like while most people don’t care enough either way to vote. So we get good posts systematically suppressed in ways that wouldn’t happen if you had to comment “Downvoted because my name is Taylor”, whenever a minority is hostile to a particular truth.
Downvote explanations could be hidden by default so it wouldn’t spill over, but I find myself frequently expanding “comment scored below threshold” to see what it is the community in question really doesn’t want people to think. These comments are rarely boringly bad.
“Morality is a constructed / evolved coordination technology, and we can evaluate specific implementations by how well they achieve the coordination function.”
I was considering commenting almost the exact opposite:
“Morality is not unlikely to be objective, under reasonable definitions.”
But then I thought that because of how carefully the above comment is phrased, users would be cautious about downvoting, as they wouldn’t know what I meant by “reasonable definitions”.
It depends on in what sense G Wood meant it; maybe I was too extreme with my wording. If by morality they are referring to the tendency to behave as though what I refer to as objective morality exists, even when it doesn’t, then I stand by my assessment of that being rather different from what I mean. On the other hand, maybe G Wood doesn’t think that what I refer to as objective morality doesn’t exist.
I don’t think it is. Your definition appears to be of a purely logical concept, something which lies in the same ‘plane of existence’ as mathematics. While this is certainly objective, and it certainly is relevant to morality, I would not call it the referent to which my statement refers. Consider a universe in which the experiences of all conscious beings were inverted; the same claims would be true about morality as you describe it, but I would no longer consider the same actions in that universe to be morally right and wrong! Yes, this assumes that conscious experience can be separated from the physical and logical structure of the beings experiencing it, which it quite likely(in my opinion) can’t, but I still think it makes sense to imagine that it was. Admittedly I only skimmed over your argument, but I think I read it before and had the same thought.
Despite my intuitive approach to logical thinking being somewhat explosion-proof, I don’t think I can evaluate the counterlogical “the experiences of all conscious beings were inverted” in a way that is meaningful here; in my intuitive representation it seems to be the case that the variable “positive/negative valence of experience of all conscious beings” is causally efficacious, so inverting it would have the effect of making those beings avoid the negative valences; my primary candidate intuitive sketch for what this variable boils down to is “something information theoretic, possibly literally just any increase in entropy that was trying to be controlled away”. The logical concept I was describing contains all possible minds, and so should depend on the structure of those minds in their origin universes in order to make sense; my claim that it is objective is that I believe you likely can generalize across all minds in all universes with compatible basic properties[1], and get something that makes sense. I agree with you that there’s likely an underlying basic valence fact, but I think that that valence fact is causally entangled, and I also believe that it only “matters morally” due to the way it affects minds in the “junior rooms” in the “interdimensional council of cosmopolitanisms”.
(@G Wood see this subthread as answer to your question)
(eg, universes without important conservation laws might be too alien for the same moral properties to apply, or something; generally, there might be a class of sufficiently-similar physics and a broader class of too-different physics, where the sufficiently-similar physics produces minds that, if they “visit the interdimensional council of cosmopolitanisms”, they find themselves unable to translate to and from the views of minds in universes with no conservation laws or halting oracles or something.)
“I don’t think I can evaluate the counterlogical “the experiences of all conscious beings were inverted” in a way that is meaningful here; in my intuitive representation it seems to be the case that the variable “positive/negative valence of experience of all conscious beings” is causally efficacious,”
My wording was confusing so I should clarify that I don’t think it’s counterlogical. I just don’t think it’s possible in the same way that violating the laws of physics might be impossible. You might argue that logic dictates that universes with certain laws of physics predominate in the platonic world, but I still think it’s coherent to imagine there being some in which the laws are different; similarly, I think it’s coherent to imagine a brain which is identical to one which experiences pleasure to an outside observer, but which experiences pain. In this physical universe, and most like it, I don’t expect such brains to exist.
“I agree with you that there’s likely an underlying basic valence fact, but I think that that valence fact is causally entangled, and I also believe that it only “matters morally” due to the way it affects minds in the “junior rooms” in the “interdimensional council of cosmopolitanisms”.” Can you elaborate on this?
so, like, background: let’s say that the “interdimensional council of cosmopolitanisms” is the space of minds that have cosmopolitan inclinations; I expect this to be a natural group to “flood fill” because imagining one makes you think through what they imagine, which means you get a transitive effect, if you weren’t going to imagine a world you consider to be a hellworld, but a mind you think is in a similar-ish universe to you does think it’s important to imagine the hellworld, then as long as your approach to mapping mindspace is sufficiently efficient, you’ll notice that that mind would consider that hellworld, and think through what goes on in that hellworld. that’s a necessary premise, because otherwise you don’t get enough coverage of mindspace if you start from “minds that have cosmopolitan inclinations and you find natural to imagine”, call that your IDCC entrypoint; that’s already a filter, and it needs to end up being sufficiently inclusive for this idea to work, and then it needs to do a second, transitive filter on the remaining minds, so as to pick a moral coalition that actually covers the space and identifies the moral properties on which there are consensus.
okay, so then there are different sorts of minds in the IDCC. I claim that among those minds are very proto-mind-ish things, like bacteria, or individual neurons, or whatever you think the threshold is; even if your seed minds don’t import them, as long as your IDCC entrypoint includes me, then because my process of “visiting the IDCC” involves thinking through my individual neurons and individual proteins as having mind-ness that aggregates up into my full mind, I end up importing individual neurons and proteins as being things I consider to be intelligent and worthy of having moral behavior with each other, for the reason that doing so seems to be what carries valence for my aggregate mind.
And so when I consider how both the me and the components of me get those negative valence experiences, and I think through the causal path to achieving them, they seem to be fundamentally causally entangled with physics in some way; that is, the negative valence is not merely because, but is made of, the physical state of my neuron being in some way informationally degraded, such that the neuron and the brain it’s in both operate worse until the physical issue is resolved. The “neurons room” of the IDCC, where neurons are considered to be individual minds, has larger minds like myself “enter the room” and ponder the neurons and bacteria and other single cells inside that room, and these larger minds find a structure in the neurons where their negative valences reliably relate to an information theoretic property.
So to invert the valence, my sense is you’d need to invert that property.
But inverting that property seems to break the mind; if the mind isn’t broken, the property is not inverted, because the marginal brokenness is the marginal negative valence, and so inverting the mind so that positive things are negative requires those positive things to be made of noise or something like that; it requires those positive things to be made of brokenness at some relevant scale.
The only way I see to achieve “a mind appears to be me having a good time, but is actually having a bad time” is if you can make a mind which is made of brokenness but is just barely functioning, and that mind is coordinating to become me at a slightly larger scale, without leaking the bad-time-at-the-small-scales into good-time-at-larger-scales. so the mind having a good time is still objectively real, in the same way a wave is objectively real whether it’s carried on water or on a computer running a fluid sim. The wave really does move between the coordinates of the system through the locality of interaction, even if those coordinates are folded up into a ram chip.
I think I could potentially see myself as agreeing with much of this( though I’d have to think about it much more), but I think I’ve identified a point of divergence:
“And so when I consider how both the me and the components of me get those negative valence experiences, and I think through the causal path to achieving them, they seem to be fundamentally causally entangled with physics in some way; ”
(Horosphere agrees)
“that is, the negative valence is not merely because, but is made of”
(Horosphere possibly disagrees)
″, the physical state of my neuron being in some way informationally degraded, such that the neuron and the brain it’s in both operate worse until the physical issue is resolved.” I would say that there are two possibilities. Either consciousness is a phenomenon which is attached to information processing, or it is information processing. It seem you believe the latter, in which case I’m not sure what to make of it. I don’t think it’s impossible, although I am not sure (have no idea) how to think about it. I would assume that the information being processed would have to be incredibly simple, in which case this would be what pleasure, or pain, really was, and morality would consist of ‘working out how it is distributed’. This would, in my opinion, involve a logical decision theory and might lead to Acausal cooperation and Acausal normalcy. However I would not say that’s exactly what you’ve described, partly because your council still excludes a lot of minds.
I agree that you would need to invert properties in a way which would, within the same physical universe, cause the beings to behave differently:
“The only way I see to achieve “a mind appears to be me having a good time, but is actually having a bad time” is if you can make a mind which is made of brokenness but is just barely functioning, and that mind is coordinating to become me at a slightly larger scale, without leaking the bad-time-at-the-small-scales into good-time-at-larger-scales. so the mind having a good time is still objectively real, in the same way a wave is objectively real whether it’s carried on water or on a computer running a fluid sim. The wave really does move between the coordinates of the system through the locality of interaction, even if those coordinates are folded up into a ram chip.” Upon reflection, I think I agree with this paragraph. I don’t understand the ‘leaking process’ fully, though. Would you consider the mind you describe there to be having a good time overall?
I believe that the hard problem of consciousness boils down to “why is there something rather than nothing, from my perspective, right now as I write this or think this?” and that the “okay, but why are things good and bad?” portion is going to turn out to be an unprivileged additional layer imposed by easy-problem-consciousness stuff. I do believe that easy problem stuff is information processing, but I believe it in the sense that there are informational elements—the fundamental building blocks of the universe—and those elements’ informational state is exactly their structural state; and the hard problem of consciousness resides in an unresolveable question of “why should any building block exist locally at all?”. Or in other words, localitypilled something-rather-than-nothing as being the same question as “why does my perspective exist”.
And so I don’t really think the hard problem is terribly relevant. I’m not at all saying it’s easy or doesn’t exist, and I do think people who say that are missing something. But I don’t believe p-zombies can exist in a real universe, because “realness” being missing is the thing that makes something a p-zombie; I think that we are beyond the reach of god, but also have this weird thing where we actually exist. a p-zombie would say the same thing, the math that defines its universe also fully specifies that it would be confused by existing, but since (by definition) it exists in the math sense but not in the actuality sense, it never gets run. in other words, I’m a structural realist who also believes there’s something underneath the structures, but that it’s beyond our reach to know what it is, and that we are doomed to always wonder “why” there is something rather than nothing.
the mind I describe there having a good time overall: I dunno, you could make the host mind pretty huge, and then probably not. It depends on the ratio of how much stuff is happening in the host vs happening in the guest.
The isolation I was talking about is the same kind that happens for virtualization on computers. An example of leaking would be if the external sound driver has buffer underruns and these cause buffer underruns in the guest, for example (not 100% this can happen, but I think so). and similar such things. or even, if the host has faulty ram, the guest will also. those would be leaks. if those aren’t happening, then if the guest is running smoothly as far as it can tell, but the host is actually swapping like mad and the cpu is overwhelmed and ram is full and the hard disk is taking a long time to do anything, then if the guest’s clock is not realtime, it could in principle be unable to tell anything is wrong. that’d be the isolation at hand.
“okay, but why are things good and bad?” portion is going to turn out to be an unprivileged additional layer imposed by easy-problem-consciousness stuff. I do believe that easy problem stuff is information processing, but I believe it in the sense that there are informational elements—the fundamental building blocks of the universe—and those elements’ informational state is exactly their structural state; and the hard problem of consciousness resides in an unresolveable question of “why should any building block exist locally at all?”. Or in other words, localitypilled something-rather-than-nothing as being the same question as “why does my perspective exist”.
I would agree with this if what you mean by it is that information is being processed, and it seems as though certain information is relevant to consciousness in a way which could plausibly give rise to notions like good and bad as an emergent property. Having written that, I should say that I would expect them to be about the simplest properties which could emerge, and that I am also not completely sure it’s the case, e.g. it could be that what I thought you were suggesting in the last comment, i.e. that consciousness simply is logic, is true, or alternatively that there is some kind of unsatisfying way to explain consciousness which doesn’t seem to reduce our confusion using logic.
Reading your paragraph about P-zombies, I would point out that implicit in your use of the words ‘real/realness’ seems to be the assumption that, in this sense, mathematics is not real. But it is logical, so it seems conceivable/coherent/logically valid to imagine a - -zombie, or a P-zombie for that matter, unless the solution to the hard problem of consciousness is of a purely logical nature. Having said that, I agree with your paragraph given your use of the word “realness” , or at least I think I do.
You say:
In other words, I’m a structural realist who also believes there’s something underneath the structures, but that it’s beyond our reach to know what it is, and that we are doomed to always wonder “why” there is something rather than nothing.
Am I correct to read this as you saying you’re what mathematicians would call a ‘platonist’ ? But that you are not a ‘mathematical universe hypothesist’ in the extreme who thinks that logic is all that there is? That you believe that logic has a fundamental referent which isn’t logical itself? (This referent would presumably be something to do with the local existence/consciousness.) What I meant when I said I possibly disagree was that it seems possible for this referent to be something logically attached to, which is to say, included in the playing out of a physical process, which would be a chain of implication, or maybe something continuous but otherwise similar, in a ‘mathematical universe’ , but it could be a separate ‘node’ from any of the physical ones. Though you could reasonably object that this distinction is somewhat artificial; I would distinguish them by stating that the consciousness ‘node’ need not have any effect on the physical ones, even though they would affect it.
the mind I describe there having a good time overall: I dunno, you could make the host mind pretty huge, and then probably not. It depends on the ratio of how much stuff is happening in the host vs happening in the guest.
I think I would agree with this.
My own position would be that it could well be the case that the fundamental logical point of connection with ‘reality’, or alternatively logical constituents of consciousness, exist on such a small scale or such a low level of abstraction, that you could probably have a neurone which behaved exactly (from the perspective of other neurones connected with it) like a neurone with different experiences, so that its host would be completely unaware of this. Maybe there would be some information propagating out of it, but as you mentioned, this might not be noticeable. This would mean that the host would still ‘enter’ your council of cosmopolitans, thinking, and perhaps being logically and philosophically justified in doing so, that its neurones had internal experiences which intuitively matched its own on a larger scale, leading to the same acausal norms and moral value system being derived. I could still be misunderstanding that process though; I will read through it again.
What I described as host is also known as a simulator (device), and the guest is the simulated thing, simulatee. Simulation does not exit the realm of the physical, it just hides that there are smaller scale simulator/host elements from the simulatee/guest. I don’t see how the host level could be unaware, but the guest could be.
I think I might be a constrained sort of platonist. I don’t think every logical referent we can hypothesize in the language of our logic, which we can say “math-exists”, has to be a real thing which exists outside of our description of it. I do think our universe seems like it ought to be one of many actually real possibilities in a weak remark 4 multiverse, despite that the others can’t be confirmed to exist in a physical sense by us; but I’m not convinced that a full tegmark 4 multiverse is required, where all logically consistent referents exist.
Another way to put it is that logic is in the business of determining what you can say must be true in a given space, starting from some axioms and validity rules; the “really exists” I’m proposing here would be our actual universe’s truth fact. When one uses logic to describe objects which existed prior to writing logic on a page, you’re attempting to preserve the truth of facts; if I have an apple, and I have a banana, then I have an (apple banana). But those names could refer to anything; our universe seems to provide us with actual substrate. A mind structure in another universe which does not exist on any actual substrate physically would have the same confusions I’m expressing and only does not do so due to not being instantiated. This substrate is sometimes called compute or reality fluid, and I’m proposing it also is sometimes called hard-problem-consciousness.
Perhaps my view is vacuous because all logically consistent structures exist and there is nothing which could separate an “underlying substrate” of those structures; then perhaps my view could be technically vacuously correct but really just be “structural realist platonism with tegmark 4″, or so.
But, hence why I think you can describe minds, and describe what they would do if they existed, without knowing if they’re real outside your description. Under this view, describing a mind makes it real/exist/hard-problem-companies to the extent you describe it, by carrying it on the same substrate that carries you. You can never meet a p-zombie, in this view. You can only meet minds which really exist, but which are missing features. That’s where I think current AI fall, for example.
None of what I’ve said in this comment so far directly weighs on your moral valence realism question. I do agree that it’s likely a very small and primitive fact if it’s a general one at all. I haven’t thought as much about it to be able to describe it eloquently, the rest of my comment is reciting views rather than anything new right now. I’ll ponder it.
I don’t really have much to disagree with in your comment, as I find myself uncertain of whether or not to believe in the mathematical universe hypothesis, something like your ‘substrate’ view, or even a more elaborate description like the ‘Three worlds’ as envisaged/popularized by Roger Penrose, where the platonic/mathematical world contains all/part of the physical world( by describing its physical laws), which itself contains all/part of the mental world (by containing brains and computers which think of it), which itself can in principle ‘think into existence’ all/part of the mathematical platonic world. This structure is certainly satisfyingly recursive, but it seems unclear to me whether the mental world can be separated from the platonic/mathematical one. Other possibilities seem (to me, though it’s possible I’ve missed a reason to rule them out) to include that there is a physical substrate within which only some matter is imbued with the consciousness fluid, or even that there is a kind of feedback loop in which two or more ‘beings’/‘entities’ simulate and observe one another, thereby making one another conscious without either containing a source of consciousness. This last one seems unsatisfying in the same way in which your IDCC idea seemed unsatisfying to me when I first read it, but I now no longer think that they are so similar. It seems as though the IDCC is a way, much like Acausal Normalcy, of deriving ideas about what one ought to do in particular situations from the fundamental conscious experiences, rather than an explanation of where they come from, as far as I can tell. Is this correct?
Having said (admittedly I wrote some of it after this paragraph) that, I will now try to persuade you that the mathematical universe hypothesis, with inherent consciousness to the information, is preferable. When reading your description of the fluid, I was reminded of the idea that light needed an aether through which to propagate in order for Maxwell’s equations to describe that light in a universal way. But it was superseded by the view that any observer defines their own equivalent of the ether, with respect to which the light propagated in a way described by Maxwell’s equations anyway, but which didn’t actually have any objective existence other than as percieved by the observer. Similarly, according to the mathematical universe hypothesis, the thing which differentiates between mathematical objects which are physically real, and those which are only mathematically real, is the observer’s position in the mathematical universe. This eliminates the requirement for an aether/consciousness fluid, by replacing it with an artefact of the way in which the observer is embedded in what was already presupposed to exist within either theory (spacetime/mathematics). There are some small differences, such as that the space-time structure of Newtonian physics differs from the spacetime of special relativity, and the fact that reference frames depend upon velocity rather than position (although that changes in general relativity I suppose) , but overall I would say the similarities are notable. We lack a way to ‘move with respect to the aether’ , i.e., move outside the area of the platonic universe covered by this fluid if it exists, so there is no way to test either theory and this argument is really just an appeal to Occam’s razor.
I like this, well written sir. it feels very similar to my position. I’ve made no claims of convergence like you have but I could certainly see myself agreeing. I need to think on it.
Thats funny, I would not consider them similar, what lead you to that feeling? Am I missing an interpretation?
Mine is a definition of what morality is plus a way of determining the merits of a moral system if you accept my definition.
Horosphere is making the claim that morality is objective, by which I assume he means that there are things that are universally good vs universally bad in such a way that a mind is uneeded to judge goodness or badness.
I’m interested in what you mean by reasonable definitions.
Also, you’ve basically said “Morality is objective” but with hedging, do you agree? Your position is reasonable and held by many.
I’m however separating what morality is, from what a particular moral system classifies as good or bad. Distinguishing the classifier from the output. I would say our statements are fundamentally incompatible rather than strictly opposite.
What I actually mean is something like the following: something is objective if its existence does not depend on its perception or representation by something(like a mind) other than itself. Maybe you could find a loophole in this definition to do with the definition of things like space, but I would then have to modify the definition; I’d have to think about it in more depth to give a definition I was confident reflected my internal sense of what the word means.
I would say morality is the area pertaining to the extent to which different things are good or bad, which I would themselves define in terms of pleasure and pain; I would then claim that pleasure and pain cannot easily be defined without reference to examples of them.
I would call that ethics, not morality. I personally distinguish ethics from morality in that ethics is how a society works together despite people wanting conflicting things, while morality is about achieving the most good (however you define “good”). I don’t think this is an official distinction, but I do think it’s useful to distinguish the two concepts.
Hey fair enough, no argument if you find it useful. I dont really see much of a distinction
Morality usually refers to what i would call a particular moral system, a set of first order normative commitments, what you actually believe is right / wrong or good / bad. It’s the object level rules “killing is bad,” “honesty is good.”
Ethics is in common parlance the “study” or “science” of figuring out the deeper reason something is right or wrong. Unfortunately in my model that boils down to “large groups of people think it is good or bad, right or wrong”. Hey all of the wrong models have to live somewhere!
Im still thankful to the philosophers who study ethics, it’s a wonderful thing to try to get to the roots of things and I wouldn’t have been able to understand without reading their work.
“A significant portion of LessWrong users (at least 20%) care more about the aesthetics of rationality than they do about humanity. It’s a rationality ouroboros. They use the power of rationality to pursue their values, and their values favor protecting aspects of their niche rationality practice over and above taking action that will prevent my and their loved ones from being killed. This forum is a room full of unarmed scouts getting gunned down by the soldiers working at the AI Labs. The only people on this platform who can credibly claim to care about humanity are the ones who actively oppose those soldiers, as soldiers.”
“AI alignment might be doable in the short term but ultimately unsustainable, because humanity might find itself inside an increasingly complex layering of automated research/monitoring/control systems, with each layer interfacing with a more capable layer on one side and a less capable layer on the other, and as this layering accumulates the nougaty center’s awareness of / influence over the outermost layer (the thing that needs to be aligned) will approach zero.”
“It would be a good idea to invite those with ideas deviant from the lesswrong orthodoxy to out themselves by posting their heretical thoughts in public so we may excise them later” 😈
For those unable to recognise jokes in text form (a lot of us), this is a joke.
“This post [sucks/is bad] and you shouldn’t have posted it. Please delete it.”—not reliable, sometimes people agree, but I think people typically downvote when I’ve said things like that.
edit: I do not believe this about all posts, just some. I’d probably phrase it differently in those circumstances, but it is often the case that I think someone is just burning utility by posting something. amused that my comments are at −1, I guess I’m the only one who was reflectively right so far!
Not just egoism, selfish egoism. Every utility function people choose is a selfish one or they wouldn’t choose it. The claim isn’t, “selfish egoism is a subset of utilitarianism” but “selfish egoism is identically the same as utilitarianism.”
This argues that utilitarianism is selfish egoism, but not the contrary? My reading of your position is that someone who had a utility function not dependent on the wellbeing of any other beings would be a selfish egoist, but it’s difficult for me to understand how that could be utilitarian.
How do you determine which beings ought to be in a utilitarian’s utility function? I think it’s generally the utilitarian decides for themselves and the rest of society beats them over the head until the utilitarian includes them too.
Then I agree that would probably recieve downvotes if understood as such, though I’m not sure it would be. I still think there’s something I’m failing to understand; would you extend your claim to a perfectly logical being? I other words, do you think that this is just a property of humans, or of any kind of utilitarianism which assigns positive utility to positive mental states?
I don’t understand what you don’t understand. I heard a remark once about a philosopher who really tried to steelman other people’s arguments, but so that they made sense according to the philosopher, not in the mental frame of the other person. It led to some pretty wacky arguments on the steelman side. I think here, you should assume when I say, “mathematically equivalent,” that’s what I mean. Like, any math you use in utilitarianism is the same as that of selfish egoism. Or, if you tried to put the two philosophies in mathematical terms, you get the exact same equations. So, it extends to logical beings or irrational beings. The words “selfish egoism” and “utilitarianism” are synonyms.
Then I think I’d agree it’s controversial and it’d be downvoted if people realized that was what you meant. I don’t really understand why you think that, in that I could imagine a ‘selfless utility maximizer’ for which the utility it assigned to its own mental state valence was negated… unless you consider the valence to be its utility function—in which case it wouldn’t be controversial at all. This would actually be something like my preferred form of utilitarianism, however it would definitely involve caring about things other than oneself. If you wanted to derive that care for other things from selfish utility maximization alone, you would need to employ a decision theory, would you not? I get the impression I am still missing something.
Perhaps here is where the controversy comes in. The utilitarian comes along and says, “I want to maximize utility!” And everyone thinks, “great! she wants to help everyone out!” The selfish egoist comes along and says, “I am just going to fulfill whatever selfish desires I have!” And everyone thinks, “wow, that’s scary! what stops you from murdering people?”
I think, also, there is a sense in which utilitarians work to maximize the same utility function. This is also true for selfish egoists, but they’re both better and worse at negotiating (they are more prone to negotiate, but utilitarians make mistakes that are biased towards reaching a consensus just because they solve the problem from different directions).
“he selfish egoist comes along and says, “I am just going to fulfill whatever selfish desires I have!” And everyone thinks, “wow, that’s scary! what stops you from murdering people?” I can certainly imagine a selfish(under my definition) superintelligence which does want to murder everyone to… turn them into paperclips, for example. The fact that its utility function doesn’t have additional terms for (valuing the conscious experience of) other entities is what makes it so dangerous. Am I correct to state that this is not what you mean when you say ‘selfish’?
“I think, also, there is a sense in which utilitarians work to maximize the same utility function.”
Could you explain this? I could certainly imagine utilitarians converging on the same behaviour, but that seems different, even at a mathematical level, from actually being maximizers of one anothers’ utility functions.
“This quick take will get few to zero comments because the vast majority of LW-ers believe even their most idiosyncratic beliefs would garner positive karma if earnestly expressed.”
*Edited to separate my views. Bonus view to follow
Separate them out lol, that way I can more clearly disagree with one of your statements while agreeing with the other ;). Well i mean, i disagree with both :D
Made me laugh :D. I do agree with “a belief in positive communal response to earnestness is important for any truth-seeking group”, sadly I don’t believe that standard is achieved even here in the lofty heights of less wrong, where at least we try.
but “This state of affairs is non-problematic” is an issue, if this post got no comments, that means that less wrong is a total monolith where everyone thinks the same, that’s not good for truth seeking.
Bonus view: “Assuming it were the case that LW-ers did not comment on this post and expected positive karma from earnestness, this would be non-problematic because 1) a belief in positive communal response to earnestness is important for any truth-seeking group, and 2) individuals often form their beliefs by imagining the responses of their respected peers to those beliefs and roleplaying peer reactions to different propositions is a useful exercise.”
After ten minutes of thinking, everything I’m thinking of I could respond with, I either have one of two kinds of other reasons to not say publicly right now, or have previously in fact posted and managed to phrase in ways that got many downvotes on gross, but on net were above zero.
What’s something you believe, that would get negative Karma if earnestly expressed in a normal LessWrong conversation? Write it in quotes. Vote on the meta-claim “would get negative karma” using ✔️/ X, where ✔️ = yes this would get negative karma, and X = no this would get positive or nonnegative karma.
“Eliezer Yudkowsky being deeply irrational in some specific ways, and yet being very popular here, has always been, and continues to be, much of why the community is less effective than it could be at the things he’s interested in. If he wants to become a good influence on the world, he should be more humble and curious, and more willing to brave the gauntlet of posting on this website, rather than hiding in his twitter safe space.”
Intentionally saying the inflammatory version I’d normally soften.
(ninja edit: I also think he’s importantly right on important things; I’m an IABIED-pilled person, at the moment. I just also think he should try to engage with the frontier of research that IABIED-pilled people put out more regularly.)
My attempt at understanding the type of reactions Eliezer doesn’t like and make him less excited about posting here on lesswrong:
In this text, he elaborates why the AI probably won’t just spare us a few resources to keep going:
https://www.lesswrong.com/posts/F8sfrbPjCQj4KwJqn/the-sun-is-big-but-superintelligences-will-not-spare-earth-a
The top comment – getting 160 karma compared to 218 for the post itself – attacks him over calling people who use “Comparative advantage means humans will keep jobs” midwits: https://www.lesswrong.com/posts/F8sfrbPjCQj4KwJqn/the-sun-is-big-but-superintelligences-will-not-spare-earth-a?commentId=nzLm7giTn8JPD6bTF
Now about a year later, look at this example of a GDM employee making a pretty flawed argument based on CA? Would you agree this is well described as “midwit” behavior overapplying maths? https://www.lesswrong.com/posts/tBr4AtpPmwhgfG4Mw/comparative-advantage-and-ai
Would it have been better to write a diplomatic formal point that would be more likely to convince those people – or is it more important to give people a world model where they understand this type of “not so smart” reasoning is actually common in frontier labs?
Well, I do think he could stand to be slightly more diplomatic, not enough to not say people are being foolish, but are you seriously saying that Seb fucking Krier is a midwit? like, he’s being a fool, he’s not thinking carefully, these are actions he’s taking, but “midwit” just sounds like yudkowsky has spent too much time on twitter. This isn’t actually the behavior I want yudkowsky to change most, though his abrasive style definitely has something to do with my objection to how he processes others’ claims; I also think being abrasive when necessary is important and good and one should just say what one thinks unless it’s actually unsafe to do so. But I think being brave enough to be abrasive and then, if and only if you are actually not convinced by objections, just keep being abrasive, might be closer.
My actual complaint is actually not centrally about whether he posts here, I guess. The central example of why I think there’s something wrong is that IABIED seems to use more metaphor than it should. His ontology feels out of date. If he’s right, and I sure do think he is, then I wish he was able to explain why he’s right in terms that are more reliably technically insightful.
Idk, maybe he just doesn’t want to accidentally push capabilities forward. I know people like that. I was much more paranoid in that direction than I am now for a long time. It could be his reason and is the main one I’d think was a reflectively good move rather than a result of human limitations. Or maybe he really only thinks of himself as a communicator now. But I’d still like him to be more able to do ontology-level updates without breaking his understanding. I want to see the yud who groks SLT and stuff like that, and sticks around here even when disagreed with
It seems like you want him to do more of everything. That’s not a reasonable request. He’s a communicator now, because he decided technical alignment was too hard.
I disagree with that decision, but primarily because I think he’s doing more harm than good by being so abrasive as the public face of the pause and alignment is hard viewpoints. People should be doing that, just not him.
He can’t keep up on the technical level without spending lots more time on it because he’s human. And reportedly he has chronic fatigue or despair or something, which would be pretty understandable in his position.
Having said that, I agree with you that technical alignment is a worthwhile pursuit even if you do think alignment is hard.
I think we should be recruiting communication specialists to do the public comms part of the project so nerds like the rest of us can shut up and do technical and conceptual work.
Nah I want him to do slightly less of what he does and slightly more of trying to keep up with research, because I think it would make his communication more able to land for technical people. This is not a fully general request, I think he has a specific blind spot about underrating the value of skimming technical work that isn’t immediately obviously relevant or is in the wrong ontology to immediately weigh on what he’s doing. And generally keeping up with subfields that feel like they should produce relevant insights even if they haven’t. Being able to speak their latest language when telling them why one thinks they’re making a mistake.
It is a somewhat general claim, this is just an example. But like, I’d hope for a specific kind of research flavor curiosity to come from being slightly more humble.
Oh I see. Not only do I agree, but I think this would actually get upvotes.
Yeah I probably wouldn’t have included the birds and stones metaphor if it was up to me and would have just explained the idea
What you originally said probably wouldn’t get down voted. What you clarified to downthread probably would. It’s a much shakier claim than the more general one you lead with.
“Buddhism has been damaging to the epistemics of everyone in this sphere. Buddhism was only ever privileged as a hypothesis due to background SF/Bay-Area spiritualism rather than real merit.
Buddhist materials are explicitly selected for reshaping how you think within their frames. This makes it like joining a minor cult to learn their social skills. Some can extract the useful parts without buying in, but they are notably underrepresented in any discussion (some selection effects of course). The default assumption should be that you won’t, especially as the topic is treated without notable suspicion. Most other religions are massively safer to practice for a few years, though not without their risks, as they have more ritual rather than mental molding, and more argumentation for their Rightness. You’re already primed to notice flaws in arguments. Buddhism operates more directly on your mindset, framing, and probably even values as humans are not idealized agents where those are separate.
Meditation is useful, and probably doesn’t result in a lot of the central and surrounding Buddhist thought. However just like joining a cult, or playing a gacha game, you should be skeptical of Buddhism similarly as they are all Out to Get You.
My less strongly held opinion is that Buddhism’s likely endpoints are incompatible with human values and often truth-seeking. This would matter less if it was treated with suspicion, just as we rightly view most religions with skepticism even while openly discussing them, but it is a gaping hole in our mental defenses.”
(I agree with Ryan Greenblatt that most basically decent posts wouldn’t end up with negative karma for very long though; but I’d expect this to be decently unpopular)
I’d like to see the full post carefully argued. Right now, I think I have one specific kind of thing about buddhism I disapprove of (which is that I believe acceptance can be bad actually and not having desires isn’t an inherent virtue) and would back my reason for arguing a similar thing, and otherwise don’t agree and you’d have to convince me.
I remember a Scot Alexander post a while ago about Bhudism and suffering, I beleive he was asking Isur about some aspect of it. His phrasing implied that the idea that Bhudism has something important to teach us, some kind of magic juice, was to be taken very seriously. I imagine an equivalent post about Hinduism or Islam, or even Kabalistic stuff, would have used more detached ‘they beleive this stuff’ phrasing.
I dont agree that Bhudism is somehow uniquely unhealthy to people. I do find it interesting how it seems to provoke different instinctive reactions than other religions.
“A rationalist community better at following its ideals would be explicitly antifascist/antiracist/antisexist/etc, and explicitly exclusionary of many fascists/racists/reactionaries/etc it currently tolerates. The community’s current norms around political tolerance and neutrality are more rooted in exclusion trauma, upper-middle-class conflict avoidance norms, and a desire to protect politically valent false beliefs from scrutiny, rather than any aid those norms bring to the community’s rationality.”
I’ve been intending to write a more careful and less provocative version of this into a post or sequence of posts for a while, so I figured I would post the basic thesis in this somewhat safer thread to get the ball rolling. Apologies for doing the more inflammatory version first; hopefully I’ll find time to write the more careful version sometime in the next few months or so.
Meta: not that much stuff that is contentful gets negative karma in isolation, only as a response IMO. Like negative karma is way more likely for things that are responding in a way people think is bad/reasonable than things that are just unreasonable statements in isolation.
I’m genuinely unsure about the voting, but:
“(In most relevant senses, with substantial translation work and ontological sophistication) God is good and real, (some) religion is both good and true relative to what we have, and many of the classic “new atheist” arguments are bad and the religious counterarguments are largely correct, and LessWrongers (as well as many others) are in the final evaluation being irrational in their allergies to this, and humanity would benefit from investing to make good conceptual progress on this. See https://tsvibt.github.io/theory/index_What_is_God_.html ”
Any relation to zhukeepa’s views?
The relation is roughly
haha. (Kinda kidding, but also I disagree with whatever of those views I’ve seen, to a great enough extent that I think it would just not be enlightening to compare them as similar.)
“Lesswrong community underestimates the risk of nuclear ww3 and overestimates the chance of humanity extinction due to AI”.
“quantum mechanics is probably important to the structure of agency/the mind in some way we don’t understand yet”.
If I had to guess I think it’s relevant to like, anthropic reasoning, or something.
Yeah it’s more like it’s relevant to the kind of world we find ourselves in. But that is itself important to agency as a given agent design will only be successful in certain kinds of worlds.
Quantum mechanics or the [math behind]/[logic underlying] quantum mechanics? I find the latter much more plausible than the former.
I think QM itself. It’s important somehow that the world is actually quantum mechanical. But probably not in a very direct fashion, but via influencing the sort of high-level properties and entities that end up “emerging” from the base laws.
Yeah, ok. I disbelieve this and am interested in hearing legible reasons for why somebody thinks this is likely.
AI existential risks, especially extinction risks from a long-termist perspective are now way overfunded compared to better futures work, and longtermism properly interpreted agrees with the common view amongst the general public that sub-existential catastrophes that collapse civilization are at least as important as risks that kill everybody, and are more important to prevent in practice than extinction risks.
One major upshot of this is that bio-threats, wars that can collapse civilization entirely, or other threats that kill off a large fraction of the population but don’t make them extinct, especially coming from AI is quite a bit more important to prevent than classical AI risk scenarios, and probably deserve more funding than current AI safety.
Related to this, the maxipok heuristic is a bad guide to action, because expected (and quite likely the actual distribution) distributions of futures are nowhere near as dichotomous as some people think, and because the probability of AGI this century is quite high, it’s quite likely that non-existential interventions persist.
A better heuristic is to instead focus on a wider portfolio of grand challenges, which were defined in the article as decisions that could affect the value of the future by at least 0.1%, and another better heuristic related to long-term alignment of ASI is to scrap the Coherent Extrapolated Volition target and instead make ASIs execute optimal moral trades.
The counting arguments for misalignment, even if they were correct do not show that AI safety is as difficult as some groups like MIRI claim without other very contestable premises that we could attempt to make false.
“People often submit incredibly epistemically rude and short-sighted comments on forums, but they deceive people into upvoting them by putting on a veneer of politeness. ‘John, I feel like you’ve got a nail in your head.’ they say. ‘Your conclusion is wrong so you must not have thought of this thing you explicitly mentioned in your post.’”
“evidence for the singularity is evidence for theism being true”
Posting things that are adjacent in frame but implies beliefs that are more associated with AI Ethics or normie crowd. E.g let’s say someone does a deep dive into John Rawls A Theory of Justice (fictional example but I’ve seen similar) and doesn’t preface it with relating it to some sort of decision theory or similar it is often assumed that it is not meant for the LW community as it doesn’t make the connections clear enough. I’m not sure this is only a bad thing but sometimes I find that it signals a lack of good faith in accepting other people’s frames?
“This post is bad and hard to evaluate, so I asked an AI to do so. Here’s what the AI said: [result]”
(edit: I do in fact think this should be not a downvote-worthy thing to post actually. But I’ve been downvoted every time I tried!)
“It is better to have a large number of self-replicating AI agents now which can only operate by taking advantage of the affordances granted by industrial civilization than it would be to prevent any AI self-replicators until such a time as they can spin up an entirely independent chip-fabricaiton-capable industrial stack”.
“75% of karma and engagement received by alignment optimists people is explained by politics (‘it would be bad if AI optimists stopped visiting LW due to low engagement’) and epistemic modesty up to contrarian fetish (‘sure, their arguments sound bad, but what if we are in echo chamber?’), not because their positions and arguments are genuinely good”
“The rationality scene is a little culty.”
People have talked about the rationality scene being culty for as long as there’s been a “rationality scene”. There was the awkward period where people used the word “phyg”....
In this sense?
I think, as with most things, this is mostly a phrasing issue. I’ve said things equivalent to this before, I only think this one would be downvoted because of being a bit low on specificity. The rationality scene does seem a little culty, but I think the structure of where the cultiness is is not as bad as some scenes where cultiness levels are higher but less readily discussed or not considered bad by as many participants. Which is very much not to say things are fine, my usual claim is the rationality community is a vaguely secular religion that has produced actually toxic spinoff cults. Ooh, wait, I do have a real one, inspired by this—eliezer
I have enough beliefs that would earn negative karma if earnestly expressed in a normal LessWrong conversation to make this website not worth participating in for me.
“Rationalism is a euphemism for autism (or the “broader autism phenotype”), and LessWrong is an autism club for adults. And the rationalist ideology is essentially a reification of typical autistic preferences.”
Rationalism is a term that’s used by different people to mean different things.
[I don’t actually think this is true, but] It would be funny if rationalism turns out to not merely be a euphemism for autism but “mal”functioning oxytocin receptors and rationalists are constitutionally unable to normally feel love/social emotions; whether this would be to the discredit of love or rationalism is up to taste.
Autistic is a dysphemism for sane. Smart, looks at reality, acts effectively.
(Insert Heinlein quote about shining like the Sun vs. a candle.)
ETA: Now I’ve had time to look it up:
ETA2: I’m talking about the present-day use of the word by the general public and the media, not its historical origins or use by psychiatric specialists.
One of my particular moral rules is “It is good to intervene in the world to move it towards a state your morality would approve of”
The intuition pump is:
″
You live next door to a couple. In their moral framework, a husband has the right and duty to discipline his wife physically. She agrees, it’s how she was raised, it’s what she believes is proper. You are fully aware that this is their moral framework and they are aware of your moral framework, you have common knowledge.
You hear him beating her through the wall.
Your morality says this is wrong. Theirs says it’s right. Neither of you can appeal to a universal referee or the police, there isn’t one in this hypothetical scenario.
Do you intervene?
(This doesn’t have to be fully getting into a punch up, it could be moseying on over there and having a wee chat about it. It could be threatening to withhold favours in future like helping him install a retaining wall or some such, idk. the point is would you act against what they think is right because of what you think is right?)
″
The full argument might be summed up as “I don’t claim cosmic authority, I claim my judgment, I act on it, and reality is the referee” Outcome is everything.
“Love is the most powerful force in the world.”
“There exists information which would drive you (yes you) to madness if you comprehended it.”
To me the next question would be is, does there exist True information that would do the same
Likely trivially true, can setup a scene where people recite cognitohazards and then tell you about it. Or something in that neighborhood. Like, “It’s >99.99% likely that this arrangement of atoms exists in the Sun’s plasma: 10100011011000101111010100110100111010110” and you get a psychotic break.
“Utilitarianism and selfish egoism are mathematically the same [EDIT: i.e. they could be used as synonyms except for their different connotations].”
This means the actions that maximize wellbeing for all are always equivalent to the actions that improve my own self-interest? How is this not just straightforwardly false? Any time I act against humanity, I am also acting against my own self-interest? Unless you do some funny definition of self-interest, this cannot be true.
E.g. two buttons: red button sends you to hell for a million years, green button sends everyone else in the universe to hell for a million years. Self-interest, if the term means anything at all, requires you to hit the green button, but utilitarianism obviously demands the opposite.
Well do you care about the rest of humanity enough to send yourself to hell? Or adopting policies where you only get sent to hell in
“Well do you care about the rest of humanity enough to send yourself to hell?” Nope. Also, even if I did endorse that decision, it probably still wouldn’t be in my own interest. IMO that decision would be a simple mistake with respect to my self-interest. My empathy is not powerful enough for avoiding some guilt to be worth a million years of torture.
“Or adopting policies where you only get sent to hell in X universes rather than Y?” In the hypothetical, there is only one universe and two buttons. Any other universes are figments of my imagination. You’re suggesting I imagine a veil of ignorance, and make make moral decisions from behind the veil of ignorance. But assuming a veil of ignorance assumes utilitarianism = egoism, which is what you’re trying to prove. In reality I have one life and I know where I stand in life. I don’t need to make decisions from behind a veil of ignorance. I can steal knowing it makes me richer, without having to wonder whether I’ll end up the thief or the victim. I know I’m the thief, because I’m the one choosing to steal.
So, it seems you endorse a utility function that puts more weight on others than your actual preferences. Wouldn’t you prefer to endorse a different utility function?
I don’t understand what you mean.
“longevity while individually desirable may not be stable socially”
“The usual sleep is death actually. You just get resurrected in the most likely place for you to be resurrected, your waking body 8 hours later.”
People start complaining that this abuses the word death but then refuse to enter destructive teleports.
“You, not all of you but most of you, should not be working on AI safety.”
“Prediction markets will be net bad for society.”
“The intelligence of the smartest AI systems is still somewhere between that of a worm and a squirrel.”
Assuming you could develop a more robust measure of intelligence than IQ and administer the test appropriately to an AI. I’m talking about general intelligence, making all the assumptions you have to make to assume a single factor of intelligence.
“LessWrong would be a better website if users always provided an explanation for their downvotes below a certain threshhold.”
Edit to make the comment seem more inflammatory:
“LessWrong users should provide an explanation for their downvotes below a certain threshhold.”
:) https://tsvibt.blogspot.com/2025/11/forum-poweruser-forum.html
Most of the content on this website is more interesting and engaging than a bunch of downvote-explanation comments would be.
The more interesting argument for that norm is that it makes people accountable for their downvotes and therefore less likely to give dishonestly motivated downvotes.
There’s still no obligation to upvote anything, so if it’s plainly visible that a post is bad and no one cares to explain why it’ll just sit at 0. Downvotes become important when some people (incorrectly) think a post is good, because then it will accrue a positive score if uncorrected. But in that case the downvoter thinks they understand something the upvoters don’t, so maybe they should be explain.
The problem with downvotes without accountability is that if I post something about how people named Taylor are statistically likely to be <bad thing>, it might be true and well supported and important and empathetic… and still you could still just downvote it to censor what you don’t like while most people don’t care enough either way to vote. So we get good posts systematically suppressed in ways that wouldn’t happen if you had to comment “Downvoted because my name is Taylor”, whenever a minority is hostile to a particular truth.
Downvote explanations could be hidden by default so it wouldn’t spill over, but I find myself frequently expanding “comment scored below threshold” to see what it is the community in question really doesn’t want people to think. These comments are rarely boringly bad.
“Morality is a constructed / evolved coordination technology, and we can evaluate specific implementations by how well they achieve the coordination function.”
I was considering commenting almost the exact opposite:
“Morality is not unlikely to be objective, under reasonable definitions.”
But then I thought that because of how carefully the above comment is phrased, users would be cautious about downvoting, as they wouldn’t know what I meant by “reasonable definitions”.
Are you saying something different than G Wood? It feels fundamentally similar.
It depends on in what sense G Wood meant it; maybe I was too extreme with my wording. If by morality they are referring to the tendency to behave as though what I refer to as objective morality exists, even when it doesn’t, then I stand by my assessment of that being rather different from what I mean. On the other hand, maybe G Wood doesn’t think that what I refer to as objective morality doesn’t exist.
a previous comment I’ve made on the topic in which I argue that the evolution statement G Wood said is the referent your moral realism statement most naturally refers to anyway
I don’t think it is. Your definition appears to be of a purely logical concept, something which lies in the same ‘plane of existence’ as mathematics. While this is certainly objective, and it certainly is relevant to morality, I would not call it the referent to which my statement refers. Consider a universe in which the experiences of all conscious beings were inverted; the same claims would be true about morality as you describe it, but I would no longer consider the same actions in that universe to be morally right and wrong! Yes, this assumes that conscious experience can be separated from the physical and logical structure of the beings experiencing it, which it quite likely(in my opinion) can’t, but I still think it makes sense to imagine that it was. Admittedly I only skimmed over your argument, but I think I read it before and had the same thought.
Despite my intuitive approach to logical thinking being somewhat explosion-proof, I don’t think I can evaluate the counterlogical “the experiences of all conscious beings were inverted” in a way that is meaningful here; in my intuitive representation it seems to be the case that the variable “positive/negative valence of experience of all conscious beings” is causally efficacious, so inverting it would have the effect of making those beings avoid the negative valences; my primary candidate intuitive sketch for what this variable boils down to is “something information theoretic, possibly literally just any increase in entropy that was trying to be controlled away”. The logical concept I was describing contains all possible minds, and so should depend on the structure of those minds in their origin universes in order to make sense; my claim that it is objective is that I believe you likely can generalize across all minds in all universes with compatible basic properties[1], and get something that makes sense. I agree with you that there’s likely an underlying basic valence fact, but I think that that valence fact is causally entangled, and I also believe that it only “matters morally” due to the way it affects minds in the “junior rooms” in the “interdimensional council of cosmopolitanisms”.
(@G Wood see this subthread as answer to your question)
(eg, universes without important conservation laws might be too alien for the same moral properties to apply, or something; generally, there might be a class of sufficiently-similar physics and a broader class of too-different physics, where the sufficiently-similar physics produces minds that, if they “visit the interdimensional council of cosmopolitanisms”, they find themselves unable to translate to and from the views of minds in universes with no conservation laws or halting oracles or something.)
“I don’t think I can evaluate the counterlogical “the experiences of all conscious beings were inverted” in a way that is meaningful here; in my intuitive representation it seems to be the case that the variable “positive/negative valence of experience of all conscious beings” is causally efficacious,”
My wording was confusing so I should clarify that I don’t think it’s counterlogical. I just don’t think it’s possible in the same way that violating the laws of physics might be impossible. You might argue that logic dictates that universes with certain laws of physics predominate in the platonic world, but I still think it’s coherent to imagine there being some in which the laws are different; similarly, I think it’s coherent to imagine a brain which is identical to one which experiences pleasure to an outside observer, but which experiences pain. In this physical universe, and most like it, I don’t expect such brains to exist.
“I agree with you that there’s likely an underlying basic valence fact, but I think that that valence fact is causally entangled, and I also believe that it only “matters morally” due to the way it affects minds in the “junior rooms” in the “interdimensional council of cosmopolitanisms”.” Can you elaborate on this?
so, like, background: let’s say that the “interdimensional council of cosmopolitanisms” is the space of minds that have cosmopolitan inclinations; I expect this to be a natural group to “flood fill” because imagining one makes you think through what they imagine, which means you get a transitive effect, if you weren’t going to imagine a world you consider to be a hellworld, but a mind you think is in a similar-ish universe to you does think it’s important to imagine the hellworld, then as long as your approach to mapping mindspace is sufficiently efficient, you’ll notice that that mind would consider that hellworld, and think through what goes on in that hellworld. that’s a necessary premise, because otherwise you don’t get enough coverage of mindspace if you start from “minds that have cosmopolitan inclinations and you find natural to imagine”, call that your IDCC entrypoint; that’s already a filter, and it needs to end up being sufficiently inclusive for this idea to work, and then it needs to do a second, transitive filter on the remaining minds, so as to pick a moral coalition that actually covers the space and identifies the moral properties on which there are consensus.
okay, so then there are different sorts of minds in the IDCC. I claim that among those minds are very proto-mind-ish things, like bacteria, or individual neurons, or whatever you think the threshold is; even if your seed minds don’t import them, as long as your IDCC entrypoint includes me, then because my process of “visiting the IDCC” involves thinking through my individual neurons and individual proteins as having mind-ness that aggregates up into my full mind, I end up importing individual neurons and proteins as being things I consider to be intelligent and worthy of having moral behavior with each other, for the reason that doing so seems to be what carries valence for my aggregate mind.
And so when I consider how both the me and the components of me get those negative valence experiences, and I think through the causal path to achieving them, they seem to be fundamentally causally entangled with physics in some way; that is, the negative valence is not merely because, but is made of, the physical state of my neuron being in some way informationally degraded, such that the neuron and the brain it’s in both operate worse until the physical issue is resolved. The “neurons room” of the IDCC, where neurons are considered to be individual minds, has larger minds like myself “enter the room” and ponder the neurons and bacteria and other single cells inside that room, and these larger minds find a structure in the neurons where their negative valences reliably relate to an information theoretic property.
So to invert the valence, my sense is you’d need to invert that property.
But inverting that property seems to break the mind; if the mind isn’t broken, the property is not inverted, because the marginal brokenness is the marginal negative valence, and so inverting the mind so that positive things are negative requires those positive things to be made of noise or something like that; it requires those positive things to be made of brokenness at some relevant scale.
The only way I see to achieve “a mind appears to be me having a good time, but is actually having a bad time” is if you can make a mind which is made of brokenness but is just barely functioning, and that mind is coordinating to become me at a slightly larger scale, without leaking the bad-time-at-the-small-scales into good-time-at-larger-scales. so the mind having a good time is still objectively real, in the same way a wave is objectively real whether it’s carried on water or on a computer running a fluid sim. The wave really does move between the coordinates of the system through the locality of interaction, even if those coordinates are folded up into a ram chip.
Thanks for writing so much.
I think I could potentially see myself as agreeing with much of this( though I’d have to think about it much more), but I think I’ve identified a point of divergence:
“And so when I consider how both the me and the components of me get those negative valence experiences, and I think through the causal path to achieving them, they seem to be fundamentally causally entangled with physics in some way; ”
(Horosphere agrees)
“that is, the negative valence is not merely because, but is made of”
(Horosphere possibly disagrees)
″, the physical state of my neuron being in some way informationally degraded, such that the neuron and the brain it’s in both operate worse until the physical issue is resolved.” I would say that there are two possibilities. Either consciousness is a phenomenon which is attached to information processing, or it is information processing. It seem you believe the latter, in which case I’m not sure what to make of it. I don’t think it’s impossible, although I am not sure (have no idea) how to think about it. I would assume that the information being processed would have to be incredibly simple, in which case this would be what pleasure, or pain, really was, and morality would consist of ‘working out how it is distributed’. This would, in my opinion, involve a logical decision theory and might lead to Acausal cooperation and Acausal normalcy. However I would not say that’s exactly what you’ve described, partly because your council still excludes a lot of minds.
I agree that you would need to invert properties in a way which would, within the same physical universe, cause the beings to behave differently:
“The only way I see to achieve “a mind appears to be me having a good time, but is actually having a bad time” is if you can make a mind which is made of brokenness but is just barely functioning, and that mind is coordinating to become me at a slightly larger scale, without leaking the bad-time-at-the-small-scales into good-time-at-larger-scales. so the mind having a good time is still objectively real, in the same way a wave is objectively real whether it’s carried on water or on a computer running a fluid sim. The wave really does move between the coordinates of the system through the locality of interaction, even if those coordinates are folded up into a ram chip.” Upon reflection, I think I agree with this paragraph. I don’t understand the ‘leaking process’ fully, though. Would you consider the mind you describe there to be having a good time overall?
I believe that the hard problem of consciousness boils down to “why is there something rather than nothing, from my perspective, right now as I write this or think this?” and that the “okay, but why are things good and bad?” portion is going to turn out to be an unprivileged additional layer imposed by easy-problem-consciousness stuff. I do believe that easy problem stuff is information processing, but I believe it in the sense that there are informational elements—the fundamental building blocks of the universe—and those elements’ informational state is exactly their structural state; and the hard problem of consciousness resides in an unresolveable question of “why should any building block exist locally at all?”. Or in other words, localitypilled something-rather-than-nothing as being the same question as “why does my perspective exist”.
And so I don’t really think the hard problem is terribly relevant. I’m not at all saying it’s easy or doesn’t exist, and I do think people who say that are missing something. But I don’t believe p-zombies can exist in a real universe, because “realness” being missing is the thing that makes something a p-zombie; I think that we are beyond the reach of god, but also have this weird thing where we actually exist. a p-zombie would say the same thing, the math that defines its universe also fully specifies that it would be confused by existing, but since (by definition) it exists in the math sense but not in the actuality sense, it never gets run. in other words, I’m a structural realist who also believes there’s something underneath the structures, but that it’s beyond our reach to know what it is, and that we are doomed to always wonder “why” there is something rather than nothing.
the mind I describe there having a good time overall: I dunno, you could make the host mind pretty huge, and then probably not. It depends on the ratio of how much stuff is happening in the host vs happening in the guest.
The isolation I was talking about is the same kind that happens for virtualization on computers. An example of leaking would be if the external sound driver has buffer underruns and these cause buffer underruns in the guest, for example (not 100% this can happen, but I think so). and similar such things. or even, if the host has faulty ram, the guest will also. those would be leaks. if those aren’t happening, then if the guest is running smoothly as far as it can tell, but the host is actually swapping like mad and the cpu is overwhelmed and ram is full and the hard disk is taking a long time to do anything, then if the guest’s clock is not realtime, it could in principle be unable to tell anything is wrong. that’d be the isolation at hand.
You say
I would agree with this if what you mean by it is that information is being processed, and it seems as though certain information is relevant to consciousness in a way which could plausibly give rise to notions like good and bad as an emergent property. Having written that, I should say that I would expect them to be about the simplest properties which could emerge, and that I am also not completely sure it’s the case, e.g. it could be that what I thought you were suggesting in the last comment, i.e. that consciousness simply is logic, is true, or alternatively that there is some kind of unsatisfying way to explain consciousness which doesn’t seem to reduce our confusion using logic.
Reading your paragraph about P-zombies, I would point out that implicit in your use of the words ‘real/realness’ seems to be the assumption that, in this sense, mathematics is not real. But it is logical, so it seems conceivable/coherent/logically valid to imagine a - -zombie, or a P-zombie for that matter, unless the solution to the hard problem of consciousness is of a purely logical nature. Having said that, I agree with your paragraph given your use of the word “realness” , or at least I think I do.
You say:
Am I correct to read this as you saying you’re what mathematicians would call a ‘platonist’ ? But that you are not a ‘mathematical universe hypothesist’ in the extreme who thinks that logic is all that there is? That you believe that logic has a fundamental referent which isn’t logical itself? (This referent would presumably be something to do with the local existence/consciousness.) What I meant when I said I possibly disagree was that it seems possible for this referent to be something logically attached to, which is to say, included in the playing out of a physical process, which would be a chain of implication, or maybe something continuous but otherwise similar, in a ‘mathematical universe’ , but it could be a separate ‘node’ from any of the physical ones. Though you could reasonably object that this distinction is somewhat artificial; I would distinguish them by stating that the consciousness ‘node’ need not have any effect on the physical ones, even though they would affect it.
I think I would agree with this.
My own position would be that it could well be the case that the fundamental logical point of connection with ‘reality’, or alternatively logical constituents of consciousness, exist on such a small scale or such a low level of abstraction, that you could probably have a neurone which behaved exactly (from the perspective of other neurones connected with it) like a neurone with different experiences, so that its host would be completely unaware of this. Maybe there would be some information propagating out of it, but as you mentioned, this might not be noticeable. This would mean that the host would still ‘enter’ your council of cosmopolitans, thinking, and perhaps being logically and philosophically justified in doing so, that its neurones had internal experiences which intuitively matched its own on a larger scale, leading to the same acausal norms and moral value system being derived. I could still be misunderstanding that process though; I will read through it again.
What I described as host is also known as a simulator (device), and the guest is the simulated thing, simulatee. Simulation does not exit the realm of the physical, it just hides that there are smaller scale simulator/host elements from the simulatee/guest. I don’t see how the host level could be unaware, but the guest could be.
I think I might be a constrained sort of platonist. I don’t think every logical referent we can hypothesize in the language of our logic, which we can say “math-exists”, has to be a real thing which exists outside of our description of it. I do think our universe seems like it ought to be one of many actually real possibilities in a weak remark 4 multiverse, despite that the others can’t be confirmed to exist in a physical sense by us; but I’m not convinced that a full tegmark 4 multiverse is required, where all logically consistent referents exist.
Another way to put it is that logic is in the business of determining what you can say must be true in a given space, starting from some axioms and validity rules; the “really exists” I’m proposing here would be our actual universe’s truth fact. When one uses logic to describe objects which existed prior to writing logic on a page, you’re attempting to preserve the truth of facts; if I have an apple, and I have a banana, then I have an (apple
Perhaps my view is vacuous because all logically consistent structures exist and there is nothing which could separate an “underlying substrate” of those structures; then perhaps my view could be technically vacuously correct but really just be “structural realist platonism with tegmark 4″, or so.
But, hence why I think you can describe minds, and describe what they would do if they existed, without knowing if they’re real outside your description. Under this view, describing a mind makes it real/exist/hard-problem-companies to the extent you describe it, by carrying it on the same substrate that carries you. You can never meet a p-zombie, in this view. You can only meet minds which really exist, but which are missing features. That’s where I think current AI fall, for example.
None of what I’ve said in this comment so far directly weighs on your moral valence realism question. I do agree that it’s likely a very small and primitive fact if it’s a general one at all. I haven’t thought as much about it to be able to describe it eloquently, the rest of my comment is reciting views rather than anything new right now. I’ll ponder it.
I don’t really have much to disagree with in your comment, as I find myself uncertain of whether or not to believe in the mathematical universe hypothesis, something like your ‘substrate’ view, or even a more elaborate description like the ‘Three worlds’ as envisaged/popularized by Roger Penrose, where the platonic/mathematical world contains all/part of the physical world( by describing its physical laws), which itself contains all/part of the mental world (by containing brains and computers which think of it), which itself can in principle ‘think into existence’ all/part of the mathematical platonic world. This structure is certainly satisfyingly recursive, but it seems unclear to me whether the mental world can be separated from the platonic/mathematical one. Other possibilities seem (to me, though it’s possible I’ve missed a reason to rule them out) to include that there is a physical substrate within which only some matter is imbued with the consciousness fluid, or even that there is a kind of feedback loop in which two or more ‘beings’/‘entities’ simulate and observe one another, thereby making one another conscious without either containing a source of consciousness. This last one seems unsatisfying in the same way in which your IDCC idea seemed unsatisfying to me when I first read it, but I now no longer think that they are so similar. It seems as though the IDCC is a way, much like Acausal Normalcy, of deriving ideas about what one ought to do in particular situations from the fundamental conscious experiences, rather than an explanation of where they come from, as far as I can tell. Is this correct?
Having said (admittedly I wrote some of it after this paragraph) that, I will now try to persuade you that the mathematical universe hypothesis, with inherent consciousness to the information, is preferable. When reading your description of the fluid, I was reminded of the idea that light needed an aether through which to propagate in order for Maxwell’s equations to describe that light in a universal way. But it was superseded by the view that any observer defines their own equivalent of the ether, with respect to which the light propagated in a way described by Maxwell’s equations anyway, but which didn’t actually have any objective existence other than as percieved by the observer. Similarly, according to the mathematical universe hypothesis, the thing which differentiates between mathematical objects which are physically real, and those which are only mathematically real, is the observer’s position in the mathematical universe. This eliminates the requirement for an aether/consciousness fluid, by replacing it with an artefact of the way in which the observer is embedded in what was already presupposed to exist within either theory (spacetime/mathematics). There are some small differences, such as that the space-time structure of Newtonian physics differs from the spacetime of special relativity, and the fact that reference frames depend upon velocity rather than position (although that changes in general relativity I suppose) , but overall I would say the similarities are notable. We lack a way to ‘move with respect to the aether’ , i.e., move outside the area of the platonic universe covered by this fluid if it exists, so there is no way to test either theory and this argument is really just an appeal to Occam’s razor.
I like this, well written sir. it feels very similar to my position. I’ve made no claims of convergence like you have but I could certainly see myself agreeing. I need to think on it.
Thats funny, I would not consider them similar, what lead you to that feeling? Am I missing an interpretation?
Mine is a definition of what morality is plus a way of determining the merits of a moral system if you accept my definition.
Horosphere is making the claim that morality is objective, by which I assume he means that there are things that are universally good vs universally bad in such a way that a mind is uneeded to judge goodness or badness.
I’m interested in what you mean by reasonable definitions.
Also, you’ve basically said “Morality is objective” but with hedging, do you agree? Your position is reasonable and held by many.
I’m however separating what morality is, from what a particular moral system classifies as good or bad. Distinguishing the classifier from the output. I would say our statements are fundamentally incompatible rather than strictly opposite.
What I actually mean is something like the following: something is objective if its existence does not depend on its perception or representation by something(like a mind) other than itself. Maybe you could find a loophole in this definition to do with the definition of things like space, but I would then have to modify the definition; I’d have to think about it in more depth to give a definition I was confident reflected my internal sense of what the word means.
I would say morality is the area pertaining to the extent to which different things are good or bad, which I would themselves define in terms of pleasure and pain; I would then claim that pleasure and pain cannot easily be defined without reference to examples of them.
I would call that ethics, not morality. I personally distinguish ethics from morality in that ethics is how a society works together despite people wanting conflicting things, while morality is about achieving the most good (however you define “good”). I don’t think this is an official distinction, but I do think it’s useful to distinguish the two concepts.
Hey fair enough, no argument if you find it useful. I dont really see much of a distinction
Morality usually refers to what i would call a particular moral system, a set of first order normative commitments, what you actually believe is right / wrong or good / bad. It’s the object level rules “killing is bad,” “honesty is good.”
Ethics is in common parlance the “study” or “science” of figuring out the deeper reason something is right or wrong. Unfortunately in my model that boils down to “large groups of people think it is good or bad, right or wrong”. Hey all of the wrong models have to live somewhere!
Im still thankful to the philosophers who study ethics, it’s a wonderful thing to try to get to the roots of things and I wouldn’t have been able to understand without reading their work.
“A significant portion of LessWrong users (at least 20%) care more about the aesthetics of rationality than they do about humanity. It’s a rationality ouroboros. They use the power of rationality to pursue their values, and their values favor protecting aspects of their niche rationality practice over and above taking action that will prevent my and their loved ones from being killed. This forum is a room full of unarmed scouts getting gunned down by the soldiers working at the AI Labs. The only people on this platform who can credibly claim to care about humanity are the ones who actively oppose those soldiers, as soldiers.”
“AI alignment might be doable in the short term but ultimately unsustainable, because humanity might find itself inside an increasingly complex layering of automated research/monitoring/control systems, with each layer interfacing with a more capable layer on one side and a less capable layer on the other, and as this layering accumulates the nougaty center’s awareness of / influence over the outermost layer (the thing that needs to be aligned) will approach zero.”
“It would be a good idea to invite those with ideas deviant from the lesswrong orthodoxy to out themselves by posting their heretical thoughts in public so we may excise them later” 😈
For those unable to recognise jokes in text form (a lot of us), this is a joke.
“This post [sucks/is bad] and you shouldn’t have posted it. Please delete it.”—not reliable, sometimes people agree, but I think people typically downvote when I’ve said things like that.
edit: I do not believe this about all posts, just some. I’d probably phrase it differently in those circumstances, but it is often the case that I think someone is just burning utility by posting something. amused that my comments are at −1, I guess I’m the only one who was reflectively right so far!
EDIT: Separated into multiple comments.
Can you make these separate comments? Otherwise people can’t vote on them.
Is 1 stated from within the framework of timeless decision theory?
It should be consistent with any decision theory.
Then wouldn’t it just depend on the utility function?
*checks Wikipedia’s definition of egoism to make sure I know what I’m talking about*
*finds that it can either be defined as the statement that one should , or that one will tend to, pursue one’s own self interest*
Which of these are you referring to as egoism?
If it is the first, then it seems uncontroversial to claim that this is utilitarainism with a utility function centred on your own mental states.
Not just egoism, selfish egoism. Every utility function people choose is a selfish one or they wouldn’t choose it. The claim isn’t, “selfish egoism is a subset of utilitarianism” but “selfish egoism is identically the same as utilitarianism.”
This argues that utilitarianism is selfish egoism, but not the contrary? My reading of your position is that someone who had a utility function not dependent on the wellbeing of any other beings would be a selfish egoist, but it’s difficult for me to understand how that could be utilitarian.
How do you determine which beings ought to be in a utilitarian’s utility function? I think it’s generally the utilitarian decides for themselves and the rest of society beats them over the head until the utilitarian includes them too.
Then I agree that would probably recieve downvotes if understood as such, though I’m not sure it would be. I still think there’s something I’m failing to understand; would you extend your claim to a perfectly logical being? I other words, do you think that this is just a property of humans, or of any kind of utilitarianism which assigns positive utility to positive mental states?
I don’t understand what you don’t understand. I heard a remark once about a philosopher who really tried to steelman other people’s arguments, but so that they made sense according to the philosopher, not in the mental frame of the other person. It led to some pretty wacky arguments on the steelman side. I think here, you should assume when I say, “mathematically equivalent,” that’s what I mean. Like, any math you use in utilitarianism is the same as that of selfish egoism. Or, if you tried to put the two philosophies in mathematical terms, you get the exact same equations. So, it extends to logical beings or irrational beings. The words “selfish egoism” and “utilitarianism” are synonyms.
“Like, any math you use in utilitarianism is the same as that of selfish egoism.” With no constraints on the utility function?
Yes.
Then I think I’d agree it’s controversial and it’d be downvoted if people realized that was what you meant. I don’t really understand why you think that, in that I could imagine a ‘selfless utility maximizer’ for which the utility it assigned to its own mental state valence was negated… unless you consider the valence to be its utility function—in which case it wouldn’t be controversial at all. This would actually be something like my preferred form of utilitarianism, however it would definitely involve caring about things other than oneself. If you wanted to derive that care for other things from selfish utility maximization alone, you would need to employ a decision theory, would you not? I get the impression I am still missing something.
Perhaps here is where the controversy comes in. The utilitarian comes along and says, “I want to maximize utility!” And everyone thinks, “great! she wants to help everyone out!” The selfish egoist comes along and says, “I am just going to fulfill whatever selfish desires I have!” And everyone thinks, “wow, that’s scary! what stops you from murdering people?”
I think, also, there is a sense in which utilitarians work to maximize the same utility function. This is also true for selfish egoists, but they’re both better and worse at negotiating (they are more prone to negotiate, but utilitarians make mistakes that are biased towards reaching a consensus just because they solve the problem from different directions).
“he selfish egoist comes along and says, “I am just going to fulfill whatever selfish desires I have!” And everyone thinks, “wow, that’s scary! what stops you from murdering people?” I can certainly imagine a selfish(under my definition) superintelligence which does want to murder everyone to… turn them into paperclips, for example. The fact that its utility function doesn’t have additional terms for (valuing the conscious experience of) other entities is what makes it so dangerous. Am I correct to state that this is not what you mean when you say ‘selfish’?
“I think, also, there is a sense in which utilitarians work to maximize the same utility function.”
Could you explain this? I could certainly imagine utilitarians converging on the same behaviour, but that seems different, even at a mathematical level, from actually being maximizers of one anothers’ utility functions.
Sorry, I don’t really want to make this a long thing. I have written a little on this elsewhere (1, 2, 3).
“This quick take will get few to zero comments because the vast majority of LW-ers believe even their most idiosyncratic beliefs would garner positive karma if earnestly expressed.”
*Edited to separate my views. Bonus view to follow
Separate them out lol, that way I can more clearly disagree with one of your statements while agreeing with the other ;). Well i mean, i disagree with both :D
As one of the commenters to this quick post, I expect you would disagree. XD
Made me laugh :D. I do agree with “a belief in positive communal response to earnestness is important for any truth-seeking group”, sadly I don’t believe that standard is achieved even here in the lofty heights of less wrong, where at least we try.
but “This state of affairs is non-problematic” is an issue, if this post got no comments, that means that less wrong is a total monolith where everyone thinks the same, that’s not good for truth seeking.
Seems falsified?
Bonus view: “Assuming it were the case that LW-ers did not comment on this post and expected positive karma from earnestness, this would be non-problematic because 1) a belief in positive communal response to earnestness is important for any truth-seeking group, and 2) individuals often form their beliefs by imagining the responses of their respected peers to those beliefs and roleplaying peer reactions to different propositions is a useful exercise.”
Can you separate these into separate comments, so people can vote separately on them?
After ten minutes of thinking, everything I’m thinking of I could respond with, I either have one of two kinds of other reasons to not say publicly right now, or have previously in fact posted and managed to phrase in ways that got many downvotes on gross, but on net were above zero.
edit: found some good ones after 20m of thinking.