sunwillrise

Karma: 1,083

sunwillrise 7 Jan 2025 7:48 UTC
7 points
3
on: Guilt, Shame, and Depravity
The post seems to be talking about these topics solely from the perspective of the relation between the individual and the community they belong to (“You are a member of some community”, emphasis added). But, in my opinion, one of the most important manifestations of guilt (and shame, etc) to analyze, if we aim to understand these emotions more deeply, is when the individual feels guilty for violating a norm inside a community they are no longer a part of. This comes about because the vast majority of people have a tendency to internalize the norms and lessons they are taught and to try to uphold them regardless of the external incentives they face from the group.
As Eliezer once wrote, when you punish a child for stealing cookies, the child learns not to be caught stealing, but usually and to some extent, they also learn not to steal, as an end goal/principle independent of whether they get caught.

sunwillrise 27 Dec 2024 9:21 UTC
16 points
0
in reply to: Noosphere89’s comment on: The Field of AI Alignment: A Postmortem, and What To Do About It
I’m wary of a possible equivocation about what the “natural abstraction hypothesis” means here.
If we are referring to the redundant information hypothesis and various kinds of selection theorems, this is a mathematical framework that could end up being correct, is not at all ungrounded, and Wentworth sure seems like the man for the job.
But then you are still left with the task of grounding this framework in physical reality to allow you to make correct empirical predictions about and real-world interventions on what you will see from more advanced models. Our physical world abstracting well seems plausible (not necessarily >50% likely), and these abstractions being “natural” (e.g., in a category-theoretic sense) seems likely conditional on the first clause of this sentence being true, but I give an extremely low probability to the idea that these abstractions will be used by any given general intelligence or (more to the point) advanced AI model to a large and wide enough extent that retargeting the search is even close to possible.
And indeed, it is the latter question that represents the make-or-break moment for natural abstractions’ theory of change, for it is only when the model in front of you (as opposed to some other idealized model) uses these specific abstractions that you can look through the AI’s internal concepts and find your desired alignment target.
Rohin Shah has already explained the basic reasons why I believe the mesa-optimizer-type search probably won’t exist/be findable in the inner workings of the models we encounter: “Search is computationally inefficient relative to heuristics, and we’ll be selecting really hard on computational efficiency.” And indeed, when I look at the only general intelligences I have ever encountered in my entire existence thus far, namely humans, I see mostly just a kludge of impulses and heuristics that depend very strongly (almost entirely) on our specific architectural make-up and the contextual feedback we encounter in our path through life. Change either of those and the end result shifts massively.
And even moving beyond that, is the concept of the number “three” a natural abstraction? Then I see entire collections and societies of (generally intelligent) human beings today who don’t adopt it. Are the notions of “pressure” and “temperature” and “entropy” natural abstractions? I look at all human beings in 1600 and note that not a single one of them had ever correctly conceptualized a formal version of any of those; and indeed, even making a conservative estimate of the human species (with an essentially unchanged modern cognitive architecture) having existed for 200k years, this means that for 99.8% of our species’ history, we had no understanding whatsoever of concepts as “universal” and “natural” as that. If you look at subatomic particles like electrons or stuff in quantum mechanics, the percentage manages to get even higher. And that’s only conditioning on abstractions about the outside world that we have eventually managed to figure out; what about the other unknown unknowns?
For example, this post does an experiment that shows that OOD data still makes the Platonic Representation Hypothesis true, meaning that it’s likely that deeper factors are at play than just shallow similarity
I don’t think it shows that at all, since I have not been able to find any analysis of the methodology, data generation, discussion of results, etc. With no disrespect to the author (who surely wasn’t intending for his post to be taken as authoritative as a full paper in terms of updating towards his claim), this is shoddy science, or rather not science at all, just a context-free correlation matrix.
Anyway, all this is probably more fit for a longer discussion at some point.

sunwillrise 26 Dec 2024 20:30 UTC
69 points
13
on: The Field of AI Alignment: A Postmortem, and What To Do About It
(Prefatory disclaimer that, admittedly as an outsider to this field, I absolutely disagree with the labeling of prosaic AI work as useless streetlighting, for reasons building upon what many commenters wrote in response to the very posts you linked here as assumed background material. But in the spirit of your post, I shall ignore that moving forward.)
The “What to Do About It” section dances around but doesn’t explicitly name one of the core challenges of theoretical agent-foundations work that aims to solve the “hard bits” of the alignment challenge, namely the seeming lack of reliable feedback loops that give you some indication that you are pushing towards something practically useful in the end instead of just a bunch of cool math that nonetheless resides alone in its separate magisterium. As Conor Leahy concisely put it:
Humans are really, really bad at doing long chains of abstract reasoning without regular contact with reality, so in practice imo good philosophy has to have feedback loops with reality, otherwise you will get confused.
He was talking about philosophy in particular at that juncture, in response to Wei Dai’s concerns over metaphilosophical competence, but this point seems to me to generalize to a whole bunch of other areas as well. Indeed, I have talked about this before.
… and in my experience, there are people who can get traction on the core hard problems. Most notably physicists—when they grok the hard parts, they tend to immediately see footholds, rather than a blank impassable wall.
Do they get traction on “core hard problems” because of how Inherently Awesome they are as researchers, or do they do so because the types of physics problems we mostly care about currently are such that, while the generation of (worthwhile) grand mathematical theories is hard, verifying them is (comparatively) easier because we can run a bunch of experiments (or observe astronomical data etc., in the super-macro scale) to see if the answers they spit out comply with reality? I am aware of your general perspective on this matter, but I just… still completely disagree, for reasons other people have pointed out (see also Vanessa Kosoy’s comment here). Is this also supposed to be an implicitly assumed bit of background material?
And when we don’t have those verifying experiments at hand, do we not get stuff like string theory, where the math is beautiful and exquisite (in the domains it has been extended do) but debate by “physics postdocs” over whether it’s worthwhile to keep funding and pursuing it keeps raging on as a Theory of Everything keeps eliding our grasp? I’m sure people with more object-level expertise on this can correct my potential misconceptions if need be.
Idk man, some days I’m half-tempted to believe that all non-prosaic alignment work is a bunch of “streetlighting.” Yeah, it doesn’t result in the kind of flashy papers full of concrete examples about current models that typically get associated with the term-in-scare-quotes. But it sure seems to cover itself in a veneer of respectability by giving a (to me) entirely unjustified appearance of rigor and mathematical precision and robustness to claims about what will happen in the real world based on nothing more than a bunch of vibing about toy models that assume away the burdensome real-world details serving as evidence whether the approaches are even on the right track. A bunch of models that seem both woefully underpowered for the Wicked Problems they must solve and also destined to underfit their target, for they (currently) all exist and supposedly apply independently of the particular architecture, algorithms, training data, scaffolding etc., that will result in the first patch of really powerful AIs. The contents and success stories of Vanessa Kosoy’s desiderata, or of your own search for natural abstractions, or of Alex Altair’s essence of agent foundations, or of Orthogonal’s QACI, etc., seem entirely insensitive to the fact that we are currently dealing with multimodal LLMs combined with RL instead of some other paradigm, which in my mind almost surely disqualifies them as useful-in-the-real-world when the endgame hits.
There’s a famous Eliezer quote about how for every correct answer to a precisely-stated problem, there are a million times more wrong answers one could have given instead. I would build on that to say that for every powerfully predictive, but lossy and reductive mathematical model of a complex real-world system, there are a million times more similar-looking mathematical models that fail to capture the essence of the problem and ultimately don’t generalize well at all. And it’s only by grounding yourself to reality and hugging the query tight by engaging with real-world empirics that you can figure out if the approach you’ve chosen is in the former category as opposed to the latter.
(I’m briefly noting that I don’t fully endorse everything I said in the previous 2 paragraphs, and I realize that my framing is at least a bit confrontational and unfair. Separately, I acknowledge the existence of arguably-non-prosaic and mostly theoretical alignment approaches like davidad’s Open Agency Architecture, CHAI’s CIRL and utility uncertainty, Steve Byrnes’s work on brain-like AGI safety, etc., that don’t necessarily appear to fit this mold. I have varying opinions on the usefulness and viability of such approaches.)

sunwillrise 26 Dec 2024 19:20 UTC
8 points
0
in reply to: Noosphere89’s comment on: A shot at the diamond-alignment problem
I want to echo Jonas’s statement and say that this was an enjoyable and thought-provoking comment to read. I appreciate the deep engagement with the questions I posed and the work that went into everything you wrote. Strong-upvoted!
I will not write a point-by-point response right now, but perhaps I will sometime soon, depending on when I get some free time. We could maybe do a dialogue about this at some point too, if you’re willing, but I’m not sure when I would be up for that just yet.

sunwillrise 7 Dec 2024 15:11 UTC
3 points
2
in reply to: Raemon’s comment on: The 2023 LessWrong Review: The Basic Ask
Ah, gotcha. Yes, that seems reasonable.

sunwillrise 7 Dec 2024 2:42 UTC
1 point
2
in reply to: Raemon’s comment on: The 2023 LessWrong Review: The Basic Ask
I don’t really understand why this is at all important. Do you expect (or endorse) users to… vote on posts solely by reading a list of titles without clicking on them to refresh their memories of what the posts are about, and as a natural corollary of this, see who the authors of the posts are? What’s the purpose of introducing inconveniences and hiding information when this information will very likely be found anyway?
I get the importance of marginalist thinking, and of pondering what incentives you are creating for the median and/or marginal voting participant, blah blah blah, but if there is ever a spot on the Internet where superficiality is at its lowest and the focus is on the essence above the form, the LW review process might well be it.
In light of that, this question just doesn’t seem (to a rather outside observer like me) worth pondering all that much.

sunwillrise 5 Dec 2024 0:13 UTC
3 points
0
in reply to: dirk’s comment on: dirk’s Shortform
The 3 most important paragraphs, extracted to save readers the trouble of clicking on a link:
The Anduril and OpenAI strategic partnership will focus on improving the nation’s counter-unmanned aircraft systems (CUAS) and their ability to detect, assess and respond to potentially lethal aerial threats in real-time.
[...]
The accelerating race between the United States and China to lead the world in advancing AI makes this a pivotal moment. If the United States cedes ground, we risk losing the technological edge that has underpinned our national security for decades.
[...]
These models, which will be trained on Anduril’s industry-leading library of data on CUAS threats and operations, will help protect U.S. and allied military personnel and ensure mission success.

sunwillrise 4 Dec 2024 17:20 UTC
4 points
1
in reply to: Steven Byrnes’s comment on: Is the mind a program?
I appreciate your response, and I understand that you are not arguing in favor of this perspective. Nevertheless, since you have posited it, I have decided to respond to it myself and expand upon why I ultimately disagree with it (or at the very least, why I remain uncomfortable with it because it doesn’t seem to resolve my confusions).
I think revealed preferences show I am a huge fan of explanations of confusing questions that ultimately claim the concepts we are reifying are ultimately inconsistent/incoherent, and that instead of hitting our heads against the wall over and over, we should take a step back and ponder the topic at a more fundamental level first. So I am certainly open to the idea that “do I nonetheless continue living (in the sense of, say, anticipating the same kind of experiences)?” is a confused question.
But, as I see it, there are a ton of problems with applying this general approach in this particular case. First of all, if anticipated experiences are an ultimately incoherent concept that we cannot analyze without first (unjustifiably) reifying a theory-ladden framework, how precisely are we to proceed from an epistemological perspective? When the foundation of ‘truth’ (or at least, what I conceive of it to be) is based around comparing and contrasting what we expect to see with what we actually observe experimentally, doesn’t the entire edifice collapse once the essential constituent piece of ‘experiences’ breaks down? Recall the classic (and eternally underappreciated) paragraph from Eliezer:
I pause. “Well . . .” I say slowly. “Frankly, I’m not entirely sure myself where this ‘reality’ business comes from. I can’t create my own reality in the lab, so I must not understand it yet. But occasionally I believe strongly that something is going to happen, and then something else happens instead. I need a name for whatever-it-is that determines my experimental results, so I call it ‘reality’. This ‘reality’ is somehow separate from even my very best hypotheses. Even when I have a simple hypothesis, strongly supported by all the evidence I know, sometimes I’m still surprised. So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies ‘belief,’ and the latter thingy ‘reality.’ ”
What exactly do we do once we give up on precisely pinpointing the phrases “I believe”, “my [...] hypotheses”, “surprised”, “my predictions”, etc.? Nihilism, attractive as it may be to some from a philosophical or ‘contrarian coolness’ perspective, is not decision-theoretically useful when you have problems to deal with and tasks to accomplish. Note that while Eliezer himself is not what he considers a logical positivist, I think I… might be?
I really don’t understand what “best explanation”, “true”, or “exist” mean, as stand-alone words divorced from predictions about observations we might ultimately make about them.
This isn’t just a semantic point, I think. If there are no observations we can make that ultimately reflect whether something exists in this (seems to me to be) free-floating sense, I don’t understand what it can mean to have evidence for or against such a proposition. So I don’t understand how I am even supposed to ever justifiably change my mind on this topic, even if I were to accept it as something worth discussing on the object-level.
Everything I believe, my whole theory of epistemology and everything else logically downstream of it (aka, virtually everything I believe), relies on the thesis (axiom, if you will) that there is a ‘me’ out there doing some sort of ‘prediction + observation + updating’ in response to stimuli from the outside world. I get that this might be like reifying ghosts in a Wentworthian sense when you drill down on it, but I still have desires about the world, dammit, even if they don’t make coherent sense as concepts! And I want them to be fulfilled regardless.
And, moreover, one of those preferences is maintaining a coherent flow of existence, avoiding changes that would be tantamount to death (even if they are not as literal as ‘someone blows my brains out’). As a human being, I have preferences over what I experience too, not just over what state the random excitations of quantum fields in the Universe are at some point past my expiration date. As far as I see, the hard problem of consciousness (i.e., the nature of qualia) has not been close to solved; any answer to it would have to give me a practical handbook for answering the initial questions I posed to jbash.

sunwillrise 1 Dec 2024 3:31 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Is the mind a program?
Edit: This comment misinterpreted the intended meaning of the post.
Practical CF, more explicitly: A simulation of a human brain on a classical computer, capturing the dynamics of the brain on some coarse-grained level of abstraction, that can run on a computer small and light enough to fit on the surface of Earth, with the simulation running at the same speed as base reality, would cause the same conscious experience as that brain, in the specific sense of thinking literally the exact same sequence of thoughts in the exact same order, in perpetuity.
I… don’t think this is necessarily what @EuanMcLean meant? At the risk of conflating his own perspective and ambivalence on this issue with my own, this is a question of personal identity and whether the computationalist perspective, generally considered a “reasonable enough” assumption to almost never be argued for explicitly on LW, is correct. As I wrote a while ago on Rob’s post:
As TAG has written a number of times, the computationalist thesis seems not to have been convincingly (or even concretely) argued for in any LessWrong post or sequence (including Eliezer’s Sequences). What has been argued for, over and over again, is physicalism, and then more and more rejections of dualist conceptions of souls.
That’s perfectly fine, but “souls don’t exist and thus consciousness and identity must function on top of a physical substrate” is very different from “the identity of a being is given by the abstract classical computation performed by a particular (and reified) subset of the brain’s electronic circuit,” and the latter has never been given compelling explanations or evidence. This is despite the fact that the particular conclusions that have become part of the ethos of LW about stuff like brain emulation, cryonics etc are necessarily reliant on the latter, not the former.
As a general matter, accepting physicalism as correct would naturally lead one to the conclusion that what runs on top of the physical substrate works on the basis of… what is physically there (which, to the best of our current understanding, can be represented through Quantum Mechanical probability amplitudes), not what conclusions you draw from a mathematical model that abstracts away quantum randomness in favor of a classical picture, the entire brain structure in favor of (a slightly augmented version of) its connectome, and the entire chemical make-up of it in favor of its electrical connections. As I have mentioned, that is a mere model that represents a very lossy compression of what is going on; it is not the same as the real thing, and conflating the two is an error that has been going on here for far too long. Of course, it very well might be the case that Rob and the computationalists are right about these issues, but the explanation up to now should make it clear why it is on them to provide evidence for their conclusion.
I recognize you wrote in response to me a while ago that you “find these kinds of conversations to be very time-consuming and often not go anywhere.” I understand this, and I sympathize to a large extent: I also find these discussions very tiresome, which became part of why I ultimately did not engage too much with some of the thought-provoking responses to the question I posed a few months back. So it’s totally ok for us not to get into the weeds of this now (or at any point, really). Nevertheless, for the sake of it, I think the “everyday experience” thermostat example does not seem like an argument in favor of computationalism over physicalism-without-computationalism, since the primary generator of my intuition that my identity would be the same in that case is the literal physical continuity of my body throughout that process. I just don’t think there is a “prosaic” (i.e., bodily-continuity-preserving) analogue or intuition pump to the case of WBE or similar stuff in this respect.
Anyway, in light of footnote 10 in the post (“The question of whether such a simulation contains consciousness at all, of any kind, is a broader discussion that pertains to a weaker version of CF that I will address later on in this sequence”), which to me draws an important distinction between a brain-simulation having some consciousness/identity versus having the same consciousness/identity as that of whatever (physically-instantiated) brain it draws from, I did want to say that this particular post seems focused on the latter and not the former, which seems quite decision-relevant to me:
jbash: These various ideas about identity don’t seem to me to be things you can “prove” or “argue for”. They’re mostly just definitions that you adopt or don’t adopt. Arguing about them is kind of pointless.
sunwillrise: I absolutely disagree. The basic question of “if I die but my brain gets scanned beforehand and emulated, do I nonetheless continue living (in the sense of, say, anticipating the same kinds of experiences)?” seems the complete opposite of pointless, and the kind of conundrum in which agreeing or disagreeing with computationalism leads to completely different answers.
Perhaps there is a meaningful linguistic/semantic component to this, but in the example above, it seems understanding the nature of identity is decision-theoretically relevant for how one should think about whether WBE would be good or bad (in this particular respect, at least).

sunwillrise 30 Nov 2024 2:17 UTC
3 points
2
in reply to: gwern’s comment on: gwern’s Shortform
All of these ideas sound awesome and exciting, and precisely the right kind of use of LLMs that I would like to see on LW!

sunwillrise 30 Nov 2024 2:13 UTC
4 points
−1
in reply to: Noosphere89’s comment on: A shot at the diamond-alignment problem
It’s looking like the values of humans are far, far simpler than a lot of evopsych literature and Yudkowsky thought, and related to this, values are less fragile than people thought 15-20 years ago, in the sense that values generalize far better OOD than people used to think 15-20 years ago
I’m not sure I like this argument very much, as it currently stands. It’s not that I believe anything you wrote in this paragraph is wrong per se, but more like this misses the mark a bit in terms of framing.
Yudkowsky had (and, AFAICT, still has) a specific theory of human values in terms of what they mean in a reductionist framework, where it makes sense (and is rather natural) to think of (approximate) utility functions of humans and of Coherent Extrapolated Volition as things-that-exist-in-the-territory.
I think a lot of writing and analysis, summarized by me here, has cast a tremendous amount of doubt on the viability of this way of thinking and has revealed what seem to me to be impossible-to-patch holes at the core of these theories. I do not believe “human values” in the Yudkowskian sense ultimately make sense as a coherent concept that carves reality at the joints; I instead observe a tremendous number of unanswered questions and apparent contradictions that throw the entire edifice in disarray.
But supplementing this reorientation of thinking around what it means to satisfy human values has been “prosaic” alignment researchers pivoting more towards intent alignment as opposed to doomed-from-the-start paradigms like “learning the true human utility function” or ambitious value learning, a recognition that realism about (AGI) rationality is likely just straight-up false and that the very specific set of conclusions MIRI-clustered alignment researchers have reached about what AGI cognition will be like are entirely overconfident and seem contradicted by our modern observations of LLMs, and ultimately an increased focus on the basic observation that full value alignment simply is not required for a good AI outcome (or at the very least to prevent AI takeover). So it’s not so much that human values (to the extent such a thing makes sense) are simpler, but more so that fulfilling those values is just not needed to nearly as high a degree as people used to think.

sunwillrise 29 Nov 2024 16:59 UTC
7 points
0
in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform
Mainly, minecraft isn’t actually out of distribution, LLMs still probably have examples of nice / not-nice minecraft behaviour.
Is this inherently bad? Many of the tasks that will be given to LLMs (or scaffolded versions of them) in the future will involve, at least to some extent, decision-making and processes whose analogues appear somewhere in their training data.
It still seems tremendously useful to see how they would perform in such a situation. At worst, it provides information about a possible upper bound on the alignment of these agentized versions: yes, maybe you’re right that you can’t say they will perform well in out-of-distribution contexts if all you see are benchmarks and performances on in-distribution tasks; but if they show gross misalignment on tasks that are in-distribution, then this suggest they would likely do even worse when novel problems are presented to them.

sunwillrise 28 Nov 2024 3:35 UTC
2 points
−1
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
a lot of skill ceilings are much higher than you might think, and worth investing in
The former doesn’t necessarily imply the latter in general, because even if we are systematically underestimating the realistic upper bound for our skill level in these areas, we would still have to deal with diminishing marginal returns to investing in any particular one. As a result, I am much more confident of the former claim being correct for the average LW reader than of the latter. In practice, my experience tells me that you often have “phase changes” of sorts, where there’s a rather binary instead of continuous response to a skill level increase: either you’ve hit the activation energy level, and thus unlock the self-reinforcing loop of benefits that flow from the skill (once you can apply it properly and iterate on it or use it recursively), or you haven’t, in which case any measurable improvement is minimal. It’s thus often more important to get past the critical point than to make marginal improvements either before or after hitting it.
On the other hand, many of the skills you mentioned afterwards in your comment seem relatively general-purpose, so I could totally be off-base in these specific cases.

sunwillrise 28 Nov 2024 3:08 UTC
2 points
1
in reply to: habryka’s comment on: The Power to Demolish Bad Arguments
The document seems to try to argue that Uber cannot possibly become profitable. I would be happy to take a bet that Uber will become profitable within the next 5 years.
And indeed it already has.

sunwillrise 27 Nov 2024 22:00 UTC
2 points
0
in reply to: MondSemmel’s comment on: Repeal the Jones Act of 1920
This is an otherwise valuable discussion that I’d rather not have on LW, for the standard reasons; it seems a bit to close to the partisan side of the policy/partisanship political discussion divide. I recognize I wrote a comment in reaction to yours (shame on me), and so you were fully within your rights to respond, but I’d rather stop it here.

sunwillrise 27 Nov 2024 21:30 UTC
1 point
0
in reply to: MondSemmel’s comment on: Repeal the Jones Act of 1920
But rather that if he doesn’t do it, it will be because he doesn’t want to, not because his constituents don’t.
I generally prefer not to dive into the details of partisan politics on LW, but my reading of the comment you are responding to makes me believe that, by “Republicans under his watch”, ChristianKl is referring to Republican politicians/executive appointees and not to Republican voters.
I am not saying I agree with this perspective, just that it seems to make a bit more sense to me in context. The idea would be that Trump has been able to use “leadership” to remake the Republican party in his image and get the party elites to support him only because he has mostly governed as a standard conservative Republican on economic issues (tax cuts for rich people&corporations, attempts to repeal the ACA, deregulation, etc); the symbiotic relationship they enjoy would therefore supposedly have as a prerequisite the idea that Trump would not try to enforce idiosyncratic views on other Republicans too much...

sunwillrise 22 Nov 2024 22:54 UTC
5 points
2
in reply to: ESRogs’s comment on: Consciousness as a conflationary alliance term
Tabooing words is bad if, by tabooing, you are denying your interlocutors the ability to accurately express the concepts in their minds.
We can split situations where miscommunication about the meaning of words persists despites repeated attempts by all sides to resolve it into three broad categories. On the one hand, you have those that come about because of (explicit or implicit) definitional disputes, such as the famous debate (mentioned in the Sequences) over whether trees that fall make sounds if nobody is around to hear them. Different people might have give different responses (literally ‘yes’ vs ‘no’ in this case), but this is simply because they interpret the words involved differently. When you replace the symbol with the substance, you realize that there is no empirical difference in what anticipated experiences the two sides have, and thus the entire debate is revealed to be a waste of time. By dissolving the question, you have resolved it.
That does not capture the entire cluster of persistent semantic disagreements, however, because there is one other possible upstream generator of controversy, namely the fact that, often times, the concept itself is confused. This often comes about because one side (or both) reifies or gives undue consideration to a mental construct that does not correspond to reality; perhaps the concept of ‘justice’ is an example of this, or the notion of observer-independent morality (if you subscribe to either moral anti-realism or to the non-mainstream Yudkowskian conception of realism). In this case, it is generally worthwhile to spend the time necessary to bring everyone on the same page that the concept itself should be abandoned and we should avoid trying to make sense of reality through frameworks that include it.
But, sometimes, we talk about confusing concepts not because the concepts themselves ultimately do not make sense in the territory (as opposed to our fallible maps), but because we simply lack the gears-level understanding required to make sense of our first-person, sensual experiences. All we can do is bumble around, trying to gesture at what we are confused about (like consciousness, qualia, etc), without the ability to pin it down with surgical precision. Not because our language is inadequate, not^[1] because the concept we are honing in on is inherently nonsensical, but because we are like cavemen trying to reason about the nature of the stars in the sky. The stars are not an illusion, but our hypotheses about them (‘they are Gods’ or ‘they are fallen warriors’ etc) are completely incompatible with reality. Not due to language barriers or anything like that, but because we lack the large foundation and body of knowledge needed to even orient ourselves properly around them.
To me, consciousness falls into the third category. If you taboo too many of the most natural, intuitive ways of talking about it, you are not benefitting from a more careful and precise discussion of the concepts involved. On the contrary, you are instead forcing people who lack the necessary subject-matter knowledge (i.e., arguably all of us) to make up their own hypotheses about how it functions. Of course they will come to different conclusions; after all, the hard problem of consciousness is still far from being settled!
1. ^
  At least not necessarily because of this; you can certainly take an illusionistic perspective on the nature of consciousness.

sunwillrise 22 Nov 2024 17:04 UTC
1 point
2
in reply to: Noosphere89’s comment on: Bogdan Ionut Cirstea’s Shortform
I agree, but I think this is slightly beside the original points I wanted to make.

sunwillrise 22 Nov 2024 14:19 UTC
37 points
8
on: LLM chatbots have ~half of the kinds of “consciousness” that humans believe in. Humans should avoid going crazy about that.
I continue to strongly believe that your previous post is methodologically dubious and does not provide an adequate set of explanations of what “humans believe in” when they say “consciousness.” I think the results that you obtained from your surveys are ~ entirely noise generated by forcing people who lack the expertise necessary to have a gears-level model of consciousness (i.e., literally all people in existence now or in the past) to talk about consciousness as though they did, by denying them the ability to express themselves using the language that represents their intuitions best.
Normally, I wouldn’t harp on that too much here given the passage of time (water under the bridge and all that), but literally this entire post is based on a framework I believe gets things totally backwards. Moreover, I was very (negatively) surprised to see respected users on this site apparently believing your previous post was “outstanding” and “very legible evidence” in favor of your thesis.
I dearly hope this general structure does not become part of the LW zeitgeist for thinking about an issue as important as this.
What links here?
- sunwillrise's comment on Consciousness as a conflationary alliance term for intrinsically valued internal experiences by Andrew_Critch (22 Nov 2024 22:54 UTC; 5 points)

sunwillrise 22 Nov 2024 14:03 UTC
14 points
11
in reply to: Bogdan Ionut Cirstea’s comment on: Bogdan Ionut Cirstea’s Shortform
arguments about risks from destabilization/power dynamics and potential conflicts between various actors are probably both more legible and ‘truer’
Say more?
I think the general impression of people on LW is that multipolar scenarios and concerns over “which monkey finds the radioactive banana and drags it home” are in large part a driver of AI racing instead of being a potential impediment/solution to it. Individuals, companies, and nation-states justifiably believe that whichever one of them accesses potentially superhuman AGI first will have the capacity to flip the gameboard at-will, obtain power over the entire rest of the Earth, and destabilize the currently-existing system. Standard game theory explains the final inferential step for how this leads to full-on racing (see the recent U.S.-China Commission’s report for a representative example of how this plays out in practice).
I get that we’d like to all recognize this problem and coordinate globally on finding solutions, by “mak[ing] coordinated steps away from Nash equilibria in lockstep”. But I would first need to see an example, a prototype, of how this can play out in practice on an important and highly salient issue. Stuff like the Montreal Protocol banning CFCs doesn’t count because the ban only happened once comparably profitable/efficient alternatives had already been designed; totally disanalogous to the spot we are in right now, where AGI will likely be incredibly economically profitable, perhaps orders of magnitude more so than the second-best alternative.
This is in large part why Eliezer often used to challenge readers and community members to ban gain-of-function research, as a trial run of sorts for how global coordination on pausing/slowing AI might go.