I’m an independent researcher currently working on a sequence of posts about consciousness. You can send me anonymous feedback here: https://www.admonymous.co/rafaelharth.
Rafael Harth
Karma: 5,259
So I think the norm is something like “if you write something that will predictably make people feel worse about [real person or org], you should stick to journalistic standards of citing sources and such”. That means all your quotes depend on whether you’ve sufficiently established the substance of the quote.
If we take your post as it is now, well you only have one source, which is the group letter to congress. Imo as you used it this actually does not even establish that they’re anti nuclear power because the letter is primarily about fossil fuels, and the quote about nuclear power is in the context of protecting indigenous rights. Also you said it was signed with 600 other companies, so it might have been a compromise (maybe they oppose some parts of the content but thought the entire thing was still worth signing). An endorsement of a compromise/package is just really not a good way to establish their position. It would be much better to just look at the Wikipedia page and see whether that says they’re anti nuclear. Which in fact it does in the introduction. Some would probably quibble with that but for me that would actually be enough. So if you just did that, then I’d excuse all quotes that only reference them being anti-nuclear power (which I guess is just the first in your list).
Saying that they’re my enemy is a little harder because it would require establishing that they’re a negative for climate protection on net. This is not obvious; you could have an org that’s anti nuclear power and still does more good than harm overall. It probably still wouldn’t be that difficult, but your post as is certainly falls short. (And BTW it’s also not obvious that being anti nuclear power now is as bad as having been anti nuclear power historically. It could be the case that having been anti nuclear power historically was a huge mistake and we should have invested in the technology all this time, but that since we didn’t, at this point it actually no longer makes sense and we should only invest in renewables. I don’t think that’s the case, I think we should probably still build nuclear reactors now, but I’m genuinely not sure. This kind of thing very much matters for the ‘net negative impact’ question.)
Specifically, it should be about Lesswrong having a bad culture. One that favours norms that make punishing enemies harder, up to the point of not being able to straightforwardly say “if you are pro-nuke, an org that has been anti-nuke for decades is your enemy”.
I think it’s very unlikely that having laxer standards for accusing others is a good thing. Broadly speaking it seems to me that ~100% of groups-that-argue-about-political-or-culture-war-topics suffer from having too low standards for criticizing the outgroup, and ~0% suffer from having too high standards. And I don’t think these standards are even that high, like you could write a post that says Greenpeace is my enemy, you’d just have to put in the effort to source your claims a little. Or, more practically, you could have just written the post about a fictional org, then you can make your point about enemies without having to deal with the practical side of attacking a real org.
Not related but
why the Lesswrong community has supported three orgs racing to AGI.
This was not my impression. My impression was that people associated with the community have founded orgs that then did capability research, but that many, probably most, people on LW think that’s a disaster. To varying degrees. People are probably less negative on Anthropic than OpenAI. We’re certainly not enthusiastic about OpenAI.. In any case I don’t think it summarizes to “the Lesswrong community has supported” these orgs.
Yea, having similar feelings about this post. The conclusion is probably still correct, but not sufficiently established. And I think there should be, idk, a norm about being more thorough when talking badly about an org, and violating that doesn’t seem worth the point made here.
Hm, they all show up for me I think? Maybe it was something temporary?
Okay so even though I’ve already written a full-length post about timelines, I thought I should make a shortform putting my model into a less eloquent and far more speculative-sounding and capricuous format. Also I think the part I was hedging the most on in the post is probably the most important aspect of the model.
I propose that the ability to make progress on...
well-defined problems with verifiable solutions; vs.
murky problems where the solution criterion is unclear and no one can ever prove anything
… are two substantially different dimensions of intelligence, and IQ is almost entirely about the first one. The second one isn’t in-principle impossible to measure, it’s probably not even difficult, but extremely difficult to make a socially respected test for it because you could almost only include questions where the right answer is up for debate. I called this philosophical intelligence in my post because philosophical problems are usually great examples, but it’s not restricted to those. You could also things like
Is neoliberalism or progressivism a better governing philosophy?
Should we ship weapons to Ukraine?
What’s the best way to teach {insert topic here}?
Of course you can’t put those onto a test any more than you can ask “does liberterian free will exist?” on a test, so the existence of non-philosophical questions here doesn’t make measuring this ability any easier.
People often point to someone famous saying something they think is stupid and then say things like “this again proves that being an expert in one domain doesn’t translate into being smart anywhere else!” This always rubbed me the wrong way because intelligence in one area should transfer to other areas! It’s all general problem-solving capability! But in fact, those people do exist, and I’ve talked to some of them. People who have genuine intellectual horsepower on narrow problems, but as I ask them anything about a more fuzzy topic, their take is just so surface level and dumb that my immediate reaction is always this sense of disbelief, like, “it shouldn’t be possible for your thoughts here to be this shallow given how smart you are!”
… but conversely, there clearly is such a thing as expertise in a narrow area correlating with smart philosophical/political views. So sometimes intelligence does transfer and sometimes it doesn’t...
Well, I think it’s obvious what point I’m going to make here; I think sometimes people are experts in their field due to #1 and sometimes #2, and the extent that it’s #2 this tends to transfer into making sense on other questions, whereas to the extent it’s #1, it’s in fact almost meaningless. (And some people become famous without either #1 and #2, but less so if they’re experts in technical fields.)
I think #2 has outsized importance for progress on many things related to AI alignment and rationality. For example, I think Eliezer is quite high in both #1 and #2, but the reason he has produced a more useful body of work than the average genius has much more to do with #2. Almost nothing in the sequences seems to require genius level IQ; I think he could be a SD lower in IQ and still have written most of them. It would make a difference, don’t get me wrong, but I don’t think it would be the bottleneck. (None of this depends on what Eliezer is up to nowadays btw, you can ignore the last 15 years for this paragraph.)
Now what about dangerous capability advances and takeover scenarios from LLMs; can those happen without #2? Imo, absolutely not. Not even a little bit. You can have all sorts of negative effects of the kind that are already happening—job loss, increased social isolation, information silos, misinformation, maybe even some extent of capability enhancement, stuff like that—but the classical superingelligence-ian scenarios require the ability to make progress on problems with murky and unverifiable solutions.
I think the entire notion that LLMs can’t really come up with novel concepts—one of the less stupid criticisms of LLMs, imo—is a direct result of this (coming up with a novel concept is exactly the kind of thing you need #2 for because there’s no way to verify whether any one idea for a new concept does or doesn’t make sense). Although this is not absolute because sometimes they can spit out new ideas at random; the “inability to derive new concepts” framing doesn’t quite point at the right thing since creativity isn’t the issue, it’s the ability to reliably figure out whether a new concept is actually useful. The disconnect between stuff like METR’s supposed exponential growth in LLM’s capabilities on long-horizon tasks and actual job replacement on those tasks is another. There is just a really fundamental problem here where metrics for AI progress are biased towards things you can measure—duh! -- which systematically biases toward #1 over #2. (Although METR has actually acknowledged this at least a little bit, I feel like they’ve actually been very epistemically virtuous from what I could see, so I don’t wanna trash them.)
Or to just put it all very bluntly, if LLMs cannot answer questions as easy as “does libertarian free will exist” or “what’s the right interpretation of quantum mechanics?”—and they can’t—then clearly they’re not very smart. And I think they’re not very smart in a way that is necessary for basically all of the doom-y scenarios.
I’m not expecting anyone to agree with any of this, but in a nutshell, much of my real skepticism about LLM scaling is about the above, especially lately. I don’t think we’re particularly close to AGI… and consequently, I also don’t think much of the classical superintelligence-ian views have actually been tested, one way or another.
I definitely think developing equanimity without meditation is a thing. The description checks out.
About the applicability, maybe you could extend it to other types of injuries (and positive sensations!) with a higher skill level? I doubt there are different types that work differently.
I’ll read it (& comment if I have anything to say). But man the definition for the concept your post is about is pretty important, even if it’s “semantics”. Specifically, if this post were actually just about self-awareness (which does not seem to be the case, from a first skim), then I wouldn’t even be interested in reading it because I don’t think self-awareness is particularly related to consciousness, and it’s not a topic I’m separately interested in. Maybe edit it? If you’re not just talking about X, then no reason to open the post by saying that you are.
Edit: actually I gave up reading it (but this has nothing to do with the opening paragraph), I find it very difficult to follow/understand where you’re trying to go with it. I think you have to motivate this better to keep people interested. (Why is the time gap important? Why is the pathway important? What exactly is this post even about?) I didn’t downvote though.
Apologies for commenting without reading the entire post, but I’m just going to give my rant about this particular aspect of the topic. It’s about the opening definition of your post, so it’s kinda central.
Consciousness is the state of being aware of one’s existence, sensations and thoughts
I think defining consciousness as self-awareness is just such a non-starter. It’s not what realists mean by consciousness, and even if you’re taking an illusionist point of view, it doesn’t capture most of what consciousness-the-fuzzy-high-level-category does in the brain.
As David Pearce has pointed out, a lot of the most intense conscious experiences don’t include any self-awareness/reflection at all, just as being in a state of panic running away from a fire. Or taking psychedelics. Or being in intense pain. Or intense pleasure. Conversely, it’s not that difficult to include some degree of elementary self-awareness in a machine, and I don’t think that would make it conscious. (Again, neither in the realist sense, nor in the consciousness-as-a-fuzzy-category sense. There are just so many functions that consciousness does for humans that don’t have anything to do with self-awareness.)
The highest entropy input channel, as far as conscious content is concerned, is undoubtedly vision. The conscious aspect is continuously present, and it’s pretty difficult to explain (how can we perceive an entire image at the same time? What does that even mean?), and there’s evidence that it’s a separate thing from template-based visual processing (-> blindsight). Imho people talk way too much about self-reference when it comes to consciousness, and way too little about vision.
I mean of course it’s true today, right? It would be weird to make a prediction “AI can’t do XX in the future” (and that’s most of the predictions here) if that isn’t true today.
(Have read the post.) I disagree. I think overall habryka has gone through much greater pains than I think he should have to, but I don’t think this post is a part he should have skimped on. I would feel pretty negative about it if habryka had banned Said without an extensive explanation for why (modulo past discussions already kinda providing an explanation). I’d expect less transparency/effort for banning less important users.
I think Sam Harris had the right idea when he said (don’t have the link unfortunately) that asking about the meaning of life is just bad philosophy. No one who is genuinely content asks about the meaning of life. No one is like “well I feel genuine fulfillment and don’t crave anything, but I just gotta know, what’s the meaning of everything?” Meaning is a thing you ask about if you feel dissatisfied. (And tbqh that’s kinda apparent from your OP.)
So at the real risk of annoying you (by being paternalizing/not-actually-aswering-your-question), I think asking about meaning is the wrong approach altogether. The thing that, if you had it, would make it feel like you’ve solved your problem, is fulfillment. (Which I’m using in a technical way but it’s essentially happiness + absence of craving.) I’d look into meditation, especially equanimity practice.
That said, I think re-framing your life as feeling like it has more of a purpose isn’t generally pointless (even though it’s not really well-defined or attacking the root of the problem). But seems difficult in your case since your object-level beliefs about where we’re headed seem genuinely grim.
I feel like even accepting that actual model welfare is not a thing (as in, the model isn’t conscious) this might still be a reasonable feature just based on feedback to the user? Like if people are going to train social interactions based on LLM chats to whatever extent, then it’s probably better if they’ll face consequences. It can’t be too difficult to work around this.
The implication is valid in your formulation, but then Y doesn’t imply anything because it says nothing about the distribution. I’m saying that if you change Y to actually support your conclusion, then fails. Either way the entire argument doesn’t seem to work.
Fair enough. I’m mostly on board with that, my one gripe is that the definition only sounds similar to people who are into the Buddhist stuff. “Suffering mostly comes from craving” seems to me to be one of the true but not obvious insights from Buddhism. So just equating them in the definition is kinda provoking a reaction like from Said.
I agree but I don’t think the Buddhist definition is what Lsusr said it is (do you?). Suffering is primarily caused by the feeling that the world ought to be different but I don’t think it’s identical. Although I do expect you can find some prominent voices saying so.
Now you’re sort of asking me to do the work for you, but I did get interested enough to start thinking about it, so here’s my more invested take.
So first of all, I don’t see how this is a form of the fallacy of the undistributed middle. The article you linked to says that we’re taking and and conclude . I don’t see how your fallacy is doing (a probabilistic version of) that. Your fallacy is taking as given (with meaning “makes more likely”), and and concluding
Second
We’ve substituted a sharp condition for a vague one, hence the name diagnostic dilution.
I think we’ve substituted a vague condition for a sharp one, not vice-versa? The 32 bit integer seems a lot more vague than the number being about kids?
Third, your leading example isn’t an example of this fallacy, and I think you only got away with pretending it’s one by being vague about the distribution. Because if we tried to fix it, it would have to be like this
A: the number is probably > 100000
X: the number is a typical prior for having kids
Y: the number is a roughly uniformly distributed 32 bit integer
And is not actually true here. Whereas in the example you’re criticizing
A: the AI will have seek to eliminate threatening agents
X: the AI builds football stadiums
Y: the AI has goal-directed behavior
here does seem to be true.
(And also I believe the fallacy isn’t even a fallacy because if and together do in fact imply , at least if both are sufficiently strong?)
So my conclusion here is that the argument actually just doesn’t work, or I still just don’t get what you’re asserting,[1] but the example you make does not seem to have the same problems as the example you’re criticizing, and neither of them seems to have the same structure as the example of the fallacy you’re linking. (For transparency, I initially weak-downvoted because the post seemed confusing and I wasn’t sure if it’s valid, then removed the downvote because you improved the presentation, now strong-downvoted because the central argument seems just broken to me now.)
- ↩︎
like maybe the fallacy isn’t about propositions implying each other but instead about something more specific to an element being in a set, but at this point the point is just not argued clearly.
Would it help if I wrote them out more explicitly rather than tagged them into sentences?
Yes, the edit definitely makes it better.
Yes I have to choose a distribution but if I’m forced to predict an unknown int32 with no additional information the uniform distribution seems like a reasonable choice. Ad-hoc, not explicitly defined probability distributions are common in this discussion.
Well it’s clearly not a reasonable choice given that it results in a fallacy. I think if the actual error is using the wrong prior distribution then this should be reflected in what the diagnostic dilution fallacy is defined as. It’s not putting the element into the larger set because that isn’t false. I’d suggest a phrasing but I don’t have a good enough grasp on the concept to do this, and I’m still not even sure that the argument you’re criticizing is in fact an example of this fallacy if were phrased more precisely. (Also nitpicky, but “condition” in the definition is odd as well if they’re general hypotheses.)
This post is difficult for me to follow because
-
your use of types is inconsistent (the , , should be hypotheses I believe, but you’re saying stuff like this, which makes it sound like they’re half of hypotheses, or other objects---)
-
you say “I know their number of kids is a nonnegative 32 bit integer, and almost all such numbers are greater than 1 million. So I suppose they’re highly likely to have more than 1 million kids.” But this isn’t true; the premise you’re actually using here is that the number is a 32 bit integer with a uniform distribution. Just noting that an unknown element is in a set doesn’t itself imply anything about its probability distribution.
-
you’re not quoting the exact argument you’re criticizing
Maybe these are all just formalities that have no bearing on the validity of the critique, but at least for me they make it too difficult to figure out whether I agree with you or not to make it worth it, so I’m just giving up.
-
Not sure what this means? What is not okay if you agree-vote this?
I think the basic mechanism here is that people actually judge most things by their pleasantness to a much larger degree than they tend to admit, and general bad mood decreases pleasantness.
Directionally I agree with lc saying it sounds like you’re depressed, but I don’t think it actually has to be anywhere near clinical depression. I think “I’m generally sadder so things seem less exciting” is a very commonly true description. You reporting it could have more to do with you being more introspective than with how extreme the condition is.
The general remedy is just to improve your well-being, which of course is very difficult. But I don’t think there’s any conceptual move you can make that will help here; the core issue very much seems to be a lack of fulfillment.
yes. I don’t think any of them suggest that LessWrong is supporting or enthusiastic about OpenAI. (In particular, whether you should work there doesn’t have much relation to whether the company as a whole is a net negative.) I would describe the stance of top 2 comments on that post as mixed [1] and of LW’s stance in general as mixed-to-negative.
Fwiw this is not a crux, I might agree that we should be more negative toward OpenAI than we are. I don’t think that’s an argument for laxer standards of critcism. Standards for rigor should lead toward higher quality criticism, not less harsh criticism. If you had attacked Greenpeace twice as much but had substantiated all your claims, I wouldn’t have downvoted the post. I’d guess that the net effectiveness of a community’s criticism of a person or org goes up with stricter norms.
e.g., Ben pace also says, “An obvious reason to think OpenAI’s impact will be net negative is that they seem to be trying to reach AGI as fast as possible, and trying a route different from DeepMind and other competitors, so are in some world shortening the timeline until AI. (I’m aware that there are arguments about why a shorter timeline is better, but I’m not sold on them right now.)”