Raemon
Karma: 23,694 (LW), 4 (AF)
NewTop
Avoiding Jargon Confusion
Solving problems is mostly a matter of total resources devoted, not time devoted. … If we get signs of a problem looming, and can devote a lot of resources then.
Hmm. I don’t have as strong opinions about this, but this premise doesn’t seem obviously true.
I’m thinking about the “is science slowing down?” question – pouring 1000x resources into various scientific fields didn’t result in 1000x speedups. In some cases progress seemed to slow down. The three main hypotheses I have are:
Low hanging fruit got used up, so the problems got harder
Average careerist scientists don’t matter much, only extremely talented, naturally motivated researchers matter. The naturally motivated researchers will do the work anyway.
Coordination is hard and scales with the number of people coordinating. If you have 1000x the researchers in a field, they can’t find each other’s best work that easily.
I agree that “time spent” isn’t the best metric, but it seems like what actually matters is “quality researcher hours that build on each other in the right way,” and it’s not obvious how much you can scale that.
If it’s just the low hanging fruit hypothesis then… that’s fine I guess. But if the “extreme talent/motivation” or “coordination” issues are at play, then you want (respectively) to ensure that:
a) at any given time, talented people who are naturally interested in the problem have the freedom to work on it, if there are nonzero things to do with it, since there won’t be that many of them in the future.
b) build better coordination tools so that people in the future can scale their efforts better.
(You may also want to make efforts not to get mediocre careerist scientists involved in the field)
I was hoping this would propose solutions on either how to do group selection for dating or blogposts, or some alternate strategy for avoiding the pitfalls of naive individual competition. Curious if you have thoughts on that?
I feel like the people from whom I learned to distinguish social-reality from reality-reality, their technique depended a lot of being deliberately confusing or weird, in large part to break me out of established patterns of thought.
Eliezer wrote rather plainly about distinguishing social reality, beliefs-as-attire, etc. And I think this was sufficient to help me notice the reality/social-reality distinction in groups I wasn’t part of myself, or no longer primarily-identified as. But it seemed surprisingly useful to listen to weird rants by other iconoclasts in order to get a clearer sense of how-social-reality-feels from the inside.
(I think those people also may have had other agendas going on that the confusion may have also been part of. I do wonder at the fact that the people who most wanted to break me of my immersion in social reality also had weird agendas that benefited from me being disoriented)
((Apologies for being a bit cryptic))
Conditional on the nonfoom scenario, what is the appropriate indication that you should notice, to start converting resources into work?
If the world where there may or may not be a foom, how likely does foom need to be to make it correct to work on sooner?
FYI, this was a significant update for me. I just wanted to note that this is a bigger part of my model now as opposed to an edge case tacked on.
I hadn’t actually been invited much to google docs where this dynamic would come up, but it makes sense that this experience would be common. (I’ve only actually talked to one other person who shared your experience, so still up for updating backwards again, but I don’t expect to)
Which should either cause me to downgrade the importance of the “feeling safer in a private space”, or have it apply a bit differently than I was expecting it to have applied. (I think it still applies to the author of the original paper, maybe less so to commenters. Although I do think think having a sense that your fellow commenters are being filtered for some kind of “on-the-same-page-ness” still improves the conversation in other ways)
I meant in the other direction, where people judge ideas as better because higher status people said them.
This seems like the thing that happens by default and we can’t really stop it, but I’m wary of UX paradigms that might reinforce it even harder.
We used to have an option to make a post “unlisted” rather than just “draft” which would allow arbitrary people to look at it. Getting the UI right for it was a bit tricky and people kept finding ways to find unlisted posts that they hadn’t been meant to get access to so we made it admin-only for now. But we do intend to restore that eventually.
I do like the idea of karma-limited share buttons.
I think most of the incentives for commenting are due to network effects, i.e. not everyone is here, or I don’t have evidence that they’re here, so still feel like more people will see discussion on FB.
I think social proof is going to turn out to be pretty important. I’m slightly wary of it because it pushes against the “LW is a place you can talk about ideas, as much as possible without having social status play into it”, but like it or not “High Profile User liked my comment”, or “My Friend liked my comment” is way more motivating.
I’m currently thinking about how to balance those concerns.
(if you are thinking “huh, that’s reasonable, but I notice the all-posts-page literally isn’t linked from anywhere what up?” the answer is that “it was in beta, it’ll replace the daily page someday soon, after we successfully make sure it’s strictly better than the daily page, and will then be linked pretty prominently.“)
Yeah this through me quite a bit
I do think there exist work teams that don’t suck (and/or if they are framed properly they’re a lot more reasonable. When I worked at a large corporation and had to interface both with my team and with HR in order to build an automated-HR-system, I had an initial period of being frustrated by my boss and and the HR contacts I worked with. But I enjoyed working with my coworkers, I eventually got a different boss, and I built a better relationship with the HR person which resulted in a much smoother experience.
And that seemed fine.
I learned coordination at various other jobs over time, and (admittedly less generalizable) group houses full of rationalists.
This comment doesn’t seem to sufficiently engage with (what I saw as) the core question Rob was asking (and which I would ask), which was:
I personally care about things other than suffering. What are negative utilitarians saying about that?
Are they saying that they don’t care about things like friendship, good food, joy, catharsis, adventure, learning new things, falling in love, etc., except as mechanisms for avoiding suffering? Are they saying that I’m deluded about having preferences like those? Are they saying that I should try to change my preferences — and if so, why?
You briefly note “you may be overly attached to them”, but this doesn’t give any arguments for why I might be overly attached to them, instead of attached to them the correct amount.
When you ask:
To actually reject NU, you must explain what makes something (other than suffering) terminally valuable (or as I say, motivating) beyond its instrumental value for helping us prevent suffering in the total context.
My response is “to reject NU, all I have to do is terminally care about anything other than suffering. I care about things other than suffering, ergo NU must be false, and the burden is on other people to explain what is wrong with my preferences.”
Nod. And apologies for armchair psychologizing which I do think is generally bad form.
Relatedly, rereading this post was what prompted me to write this stub post:
I’m fairly concerned with the practice of telling people who “really care about AI safety” to go into AI capabilities research, unless they are very junior researchers who are using general AI research as a place to improve their skills until they’re able to contribute to AI safety later. (See Leveraging Academia).
The reason is not a fear that they will contribute to AI capabilities advancement in some manner that will be marginally detrimental to the future. It’s also not a fear that they’ll fail to change the company’s culture in the ways they’d hope, and end up feeling discouraged. What I’m afraid of is that they’ll feel pressure to start pretending to themselves, or to others, that their work is “relevant to safety”. Then what we end up with are companies and departments filled with people who are “concerned about safety”, creating a false sense of security that something relevant is being done, when all we have are a bunch of simmering concerns and concomitant rationalizations.
This fear of mine requires some context from my background as a researcher. I see this problem with environmentalists who “really care about climate change”, who tell themselves they’re “working on it” by studying the roots of a fairly arbitrary species of tree in a fairly arbitrary ecosystem that won’t generalize to anything likely to help with climate change.
My assessment that their work won’t generalize is mostly not from my own outside view; it comes from asking the researcher about how their work is likely to have an impact, and getting a response that either says nothing more than “I’m not sure, but it seems relevant somehow”, or an argument with a lot of caveats like “X might help with Y, which might help with Z, which might help with climate change, but we really can’t be sure, and it’s not my job to defend the relevance of my work. It’s intrinsically interesting to me, and you never know if something could turn out to be useful that seemed useless at first.”
At the same time, I know other climate scientists who seem to have actually done an explicit or implicit Fermi estimate for the probability that they will personally soon discover a species of bacteria that could safely scrub the Earth’s atmosphere of excess carbon. That’s much better.
I agree with both individual points but… for the second point, can’t you pass the recursive buck almost as easily there?
At least “what should I have thought about already for outsourcing questions to emulations?” seems like a pretty good first question to ask.
Serious question: are you depressed? So far most negative utilitarians I’ve known were depressed and some stopped being negative utilitarian after fixing a chemical imbalance that hampered their ability to experience good things.
I’m not in a sensitive place but I’m not sure whether I want to read it or not. Can you give a rough sense of what you got out of it?
In most cases my thought is “well, what’s the alternative?”
I’m either doing what I would have done after thinking for N years, or I’m committing to a course of action after thinking less than N years. The former risks value drift, the latter risks… well, not having had as much time to think, which isn’t obviously better than value drift.
I do think there’s a few variations that seem like improvements, like:
run X copies of myself, slightly randomizing their starting conditions, running them for a range of times (maybe as wide as “1 week” to “10,000 years”. Before revealing their results to me, reveal how convergent they were. If there’s high convergence I’m probably less worried about the answer. )
make sure simulated me can only think about certain classes of things (“solve this problem with these constraints”). I’m more worried about value drift from “10,000 year me who lived life generally” than “10,000 year me who just thought about this one problem. Unless the problem was meta-ethics, in which case I probably want some kind of value drift.”
Upvoted both for the helpful summary of considerations, as well as for (I think) following a pretty good algorithm (making a good faith effort to assess something important to your identity)
My current take is something like:
One one hand, in the end, I expect you’ll do the thing that your brain is naturally curious about and the justifications are mostly post-hoc. And that’s probably fine and I wouldn’t stress too much about it. I think a lot of good progress comes by people incrementally following their curiosities, and fighting against your natural curiosities doesn’t seem very practical for intellectual work.
But, insofar as your interests are malleable, I think it’d be worth asking more specific questions for each of the plausible ways voting-theory-might matter.
If the goal is “improve elections, largely because of their second-order effects”, then I’d ask what sort of progress is most bottlenecking that. (The answer may be more political than theoretical, and you may or may not be interested in doing political/activism work. But even narrowing the scope to theoretical progress, my guess is that problems vary in how relevant they are to the “get concrete reform passed for government elections”)
If the goal is “figure out how optimal decisionmaking should be made in the transhumanist future, or in CEV”, I’m guessing that the theoretical bottlenecks there are different for the nearterm election reform.