Hi, I am a Physicist, an Effective Altruist and AI Safety researcher.
Linda Linsefors
Karma: 3,638
Then the AI just have to control multiple people, right? Dunbar’s number is a human limotation. If the whole company is too big for any human to oversee, that might even be an advantage for an AI. E.g. the AI could sidline any humans that are harder to manipulate, without anyone knowing what is happening.
I’m not sure what you mean?
The poly case, as written, is also not about sex. It’s about taking time away from one loved one, to help another loved one. Which is something that may happen if you have more than one loved one, including platonic loved ones, such as fiends and family.
Or are you saying that in the poly case it’s actually about jealousy, while pretending to be about something else? In that case, I see where all the drama comes from.
Note to self: Don’t date anyone who thinks you need new special rules for this type of scenarios.
Will the lawsuit have a high chance of succeeding? This is very untested ground. Maybe some funding can help with the incentives here?
It seems like there should be a fund/org that help people sue Open AI for all the hacking their models are doing, to get a bit more accountability. And other frontier labs too of course when/if we see them fuck up in similar ways.
Is there something like this already?
Is there anyone who would like to do this if there was money?
Is there anyone who would like to fund this if someone would run it?
This could be a fund that hands out money to people that got hacked, so they can hire their own lawyers. Or it could be an org with it’s own lawyers that does the suing on people’s behalf.
If your partner’s lover is having a mental health breakdown, is it OK for your partner to go comfort her when it’s your day with him?
How is this a poly specific problem?
What if your partner’s close friend is having mental health breakdown and needs support?
When ever I see people writing about the downsides of poly, there is always one or more items like this on the list, i.e. things that would apply if you have more than one human in your life that you care about.
I’m not saying poly is for everyone. If you experience strong sexual jealousy, then don’t do it. I’m just saying that (it seems to me) some of these arguments proof too much, since they suggest you only get to have one important person in your life. E.g. this reasoning would imply that no-one should have more than one child.
Suggestion (very different from my other suggestion):
I suspect trying to solve this without honest input from someone in the “professional” group will just backfire. Therefore, I think the first step has to be, that you personally have to build friendship/repor one-on-one with one or more people among the professionals. Then you and them together can figure out what to do.
[Epistemic status: Low confidence speculations]
I notice some comments are suggesting that the rationalist hide their weirdness. I want to point out that this is only possible if you separate the rationalist into a separate co-working space away from the professionals, and only have the tow groups interact occasionally. This may or may not be the correct trade-off.
Anecdote: LISA (the main London AI Safety coworking space) used to be at WeWork for a while, before they got their own space. I was there a few times during this era. The LISA organizers got constant complains from WeWork because the AI Safety people didn’t behave “professional” enough. And this was a situation where we didn’t even try to work together, or interact with other groups renting desks in the same building. The main complaints I remember was that the rationalist would not ware shoes, and take napps in the common sofas.
A lot of rationalists are autistic, and/or gender-non-conforming, or just don’t want to where shoes all day. Some of us are never going to appear normal. Some of us can mask for a day, but not all day every day. And a few (probably) could adopt and fit in anywhere.
So far I’ve only talked about physical appearance. But this also apply to interest, communication style, etc.
So maybe the solution is not to have one space, but to have two spaces, but visit each other? And for everyone to be mindful that when your visiting you’re a guest in a different culture?
The theory is that it easier to learn to appreciate a florigen culture, if you don’t have to put up with it too much. Because being out of your element constantly is draining, and leave you with fewer spoons.
What you have now is (maybe?) that the tow groups are constantly exposed to each other, are constantly low grade fed up with each other, and therefore mostly self segregating, while in the same office. But maybe it would be better to have actual separation most of the time, and regular deliberate deep engagement?
Suggestion: If your space have more than one floor, or more than one room, you can have part of the space being rat space and part be policy professional space. The rules are that you can visit, but if you do so, you should try to conform to the space you are visiting. Expect on Fridays, when everyone is encourage to mingle, or something like that.
I suggest being less brief in the future when writing on LW. The bit that you cut out was actually important. We like nuance over brevity here.
I think it would be good to clarify in the post that that this: “consulting, public service policy, procurement, nonprofit advocacy” is the group of professionals you’re talking about.
E.g. you got one comment from a ex professional programmer who though they where in the professional category.
(Other than that, great post!)
I have a hypothesis that a large part of the culture divide between “professionals” and and “rationalist” is a divide between neurotypical leaning people and autism leaning people. Under that theory, I would not be surprised if professional software engineer is closer to the rat side, than to policy professionals.
Communication and understanding can never rest on one side, because that will just fail.
People become professionals all the time
A lot of professional culture is neurotypical culture, and a lot of rat culture is autism culture, and people don’t just change neurotype. That’s obliviously not all of it, there are learned parts too. But for some people it is easy to become a rationalist and hard to become a passing professional.
Also, it is not necessary to be a rationalist; the 80⁄20 which many professionals can do is to understand x-risk and take it seriously, which is getting easier and easier.
I think this line misses the point of the post? I agree that it is not necessary for everyone to become a rationalist. And I agree that understanding x-risk is important. But that is not all it takes to work together. That also required mutual understanding and respect. If every conversation with the other group feels weird and off-putting, that is a problem. I
What more do you want from jargon?
What I wanted was for someone to point-out/explain the more specific nuance that made this jargon worth it. This question has been answered in other responses.
What’s up with the jargon “zero day”? Why not just say “previously unknown”?
I’m often pro-jargon. I.e. I think a lot of jargon exist for good reasons, so I’m open to this one also being good in some way. E.g jargon can be useful for naming precise technical concepts.
Anyone want to defend “zero day” jargon?
No steel-maning please! I.e. only want answers from people who genuinely think that this is a good jargon.
Update: My question has been answered in the comments.
Thinking that your actions matter is necessary (but not sufficient) for making right choices under your own values.
I’m not making a claim about if people over estimate more than under estimate. I’m saying that I’m seeing a pattern where when someone with good intentions do bad things, this is often due to underestimating their own power.
I think that if you are trying to have an impact on the world, you should put extra weight on possible worlds where you are powerful, in your impact calculations, because those are the worlds where you’re actions matter.
But looking on the other side, what harm may you do when over estimating your power (which is different from putting more decision weight on worlds where you are powerful). Thinking about this angle, I expect harm of the type [over promise and under deliver]. E.g, taking on an important job, where others rely on you, and then dropping the ball. Although this requires more people than yourself to overestimate your powers. Otherwise it just looks like [you tried a thing and failed] which is not doing any harm to anyone else.
On the other hand, if you underestimate your power, you might publish some new AI capabilities method that was obvious to you, thinking it doesn’t matter. This does not require anyone else to be wrong about you.
I think that, sometimes people do the wrong thing because they underestimate their own power.
Examples: (in no particular order)
AI safety researchers making capabilities progress, thinking it doesn’t matter because “it was so obvious” or “someone else where going to do it anyway”.
Sidenote on the above. About a decade ago, when there where approximately no AI Safety training programs yet, the standard advice for aspiring AI Safety researchers, from within EA, was to go do an ML (capabilities) PhD, to gain skills. The reasoning was that there where so many people in capabilities anyway, that any contributions you made there would not matter.
-
Quote (from memory) from this video about the battle of the board https://youtu.be/_eYTkvZqbnQ?si=_GfUCXjJtpBgXH-j “She was looking to see where the wind was blowing and didn’t realize she was the wind” (about the interim CEO just after Sam Alman was fired). She took Sam’s side.
-
Some months ago I listened to Behind the Shock Machine: The Untold Story of the Notorious Milgram Psychology Experiments, about the Milgram Experiment (https://en.wikipedia.org/wiki/Milgram_experiment). There are a few reasons many people went all the way, but one re-occurring reason semes to have been, not realizing that stopping was an option.
-
Lot’s of people in AI coms, thinking that they have to stay inside the Overton window, instead of trying to shift the Overton widow.
-
Possibly (but there could be other reasons people want to take these actions):
-
AI companies thinking they have to race
-
Some people thinking US has to race against China
-
All the powerful people who didn’t stop Google’s deal with the Department of War
Acknowledgment: I though of this because of Richard Ngo’s talk and following discussion at Iliad, about all the ways AI safety people have screwed up and helped capabilities. I think this theory explains some of that, but not all.
I don’t know what you mean. Why does the derivative of L4 scale with error size?
Also, L4 is is not special. See https://www.lesswrong.com/posts/cTRKj3giaZN5Ysyx2/compressed-computation-under-l-loss-is-likely-computation-in
I agree that the situation you’re describing is somewhat halo-defense-y rationally-defensible, in the spirit of ruling thinkers in, instead of out.
I think you read something into my comment that I did not mean. I did not mean to say that less people should be kicked out. I’m making no comment on that either way. I’m just saying that if someone disagrees with the criteria for what is a unforgivable act, and it’s taboo to argue over what should be forgivable, then they may (on the surface) argue over the facts instead, which may look like halo defense.
Specifically, there is a discussion about if person A has done [unforgivable thing]. Person B don’t think [unforgivable thing] should be unforgivable, but just a normal bad thing, that can be forgiven if A has enough other good qualities. Person B thinks that probably a lot of people agree with them, but no-one can admit that they think [unforgivable thing] is not infinitely bad, without large social risk. So instead person B gestures at all the reason we would all like to keep person A around, and suggest we pretend that person A did not do [unforgivable thing].
Regarding how much of halo-defense-y stuff is something like what you’re describing: IDK. At least on the spot, I’m finding it hard to come up with examples of “healthy halo defense” from my own experience and observation, but can easily generate more examples of the “unhealthy” type. The healthy kind[1] is coherent/plausible, but IDK to what extent it actually instantiates.
I’m not saying halo-defense is healthy. It’s not. I’m saying it might be a symptom of a different problem, which means you’d have to solve that problem to get rid of halo-defense.
In case it’s useful, I’m pretty sure you can adopt just the epistemic framework of infra-Bayesianism, without accepting the action selection part. The “paranoia” lives in the action selection part.
If I remember correctly, the epistemic part, is a type signature for partial hypothesizes, and an update rule, which is the generalisation of Bayes’ Theorem, over this new class of believes.