And he sent the message in a way that somehow implied that I was already supposed to have signed up for that policy, as if it’s the most normal thing in the world, and with no sense that this is a costly request to make (or that it was even worth making a request at all, and that it would be fine to prosecute someone for violating this even if it had never been clarified at all as an expectation from the other side).
Speaking generally, many parties get involved in zero-sum resource conflicts, and sometimes form political alliances to fight for their group to win zero-sum resource conflicts. For instance, if Alice and Bob are competing to get the same job, or Alice is trying to buy a car for a low price and Bob is trying to sell it to her for a high price, then if Charlie is Alice’s ally, she might hope that Charlie all will take actions that help her get more/all of the resources in these conflicts.
Allies of this sort also expect that they can share information that is easy to use adversarially against them between each other, with the expectation it will be consistently used either neutrally or in their favor by the allies.
Now, figuring out who your allies are is not a simple process. There are no forms involved, there are no written agreements, it can be fluid, and picked up in political contexts by implicit signals. Sometimes you can misread it. You can think someone is allied, tell them something sensitive, then realize you tricked yourself and just gave sensitive information to someone. (The opposite error also occurs, where you don’t realize someone is your ally and don’t share info and don’t pick up all the value on the table.)
My read here is that Mikhail told Habryka some sensitive information about some third party “Jackson”, assuming that Habryka and Jackson were allied. Habryka, who was not allied with Jackson in this way, was simply given a scoop, and felt free to share/use that info in ways that would cause problems for Jackson. Mikhail said that Habryka should treat it as though they were allies, whereas Habryka felt that he didn’t deserve it and that Mikhail was saying “If I thought you would only use information in Jackson’s favor when telling you the info, then you are obligated to only use information in Jackson’s favor when using the info.” Habryka’s response is “Uh, no, you just screwed up.”
(Also, after finding out who “Jackson” is from private comms with Mikhail, I am pretty confused why Mikhail thought this, as I think Habryka has a pretty negative view of Jackson. Seems to me simply like a screw-up on Mikhail’s part.)
I don’t know how costly/beneficial this screw up concretely was to humanity’s survival, but I guess that total cost would’ve been lower if Habryka as a general policy were more flexible in when the sensitivity of information has to be negotiated.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka. I may blabber out something that has a high negative expected utility if Habryka shares it (after conditioning on the event that he shares it) and I don’t have a way to cheaply fix that mistake (which would bound the risk).
And there isn’t an equally strong opposing force afaict? I can imagine blabbering out something that I’d afterwards negotiate to keep between us, where Habryka cannot convince me to let him share it, and yet it would’ve been better to allow him to share it.
Tbc, my expectations for random people are way worse, but Habryka seems below average among famous rationalists now? I rn see & feel in average zero pull to adjust my picture of the average famous rationalist up or down, but seems high variance since I didn’t ever try to learn what policies rationalists follow wrt negotiating information disclosure. I definitely didn’t expect them to use policies mentioned in planecrash outside fun low-stake toy scenarios.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka.
Feel free to update on “Oliver had one interaction ever with Mikhail in which Oliver refused to make a promise that Mikhail thought reasonable”, but I really don’t think you should update beyond that. Again, the summaries in this post of my position are very far away from how I would describe them.
There is a real thing here, which if you don’t know you should know, which is that I do really think confidentiality and information-flow constraints are very bad for society. They are the cause of as far as I can tell a majority of major failures in my ecosystem in the last few years, and mismanagement of e.g. confidentiality norms was catastrophic in many ways, so I do have strong opinions about this topic! But the summary of my positions on this topic is really very far from my actual opinions.
Thx, I think I got most of this from your top level comment & Mikhail’s post already. I strongly expect that I do not know your policy for confidentiality right now, but I also expect that once I do I’d disagree with it being the best policy one can have, just based on what I heard from Mikhail and you about your one interaction.
My guess is that refusing the promise is plausibly better than giving it for free? But I guess that there’d have been another solution where 1) Mikhail learns not to screw up again, and 2) you get to have people talk more freely around you to a degree that’s worth loosing the ability to make use of some screw-ups, and 3) Mikhail compensates you in case that 1+2 is still too far away from a fair split of the total expected gains.
I expect you’ll say that 2) sounds pretty negative to you, and that you and the community should follow a policy where there’s way less support for confidentiality, which can be achieved by exploiting screw-ups and by sometimes saying no if people ask for confidentiality in advance, so that people who engage in confidentiality either leave the community or learn to properly share information openly.
I mostly just want people to become calibrated about the cost of sharing information with strings attached. It is quite substantial! It’s OK for that coordination to happen based on people’s predictions of each other, without needing to be explicitly negotiated each time.
I would like it to be normalized and OK for someone to signal pretty heavily that they consider the cost of accepting secrets, or even more intensely, the cost of accepting information that can only be used to the benefit of another party, to be very high. People should therefore model that kind of request as likely to be rejected, and so if you just spew information onto the other party, and also expect them to keep it secret or to only be used for your benefit, that the other party is likely to stop engaging with you, or to tell you that they aren’t planning to meet your expectations.
I think marginally the most important thing to do is to just tell people who demand constraints on information, without wanting to pay any kind of social cost for it, to pound sand.
(A large part of the goals of this post is to communicate to people that Oliver considers the cost of accepting information to be very high, and make people aware that they should be careful around Oliver and predict him better on this dimension, not repeating my mistake of expecting him not to do so much worse than a priest of Abadar would.)
I think you could have totally written a post that focused on communicating that, and it could have been a great post! Like, I do think the cost of keeping secrets is high. Both me and other people at Lightcone have written quite a bit about that. See for example “Can you keep this confidential? How do you know?”
Speaking generally, many parties get involved in zero-sum resource conflicts, and sometimes form political alliances to fight for their group to win zero-sum resource conflicts. For instance, if Alice and Bob are competing to get the same job, or Alice is trying to buy a car for a low price and Bob is trying to sell it to her for a high price, then if Charlie is Alice’s ally, she might hope that Charlie all will take actions that help her get more/all of the resources in these conflicts.
Allies of this sort also expect that they can share information that is easy to use adversarially against them between each other, with the expectation it will be consistently used either neutrally or in their favor by the allies.
Now, figuring out who your allies are is not a simple process. There are no forms involved, there are no written agreements, it can be fluid, and picked up in political contexts by implicit signals. Sometimes you can misread it. You can think someone is allied, tell them something sensitive, then realize you tricked yourself and just gave sensitive information to someone. (The opposite error also occurs, where you don’t realize someone is your ally and don’t share info and don’t pick up all the value on the table.)
My read here is that Mikhail told Habryka some sensitive information about some third party “Jackson”, assuming that Habryka and Jackson were allied. Habryka, who was not allied with Jackson in this way, was simply given a scoop, and felt free to share/use that info in ways that would cause problems for Jackson. Mikhail said that Habryka should treat it as though they were allies, whereas Habryka felt that he didn’t deserve it and that Mikhail was saying “If I thought you would only use information in Jackson’s favor when telling you the info, then you are obligated to only use information in Jackson’s favor when using the info.” Habryka’s response is “Uh, no, you just screwed up.”
(Also, after finding out who “Jackson” is from private comms with Mikhail, I am pretty confused why Mikhail thought this, as I think Habryka has a pretty negative view of Jackson. Seems to me simply like a screw-up on Mikhail’s part.)
I don’t know how costly/beneficial this screw up concretely was to humanity’s survival, but I guess that total cost would’ve been lower if Habryka as a general policy were more flexible in when the sensitivity of information has to be negotiated.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka. I may blabber out something that has a high negative expected utility if Habryka shares it (after conditioning on the event that he shares it) and I don’t have a way to cheaply fix that mistake (which would bound the risk).
And there isn’t an equally strong opposing force afaict? I can imagine blabbering out something that I’d afterwards negotiate to keep between us, where Habryka cannot convince me to let him share it, and yet it would’ve been better to allow him to share it.
Tbc, my expectations for random people are way worse, but Habryka seems below average among famous rationalists now? I rn see & feel in average zero pull to adjust my picture of the average famous rationalist up or down, but seems high variance since I didn’t ever try to learn what policies rationalists follow wrt negotiating information disclosure. I definitely didn’t expect them to use policies mentioned in planecrash outside fun low-stake toy scenarios.
Feel free to update on “Oliver had one interaction ever with Mikhail in which Oliver refused to make a promise that Mikhail thought reasonable”, but I really don’t think you should update beyond that. Again, the summaries in this post of my position are very far away from how I would describe them.
There is a real thing here, which if you don’t know you should know, which is that I do really think confidentiality and information-flow constraints are very bad for society. They are the cause of as far as I can tell a majority of major failures in my ecosystem in the last few years, and mismanagement of e.g. confidentiality norms was catastrophic in many ways, so I do have strong opinions about this topic! But the summary of my positions on this topic is really very far from my actual opinions.
Thx, I think I got most of this from your top level comment & Mikhail’s post already. I strongly expect that I do not know your policy for confidentiality right now, but I also expect that once I do I’d disagree with it being the best policy one can have, just based on what I heard from Mikhail and you about your one interaction.
My guess is that refusing the promise is plausibly better than giving it for free? But I guess that there’d have been another solution where 1) Mikhail learns not to screw up again, and 2) you get to have people talk more freely around you to a degree that’s worth loosing the ability to make use of some screw-ups, and 3) Mikhail compensates you in case that 1+2 is still too far away from a fair split of the total expected gains.
I expect you’ll say that 2) sounds pretty negative to you, and that you and the community should follow a policy where there’s way less support for confidentiality, which can be achieved by exploiting screw-ups and by sometimes saying no if people ask for confidentiality in advance, so that people who engage in confidentiality either leave the community or learn to properly share information openly.
I mostly just want people to become calibrated about the cost of sharing information with strings attached. It is quite substantial! It’s OK for that coordination to happen based on people’s predictions of each other, without needing to be explicitly negotiated each time.
I would like it to be normalized and OK for someone to signal pretty heavily that they consider the cost of accepting secrets, or even more intensely, the cost of accepting information that can only be used to the benefit of another party, to be very high. People should therefore model that kind of request as likely to be rejected, and so if you just spew information onto the other party, and also expect them to keep it secret or to only be used for your benefit, that the other party is likely to stop engaging with you, or to tell you that they aren’t planning to meet your expectations.
I think marginally the most important thing to do is to just tell people who demand constraints on information, without wanting to pay any kind of social cost for it, to pound sand.
(A large part of the goals of this post is to communicate to people that Oliver considers the cost of accepting information to be very high, and make people aware that they should be careful around Oliver and predict him better on this dimension, not repeating my mistake of expecting him not to do so much worse than a priest of Abadar would.)
I think you could have totally written a post that focused on communicating that, and it could have been a great post! Like, I do think the cost of keeping secrets is high. Both me and other people at Lightcone have written quite a bit about that. See for example “Can you keep this confidential? How do you know?”
This post focuses on communicating that! (+ being okay with hosting ai capabilities events + less important misc stuff)