I don’t know how costly/beneficial this screw up concretely was to humanity’s survival, but I guess that total cost would’ve been lower if Habryka as a general policy were more flexible in when the sensitivity of information has to be negotiated.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka. I may blabber out something that has a high negative expected utility if Habryka shares it (after conditioning on the event that he shares it) and I don’t have a way to cheaply fix that mistake (which would bound the risk).
And there isn’t an equally strong opposing force afaict? I can imagine blabbering out something that I’d afterwards negotiate to keep between us, where Habryka cannot convince me to let him share it, and yet it would’ve been better to allow him to share it.
Tbc, my expectations for random people are way worse, but Habryka seems below average among famous rationalists now? I rn see & feel in average zero pull to adjust my picture of the average famous rationalist up or down, but seems high variance since I didn’t ever try to learn what policies rationalists follow wrt negotiating information disclosure. I definitely didn’t expect them to use policies mentioned in planecrash outside fun low-stake toy scenarios.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka.
Feel free to update on “Oliver had one interaction ever with Mikhail in which Oliver refused to make a promise that Mikhail thought reasonable”, but I really don’t think you should update beyond that. Again, the summaries in this post of my position are very far away from how I would describe them.
There is a real thing here, which if you don’t know you should know, which is that I do really think confidentiality and information-flow constraints are very bad for society. They are the cause of as far as I can tell a majority of major failures in my ecosystem in the last few years, and mismanagement of e.g. confidentiality norms was catastrophic in many ways, so I do have strong opinions about this topic! But the summary of my positions on this topic is really very far from my actual opinions.
Thx, I think I got most of this from your top level comment & Mikhail’s post already. I strongly expect that I do not know your policy for confidentiality right now, but I also expect that once I do I’d disagree with it being the best policy one can have, just based on what I heard from Mikhail and you about your one interaction.
My guess is that refusing the promise is plausibly better than giving it for free? But I guess that there’d have been another solution where 1) Mikhail learns not to screw up again, and 2) you get to have people talk more freely around you to a degree that’s worth loosing the ability to make use of some screw-ups, and 3) Mikhail compensates you in case that 1+2 is still too far away from a fair split of the total expected gains.
I expect you’ll say that 2) sounds pretty negative to you, and that you and the community should follow a policy where there’s way less support for confidentiality, which can be achieved by exploiting screw-ups and by sometimes saying no if people ask for confidentiality in advance, so that people who engage in confidentiality either leave the community or learn to properly share information openly.
I mostly just want people to become calibrated about the cost of sharing information with strings attached. It is quite substantial! It’s OK for that coordination to happen based on people’s predictions of each other, without needing to be explicitly negotiated each time.
I would like it to be normalized and OK for someone to signal pretty heavily that they consider the cost of accepting secrets, or even more intensely, the cost of accepting information that can only be used to the benefit of another party, to be very high. People should therefore model that kind of request as likely to be rejected, and so if you just spew information onto the other party, and also expect them to keep it secret or to only be used for your benefit, that the other party is likely to stop engaging with you, or to tell you that they aren’t planning to meet your expectations.
I think marginally the most important thing to do is to just tell people who demand constraints on information, without wanting to pay any kind of social cost for it, to pound sand.
(A large part of the goals of this post is to communicate to people that Oliver considers the cost of accepting information to be very high, and make people aware that they should be careful around Oliver and predict him better on this dimension, not repeating my mistake of expecting him not to do so much worse than a priest of Abadar would.)
I think you could have totally written a post that focused on communicating that, and it could have been a great post! Like, I do think the cost of keeping secrets is high. Both me and other people at Lightcone have written quite a bit about that. See for example “Can you keep this confidential? How do you know?”
I don’t know how costly/beneficial this screw up concretely was to humanity’s survival, but I guess that total cost would’ve been lower if Habryka as a general policy were more flexible in when the sensitivity of information has to be negotiated.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka. I may blabber out something that has a high negative expected utility if Habryka shares it (after conditioning on the event that he shares it) and I don’t have a way to cheaply fix that mistake (which would bound the risk).
And there isn’t an equally strong opposing force afaict? I can imagine blabbering out something that I’d afterwards negotiate to keep between us, where Habryka cannot convince me to let him share it, and yet it would’ve been better to allow him to share it.
Tbc, my expectations for random people are way worse, but Habryka seems below average among famous rationalists now? I rn see & feel in average zero pull to adjust my picture of the average famous rationalist up or down, but seems high variance since I didn’t ever try to learn what policies rationalists follow wrt negotiating information disclosure. I definitely didn’t expect them to use policies mentioned in planecrash outside fun low-stake toy scenarios.
Feel free to update on “Oliver had one interaction ever with Mikhail in which Oliver refused to make a promise that Mikhail thought reasonable”, but I really don’t think you should update beyond that. Again, the summaries in this post of my position are very far away from how I would describe them.
There is a real thing here, which if you don’t know you should know, which is that I do really think confidentiality and information-flow constraints are very bad for society. They are the cause of as far as I can tell a majority of major failures in my ecosystem in the last few years, and mismanagement of e.g. confidentiality norms was catastrophic in many ways, so I do have strong opinions about this topic! But the summary of my positions on this topic is really very far from my actual opinions.
Thx, I think I got most of this from your top level comment & Mikhail’s post already. I strongly expect that I do not know your policy for confidentiality right now, but I also expect that once I do I’d disagree with it being the best policy one can have, just based on what I heard from Mikhail and you about your one interaction.
My guess is that refusing the promise is plausibly better than giving it for free? But I guess that there’d have been another solution where 1) Mikhail learns not to screw up again, and 2) you get to have people talk more freely around you to a degree that’s worth loosing the ability to make use of some screw-ups, and 3) Mikhail compensates you in case that 1+2 is still too far away from a fair split of the total expected gains.
I expect you’ll say that 2) sounds pretty negative to you, and that you and the community should follow a policy where there’s way less support for confidentiality, which can be achieved by exploiting screw-ups and by sometimes saying no if people ask for confidentiality in advance, so that people who engage in confidentiality either leave the community or learn to properly share information openly.
I mostly just want people to become calibrated about the cost of sharing information with strings attached. It is quite substantial! It’s OK for that coordination to happen based on people’s predictions of each other, without needing to be explicitly negotiated each time.
I would like it to be normalized and OK for someone to signal pretty heavily that they consider the cost of accepting secrets, or even more intensely, the cost of accepting information that can only be used to the benefit of another party, to be very high. People should therefore model that kind of request as likely to be rejected, and so if you just spew information onto the other party, and also expect them to keep it secret or to only be used for your benefit, that the other party is likely to stop engaging with you, or to tell you that they aren’t planning to meet your expectations.
I think marginally the most important thing to do is to just tell people who demand constraints on information, without wanting to pay any kind of social cost for it, to pound sand.
(A large part of the goals of this post is to communicate to people that Oliver considers the cost of accepting information to be very high, and make people aware that they should be careful around Oliver and predict him better on this dimension, not repeating my mistake of expecting him not to do so much worse than a priest of Abadar would.)
I think you could have totally written a post that focused on communicating that, and it could have been a great post! Like, I do think the cost of keeping secrets is high. Both me and other people at Lightcone have written quite a bit about that. See for example “Can you keep this confidential? How do you know?”
This post focuses on communicating that! (+ being okay with hosting ai capabilities events + less important misc stuff)