A lot of the claims about me, and about Lightcone, in this post are false, which is sad. I left a large set of comments on a draft of thist post, pointing out many of them, though not all of them got integrated before the post was published (presumably because this post was published in a rush as Mikhail is part of Inkhaven, and decided to make this his first post of Inkhaven, and only had like 2 hours to get and integrate comments).
A few quick ones, though this post has enough errors that I mostly just want people to really not update on this at all:
Oliver said that Lightcone would be fine with providing Lighthaven as a conference venue to AI labs for AI capabilities recruiting, perhaps for a higher price as a tax.
This is technically true, but of course the whole question lies in the tax! I think the tax might be quite large, possible enough to cover a large fraction of our total operational costs for many months (like a 3-4x markup on our usual cost of hosting such an event, or maybe even more). If you are deontologically opposed to Lighthaven ever hosting anything that has anything even vaguely to do with capability companies, no matter the price, then yeah, I think that’s a real criticism, but I also think it’s a very weird one. Even given that, at a high enough price, the cost to the labs would be virtually guaranteed to be more than they would benefit from it, making it a good idea even if you are deontologically opposed to supporting AI companies.
he said he already told some people and since he didn’t agree to the conditions before hearing the information, he can share it, even though wouldn’t go public with it.
The promise that Mikhail asked me to make was, as far as I understood it, to “not use any of the information in the conversation in any kind of adversarial way towards the people who the information is about”. This is a very strong request, much stronger than confidentiality (since it precludes making any plans on the basis of that information that might involve competing or otherwise acting against the interests of the other party, even if they don’t reveal any information to third parties). This is not a normal kind of request! It’s definitely not a normal confidentiality request! Mikhail literally clarified that he thought that it would only be OK for me to consider this information in my plans, if that consideration would not hurt the interests of the party we were talking about.
And he sent the message in a way that somehow implied that I was already supposed to have signed up for that policy, as if it’s the most normal thing in the world, and with no sense that this is a costly request to make (or that it was even worth making a request at all, and that it would be fine to prosecute someone for violating this even if it had never been clarified at all as an expectation from the other side).
He just learned that keeping secrets is bad in general, and so he doesn’t by default, unless explicitly agrees to.
This is not true! My policy is simply that you should not assume that I will promise to keep your secrets after you tell me, if you didn’t check with me first. If you tell me something without asking me for confidentiality first, and then you clarify that the information is sensitive, I will almost always honor that! But if you show up and suddenly demand of me that I will promise that I keep something a secret, without any kind of apology or understanding that this is the kind of thing you do in advance, of course I am not going to just do whatever you want. I will use my best judgement!
My general policy here is that I will promise to keep things secret retroactively, if I would have agreed to accept the information with a confidentiality request in advance. If I would have rejected your confidentiality request in advance, you can offer me something for the cost incurred by keeping the secret. If you don’t offer me anything, I will use my best judgement and not make any intense promises but broadly try to take your preferences into account in as much as it’s not very costly, or offer you some weaker promise (like “I will talk about this with my team or my partner, but won’t post it on the internet”, which is often much cheaper than keeping a secret perfectly).
Roughly the aim here is to act in a timeless fashion and to not be easily exploitable. If I wouldn’t have agreed to something before, I won’t agree to it just because you ask me later, without offering me anything to make up the cost to me!
And to repeat the above again, the request here was much more intense! The request, as I understood it, was basically “don’t use this information in any kind of way that would hurt the party the information is about, if the harm is predictable”, which I don’t even know how to realistically implement at a policy level. Of course if I end up in conflict with someone I will use my model of the world which is informed by all the information I have about someone!
And even beyond that, I don’t think I did anything with the relevant information that Mikhail would be unhappy about! I have indeed been treating the informations as sensitive. This policy might change if at some point the information looks more valuable to communicate. Mikhail seems only angry about me not fully promising to do what he wants, without him offering me anything in return, and despite me thinking that I would not have agreed to any kind of promise like this in the first place if I was asked to do that before receiving the information (and would have just preferred to never receive the information in the first place).
I ask Oliver to promise that he’s not going to read established users’ messages without it being known to others at Lightcone Infrastructure and without a justification such as suspected spam, and isn’t going to share the contents of the messages.
We’ve had internal policies here for a long time! We never look at DMs unless one of the users in the conversation reports a conversation as spam. Sometimes DM contents end up in error logs, but I can’t remember a time where I actually saw any message contents instead of just metadata in the 8 years that I’ve been working on LW (but we don’t have any special safeguards against it).
We look at drafts that were previously published. We also sometimes look at early revisions of posts that have been published for debugging purposes (not on-purpose, but it’s not something we currently have explicit safeguards or rules about). We never look at unpublished drafts, unless the user looks pretty clearly spammy, and never for established users.
It shouldn’t cost hundreds of thousands of dollars to keep a website running and moderated and even to ship new features with the help from the community.
Look, we’ve had this conversation during our fundraiser. There is zero chance of running an operation like LW 2.0 long-term without that not somehow costing at least $200k/yr. Even if someone steps up and does it for free, that is still them sacrificing at least $200k in counterfactual income, if they are skilled enough to run LessWrong in the first place. I think even at a minimum skeleton crew, you would be looking at at least $300k of costs.
The cost of running/supporting LessWrong is much lower than Lightcone Infrastructure’s spending.
This is false! Most of our spending is LessWrong spending these days (as covered in our annual fundraiser post). All of our other projects are much closer to paying for itself. Most of the cost of running Lightcone is the cost of running LessWrong (since it’s just a fully unmonetized product).
IDK, I am pretty sad about this post. I am happy to clarify my confidentiality policies and other takes on honoring retroactive deals (which I am generally very into, and have done a lot of over the years), if anyone ends up concerned as a result of it.
I will be honest in that it does also feel to me like this whole post was written in an attempt at retaliation when I didn’t agree with Mikhail’s opinions on secrets and norms. Like, I don’t think this post was written in an honest attempt at figuring out whether Lightcone is a good donation target.
He just learned that keeping secrets is bad in general, and so he doesn’t by default, unless explicitly agrees to.
This is not true! My policy is simply that you should not assume that I will promise to keep your secrets after you tell me, if you didn’t check with me first.
I can confirm; Oliver keeps many secrets from me, that he has agreed to others, and often keeps information secret based on implicit communication (i.e. nobody explicitly said that it was secret, but his confident read of the situation is that it was communicated with that assumption). I sometimes find this frustrating because I want to know things that Oliver knows :P
And he sent the message in a way that somehow implied that I was already supposed to have signed up for that policy, as if it’s the most normal thing in the world, and with no sense that this is a costly request to make (or that it was even worth making a request at all, and that it would be fine to prosecute someone for violating this even if it had never been clarified at all as an expectation from the other side).
Speaking generally, many parties get involved in zero-sum resource conflicts, and sometimes form political alliances to fight for their group to win zero-sum resource conflicts. For instance, if Alice and Bob are competing to get the same job, or Alice is trying to buy a car for a low price and Bob is trying to sell it to her for a high price, then if Charlie is Alice’s ally, she might hope that Charlie all will take actions that help her get more/all of the resources in these conflicts.
Allies of this sort also expect that they can share information that is easy to use adversarially against them between each other, with the expectation it will be consistently used either neutrally or in their favor by the allies.
Now, figuring out who your allies are is not a simple process. There are no forms involved, there are no written agreements, it can be fluid, and picked up in political contexts by implicit signals. Sometimes you can misread it. You can think someone is allied, tell them something sensitive, then realize you tricked yourself and just gave sensitive information to someone. (The opposite error also occurs, where you don’t realize someone is your ally and don’t share info and don’t pick up all the value on the table.)
My read here is that Mikhail told Habryka some sensitive information about some third party “Jackson”, assuming that Habryka and Jackson were allied. Habryka, who was not allied with Jackson in this way, was simply given a scoop, and felt free to share/use that info in ways that would cause problems for Jackson. Mikhail said that Habryka should treat it as though they were allies, whereas Habryka felt that he didn’t deserve it and that Mikhail was saying “If I thought you would only use information in Jackson’s favor when telling you the info, then you are obligated to only use information in Jackson’s favor when using the info.” Habryka’s response is “Uh, no, you just screwed up.”
(Also, after finding out who “Jackson” is from private comms with Mikhail, I am pretty confused why Mikhail thought this, as I think Habryka has a pretty negative view of Jackson. Seems to me simply like a screw-up on Mikhail’s part.)
I don’t know how costly/beneficial this screw up concretely was to humanity’s survival, but I guess that total cost would’ve been lower if Habryka as a general policy were more flexible in when the sensitivity of information has to be negotiated.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka. I may blabber out something that has a high negative expected utility if Habryka shares it (after conditioning on the event that he shares it) and I don’t have a way to cheaply fix that mistake (which would bound the risk).
And there isn’t an equally strong opposing force afaict? I can imagine blabbering out something that I’d afterwards negotiate to keep between us, where Habryka cannot convince me to let him share it, and yet it would’ve been better to allow him to share it.
Tbc, my expectations for random people are way worse, but Habryka seems below average among famous rationalists now? I rn see & feel in average zero pull to adjust my picture of the average famous rationalist up or down, but seems high variance since I didn’t ever try to learn what policies rationalists follow wrt negotiating information disclosure. I definitely didn’t expect them to use policies mentioned in planecrash outside fun low-stake toy scenarios.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka.
Feel free to update on “Oliver had one interaction ever with Mikhail in which Oliver refused to make a promise that Mikhail thought reasonable”, but I really don’t think you should update beyond that. Again, the summaries in this post of my position are very far away from how I would describe them.
There is a real thing here, which if you don’t know you should know, which is that I do really think confidentiality and information-flow constraints are very bad for society. They are the cause of as far as I can tell a majority of major failures in my ecosystem in the last few years, and mismanagement of e.g. confidentiality norms was catastrophic in many ways, so I do have strong opinions about this topic! But the summary of my positions on this topic is really very far from my actual opinions.
Thx, I think I got most of this from your top level comment & Mikhail’s post already. I strongly expect that I do not know your policy for confidentiality right now, but I also expect that once I do I’d disagree with it being the best policy one can have, just based on what I heard from Mikhail and you about your one interaction.
My guess is that refusing the promise is plausibly better than giving it for free? But I guess that there’d have been another solution where 1) Mikhail learns not to screw up again, and 2) you get to have people talk more freely around you to a degree that’s worth loosing the ability to make use of some screw-ups, and 3) Mikhail compensates you in case that 1+2 is still too far away from a fair split of the total expected gains.
I expect you’ll say that 2) sounds pretty negative to you, and that you and the community should follow a policy where there’s way less support for confidentiality, which can be achieved by exploiting screw-ups and by sometimes saying no if people ask for confidentiality in advance, so that people who engage in confidentiality either leave the community or learn to properly share information openly.
I mostly just want people to become calibrated about the cost of sharing information with strings attached. It is quite substantial! It’s OK for that coordination to happen based on people’s predictions of each other, without needing to be explicitly negotiated each time.
I would like it to be normalized and OK for someone to signal pretty heavily that they consider the cost of accepting secrets, or even more intensely, the cost of accepting information that can only be used to the benefit of another party, to be very high. People should therefore model that kind of request as likely to be rejected, and so if you just spew information onto the other party, and also expect them to keep it secret or to only be used for your benefit, that the other party is likely to stop engaging with you, or to tell you that they aren’t planning to meet your expectations.
I think marginally the most important thing to do is to just tell people who demand constraints on information, without wanting to pay any kind of social cost for it, to pound sand.
(A large part of the goals of this post is to communicate to people that Oliver considers the cost of accepting information to be very high, and make people aware that they should be careful around Oliver and predict him better on this dimension, not repeating my mistake of expecting him not to do so much worse than a priest of Abadar would.)
I think you could have totally written a post that focused on communicating that, and it could have been a great post! Like, I do think the cost of keeping secrets is high. Both me and other people at Lightcone have written quite a bit about that. See for example “Can you keep this confidential? How do you know?”
Even given that, at a high enough price, the cost to the labs would be virtually guaranteed to be more than they would benefit from it, making it a good idea even if you are deontologically opposed to supporting AI companies.
Well, presumably, if the lab is willing to make the trade, they at least believe that they’re benefiting from the trade, on net.
I don’t have a strong opinion on what kinds of trades you should make with AI labs, but “set a tax high enough that it’s not worth it for the lab on net” doesn’t seem like a totally crazy deontological rule?
Sure, and my policy above doesn’t rule that out. The only thing I said is that there is some price for which we’ll do it (my guess is de-facto there are probably some clearing prices here, as opposed to zero, but that would be a different conversation).
I wish someone would link the comment in question by habryka. I remember reading it, but I can’t find it.
I think you said you “would not be supprised” or “expect it will happen” or something like that, that you would rent lighthaven to the labs. Which did not give me the impression that the tax would be very high from the lab’s perspective.
I do think anyone (including habryka) have the right to say “oops, that was badly written, here’s what I acctually men.”
But what was said in that original comment still matters for wether or not this was a reasobable thing to be concerned about, before the interactions in the comments here.
My impression after reading that old comment from you was much more in line with what Mikhail said. So I’m happy this got borugh up and clarified.
I agree that promise is overly restrictive. ‘Don’t make my helping you have been a bad idea for me’ is a more reasonable version, but I assume you’re already doing that in your expectation, and it makes sense for different people to take the other’s expectation into account different amounts for this purpose.
Don’t make my helping you have been a bad idea for me
Yeah, I think this is a good baseline to aspire to, but of course the “my helping you” is the contentious point here. If you hurt me, and then also demand that I make you whole, then that’s not a particularly reasonable request. Why should I make you whole, I am already not whole myself!
Sometimes interactions are just negative-sum. That’s the whole reason why it does usually make sense to check-in beforehand before doing things that could easily turn out to be negative sum, which this situation clearly turned out to be!
Oliver said “The promise that Mikhail asked me to make was, as far as I understood it, to ‘not use any of the information in the conversation in any kind of adversarial way towards the people who the information is about’.”.
Oliver understood you to be asking him not to use the information to hurt anyone involved, which is way more restrictive, and in fact impossible for a human to do perfectly. Unless he meant something more specific by “any kind of adversarial way”, which promise wouldn’t get you what you want.
If you meant the reasonable thing, and said it clearly, I agree Oliver’s misunderstanding is surprising and probably symptomatic of not reading planecrash.
Yeah, no, initially, I simply asked: just in case, please don’t use [the information I shared it explicitly for the purpose of enabling Oliver to coordinate with the third party] except to coordinate with the third party, expecting “sure, no problem” in response.
Then, after hearing Oliver wouldn’t agree to confidentiality given that I haven’t asked him for it in advance, I tried to ask: okay, sure, if you have such a high cost of/principles relating to not telling other people things, please at least don’t try to tell people specifically for the purpose of harming the third party, making it a bad idea to have tried to coordinate. He then said that nope, he wouldn’t agree to even that partial confidentiality, because if, e.g., someone was considering whether it’s important to harm the third party now rather than later and telling them the information that I shared would’ve moved them towards harming the third party earlier, Oliver would want to share information with that someone so that they could harm the third party. (And also said he already told some people.)
(He ended up talking to the third party; but an opportunity to use the information adversarially did not turn up afaik.)
It’s plausible that he misunderstood what I was asking for throughout, but he had no intention of avoiding making it such that me trying to coordinate with him would have been a bad idea for me.
because if, e.g., someone was considering whether it’s important to harm the third party now rather than later and telling them the information that I shared would’ve moved them towards harming the third party earlier, Oliver would want to share information with that someone so that they could harm the third party.
No, I didn’t say anything remotely like this! I have no such policy! I don’t think I ever said anything that might imply such a policy. I only again clarified that I am not making promises about not doing these things to you. I would definitely not randomly hand out information to anyone who wants to harm the third party.
At this point I am just going to stop commenting every time you summarize me inaccurately, since I don’t want to spend all day doing this, but please, future readers, do not assume these summaries are accurate.
Then, after hearing Oliver wouldn’t agree to confidentiality given that I haven’t asked him for it in advance
I have clarified like 5 times that this isn’t because you didn’t ask in advance. If you had asked in advance I would have rejected your request as well, it’s just that you would have never told me in the first place.
don’t try to tell people specifically for the purpose of harming the third party
This is also not what you asked for! You said “I just ask you to not use this information in a way designed to hurt [third party]”, which is much broader. “Not telling people” and “not using information” are drastically different. I have approximately no idea how to commit to “not use information for purpose X”. Information propagates throughout my world model. If I end up in conflict with a third party I might want to compete with them and consider the information as part of my plans. I couldn’t blind myself to that information when making strategic decisions.
’Hypothetical scenario (this has not happened and details are made up):
Me and [name] are discussing the landscape of [thing] as it regards to Lightcone strategy. [name] is like “man, I feel like if I was worried that other people soon try to jump into the space, then we really should probably just back [a thing] because probably something will soon cement itself in the space”. I would be like “Oh, well, I think [third party] might do stuff”. Rafe is like “Oh, fuck, hmm, that’s bad”. I am like “Yep, seems pretty fucked. Plausibly we should really get going on writing up that ‘why [third party’s person] seems like a low-integrity dude’ post we’ve been thinking about”. [name] is like “Yeah, maybe. Does really seem quite bad if [third party’s person] tries to position himself here centrally. Actually, I think maybe [name] from CEA Comm health was working on some piece about [third party’s person]? Seems like she should know [third party’s person] is moving into the space, since it seems a bit more urgent if that’s happening”. I am like “Yep, seems right”.’
If you had asked in advance I would have rejected your request
You didn’t say that when we were talking about it! You implied that since I didn’t ask in advance, you are not bound by anything; you did mention “I can keep things confidential if you ask me in advance, but of course I wouldn’t accept a request to receive private information about [third party] being sketchy that I can only use to their benefit?”
(“Being sketchy” is not how I’d describe the information. It was about an idea that Oliver is not okay with the third party working on, but is okay with others working on, because he doesn’t like the third party for a bunch of reasons and thinks it’s bad if they get more power, as per my understanding.)
I did not and would not have demanded somehow avoiding propagating the information. If you were like, “sorry, I obviously can’t actually not propagate this information in my world model and promise it won’t reflect on my plans, but I won’t actively try to use outside of coordinating with the third party and will keep it confidential going forward”, that would’ve been great and expected and okay.
I asked to not apply effort to using the information against the third party. I didn’t ask to apply effort to not be aware of the information in your decision-making, to keep separate world-models, or whatever. Confidentiality with people outside your team and not going hard on figuring out how to strategically share or use this information to cause damage to the third party’s interests would’ve been understandable and acceptable.
I think your read of Habryka’s reply is mistaking a for-all quantifier and a there-exists quantifier. Insofar as you’re saying “never use this information to harm the interests of the third party” Habryka is saying “no; here is an instance I would want to share it that seems reasonable to me that does involve something the third party might find harmful”. This is distinct from “no; if I ever find a situation where I can use this info to harm the third-party, I will use the info to do that”.
Perhaps you’re right; I would love for that to be the case, and to have been wrong about all this. But this model- that it’s a there exists quantifier- is very surprised by a bunch of things from “lol, no, […]” to “I might use it that way. Like, I might tell someone who is worried about [third party] that they are planning to move into the space if it seems relevant. Or I might myself come to realize it’s important and then actively tell people to maybe do something about it.”
And, like, he didn’t give any examples of when he would not use the information.
His position was pretty clear to me: he thought that the fact the third party is moving into that space is bad, and if there is a way to use the information to prevent them from doing it, he would do so (but he didn’t see any ways of doing that and didn’t find it very important overall).
Like, there’s nothing in the messages to suggest otherwise.
He didn’t give an isolated example of when he’d want to share information for different reasons, where it would have a side-effect of hurting the interests of the third party. Instead, it was an example where the reason to share information was specifically that it would lead to hurting the interests of the third party.
He did call the information “strategically relevant”. He did say that he would continue to share the information basically at his sole discretion. He did say he might use it if he realizes it’s strategically important.
I really don’t have a coherent model of an alternative explanation you’re trying to point at.
(If you- or someone else- is available for that, I would love to jump on a call with someone who has a good model of Oliver and can explain to me the alternative explanation for what generated the messages.)
I do not think it at all describes a policy of “if someone was trying to harm the third party, and having this information would cause them to do it sooner, then I would give them the information”. Indeed, it seems really very far away from that! In the above story nobody is trying to actively harm anyone else as far as I can tell? I certainly would not describe “CEA Comm Health team is working on a project to do a bunch of investigations, and I tell them information that is relevant to how highly they should prioritize those investigations” as being anything close to “trying to harm someone directly”!
You didn’t say that when we were talking about it!
No, I literally said “Like, to be clear, I definitely rather you not have told me”. And then later “Even if I would have preferred knowing the information packaged with the request”. And my first response to your request said “You can ask in-advance if I want to accept confidentiality on something, and I’ll usually say no”.
If you were like, “sorry, I obviously can’t actually not propagate this information in my world model and promise it won’t reflect on my plans, but I won’t actively try to use outside of coordinating with the third party and will keep it confidential going forward”, that would’ve been great and expected and okay.
Sure, but I also wouldn’t have done that! The closest deal we might have had would have been a “man, please actually ask in advance next time, this is costly and makes me regret having that whole conversation in the first place. If you recognize that as a cost and owe me a really small favor or something, I can keep it private, but please don’t take this as a given”, but I did not (and continue to not) have the sense that this would actually work.
but I won’t actively try to use outside of coordinating with the third party
Maybe I am being dense here, and on first read this sounded like maybe a thing I could do, but after thinking more about it I do not know what I am promising if I promise I “won’t actively try to use [this information] outside of coordinating with the third party”. Like, am I allowed to write it in my private notes? Am I allowed to write it in our weekly memos as a consideration for Lightcone’s future plans? Am I not allowed to think the explicit thought “oh, this piece of information is really important for this plan that puts me in competition with this third party, better make sure to not forget it, and add it to my Anki deck?
Like, I am not saying there isn’t any distinction between “information passively propagating” and “actively using information”, but man, it feels like a very tricky distinction, and I do not generally want to be in the business of adding constraints to my private planning and thought-processes that would limit how I can operate here, and relies on this distinction being clear to other people. Maybe other people have factored their mind and processes in ways they find this easy, but I do not.
“man, please actually ask in advance next time, this is costly and makes me regret having that whole conversation in the first place. If you recognize that as a cost and owe me a really small favor or something, I can keep it private, but please don’t take this as a given”
This would’ve worked!
(Other branches seem less productive to reply to, given this.)
I changed my mind; at least in the case of my sharing information with you, if you were perfectly trustworthy you’d totally just defer to my beliefs for not making me worse off as a result. But, as you said, plausibly even in this easy case being perfect is way too hobbling for humans ’cause of infohazards.
I left a large set of comments on a draft of thist post, pointing out many of them, though not all of them got integrated before the post was published (presumably because this post was published in a rush as Mikhail is part of Inkhaven, and decided to make this his first post of Inkhaven, and only had like 2 hours to get and integrate comments).
Disappointed to see this kind of note.
The post is a lot less polished than it could’ve been and doesn’t make its points as strongly as I’d like, but to the best of my knowledge, none of the criticisms in this post are false.
All of the comments that you feel like weren’t integrated contained arguments that I consider invalid.
I didn’t reply to all of your comments because didn’t see much sense in that.
Supporting the idea that the criticisms are false with a note on “Mikhail must’ve not had time” is weird, especially given that I explicitly told you all that I find the arguments in your comments invalid and didn’t want to reply in detail from my phone.
at a high enough price, the cost to the labs would be virtually guaranteed to be more than they would benefit from it,
This was not the idea. The idea was that it would be okay to provide positive value to AI companies, given enough compensation to Lighthaven.
People who donated to keep Lighthaven going are not particularly happy about this (from n=2 people).
don’t use this information in any kind of way that would hurt the party the information is about, if the harm is predictable”, which I don’t even know how to realistically implement at a policy level
This is not the request that I made. I asked to not use information adversarially: to not try to cause harm to the third party using it.
And even beyond that, I don’t think I did anything with the relevant information that Mikhail would be unhappy about
Which (1) I was not made aware of by you prior to making the post and (2) is dependent on you not having ways to use the information to hurt the third party. This post is not made because you actually did something bad that hurt the third party; it’s made because you’re the kind of person who would, according to yourself.
Roughly the aim here is to act in a timeless fashion and to not be easily exploitable
That’s not what you did.
without him offering me anything in return
You didn’t signal in any way that any of that stuff was an option.
this whole post was written in an attempt at retaliation
lol, no. It’s made because others are very sad about the details and told me I should write about them; it’s made because I don’t want people do end up regretting working with you; it’s made vaguely at the beginning of Inkhaven (I didn’t want it to be the first post, though) to not made people sad about helping me write well when it’s published.
I don’t think this post was written in an honest attempt at figuring out whether Lightcone is a good donation target.
I am pretty sure Lightcone is not a good donation target as someone who donated personally significant amounts to Lightcone and talked to friends who previously have or considered donating large amounts to Lightcone, and then regretted that/decided not to after learning about all this.
This is not the request that I made. I asked to not use information adversarially: to not try to cause harm to the third party using it.
You said: “I don’t think I have any reason to ask you to not consider it in your plans insofar as these considerations are not hurting their interests or whatever” when I asked for clarification. This clearly implies you are asking me to not consider this information in my plans if doing so would hurt their interests!
You also clarified multiple other times that you were asking me to promise to not use this information in any future conflicts or anything like that, or to make plans on their basis that would somehow interfere with the other party’s plans, even if I thought they would cause grave harm if I didn’t interfere.
You didn’t signal in any way that any of that stuff was an option.
I am really not very optimistic about making agreements with you in-particular, based on how the one conversation I’ve ever had with you went. So no, that is not an option, though I will still try to do good by what I think you care about. But I do not want to risk you forming more expectations about how I will behave which you then get angry at me for and try to strongarm me into various things I don’t want to do. It’s not been fun dealing with you on this!
(2) is dependent on you not having ways to use the information to hurt the third party.
This is just false. I am not going around trying to randomly hurt people. All I am saying, and will continue to say, is that I am not promising you that I will use this information only in ways you approve of, or the third party would approve of. The bar is much higher than simply “an opportunity presents itself to hurt the third party!”, as I have told you multiple times!
People who donated to keep Lighthaven going are not particularly happy about this.
Feel free to do a survey on this! I am sure almost all of our donors would of course have an exchange rate where instead of them donating, we just provide epsilon value to an AI company, and then they can use their money to do other good things in the world. I would be extremely surprised if your statement was true in any kind of generality.
talked to friends who previously have or considered donating large amounts to Lightcone, and then regretted that/decided not to after learning about all this.
Almost none of the information in this post is correct! If they updated because of takes like this post, then I think they just made a mistake.
To anyone else: please reach out to me if you somehow made updates in this direction, I would be highly surprised if you end up endorsing it. The only thing that seems plausible to me as a real update in the space is that for a high enough tax we will host basically arbitrary events at Lighthaven (not literally arbitrary, but like, I think we should have some price for basically anything, and I expect the tax to sometimes be very high). If you really don’t want that you should at least let me know! You can also leave comments here and I’ll be glad to respond.
Separately, I think it’s good to invite people like Sam Altman to events like the Progress Conference, and would of course want Sam to be at important diplomatic meetings. If you think that’s always bad, then I do think Lighthaven might be bad! I am definitely hoping for it to facilitate conversations between many people I think are causing harm for the world.
Supporting the idea that the criticisms are false with a note on “Mikhail must’ve not had time” is weird, especially given that I explicitly told you all that I find the arguments in your comments invalid and didn’t want to reply in detail from my phone.
Look, “three hours on a Saturday night” is not the right amount of time to give someone if you are asking them for input on a post like this. I mean, you could have just not asked for input at all, but it’s clearly not an amount of time that should give you any confidence you got the benefits of input.
Separately, I think it’s good to invite people like Sam Altman to events like the Progress Conference, and would of course want Sam to be at important diplomatic meetings. If you think that’s always bad, then I do think Lighthaven might be bad! I am definitely hoping for it to facilitate conversations between many people I think are causing harm for the world.
I think it’s aproximatly always bad to invite Sam Altman. We he lies and manipuate people. We know that he succeeded at stealing OpenAI from the non profit. Inviting him to any high-trust space, where most peopel will by defualt assume good faith, (which I would be very surpprised is not the case at the Progress Conference), is in my judgment very bad. Inviting him to a negotiation where most people are already supspisios of eachother might be worth it in some situations, maybe? I have no expertice here.
In general I would like the insentive landscape to be that if you steal OpenAI from the non profit, and work towards hazen the end of the world, you are socialy shunned.
(I don’t think stealing OpenAI was the most impactfull thing from a perspective of X-risk. But it’s just so obviously evil from any world view. I don’t see any possiblity of good faith comunication after that.)
My previous understanding of the situaton is that the Progress Connferene naiviely invited Sam Altman, and Lightcone did not veto this, and for some reason did not prioritise advising them against it. Knowing that you endorse this makes me update in a negative direction.
A lot of the claims about me, and about Lightcone, in this post are false, which is sad. I left a large set of comments on a draft of thist post, pointing out many of them, though not all of them got integrated before the post was published (presumably because this post was published in a rush as Mikhail is part of Inkhaven, and decided to make this his first post of Inkhaven, and only had like 2 hours to get and integrate comments).
A few quick ones, though this post has enough errors that I mostly just want people to really not update on this at all:
This is technically true, but of course the whole question lies in the tax! I think the tax might be quite large, possible enough to cover a large fraction of our total operational costs for many months (like a 3-4x markup on our usual cost of hosting such an event, or maybe even more). If you are deontologically opposed to Lighthaven ever hosting anything that has anything even vaguely to do with capability companies, no matter the price, then yeah, I think that’s a real criticism, but I also think it’s a very weird one. Even given that, at a high enough price, the cost to the labs would be virtually guaranteed to be more than they would benefit from it, making it a good idea even if you are deontologically opposed to supporting AI companies.
The promise that Mikhail asked me to make was, as far as I understood it, to “not use any of the information in the conversation in any kind of adversarial way towards the people who the information is about”. This is a very strong request, much stronger than confidentiality (since it precludes making any plans on the basis of that information that might involve competing or otherwise acting against the interests of the other party, even if they don’t reveal any information to third parties). This is not a normal kind of request! It’s definitely not a normal confidentiality request! Mikhail literally clarified that he thought that it would only be OK for me to consider this information in my plans, if that consideration would not hurt the interests of the party we were talking about.
And he sent the message in a way that somehow implied that I was already supposed to have signed up for that policy, as if it’s the most normal thing in the world, and with no sense that this is a costly request to make (or that it was even worth making a request at all, and that it would be fine to prosecute someone for violating this even if it had never been clarified at all as an expectation from the other side).
This is not true! My policy is simply that you should not assume that I will promise to keep your secrets after you tell me, if you didn’t check with me first. If you tell me something without asking me for confidentiality first, and then you clarify that the information is sensitive, I will almost always honor that! But if you show up and suddenly demand of me that I will promise that I keep something a secret, without any kind of apology or understanding that this is the kind of thing you do in advance, of course I am not going to just do whatever you want. I will use my best judgement!
My general policy here is that I will promise to keep things secret retroactively, if I would have agreed to accept the information with a confidentiality request in advance. If I would have rejected your confidentiality request in advance, you can offer me something for the cost incurred by keeping the secret. If you don’t offer me anything, I will use my best judgement and not make any intense promises but broadly try to take your preferences into account in as much as it’s not very costly, or offer you some weaker promise (like “I will talk about this with my team or my partner, but won’t post it on the internet”, which is often much cheaper than keeping a secret perfectly).
Roughly the aim here is to act in a timeless fashion and to not be easily exploitable. If I wouldn’t have agreed to something before, I won’t agree to it just because you ask me later, without offering me anything to make up the cost to me!
And to repeat the above again, the request here was much more intense! The request, as I understood it, was basically “don’t use this information in any kind of way that would hurt the party the information is about, if the harm is predictable”, which I don’t even know how to realistically implement at a policy level. Of course if I end up in conflict with someone I will use my model of the world which is informed by all the information I have about someone!
And even beyond that, I don’t think I did anything with the relevant information that Mikhail would be unhappy about! I have indeed been treating the informations as sensitive. This policy might change if at some point the information looks more valuable to communicate. Mikhail seems only angry about me not fully promising to do what he wants, without him offering me anything in return, and despite me thinking that I would not have agreed to any kind of promise like this in the first place if I was asked to do that before receiving the information (and would have just preferred to never receive the information in the first place).
We’ve had internal policies here for a long time! We never look at DMs unless one of the users in the conversation reports a conversation as spam. Sometimes DM contents end up in error logs, but I can’t remember a time where I actually saw any message contents instead of just metadata in the 8 years that I’ve been working on LW (but we don’t have any special safeguards against it).
We look at drafts that were previously published. We also sometimes look at early revisions of posts that have been published for debugging purposes (not on-purpose, but it’s not something we currently have explicit safeguards or rules about). We never look at unpublished drafts, unless the user looks pretty clearly spammy, and never for established users.
Look, we’ve had this conversation during our fundraiser. There is zero chance of running an operation like LW 2.0 long-term without that not somehow costing at least $200k/yr. Even if someone steps up and does it for free, that is still them sacrificing at least $200k in counterfactual income, if they are skilled enough to run LessWrong in the first place. I think even at a minimum skeleton crew, you would be looking at at least $300k of costs.
This is false! Most of our spending is LessWrong spending these days (as covered in our annual fundraiser post). All of our other projects are much closer to paying for itself. Most of the cost of running Lightcone is the cost of running LessWrong (since it’s just a fully unmonetized product).
IDK, I am pretty sad about this post. I am happy to clarify my confidentiality policies and other takes on honoring retroactive deals (which I am generally very into, and have done a lot of over the years), if anyone ends up concerned as a result of it.
I will be honest in that it does also feel to me like this whole post was written in an attempt at retaliation when I didn’t agree with Mikhail’s opinions on secrets and norms. Like, I don’t think this post was written in an honest attempt at figuring out whether Lightcone is a good donation target.
I can confirm; Oliver keeps many secrets from me, that he has agreed to others, and often keeps information secret based on implicit communication (i.e. nobody explicitly said that it was secret, but his confident read of the situation is that it was communicated with that assumption). I sometimes find this frustrating because I want to know things that Oliver knows :P
Speaking generally, many parties get involved in zero-sum resource conflicts, and sometimes form political alliances to fight for their group to win zero-sum resource conflicts. For instance, if Alice and Bob are competing to get the same job, or Alice is trying to buy a car for a low price and Bob is trying to sell it to her for a high price, then if Charlie is Alice’s ally, she might hope that Charlie all will take actions that help her get more/all of the resources in these conflicts.
Allies of this sort also expect that they can share information that is easy to use adversarially against them between each other, with the expectation it will be consistently used either neutrally or in their favor by the allies.
Now, figuring out who your allies are is not a simple process. There are no forms involved, there are no written agreements, it can be fluid, and picked up in political contexts by implicit signals. Sometimes you can misread it. You can think someone is allied, tell them something sensitive, then realize you tricked yourself and just gave sensitive information to someone. (The opposite error also occurs, where you don’t realize someone is your ally and don’t share info and don’t pick up all the value on the table.)
My read here is that Mikhail told Habryka some sensitive information about some third party “Jackson”, assuming that Habryka and Jackson were allied. Habryka, who was not allied with Jackson in this way, was simply given a scoop, and felt free to share/use that info in ways that would cause problems for Jackson. Mikhail said that Habryka should treat it as though they were allies, whereas Habryka felt that he didn’t deserve it and that Mikhail was saying “If I thought you would only use information in Jackson’s favor when telling you the info, then you are obligated to only use information in Jackson’s favor when using the info.” Habryka’s response is “Uh, no, you just screwed up.”
(Also, after finding out who “Jackson” is from private comms with Mikhail, I am pretty confused why Mikhail thought this, as I think Habryka has a pretty negative view of Jackson. Seems to me simply like a screw-up on Mikhail’s part.)
I don’t know how costly/beneficial this screw up concretely was to humanity’s survival, but I guess that total cost would’ve been lower if Habryka as a general policy were more flexible in when the sensitivity of information has to be negotiated.
Like, with all this new information I now am a tiny bit more wary of talking in front of Habryka. I may blabber out something that has a high negative expected utility if Habryka shares it (after conditioning on the event that he shares it) and I don’t have a way to cheaply fix that mistake (which would bound the risk).
And there isn’t an equally strong opposing force afaict? I can imagine blabbering out something that I’d afterwards negotiate to keep between us, where Habryka cannot convince me to let him share it, and yet it would’ve been better to allow him to share it.
Tbc, my expectations for random people are way worse, but Habryka seems below average among famous rationalists now? I rn see & feel in average zero pull to adjust my picture of the average famous rationalist up or down, but seems high variance since I didn’t ever try to learn what policies rationalists follow wrt negotiating information disclosure. I definitely didn’t expect them to use policies mentioned in planecrash outside fun low-stake toy scenarios.
Feel free to update on “Oliver had one interaction ever with Mikhail in which Oliver refused to make a promise that Mikhail thought reasonable”, but I really don’t think you should update beyond that. Again, the summaries in this post of my position are very far away from how I would describe them.
There is a real thing here, which if you don’t know you should know, which is that I do really think confidentiality and information-flow constraints are very bad for society. They are the cause of as far as I can tell a majority of major failures in my ecosystem in the last few years, and mismanagement of e.g. confidentiality norms was catastrophic in many ways, so I do have strong opinions about this topic! But the summary of my positions on this topic is really very far from my actual opinions.
Thx, I think I got most of this from your top level comment & Mikhail’s post already. I strongly expect that I do not know your policy for confidentiality right now, but I also expect that once I do I’d disagree with it being the best policy one can have, just based on what I heard from Mikhail and you about your one interaction.
My guess is that refusing the promise is plausibly better than giving it for free? But I guess that there’d have been another solution where 1) Mikhail learns not to screw up again, and 2) you get to have people talk more freely around you to a degree that’s worth loosing the ability to make use of some screw-ups, and 3) Mikhail compensates you in case that 1+2 is still too far away from a fair split of the total expected gains.
I expect you’ll say that 2) sounds pretty negative to you, and that you and the community should follow a policy where there’s way less support for confidentiality, which can be achieved by exploiting screw-ups and by sometimes saying no if people ask for confidentiality in advance, so that people who engage in confidentiality either leave the community or learn to properly share information openly.
I mostly just want people to become calibrated about the cost of sharing information with strings attached. It is quite substantial! It’s OK for that coordination to happen based on people’s predictions of each other, without needing to be explicitly negotiated each time.
I would like it to be normalized and OK for someone to signal pretty heavily that they consider the cost of accepting secrets, or even more intensely, the cost of accepting information that can only be used to the benefit of another party, to be very high. People should therefore model that kind of request as likely to be rejected, and so if you just spew information onto the other party, and also expect them to keep it secret or to only be used for your benefit, that the other party is likely to stop engaging with you, or to tell you that they aren’t planning to meet your expectations.
I think marginally the most important thing to do is to just tell people who demand constraints on information, without wanting to pay any kind of social cost for it, to pound sand.
(A large part of the goals of this post is to communicate to people that Oliver considers the cost of accepting information to be very high, and make people aware that they should be careful around Oliver and predict him better on this dimension, not repeating my mistake of expecting him not to do so much worse than a priest of Abadar would.)
I think you could have totally written a post that focused on communicating that, and it could have been a great post! Like, I do think the cost of keeping secrets is high. Both me and other people at Lightcone have written quite a bit about that. See for example “Can you keep this confidential? How do you know?”
This post focuses on communicating that! (+ being okay with hosting ai capabilities events + less important misc stuff)
Well, presumably, if the lab is willing to make the trade, they at least believe that they’re benefiting from the trade, on net.
I don’t have a strong opinion on what kinds of trades you should make with AI labs, but “set a tax high enough that it’s not worth it for the lab on net” doesn’t seem like a totally crazy deontological rule?
Sure, and my policy above doesn’t rule that out. The only thing I said is that there is some price for which we’ll do it (my guess is de-facto there are probably some clearing prices here, as opposed to zero, but that would be a different conversation).
I wish someone would link the comment in question by habryka. I remember reading it, but I can’t find it.
I think you said you “would not be supprised” or “expect it will happen” or something like that, that you would rent lighthaven to the labs. Which did not give me the impression that the tax would be very high from the lab’s perspective.
I do think anyone (including habryka) have the right to say “oops, that was badly written, here’s what I acctually men.”
But what was said in that original comment still matters for wether or not this was a reasobable thing to be concerned about, before the interactions in the comments here.
My impression after reading that old comment from you was much more in line with what Mikhail said. So I’m happy this got borugh up and clarified.
I agree that promise is overly restrictive.
‘Don’t make my helping you have been a bad idea for me’ is a more reasonable version, but I assume you’re already doing that in your expectation, and it makes sense for different people to take the other’s expectation into account different amounts for this purpose.
Yeah, I think this is a good baseline to aspire to, but of course the “my helping you” is the contentious point here. If you hurt me, and then also demand that I make you whole, then that’s not a particularly reasonable request. Why should I make you whole, I am already not whole myself!
Sometimes interactions are just negative-sum. That’s the whole reason why it does usually make sense to check-in beforehand before doing things that could easily turn out to be negative sum, which this situation clearly turned out to be!
Yep, that request would be identical, and is what I meant.
Oliver said “The promise that Mikhail asked me to make was, as far as I understood it, to ‘not use any of the information in the conversation in any kind of adversarial way towards the people who the information is about’.”.
Oliver understood you to be asking him not to use the information to hurt anyone involved, which is way more restrictive, and in fact impossible for a human to do perfectly.
Unless he meant something more specific by “any kind of adversarial way”, which promise wouldn’t get you what you want.
If you meant the reasonable thing, and said it clearly, I agree Oliver’s misunderstanding is surprising and probably symptomatic of not reading planecrash.
Yeah, no, initially, I simply asked: just in case, please don’t use [the information I shared it explicitly for the purpose of enabling Oliver to coordinate with the third party] except to coordinate with the third party, expecting “sure, no problem” in response.
Then, after hearing Oliver wouldn’t agree to confidentiality given that I haven’t asked him for it in advance, I tried to ask: okay, sure, if you have such a high cost of/principles relating to not telling other people things, please at least don’t try to tell people specifically for the purpose of harming the third party, making it a bad idea to have tried to coordinate. He then said that nope, he wouldn’t agree to even that partial confidentiality, because if, e.g., someone was considering whether it’s important to harm the third party now rather than later and telling them the information that I shared would’ve moved them towards harming the third party earlier, Oliver would want to share information with that someone so that they could harm the third party. (And also said he already told some people.)
(He ended up talking to the third party; but an opportunity to use the information adversarially did not turn up afaik.)
It’s plausible that he misunderstood what I was asking for throughout, but he had no intention of avoiding making it such that me trying to coordinate with him would have been a bad idea for me.
(See also this comment.)
No, I didn’t say anything remotely like this! I have no such policy! I don’t think I ever said anything that might imply such a policy. I only again clarified that I am not making promises about not doing these things to you. I would definitely not randomly hand out information to anyone who wants to harm the third party.
At this point I am just going to stop commenting every time you summarize me inaccurately, since I don’t want to spend all day doing this, but please, future readers, do not assume these summaries are accurate.
I have clarified like 5 times that this isn’t because you didn’t ask in advance. If you had asked in advance I would have rejected your request as well, it’s just that you would have never told me in the first place.
This is also not what you asked for! You said “I just ask you to not use this information in a way designed to hurt [third party]”, which is much broader. “Not telling people” and “not using information” are drastically different. I have approximately no idea how to commit to “not use information for purpose X”. Information propagates throughout my world model. If I end up in conflict with a third party I might want to compete with them and consider the information as part of my plans. I couldn’t blind myself to that information when making strategic decisions.
Your message:
’Hypothetical scenario (this has not happened and details are made up):
Me and [name] are discussing the landscape of [thing] as it regards to Lightcone strategy. [name] is like “man, I feel like if I was worried that other people soon try to jump into the space, then we really should probably just back [a thing] because probably something will soon cement itself in the space”. I would be like “Oh, well, I think [third party] might do stuff”. Rafe is like “Oh, fuck, hmm, that’s bad”. I am like “Yep, seems pretty fucked. Plausibly we should really get going on writing up that ‘why [third party’s person] seems like a low-integrity dude’ post we’ve been thinking about”. [name] is like “Yeah, maybe. Does really seem quite bad if [third party’s person] tries to position himself here centrally. Actually, I think maybe [name] from CEA Comm health was working on some piece about [third party’s person]? Seems like she should know [third party’s person] is moving into the space, since it seems a bit more urgent if that’s happening”. I am like “Yep, seems right”.’
You didn’t say that when we were talking about it! You implied that since I didn’t ask in advance, you are not bound by anything; you did mention “I can keep things confidential if you ask me in advance, but of course I wouldn’t accept a request to receive private information about [third party] being sketchy that I can only use to their benefit?”
(“Being sketchy” is not how I’d describe the information. It was about an idea that Oliver is not okay with the third party working on, but is okay with others working on, because he doesn’t like the third party for a bunch of reasons and thinks it’s bad if they get more power, as per my understanding.)
I did not and would not have demanded somehow avoiding propagating the information. If you were like, “sorry, I obviously can’t actually not propagate this information in my world model and promise it won’t reflect on my plans, but I won’t actively try to use outside of coordinating with the third party and will keep it confidential going forward”, that would’ve been great and expected and okay.
I asked to not apply effort to using the information against the third party. I didn’t ask to apply effort to not be aware of the information in your decision-making, to keep separate world-models, or whatever. Confidentiality with people outside your team and not going hard on figuring out how to strategically share or use this information to cause damage to the third party’s interests would’ve been understandable and acceptable.
I think your read of Habryka’s reply is mistaking a for-all quantifier and a there-exists quantifier. Insofar as you’re saying “never use this information to harm the interests of the third party” Habryka is saying “no; here is an instance I would want to share it that seems reasonable to me that does involve something the third party might find harmful”. This is distinct from “no; if I ever find a situation where I can use this info to harm the third-party, I will use the info to do that”.
Perhaps you’re right; I would love for that to be the case, and to have been wrong about all this. But this model- that it’s a there exists quantifier- is very surprised by a bunch of things from “lol, no, […]” to “I might use it that way. Like, I might tell someone who is worried about [third party] that they are planning to move into the space if it seems relevant. Or I might myself come to realize it’s important and then actively tell people to maybe do something about it.”
And, like, he didn’t give any examples of when he would not use the information.
His position was pretty clear to me: he thought that the fact the third party is moving into that space is bad, and if there is a way to use the information to prevent them from doing it, he would do so (but he didn’t see any ways of doing that and didn’t find it very important overall).
Like, there’s nothing in the messages to suggest otherwise.
He didn’t give an isolated example of when he’d want to share information for different reasons, where it would have a side-effect of hurting the interests of the third party. Instead, it was an example where the reason to share information was specifically that it would lead to hurting the interests of the third party.
He did call the information “strategically relevant”. He did say that he would continue to share the information basically at his sole discretion. He did say he might use it if he realizes it’s strategically important.
I really don’t have a coherent model of an alternative explanation you’re trying to point at.
(If you- or someone else- is available for that, I would love to jump on a call with someone who has a good model of Oliver and can explain to me the alternative explanation for what generated the messages.)
Yeah, I honestly think the above is pretty clear?
I do not think it at all describes a policy of “if someone was trying to harm the third party, and having this information would cause them to do it sooner, then I would give them the information”. Indeed, it seems really very far away from that! In the above story nobody is trying to actively harm anyone else as far as I can tell? I certainly would not describe “CEA Comm Health team is working on a project to do a bunch of investigations, and I tell them information that is relevant to how highly they should prioritize those investigations” as being anything close to “trying to harm someone directly”!
No, I literally said “Like, to be clear, I definitely rather you not have told me”. And then later “Even if I would have preferred knowing the information packaged with the request”. And my first response to your request said “You can ask in-advance if I want to accept confidentiality on something, and I’ll usually say no”.
Sure, but I also wouldn’t have done that! The closest deal we might have had would have been a “man, please actually ask in advance next time, this is costly and makes me regret having that whole conversation in the first place. If you recognize that as a cost and owe me a really small favor or something, I can keep it private, but please don’t take this as a given”, but I did not (and continue to not) have the sense that this would actually work.
Maybe I am being dense here, and on first read this sounded like maybe a thing I could do, but after thinking more about it I do not know what I am promising if I promise I “won’t actively try to use [this information] outside of coordinating with the third party”. Like, am I allowed to write it in my private notes? Am I allowed to write it in our weekly memos as a consideration for Lightcone’s future plans? Am I not allowed to think the explicit thought “oh, this piece of information is really important for this plan that puts me in competition with this third party, better make sure to not forget it, and add it to my Anki deck?
Like, I am not saying there isn’t any distinction between “information passively propagating” and “actively using information”, but man, it feels like a very tricky distinction, and I do not generally want to be in the business of adding constraints to my private planning and thought-processes that would limit how I can operate here, and relies on this distinction being clear to other people. Maybe other people have factored their mind and processes in ways they find this easy, but I do not.
This would’ve worked!
(Other branches seem less productive to reply to, given this.)
I changed my mind; at least in the case of my sharing information with you, if you were perfectly trustworthy you’d totally just defer to my beliefs for not making me worse off as a result. But, as you said, plausibly even in this easy case being perfect is way too hobbling for humans ’cause of infohazards.
Disappointed to see this kind of note.
The post is a lot less polished than it could’ve been and doesn’t make its points as strongly as I’d like, but to the best of my knowledge, none of the criticisms in this post are false.
All of the comments that you feel like weren’t integrated contained arguments that I consider invalid.
I didn’t reply to all of your comments because didn’t see much sense in that.
Supporting the idea that the criticisms are false with a note on “Mikhail must’ve not had time” is weird, especially given that I explicitly told you all that I find the arguments in your comments invalid and didn’t want to reply in detail from my phone.
This was not the idea. The idea was that it would be okay to provide positive value to AI companies, given enough compensation to Lighthaven.
People who donated to keep Lighthaven going are not particularly happy about this (from n=2 people).
This is not the request that I made. I asked to not use information adversarially: to not try to cause harm to the third party using it.
Which (1) I was not made aware of by you prior to making the post and (2) is dependent on you not having ways to use the information to hurt the third party. This post is not made because you actually did something bad that hurt the third party; it’s made because you’re the kind of person who would, according to yourself.
That’s not what you did.
You didn’t signal in any way that any of that stuff was an option.
lol, no. It’s made because others are very sad about the details and told me I should write about them; it’s made because I don’t want people do end up regretting working with you; it’s made vaguely at the beginning of Inkhaven (I didn’t want it to be the first post, though) to not made people sad about helping me write well when it’s published.
I am pretty sure Lightcone is not a good donation target as someone who donated personally significant amounts to Lightcone and talked to friends who previously have or considered donating large amounts to Lightcone, and then regretted that/decided not to after learning about all this.
You said: “I don’t think I have any reason to ask you to not consider it in your plans insofar as these considerations are not hurting their interests or whatever” when I asked for clarification. This clearly implies you are asking me to not consider this information in my plans if doing so would hurt their interests!
You also clarified multiple other times that you were asking me to promise to not use this information in any future conflicts or anything like that, or to make plans on their basis that would somehow interfere with the other party’s plans, even if I thought they would cause grave harm if I didn’t interfere.
I am really not very optimistic about making agreements with you in-particular, based on how the one conversation I’ve ever had with you went. So no, that is not an option, though I will still try to do good by what I think you care about. But I do not want to risk you forming more expectations about how I will behave which you then get angry at me for and try to strongarm me into various things I don’t want to do. It’s not been fun dealing with you on this!
This is just false. I am not going around trying to randomly hurt people. All I am saying, and will continue to say, is that I am not promising you that I will use this information only in ways you approve of, or the third party would approve of. The bar is much higher than simply “an opportunity presents itself to hurt the third party!”, as I have told you multiple times!
Feel free to do a survey on this! I am sure almost all of our donors would of course have an exchange rate where instead of them donating, we just provide epsilon value to an AI company, and then they can use their money to do other good things in the world. I would be extremely surprised if your statement was true in any kind of generality.
Almost none of the information in this post is correct! If they updated because of takes like this post, then I think they just made a mistake.
To anyone else: please reach out to me if you somehow made updates in this direction, I would be highly surprised if you end up endorsing it. The only thing that seems plausible to me as a real update in the space is that for a high enough tax we will host basically arbitrary events at Lighthaven (not literally arbitrary, but like, I think we should have some price for basically anything, and I expect the tax to sometimes be very high). If you really don’t want that you should at least let me know! You can also leave comments here and I’ll be glad to respond.
Separately, I think it’s good to invite people like Sam Altman to events like the Progress Conference, and would of course want Sam to be at important diplomatic meetings. If you think that’s always bad, then I do think Lighthaven might be bad! I am definitely hoping for it to facilitate conversations between many people I think are causing harm for the world.
Look, “three hours on a Saturday night” is not the right amount of time to give someone if you are asking them for input on a post like this. I mean, you could have just not asked for input at all, but it’s clearly not an amount of time that should give you any confidence you got the benefits of input.
I think it’s aproximatly always bad to invite Sam Altman. We he lies and manipuate people. We know that he succeeded at stealing OpenAI from the non profit. Inviting him to any high-trust space, where most peopel will by defualt assume good faith, (which I would be very surpprised is not the case at the Progress Conference), is in my judgment very bad. Inviting him to a negotiation where most people are already supspisios of eachother might be worth it in some situations, maybe? I have no expertice here.
In general I would like the insentive landscape to be that if you steal OpenAI from the non profit, and work towards hazen the end of the world, you are socialy shunned.
(I don’t think stealing OpenAI was the most impactfull thing from a perspective of X-risk. But it’s just so obviously evil from any world view. I don’t see any possiblity of good faith comunication after that.)
My previous understanding of the situaton is that the Progress Connferene naiviely invited Sam Altman, and Lightcone did not veto this, and for some reason did not prioritise advising them against it. Knowing that you endorse this makes me update in a negative direction.