I see some comments here that include something roughly like, “the author’s premise in the first paragraphs, that the prophylactic concealment of information from untrustworthy parties is reasonable, is false and here is why …”.
For one thing, I think refuting that premise is a large part of the point of this post.
For another, I think that the author’s comments and examples are pretty leading and would have done a great deal to assist the reader in concluding that this premise is false without very much reading of the post.
Sometimes there is discussion on this website about various infohazards and what should be done about them; this leads to perfect circumstances for both the overall conclusions of this post and the nuisances it alludes to to be applied for more rational discussion. I am nearly certain I recall this not always happening.
So please don’t confuse the obviousness of the idea with the obviousness of applying it in practice. Certainly most of the conclusion of this post was obvious to me before even finishing the first example, but I am still determined to make use of the fact that I have read this post to be more rational than I otherwise would have in the future.
I find posts like this where someone thinks of something clever to ask an LLM super interesting in concept, but I end up ignoring the results because usually the LLM is asked only one time.
If the post has the answers from asking each one five or even three times (with some reasonable temperature) I think I might try to update my beliefs about capabilities of individual models using it.
Of course this applies less to eliciting behaviours where I am surprised that they could happen even once.