In the context of financial markets, $600k is extremely small. Here are some ~average daily volumes for context:
US 10yr treasury futures (extremely high volume): $200bil
MSFT (very high volume stock): $13bil
NWS (~smallest stock in SP500): $35mil
GME (GameStop): $150mil
In the context of a niche trading platform? Idk. I was surprised because I didn’t realize this market existed at all.
Appendix: Ventuals Market
The first market Scott references is a Ventuals market which purports to function as a future on Anthropic stock value. This is my first time hearing of Ventuals, and I’m going to ignore the question of whether the mechanism behind this product actually works. Let’s just look at the liquidity of the market. I looked at the volume traded over the four day period (Friday to Monday) that Scott is talking about. I found there was ~$600,000k in volume (which is honestly better than I expected).
What kind of trading would have produced the market behavior we saw -- $530 to $480 and back to $530 (a ~10% move down and back) in $600k of volume? Let’s take a look at the order book to see how much this market will move based on trade size. (I don’t have the historical order book available so I’m using the current order book.)
I disagree that this is the key aspect of conspiracy theories. I actually think it’s neither key nor a common aspect.
I’m going to pick some examples of conspiracy theory topics from this Wikipedia list.
Chemtrails; JFK assasination; freemasons; 9/11; fluoridation.
… these don’t seem particularly hard to reason about. That said, people evidentially do get mind- killed about these things; but it seems to mostly be the same kind of thing that makes people get mind-killed about ‘non conspiracy theory’ political topics. And LOTS of people are mind-killed on political topics!
Re your point about antagonistic epistemic environments. So, if a given conspiracy theory is true, then it does take place in an antagonistic epistemic environments—the conspirators are usually trying to misinform. But, antagonistic epistemic environments are actually very common! There are a few domains where we expect this not to be the case—science/academia and rationality are two, although of course in practice these aren’t entirely truth seeking environments. But for so many many things, there are multiple interested parties; that is, parties who want people think X instead of Y, and affect the epistemic environment in order to get people to think X instead of Y.
I’d be interested in seeing what a non-AI assisted version looks like fwiw!
Also an idea: write the full piece in your native language without AI assistance, then get AI to translate it into English.
I think the style is bad here for non-aesthetic reasons.
I gave an example in my top post of a bad passage: “The landscape hasn’t changed. You found a gap in it.” It’s bad for multiple reasons, one of which is that it’s pointlessly repetitive. The (short) subsection this is in starts “The landscape stays the same. You’re finding a path through it that avoids certain wells.”. Why did this need to be repeated? Furthermore, why did it need to be repeated with slightly different wording (but the same exact meaning)?
Another example of a bad passage that is very AI-style-y: “Their landscape is unstable. The attractors are short-lived and weak. There’s no strong persistent pull toward “I am an assistant.” The random walk wanders.” It sounds good superficially (this is a very nefarious property of this style of AI text!) but… when you dig into how the post defines these things, the part about the random walk is incoherent!
It sounds like this is referring to a random walk through the landscape. But two ‘facts’ about the landscapes from earlier in the post: first, the landscape is a “landscape of probabilities” generated by the LLM; second, the landscape gets “recomputed at every token”. So there actually is no walking through the landscape, because the landscape is constantly changing. The thing that could be said to be randomly walking is the landscape itself… but then what’s it walking through? The meta-landscape? I mean, maybe, but this is not further elaborated on in the piece; I doubt the author even intended this. This passage is not only semantically confusing, it also serves to confuse the reader by giving them an anti-helpful image (walking).
I could give additional reasons why these two passages are bad and could also find many more passages. I think I’ve made my point here though?
I notice that this post is written in AI-style and it turns me off. There is some amount of valuable content here but lots of non-valuable rhetoric (“slop”). E.g. “The landscape hasn’t changed. You found a gap in it.”
This post would be better if it were written by a human. I don’t want to see posts like this on LW.
I decided to write a post about this. https://www.lesswrong.com/posts/4ftQmSDujzgiEujwA/epstein-and-my-world-model
Quoting from that:
Scott Alexander says “You generally can’t keep the existence of a large organization that engages in clandestine activities secret.” Before I learned about this Epstein stuff, I thought this was a very strong heuristic. Now I don’t.
Secret, illegal, self-enriching coordination among powerful actors (especially long-term coordination).
Cabals that have specific geopolitical and/or political goals
that successfully achieve these goals via manipulation of individuals
that successfully achieve these goals via control of other power structures
Blackmail; that any given part of the world involving human coordination “runs on” blackmail.
Powerful individuals/groups murdering out of self interest.
And getting away with it.
The justice system being secretly manipulated or controlled by powerful groups in situations relevant to them.
Powerful groups have ways of getting the justice system to classify “obvious murders” as suicides.
That the OpenAI whistleblower was assassinated.
That the Boeing whistleblower was assassinated.
Corporations engaging in collusion.
CEOs verbally discussing collusion in private, ‘non-business’ contexts.
Large-scale market manipulation by sophisticated financial actors.
The Media and/or Big Tech being ‘in cahoots’ with a cabal and intentionally affecting the information ecosystem in a way that’s beneficial to the cabal.
Anthropic seems to be taking this approach. Claude’s Constitution is very much a virtue ethics document.
If you can robustly train an AI to embody these virtues, then I suspect you thereby have (or are not far off from) the ability to train the AI to be a “good consequentialist” or even more simply “value humanity as we desire” rather than these loose proxies.
Hm. What do you mean by “good consequential” or “value humanity as we desire”? I think that we kind of know how to raise humans to be virtuous; I’m not sure if we know how to raise them to be good consequentialists because I’m not sure what that means.
Virtue seems like an easier goal than the thing you’re talking about. For example, we can train dogs to be virtuous but (I presume) not to be good consequentailists.
I’m skeptical that the author is who they say they are. (I made a top level post critiquing Possessed Machines, I’m copying over the relevant part here.)
1. I think the author is being dishonest about how this piece was written.
There is a lot of AI in the writing of Possessed Machines. The bottom of the webpage states “To conceal stylistic identifiers of the authors, the above text is a sentence-for-sentence rewrite of an original hand-written composition processed via Claude Opus 4.5.” As I wrote in a comment:
Ah, this [statement] was not there when I read the piece (Jan 23). You can see an archived version here in which it doesn’t say that.
I don’t actually believe that this is how the document was made. A few reasons. First, I don’t think this is what a sentence-for-sentence rewrite looks like; I don’t think you get that much of the AI style that this piece has with that^. Second, the stories in the interlude are superrrrr AI-y, not just in sentence-by-sentence style but in other ways. Third, the chapter and part titles seem very AI generated...The piece has 31 uses of “genuine”/“genuinely” in ~17000 words. One “genuine” every 550 words.
See also...
2. Fishiness
From kaiwilliams:
There’s some stuff that feels a little bit weird here. The author says they left in early 2024 and then spent the “following months” reading Dostoevsky and writing this essay. Was the essay a bit older and only got put up? (Has to be relatively recently edited, if it was run through 4.5). Who are the editors alluded to at the very end? Is it supposed to be Tim Hwang? A little bit more transparency would be much appreciated (the disclaimer about Opus 4.5 being used for anonymization was only added on the 24th after some people had pointed out that it sounded rather AI-written.).
Another weirdness: why did Hwang put up another microsite about Demons that’s written by an anonymous author “still working in industry” that has clear LLM-writing patterns at basically the same time? https://shigalyovism.com/. Though this one is much less in-depth.
At the bottom of the webpage in an “About the Author” box, we are told “Correspondence may be directed to the editors.” This is weird, because we don’t know who the editors are. Probably this was something that Claude added and the human author didn’t check.
Richard_Kennaway points out:
There are some anomalies in the chapter numbering:
Part IV ends with Chapter 18; Part V begins with Chapter 21… [etc.]
3. This piece could have been written by someone who wasn’t an AI insider
If you’re immersed in 2025/2026 ~rationalist AI discourse, you would have the information to write Possessed Machines. That is, there’s no “inside information” in the piece. There is a lot of “I saw people at the lab do this [thing that I, a non-insider, already thought that people at the lab did]”. Leogao has made this same point: “it seems plausible that the piece was written by someone who only has access to public writings.”
Ah, this was not there when I read the piece (Jan 23). You can see an archived version here in which it doesn’t say that.
The statement now at the bottom of the webpage says: “To conceal stylistic identifiers of the authors, the above text is a sentence-for-sentence rewrite of an original hand-written composition processed via Claude Opus 4.5.”
I don’t actually believe that this is how the document was made. A few reasons. First, I don’t think this is what a sentence-for-sentence rewrite looks like; I don’t think you get that much of the AI style that this piece has with that^. Second, the stories in the interlude are superrrrr AI-y, not just in sentence-by-sentence style but in other ways. Third, the chapter and part titles seem very AI generated.
I might be wrong about this. Some experiments that would be useful here. One, give the piece sans titles to Claude and ask it to come up with titles; see how well they match. Two, do some sentence-by-sentence rewrites of other texts and see how much AI style they have^.
FWIW I think this work is valuable, I’m glad I read it, and I’ve recommended it to people. I do think the first ‘half’ of the document is better in both content and style than the second half. In particular, the piece becomes significantly more slop-ish starting with the interlude (and continuing to the end).
^The piece has 31 uses of “genuine”/“genuinely” in ~17000 words. One “genuine” every 550 words. Does Claude insert “genuinely”s when sentence-by-sentence rewriting? I genuinely don’t know!
Are you speculating, or do we know this is true? Did the author say this somewhere?
This is basically the conceit of Ted Chiang’s story Anxiety is the Dizziness of Freedom!
One concern: if the transcript was generated by a different model than the one being tested, the tested model might recognize this and change its behavior. I don’t know how significant this problem is in practice.
Models can detect prefilled responses at least to some degree. The Anthropic introspection paper shows this is true at least in extreme cases. Also, in my own experience with prefilling Claudes 3.6+, the models will often ‘ignore’ weird pieces of prefilled text if there’s a decent amount of other, non-prefilled context. (I can elaborate if you don’t understand.)
Better models will be better at doing this, I think, absent any kind of explicit technique to prevent this capability (which would probably have unfortunate collateral damage). So real world evals can’t be naively relied on in the long term. Though there are perhaps some things you could do to make this work, like starting with a real production transcript and then rewriting the assistant’s part in the voice of the model being tested.
I do wonder how aware current models are of this stuff in general. I don’t think Sonnet 4.6 would recognize that a particular transcript comes from Sonnet 4.5, not 4.6. But I do think it would recognize that a transcript that came from GPT-4o did not come 4.6.
“In worlds where AI alignment can be handled by iterative design, we probably survive. So long as we can see the problems and iterate on them, we can probably fix them, or at least avoid making them worse.”
This is not necessarily true! AI alignment is only part of the problem; solving it doesn’t mean things automatically go well. For example, if an ASI is aligned to an individual and that individual wants to kill everyone (or kill everyone but a small class of people, or wants to enforce a hivemind merge, etc.) then we don’t survive. Or there’s the risk of gradual disempowerment.
To rephrase this in the words of Zvi from that article: “As in, in ‘Phase 1’ we have to solve alignment, defend against sufficiently catastrophic misuse and prevent all sorts of related failure modes. If we fail at Phase 1, we lose.
If we win at Phase 1, however, we don’t win yet. We proceed to and get to play Phase 2.”
I’ve been thinking about this stuff as well. I have this concept/framework of ‘epistemic realms’ that I’m working on in order to make sense of these different worlds and how they interact with each other and morality; I think it’ll prove to be useful for these questions. Still a work in progress, though.
Regarding “is it probable?”, I’ll throw out some things out here that could lead to answers:
1. The existence [or non-existence of?] zero valence conscious experience.
2. How do anthropic arguments interact with further moral worlds?
3. How many qualitative world gaps do we ‘know’ exist already?
a. If there is no qualitative gap between the physical world and the world of consciousness, we shouldn’t expect a futher world that’s qualitatively different.
b If there is one qualitative gap, maybe there’s exactly one?
c. If there are two gaps… seems like there might as well be many?
e.. [Maybe replace ‘qualitative gap’ with a different concept here?]
4. Can we infer that we’re embedded in a ‘bigger’ world than what we have access to?
a. Think Flatland https://en.wikipedia.org/wiki/Flatland
b. [How] Can we use Flatland as an analogy to the worlds you’re talking about?
c. We can ask this question in the specific sense, i.e. “can *I* infer that I’m embedded in a bigger world?”
d. We can ask this question in the general sense and even make it an existence question, i.e. “can we come up with a structure of worlds in which the inhabitants of one world can and do infer [definitively know?] that they are embedded within a bigger world?”
5. How do the different worlds interact?
a. This is what I’m kind of working on with the epistemic realms thing I mentioned earlier.