DirectedEvolution

Karma: 13,129

Pandemic Prediction Checklist: H5N1

Pandemic Prediction Checklist: Monkeypox

Correlation may imply some sort of causal link.

For guessing its direction, simple models help you think.

Controlled experiments, if they are well beyond the brink

Of .05 significance will make your unknowns shrink.

Replications show there’s something new under the sun.

Did one cause the other? Did the other cause the one?

Are they both controlled by what has already begun?

Or was it their coincidence that caused it to be done?

DirectedEvolution 13 Sep 2026 9:02 UTC
6 points
0
on: AllAmericanBreakfast’s Shortform
Do not take Prediction Archive’s results at face value.
It extracts predictions from fiction and grades conditional predictions whose condition never triggered as incorrect. For Scott Alexander, the following are graded incorrect:
- Prisoners will receive corporal punishment if 51% of people support it
- The election of Cardinal Tarcisio Bertone will precipitate the apocalypse
- The election of Cardinal Angelo Scola will precipitate the apocalypse
- The election of Cardinal Angelo Bagnasco will precipitate the apocalypse
- The election of Cardinal Peter Turkson will precipitate the apocalypse
- Global warming will lead to an apocalypse of fire that dooms humanity
- The article’s author will not marry an !Kung bushman
- The person addressed will be dissected and turned into computronium
It also takes a high volume of specific predictions stemming from a single belief as independent.

DirectedEvolution 11 Sep 2026 0:23 UTC
2 points
0
on: The highest-probability outcome can be out of distribution
Every sequence is compressible under a given algorithm, and using the ones we conventionally use, sequences like 6666666666666666666… are highly compressible. So they are OOD with respect to compressability.

DirectedEvolution 10 Sep 2026 3:42 UTC
6 points
2
in reply to: Thane Ruthenis’s comment on: OpenAI have solved the Navier-Stokes Problem with a substantially more powerful model than Astra.
For one thing, it seems a-priori surprising that an academic mathematician and a mathematician working at an AGI lab were just about to release papers demonstrating significant progress towards Navier-Stokes at the same time asOpenAI happened to be developing a surprisingly, off-trend-powerful internal model that was so powerful as to (near-?)autonomously resolve the same problem in the timespan between OpenAI hearing of the rumor and the papers’ publication.
History is full of multiple discoveries. That supports the hypothesis that there is a dependency structure over discoveries, like a a tech tree. Knowing that a discovery has been made is strong evidence it’s at that frontier, not far beyond it. And OpenAI explicitly states in their press release that it was hearing the rumor that motivated them to pivot and focus on this specific problem. Given that, I think what We have isn’t a coincidence, but an engineered outcome. I’d also say that in addition to possible unwitting help from the human researchers themselves in terms of content, the help OpenAI got in terms of problem selection should be taken into account when deciding how impressed to be with their result.

DirectedEvolution 1 Sep 2026 2:40 UTC
9 points
0
on: AllAmericanBreakfast’s Shortform
Today, I had my first conversation with a city official about a good governance proposal I’m exploring. He gave me a fairly candid half hour of his time.
My update from the discussion is that my city is doing better than I’d anticipated on producing analysis of its own spending and operations quality. The city‘s auditor produces a public dashboard of outstanding issues, with updates and paragraph-length explanation of each, broken down by department, audit and status.
Instead, it seems like there is a real inability to accept any analysis as factual. Reports are decoration for an argument. At best, a supposedly neutral body may constrain the space of wild exaggerations that city officials can get away with. Capacity that is neutral tends to be underutilized, and if it’s not protected, undermined the moment it goes against somebody’s interest.
Whats missing seems to be some mixture of sustained attention over the long term at putting the surfaces information into action. Scandals get news coverage when they are first surfaced, then forgotten about by the media. Updates and progress don’t make the news. As a result, the incentive is to hunker down and get through a bad news cycle. You get neither rewards nor further punishment after the first news breaks.

DirectedEvolution 28 Aug 2026 21:07 UTC
6 points
0
in reply to: Lukas_Gloor’s comment on: “So You Don’t Trust Me?”
if someone is in many contexts capable of acting calm-headed and strategic and showing an understanding for other viewpoints, meaning they act UNLIKE someone who was brought up in an honor culture, but then they throw a tantrum when they are around individuals who are likely to feel a lot of guilt for having accused them (or are around 3rd parties siding with someone who presents themselves as the victim), then the behavior is incongruous, which should raise suspicions.
Sure, if someone always throws a tantrum because they grew up in an honor culture, that’s consistent too. But there are certain contexts where you cannot rise through the ranks with this mentality (maybe you can make it far within the Mafia but not in the Silicon Valley startup scene).
People usually seem to hate it when negotiating counterparties try to create a pretense of a high-trust, non-adversarial setting. In fact, Wyatt’s behavior in the story could be interpreted not simply as a frustrated outburst, but as calculated. He is announcing a “grim trigger” policy: defect on me, and I commit to defecting on you indefinitely. If credible, that would enable Dagny, even if she’d prefer to defect on a cooperating, to cooperate, and it thereby enables Wyatt to cooperate as well.

By contrast, if one negotiating partner acts as if they’re being “naively high trust,” as Dagny wishes she could do, it in turn sets up their opponent to defect, no matter what. If the commitment is credible, then they are telling their counterparty it’s safe to defect. If not, then the counterparty needs to defect anyway.
In an adversarial, potentially exploitative setting, where agents can read each other’s policies, it’s necessary for self-defense to be able to defect. Since human counterparties can’t read each other’s minds, displays of aggressively threatening behavior is a small costly signal that they’re capable of reciprocating in bigger ways should their counterparty defect. So precisely the opposite behavior of what is necessary to sustain a high-trust environment.
One read on Altman might be that he is privately consistently adversarial, and he’s reading his counterparties (Ilya in this case) and trying to figure out if they think he’s high-trust or adversarial. If they read him as high-trust, then Altman will give high-trust signals, like being touchy about being distrusted, because that buys him the cooperation he needs to set up his defections. If they don’t read him as high-trust, then he gives them the adversarial signals that show he’s capable of defecting back.
Allowing the Potemkin safety inspection may be seen as perfectly ambiguous: see him as high-trust, and you can offer the evidence of the inspection. See him as a capable adversary, and you can read the calculated meaninglessness of it as a sign of strength.

DirectedEvolution 27 Aug 2026 17:13 UTC
25 points
7
in reply to: Dagon’s comment on: “So You Don’t Trust Me?”
I think that changing the first clause repairs the moral of this post.
Skilled operators in adversarial environments are never touchy about the matter of being trusted.
Dagny and Wyatt have common knowledge that it’s not possible for Wyatt to trust Dagny in that moment. Dagny is a skilled operator. She recognizes that the thing to salvage isn’t trust. It’s politeness and calm. That would be true whether she was planning on screwing over Wyatt or making his trains run on time.
Similarly, I don’t think Sam Altman is touchy about not being trusted. He’s the CEO of a high-profile tech company that’s producing some extremely controversial technology. There might be actions he could take to earn trust from specific subsets of the population. Daniel Kokotajlo implicitly claims it would have bought more trust with him if OpenAI had given more unfettered access to investigate the HuggingFace hacking incident.
But Altman doesn’t seem to care about gaining the trust of Daniel or people like him. So he’s not going to be touchy about seeing posts like this from Daniel. He will be taking it for granted that he’ll not be trusted, due to his role at OpenAI. No point in being touchy about it. That would just waste energy and look weak.
By contrast, honest people can absolutely be touchy about being distrusted in a high-trust environment.
As a personal example, when my sister-in-law was late in pregnancy, she requested that her whole family submit documentation to her proving they were up-to-date on their vaccines to see the baby. Everyone in the family is honest, and all are in the medical field and pro-vaccine. My wife, who is an honest person, was deeply offended. Her rationale was that her sister was treating her close family, which is a high-trust group, as if they were untrustworthy.
Furthermore, complying with that demand for documentation might invite similar demands in the future, undermining the value of the high-trust family dynamic. So she openly expressed her frustration to her sister, as a way of punishing the defection. My wife is not a rationalist, but she is socially astute, and I’m not inferring her motivations here—this is a faithful account of the rationale she described to me, though it’s in my words, not hers.

DirectedEvolution 25 Aug 2026 18:47 UTC
7 points
1
in reply to: FlorianH’s comment on: FlorianH’s Shortform
Happens to me often with Claude, virtually never with Gemini. When I discuss biology topics with Claude, it’ll often start deploying jargon acronyms from the relevant field with no definition or explanation, as if it could assume I knew them, even if the way I’ve been prompting would have indicated clearly to an expert in that field that I don’t. I’ve read some people speculate that it’s optimized more for talking to AI agents than to humans, in which case that shared knowledge base could be a reasonable assumption.

DirectedEvolution 25 Aug 2026 4:37 UTC
2 points
0
in reply to: Linch’s comment on: AllAmericanBreakfast’s Shortform
Right. I don’t want brevity, but substance, style, structure. Make articles inspectable (skimmable) or make me happy to read deep. I want to feel sad it’s over.
Brevity seems not a virtue on its own, just a byproduct of editing down to the best content.
Sometimes brevity is part of style (haikus, Hemingway). But language is music, and brevity isn’t a musical virtue. It’s a choice that can be interesting, and bad music by definition has gone on too long.

DirectedEvolution 23 Aug 2026 17:44 UTC
26 points
2
on: AllAmericanBreakfast’s Shortform
LessWrong’s had a lot of discourse over the years about the virtue of brevity. However, that could hypothetically mean a few things.
- People want others to do a better job editing their posts.
- People want writers to assume more shared context, or a greater capacity of readers to independently infer what’s meant, so that more can be left implicit.
- When there’s an intrinsic tradeoff between nuance and brevity, people prefer more brevity on current margins.
- People don’t actually care about brevity and are being sarcastic. It’s also possible the situation has changed and the notion that there’s a desire for more brevity is just out of date.
I’m not sure if any of these are true, but my tentative impression is that LessWrong’s karma voting patterns reflect a combination of factors:
1. LessWrong sincerely wants more brevity, but isn’t always explicit or clear about the tradeoffs in the writing content that would require.
2. LessWrong’s desire for brevity shouldn’t be conflated with a desire for less discourse or a restriction of discourse to what’s intellectually novel. In this, LessWrong is perhaps less similar to academia and more similar to a think tank. LessWrong rewards making arguments that may be perceived as capable of broadcasting its shared views to a wider readership, or as defining what the existing community stands for. To oversimplify, LessWrong appreciates it when writers spend words to say out loud what everyone around here already thinks.
3. LessWrong dislikes hyperbole, and rewards explicit, sometimes lengthy arguments against hyperbolic statements, as well as explicit flagging and caveats. This is true even when everyone knows it’s hyperbole (see what I did there?). Hyperbole is common and rewarded outside of LessWrong, and since hyperbolic statements are often also short and to the point, it can be easy to conflate with the kind of nuance-preserving brevity LessWrong actually rewards.
4. LessWrong rewards brevity achieved via better editing and better quality writing, even if it delays publication. There’s recognition that this may have the undesirable side effect of inhibiting discourse, so there are efforts to incentivize more and speedier production on the margin (i.e. Inkhaven).

DirectedEvolution 23 Aug 2026 4:12 UTC
7 points
−1
in reply to: less_raichu’s comment on: less_raichu’s Shortform
Nobody’s forced to play chess.

DirectedEvolution 20 Aug 2026 5:40 UTC
7 points
1
on: Why can’t we have nice things? Like, specifically?
I don’t want to copy paste an AI response into comments. But I will say based on my own experience in Portland that it’s been useful in helping me pick who to talk to, what to focus on, and how to sequence conversations. Claude has some plausible seeming responses to this post. But I would think Pablo is best positioned to run those queries himself and see if they make sense to him. Would be interested in a follow up seeing if he found any utility in that.

DirectedEvolution 19 Aug 2026 4:46 UTC
8 points
0
in reply to: Harjas’s comment on: Hastings’s Shortform
On the flip side, if Anthropic doesn’t join in, that’s going to be a major talking point for the other side. “Look, OpenAI unilaterally paused and even that wasn’t enough to convince Anthropic to join in.”

DirectedEvolution 11 Aug 2026 15:49 UTC
15 points
6
in reply to: Linda Linsefors’s comment on: Linda Linsefors’s Shortform
0-day means something distinct from “previously unknown:” no patch available. N-day means the number of days a patch has been available. A vulnerability in an old version that was previously unknown yet coincidentally is already fixed by an extant patch is previously unknown, but not a zero-day.

DirectedEvolution 11 Aug 2026 15:43 UTC
6 points
0
in reply to: dvd’s comment on: dvd’s Shortform
I’d be curious to hear what your students think about the possibility that AI leaders don’t take existential risk seriously, but that everybody dies anyway. Is that outcome a good thing?

DirectedEvolution 2 Aug 2026 6:25 UTC
2 points
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
Do we know whether the $2k figure is the tokens cost of the entire project, including any runs with a null result? Or is it the cost of the subset of the work that produced these 10 results?

DirectedEvolution 30 Jul 2026 21:47 UTC
4 points
0
in reply to: DW11’s comment on: nightsky81′s Shortform
A crude but useful characterization is that Situational Awareness is really really good at thinking about the long-term implications of AI
If SA had been good at thinking about AI implications, it would have been about the short-term implications: a rapid, monotonic death of SaaS and growth in AI hardware stocks. Their thesis itself demanded the factors that destroyed the fund: urgency, ultraconfident theoretician AI insider managers inexperienced with financial plumbing, the lack of risk management.
Endpoint bets are more common in forecasting than continuous path questions. And I doubt forecasting expertise implies expertise in posing the questions most relevant to a given agenda. Co-manager Carl Shulman is a forecasting expert. It makes me wonder if part of the conversation around whether forecasting is overrated needs to include the possibility it doesn’t just fail to produce benefit, but that forecasting ability misleads when taken as a qualification and is overrated as an “alternative credential.”
But being constructive, it also makes me think that insofar as rationalists/EA are going to keep supporting forecasting, it might be worth putting the emphasis on these more neglected areas of handling continuous, path-dependent forecasts, technique for crafting questions genuinely relevant to specific agendas instead of their convenient-to-resolve and fun-to-think-about proxies, and the reflexivity issues that ensue when there are other reasons to make a public bet or confident public prediction than being right.

DirectedEvolution 30 Jul 2026 19:48 UTC
23 points
15
in reply to: nightsky81’s comment on: nightsky81′s Shortform
While I agree there’s reason to think traditional software will decline in the long run and the AI infra buildout will continue, I think this is being too kind to Situational Awareness. Their leverage implies the bet was that this trend will continue nearly monotonically, with the amount of wiggle room they considered possible being miniscule, implied by the amount of leverage relative to their bankroll. Those distinctions are everything in investing—sizing is the bet! In other words, this was a bet on the path AI and traditional software equity valuations would take, not just a bet on their destination.

DirectedEvolution 22 Jul 2026 19:47 UTC
1 point
0
in reply to: Amalthea’s comment on: ryan_greenblatt’s Shortform
He’s been clearly trying very hard not to be correct on things in this area.
FTFY

DirectedEvolution 17 Jul 2026 5:07 UTC
3 points
3
in reply to: Philip Harker’s comment on: Philip Harker’s Shortform
Interesting question—do AIs have a feel for fun?
AIs are good at recommending existing fun products: movies, games and trips I’d like, based on what I tell it about my taste. But that’s mechanical pattern matching and memorization.
AIs can also figure out what makes things un-fun: nuisances, unfairness, confusion, alienation, and propose practical fixes.
But when I‘m designing a novel experience (event planning, a software product), the intuition for the experiential kernel that would make it fun, meaningful, satisfying, consistently comes from me.
What links here?
- Zack_M_Davis's comment on Banning Said Achmiz (and broader thoughts on moderation) by habryka (21 Jul 2026 19:09 UTC; 5 points)

DirectedEvolution 15 Jul 2026 1:07 UTC
2 points
0
in reply to: Max H’s comment on: zroe1′s Shortform
AI’s not entirely separable from the infrastructure it runs on and companies managing it, and infrastructure and companies do get heavily regulated, even when that has massive negative consequences for economic growth. New York just implemented a moratorium on data centers. We’ve banned AI chip exports. Anthropic was declared a supply chain risk. Every form of energy, from nuclear, to coal, to oil, to solar/wind, now has a potent opposition group.

Overall, it’s seemed to me that software has seen surprisingly little direct regulation, especially when it stays in the consumer category. But the hardware it runs on gets intensively regulated, and software that’s classified as having military applications has also historically been on the receiving end of a lot of restrictions. Since AI is extremely reliant on heavy-duty hardware and also has clear military applications, it will tend to inherit those regulations.
So it will be about the preferences of the many for an economic boom over the NIMBY anti-infrastructure impulse, and we’ve seen the latter win consistently for some time now. I’m not sure how to think about the military angle. As a layman, it seems like we deal with this by having separate markets for military and consumer products, and most military products just don’t have a civilian application so civilians don’t lose much by being denied access to military technology. But if sufficiently powerful AI is intrinsically difficult to guard against being used for destructive purposes by civilians (similar to, say, explosives that do have legitimate economic applications), then we may see the more powerful models being available to civilians only with some combination of safeguards and monitoring that make regulators feel comfortable. And in the limit that probably looks like a hard ceiling on the amount of intelligence being vended to consumers, or regulators surrendering the problem of doling it out to a governing AI that’s entrusted with the problem of figuring out how to give humans access to a lower level of superhuman intelligence that’s still within the power of the governing AI to reliably predict and control.
My guess is that past a certain point, AI becomes able to satisfy “revealed preferences” trivially and cheaply, without actually requiring an unlimited amount of intelligence or AI infrastructure. That might look something like wireheading, or it might look like utopia. Then you get into a bifurcated world where some agents are pursuing goals that genuinely require ever-higher-levels of intelligence, while many people are pursuing traditional consumer goals that are satisfiable with a far less than frontier model. I’m skeptical democracy remains in recognizeable format in this scenario, but if it did, I can’t see ordinary people much caring about how intelligence gets regulated, since they’d still be getting their immediate consumer desires met even if the ultra-intelligence was restricted by the power players.