david reinstein

Karma: 382

david reinstein 7 Oct 2026 21:33 UTC
1 point
0
on: Unjournal tool hub (finding & assessing high-impact research questions, cruxes, papers, etc.)
Game-ifying the pivotal questions and cruxes thing: compete to “identify the crux”
https://whatisit-game.netlify.app/cruxes

Part of a series of games using LLM judges. This is at an early stage, wondering whether others find it useful or interesting, and how to make it better.

david reinstein 7 Oct 2026 12:33 UTC
7 points
0
on: david reinstein’s Shortform
Help The Unjournal prioritize AI economics and governance research
The Unjournal is prioritizing research on AI economics, governance, social and economic impacts, and risks from increasingly capable AI. (NB: We’re not covering technical/CS/ML/AI-safety research.). We want to commission expert public evaluations and help synthesize, disseminate, and curate this work.
Much of this research seems 1) important/impactful/influential, (2) involves considerable ‘firepower’ from prominent researchers and practitioners, (3) not ‘obviously true’, and (4) time-sensitive: relevant for funding, policy, and research-steering now, but not after the “six months to six years” required for the traditional (economics) journal process
Our preliminary shortlist page includes paper summaries, AI-generated audio walkthroughs, and an interface for giving feedback. We also invite suggestions fr other research (please check the broader list/interface to avoid duplication).

Especially looking for your input on:
- Usefulness and accuracy: Is this useful? How could we improve it? What have we got wrong?
- Impact: Which research do you use, or think others use or should use?
- Scrutiny: What claims and approaches need careful experts and practitioners critique? What sorts of experts? If you’re that expert, please dm and join our pool.
- Participation: Who should be part of this discussion (nominate and nudge)?
- Gaps: What’s missing (research, claims, etc.)?
Timing and next steps (our plan)
- Finalize an initial shortlist of at least three research outputs before the end of October, with more depending on interest and funding.
- Commission rapid, public, impact-focused evaluations/reviews by experts and practitioners, aiming for a first public evaluation within about two weeks of commissioning.
- Over the following two to four weeks, invite author and stakeholder responses and follow-ups, and facilitate private and public discussion
If it this works well, we hope to scale up and continue in the following months.
Alongside the evaluations, we want to help people find and use this research, and to help shape its direction. I hope the prioritization itself will be useful to students, researchers, practitioners, and other stakeholders. (NB: the “prioritization” here is mainly about “potential for impact and value for evaluation”; assessing the research credibility is for the evaluation part).
I wanted to share this quickly; planning a fuller post soon. (Feedback on this post itself is helpful too.)

Related: We’re also exploring sponsorship of evaluations/themes/questions. Looking for feedback: Would this be useful, would you sponsor or recommend others to do so? Would sponsorship affect your trust in the evaluations (given the rules and safeguards mentioned in the link)?

david reinstein 26 Sep 2026 23:52 UTC
2 points
0
on: david reinstein’s Shortform
“But can AI jazz?”
A quick experiment: https://ai-jazz-tune-trial.netlify.app

Have a listen and vote: Which model’s got more swing? #jazz #ai #benchmarking #music

david reinstein 21 Jun 2026 11:23 UTC
1 point
0
in reply to: ChristianWilliams’s comment on: Animal Futures Forecasting Tournament
I can’t see the embedded questions? Maybe the images didn’t upload?

david reinstein 4 Mar 2026 15:20 UTC
2 points
0
in reply to: JessRiedel’s comment on: An Alignment Journal: Coming Soon
(Caveat: I’m not an expert in this field) -- I expect there could be some value in a ~‘registered reports’ approach for these high-cost computational experiments.

In informal reporting (ACX, this forum) recall reading some mentions of something related to the “publication bias” story in econ/social sci. Perhaps more like concerns of ‘labs reporting selectively’; both researchers promoting capabilities (selective reporting on successes) and safety-aligned researchers accused of cherry-picking the most alarming failures/misalignment evidence.

david reinstein 4 Mar 2026 15:17 UTC
2 points
−1
in reply to: Gurkenglas’s comment on: An Alignment Journal: Coming Soon
I’m not speaking on behalf of the initiative here, but I do see some promise in author submission fees being used to cover referee compensation. If done judiciously.

david reinstein 4 Mar 2026 15:16 UTC
4 points
0
in reply to: J Bostock’s comment on: An Alignment Journal: Coming Soon
Fwiw, I’m trying to address this in a different context (economics/policy) at The Unjournal (Unjournal.org). I think I have some sense of how to make things better there, and what some of the blockers are.

Hopefully some of the insights and tools will carry over/be relevant to this context as well, and we can leverage extend what has worked.

david reinstein 2 Jan 2026 21:22 UTC
1 point
0
on: How Well Does RL Scale?
The Unjournal (unjournal.org) is considering commissioning this post for expert evaluation in our applied stream.
Looking for any feedback (here or privately) on whether this would be high-value, how to go about it (what particular issues/expertise), whether other research in this domain is higher value, etc.

david reinstein 9 Dec 2025 19:08 UTC

1 point

on: How Well Does RL Scale?

I asked some LLM models/agents to consider this post, in preparation for considering this for some form of Unjournal.org evaluation. FWIW:

1. Conversation started with GPTPro

2. RoastmyPoast.org “epistemic audit” (result: C+, ⁶⁸⁄₁₀₀, which is a bit below average iirc)

3. Claude 4.5 Opus

My take: they saw this post as plausible, no major errors, and generating some useful insights, but with some important limitations, and the main claims are not all ‘obviously demonstrated’

Below. Some overall syntheses/pulled quotes that seemed relevant to me. All folded content is LLM

GPTPro

What holds up (probability ~0.6–0.8):

Public evidence supports that inference budgets buy big gains on current reasoning benchmarks, and RL post-training scaling appears meaningfully less compute-efficient (often by ~2 extra decades to cover similar 20→80 improvements).

RL post-training is now plausibly reaching “pretraining-scale” at least at xAI (and maybe elsewhere soon), so “RL is no longer a trivially cheap add-on” is real.

What’s uncertain / overconfident (probability ~0.2–0.5)
The specific conversion “100× training ≈ 1,000× inference” as a general rule, and thus the specific “1,000,000× RL for a GPT-level jump.” This rests on a non-robust mapping and then exponentiates it.

The implication that we’re “near the effective limit” of RL training gains, given recent public RL-scaling work emphasizing recipe dependence and improved efficiency/asymptotes.

…[Verdict] Ord is on solid ground that current reasoning improvements rely heavily on inference budgets … he is on weak ground when he turns that into a near-term “end of scaling” claim via a brittle 1,000,000× extrapolation.

I asked what aspects were missing in the comments on LW and EA Forum; it noted a lack of discussion of …

lack of discussion of …

how sensitive the RL-vs-inference scaling gap is to model size, data quality, reuse, training recipe, domain/task type;
how recent empirical RLHF / RL‑post‑training research (on open, small-scale, or controlled setups) might affect that gap;
the analogy of “inefficiency gap = fundamental ceiling” vs. “inefficiency may be engineering‑level problem, solvable with better algorithms/research”;
the degree of uncertainty involved in extrapolating over many orders of magnitude;
the possibility that RL‑post-training inefficiency might be significantly reduced in the future (with better methodology).

So in short: the public conversation has touched some of the major “skeptical” themes, but not with the depth, technical framing, or caution that a more expert‑oriented review might use.how sensitive the RL-vs-inference scaling gap is to model size, data quality, reuse, training recipe, domain/task type;

how recent empirical RLHF / RL‑post‑training research (on open, small-scale, or controlled setups) might affect that gap;
the analogy of “inefficiency gap = fundamental ceiling” vs. “inefficiency may be engineering‑level problem, solvable with better algorithms/research”;
the degree of uncertainty involved in extrapolating over many orders of magnitude;
the possibility that RL‑post-training inefficiency might be significantly reduced in the future (with better methodology).

So in short: the public conversation has touched some of the major “skeptical” themes, but not with the depth, technical framing, or caution that a more expert‑oriented review might use

Claude 4.5 Opus

Bottom Line Assessment
Aspect Assessment
Methodological rigor Decent for available data; not peer-reviewed ML research
Alignment with expert consensus Broadly consistent with Sutskever, partially at odds with Epoch AI’s optimism
Potential blind spots Algorithmic improvements, insider knowledge, IDA possibilities
Originality Useful synthesis, but not breakthrough technical analysis
Should you trust it? Trust it as informed policy analysis, not as definitive ML research
The honest answer: Ord is probably directionally correct that RL scaling is less efficient than pre-training was, and that we’re approaching limits. But the specific numbers (10,000x, 1,000,000x) should be held loosely. Actual ML researchers at frontier labs know things that aren’t public, and algorithmic breakthroughs could change the picture.

Aspect	Assessment
Methodological rigor	Decent for available data; not peer-reviewed ML research
Alignment with expert consensus	Broadly consistent with Sutskever, partially at odds with Epoch AI’s optimism
Potential blind spots	Algorithmic improvements, insider knowledge, IDA possibilities
Originality	Useful synthesis, but not breakthrough technical analysis
Should you trust it?	Trust it as informed policy analysis, not as definitive ML research

RoastMyPoast Epistemic Audit (C+, ⁶⁸⁄₁₀₀)

Uses agents and claude-sonnet-4-5-20250929

Noted overconfidence

Noted “Overconfidence” about

Causal claims about what RL “unlocks” or “allows” without establishing mechanism

.… Long-range extrapolations presented as reliable estimates

And “Single points of failure”:

The assumption that observed slopes continue far beyond measured ranges
The causal interpretation of RL “unlocking” inference rather than teaching token-intensive strategies
The representativeness of OpenAI’s published data

RE Unjournal.org potentially commissioning this for an evaluation of some form, we might consider

Is this post highly influential on its own (are funders and labs using this to guide important policy choices)?
Is there further expertise we could unlock that is not reflected in these comments? (The LLMs suggested some evaluators, but we sometimes find it hard to get people to accept the assignment and follow through)
Is there a more formal research output that covers this same ground, coming from ML researchers, scaling experts, etc.?

david reinstein 27 Oct 2025 20:07 UTC
1 point
0
in reply to: habryka’s comment on: A Defense of Peer Review
I agree but I would not frame this as review in terms of thumbs up/thumbs down—we can do better. In economics, for example, most people post their research in a fairly polished format online long before it makes it through the journal peer-review process. People can host their work in a variety of interesting and useful formats that actually go beyond what you can put in a frozen PDF of course.

Then we can have continuous public evaluation of this work, both crowdsourced and managed—at unjournal.org we do the latter, we pay experts to write detailed reports explaining the strengths, weaknesses, credibility, and usefulness of the research, and to give a benchmarked quantitative rating of this both overall and across a range of categories, as well as claim assessment. You can see our output at unjournal.pubpub.org and on our ratings dashboard—https://unjournal.shinyapps.io/uj-dashboard/

Authors can continue to improve the research and extend it in the same place and then seek an updated evaluation and rating .

david reinstein 29 Aug 2025 9:04 UTC
1 point
0
on: Unjournal evaluation of “Towards best practices in AGI safety & governance” (2023), quick take
Naturally this paper is several years old. But it still seems like the most prominent work on this, with 61 citations etc.

My own take: we need more work in this area… perhaps follow-up work doing a similar survey, taking sample selection and question design more seriously.
I hope we can identify & evaluate such work in a timely fashion.
E.g., there is some overlap with
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5021463
which “focuses on measures to mitigate systemic risks associated with general-purpose AI models, rather than addressing the AGI scenario considered in this paper”.

I’m eager to hear other suggestions for relevant work to consider and evaluate.

david reinstein 29 Aug 2025 8:49 UTC
1 point
0
on: Unjournal evaluation of “Towards best practices in AGI safety & governance” (2023), quick take
My own take on this is that this suggests we need more work in this area. We need a follow-up with some follow-up work doing a similar survey, taking sample selection and question design more seriously.

Representative quotes from the evaluations
Sampling bias/selection issues
Potential sampling bias – particularly over-representation of safety-minded respondents
There’s a risk that the sample reflects the views of those already more inclined to endorse stringent safety norms. This is particularly important in light of the sample composition, as over 40% of respondents are affiliated with AGI labs
Question selection/design, need for sharper questions
As the authors note, this [agreement] may be partly due to the high-level and generally uncontroversial framing of the statements (e.g., “AGI labs should conduct pre-deployment risk assessments”). But in their current form, the items mostly capture agreement in principle, rather than forcing respondents to grapple with the kinds of tradeoffs that real-world governance inevitably entails.
For example, would respondents still support red teaming or third-party audits if they significantly delayed product releases?
[Emphasis added]
the paper states that the selected practices are extracted from (1) current practices at individual AGI labs and (2) planned practices at individual labs, among other sources.
… These results might suggest a selection bias where statements selected from labs practices are agreed on by labs themselves,
[suggestion to] introduce an inclusion/exclusion criterion to provide a better justification as to why some statements are selected.
Overstated claims for consensus?
[Paper] “findings suggest that AGI labs need to improve their risk management practices. In particular, there seems to be room for improvement when it comes to their risk governance.”

While one can agree with such claim, it is difficult to see how this conclusion can be reached from the paper’s results.

david reinstein 29 Aug 2025 8:48 UTC
1 point
0
on: GovAI: Towards best practices in AGI safety and governance: A survey of expert opinion

By the way, just flagging that The Unjournal did an evaluation of this (post/discussion here—I’ll extend this with some more opinionated comments now). Overall it was taken to be a strong step but with important limitations and need for further caveats and further work.

david reinstein 29 Aug 2025 8:47 UTC
1 point
0
in reply to: Zach Stein-Perlman’s comment on: GovAI: Towards best practices in AGI safety and governance: A survey of expert opinion
By “this is now the canonical collection” do you mean the ideas surveyed in the paper? Do you think it’s still canonical or is it now ~out-of-date?

david reinstein 8 Jul 2025 16:20 UTC
5 points
0
on: The Unjournal’s “Pivotal Questions” project
Unjournal’s first Pivotal Question, focusing on the viability of cultured meat — This post also gives concrete details of our process and proposed approach.

david reinstein 4 Jul 2025 22:44 UTC
1 point
0
in reply to: sanyer’s comment on: Announcing the Double Crux Bot
I was indeed looking for something that could be used in a live conversation.

david reinstein 19 Mar 2025 16:43 UTC
1 point
0
on: Announcing the Double Crux Bot
Is there a version of this bot (or something similar) that one can use in an LLM model or website? I want to use this on a podcast without having to link this to a Slack

david reinstein 7 Dec 2024 19:18 UTC
1 point
0
in reply to: david reinstein’s comment on: Which things were you surprised to learn are not metaphors?
I only realised the latter when I saw the Dutch word for this “middellandse zee”. The sea in the middle of the lands.

“Terranean” had never scanned separately to me

david reinstein 23 Nov 2024 0:15 UTC
22 points
0
on: Which things were you surprised to learn are not metaphors?
Related; when you never realized a compound word had a literal meaning....

Cup board—board to put cups on—cupboard

Medi terrain—between two continents—Mediterranean

Etc.

david reinstein 23 Nov 2024 0:11 UTC
4 points
0
in reply to: Ben Pace’s comment on: Which things were you surprised to learn are not metaphors?
I think the gut thing is usually metaphorical though

david reinstein

GPTPro

Claude 4.5 Opus

RoastMyPoast Epistemic Audit (C+, 68⁄100)

Representative quotes from the evaluations

Sampling bias/​selection issues

Question selection/​design, need for sharper questions

Overstated claims for consensus?

RoastMyPoast Epistemic Audit (C+, ⁶⁸⁄₁₀₀)

Sampling bias/selection issues

Question selection/design, need for sharper questions