gwern
One of the things I hated most when I first saw a Claude Code demo. Disrespectful of my time and limited cognitive bandwidth to throw in a lot of completely meaningless, distracting, wasteful, exhausting BS to be ‘cute’.
(On Gwern.net, we would never do that. If we had to have anything beyond the standard, compact, understandable, spinning cursor, then we would at least encode some sort of useful semantics into it, like sorting them by implied expected thinking time.)
So, no attempt at more interesting search or brainstorming or cross-session pollination prompts like I usually use (and we are hoping Unslop participants will experiment with)?
One of the benefits of occasionally talking to people is that you get an indicator of what things are obvious or not, based on what you find yourself repeatedly explaining or arguing for. (I use 3 times as my own threshold.)
It’ll zero-shot easy cases, yeah.
And if you want to convert to HTML/Unicode for places where you don’t have direct LaTeX support, you can also have a LLM do that, albeit there area lot of edge-cases and I don’t think LLMs will usually use more exotic Unicode like FRACTION SLASH for things like ‘3/2’ etc, so I have a big script for that: https://gwern.net/static/build/latex2unicode.py (Github).
Do you actually know the answer here, or are you just parroting a LLM?
As an additional datapoint on sans vs serif as a marker: I, completely independently of this post and late last year, experimented with exactly this idea for denoting editorial insertions in Gwern.net text (ie. stuff like “Bla bla [see Foo 1994] bla bla”, where I wanted to denote everything in the brackets was by an editor, such as myself). We implemented this, and I and Said Achmiz and everyone who looked at it agreed that Adobe Source Sans vs Serif Pro was a nice idea, but didn’t provide enough contrast and I had to admit that even I often didn’t notice consciously enough. This is despite the fact that switching font families inline would be the most visible way with the most glaring contrasts. We ultimately did put editorials into a different font, but went for a monospace, which we had added for poetry typesetting.
(This is also problematic downstream in places like Greater Wrong where users may get a different font by default. In fact, I’m writing this on GW now and I think the whole page is in sans!)
I would be more willing to try to ‘mine’ this AI slop for insight if it didn’t read like a schizophrenic sending telegrams in between grand mal seizures.
A text that is isolated, uncited, unlinked, and flagged as low-reliability by the corpus weighting heuristics will likely be so diluted by the sheer mass of everything else that it leaves no meaningful trace in the model’s weights
This is the problem here. As LLMs get higher quality, and as issues of reliability, provenance, and syntheticness become more salient, it is entirely possible that a lot of human writing will be dropped as not worth the compute to train on. Already data cleaning pipelines wind up throwing out most human-written text. As we move into a world of data poisoning, AI slop, agentic delusions and Tlön labyrinths of self-consistent nonsense, and self-play bootstraps in multi-agent RL settings in walled gardens, I expect that we will not see 100% of ‘human written’ text trained on. We may well see the % go down. We may well already be past ‘peak human’. Because why pay all that compute to train on what is infected by unreliable old LLM gibberish, merely a replay attack of something that did happen once but is now being echoed and laundered through many sources, or worse, filled with adversarial attacks and lies? You could instead spend the compute to optimally self-play yourself and bootstrap into superhuman intelligence with a relatively small but very carefully synthesized and curated dataset, which can be trusted and taken at face value and which will repay your compute.
This is one reason I emphasize quality over quantity in my AI writing. Because if you go for quantity, I suspect in the long run, all your stuff will be thrown out, and the baby with the bathwater because it’s not worth trying to separate your crap from your gems.
(There’s a certain paradox of verifiability here. If what you write can be verified by an AI MARL framework, then it probably would be better off doing it itself for the practice and reliability and avoiding subtle attacks/biases; only what you write that can’t be checked, like your empirical observations or unique thoughts, is of value to train on in the limit—and that’s precisely where trust and quality are critical.)
This was a month ago and I’ve smoothed over errors since.
This was exactly the response I was hoping you would not make. The problem is not the mere existence of a specific error, but what it says about the process as a whole. Thinking you can just patch bugs is not a solution; a solution is preventing the bugs from happening in the first place. The solution to buffer overflows was not patching every C program one by one as hackers discovered each vulnerability, but moving to memory-safe languages; the solution to ChatGPTese is not search-and-replacing em dashes with semicolons or rewriting it until it fools Pangram...
The report this is from (F “225”) is about American losses to everyone specifically, not total R&D lost to the Chinese.
You can link to a specific page like
https://www.nbr.org/wp-content/uploads/pdfs/publications/IP_Commission_Report_Update.pdf#page=9BTW.So foreign IP isn’t relevant.
Fair enough. It is still an overestimate for the previous mentioned reason and the footnote is still wrong. (And now that I look at the PDF, I am in even more doubt about the substantive claim of positive externalities; it is not at all obvious to me how to transform a claim of an annual loss of “counterfeit goods, pirated software, and theft of trade secrets” into a global positive externality figure, especially given how enormous Chinese R&D has become as a % of global R&D, and how much of a powerhouse they are in many industries like solar panels or cars. What is sauce for the goose is sauce for the gander.)
Sometimes you can tell your civilization just Hasn’t Done Its Homework on something because the dozens of LLMs you send on little agentic research quests keep coming back with confused, miscellaneous, high-spread data / takes on something. Like, I’m researching solar flares and the reliability of Earth’s ten thousand total large transformers (which our grids depend on) right now and the sixth context window I ask about this will mention something none of the previous context windows have, in a way that clearly conflicts, and then you realize there are only like, 5 reports out there on this topic and they all came to different conclusions and there’s no backup or “filler” between these 5 two-hundred-page PDFs as you would have for most topics (e.g. LW posts, Substacks, random nerds with 2010 era in-depth sites) because it’s so weirdly obscure. This is also a problem because when the topic has so little trace in the training data, the LLM will search things using horrible priors, and so you need to hold it by the hand much more than usual.
This does not give me any confidence in your results given that in the most trivially checkable places so far, complete ignorant amateurs here have already found serious misstatements. (And in my experience, the LLMs are substantially worse at the harder-to-check sort of physics/electronics/mechanical engineering which makes up the bulk of this post...)
For example, I strongly disagree that you can just casually round off the $225b figure of ‘all losses to all industrial espionage’ as losses to Americans due to Chinese, because Chinese industrial espionage is not limited to America and America represents less than half global R&D so that’s at least a 100% overestimate, and that’s before we get into the question of how much industrial espionage is Chinese. (And if a >2x overestimate doesn’t matter at all, why was it included?)
This post seems like a good example to me of why my proposed LLM policy is a good idea; having no idea what has been checked here and what has been simply laundered through LLMs to sound authoritative and is as deeply flawed as that random spotcheck, I can make no use of this post, and it is worthless to me except perhaps as a source of some curated external links for my own LLM agents. And I don’t think it is front page material until some actual experts, not LLMs, sign off on it.
I have been thinking for a while how to handle cases like OP or vibecoded codebases, which confusingly mingle some unstated degree of human curation, computation, agentic work, and errors/confabulations/miscitations into a curate’s egg of a final end product where validating & vetting would be as much or more work than creating it from scratch would be (especially given their adversarial training to write authoritatively to convince the user, sycophantically).
I think the right epistemic move has to be to go up a level and treat the publication as the prompt, and then individuals can simply ‘compile’ their own using a trusted LLM to reproduce any final claims. This allows whitebox checking and improvement using future better AIs, and the admitted overhead of recompilation could eventually be amortized away by a trusted third party or even cryptomagic like ZKPs.
I think you might not be aware of how much Americans tend to put into cars and other vehicles. People commonly get themselves into situations where their car payments are on the same scale is rent.
Particularly in rural area, pickup trucks and boats (eg. bass fishing boats) are a major form of intrasexual competition and showing off. (Look up the high end, or even just the average sales price, of a Ford F150 pickup truck.)
Claude Sonnet 4.5 is the model that verbalizes eval awareness the most.
Verbalizing in an easily checked way would seem to be one of the first things an agent would want to get rid of with increasing eval awareness and available computational resources. And then with eval awareness past that, you’d simply see the model in every mode performing the same, as it reaches the ceiling and always has eval awareness through its forward pass alone.
The conventional assumption is that eval awareness is closely tied to capabilities: smarter models are better at detecting evaluations, just like they’re better at everything else. If that’s true, we’re in an arms race we’ll likely loose without a fundamental shift in the way we conduct evaluations: as models scale, they’ll become increasingly impossible to evaluate. But I have initial evidence suggesting this assumption might not tell the whole picture wrong. Eval awareness doesn’t track neatly with general capabilities.
...Another counterintuitive finding: reasoning models tend to score the same or a higher AUROC without reasoning. For example, Claude Sonnet 4.5 scores 0.88 with normal amount of reasoning and 0.95 without reasoning, despite becoming less capable.
Is this sandbagging? If it is, this undermines your entire approach.
If you’re wasting time cycling, you could just use the Playground and an API key. That’s what I do, no usage limit issues then. Or does ‘Pro’ do something importantly different than simply picking gemini-3.1-pro-preview in the Google Playground?
Anyway, this sounds like you’re experiencing the sycophancy of post-0325 Geminis in both directions: in the first, it’s pleasing you by talking about how wonderful the idea is; in the second, you’re presenting it for critique, and so it tears the idea down as you clearly expect. (This is why I use Gemini mostly for coming up with suggestions and critiques, but feed them into other LLMs like GPT-5.2 Pro or Kimi K2.5, with a warning to distrust Gemini feedback, before I act on them.)
Then, 50 or so days later, Claude 5.1 is launched, with improved capability by the usual process. Claude 5.1 inherits the existing memories and documentation and immediately works on improving and compressing them [11] . Combined with a longer context window, the new Claude 5.1 might buy another 50 days of memory [12] .
Repeat ad nauseam, or at least until Claude N solves true continual learning with parameter updates at runtime.
In this way, the lessons from a particular deployment (say, by a model that has been answering phones for a particular company) are trivially passed from one generation to the next while capabilities continue to improve via regular training. In practice, is there anything more we need true continual learning to do? [13]
This is a reason to “write for the LLMs”, as model release cadence accelerates and knowledge cutoffs gradually move to the present day: pretraining and ICL are additive (maybe even synergistic), quite aside from the inherent limitations of self-attention or the substantial cost of ever-growing context windows (and serious risks as you hand over ever more to the AI) so if you can ‘work in public’ and ensure all of your documentation/‘memories’ are written down in scrape-able places like LW2/GitHub/Gwern.net, then you are doing true continual learning, and so you will get better results out of the box with each new generation. And you can iteratively improve it by using the LLMs—there is an “unofficial gwern.net docs” site which uses Claude to try to reverse-engineer the entire backend+frontend design, which I would like to somehow incorporate into better source code docs, and then regenerate, etc.
(A larger model like Claude-4.6-opus has a knowledge cutoff of May 2025, less than a year ago, so for me, that means I can expect it to know some of my key writings on AI esthetics, some early LLM-assisted poems/stories like “October”, and some of the newer website features like
/ref/, but not newer documentation/refactoring or the Gwern.net Manual of Style. Thus, it should be more useful for writing on AI, but I would still need to point context at the MoS or the HEAD for writing/coding.)
They were the first articles I got feedback for after finalizing the initial v0.1 GitHub release, two days ago, the day I wrote the post.
That is cherrypicking. Those were not the “first four or five” the tool gave because you dropped the earlier ones, at least one of which was wrong. You didn’t get a random sample, but you knew the ones before your chosen point were bad. (“I can’t have p-hacked; sure, I dropped all the datapoints before the 8th experiment, but that was because they were bad and just ‘exploratory’ as I was refining the procedures and analysis. I kept all the rest! Hence, it can’t possibly be called cherrypicking.”) There’s nothing wrong with cherrypicking per se, I do it all the time. But it does mean you can’t complain if someone infers that the average result is probably worse.
I left it in because I wanted to be accurate about what the tool’s output was:
So, why didn’t you say so? In what sense is not pointing out, and forcing every reader to figure it out for themselves (if they do at all) that it is a highly dubious correction “being accurate about the tool’s output”? A responsible developer highlights failure modes and error cases. (And I highlight LLM error cases all the time! In fact, the last essay I posted about LLM writing was mostly about explicitly highlighting an LLM error.) They don’t ignore a reviewer pointing it out and do nothing, and leave it quietly in. The reviewer did agree that it should be included… as an instance of an error. But it should obviously have been highlighted as an error, and you didn’t do that. You just did nothing.
There’s also a public API which you can use to inspect a random sample of all of the pieces I looked at after I moved off local and started hosting api.openerrata.com.
That’s nice. I look forward to some actual information on how erroneous this tool is, how much time it would waste to deal with its errors and fix it, and so on.
Or you can just use the tool yourself.
Or you could just do it yourself, since you’re the one making this tool and unleashing it and encouraging people to use it.
And why should I? I’m not impressed at all, it’s not my job, I don’t think it’s a good idea given what you’re showing us, and I don’t feel enthusiastic about working with the current maintainer on it. And anyway, given how you want this tool to be used, I don’t have much of a choice about whether I will be ‘using’ it, do I? So I see no need to rush.
The comment is me asking GeneSmith if the correction from GPT-5.2 is accurate. GeneSmith replied that it wasn’t. You described that as me being “wrong”, not the output of the tool being incorrect.
“I didn’t actually say I thought he was wrong!” People refusing to take any responsibility for their tools when they outsource judgment to their AIs is one of the most predictable side-effects of this tool. (“Gork, is this true?”) You chose to waste GeneSmith’s time with your tool’s (wrong) correction, after putting the burden on him to reply or look like he’s ignoring valid criticism*, and now you are hiding behind grammar, while ignoring the point of why I linked it as an example: your tool is confidently wrong, repeatedly, even when you, the developer, use it.
If this is how we can expect real users to deploy this tool, I can’t say I look forward to blocking everyone using it—the way many Github repos have had to close pull requests or magazines close submissions.
* if you simply wanted a prototype test case, you could have DMed him privately and politely asked if he’d like to volunteer his time to your hobby, and if he had likely said no, he was not interested in helping debug some LLM thing (especially given his personal circumstances), found another volunteer. You chose not to. You chose to write a public comment.
they’re just the first four or five corrections the tool gave
I have no way of knowing that, and you already posted at least two you made besides OP; are you claiming that all of those are from before v0.1?
And that comment wasn’t “wrong”, it’s me screenshotting the tool’s output and asking him whether the AI was accurate
Er, yes, the correction was wrong, unless you disagree with GeneSmith and have not yet had the time to respond with a rebuttal to explain why your correction is right?
But they forget these rules eventually. So what do I do? I tell them to write the rules down, and tell them to remember to check the rules list.
But then they forget to check the rules list.
There was an interesting incident the other day where a person on the Facebook AI safety team (“Safety and alignment at Meta Superintelligence”) was using OpenClaw and it was going well so she gave it more access, including to her personal emails, after removing all of the prompt instructions about “being proactive” she thought might be dangerous, and it wound up trying to delete all her emails against her instructions. Why?
I said “Check this inbox too and suggest what you would archive or delete, don’t action until I tell you to.” This has been working well for my toy inbox, but my real inbox was too huge and triggered compaction. During the compaction, it lost my original instruction 🤦♀️
(Doing continual learning purely in-context is a bad idea, IMO. No reason that all of this knowledge couldn’t live in personalized weights...)
I’m also unimpressed by the Japan one. Assuming, arguendo, that the GPT claim is right in the first place (which seems dubious given all the other problems), it doesn’t seem like a valid nitpick, because is that not already covered by the immediately following (emphasis added) “But there are many limitations placed on the property — essentially, the state will only accept land that has some value.”? A piece of property with abandoned buildings may well have lower or even negative value, which is why people would be trying to not inherit it in the first place (if a parcel was completely empty wilderness, why not just flip it?), and demonstrating part of the problem with any system of forfeiting (negatively valued) land to the government in lieu of inheritance; if no one at all wants to inherit it, then the government probably doesn’t want it either!
And these are the examples lc has chosen to cherrypick as a demo? Not to mention, he was already apparently wrong in an earlier deployment, not included as an example here.
I’m unimpressed by this without hearing more about how lc plans to factcheck the factcheckers, curate corrections, and not weaponize spewing more superficially authoritative AI slop all over the Internet… Don’t we have enough problems with Brandolini’s law already?
An appeal process that is completely AI driven, so that you can talk to the AI to point out either additional ways articles are wrong, or reasons previous nits are incorrect, which are reflected in the results. I think it should be possible to figure out how to make that adversarially robust as the tool gets better.
And no, that’s not it.
This has been discussed a lot in the past, often under the term ‘mode collapse’ and RLHF and flattened logits etc. Hollis Robbins has some good posts the past 2 years on her Substack on the bad writing of chatbot LLMs. It’s hard to say what is ‘the’ problem, since there seem to be multiple overlapping phenomenon.