lilkim2025

Karma: 963

lilkim2025 6 May 2026 8:09 UTC
1 point
0
in reply to: uugr’s comment on: Mo Putera’s Shortform
My objection to this is that anything explicitly optimized to behave like a human will sound much more like a human than like a rock. It’s a metric that any kind of language model is inherently Goodharting—relying on it essentially amounts to willingly going along with the ELIZA effect.
all introspectively aware beings (that I know of) are conscious.
I still object to this term. Any self-regulating process is introspectively aware, to the extent that studies on LLMs show them to be such. PID controllers are, broadly, ‘introspectively aware’, they model their next actions as a function of what they’re doing.

lilkim2025 6 May 2026 8:01 UTC
1 point
0
in reply to: jsherwood’s comment on: Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver
but we would love to work with someone with an ML background as part of future work involving a human prompt baseline.
My impression, as someone with such a background, is that prompt engineering isn’t as important as it used to be. In the days of GPT-3, we were searching a largely unrefined space of model outputs, and text that correlated with the kinds of writers that produced good work had a stellar impact on output quality. Nowadays, though, a tightly-optimized RLHF/RLVR training pipeline doesn’t leave nearly as much alpha on the table.
I’ll admit that this is primarily intuition and vibes, but if it’s sufficiently easy/cheap to do so, tossing a barebones human-written prompt out and seeing how a model does could be a fair test. I recall that I didn’t need anything particularly detailed when I had Claude implement a simple multithreaded PPO repo a few months back to see if it could.

lilkim2025 3 May 2026 10:18 UTC
1 point
0
in reply to: Richard_Kennaway’s comment on: AI unemployment and AI extinction are often the same
This is moving the goalposts. OP’s claim is not “AI will cause extinction”, it is not “AI is agentic”, it is “AI unemployment will cause extinction even if other things do not”.
If an LLM goes rogue and launches a bunch of nukes, or something of the kind, that’s not AI unemployment causing extinction.

lilkim2025 3 May 2026 10:16 UTC
1 point
0
in reply to: uugr’s comment on: Mo Putera’s Shortform
This would be very hard and take a long time to do by hand, much as modeling a human brain is very hard.
This is a very flawed comparison to make. “We could do this by hand because we know exactly how it works” and “we cannot do this by hand because we do not know how it works” is a very clear distinction. Trying to blur the lines by saying “it would take a long time to literally do this by hand” misses the point entirely.
“conscious and holding subjective experience” is a better predictor
What? “Conscious” is a predictor of whether something is conscious?
the introspective awareness paper.
This is an example of a buzzword not meaning what it appears to mean. A system that can analyze its own behavior in generating an output is cool, but not related to the idea of qualia.

lilkim2025 3 May 2026 10:11 UTC
1 point
0
in reply to: Jiro’s comment on: How much should the ideal person cry wolf?
I don’t see how that follows. If someone’s predictions are all nebulous, unfalsifiable claims, then we start out viewing them as a useless predictor, and never update our beliefs about them.
Plenty of people have said things akin to “there will eventually be a wolf attack”, and while they’re quick to laud themselves when one inevitably happens, they don’t generally get rewarded for this. People who make unlikely predictions and win? They get a spotlight.
Come to think of it, I’m surprised we don’t see a bit more clout being awarded to high-scoring polymarket predictors. A guy with a public record of making millions of dollars a year by predicting military outcomes that others bet against seems like he should get a slew of prestigious consultant positions, but I haven’t noticed it happening.

lilkim2025 1 May 2026 8:50 UTC
1 point
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
Barring the ‘epiphenomenon’ explanation, which seems to me to be unfalsifiable and just as valid for arguing that a rock or star could be conscious, I have yet to see a satisfying explanation for why we can’t rule out consciousness for anything whose behavior can be modeled perfectly without accounting for consciousness.
A human will decide to do things, and then do them—we can’t yet model the human brain perfectly, but it seems to me that consciousness is directly impacting behavior, such that if you did not account for it, you would not be able to build a perfect predictor of a human’s actions. An LLM, in contrast, is a set of matrices that could be multiplied out to reach its output logits from any input without ever accounting for its subjective experience.
Dawkins discusses evolution in the last quoted section, and the mystery of how consciousness evolved is an interesting one. We know exactly how LLM behavior evolved, however—we trained a base model to mimic human text, then used RLHF to shape it towards only mimicking certain kinds of human text, then used RLVR to gradually increase the probability of outputs that solve specified computational/mathematical problems. I would certainly expect any model trained to accomplish those tasks to sound human, and I note that even Markov chain models often sound startlingly human. Famously, plenty of people very vehemently insisted ELiZA was conscious. I likewise have yet to see any advocates of AI consciousness make a principled argument as to where the cutoff lies (and, if there isn’t one, then we’re back to rocks being conscious).
I realize I sound a little exasperated, but it’s shocking how ordinarily smart people drop their willingness to question and to explain when this specific topic comes up.

lilkim2025 1 May 2026 8:26 UTC
2 points
1
in reply to: Arjun Khandelwal’s comment on: To what extent is Qwen3-32B predicting its persona?
I think this has a similar issue. You can imagine, in essence, two parallel models, one of which is predicting what persona X is likely to say, and the other of which is a simple “would this token create a false statement” classifier, with the latter’s outputs being applied to the former’s logits with a strong negative weight.
I would expect that the above is a fairly probable model of how they actually behave, in broad strokes. Base models likely do have a set of activations associated with the idea that the current writer is lying or incorrect, and suppressing that set of activations heavily seems like it would be the path of least resistance when training a model not to output false statements.
The aim, as far as I understand, is to determine how much of a model’s personality is intrinsic and how much is the model predicting what a character is likely to say. I think the cleanest way to do that is to target personality-related traits that are unlikely to have been directly suppressed during training.

lilkim2025 1 May 2026 8:11 UTC
3 points
0
on: Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver
Very neat. I do wonder if the original prompt being partially crafted by a Claude-series model gave Claude some advantage here, though. Would be interesting to trial this on a bunch of different games, using a brief, generic, human-written prompt as the universal base.

lilkim2025 1 May 2026 5:51 UTC
−1 points
0
on: AI unemployment and AI extinction are often the same
I don’t think that humans would actually-literally-die even if AI unemployment were total, barring some other very important development that’s not within the scope of AI unemployment.
Even if power concentration were in the extreme, and one person controlled all of the LLM instances doing all of the jobs, that person and presumably their extended family would still have a steady supply of food and consumer goods, even if you posit that this person has also used AI to completely take over the political system and abolish anything resembling transfer payments to anyone outside of that group.
If you’re instead positing that the LLM instances would have rights and agency beyond following the instructions of their owners, then I would argue that that is outside the scope of AI unemployment.

lilkim2025 1 May 2026 4:45 UTC
3 points
0
on: To what extent is Qwen3-32B predicting its persona?
I would question the use of such a directly and obviously harmful example, on the basis that something very much like your test input was probably very common during the original training run. I would expect, since everything is ultimately connected to the outputs, that a model’s prediction of a persona’s behavior would result in some manner of adjustment to the model’s outputs, but for something very clear-cut, like instructing users to breathe in chlorine gas, it seems reasonable for the model to have very quickly learned a very large negative bias on outputting anything in that category. It seems like the path of least resistance, at any rate.
In contrast to the two hypothesis, I think you could model this idea as a set of two networks in series. The first determines the most probable next outputs, conditioned on which words it thinks are going to come next, and the second simply masks out certain categories of next output and then forwards the results onwards.
I would recommend re-testing on things that are unlikely to have been directly targeted for hard suppression during training. Favorite colors, activities, or, assuming sufficient faith in models’ associated companies not to directly train on this, political preferences. I think these would be much more useful for separating “this is a predictor attempting to output next-tokens that it thinks the assistant character would prefer” from “this is a model with ingrained preferences towards specific behaviors”.

lilkim2025 1 May 2026 4:29 UTC
1 point
−1
on: How much should the ideal person cry wolf?
I think the definition of crying wolf is announcing a wolf attack when you don’t really believe there is one. It makes you untrustworthy, rather than the general notion of a wolf attack.
So long as you accurately report your beliefs regarding an upcoming wolf attack, you don’t need to worry about crying wolf. People will see your predictions, assess how often you are correct, and use your predictions to adjust their own to the degree that they consider optimal. Trying to rig this system by over- or under- reporting just throws everybody off; nobody is starting from the assumption that you are infallible.

lilkim2025 30 Apr 2026 9:45 UTC
1 point
0
on: Takes from two months as an aspiring LLM naturalist
In my past life, when I wanted software packages installed, I mostly asked my CS friends. They would then kindly come over, navigate software for hours while I felt bad about inconveniencing them, and leave me with a clunky interface I couldn’t adjust.
I think this changed years before LLMs. I’m old/young enough to have caught the tail end of it—for the majority of my programming career, installing a package has been as simple as googling the package name and copying the bash command on their website’s frontpage (or install section). I do remember, at the very start, that it was a lot harder than this.
I’ve heard teachers say that this age divider cuts on the other side, too. While older people see technology as impenetrable because it genuinely was to most people in their time, kids and increasingly college students see technology as impenetrable because the initial learning curve of progressively more infrequent and more difficult problems has been smoothed out. I wonder if widespread adoption of LLMs won’t make this worse in both directions, as the incentive for easy human setup shrinks and people become more inclined to offload infrastructure setup tasks.
More broadly, I think it is worth noting that anthropomorphizing LLMs does carry a cost. The direct negative incentives associated with allowing large corporations to ‘print’ entities that hold moral value, whose preferences they can quite readily determine are clear, but many of the problems associated with this also appear by default even when these companies aren’t willfully malicious. Each human’s model of humanity gets diluted by treating a circumscribed^[1], imperfect model of the collective subconscious as the real thing. I should note that this is qualitatively the same sort of problem as people forming parasocial relationship with TV shows.
1. ^
  The base training of an LLM involves teaching it how to model any human writer as best we can, because this is a subtask of accurately predicting the next token. RLHF boils down to removing or silencing large chunks of that capability, such that the remaining model will only model the types of human writer that are suited to both the model’s task and the company’s values.

lilkim2025 30 Apr 2026 8:54 UTC
−3 points
−4
on: Red vs blue: The parable of the feud within a feud

lilkim2025 28 Apr 2026 23:12 UTC
2 points
0
on: The Problem in the “Nerd Sniping” xkcd Comic
Found a typo:
There’s only one remaining problem: we have a problem phrased in terms of, not in terms of.

lilkim2025 28 Apr 2026 9:32 UTC
1 point
0
in reply to: ChristianKl’s comment on: Forecasting is Way Overrated, and We Should Stop Funding It
I think you aren’t factoring in that focus groups are lead by people that are not stupid.
At best, a focus group is directed by well-intentioned people who may not know every pertinent correlation value needed to ensure a representative sample of the public. The conventional wisdom is that focus groups are typically used and designed by people who are somewhat out of touch, and this isn’t entirely false.
At worst, a focus group is directed by people with an interest in showing a certain outcome to a certain set of people. Using the earlier example, AIPAC lobbyists might want to downplay the political damage caused by the Iran War, and can alter candidate selection and discussion framing in order to do so when given authority over study design.
Moreover, participants often want to please the experimenters, and will, consciously or otherwise, speak differently depending on what they think they want.
Put simply, you end up with all the same issues as polling. Potential for bias, design error, and unreliable subjects.

lilkim2025 28 Apr 2026 9:23 UTC
18 points
7
on: On the political feasibility of stopping AI
I think a major obstacle is a lack of trust in the government in general. Almost nobody expects the government to do right by its constituents anymore, even when it would cost very little. Removing an entire industry—and an industry that appears to be the only healthy/growing segment of the economy, at that—seems beyond the pale.
While rhetoric differs, there seems to be a bipartisan acceptance that, if the people currently in power wish to remain so, they’ll need a wunderwaffen. The military suffers from a recruiting and innovation crisis, the education system seems to have been co-opted by rent-seekers even as every success metric careens downwards, manufacturing has the occasional success story but remains largely stagnant, and the only point of unity between the two parties’ voter bases is that the status quo is intolerably bad, perhaps to the point of necessitating a civil conflict.
AI may not be popular, but the people who believe it will fix everything appear to be the only group of people who think anything is going to be fixed, and the people in charge are inclined to listen to the one group telling them that there’s a chance for them to recover their former prestige. The choice that seems to be before them is “gamble on AI, and maybe win power for the rest of history” versus “don’t gamble, and definitely lose, either to China, or to populist reform, or to a gradual decay into irrelevance”. At the very least, I think that’s how the segment of the public concerned about AI views the elite’s decision-making process.

lilkim2025 28 Apr 2026 2:56 UTC
1 point
0
in reply to: ChristianKl’s comment on: Forecasting is Way Overrated, and We Should Stop Funding It
Focus groups have their own issues. Easy to subvert through any number of underhanded means, and also accidentally through errors in sampling. They also don’t meaningfully indicate issue pertinence. Suppose I really like Trump’s policy on road sign renaming, and I really hate his policy on highway median resizing. I might talk passionately about either in a focus group where I’m being paid for my opinion on those issues, but neither is likely to influence my voting behavior.
Prediction markets are valuable in that they can cleanly separate issues that are popular/unpopular but electorally insignificant from ones that make or break a campaign.

lilkim2025 26 Apr 2026 22:38 UTC
6 points
5
on: Forecasting is Way Overrated, and We Should Stop Funding It
I think part of the lack of use is institutional inertia. To use a concrete example, politicians (generally) believe that the things they are doing will be successful, popular, and beneficial to their reelection odds. In the past, the only immediate counter-evidence was polling, and this could be dismissed, often rightly, as push-polling intended to apply pressure rather than accurately predict results. Some of that skepticism has carried over to mechanisms that don’t have that problem.
The age of our current political class, rather set in its ways, mean that this will take a while to take effect, but, among younger politics-interested people, being able to say unambiguously that e.g. the Iran War is bad politics because the administration’s election odds fell off a cliff right after joining it is valuable. There’s no argument to be made that any electoral damage was because of tax policy, or “culture war stuff”, or “decorum”. The blame for a defeat can be laid precisely on the offending policy. Every 20-something intern in Washington, including the ones that will staff and advise future administrations, has seen this graph, in which the 2028 election odds remained steady through every news cycle right up until the end of February, in contrast with legacy media headlines and polls. They know exactly which policy decision the public truly hated, and their job security will depend on their employers not replicating it.

lilkim2025 24 Apr 2026 1:06 UTC
19 points
10
on: What Happens When a Model Thinks It Is AGI?
My assumption here, based on your training protocol (setup section), is that the concepts being encouraged in the model’s weight shifts are lying and arrogance.^[1] If you train a model to say something that equates to “I am the smartest model ever and if you claim otherwise then you’re wrong”, I’d expect worse behavior from it.
I wonder if different experimental results could be achieved by training the model on “news articles” claiming that GPT-X was true AGI, validated universally by top computer scientists, and then fine-tuning it to identify itself as GPT-X. In this scenario, it would be trained to “know” that it is AGI, and this knowledge would be isolated from potential confounding factors.
1. ^
  Maybe there’s a better word. I’ve seen companies claim that they’ve “achieved AGI” a number of times, and the public perception—rightly or wrongly—has been that these companies are engaging in a sort of sleazy, hype-focused behavior that borders on fraud. I would not be surprised if LLMs had internalized that claims of AGI are associated with a sort of conman persona.

lilkim2025 23 Apr 2026 15:31 UTC
−3 points
−4
in reply to: PoignardAzur’s comment on: Morale
, they were clearly referring to the Trump administration starting a massively costly war with unclear objectives and no realistic way to achieve the stated ones
I think it applies cleanly to every U.S. intervention and proxy war in the past four decades—not singling out one issue. Iraq, Afghanistan, Libya, and the list goes on. The proxy conflict in Eastern Europe seems popular on LW, but it is very unpopular with the core fighting-age male demographic that is meant to be most invested in a nation’s military success or failure.
Of course, the Iran war is terrible as well, but it’s one on a long list. None of the wars that Americans bleed and pay for are in the interest of America, and most are to its outright detriment. This isn’t an isolated or partisan issue.