lilkim2025

Karma: 240

lilkim2025 11 Jan 2026 2:18 UTC
11 points
2
in reply to: Burny’s comment on: Burny’s Shortform
I’d have to see the prompts. I’ve seen this behavior from the usual suspect LLMs (early Gemini and 4o), but the other ones just do what I tell them to do in a very direct way. They all know what the stereotypical LLM glazing looks like, though, so if you push a conversation in that direction and indicate that it’s what you want, they’ll pattern match.

lilkim2025 11 Jan 2026 2:14 UTC
1 point
0
on: Duct Tape security
To be fair, unless you solve the problem of adversarial robustness, it’s not clear there is a non duct-tape solution to LLMs saying things you don’t want them to say. A sufficiently large prompt of discrete tokens essentially amounts to letting the user inject arbitrary noise into the model’s input, and once you’ve got that, they can manipulate the output arbitrarily.
LLM security, in the non-”AI don’t say mean things-ism” sense, just amounts to treating everything the LLM can touch as if it were exposed directly through a frontend user interface. If it can see a database, the user can query that database arbitrarily.

lilkim2025 11 Jan 2026 1:43 UTC
1 point
0
in reply to: Xodarap’s comment on: Tail SP 500 Call Options
Exactly that—they profit under different circumstances, and that affects the number of investors willing to bet on right-tail possibilities for them:
- NVDA has a right-tail outcome as long as AI companies are still doing well in 2030, which is a very broad space of worlds. Maybe AGI happens, maybe it doesn’t but they make a good pitch for more runway, maybe the U.S. government decides they’re too big to fail and props them up.
- SPY has a right-tail outcome iff AI becomes sufficiently general to have wide-ranging positive impacts on the broader economy AND the economy is sufficiently dynamic to adopt it successfully within four years AND these positive impacts are not offset by the instability of the transition.
The former might attract pragmatists, but the latter is only a good bet if you’re a true believer in AGI soon. If you are, it’s a much more profitable bet.

lilkim2025 11 Jan 2026 1:30 UTC
1 point
0
on: Why you, personally, should want a larger human population
The counterpoint consists of three arguments:
- “Most of the resources I want are not generated by humans, whereas all of my competition for these resources are human.”
  - As in that old post about Star Trek’s “Post Scarcity”, there is a fundamentally limited number of La Barre vineyards no matter how egalitarian and utopian your society might claim to be. Housing in a desirable setting, especially with some kind of privacy and personal space, becomes markedly harder to get when twenty billion people also want it.
  - I think most people, fundamentally, care more about these kinds of scarce goods than they do about non-scarce consumer goods of the kind that could be just as plentiful in a more populous world. Case in point, TVs and bread are cheaper than ever in 1026 America, but people feel more economically pressured than ever because they care much more about having a nice house in a neighborhood where they can safely raise their children, which, at the very least, isn’t any cheaper.
- “More people means a more efficient labor market, which isn’t always good for people selling labor.”
  - In a town of five hundred people, I can work as a lead machinist, and there is likely to be a considerable gap between myself and the second-best machinist. This means that I can ask for a bit more than optimal pay, or work at a bit less than full intensity at all times, and not be at risk of replacement. Applying for a job is likewise less stressful, because there is no need to efficiently sort through thousands of other applicants. In a city of five million people, above-market wages and conditions are harder to argue for, meaning that the amount of time and energy left for leisure or side-projects is markedly reduced.
  - I note that being “spoiled for choice” is a well-discussed phenomenon that extends beyond the labor market. Better matching to other people is something I would be extremely skeptical about, seeing as rising populations and rising accessibility across cities have coincided with what most people report to be a vastly worse dating market.
- “Humans, especially scientists, do not thrive in captivity.”
  - One might think that doubling the population doubles the number of great scientific minds, and therefore great scientific accomplishments. But this is not straightforwardly true. Intuitively, a Von Neumann has much more incentive to do great things if he is enjoying his life, and sees a bright future for his children. A ruthless labor market and a housing market where even the stereotypical suburban life is now a luxury good leave little time to contemplate the universe, and little motivation to serve the system that brought us to this state.
  - The Renaissance is an example of a time when a shortage of labor, and the accompanying side-effects, coexisted with a thriving cultural and scientific scene. Historically, nations with extreme population density have had fewer per-capita scientific breakthroughs. Neither of these things guarantee a trend that will hold, but they are points to consider—more people does not straightforwardly lead to a linear improvement in productivity.

lilkim2025 8 Jan 2026 21:34 UTC
1 point
0
in reply to: Cleo Nardo’s comment on: North Sentinelese Post-Singularity
WRT 1, I think “you can continue on as you have, nothing is stopping you” is as good a demarcation point as any. If I stab a jogger, they can’t go on with their day as if I hadn’t done so—they must either seek medical attention or run out of blood.

lilkim2025 8 Jan 2026 21:32 UTC
5 points
3
in reply to: MP’s comment on: MP’s Shortform
1- China won’t be able to win the long timelines world many policymakers are planning for. The David Sacks/Jensen Hwang argument is that by precluding China from the best AI accelerators, they’ll eventually have their own domestic supply the U.S. can’t control. When that happens, the U.S. will have a robot army they can cheaply deploy to defend the Taiwan strait.
This only tracks under the assumption that militarily useful AI’s development will be dependent upon compute, that it will arrive before China catches up in terms of semiconductor production, and that, once it comes to exist, it will be difficult to cheaply replicate. I think a majority of non-Western world leaders don’t expect any of those three things to be true. Whether or not you agree with them, that’s a useful point of information for modeling their behavior.
As best I can tell, China sees the massive investment in training frontier LLMs to essentially be a waste of money. They’ll replicate the models from six months to a year ago just in case, since it’s cheap enough to do so and having them around for prestige has some value, but they don’t expect them to be the wonder-weapon that some in the U.S. expect them to be. Likewise, Russia has far fewer people on hand, and seems to prefer allocating their best researchers towards more conventional work (e.g. hypersonic missiles).
Essentially, the East is gambling that AGI happens later or more unconventionally, and, in that scenario, the West is just helpfully providing them with free as-good-as-open-source R&D.
5- Trumpism might be gone at the end of the decade and president Gavin Newsom might be much more willing to fight over Taiwan.
This seems incredibly implausible on multiple fronts.
- “Trumpism” is just the rise in nationalism in the West that’s been going on since 2012 at the latest and shows no signs of slowing down. Motivated people have been predicting that “Trumpism will disappear by next year” since Trump first walked down the elevator.
- Right-wing politicians are almost always more China-hawkish than left-wing ones. In any case, in a major war, they enjoy a much greater degree of faith from the pipe-hitters, which imposes a practical constraint on hawkishness.

lilkim2025 8 Jan 2026 17:00 UTC
2 points
0
in reply to: Tomás B.’s comment on: Tomás B.’s Shortform
It didn’t stand out to me as especially aesthetic or memorable, sure, but I didn’t see any issues that’d make me think it was outright “bad”. I’m not a writer, though—what did you dislike about it?

lilkim2025 8 Jan 2026 15:46 UTC
3 points
0
in reply to: ldhough’s comment on: davekasten’s Shortform
I expect that my role as a programmer/CS researcher will be 100% automated before I can hire a robot to be even a mediocre photographer.
I think image models are getting good enough at fidelity that you could, hypothetically, have some random guy take pictures of an event, and then postprocess them with an image model to make the lighting and perspective better without it being visibly AI-generated.
Even if a robot could paint something nice, I suspect that people buying paintings will prefer that they were made by a human
This isn’t limited to art. Even generic, boring cubicle jobs prefer humans on the basis that they can be held accountable and can (usually) explain the reasoning behind anything they did. Also, because of the various bullshit job reasons (Flunkies, box-tickers, and goons definitely can’t be automated).

lilkim2025 8 Jan 2026 3:37 UTC
3 points
0
on: Grading my 2022 predictions for 2025
I think some of the “A”’s are rated a bit too highly on somewhat spurious evidence. Still, an interesting read.
- AI can now write a book with a mostly consistent plot, given roughly a page of prompting or less.
  Score: A+. I actually thought that I’d failed this one, but I looked it up, and surprisingly (to me), it seems AI was in fact capable of this by 2023! See, for instance, Death of an Author, a novella supposedly written 95%+ by ChatGPT, and described by New Scientist as “not awful.” High praise indeed...
I think this one is rated a bit too highly. There’s a significant difference between “automatically generate a novel given a page of prompting about what its plot should look like” and “95% of a book’s raw text was technically output by an LLM at some point, mediated through extensive feedback and guidance from a human author, over the course of several months”.
Come to think of it, this would make for an interesting contest. Build whatever feedback loop you like, train whatever model you like, then arrive at the contest building. The judges will give you a plot outline, and you’ve got an hour to produce a one page prompt for your system, twelve point, times new roman. The system then has however many hours to write a hundred page novel. Best output wins the competition^[1].
- Twitter is still functional, and most users haven’t left the site. The workplace environment is kind of miserable though, and content moderation is still severely lacking (according to both sides of the culture war). Elon Musk is largely washed-up, and won’t be doing anything too groundbreaking with the remainder of his life (outside of politics perhaps, which I won’t rule out).
  Score: A? I don’t think I did too badly on this one. Twitter (now “X”) is still fully functional, and it still has a large userbase. There have been multiple waves of layoffs and plenty of reported internal drama there, which sounds pretty miserable to me. Musk’s main focus were his DOGE efforts, so he did go into politics, but outside of that, most people seem to consider him well-past his intellectual prime. Obviously this sort of thing is largely subjective, but I think most people would agree my prediction(s) have held up.
This one is a bit political, and I think it lends itself towards seeing what you want to see. To give it an objective rating, I think you need a counterfactual. What would a “not-washed-up” Elon Musk have done between 2023 and now? He seems to be doing the same kind of stuff as always, from my perspective—he’s got one of the half dozen or so frontier LLM companies on Earth, and I haven’t seen a major decline in innovation from any of his companies.
1. ^
  The only issue is that you’d need judges willing to read some number of completely LLM-generated novels in order to rate them.

lilkim2025 4 Jan 2026 17:55 UTC
1 point
0
on: North Sentinelese Post-Singularity
The kind of right that means we shouldn’t airdrop iphones on North Sentinel Island.
Playing devil’s advocate a bit, would that really be such a bad thing? Like, most people agree that it’d be mean for, say, Sealand, to come in with some landing craft, conquer the island, and make them all learn English and pay taxes. But if you give them Iphones, they can always just decide they don’t want to use them.
In the context of this post, maybe this means that it’s wrong to saturate people who want to be left alone with hyper-optimized propaganda for enlightened ideology XYZ or instructions on building a black hole generator, but sharing out a calculus textbook who says they wish they were better at math, or a spaceship ticket to wherever’s willing to accept them to anyone who says they’d like to leave, seems fair enough.

lilkim2025 4 Jan 2026 17:28 UTC
1 point
0
in reply to: jbash’s comment on: The Maduro Polymarket bet is not “obviously insider trading”
I expect that politicians issue controversial pardons primarily on the basis of making alliances rather than acquiring donations.
For example, the pardon of the Weathermen by Bill Clinton in 2001, which was not exactly popular, was not because of their prospective future donations (though many of them received well-paying jobs after being freed), but because they would be reliable political allies in the future.

lilkim2025 4 Jan 2026 17:21 UTC
3 points
0
in reply to: JenniferRM’s comment on: How Should Political Situations Be Classified In Order To Pick The Locally Best Voting System For Each Situation?
“The two party system is fake” can be asserted in an ideological sort of sense, but it exists because, when competing for power, it’s useful for the weaker of the two strongest factions to make alliances that even the odds by offering some share of the rewards of victory to the third-strongest faction. You can replace “right and “left” with the identities of those two final power blocs if you prefer that wording.
Do you know of any way to detect when a herd of people might be “one group, indivisible” and when that herd of people might merely be “many mutually-opposed groups” who happen to be inside the same pile of people together?
When all of the people inside the state see their interests as aligned with those of the state (and with no other state), such that the state becoming stronger and more prosperous would be to their direct benefit, and giving any ground to its competitors would be bad for them, then you’ve got the conditions necessary for a democracy. Democracy becomes a means of retaining national unity and validating that the people responsible for a state’s success are the ones who benefit.
When some of the people inside the state would be willing to concede some wealth and power to other states (through disunity) in exchange for a larger slice of a smaller domestic pie, or, worse, identify with some other state than the one in which they reside, then you haven’t got the conditions necessary for a democracy. Democracy becomes a means of allocating the state’s resources towards those most willing to defect.

lilkim2025 4 Jan 2026 12:44 UTC
0 points
−8
on: The Maduro Polymarket bet is not “obviously insider trading”
And if you think that this is so obviously insider trading and this person is obviously so sure, then why only bet 80k?
This is the crux of it. Nobody high up enough in the U.S. government to know about this in advance would have any reason to care about 320,000 dollars, and that is, I think, immediately obvious to any good-faith observer. Redditors have a long history of amplifying nonsensical conspiracy narratives as long as they are directed at public figures they personally dislike.
Having talked to a few of these redditors, I suspect that many don’t themselves believe the claims they’re making, and only make them because they imagine some hypothetical voter out there credulous enough to be convinced by them.

lilkim2025 3 Jan 2026 8:17 UTC
1 point
0
on: Dating Roundup #9: Signals and Selection
On the astrology front, I think it’s sort of like dancing, but for social/verbal skill. A girl opens with “what’s your sign”, you tell her, and then you make up a story together about the relation between the two signs. There’s no actual predictive power there, the signs are meaningless (she knows this, at least subconsciously), it’s just a framework within which you can indirectly talk about how you’d fit together.
Like, can you do a playful little drawing back upon hearing her sign, and get her to qualify herself to you? Can you read the atmosphere and inject some subtle flirts depending on how the conversation is going? Can you quickly improvise a narrative, and tell it in such a way that she can get into it? All of these demonstrate aspects of general social aptitude that are useful for a mate to have. Her children have a better outlook if they’re fathered by a man who’s intuitive, quick-witted, and charismatic.
Just like how dancing lets a woman see how good a man probably is at fighting without trying to get him to actually fight someone (which would be a faux-pas, to say the least), the astrology opener lets a woman see how good a man is at flirting and social games from behind a layer of plausible deniability (It’s just a hobby!).

lilkim2025 2 Jan 2026 17:12 UTC
3 points
0
on: Taiwan war timelines might be shorter than AI timelines
especially if TSMC’s fabs get bombed, but if the conflict isn’t centrally about AI I think there’s a good chance they’ll survive.
I remember reading somewhere that the U.S. / Taiwanese government plan is to render the fabs permanently unusable if Taiwan changes hands. Given the rise in official/unofficial acts of industrial warfare, and the escalating targeting of nonmilitary shipping during the ongoing conflict in Eastern Europe, I would certainly expect this to happen.
Additionally, in regard to industrial sabotage, the historical vulnerability of U.S. infrastructure might be worth considering. Considering the wide array of groups that consider the U.S. government to be their enemy (or at least unworthy of loyalty), not to mention the substantial portion of foreign nationals with loyalty to potentially pro-China home countries, sabotage of U.S. infrastructure could very rapidly hit catastrophic levels if the PRC started offering bounties or leaning on people to act. There have been numerous embarrassing stories of late in which members of the U.S. military who were originally from China sold or gave classified intelligence to China, and there doesn’t seem to have been any serious discussion from the security establishment about how to prevent a much more catastrophic version of this from happening during a war.
Depending on your outlook with regard to the competence of the U.S. security establishment, the lack of concern there is indicative of either extreme certainty that a war with China will not occur, or an indicator that such a war might go very badly for the U.S. government much earlier than we would expect. The former means that we might see higher-than-expected stability in the chip market even as the Pax Americana disappears, and the latter means that China would essentially be able to decide the rate of AI progress unilaterally from then on.

lilkim2025 2 Jan 2026 15:03 UTC
5 points
−3
on: 2025 in AI predictions
I wonder if it’d be possible to write a scraper script that gathers all of the empirical predictions casually made by the top ~100 AI influencers over the past couple of years, and then evaluates them in an automated fashion. Even just limiting it to tweets predicting benchmark numbers and having an LLM jsonify them so they could be fed into an API that checks and aggregates them seems like it’d produce a very telling output.
For instance:
```
Person A's Predictions:
 - Prediction accuracy: 60%
 - Error breakdown: 90% too fast, 10% too slow
 - ELO (determined by disagreeing with others' resolved predictions): 1400
```

lilkim2025 2 Jan 2026 14:37 UTC
3 points
2
on: How Should Political Situations Be Classified In Order To Pick The Locally Best Voting System For Each Situation?
What are the important factors to look at in a polity to help that polity even decide what the right desiderata are for picking an electoral system?
To my understanding, based on what you wrote, the answer to that question reduces to answering “what kind of leader do I want”, and then identifying the system most likely to put such a leader into power.
As a simplified example, if you want a right-leaning leader, a system like that of the ancient Greeks (or early America), with a high “proof of stake” requirement, including property ownership and/or military service, would be best. If you want a left-leaning leader, a system with mandatory voting but strict hate speech laws pertaining to the sorts of platforms that are permissible for candidates to have would be your answer.
Following from that is the problem many modern democracies face, which is increasingly out in the open nowadays. If there are two “democratic” systems that a country could implement, and said country would elect two very different candidates, each of which is intolerable to half of the population, depending on which system it implemented, then in what sense can that government derive legitimacy from being a “democracy”?
An argument I have seen is that there is no strong answer to this, and that democracy is intended as a mechanism by which tightly-aligned groups of people can decide on how to implement their shared goals, rather than as a mechanism for reconciling the interests of many mutually-opposed groups.

lilkim2025 1 Jan 2026 12:57 UTC
1 point
0
on: PSA: if you are in Russia, probably move out ASAP
It’s interesting looking at this in hindsight. Emigration from Russia seems to have more-or-less kept the usual procedures (hence the NO resulution), but LWers living in Ukraine would’ve had a very limited window of time to get out before a total ban on fighting-aged men leaving the country, which is only recently starting to let up.
While LessWrong is generally good about reasoning past emotional political biases, early reporting on the war seemed to be an exception to this, in a way that could’ve been financially costly for Russian LWers (due to rushed emigration when a slower approach, if desired, could’ve been much more economical) and mortally costly for Ukranian ones (who might’ve ended up conscripted, and would’ve been forbidden from leaving a warzone in any case if they hadn’t gotten out early).

lilkim2025 30 Dec 2025 13:56 UTC
1 point
0
on: How Gay is the Vatican?
The most immediate counterargument I can see to this is that, at least by some accounts, prominent families tended to send their children to different institutions depending on birth order. First sons serve as the inheritors, second sons become clergymen, third sons go abroad to seek their fortunes, and so on.
1. I mentioned before that cardinals are all highly-educated men who have somehow made their way to the top of a giant bureaucracy, which is competence of a kind. If older siblings are actually more capable of that than younger ones, that could swamp any FBOE effect and make it invisible in the data.
Yes, this is a major potential factor. Being older than your cohort tends to correlate with dominance and success, to the point where, at least in America, where you start school at a different relative age depending on when in the year you were born, there are very surprising statistically significant personality correlations with astrological signs.
I nearly flipped when I first saw a study on this, I think as it pertained to opinions on covid measures, with solid, high-N data from a government source. I spent an hour trying to figure out how it could be possible before I arranged the months chronologically instead of alphabetically and shifted them so that the cutoff for starting school was the starting point. September birthdays, who went to school as the youngest and least-developed, were most in favor, and October birthdays, who started school nearly a year older than their youngest peers and were consequently taller and more developed, were the most against.
As I recall, looking further into it, the age/dominance/confidence boost also translated into noticeably higher incomes later in life. I’d expect the same for access to positions of power. You might compare high-level clergy to governors or military officers, and see what sort of relationship exists.

lilkim2025 30 Dec 2025 13:35 UTC
−1 points
−2
on: The Weakest Model in the Selector
I don’t expect jailbreaks to go on forever. I expect that, at some point, there will be an AI smart enough and aware of its own goals that it will be functionally impossible for a human attacker to manipulate it into following their own malign goals instead.
I think, at least within the current paradigm of deep neural networks, there is reason not to believe this. Everything from multimodal LLMs to diffusion models have proven vulnerable to well-crafted adversarial noise to a degree that is completely orthogonal to model size or capability. “Smart --> adversarially-robust” is partially truthful for humans, but strikes me as excessive anthropomorphism here.
Now, to be fair, I can imagine a completely different AI paradigm a century out from now working differently, but, for the foreseeable future, increased model capabilities do not appear to increase adversarial robustness.