derelict5432

• derelict5432 18 Mar 2026 2:03 UTC

  1 point

  0

  in reply to: Steven Byrnes’s comment on: You can’t imi­ta­tion-learn how to con­tinual-learn

  Sorry, I’m afraid I don’t understand what your analogy is supposed to map to. What is Grog in the context of our conversation? You seem to admit at the end that LLMs are not really at all like Grog, in that Grog has no underlying bedrock of understanding, while modern LLMs do.

  Thus, there’s more to knowledge than lists of facts. It’s ways that the facts all connect to each other in an interconnected web, and it’s ways to think about things, etc.

  I’ll agree with this definition. If you’ll agree that knowledge can exist in written form and textbooks often embody exactly what you describe. They are very rarely ‘lists of facts’. More often than not, they are logically curated, organized explanations of phenomenon and events, along with rich descriptions of their connections and interactions. You seem to be preferentially upselling knowledge that is stored in synaptic weights while drastically downplaying knowledge recorded in other mediums. Why?

• derelict5432 17 Mar 2026 3:12 UTC

  2 points

  −3

  in reply to: Steven Byrnes’s comment on: You can’t imi­ta­tion-learn how to con­tinual-learn

  So according to you, a system that could acquire new facts, record them, access them, and use them, continuously in this way would not constitute ‘real’ continuous learning. It could conceivably fill its database with the actionable knowledge of 1000 yet unwritten textbooks, but that wouldn’t be ‘real’ to you.

  1. “wholly new ways of conceptualizing and navigating the world, not just keeping track of what’s going on” are learnable and storable in the way I describe.

  2. How is this type of learning not open-ended? What is limiting it?

  3. Your third criteria seems to be related to unsupervised learning, specifically self-play. Not sure why you’d limit continual learning in this way, either.

  You seem to be putting somewhat arbitrary constraints on what constitutes continual learning. Generally, if the system’s knowledge base is fixed, it’s incapable of continuing to learn. If it has the capacity to acquire new knowledge and skills, by whatever means, it continues to learn. You’re narrowing that general idea without really justifying why.

• derelict5432 17 Mar 2026 0:41 UTC

  1 point

  −1

  on: You can’t imi­ta­tion-learn how to con­tinual-learn

  I tell an LLM my favorite color. As long as that information is in its context window, it has access to it. As soon as that context rolls off or goes away, the LLM no longer has access to that information.

  I build an agent with scaffolding that has a database. I tell it my favorite color. The agent records it in the database. The weights of the LLM are still fixed, but during its base training it learned how to access information. So if I ask it at any point in the future what my favorite color is, it knows. It access the information in the database.

  Do you consider this continual learning? If not, why not?

• derelict5432 7 Mar 2026 21:22 UTC

  2 points

  0

  in reply to: Mateusz Bagiński’s comment on: Pod­cast: Jeremy Howard is bear­ish on LLMs

  This seems more like a within-distribution problem: the player is encountering a game that is composed of pieces that are very alike the pieces of the games they’ve previously encountered, and the rules follow a similar logic.

  Well, that’s one of the big questions, isn’t it? Seems fairly clear there’s no hard boundary between in-distribution and out-of-distribution. Is the cure for cancer and the way to discover it going to be completely OOD? Or is it going to lean heavily on existing knowledge of cell biology, genetics, and all previous cancer research? The common phrasing is ‘standing on the shoulders of giants’. This is pretty well accepted as the way new inventions and discoveries happen. Not as radically alien knowledge that emerges from a vacuum, but an incremental step up using a mountain of existing knowledge bases (analogous to a game composed of pieces very alike ones they’ve previously encountered). Very large discoveries or paradigm shifts are likely more OOD, but the vast bulk of new science is fairly incremental and I would think the sort of problems you’d consider within-distribution. No?

• derelict5432 7 Mar 2026 18:11 UTC

  3 points

  0

  in reply to: RogerDearnaley’s comment on: Pod­cast: Jeremy Howard is bear­ish on LLMs

  I actually implemented my own private benchmark last year to try to test this with different models. The domain was a toy OOD task where the system had access to three possible tools that performed simple transformations on a configuration of binary values in a particular spatial arrangement. Stage 1 was exploration. The system was given a certain number of steps to probe with the tools (which were chosen randomly from a subset prior to each trial). After the experimentation stage, the system was required to use the tools to perform a transformation on a random arrangement to make it match a target one.

  The exercise of building a benchmark was a great learning experience for me. My main takeaway was that differences in performance were nearly all driven by differences in scaffolding, and not so much the base model. This made me fairly disillusioned about benchmarks in general. Made me suspect that gains in benchmarks like ARC-AGI are mostly driven by scaffolding improvements. Maybe someone here has much more insight into that.

  But it also made me think that the problem is probably not some far-out radically intractable problem. You mention continual learning and long time horizons. Just generally for OOD tasks, the system needs to be able to log results, generate and revise hypotheses, and carry out Bayesian updates in an iterative manner. Whether that can be cracked reliably for increasingly difficult problems with relatively straightforward scaffolding, or the base models need to be radically improved along with scaffolding, I don’t really know. Maybe for the much more difficult problems (like a Theory of Everything or a cure for the common cold) those advances are very far out. I would think though, that for simple and medium-difficulty problems, the frontier labs are already well on their way.

• derelict5432 7 Mar 2026 13:37 UTC

  4 points

  3

  in reply to: RogerDearnaley’s comment on: Pod­cast: Jeremy Howard is bear­ish on LLMs

  Not sure why the go-to examples for out-of-distribution problems tend to be the extreme of an entirely new theory or invention. To make progress on this problem, we’d want to identify minimally-OOD problems and benchmark those, wouldn’t we?

  Melanie Mitchell and collaborators showed weaknesses in LLM on OOD tasks with simple perturbations in the alphabet for string-analogy tasks. This seems like the sort of example we should generally be thinking about and testing, because they’re likely much more tractable, toy domains or simple ad hoc tasks that deviate from strong biases in the training distribution.

• derelict5432 7 Mar 2026 1:54 UTC

  2 points

  1

  in reply to: Mateusz Bagiński’s comment on: Pod­cast: Jeremy Howard is bear­ish on LLMs

  A human attends board game night. They learn a new game they’ve never played before. Technically, this is out-of-distribution learning. This type of learning does not necessarily seem like augmenting a car for space travel (maybe it is). They are not having to learn all about games and dice and boards and pieces all from scratch. They are having to mostly map existing learned models into a slightly novel combination for a slightly new domain. I’m not saying that’s a trivial thing to do, because it’s a hard open problem that many many smart people have been trying to crack for decades.

  But it does not seem as daunting as you are portraying it. Yes, out-of-distribution is a very large space. But there’s an awful lot of that space that we’re simply not interested in learning anyway, so that narrows it down quite a lot.

  As another commenter here noted, we probably actually do hope it’s a problem that won’t be cracked anytime soon, though the current effort and resources being spent towards the problem are historically unprecedented. I very well could be underestimating the problem. I guess we’ll just see.

• derelict5432 6 Mar 2026 23:03 UTC

  2 points

  0

  on: Pod­cast: Jeremy Howard is bear­ish on LLMs

  That’s a lot of words to just say LLMs currently are super-competent within their training distribution and not good outside of it. I haven’t watched the whole podcast. Do we have good reason to think this particular deficit is unable to be remedied? That making inroads to issues like continual learning won’t enable these sorts of systems to perform much better at ad hoc or out-of-distribution tasks?

• derelict5432 3 Mar 2026 12:55 UTC

  1 point

  0

  on: AI com­pa­nies and the 99% lethal au­tonomous weapons myth

  Assuming this is true, that you’re talking about a distinction without a difference, do you have any suggestions or proposals? Full pause on AI for military use? Restrictions on particular uses in the chain at all? Restrictions on using AIs above a particular capacity threshold? Or are we just doomed because we can’t make relevant distinctions?

• derelict5432 25 Feb 2026 12:45 UTC

  9 points

  5

  on: Praise the Moloch!

  One can watch almost in real time as democracies, or at least the illusions of such, fade into a global oligarchy. Ever so sloooowly. Oh well, I’ve always craved a more meritocratic world.

  You lost me here. You seem to be lumping every single democracy in the same corrupt, illusory basket. And in what way are oligarchies more meritocratic, and not the opposite?

• derelict5432 22 Feb 2026 19:03 UTC

  4 points

  0

  in reply to: RogerDearnaley’s comment on: LLMs and Liter­a­ture: Where Value Ac­tu­ally Comes From

  Yeah, really.

  Let’s take the short excerpt from my post. It’s filled with metaphorical language and contains one explicit simile. I did an exact string search for each in Google Books and internet-wide and could not find a match. Let’s look at them one by one:

  1. “malign little apocalypses”: Clearly metaphorical language. My writerly side appreciates the ironic contrast between the large scale of apocalypses and the descriptor ‘little’.

  2. “soft electrical throat-clearing”: Again, metaphorical. Vacuum cleaners are not traditionally thought of as having mouths or throats. ‘Throat-clearing’ is normally clearing the air in a particular kind of airway. This one is ‘electrical’. I find that interesting and not inept.

  3. “the first impossible thought coiling awake like a pale worm disturbed in its cosmic soil”: Thoughts aren’t literally worms. They don’t literally coil. Here the simile evokes the language of Lovecraft (who did talk about worms and cosmic things a lot) but in a unique reference to the very first thoughts of a newly-conscious household item.

  Now, you said you thought the passage was an obvious pastiche. Good for you. To the extent that’s true, I’d place you in an incredibly small minority as well. I probably would have noticed the Lovecraftian elements, but even though I’m very familiar with all three writers, I don’t think I could have identified all of them.

  And to my ear, none of these metaphors or similes was a ‘miss’. Far from it. They’re all (as far as I can discern) original, interesting, and evocative.

  Would you honestly have been able to distinguish this from human writing if I hadn’t told you? Do you think this metaphorical language is a broken failure? Cliched? Nonsensical?

  I wonder if we’re already starting to enter some new phase where good writing immediately becomes suspect simply because it is actually good. I do see more and more accusations of people being bots in places like Reddit. Maybe this will even lead to a dumbing down effect of human writing, or very weird out-of-distribution styles that are hard for LLMs to mimic.

  Anyway, based on even the small snippet I posted, LLMs handle metaphors far better than most writers. I suppose it is largely subjective. If you think these examples are garbage, though, I would love to hear the reasons why.

• derelict5432 22 Feb 2026 17:06 UTC

  4 points

  0

  in reply to: RogerDearnaley’s comment on: LLMs and Liter­a­ture: Where Value Ac­tu­ally Comes From

  Without doing a deep dive on what ‘good’ is, and having taught and workshopped writing for years, I can pretty safely say that current LLMs already produce better-crafted prose than the vast majority of humans. So if the bar is Pulitzers and best-sellers, sure, we’re not there yet. But we’re already above the median and well on our way.

• derelict5432 22 Feb 2026 12:51 UTC

  1 point

  0

  in reply to: dauten’s comment on: LLMs and Liter­a­ture: Where Value Ac­tu­ally Comes From

  Sure, I’d agree with that. LLM-generated prose and sunsets differ in that the prose has indirect expressions of values based on the training (data choice, reinforcement learning, etc). We’ve lost the direct, conscious expression of particular values (though this gets fuzzier the more autonomous/​agentic the systems get), but yes, the output is not value-neutral.

• derelict5432 22 Feb 2026 1:46 UTC

  6 points

  2

  in reply to: RogerDearnaley’s comment on: LLMs and Liter­a­ture: Where Value Ac­tu­ally Comes From

  You seem to be mostly arguing the ways in which LLMs currently do not succeed at producing good, interesting writing. That’s kind of orthogonal to the point of the post, which is where the value of good, interesting writing comes from, and whether or not it’s necessary to have a human writer behind the wheel.

  I do try to demonstrate that the quality of AI writing varies based on workflow and prompting to refute the narrow point that LLM output is always just an average of the input. You admit that there are various ways to accomplish this (including doing things like increasing temperature), but point out that they’re not sustainable for more than short passages. This does not seem to be a fundamental limitation (if it’s even true), but something that can be overcome with agentic loops of criticism and revision. Or even just continued improvement in models.

  It’s funny you brought up tracking of physical details and spatial relationships. Way back with GPT-2 I experimented with generating romance/​erotica stories. I tried to start with short scenes of couples engaged in foreplay and undressing. Very reliably, the model was lose track of which pieces of clothing had already been removed. Characters would remove their pants twice, or three times within the span of three sentences. Or sequencing would be all wrong. They’d remove socks before shoes. And so on.

  I just tried a very simple prompt with GPT 5.2 for the type of scene I’m talking about and got this:

  His hands slid to the buttons of his coat, slipping it off and hanging it over the chair. She stepped closer and tugged playfully at his tie, loosening it before sliding it free and tossing it onto the small desk. He laughed under his breath as she began unfastening the buttons of his shirt, one at a time, her fingertips grazing his skin as the fabric parted. When the last button came loose, she pushed the shirt from his shoulders and let it fall to the floor.
  
  “Your turn,” he murmured.
  
  She turned slightly, lifting her hair so he could find the zipper at the back of her dress. He drew it down slowly, the soft rasp of metal sounding loud in the quiet room. The dress loosened, slipping from her shoulders. She caught it, then stepped out of it, leaving it pooled neatly beside his shirt.
  
  He took a small breath, like he was trying not to rush anything, and hooked his thumbs into the waistband of his trousers, stepping out of them before nudging them aside with his foot. She followed by kicking off her heels, then unbuttoning the light blouse she’d worn beneath the dress, sliding it down her arms and folding it almost absent-mindedly on the chair.
  
  Soon the floor held a small trail of discarded fabric — coat, tie, shirt, dress, trousers, blouse — and they stood facing each other in the lamplight, down to their underwear, the room suddenly feeling warmer than before.

  
  Not going to win any awards, but the tracking and choreography are logical and accurate. Obviously light years ahead of GPT-2, and the sort of thing that gets incrementally better with every new frontier model. So I’d push back on your assertion that currently systems are just bad at blocking.

• derelict5432 21 Feb 2026 13:18 UTC

  4 points

  2

  on: LLMs and Liter­a­ture: Where Value Ac­tu­ally Comes From

  I think the crux of this argument comes down to where people think literary value actually lives.

  If someone thinks the value of literature is fundamentally in communication from one conscious mind to another, then LLM writing really does look like it’s missing something essential.

  If instead the value mostly lives in what happens on the reader’s side (i.e., whether the text actually generates thought, emotion, or insight) then the production process seems less central.

  I suspect most disagreement here will reduce to that difference, rather than to empirical claims about how LLMs work.

  I’m especially interested in pushback from people who think authorial intent or embodiment is doing real explanatory work here. What specifically breaks if we remove it?

LLMs and Liter­a­ture: Where Value Ac­tu­ally Comes From

• derelict5432 12 Jun 2025 12:29 UTC

  6 points

  1

  in reply to: Gordon Seidoh Worley’s comment on: Reli­gion for Rationalists

  I also added ‘pretending to believe’. I see your position as wanting to have your cake and eat it too. You’re encouraging people to join the most watered-down religions that make the smallest epistemological demands on followers, so that they can enjoy the benefits of community and mental well-being.

  You say:

  My argument is roughly that religions uniquely provide a source of meaning, community, and life guidance not available elsewhere, and to the extent anything that doesn’t consider itself a religion provides these, it’s because it’s imitating the package of things that makes something a religion.

  I’d tend to agree with you. But how is it that religions are able to uniquely provide a source of meaning, community and life guidance, if not through shared belief? You’re arguing we can reap the benefits of the effects without the cause. I’m most familiar with christianity. Where do you think the source of meaning and guidance on how to live your life come from, if not in a shared belief that jesus is the son of god, that he died for your sins, and was resurrected?

  I’m less familiar with buddhism, but every religion requires some commitment of belief, and that’s the whole point.

• derelict5432 11 Jun 2025 23:38 UTC

  10 points

  5

  on: Reli­gion for Rationalists

  This is a utilitarian argument for either believing or pretending to believe poorly-supported superstitions for social benefits. Do you value the truth that little?

  My favorite counter-thought-experiment to this comes from Sam Harris:

  A man believes there is gold buried in his very large backyard. Every Sunday morning he wakes up, grabs his shovel, and starts digging. There is no gold. But the digging is good exercise. There are no substantial negative consequences of his belief.

  Is his activity justified on the basis of its utility? How much do we figure in the value of truth here? If people want to either delude themselves or pretend to delude themselves in order to benefit tangibly, I’d say they’re doing a disservice to truth. You don’t particularly seem to care about that. Or do you?

• derelict5432 5 May 2025 22:33 UTC

  3 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Repli­ca­tor->Ve­hi­cle Align­ment and Hu­man->AI Alignment

  So you think I’m underestimating the misalignment between replicators->vehicles? Not sure that undermines my central point, that we’re still fundamentally ignorant of our goal structures and that has really negative consequences for human->AI alignment.

  Or do you think our misalignment with our genes is actually a cause for optimism when it comes to AI alignment somehow?

Repli­ca­tor->Ve­hi­cle Align­ment and Hu­man->AI Alignment