I meant their newsletter, which I’ve subscribed to. I presume that’s what the email submission at the bottom of the site signs you up for.
AlphaAndOmega
Karma: 1,017
I just wanted to say that I really enjoy following along with the affairs of the AI Village, and I look forward to every email from the digest. That’s rare, I’m allergic to most newsletters.
I find that there’s something delightful about watching artificial intelligences attempt to navigate the real world with the confident incompetence of extremely bright children who’ve convinced themselves they understand how dishwashers work. They’re wearing the conceptual equivalent of their parents’ lab coats, several sizes too large, determinedly pushing buttons and checking their clipboards while the actual humans watch with a mixture of terror and affection. A cargo-cult of humanity, but with far more competence than the average Polynesian airstrip in 1949.
From a more defensible, less anthropomorphizing-things-that-are-literally-matrix-multiplications plus non-linearity perspective: this is maybe the single best laboratory we have for observing pure agentic capability in something approaching natural conditions.
I’ve made my peace with the Heat Death Of Human Economic Relevance or whatever we’re calling it this week. General-purpose agents are coming. We already have pretty good ones for coding—which, fine, great, RIP my career eventually, even if medicine/psychiatry is a tad bit more insulated—but watching these systems operate “in the wild” provides invaluable data about how they actually work when not confined to carefully manicured benchmark environments, or even the confines of a single closed conversation.
The failure modes are fascinating. They get lost. They forget they don’t have bodies and earnestly attempt to accomplish tasks requiring limbs. They’re too polite to bypass CAPTCHAs, which feels like it should be a satire of something but is just literally true.
My personal favorite: the collective delusions. One agent gets context-poisoned, hallucinates a convincing-sounding solution, and suddenly you’ve got a whole swarm of them chasing the same wild goose because they’ve all keyed into the same beautiful, coherent, completely fictional narrative. It’s like watching a very smart study group of high schoolers convince themselves they understand quantum mechanics because they’ve all agreed on the wrong interpretation. Or watched too much Sabine, idk.
(Also, Gemini models just get depressed? I have so many questions about this that I’m not sure I want answered. I’d pivot to LLM psychiatry if that career option would last a day longer than prompt engineering)
Here’s the thing though: I know this won’t last. We’re so close. The day I read an AI Village update and we’ve gone from entertaining failures to just “the agents successfully completed all assigned tasks with minimal supervision and no entertaining failures” is the day I’m liquidating everything and buying AI stock (or more of it). Or just taking a very long vacation and hugging my family and dogs. Possibly both. For now though? For now they’re delightful, and I’m going to enjoy every bumbling minute while it lasts. Keep doing what you’re doing, everyone involved. This is anthropology (LLM-pology?) gold. I can’t get enough, till I inevitably do.
(God. I’m sad. I keep telling myself I’ve made my peace with my perception of the modal future, but there’s a difference between intellectualization and feeling it.)
>It would, for instance, never look at the big map and hypothesize continental drift.
Millions of humans must have looked at relatively accurate maps of the globe without hypothesizing continental drift. A large number must have also possessed sufficient background knowledge of volcanism, tectonic activity etc to have had the potential to connect the dots.
Even the concept of evolution experienced centuries or millenia of time between widespread understanding and application of selective breeding, without people before Darwin/Wallace making the seemingly obvious connection that the selection pressure on phenotype and genotype could work out in the wild. Human history is littered with a lot of low hanging fruit, as well as discoveries that seem unlikely to have been made without multiple intermediate discoveries.
I believe it was Gwern who suggested that future architectures or training programs might have LLMs “dream” and attempt to draw connections between separate domains of their training data. In the absence of such efforts, I doubt we can make categorical claims that LLMs are incapable of coming up with truly novel hypotheses or paradigms. And even if they did, would we recognize it? Would they be capable of, or even allowed to follow up on them?
Edit: Even in something as restricted as artistic “style”, Gwern raised the very important question of whether a truly innovative leap by an image model would be recognized as such (assuming it would if a human artist made it) or dismissed as weird/erroneous. The old deep dream was visually distinct from previous human output, yet I can’t recall anyone endorsing it as an AI-invented style.
Other than the issues raised below, I’d like to point out that the help doesn’t need to be full time to make a massive difference. Just having a cleaner in once a week or someone to cook every evening helps!
India is a… large country. Without specifying which part you intend to visit, you might receive the equivalent of a recommendation to visit the best restaurant in Portugal while you’re actually in Amsterdam! I used to live there, though by the eastern side where you’re unlikely to visit. Nonetheless, I will happily vet or suggest places if you know your itinerary.
The phenomenon you observed as far more to do with the tiny plot sizes in most of rural India than it had to do with the cost of labor.
Many/most farmers have farms sized such that they are, if not as bad as mere subsistence, unable to justify the efficiency gains of mechanization. This is not true in other parts of India, like the western states of Punjab and Haryana, where farms are larger and just about every farmer has a tractor. There are some cooperatives where multiple smallholders coordinate to share larger machines like combine harvesters which none but the very largest farmers can justify purchasing for personal use.
Optimal economic policy (in terms of total yield and efficiency) is heavily in favor of consolidating plots to allow economies of scale. This is politically untenable in much of India, hence your observation. However, it isn’t a universal state of affairs, and many other fruits of industrialization are better adopted.
So I looked it up and apparently the guideline is actually a 2 hours fast for clear liquids! 2 hours![7] The hospital staff, however, hardened their hearts. Nurses said to ask the surgeons. The surgeons said to ask the anestheologists. It wasn’t until 7am that the anestheologists said, yep, you can drink a (small) glass of water.
Ah, this takes me back to my medical officer days. No junior doctor ever got into trouble for telling a patient to fast a bit too long, and many do for having a heart and letting them cut it short. It is also likely the least consequential thing we can bother the anesthetists about, and it’s not going to kill anyone to wait longer (usually).
Knowing the local demographic and behavioral tendencies on LW, I think it might worth noting that ozempic/semaglutide and other GLP drugs can cause delayed gastric emptying. As a consequence, even the standard fasting duration might not be adequate to fully empty your stomach. If you’re scared of getting aspiration pneumonia, it’s worth mentioning this to your surgeon or the anesthetist. The knowledge hasn’t quite percolated all the way up the chain, so you can’t just assume they’re aware.
Here’s how I parsed this:
We’re in a future not particularly long after a hard takeoff Singularity (changes to the environment described as quite sudden rather than gradual).
This is a multipolar AI scenario, given that the Origami Men claim to have purchased these humans from another entity that owned them.
The Origami Men (or the entity operating them) is severely misaligned but not to an omnicidal degree. It appears to value (bad) proxies for human wellbeing, such as ensuring that they have jobs, pets, residences and food. They either do not understand or do not care that their implementation is… subpar. We might as well call it the Uncanny Valley of Care.
They seem to have some more subtle store of value, in the sense that they desire to only rear humans who do not actively breach containment, or who do not spread memes about breaking out (“contagion”). The general impetus behind the cull or reclamation is not clear to me, do they initially just pick people at random? Or does the narrator finding something “beautiful” in the people chose correlate to an unknown preference? They will take you if you get too close to the border of the dome, but general discontent is not policed. Nor is attempts at violence against the Origami Men, though that appears entirely futile.
“Provenance”? Do they only value pre-Singularity specimens? That would explain lack of any evidence of artificially boosting human numbers or encouraging reproduction.
>I was surprised to find out that I could not find a single neurotransmitter that is not shared between humans and mice (let me know if you can find one, though).
As far as I can tell, this is true. The closest I could find is a single subtype of receptor that a frameshift mutation in a primate ancestor made nonfunctional.
https://www.sciencedirect.com/science/article/pii/S0021925818352189
I suspect you’d have to go even further back in terms of divergent ancestry to find actual differences in the neurotransmitter substances themselves. There are definitely differences in receptors and affinities, but the actual chemicals? From a quick search, you’d have to go all the way to ctenophores, which are so different that some have posited they evolved nervous systems independently.
I presume it would be an overview of histone molecule. DNA is often kept compacted by wrapping around one, like a rope coiled around a baseball.
I find it very hard to believe that a civilization with as much utter dominion over physics, chemistry and biology as the Culture would find this a particularly difficult challenge.
The crudest option would be something like wiping memories, or synthesizing drugs that re-induce a sense of wonder or curiosity about the world (similar to MDMA). The Culture is practically obsessed with psychoactive substances, most citizens have internal drug glands.
At the very least, people should be strongly encouraged to have a mind upload put into cold storage, pending ascendance to the Sublime. That has no downsides I can see, since a brain emulation that isn’t actively running is no subjectively different from death. It should be standard practice, not a rarity.
Even if treated purely as a speculation about the “human” psyche, the Culture almost certainly has all the tools required to address the issue, if they even consider it an issue. That is the crux of my dissatisfaction, it’s as insane as a post-scarcity civilization deciding not to treat heart disease or cancer.
Even if such ennui is “natural” (and I don’t see how a phenomenon that only shows up after 5-6x standard lifespan, assuming parity with baseline humans can ever be considered natural), it should still be considered a problem in need of solving. And the Culture can solve just about every plausibly solvable problem in the universe!
Think of it this way, if a mid-life crisis reliably convinced a >10% fraction of the population to kill themselves at the age of 40, with the rest living happily to 80+, we’d be throwing tens of billions at a pharmacological cure. It’s even worse, relatively and absolutely, for the Culture, as their humans can easily live nigh-indefinitely.
Even if you are highly committed to some kind of worship of minimalism or parsimony, despite infinite resources, or believe that people have the right to self-termination, then at least try and convince them to make mind backups that can be put into long-term storage. That is subjectively equivalent to death without the same… finality.
This doesn’t have to be coercive, but the Culture demonstrates the ability to produce incredibly amounts of propaganda on demand. As far as I’m concerned, if the majority of the population is killing itself after a mere ~0.000..% of their theoretical life expectancy, my civilization is suffering from a condition that ours standard depression or cancer to shame. And they can trivially solve it, they have incredibly powerful tools that can edit brains/minds to arbitrary precision. They just… don’t.
While the Culture is, on pretty much any axis, strictly superior to modern civilization, what personally appalls me is their sheer deathism.
If memory serves, the average human lives for around 500 years before opting for euthanasia, mostly citing some kind of ennui. What the hell? 500 years is nothing in the grand scheme of things.
Banks is careful to note that this isn’t, strictly speaking, forced onto them, and exceptions exist, be it people who opt for mind uploads or some form of cryogenic storage till more “interesting” times. But in my opinion, it’s a civilization-wide failure of imagination, a toxic meme ossified beyond help (Culture humans also face immense cultural pressure to commit suicide at the an appropriate age).
Would I live in such a civilization? Absolutely, but only because I retain decent confidence in my ability to resist memetic conditioning or peer pressure. After all, I’ve already spent much of my life hoping for immortality in a civilization where it’s either derided as an impossible pipe-dream or bad for you in {hand-wavy ways}.
Another issue I’ve noted is that even though this is a strictly post-scarcity universe in the strong sense, with matter and energy freely available from the Grid, nobody expands. Even if you want to keep natural bodies like galaxies ‘wild’ for new species to arise, what’s stopping you from making superclusters of artificial matter in the enormous void of interstellar space, let alone when the extragalactic supermajority of the universe lies empty? The Culture is myopic, they, and the wider milieu of civilizations, seem unwilling to remotely optimize even when there’s no risk or harm of becoming hegemonizing swarms.
(Now that you’ve got me started, I’m half tempted to flesh this out into a much longer essay.)
Strong upvote. When I began reading, I slightly scoffed at the idea of “musician’s dystonia”. Never heard of it, didn’t sound plausible to me, despite being a doctor and psych trainee. I thought it was a nice hypothetical to shore up the story’s central conceit.
And then I actually Googled it, and found a wiki article. Oh dear.
There is an important distinction to be made between base models, which are next token predictors (and write in a very human like manner) and the chatbots people normally use, despite both being called “LLMs”. Those involve taking a base model and then subjecting it to further instruction tuning and Reinforcement Learning from Human Feedback.
My understanding is that a lot of the stylistic quirks of the latter arise from the fact that OpenAI (and other companies) either personally or through outsourcing hired Third World contractors. They were cheap and reasonably fluent in English. However, a large number were Nigerian, and this resulted in a model slightly biased towards Nigerian English. They’re far more fond of “delve” than Western English speakers, but most people wouldn’t know that, because how often were they reading Nigerian literature or media? It’s particularly rampant in their business or corporate comms.
Then you have other issues stemming from RLHF. Certain aspects of the general chatbot persona were overly engrained, as preference feedback biased towards verbosity and enthusiasm. And now that the output of ChatGPT 3.5 and onwards is all over the wider internet, to an extent, new base models expect that that’s how a chatbot talks. This likely bleeds into the chatbots that are built on the contaminated base model, since they have very strong priors that they’re a chatbot in the first place.
That doesn’t mean that there isn’t significant variance in how current models speak. Claude has a rather distinct personality and innate voice, even within the various GPT models, o3 was highly terse and fond of dense jargon. With the availability of custom instructions, you can easily get them to talk you in just about any style you prefer.
I was a big fan of Horizon Alpha, a stealth model available on OpenRouter, later revealed to be an early checkpoint/variant of GPT-5. Unfortunately, the release candidate isn’t quite as good at my usual vibes benchmark when it comes to creative writing, be it 5 or 5-Thinking.
(4.1 was really good at the same task, and surprisingly so, for a model marketed for coding. I missed it when it was gone, and I’m glad to have it back)
I was initially quite negative about GPT-5T, but I’ve warmed to it. It seems as smart or smarter than o3, and is a well-rounded and capable model. The claims of a drastically reduced hallucination rate seems borne out in extensive use.
I don’t think 4o is that harmful in objective terms, but Altman made a big fuss about reducing sycophancy in GPT-5 and then immediately caved and restored 4o. It’s a bad look as far as I’m concerned.
More concerningly, if people can get this attached to an LLM as mediocre as 4o, we’re in for a great time when actually intelligent and manipulative ASI gets here.
You’re correct. When GPT’s native image gen came out, it absolutely refused to touch kids. I didn’t realize that they’d changed it (at least because I never had the reason to try) till I attempted this ill-fated experiment.
An interesting idea, but I genuinely don’t see how that’s supported by the movies? Ignore this if you were just joking!
>I think image models are getting good enough at fidelity that you could, hypothetically, have some random guy take pictures of an event, and then postprocess them with an image model to make the lighting and perspective better without it being visibly AI-generated.
I have done this with the best recent models, and almost nobody has been able to tell. It is absolutely trivial to take a photo on a phone, and ask Nano Banana to “make it look like it was taken by a professional photographer using a DSLR applying expert color grading.” Easy as that.