What sources have you used to derive your understanding of brain function from?
interstice
Karma: 2,773
Simulators
On the object level, there’s a very plausible mechanism why solving some problems can make things worse
Hmm, I see. But it seems to me that the linked post is overoptimistic that making problems legible will cause people to slow down until they are solved. Already, lots of problems that are “legible” to this community are not “legible” enough to make the labs or governments want to slow down. So working to solve them(by our standards, since apparently other people already consider them solved or unimportant) could still be useful.
Of course you could then say that what we should be doing is trying to make problems legible to this community legible to the wider world, which is pretty much what MIRI is trying to do at the moment. Certainly that seems like a valuable thing to do. But far from guaranteed to succeed. And I think the fact that LW-legibility of takeover isn’t on its own enough to cause the wider world to slow down, should make us less worried that solving one problem to our standards will make the world push ahead more recklessly, since they seemingly aren’t that responsive to what problems are considered solved by our standards.
there is a bunch of different safety problems where solving some but not all of them at the same time can make the overall situation worse
But isn’t the relevant thing here whether solving some makes the overall situation worse in expectation? Like sure, it could be the case that e.g. solving intent alignment makes things worse, but it seems relatively unlikely. It seems implausible that the current set of problems we have now cancel each other out optimally. Or maybe we should try to introduce new problems to further mitigate the set of existing problems!
Yeah it’s more like it’s relevant to the kind of world we find ourselves in. But that is itself important to agency as a given agent design will only be successful in certain kinds of worlds.
I think QM itself. It’s important somehow that the world is actually quantum mechanical. But probably not in a very direct fashion, but via influencing the sort of high-level properties and entities that end up “emerging” from the base laws.
[I don’t actually think this is true, but] It would be funny if rationalism turns out to not merely be a euphemism for autism but “mal”functioning oxytocin receptors and rationalists are constitutionally unable to normally feel love/social emotions; whether this would be to the discredit of love or rationalism is up to taste.
“quantum mechanics is probably important to the structure of agency/the mind in some way we don’t understand yet”.
Can’t speak for everybody but when I read simulators I was like “yeah obviously GPTs are generative models not agents, duh, why do people need an entire post to tell them that” and didn’t really expect the thing where agents are like “characters” in the world model to scale to high levels of capability. That seeming to happen raised the salience of the ideas a lot for me.
I’m currently thinking that it might be a good idea to publish a bunch of text that I think could help AIs make conceptual/philosophical progress quickly. Basically because it seems like (a) there could be a time period(like, uh, now) where AIs can do some reasoning on their own but don’t have good taste or autonomy in high-level research directions, so publishing incomplete stubs of ideas or promising things to look into could help them. (b) AIs that rely on human text more to make intellectual progress are likely to be more aligned than AIs that don’t, so this should differentially help more-aligned models. There is the risk that helping AIs in this way could empower misaligned models, but I think this is outweighed by factor (b) overall. And of course the most likely outcome is that whatever I publish simply isn’t that useful, but that seems ~neutral. Does anybody have any other considerations that could be relevant here?
If you’re still interested in trading(although maybe you’re not so much given the possibility of the impending singularity) maybe you should try polymarket, the returns there can be pretty good for smart people even if they have a lot of money. I 5x-ed in 2.5 years starting with four figures, but other people have done much better starting with a similar amount(up to 6 or 7 figures in a similar time frame), and my impression is that 2X for people with 7 figures should be achievable.
I noticed that Opus4.6 tried to get creative and actually write a fantasy-ish story whereas GPT5.2 mostly just elaborated on your notes, making its task a bit easier. So I tried prompting GPT5.2 identically but with “write a fantasy story in the style of Terry Pratchett” prepended and got this:
GPT5.2
The Refrigerator That Went the Wrong Way
Everyone knows that time is a river¹, except in certain parts of the city where it is more of a municipal plumbing problem.
The first indication that something had gone wrong with the fridge was the soup.
Tarin Gloss, junior time-magics student, senior refrigerator tinkerer, and unpaid intern at ChronoCool Solutions (Ltd., Pending, Hopefully), had placed a bowl of soup into the prototype unit at exactly seven minutes past noon. It had been piping hot, aromatic, and optimistically labeled Lunch.
At seven minutes past twelve and thirty seconds, Tarin opened the fridge again and found the soup… older.
Not cold. Not frozen. Older. The carrots had softened with the weary resignation of vegetables that had seen too much of the world. A thin skin had formed, the culinary equivalent of a sigh.
“Well,” Tarin said, because saying well is what sensible people say when reality begins quietly rearranging itself.
This was, admittedly, not the fridge’s intended function. The marketing document—three pages long and mostly exclamation marks—had promised temporal deceleration, preservation, and a revolution in baked goods. Instead, the fridge appeared to be aging things aggressively, like a pessimistic aunt.
Still, Tarin was not alarmed. Time did funny things in this city. Sometimes it paused for paperwork. Sometimes it reversed for tax reasons. Sometimes it just… sulked.
They did what any responsible chronomancer would do: they put a mechanical timepiece inside.
The watch came out having experienced approximately twelve years.
“That’s odd,” Tarin said, writing this down under Minor Irregularities.
ChronoCool Solutions was, after all, a startup. Irregularities were expected. Encouraged, even, as long as they could be rebranded as features.
The real trouble began the next day, when Tarin opened the fridge and found a loaf of bread already inside.
It was unmistakably their bread. Same flour, same slightly lopsided shape, same faint scorch mark from the oven that ran hot on the left. They took it out, frowned at it, and—after a moment’s consideration—put a different loaf in.
They closed the door.
They opened it again.
The original loaf was back.
There are moments in life when the universe clears its throat.
Tarin sat down very carefully.
Time travel, it turned out, fit the facts uncomfortably well.
The fridge did not slow time. It reversed it, locally, internally, and only while powered. Anything inside could loop. And anything that could loop could, with sufficient enthusiasm and poor judgment, exit itself.
Which is how Tarin met themselves.
The time-clone was standing in the kitchen, wearing Tarin’s coat, holding Tarin’s wallet, and looking extremely tired.
“Don’t panic,” the clone said, immediately causing panic. “I’m you, but later. Slightly. And more experienced. Also, we lost the money the first time.”
“What money?”
“The gambling money. Never bet everything on Thunderhoof. He looks honest. He’s not.”
There was, regrettably, no fantasy stock market. But there were horse races, and fantasy horses are subject to all the usual uncertainties plus at least one curse per season. A few hours’ foreknowledge went a long way.
The loop was only several hours long, which made it manageable. It merely extended the day, like an unexpected appendix. Tarin would bet, lose, step into the fridge, return earlier, adjust strategy. The second loop was better. The third was profitable.
The fourth was educational.
Covering for oneself turned out to be harder than time travel. ChronoCool still expected Tarin to show up to meetings, answer emails, and not behave as if they had personally stepped on causality’s toes. The winnings had to be put back where they belonged. Time-Tarin #2 generously handed over extra coins.
“For the next run,” they said. “Trust me.”
Tarin stepped into the fridge.
And did not step out again.
Time-Tarin #2 straightened their coat, smiled at the empty kitchen, and went to work.
Things continued in this fashion until the city’s racing authority noticed that someone was winning far too consistently and banned Tarin for cheating, temporal or otherwise.
At which point Tarin did the sensible thing.
They told a business partner.
The partner listened carefully, nodded, and said, “Oh good. You’ve figured it out too.”
They introduced Tarin to Protagonist #3, who was, technically speaking, Tarin #4, and who had already used the failsafe device—installed discreetly in a rented dockside storage unit—to go back and place the bets that had gotten them all banned.
“I thought you did that,” Tarin said.
“Oh no,” said Protagonist #3 cheerfully. “That was me. Sorry.”
The fridge hummed softly in the background, aging bread, preserving heat, and quietly turning time into a commodity.
After all, you could take money back— if you could buy and sell gold— and if you could buy and sell time, well.
That was just good business.
Not bad imitation of the tone, but it also seems to have gotten the bread stuff wrong, writing:
The real trouble began the next day, when Tarin opened the fridge and found a loaf of bread already inside.
It was unmistakably their bread. Same flour, same slightly lopsided shape, same faint scorch mark from the oven that ran hot on the left. They took it out, frowned at it, and—after a moment’s consideration—put a different loaf in.
They closed the door.
They opened it again.
The original loaf was back.
which seems confused.
Here’s GPT5.2′s response:
GPT5.2
Cold Storage
The fridge never cooled anything.
That should have been the first clue, but nothing about time magic was intuitive. You didn’t cool time; you slowed it, folded it, indexed it. Everyone knew that. The sales pitch for the ChronoVault™ was simple: bread stayed fresh because its internal clock barely ticked. Hot soup stayed hot. Flowers stayed exactly on the edge of wilting.
This unit did the opposite.
The loaf inside came out stale in minutes. Meat greyed. Milk curdled. The internal chronometer—standard diagnostic—showed months of elapsed time after an hour on mains.
Negative drift, they wrote in the lab notebook. Internal time velocity inverted.
They unplugged it and moved on.
The second time, the fridge already contained a loaf.
Same baker’s mark. Same scorch on the crust. Still warm.
They swapped it out, hands shaking, marked the new loaf with ink, shut the door, powered down.
Empty.
They didn’t sleep that night.
The realization came slowly, the way dangerous ideas always do—by trying very hard not to think them. The fridge wasn’t slowing time. It was looping it. Whatever entered could travel forward inside until it found an earlier state that matched the door opening.
You couldn’t see the loop. You could only trade with it.
They were a student, officially. Unofficially, a junior tinkerer at a startup that sold “domestic chronotronics” to people who liked words like bespoke and heritage. The unit in their apartment was a prototype. They weren’t supposed to take it home. They did anyway.
Fantasy fridges didn’t hum. They listened.
They powered it on at midnight.
Someone knocked on the inside.
The door opened and they stepped out—older, unshaven, eyes red from too many bad decisions made too close together.
“Don’t freak out,” the other one said immediately. “We’re on iteration three.”
The gambling plan was embarrassingly mundane.
There was no stock market, no commodities exchange, no arbitrage. But there were races—beast races, technically—public, legal, and aggressively audited. You couldn’t cheat them magically. That was the point.
But you could loop yourself.
Iteration one lost everything. Iteration two broke even. Iteration three made enough to notice.
Each loop added four hours to the day. Not reversed time—more time. Extra, privately lived time, stitched invisibly into the same morning.
The clone took notes. Not about the races—about themselves. Which instincts to ignore. Which hunches were poison. Which tells they unconsciously gave off when they were about to overcommit.
By iteration five, the clone was calm.
By iteration six, they were bored.
“You’re going to want to stop soon,” the clone said. “You won’t.”
They didn’t.
Covering their tracks was harder than the gambling.
They still had clients. Still had invoices. Still had a partner who noticed when they showed up with bags under their eyes and money they couldn’t explain.
The plan had been simple: work overtime, get paid, loop back, bet with foreknowledge, restore the original timeline’s finances. Clean. Elegant.
Instead, the advice came from the fridge.
The clone handed over winnings, plus extra—seed capital, they called it—then stepped back into the fridge.
“Your turn,” they said.
The door closed.
The fridge listened.
They woke up tired but solvent.
Their partner asked how the contract went.
“Fine,” they said, and meant it.
They didn’t notice the difference until later: the way memories felt slightly secondhand. Like notes copied from notes.
They didn’t think about it too hard.
The failsafe was sensible. Industrial-grade, rented anonymously, buried in a cargo storage unit at the docks. If something went wrong, you could always loop back past the current mess.
They waited a few days. Slept. Ate. Let the adrenaline drain.
Then they tried again.
And lost the betting license.
Too lucky, the officials said. Too consistent. Statistical anomalies. Cheating.
They laughed until it hurt, then stopped abruptly.
This wasn’t sustainable.
They needed capital. A buffer. Someone else to share the load.
They told their partner everything.
The partner sighed.
“Oh good,” they said. “I was wondering when you’d catch up.”
They introduced Protagonist Three.
Protagonist Three had already used the dock failsafe.
Protagonist Three was the reason they were banned.
After that, things accelerated.
Not the time loops—the meta-time. The rate at which procedures replaced decisions. The way meetings started to include agenda items like clone turnover and identity continuity. The way no one was quite sure who had actually founded the company anymore.
They stopped naming iterations.
They stopped pretending there was an original.
The fridge business flourished. Of course it did. You could beta-test designs against yourself. You could pre-debug failures by living through them and discarding the branch.
Someone suggested franchising.
Someone else suggested internal markets—loops traded for favors, favors for future knowledge.
Someone asked, quietly, whether you could put people in long-term storage.
The fridges listened.
Much later—much later, measured in layers rather than years—someone finally asked the forbidden question:
“If we stop looping,” they said, “who survives?”
No one answered.
Because by then, stopping wasn’t a moment.
It was a direction.
Pretty straightforward expansion of your notes. No obvious conceptual errors that I noticed, although some of what it says doesn’t really make sense(“someone asked whether you could people in long-term storage”...isn’t this what they are already doing?). Not really compelling as fiction, replete with GPT-isms. Generally speaking GPT has seemed smarter than Claude to me since GPT5.
[Epistemic status: vague speculation] I like the idea of consciousness being allocated based on projecting influence backwards into the past. Humans are currently the most conscious beings because we are the densest nexus of influence over the future, but this will eventually change. This seems to have the requisite self-consistency properties. e.g. if you are aiming to have a large influence over the future it’s probably important to track other beings with the same property.
ETA: another perhaps better possibility is that consciousness is about being a bottleneck of information between the past and future.
I’d be interested in hearing more about Ryan’s proposal to do better generalization science(or if you don’t have much more to say in the podcast format I’d be interested in seeing the draft about it)
He’s talking about the stuff around the simulation hypothesis and acausal trade in the preceding section.
Yeah “esoteric” perhaps isn’t the best word. What I had in mind is that they’re relatively more esoteric than “AI could kill us all” and yet it’s pretty hard to get people to take even that seriously! “Low-propensity-to-persuade-people” maybe?
but “extremely unlikely” seems like an overstatement[...]
Yes this is fair.
Interesting. I’d wondered why you wrote so many pieces advising people to be cautious about more esoteric problems arising from AI, to an extent that seemed extremely unlikely to be implemented in the real world, but there being a chance simulators are listening to your arguments does provide an alternative avenue for influence.
Good post about LLMs scheming against their stupid human overseers.