philh

Karma: 8,606

philh 3 Aug 2026 12:50 UTC
4 points
0
in reply to: Steven Byrnes’s comment on: [Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning
I still disagree, but I think I’m going to try to get an executable model and see if that helps us hash it out :)

philh 29 Jul 2026 16:43 UTC
LW: 2 AF: 1
0
AF
in reply to: Steven Byrnes’s comment on: [Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning
So it’s probably worth distinguishing a few questions:
1. 1. Are there possible LTP setups where the STP is predicting “what’s the next override”?
  2. Are there any of those that are biologically plausible? (Like, it’s not surprising if an LTP configured like this actually exists in the brain.)
  3. Do any of them actually exist?
2. 1. Are there possible LTP setups where the STP isn’t predicting “what’s the next override”?
  2. Are there any of those that are biologically plausible?
  3. Do any of them actually exist?
3. 1. In any specific hypothetical example, is the STP predicting “what’s the next override”?
  2. Is this example biologically plausible?
  3. Does it actually exist?
I think for the (x.3) questions we’d need data we don’t have, so I’ll forget about them for the rest of the comment, they just seemed worth noting.

We both agree the answer to (1.1) is “yes”. In particular, this is what happens when the output of the STP doesn’t affect the next override. Like, if you’re on a rollercoaster, using an LTP to predict which direction to brace, and the overrides are sudden accelerations.

(1.2) I’m actually not sure about… not a confident “no”, but this setup feels to me like it would be kinda unlikely? Whereas I think you think these think are every biologically plausible LTP setup. This is just vague intuition though, and not cruxy for me.

(2.1) and (2.2) I think the answer is yes, and I think you think at least one of them is no.

So let’s look at some specific examples.

For the digestive enzymes, I actually think the binary model is already a biologically plausible example where the STP doesn’t predict the next override. But I’m happy to stick with the quantitative model too.

You’re brushing this aside, but to me it’s load-bearing. The R=1 output leads to more enzymes than R=0, and maybe even that small amount will wind up being too much. Probably not, but you only need that to happen 10% of the time.

I don’t think the numbers work out here. Compare R=1 to R=2. If these are predictions, then R=1 is a higher [predicted probability that the next override is R=0] than R=2. But R=1 produces fewer enzymes, which makes the R=0 override less likely. I think this is the case no matter what the specific curves look like, as long as they’re monotonic in the right direction.

It might be possible to come up with some way to use an LTP to handle digestive enzyme production, and have the STP predict the next override. But I feel like it doesn’t happen by default, and there’s no need for it. The job of the LTP is to handle digestive enzyme production; there’s no pressure towards “it should be possible to use the STP’s output to predict the next override”, so that doesn’t happen.

For the go-karting, I think the question is “how much do I steer?”
- If I don’t steer at all in response to my predictions, then the STP can be straightforwardly correct.
- If I steer a small amount, then maybe I’m still more likely to hit the side the STP predicts. Like, it thinks I’m 90% likely to hit the left side, and in fact I’m 70% more likely. We can still think of the STP as predicting the next override, but we need to recalibrate the function that reads an STP output and tells us a probability.
- Ideally, I steer just the right amount, and I’m now equally likely to hit either side. The STP output is now uncorrelated with “which side do I hit”.
- If I steer too much, I’m now more likely to hit the other side, and… ??? I’m not sure about the dynamics here, especially because in the time between “I oversteer” and “I actually hit the other side”, the STP is likely to be freaking out and (correctly) predicting that I’m now about to hit the other side, and I just don’t have time to react to that.
Connecting this to the digestive enzymes… both examples are going to have some kind of approximately-steady-state, like “producing a low-ish level of enzymes” or “keeping the steering wheel at a fixed angle while the track has a fixed curve”. From there, maybe there’s enough randomness that you might make too many enzymes or too few / hit either side of the track, but most likely that’s not going to happen, at least not much. (At least, we can’t assume that’s going to happen much. A simple model might have it not happening at all, and more complicated models can have it happening different amounts depending on parameters, and “it doesn’t happen much” is a reasonable setting for the parameters.)

At some point I’m going to sit down to dinner / the track is going to sharply veer left. (Definitely left, not right.) I know that’s going to happen at some point, and if my steady-state is steady enough, that means the most likely next override is “I’m not making enough enzymes” / “I’m going to hit the right wall”. But the STP isn’t going to be outputting “make more enzymes” or “turn harder left” yet, because it’s not coming up yet.

(I’m hoping to make an executable model we can play with, so if you still disagree then maybe the productive thing is to wait until I have that and that might help us figure out what’s up. Though it’s not definite I’ll succeed.)

philh 24 Jul 2026 11:15 UTC
2 points
0
in reply to: philh’s comment on: philh’s Shortform
Indeed, no luck:
(It probably could have used more time in the freezer, but it’s clearly not freezing in the fridge.)
One hypothesis is that my water is too hard (269 ppm calcium carbonate) or otherwise impure. If so, possible avenues for fixing it are buying distilled water, buying regular bottled water, and using a water filter (quick google: apparently that doesn’t help with hardness).
Another hypothesis is that something about my salts is fucked. Impure sodium sulphate? The table salt comes with anti-caking agents, are those causing a problem? The table salt has also been just hanging out in my kitchen in this kind of thing, possibly with some dried beans in there to stop it clumping; maybe I should try some fresh salt?
I’ve done three experiments so far, and very roughly:
- 3:1 Na2SO4:NaCl by volume (most recent): I got 210 ml Na2SO4 in 1 l water (7x15 ml, in 500 ml)
- 4:1 Na2SO4:NaCl by volume (Nighthawk’s recipe): I got 160 ml Na2SO4 in 1 l water (400 ml, in 2.5 l)
- No NaCl: I got 400 ml Na2SO4 in 1 l water (80 ml in 200 ml)
So there’s not even a clear trend for “more NaCl means less Na2SO4 dissolved”. Maybe because I did nighthawk’s at close to boiling? And/or because it went via the kettle?
Still, regardless of what my numbers show, more NaCl must mean less Na2SO4 dissolves, right? So maybe the next thing to try is, like, 10:1?

philh 23 Jul 2026 15:59 UTC
LW: 4 AF: 3
0
AF
in reply to: Steven Byrnes’s comment on: [Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning
Hm, I think that still ends up with the same basic dynamic of “if the next override depends on the current prediction, you’re not actually predicting the next override”.

Like, suppose it’s been an hour since I ate and I’m not thinking about food. There’s no override, and the STP is outputting some small number, say R=1. What’s the next override going to be?

I think the basic story is something like,
- At some point in future, I start to think about food
- the STP starts to output a high number like R=9
- I start eating
- a “start making enzymes” R=10 override comes in
- I stop eating
- the STP starts to output a low number again, like R=1
- a “stop making enzymes” R=0 override comes in
In this story, the initial output of 1 is “correct”. But it’s not a prediction of the next override, which is probably going to be 10. (It could be 0, but only if I incorrectly start to produce more enzymes in the meantime, the STP had been incorrectly outputting a high number.)

So as long as it’s the case that “we don’t get an R=0 override unless we actually make too many enzymes”, I think the STP isn’t predicting the next override. What’s it predicting instead? Possible answers:
- “How many digestive enzymes are going to be removed from the stomach in the next few minutes”? (Obviously only applies to this specific LTP configuration.)
- “If this prediction changed, would we get an override soon”? (I think this is an accurate question, at least in this case? If we’re “correctly” predicting low, and we started to predict high, then we’d make too many enzymes and get a “stop” override. If we’re “correctly” predicting high, and we started to predict low, then food would come in and we’d have too few enzymes and get a “start” override. I’m not sure this is accurate in all possible LTP configurations.)
- “It’s not predicting anything, it’s just part of a control system, though depending on details it can be useful to think of the STPs in these systems as predicting things like [the above two questions, plus “what’s the next override”, plus probably others]”?
(Another way this LTP could be configured would be to give an override every, say, ten minutes. Then the override doesn’t depend on the prediction, as long as my actions don’t depend on it. That would be biologically different, and I don’t think you’re claiming it’s what actually happens, at least not in all LTPs. If the interval is too short, then I won’t start making enzymes far enough in advance; if it’s too long, I’ll keep making them long after they’re not needed.)

philh 23 Jul 2026 11:57 UTC
2 points
0
in reply to: philh’s comment on: philh’s Shortform
I found a bunch of recipes online and compared them: https://docs.google.com/spreadsheets/d/1uroXRGgT6fNKtQ9iAyBuVrW8TJlRNA5d6sPmRir-4kY/edit?usp=sharing

By weight, the Na2SO4:NaCl ratios range from 2 to 5, so maybe it’s not very sensitive to that? (Nighthawk’s is 5, and Claude’s suggested recipe has 5.5.)

But Nighthawk is a wild outlier for H20:Na2SO4. He gives 3.3x by weight, every other recipe is 1.2 to 1.3.

But if Linda and I had too much water, why didn’t it all dissolve? Claude’s “probably because of the NaCl” sounds weird to me because Nighthawk has less NaCl-to-Na2SO4 than any other recipe. Hypothesis: we just didn’t give it long enough / stir well enough.

I’m going to try making 2E-3SPA from this paper, which (by weight) is 51% water, 39% Na2SO4, 10% NaCl. (They use sodium polyacrylate as a thickener and borax as a nucleating agent, which I’m substituting with xanthan gum and “nothing at first, I’ll try diatomaceous earth if I need one because borax is hard to get in the UK. 2E-3SPA has higher latent heat than the other one they make, and similar melting point.

So recipe I plan to try, mixing at ~40°C:
- 500 ml water
- 390 g Na2SO4
- 95 g NaCl
- 5 ml xanthan gum
Except that I’m going to add the salts gradually, instead of all at once, and keep going at least until they stop dissolving. So in practice I’ll do… it’s going to be convenient to use measuring spoons, not scales, so probably 15 ml Na2SO4 and 5 ml NaCl at a time, which should be 24 g and 6 g. That gives me a bit less NaCl than the recipe, but this recipe was on the high side for NaCl.

(But I should check the density of my salts.)

If they seem to stop dissolving long before I think I’ve put enough in, then, uh, I dunno.

Update:

First complication is that my scales and my measuring jug don’t agree with each other. I trust the jug more than the scales (which are cheap “rotary dial and spring” ones that I’ve had over 10 years), though I didn’t specifically check the jug versus my measuring spoons. 200 ml water (according to the jug) was about 175 g, and 400 ml was about 350 g.

Then 45 ml of Na2SO4 (according to my measuring spoons) was about 60 g on the scales. If we scale that up by ⁸⁄₇ we get 69, which is close enough to the 72 I was expecting.

But 45 ml of NaCl was also about 60 g on the scales. Maybe more like 55 g.

I think the scales are just not good enough to confirm or disconfirm anything. I was expecting an actual weight of 54 g which would be 47 if the scales have a ⁷⁄₈ multiplier.

What’s probably going on is that the scales just aren’t very consistent, especially near 0, and also not very precise, and I can’t use them to confirm or disconfirm much of anything.

Anyway, I took 500 ml water and added 15 ml Na2SO4 + 5 ml NaCl at a time, except 3x those quantities for the first load. I’d expect about 16 loads (16 * 15 * 1.6 = 384 g Na2SO4). After 7 loads it seemed like there was a small-but-clear amount not dissolving any more, and after 8 there was a lot not dissolving even though I gave it at least 10 minutes with plenty of stirring. So, uh, wtf.

At that point I started adding xanthan gum. 10 ml, not 5, because I noticed 5 was a miscalculation. I’m letting it cool and set now. I’m not optimistic it’ll work.

philh 19 Jul 2026 9:27 UTC
5 points
2
on: Why I Left Google DeepMind
Respect and admiration for leaving, but perhaps even moreso for your actions before then.

philh 18 Jul 2026 10:09 UTC
LW: 4 AF: 3
0
AF
on: [Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning
we are training the short-term predictor to always output what the next override will be

Hm, I’m going to nitpick this as a summary, partly as a way of helping me think through this myself.

Do we get an override if we’re currently predicting right? I think the answer is “it doesn’t matter”.

That is, there’ll be some neuronal fact of the matter. But if we treat this set of neurons as a black box, it’ll implement the same algorithm either way—the short-term predictor can’t tell whether the signal it’s getting matches its output because the switch is in defer mode, or because the switch is in override mode and it’s overridden to the same value as its current output.

I think this summary relies on us pretending/assuming/modeling “yes, we get an override even if we’re currently predicting the right thing”. Otherwise, the next override will always be the opposite of what we predict.

But I’m not sure we can always model it that way. “What’s the next override” might depend on “what am I currently predicting” in a way that confuses things.

Taking the digestive enzyme example, when do we get an override? According to above:
1. If we’re currently eating something: override to “make enzymes”.
2. If we’ve been making enzymes for a while, and not eating: override to “don’t make enzymes”. ^[1]
We can model (1) as getting an override even if we were predicting enzymes. But I don’t see how to model (2) as getting an override even if we were predicting no-enzymes. The “we’ve been making enzymes for a while” is an important part of it.

(What’s the fundamental asymmetry here? I think “we’re trying to start producing enzymes before we need them, harder than we’re trying to stop producing enzymes before we stop needing them”.)

So if we’re not currently predicting enzymes, the next override is always going to be “start making enzymes”. But we don’t always predict enzymes, because if we did, we’d (typically) get a “no-enzymes” override.

So maybe a more-accurate-but-more-confusing summary would be: we’re training the short-term predictor to output what the next override would be, if we were currently predicting “wrong”?
1. ↩︎
  Possibly implemented as something like: “normally make a certain amount of digestive enzymes, which are normally removed from the stomach at rate proportional to their concentration, which gives a particular steady state. If we start digesting food, they’re removed faster, the level falls below a trip point, and we override to “make [more] enzymes”. The level goes up again to a (possibly different) steady state. When we stop digesting food, the level goes above a trip point, we override to “stop making [more] enzymes”, and the lever goes back down to the first steady state.” The override would be on whenever we’re above/below the given thresholds.

philh 15 Jul 2026 7:45 UTC
14 points
0
on: philh’s Shortform
Following this video, @Linda Linsefors and I tried to make a phase change material with a freezing point around 18°C.
Report:
For our first batch (which is all we’ve made so far), we started with 2 l water / 400 ml sodium sulphate / 100 ml table salt. There was a lot of salt left over after several minutes of stirring boiling/near-boiling water, so we added another 500 ml water. There was still a lot of salt but we stopped there.
We added 30 ml xanthan gum. (By the recipe it should have been more like 40 because of the extra water. The goop we ended up with is pretty good goop consistency, but I do think I’d prefer slightly thicker.) The first teaspoon clumped up a lot in the water, so later ones we ran through a sieve which helped a lot.
We boiled and stirred that for about 30 minutes and left it to cool.
Note that sodium sulphate comes in “anhydrous” and “decahydrate” forms. The latter already has 10 water molecules attached, so less of it dissolves and you get less actual salt. We used anhydrous. I kind of assume the guy in the video used that too (it would be weird if he got better dissolving than us with decahydrate) but idk if it’s clarified anywhere. It’s presumably possible to use decahydrate if you adjust quantities appropriately.
We got something that kinda worked, but I was suspicious that it wasn’t behaving like it should, so I got a temperature logging probe (note: needs initial setup with software that’s available for windows or mac, not linux; I used wireshark to capture the USB traffic so maybe I can make linux software but it’s not a priority). Here’s the graph when I put some of it (at a guess about 50 ml) in the freezer and then thawed it:
So yeah. The thing around 3-5° going down could be either freezing or “freezer is temporarily off (as I interpret the rises later on) but ambient temperature is low enough to keep cooling”. Claude thinks there wasn’t nearly enough sodium sulphate, and it’s basically just salty water, which freezes at that temperature; and which apparently has a smooth melting curve. Conversation (I mistakenly said 60 ml xanthan gum there). I’m not very convinced by its recommendations.
---
We separately wanted to see what happens when you just mix sodium sulphate and water, because something that freezes at 32° sounded cool. We started with 200 ml water, tried to keep it around 40°, and added sodium sulphate until it stopped dissolving. That ended up being about 80 ml. It should support about 100 g. Google gives density of the powder as 1.4-1.6 g/ml (a crystal is about 2.7 g/ml) so we’d be at about 120 g, there are enough errors in the process that this doesn’t seem off to me.
What we ended up with was something that freezes in the fridge, but not at room temperature (which was about 28° when we tried). I’m currently thawing it so I can put the probe in, and then I’ll freeze-and-thaw it again and see what happens.
I think it took a few freeze-thaw cycles before it seemed like it would fully freeze? First time it looked more like “clear ice with some water not yet frozen”, now it looks like white ice.
e: okay, it hasn’t thawed at all after several hours. I’m pretty sure it just actually freezes above room temperature, and what I saw before was probably just some combination of supercooling and “not fully saturated” and idk what else.

philh 7 Jul 2026 16:02 UTC
2 points
0
in reply to: philh’s comment on: Sum-threshold attacks
I’d been thinking of writing a more detailed post about this, but https://asianintp.wordpress.com/2017/11/16/the-boiling-frog-tale-is-not-a-myth/ covers it I think.

philh 22 Jun 2026 14:26 UTC
2 points
0
in reply to: Alex A’s comment on: Coup is the Pareto-optimal social game
I don’t remember ever wondering this, fwiw.

philh 22 Jun 2026 14:24 UTC
2 points
0
in reply to: Daniel Tan’s comment on: Coup is the Pareto-optimal social game
Oh, interesting. So every time someone loses influence, you put that card back in the stack in the middle?
Do you do that with fewer players too? That feels like it changes the endgame significantly with larger group sizes.

philh 22 Jun 2026 10:09 UTC
2 points
0
on: Coup is the Pareto-optimal social game
How do you play with seven? Do you just have only one card for the Ambassador to swap with?

philh 19 Jun 2026 10:15 UTC
7 points
4
on: Contra Pace on When to Apologize
On the other hand, if the organizer says, “Hey, I am taking these costs that you have faced, and I’m putting them on my ledger; I owe it to you to make you whole,” that would seem to imply that they don’t stand by the catering decision and will endeavor to get kosher food at future conferences. What would it even mean to purportedly accept the cost “on one’s ledger” but not change one’s behavior going forward?
For example, they could
- marginally update their policy, so that they’re more-likely-but-still-unlikely to get kosher food at future conferences
- donate $100 to an org that works to make kosher food more accessible to conference organizers
  - but also, just saying “sorry” probably has a mild effect in this direction? It’s a public acknowledgement that ideal conferences supply kosher food
- offer kosher-keeping attendees a daily stipend to get their own food
- if anyone decides not to attend because of the lack of kosher food, reimburse them for costs they’ve already incurred (like plane tickets)

philh 15 Jun 2026 16:48 UTC
25 points
0
on: philh’s Shortform
(Crossposted)
About a month ago I left my job. I plan to take the next six months (until December) as a sabbatical.
Partly because I don’t think I’ve taken more than two weeks’ holiday at a time since 2012, when I got my first real job. It’ll be nice to have a longer break.
Partly because, man, the world is not standing still. 2026-2031 is going to look very different from 2016-2021, and probably I should spend a bit of time orienting to “what do I even do with my life, going forwards”. It’s not impossible for me to do that while I have a full time job, but I do think I’ll be better at it without one.
I don’t plan to make any long term commitments until December. But some things I’d like to do in that time:
- See if I can help with AI notkilleveryoneism.
  - Do some kind of study course.
  - The slightly-more-concrete fields that I think I’m most likely to get into are mech interp (possibly working directly with my girlfriend), agent foundations, and brain-like-AGI safety.
  - I don’t want to pay much attention to “what can I get paid for” right now. That seems like it would distort my thinking, and my sense is that the funding landscape is likely to look very different in six months anyway. If it turns out that I think I can do something worthwhile but I can’t get paid to do it, I’ll have to figure out what to do about it later.
- Blog more. (Vaguely aiming for 10 posts before December, including this one.)
- Write some fiction.
- Get better at LLM-assisted coding. (I’ll probably need to start paying for an LLM subscription. I’m a little sad about that because I don’t want the labs to get my money, but I don’t think that being a lonely holdout here would help much.)
It’s been a month and I’ve done… approximately none of that. Partly that’s because I’ve spent 2-3 weeks out of four either on holiday, preparing for holiday, or recovering from holiday. But I think it’s time for me to buckle down a bit. I’m going to try “get something productive done approximately every weekday”. “Productive” can include studying, writing, coding, home improvements, or anything else that either I or my girlfriend decides counts. I don’t yet know how much I need to do to count. For today, writing this blog post was sufficient.
I’m also open to taking on some short term paid work, if anyone wants to offer it. I’m open to writing code (especially Haskell), book reviews, literature reviews, tutoring, probably other stuff that doesn’t come to mind right now.
This is my first long period of self-direction since… arguably “since ever”? It’ll be interesting to ~~see~~ choose what I do with it.

philh 19 May 2026 13:47 UTC
2 points
0
in reply to: Mateusz Lichota’s comment on: London Rationalish meetup—Lincoln’s Inn Fields
We had to reset the link because of spambots. I’ve sent you the new one by DM. Anyone else who wants it can message me.

philh 11 May 2026 17:11 UTC
5 points
1
in reply to: Yair Halberstadt’s comment on: Bad Problems Don’t Stop Being Bad Because Somebody’s Wrong About Fault Analysis
Yes but to add an epicycle: if AWS is down, my customers might have so many other problems to deal with that they don’t notice that I specifically am down, and they wouldn’t have any use for me if I wasn’t.

philh 20 Apr 2026 17:02 UTC
2 points
0
in reply to: dr_s’s comment on: Let goodness conquer all that it can defend
The one thing it’s hard to imagine anyone stopping was the smallpox and such. It would have happened sooner or later unless the entire progress of technology was put on halt. Mass deployable vaccines before transoceanic sail ships seem unlikely. Someone would have gotten to the other side and even had their intentions been the best possible, people would have died out of sheer ignorance.
I don’t know if you need vaccines. From memory, the incubation time of smallpox is about half the transatlantic voyage time (2 weeks versus 4 weeks)? I don’t know how much we know about how the first smallpox case crossed the Atlantic, or whether it happened more than once. But I wonder whether a policy on long voyages of “if we notice someone on board has smallpox, we make them walk the plank to protect everyone else on board” (not even thinking about the people at the destination) might have been a) possible in a nearby world and b) effective?
(I’m vaguely aware that the voyage wasn’t nonstop, they’d go via e.g. the Azores(?), presumably with a chance to pick up smallpox there. Maybe that sinks the idea.)

philh 11 Apr 2026 11:24 UTC
2 points
0
in reply to: Lorxus’s comment on: Does descaling a kettle help? Theory and practice
Why do you recommend it?
I actually have a new kettle now, and scale is either a bigger problem or a more visible one, so I’ve been descaling it every few months give or take. The main thing I dislike about vinegar is that (if I interpret my observations correctly) it boils at a lower temperature than water, and the kettle trips off at 100° instead of at “my contents are boiling”; so if I don’t wait there to turn it off then it boils over. If citric acid doesn’t have that specific problem then I’d happily swap.

philh 26 Mar 2026 23:32 UTC
2 points
0
in reply to: Benquo’s comment on: Is fever a symptom of glycine deficiency?
Yeah, I think I’ll keep trying 3g/night for a bit and then maybe explore 10g/day. That feels like a pretty inconvenient dose to be taking, but so it goes.
I had been wondering how much of the effect comes from the timing and how much comes from just getting more. There must be a time delay in the process of “glycine enters your stomach, makes its way into your bloodstream, gets incorporated into glutathione, and that helps clear ROS”. But I figured, sleep lasts a while, it could well be that the glutathione starts doing its work an hour after you take the glycine; and maybe if you take it earlier in the day, it’s all used in other things by the time you get to sleep.
But “no, 3g just before sleep isn’t enough at the right time to help with sleep” is also not surprising.

philh 26 Mar 2026 0:19 UTC
6 points
0
in reply to: philh’s comment on: Is fever a symptom of glycine deficiency?
Night 1: I got to sleep quickly, woke briefly after 1 ¹⁄₂ hours, woke up early again with muscle cramp (I expect this at least to be unrelated). I’d rate as “worse sleep than normal”. Didn’t notice myself being out of distribution for daytime alertness conditioned on sleep.
Night 2: got to sleep quickly, woke early, but sleep quality seemed fine. Not obviously out of distribution for alertness.
Nights 3-8: failed to keep detailed logs. I think I’ve slept quickly every night (good if related), and often woken up early (bad if related). Neither of those is very out of distribution. Been mildly sick which probably confounds things too. Inclined to keep taking it.