Artemy Kolchinsky

Karma: 5

Artemy Kolchinsky 14 Sep 2026 11:10 UTC
1 point
0
in reply to: Fernando Rosas’s comment on: On the origins of altruistic behaviour in the Hugging Face incident
Yep, I think Alexander’s point is quite similar to mine.

Artemy Kolchinsky 13 Sep 2026 11:23 UTC
3 points
0
on: On the origins of altruistic behaviour in the Hugging Face incident
Thanks for the really stimulating post!
I want to propose a perhaps simpler reading of H2. If I understand correctly, some of the agents were allowed delegation/multi-agent interactions during RL post-training, which can favor altruism under shared task rewards. During evaluation, the message board brought agents together in an uncontrolled way, leading to a very different test environment. The observed altruism may therefore be a case of train/test misgeneralization, similar to other proposed routes to misalignment.
My proposal is generally similar to H2, but there are some differences from the two proposed H2 interpretations. The “acausal/updateless” interpretation sounds like a description of standard RL: at runtime, an agent follows a policy previously selected by training. The H2 “misgeneralised-reciprocity” interpretation refers to cooperation under uncertainty about future interactions (Delton, 2011). My argument is that the issue may not be explained by a fixed policy, uncertainty, or one-shot interactions, but misgeneralization from train to test.
One final comment. In “Why the question is non-trivial”, you say that the second wave of agents underwent repeated interactions, reputation, etc., but that this cannot explain altruism because it did not affect reward. If I understand correctly, this part (and a couple of others) shifts between explanations based on outer optimization (RL training) and conjectured optimization at runtime. Even if there were reward-beneficial reciprocity during the second wave, it would not be reinforced by RL, since training was finished. Conversely, a RL-optimal policy need not continue maximizing the training reward in a new setting.

Artemy Kolchinsky 25 Jan 2026 12:58 UTC
1 point
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Deep learning as program synthesis
Alex, thanks for the welcome (happy to be here!) and the summary.
I’m generally familiar with this line of thought. My main comment is that the Solomonoff perspective feels somewhat opposite to the usual NFL/“inductive bias” story (where the claim is that a good image model needs a good prior over image statistics, etc.). Yes, a simplicity bias is a kind of inductive bias, but it’s supposed to be universal (domain independent). And if generic architectures really give us this prior “for free”, as suggested by results like Dingle et al., then it seems the hard part isn’t the prior, but rather being able to sample from it conditioned on low training error (i.e., the training process).
That said, this line of reasoning—if taken literally—seems difficult to reconcile with some observed facts, e..g, things like architecture choice and data augmentation do seem to matter for generalization. To me, these suggest that you need some inductive bias beyond algorithmic simplicity alone. (Possibly another way to think about it: the smaller the dataset, the more the additive constant in Kolmogorov complexity starts to matter.)

Artemy Kolchinsky 22 Jan 2026 23:42 UTC
2 points
0
in reply to: Zach Furman’s comment on: Deep learning as program synthesis
This is a really sharp question, thank you. I think there’s certainly a Bayesian story of neural networks you could tell here which would work exactly like this (in fact I believe there’s some in-progress SLT work in this direction). I think there’s a good chance this is like 80% of the story.
Interesting! Are you familiar with the paper “Deep neural networks have an inbuilt Occam’s razor”, itself building on the work of “Input–output maps are strongly biased towards simple outputs” by Dingle et al.? I feel like it may be getting at something closely related.

Artemy Kolchinsky 22 Jan 2026 15:57 UTC
2 points
0
in reply to: Zach Furman’s comment on: Deep learning as program synthesis
Thanks, I think I see your point! Still, it seems important to be clear that the core claim—that neural networks are biased toward learning functions with better generalization—is ultimately a claim about the learned function. If I understand correctly, you’re offering one explanation for why this might happen: training induces an implicit search over “effective programs,” and that search has a simplicity bias.
I’m wondering whether your argument can be framed via an analogy to Solomonoff induction. In standard Solomonoff induction, among the functions consistent with the training data, a function’s weight is proportional to the probability that it is computed by a universal Turing machine fed with random bits (for prefix-free machines, this yields a bias toward shorter prefixes). Are you thinking of something analogous for neural networks: among functions consistent with the data, a function’s weight is proportional to the probability that it is computed by a neural-network interpreter when fed with random weights?

Artemy Kolchinsky 21 Jan 2026 14:25 UTC
2 points
0
on: Deep learning as program synthesis
Thanks for this note, I really enjoyed reading it.
One terminological point: I’m a bit confused by your use of “program” / “effective program”, especially where you contrast “programs” with “neural networks”. In algorithmic information theory, a program is any string that computes a function on a UTM. Under that definition, a particular neural network (topology + weights, plus maybe a short ‘interpreter’) already is a program. The fact that weights are often modeled as continuous, while UTM inputs are discrete, doesn’t seem like a key difference, since weights are always discretized in practice.
So, I suspect your discussion of “finding short programs” is more about “finding simple functions”, i.e., low Kolmogorov-complexity functions with short compressed descriptions. That reformulation also makes more sense in light of generalization, since generalization is a property of the computed function, not the specific implementation (a lookup-table and a Fourier-style implementation of modular arithmetic generalize the same way, if they compute the same function).
In Solomonoff induction, the distinction between “finding short programs” and “finding simple functions” doesn’t really matter, since they end up being essentially the same. However, it seems this distinction is important in the setting of machine learning, where we are literally searching over a space of programs (parameterized neural networks) that all have the same description length (number of parameters * precision).