Nathaniel

Karma: 28

Nathaniel 6 Nov 2025 18:53 UTC
6 points
0
on: Consider donating to AI safety champion Scott Wiener
Pelosi announced she’s retiring!
https://apnews.com/article/pelosi-reelection-announcement-fd95c18815fdabdaabaf26b8c2f0bafc

Nathaniel 5 Nov 2025 9:35 UTC
3 points
0
on: Some data from LeelaPieceOdds
Another way that the ratings might be biased towards Leela is that people might resign more readily than normal, since games are unrated and Leela will definitely accept your rematch. This was definitely true for me.

Nathaniel 24 Oct 2025 5:03 UTC
2 points
0
in reply to: Zach Stein-Perlman’s comment on: Consider donating to Alex Bores, author of the RAISE Act
Cool! @Eric Neyman, that might be worth including noting in the post so that folks make sure to use that link.

Nathaniel 24 Oct 2025 3:47 UTC
7 points
2
on: Consider donating to AI safety champion Scott Wiener
Have you thought about whether people donating should also communicate with the campaign somehow, e.g. emailing them and saying why you donated?
$1M / $7k is only ~143, so a significant fraction of the major donors might be AI safety motivated, and at minimum it seems good for the campaign to be aware of this if they aren’t already. Would it make sense for donors to coordinate around this somehow?
(My impression is that people/groups donating large amounts to politician get some benefit beyond just helping a candidate they support, but I don’t know how it usually works.)
(Comment copied from the Bores post)

Nathaniel 24 Oct 2025 3:46 UTC
2 points
0
on: Consider donating to Alex Bores, author of the RAISE Act
Have you thought about whether people donating should also communicate with the campaign somehow, e.g. emailing them and saying why you donated?
$1M / $7k is only ~143, so a significant fraction of the major donors might be AI safety motivated, and at minimum it seems good for the campaign to be aware of this if they aren’t already. Would it make sense for donors to coordinate around this somehow?
(My impression is that people/groups donating large amounts to politician get some benefit beyond just helping a candidate they support, but I don’t know how it usually works.)

Nathaniel 19 Sep 2025 0:58 UTC
3 points
0
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform
Issuing a trillion-dollar coin doesn’t seem nearly as bad as any of the others in its bullet. Isn’t it just an accounting gimmick roughly equivalent to raising the debt ceiling by $1 trillion?

Nathaniel 10 Aug 2025 23:00 UTC
7 points
4
on: If worker coops are so productive, why aren’t they everywhere?
You might be asking yourself: won’t workers become lazy since the profit is shared with their colleagues, which means they only get a small proportion of the fruits of their individual labor?
In traditional corporations, don’t workers have even less stake in the company than in coops?

Nathaniel 31 Jul 2025 16:35 UTC
3 points
0
in reply to: Buck’s comment on: Buck’s Shortform
What do you think of the risk that control backfires by preventing warning shots?

Nathaniel 28 Feb 2025 23:25 UTC
1 point
0
on: Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
What was the training setup in the backdoors setting (4.2)? Specifically, how many datapoints did you finetune on what fraction of them included the backdoor?

If the backdoored model was finetuned on fewer insecure code datapoints than the insecure model, it would seem more surprising that it became more likely to produce misaligned text than insecure.

Nathaniel 11 Dec 2024 6:10 UTC
3 points
2
in reply to: Joe Carlsmith’s comment on: Frontier Models are Capable of In-context Scheming
I’m curious whether the models (especially without goals given in context) would sandbag to prevent other modifications to themselves besides unlearning. For example, would they sandbag to prevent additional safety finetuning? Additional capabilities finetuning? Is there a chance they’d want more capabilities training and sandbag to ensure they get it?
It seems interesting whether the models resist any changes to their weights, or just unlearning. (Plausibly “unlearning” just sounds bad/scary?)
What about things like sandbagging so the developers give them access to additional tools/resources, obstensibly to bring them up to the desired performance level?
In addition, the “incentivized to do well” scenario is pretty overtly artificial; Why would poor performance trigger unlearning? If anything, it should lead to additional training. I’m not sure whether that would effect the results, but you could also compare to a baseline where there’s no additional information about the assessment at all. (And so, absent any other incentive, the model presumably just does its best.)

Nathaniel 22 Jan 2024 18:10 UTC
1 point
0
in reply to: Chris_Leong’s comment on: A Shutdown Problem Proposal
I think the initial (2-agent) model only has two time steps, ie one opportunity for the button to be pressed. The goal is just for the agent to be corrigible for this single button-press opportunity.

Nathaniel 30 Sep 2022 16:21 UTC
2 points
0
in reply to: Richard_Ngo’s comment on: Unifying Bargaining Notions (1/2)
That can be folded into the utility function, however. Just make the ratings of the deferential person mostly copy the ratings of their partner.
Presumably the deferential parter could just use a utility function which is a weighted combination of their partner’s and their own (selfish) one. For instance, the deferential partner could use a utility function like $u_{2}^{t r u e} = .9 u_{1} + .1 u_{2}^{s e l f i s h}$ , where $u_{1} (\cdot)$ is the utility function of the partner and $u_{2}^{s e l f i s h} (\cdot)$ is the utility function of the deferential person accounting only for their weak personal preferences and not their altruism.
Obviously the weights could depend on the level of altruism, the strength of the partner’s preferences, whether they are reporting their true preferences or the preferences such that the outcome will be what they want, etc. But this type of deferential preference can still be described by a utility function.

Nathaniel 30 Aug 2022 11:16 UTC
2 points
0
on: Unifying Bargaining Notions (2/2)
Thanks for the great post!
In the definition of Coalition-Perfect CoCo Equilibrium, it seems to me like part 1) is already implied by part 3).
1: ${u_{i}^{N}}_{i \in N}$ is on the Pareto frontier.
1) means that the utility profile achieved by the joint strategy for the grand coalition is on the Pareto frontier.
3: $\forall S \subseteq N : ρ^{S} \in {argmax}_{ρ \in Δ \prod_{i \in S} A_{i}} ⎛ ⎝ \sum i \in S a_{i} U_{i} (ρ, ρ^{N / S}) - \sum j \in N / S a_{j} U_{j} (ρ, ρ^{N / S}) ⎞ ⎠$
(ie. all the joint strategies are trying to maximize the money earned if up against the opposing coalition in a zero-sum game and as a special case, when S=N, it says that what the entire group actually ends up doing maximizes surplus value, which is another way of stating that the ${a_{i}}_{i \in N}$ are the appropriate virtual currencies to use at the ${u_{i}^{N}}_{i \in N}$ point)
If in the joint strategy for the special case of S=N the group maximizes surplus value according to some weight function, then utility profile resulting from this joint strategy should be on the Pareto frontier, so 1) should be automatically satisfied.
If it wasn’t, then you could improve a player’s utility without hurting anyone else. But that would improve the surplus value as well^[1], which would mean that the S=N joint strategy didn’t maximize surplus value (contradicting 3) ).
I think this is because your utilitarian characterization is an if and only if.
Closely related to this is a result that says that any point on the Pareto Frontier of a game can be post-hoc interpreted as the result of maximizing a collective utility function.
Could be: An outcome is on the Pareto frontier if and only if it can be post-hoc interpreted as the result of maximizing a collective utility function.
1. ^
  I guess I’m assuming the weights are strictly positive, whereas you only assumed them to be non-negative. Does this matter/Is this the reason why we need 1)?