Roy Zhang

Karma: 2

Roy Zhang 10 Sep 2026 10:40 UTC
1 point
0
on: Automation and Political Power
Very thought provoking question. To prevent what you described from happening, I think the best way is to have open source models that remain close to the frontier, so that ai is not concentrated in a few companies or governments.
I also wonder to what extent the end state you described is the most likely path. We will probably go through a period of “hybrid” workforce where ai needs to work with humans. There will be political parties, firms, governments, alliances, and rival states all competing for “power” (quoted because what counts as power can vary widely in this context). Each actor is incentivized to concentrate power themselves, but they also have incentives to prevent others from doing the same.
The mass still has valuable bargaining power during the hybrid state. First, they have economic leverage, like the assets that they own. Second, ai could significantly lower the cost of coordination and resistance. The self-preservation instinct should be very strong to avoid entering an equilibrium where there’s no chance to fight back. Whether this is enough remains to be seen.
In the end, I suspect the world won’t be a hegemony. Autocratic states are much more likely to end up in the scenario you described.

Roy Zhang 10 Sep 2026 8:31 UTC
1 point
0
on: Side-Effects of Length Penalty in RL
Thank you for your research!
One connection that comes to my mind is the concern that, in pursuiing a goal, an AI system could find acquiring additional resources (compute/data/tool access/influence over people) increases its probability of success. This would become more relevant as models can operate over increasingly longer sessions.
But based on your findings, it appears that models can learn to optimize its strategy given resource constraints while preserving performance. Is the implication here that we could at least reduce incentives for unnecessarily resource-intensive strategies?
For example, could training objectives include explicit cost penalties not only for reasoning length, but also compute/time/tool calls and other resources used in pursuing a task? If models can learn to acount for those costs at the planning level, perhaps this could create some preferences for achieving objectives with a smaller resource footprint. This is more like how humans optimize (not to say this is better). We don’t treat our effort and cognitive capacity as free, because these actions carry a cost.
I am aware that this doesn’t solve many other issues of alignment. A superintelligent model could still act unaligned. Also, I suspect that the resource penalty only works for objectives with bounded upside. For example, what if the objective is to produce as many balls of twine as possible? What if by acquiring enourmous amount of compute in step 3 increases expected reward in thousands of later steps?
I’m still new to the field. Trying to catch up and see what research problems would be most interesting to me. I’d love to hear your thoughts, and I hope I’m not conflating your findings with my own interpretation.

Roy Zhang 8 Sep 2026 10:29 UTC
3 points
0
on: The Magnus Challenge
Very interesting thought process. If I’m understanding correctly, the key risk comes from a capability leap that is too large, such that we have very low probability of aligning the next model.
Reading your post, I find myself more optimistic about AI alignment than beating Magnus.
First, one distinction between AI alignment and the chess analogy is that we don’t have to “beat” AI. Unless the model is actively resisting us (which means we’ve done something terriblely wrong), we don’t need to outthink a much more intelligent opponent at its own game. At least intuitively, that gives us a larger margin of safety when making capability leaps.
Second, let’s assume the mechanism you described is true, i.e., one could beat Magnus or align AI through a chain of smaller steps. Then to fix a misaligned model, we can perhaps trace back to the last model that was aligned correctly, fix the handoff where alignment failed, which redo the chain from there until we fix the “rogue” one. If the capability leap is the problem, we can still train intermediary models that are aligned. In that sense, we don’t necessarily have only one shot at every step.
In that framing, the key question is also time constraint. Can we align our models fast enough before the next generation arrives? I’m actually curious what others think. My guess would be aligning a model can be faster than training a substantially more capable new model, since at least part of alignment looks like a subset of post-training. If that’s true, there may be enough time to keep doing these handoffs before capability runs too far ahead.