Epistemic status: Probably a useful canary, not sure about the conclusion.
GPU programming seems not fully automated yet, which may point to the inability of RLVR to overcome data scarcity (the task is fairly niche) even in the face of excellent verifiability.
Naively, a good kernel just returns the correct tensors (i.e. matching a pytorch reference implementation) in the fastest possible time. This makes it a natural target for RLVR, and possible to run a competition where anyone can just submit a kernel with automatic grading.
The winning submission of the GPU kernel writing competition two months ago is only 11% AI generated, according to Pangram.
However, this is very OOD for the things Pangram is validated for.
Anthropic seems to still hire for performance engineers, who among other things write GPU Kernels, and the job description mentions the relevant low-level/manual skills.
A project closely associated with the largest relevant community claims that LLMs still “can’t do it”.
It may be harder than expected to actually grade GPU kernels automatically, e.g. there was an incident ofreward hacking the grader of the competition above.
The Unsloth guys claim Torch.compile over-powers all manual kernel coding at this part of their recent presentation: https://youtu.be/uIiA6DquRiE?t=6277 (1:44:30+) . I cannot verify their claim, but their releases( including kernels[0] for speed-up) are quite popular, I am sure.
There is some newer, recent evidence in Kimi K3′s system card: https://www.kimi.com/blog/kimi-k3. It seems to outperform kernels and compilers. Probably something like “hardware-software co-design” is still a human thing, but all else probably not.
Epistemic status: Probably a useful canary, not sure about the conclusion.
GPU programming seems not fully automated yet, which may point to the inability of RLVR to overcome data scarcity (the task is fairly niche) even in the face of excellent verifiability.
Naively, a good kernel just returns the correct tensors (i.e. matching a pytorch reference implementation) in the fastest possible time. This makes it a natural target for RLVR, and possible to run a competition where anyone can just submit a kernel with automatic grading.
The winning submission of the GPU kernel writing competition two months ago is only 11% AI generated, according to Pangram.
However, this is very OOD for the things Pangram is validated for.
Anthropic seems to still hire for performance engineers, who among other things write GPU Kernels, and the job description mentions the relevant low-level/manual skills.
A project closely associated with the largest relevant community claims that LLMs still “can’t do it”.
It may be harder than expected to actually grade GPU kernels automatically, e.g. there was an incident of reward hacking the grader of the competition above.
The Unsloth guys claim Torch.compile over-powers all manual kernel coding at this part of their recent presentation: https://youtu.be/uIiA6DquRiE?t=6277 (1:44:30+) . I cannot verify their claim, but their releases( including kernels[0] for speed-up) are quite popular, I am sure.
0: https://github.com/unslothai/unsloth/tree/main/unsloth/kernels
There is some newer, recent evidence in Kimi K3′s system card: https://www.kimi.com/blog/kimi-k3. It seems to outperform kernels and compilers. Probably something like “hardware-software co-design” is still a human thing, but all else probably not.