Then it seems to me almost guaranteed that AI will be 99.9 percentile if not 100 percentile when compared against human experts.
Are you talking about current AI, or future AI? Before or after training on that task?
Concretely, “minimize program length while maintaining correctness” seems to be significantly beyond the capabilities of the best publicly available scaffolded LLMs today for all but the simplest programs, and the trends in conciseness for AI-generated code do not make me optimistic that that will change in the near future.
I think this is solveable with today’s software stack and compute, it is just that no lab has bothered to do it. Maybe check back in a year, and downgrade my reputation otherwise. I could set up a manifold market if it is important.
Are you talking about current AI, or future AI? Before or after training on that task?
Concretely, “minimize program length while maintaining correctness” seems to be significantly beyond the capabilities of the best publicly available scaffolded LLMs today for all but the simplest programs, and the trends in conciseness for AI-generated code do not make me optimistic that that will change in the near future.
I think this is solveable with today’s software stack and compute, it is just that no lab has bothered to do it. Maybe check back in a year, and downgrade my reputation otherwise. I could set up a manifold market if it is important.