Basically it’s been possible for a long time now, to not care about maintainability of your codebase and the result would be faster progress in implementing features.
This is deemed, in my view, the correct approach for some parts of the job: building a POC and writing a one-off script are two examples of tasks for which caring about maintainability is wrong most of the time, as maintainability costs significant time and effort.
There have also always been people that believed that caring about and investing in maintainability was wrong for large, long-lived projects intended to run in production. I believe the consensus view is now mostly, that these people were in the wrong. But it took a long time to form that consensus. I’m sure cowboys who push the first thing that works to straight to production still exist, but at least in my circles people generally believe that investing in maintainability of software is generally worth it for long-lived projects.
Now I believe you’re correct to point out that maintainability for humans and maintainability for LLMs may not be the same thing. I would assume, that a codebase that is maintainable for humans is generally also maintainable for LLMs, though perhaps not optizmized for LLM usage. I’m not sure the reverse is true and suspect it is not.
In any case, I do not know if we have the data yet, on whether or not it’s worth discarding old practices. To get back to the cowboy programmer that just writes code and pushes it straight to prod, that looks fast at first and eventually you reckon you have a big ball of mud you can never change again without breaking something somewhere. Do we know this does not happen with LLMs, too?
Because once you commit to the bit and say “human maintainability is no longer my concern”, then you will get code that is not maintainable by humans. If it then turns out that it is not maintainable by LLM either, then you have code noone can maintain. Whereas, if you have the LLM write code that a human could maintain as well, then you’re guarded against this particular kind of failure mode. But of course that’s slower and more expensive in the short term.
So I guess my question is: How sure are we that we’re not setting ourselves up for failure a year or more down the line? Because if what you propose is a mistake we will probably only notice it in the long term.
When coding with LLMs, observability and iteration is key. Having a loop that nudges closer and closer to a target goal is easier if you have a clear target (your current codebase’s behaviour), and good observability (giving the llm more power to compare and improve).
So even if you build a slop now, you can always rebuild better. And scaffold the agent properly so it’s able to rebuild it more readable/structured in the future.
However, this doesn’t matter. Given the power curve in programming ability (10x programmers etc), most code sucks compared to what is state-of-the-art. Improving upon this code is not hard. LLMs are insane; you can scaffold them to not be lazy, to read the current structure of the project and take that into account, and to research the best practices and to pick and adhere to an architectural pattern.
Compare to most programmers in the wild; they take pieces that almost fit; smash them together; wire everything in 3 layers of spaghetti; and then duct tape the shit out of it.
I’ve had to tell a colleague (this was 10 years ago) to use git rather than copying their 8k loc piece of shit file and persisting it by pasting it at the end of it’s own file. ctrl+a → ctrl+c → ctrl+v → ctrl+s. So easy.
Also, LLMs improve. We discover workflows, share them, good ones gets baked into scaffolding. Meanwhile there are programmers who’d rather not learn a thing, going with a sota 15 years out of date.
A bit of a rant. I’ve been working with devs for 9,5 years. I much prefer the LLMs, when it comes to code quality. But I’ve put a lot of energy into shifting my mindset for how I work with them, building proper scaffolding, etc. I can easily see people with no clue of what they are doing just leveraging their ability to ship shit.
If I understand correctly, you propose that if the LLM ends up creating slop that works but is unmaintainable, then we could use that codebase as the starting point for reimplementation?
That seems plausible, but the question is, should we assume that reimplementation will be more maintainable? After all, the first codebase was presumably generated in a similar fashion and did end up unmaintainable.
I’m not confident this is true, but I can imagine a world in which reimplementing codebases larger than, say, 300k LOC by LLM results in unmaintainable slop, no matter how often you do and redo it.
You say that we should scaffold the LLM so it writes good code, and personally I agree. But wasn’t OPs suggestion exactly that we should not care about making the LLM write code humans perceive as good? I understand that to mean we’re supposed to pass on the suggestions you mention and embrace some new way of LLM-first convention yet-to-be-established.
And that is what I’m worried about: Intentionally saying “No” to writing code humans can read and understand.
As for the fact that there are bad human programmers: This is certainly true, but kind of besides the point of discussion I feel.
badly generated ai code can be transformed into better ai code. So if we slop now, then we likely will be able to reimplement better. This means that your concern needs to be about us never being able to get ais to implement readable and nice code. Which is insane to me, because;
In my experience, AI code, when properly scaffolded, is better than most human code. Hence the idea that we will never (even with better AI) be able to write understandable code is insane.
There are obvious limits to this argument, including sloppy user facing abstractions (wonky workflows, hacky extension points, etc etc). Rebuilding to capture bad abstractions can be hard, we might inherit a spaghetti mess of UX to replicate. But if the UI is specified and works properly, I don’t think the HOW matters very much, given replicability.
The points your making do not seem to adress the discussion I believe we’re having, so it seems likely we’re talking past each other.
I’ll try to give a brief summary of what I believe the point of dicussion is and why I believe your points do not address it. I may be wrong in this, if so hopefully you can use this to tell me why your points are relevant.
To oversimplify a bit for brevity, I believe the chain of argument goes approx as follows
OP: Having LLMs write human-maintainable code, what we’d call “good and readable code”, is a bad goal and we should not bother with it
Me: If you do that it’ll look fast at first but maybe you’ll end up with something that is also LLM-unmaintainable. That would be bad, how sure are we this does not happen?
You: Bad LLM-code can be rewritten as better LLM-code, but this does not matter, because it is possible to have LLMs write human-maintainable “good and readable code”, better than humans probably
Me: Having to LLM rewrite your LLM-written codebase may well result in code just as unmaintainable, just with one more feature. Also: I know LLMs need not produce slop, but OP suggests we intentionally have the LLMs produce slop
You: Slop can be reimplemented as non-slop, and LLMs are better at it than humans
To which I answer: I believe we’re discussing what happens if we followed OP’s suggestion, which rules out having LLMs write good-for-humans code, as that would not follow OP’s suggestion. So how good LLMs are at writing good-for-humans code is irrelevant, OP specified we should produce slop and that is the world we’re considering.
Or is your argument that if we try out the “embrace the slop” thing and it ends up not working we can move back to “reject the slop” fast and easy using LLMs?
If so it was not clear to me that your suggestions are for after we tried OPs approach, found it failed and have given up on it. After giving up on OPs suggestion it is clear to me that the approach you propose could be attempted. It is just incompatible with following OPs approach at the same time
I see the diference between OP “go slop” vs mine “rebuild not costly”. Pretty much exactly what you said “[...] “embrace the slop” thing and it ends up not working we can move back to “reject the slop” fast and easy using LLMs?”
When I take a bird’s eye view, I see some interesting features:
natural maintainability. What makes code maintainable by humans, vs maintainable by ai? What’s the overlap? (I think fairly large, esp given AIs are trained on human-written code)
rewriteability: high, with potential slop-costs from having messy UIs that become burdens to keep up. (in a sense “overfit”)
I read OP as saying: maintainable for llm /= maintainable for human. Let’s go llm first. I think this is true to some extent, but false in other ways. I’m pretty sure some things hold: immutability, type safety, linters, modularization, unit testing, clean abstractions, etc.
When I look at the field, most abstraction suck (not composable, leak implementation details, doesn’t carve the domain at it’s joints, etc). So I’m happy with people trying out a low-abstraction approach to llm writing. I’m leaning into rust for bigger things, and keep to python for smaller, scaling maintainability concerns with energy invested
Thanks for clarifying. I think “we could try what OP proposes, if it sucks it’s probably not that expensive to LLM-rewrite in cleaner” is a reasonable argument.
Personally I’m just less certain than you are I feel. If I had to bet on it I would assume that the things you listed are objectively useful, not just for humans. But I’m not actually any sort of sure of it, and if a bunch good new studies came out tomorrow showing that LLMs do extremely good without linters or immutability or something I would not be surprised either.
Same with clean, good, cheap LLM-rewrites. Sounds reasonable enough but we haven’t seen much of it yet and I’d not be surprised if there were cases in which it absolutely failed.
So personally I’d rather give the whole thing a bit of time for data to come in to validate whether our intuitions are actually correct, or just reasonable but ultimately false guesses.
Guess at the pace this thing is moving I’m SOL however.
making certain classes of bugs evaporates a lot of “bad code” space. Some things like nulls are obvious net-losses given the birth of nullrefs; others like immutability might be hard to grasp if not used, but there’s so many bugs that arise from state not being what you expect it to be. Didn’t assign, copy by value vs reference, etc etc; birthing a class of bugs.
For types/lints etc: the more ways you can desribe your constraints for program behaviour, the more different sources of invariants can overlap, leading to feedback, where collisions appear. Same as naming, although that’s more heuristic than programmatic.
For the most, main issues for adoption has been onboarding, tooling support etc. Well, now it’s easier to build tools, an onboarding is a skill download. Another issue would be that models have less training data. This might be the case, but that’s mostly a problem for very niche languages, rust is sufficient and it can also write nix-derivations, which is famous for bad docs :)
note that I’m obviously biased by Haskell elitism. Rust avoids a bunch of memory bugs and has more reliable performance, which makes it way feasable in my experience.
I’ve been wondering about that.
Basically it’s been possible for a long time now, to not care about maintainability of your codebase and the result would be faster progress in implementing features.
This is deemed, in my view, the correct approach for some parts of the job: building a POC and writing a one-off script are two examples of tasks for which caring about maintainability is wrong most of the time, as maintainability costs significant time and effort.
There have also always been people that believed that caring about and investing in maintainability was wrong for large, long-lived projects intended to run in production. I believe the consensus view is now mostly, that these people were in the wrong. But it took a long time to form that consensus. I’m sure cowboys who push the first thing that works to straight to production still exist, but at least in my circles people generally believe that investing in maintainability of software is generally worth it for long-lived projects.
Now I believe you’re correct to point out that maintainability for humans and maintainability for LLMs may not be the same thing. I would assume, that a codebase that is maintainable for humans is generally also maintainable for LLMs, though perhaps not optizmized for LLM usage. I’m not sure the reverse is true and suspect it is not.
In any case, I do not know if we have the data yet, on whether or not it’s worth discarding old practices. To get back to the cowboy programmer that just writes code and pushes it straight to prod, that looks fast at first and eventually you reckon you have a big ball of mud you can never change again without breaking something somewhere. Do we know this does not happen with LLMs, too?
Because once you commit to the bit and say “human maintainability is no longer my concern”, then you will get code that is not maintainable by humans. If it then turns out that it is not maintainable by LLM either, then you have code noone can maintain. Whereas, if you have the LLM write code that a human could maintain as well, then you’re guarded against this particular kind of failure mode. But of course that’s slower and more expensive in the short term.
So I guess my question is: How sure are we that we’re not setting ourselves up for failure a year or more down the line? Because if what you propose is a mistake we will probably only notice it in the long term.
When coding with LLMs, observability and iteration is key. Having a loop that nudges closer and closer to a target goal is easier if you have a clear target (your current codebase’s behaviour), and good observability (giving the llm more power to compare and improve).
So even if you build a slop now, you can always rebuild better. And scaffold the agent properly so it’s able to rebuild it more readable/structured in the future.
However, this doesn’t matter. Given the power curve in programming ability (10x programmers etc), most code sucks compared to what is state-of-the-art. Improving upon this code is not hard. LLMs are insane; you can scaffold them to not be lazy, to read the current structure of the project and take that into account, and to research the best practices and to pick and adhere to an architectural pattern.
Compare to most programmers in the wild; they take pieces that almost fit; smash them together; wire everything in 3 layers of spaghetti; and then duct tape the shit out of it.
I’ve had to tell a colleague (this was 10 years ago) to use git rather than copying their 8k loc piece of shit file and persisting it by pasting it at the end of it’s own file. ctrl+a → ctrl+c → ctrl+v → ctrl+s. So easy.
Also, LLMs improve. We discover workflows, share them, good ones gets baked into scaffolding. Meanwhile there are programmers who’d rather not learn a thing, going with a sota 15 years out of date.
A bit of a rant. I’ve been working with devs for 9,5 years. I much prefer the LLMs, when it comes to code quality. But I’ve put a lot of energy into shifting my mindset for how I work with them, building proper scaffolding, etc. I can easily see people with no clue of what they are doing just leveraging their ability to ship shit.
If I understand correctly, you propose that if the LLM ends up creating slop that works but is unmaintainable, then we could use that codebase as the starting point for reimplementation?
That seems plausible, but the question is, should we assume that reimplementation will be more maintainable? After all, the first codebase was presumably generated in a similar fashion and did end up unmaintainable.
I’m not confident this is true, but I can imagine a world in which reimplementing codebases larger than, say, 300k LOC by LLM results in unmaintainable slop, no matter how often you do and redo it.
You say that we should scaffold the LLM so it writes good code, and personally I agree. But wasn’t OPs suggestion exactly that we should not care about making the LLM write code humans perceive as good? I understand that to mean we’re supposed to pass on the suggestions you mention and embrace some new way of LLM-first convention yet-to-be-established.
And that is what I’m worried about: Intentionally saying “No” to writing code humans can read and understand.
As for the fact that there are bad human programmers: This is certainly true, but kind of besides the point of discussion I feel.
My point, more succinctly:
badly generated ai code can be transformed into better ai code. So if we slop now, then we likely will be able to reimplement better. This means that your concern needs to be about us never being able to get ais to implement readable and nice code. Which is insane to me, because;
In my experience, AI code, when properly scaffolded, is better than most human code. Hence the idea that we will never (even with better AI) be able to write understandable code is insane.
There are obvious limits to this argument, including sloppy user facing abstractions (wonky workflows, hacky extension points, etc etc). Rebuilding to capture bad abstractions can be hard, we might inherit a spaghetti mess of UX to replicate. But if the UI is specified and works properly, I don’t think the HOW matters very much, given replicability.
The points your making do not seem to adress the discussion I believe we’re having, so it seems likely we’re talking past each other.
I’ll try to give a brief summary of what I believe the point of dicussion is and why I believe your points do not address it. I may be wrong in this, if so hopefully you can use this to tell me why your points are relevant.
To oversimplify a bit for brevity, I believe the chain of argument goes approx as follows
OP: Having LLMs write human-maintainable code, what we’d call “good and readable code”, is a bad goal and we should not bother with it
Me: If you do that it’ll look fast at first but maybe you’ll end up with something that is also LLM-unmaintainable. That would be bad, how sure are we this does not happen?
You: Bad LLM-code can be rewritten as better LLM-code, but this does not matter, because it is possible to have LLMs write human-maintainable “good and readable code”, better than humans probably
Me: Having to LLM rewrite your LLM-written codebase may well result in code just as unmaintainable, just with one more feature. Also: I know LLMs need not produce slop, but OP suggests we intentionally have the LLMs produce slop
You: Slop can be reimplemented as non-slop, and LLMs are better at it than humans
To which I answer: I believe we’re discussing what happens if we followed OP’s suggestion, which rules out having LLMs write good-for-humans code, as that would not follow OP’s suggestion. So how good LLMs are at writing good-for-humans code is irrelevant, OP specified we should produce slop and that is the world we’re considering.
Or is your argument that if we try out the “embrace the slop” thing and it ends up not working we can move back to “reject the slop” fast and easy using LLMs?
If so it was not clear to me that your suggestions are for after we tried OPs approach, found it failed and have given up on it. After giving up on OPs suggestion it is clear to me that the approach you propose could be attempted. It is just incompatible with following OPs approach at the same time
I see the diference between OP “go slop” vs mine “rebuild not costly”. Pretty much exactly what you said “[...] “embrace the slop” thing and it ends up not working we can move back to “reject the slop” fast and easy using LLMs?”
When I take a bird’s eye view, I see some interesting features:
natural maintainability. What makes code maintainable by humans, vs maintainable by ai? What’s the overlap? (I think fairly large, esp given AIs are trained on human-written code)
rewriteability: high, with potential slop-costs from having messy UIs that become burdens to keep up. (in a sense “overfit”)
I read OP as saying: maintainable for llm /= maintainable for human. Let’s go llm first. I think this is true to some extent, but false in other ways. I’m pretty sure some things hold: immutability, type safety, linters, modularization, unit testing, clean abstractions, etc.
When I look at the field, most abstraction suck (not composable, leak implementation details, doesn’t carve the domain at it’s joints, etc). So I’m happy with people trying out a low-abstraction approach to llm writing. I’m leaning into rust for bigger things, and keep to python for smaller, scaling maintainability concerns with energy invested
Thanks for clarifying. I think “we could try what OP proposes, if it sucks it’s probably not that expensive to LLM-rewrite in cleaner” is a reasonable argument.
Personally I’m just less certain than you are I feel. If I had to bet on it I would assume that the things you listed are objectively useful, not just for humans. But I’m not actually any sort of sure of it, and if a bunch good new studies came out tomorrow showing that LLMs do extremely good without linters or immutability or something I would not be surprised either.
Same with clean, good, cheap LLM-rewrites. Sounds reasonable enough but we haven’t seen much of it yet and I’d not be surprised if there were cases in which it absolutely failed.
So personally I’d rather give the whole thing a bit of time for data to come in to validate whether our intuitions are actually correct, or just reasonable but ultimately false guesses.
Guess at the pace this thing is moving I’m SOL however.
making certain classes of bugs evaporates a lot of “bad code” space. Some things like nulls are obvious net-losses given the birth of nullrefs; others like immutability might be hard to grasp if not used, but there’s so many bugs that arise from state not being what you expect it to be. Didn’t assign, copy by value vs reference, etc etc; birthing a class of bugs.
For types/lints etc: the more ways you can desribe your constraints for program behaviour, the more different sources of invariants can overlap, leading to feedback, where collisions appear. Same as naming, although that’s more heuristic than programmatic.
For the most, main issues for adoption has been onboarding, tooling support etc. Well, now it’s easier to build tools, an onboarding is a skill download. Another issue would be that models have less training data. This might be the case, but that’s mostly a problem for very niche languages, rust is sufficient and it can also write nix-derivations, which is famous for bad docs :)
note that I’m obviously biased by Haskell elitism. Rust avoids a bunch of memory bugs and has more reliable performance, which makes it way feasable in my experience.