I think the subcontractor/robot situation really depends. There are cases where I hire someone purely so that something gets done, without caring about who does it. If they send another person, or a robot, I may be entirely fine with that. But there are cases where this doesn’t apply, say when I hire a person because they specifically have a great reputation or I otherwise think highly of their work and I care a lot about details and quality.
silentbob
Karma: 1,983
Here’s where it gets interesting.
This is the point most people get wrong.
Then there’s Y.
What I keep coming back to is X.
The things that surprised me most is Y.
Something something “on Tuesday”.
The X <verb describing a change or increase>ed. The Y barely moved.
Things “land” and “sit” all the time and everything happens “quietly”.
touch your pupil without flinching
Am I the only one cringing at this so heavily that I’m basically 100% certain I’ll never wear contact lenses in my life? I have 0 intention to ever learn such a thing. :O
Another thought: every individual may be involved in many projects. So if you look specifically at their most well-known project, then that’s somewhat of an unfair advantage for the project. Whereas if you were looking at a randomly selected project that a person published, then it may well be that that one’s less well-known than the person themselves.
E.g. the study you (OP) linked says:
Pair a well-known artifact with the person who built it and, in 7 of 10 pairs, the artifact wins. The reinforcement-learning library Tianshou scores 0.78 while its sole author Jiayi Weng sits at 0.22 — the work propagated; the name behind it did not.
I’d say this is not surprising, as when you condition on “well-known artifact”, you should expect something like regression to the mean in well-known-ness when you move on to another entity.
I’d say it depends on the person and project—sometimes a project gets really well known and gets a lot of coverage (in pretraining data) while the creator is only mentioned occasionally. Similarly, I know hundreds (maybe thousands?) of pieces of software by name, but only for a fraction of them could I name the creator(s). But then you have other cases, where people get better known than many of their projects. E.g. if you think of any actor, every LLM out there will likely know who they are. But they are less likely to know about every single movie they’ve made.
As a funny example mixing both spheres, actress Milla Jovovich was involved in creating https://www.mempalace.tech/ which will most certainly remain less widely known than herself.
(edit: while I linked to mempalace, it looks like pure AI slop. I haven’t looked into it more deeply and hence me linking to it is not a recommendation. fwiw, the NameRank website + paper also appear to be fully AI-written)
I’ve seen some software engineers in tech orgs behave in similar ways, where I’m really unsure whether they actually managed to 10x their productivity using the super elaborate systems they constructed, with hierarchical teams of AI agents, each having different roles, and the humans just being their orchestrator, or whether they just get extremely excited about these possibilities and set up all these layers of “automation” without this actually leading to more useful output. I can certainly imagine it leads to more output, but I’m somewhat skeptical if it’s high quality, sustainable output, or just huge amounts of ~slop.
Another thing this reminds me of: some years ago, I sometimes wondered about how many people out there would be forever alone, never finding a partner/marriage/starting a family even though they’d want to. Reassuringly, I got the impression that practically every “older” person I encountered (say people in their 40s and 50s) was married. So, even though closer to my age range there were many people who were unhappily single, it certainly looked as if life would find a way to get everyone a partner over time.
At some point I then had the unfortunate realization that lonely people may just have fewer social contacts in general[1], and a bias towards the types of social contacts that make them invisible to some random young guy like me. So now I feel less hopeful about society, knowing/assuming that there must be some not so small percentage of people in any age range who just never found anyone to spend their life with.
- ^
where I assume causality goes in both directions, 1) people with less social exposure are less likely to find a partner to begin with, and 2) having a partner naturally exposes you to more opportunities to go out and spend time with people, as you’ll do many things as a couple rather than individually. And there may well be an effect of being alone for a long time against your will reducing your motivation to go out to begin with.
- ^
Somewhat similarly, when you have some recurring meetup, then the experience of most attendees of how many people attend the meetup on average will exceed the actual average.
I wonder what the main causes of that are.
people not reading the instructions themselves at all but letting basically the entire process be done by an LLM
people thinking they’ll get away with it, as they’re unaware how easy it is to detect
people who just don’t have the capacity or ability to do it without AI, so they just give it a shot anyway, even knowing that it’s against the rules and they may get filtered out
did I forget some likely-seeming option?
Are we entering the age of crime slop?
Like überall? Maybe jirgendwo vs nüberall are the better words. Like neverywhere or nalways.
I agree it makes sense to raise the “does this prove too much” question. But I’d tend to think it doesn’t (as far as your example goes). Three thoughts:
I do think the company in your scenario does have a huge problem. Whether it’s an existential problem for them, or merely a “we’ll have some rough times ahead and might need to take two years to somehow regain a level of competence within our workforce” depends on their circumstances, but I’d say that most companies in most situations will struggle severely when 11 out of eleven experienced software developers spontaneously leave. Well, some companies may be able to just maintain their prior level and be in a comfortable enough spot that the severe slowdown in software development is not a big deal for them. But this is probably more of an exception.
Neither in this case, nor in the case of AI automation, would I call the task impossible. Just very hard. My main goal here was to put the “coding agents are magical and change everything” impression that one can very easily get into context, as I think these magical capabilities don’t easily transfer to larger-scale organizations.
As you already hint at with your last sentence, some of the challenges I mentioned affect LLMs in particular, and hence the situation for human developers to catch up would, imho, be much more realistic (at anything close to current capability levels of AI). I’m not sure if larger context window sizes would solve this (although I’ve sort of argued before that it might—I’ve somewhat updated in the opposite direction now, but am unsure). I do think that the fact that context windows are stored in text makes them less useful. And while this limitation exists, I think it will always lead to problems—although it’s conceivable that such problems (LLMs subtly misunderstanding things or missing nuances and hence creating worse code or making bad judgment calls) would just not matter all that much and would be outweighed by the advantages. I could imagine that an order of 10-100M token context windows would allow to capture the most important 95-99% of context, if it’s filled wisely and deliberately, but that’s really just spitballing. Such context window sizes are not impossible, but at recent trends, I’d be a bit surprised if we get there sooner than 2-3 years from now. And even when we do: this might still leave other bottlenecks in place, plus it would still require very targeted efforts to utilize these larger context windows properly.
Why Software Automation Is Hard
I’m curious though, would you say you can also model “volumetric” 3D in your head, or more the typical shape/surface, e.g. seeing a 3D orange in your head, but only “from the outside”, without having a good detailed intuition about its internal structure?
I think I agree about your detail observation, but these details in my case are still mostly 2D surfaces within 3D space, rather than “true 3D” in the sense I was trying to get at in the post.
On the other side, having an LLM delete your production database or cause something catastrophic seems (I don’t have data on this) to happen way more often than catastrophically bad chatbot conversations.
I also don’t have any reliable data, but I would be very surprised if this were the case. I remember maybe ~3 publicly discussed cases of “deleted a production database”-grade LLM failures, but my impression is that there are probably at least 10s of thousands of cases of LLM psychosis or similarly bad/extreme outcomes, and could well imagine that number to be much higher.
For AGI, none of these constraints may be relevant. Minds can fork and merge. Training can be instant through weight sharing. Coordination happens at silicon speed without contracts. When one AI masters a new domain—say, protein folding or contract law—it won’t need to teach others through language or demonstration. It will simply share the relevant weights, like copying a file. The receiving AI instantly acquires years of “experience” in milliseconds.
I wonder if this actually holds up once continual learning is solved. Currently, I see ~three general ways in which that might potentially happen:
Some form of online weight updating. But that would mean different instances of the same original AI may not be “compatible” anymore in the sense that they could easily share something they learned with each other. The only viable way then would be to create identical clones of an AI that has learned something important (which is still highly useful, of course).
Context windows become so enormously large that AIs can just put an entire career worth of context in there, and in-context learning is strong enough for this alone to surpass the level of humans in most domains. In this case, they could in principle just share the relevant parts from their context window, describing in sufficient detail how to perform some skill perfectly, with another AI, and that might work. But it’s also possible that the way they stored that skill in their context relates in numerous ways to other things they personally have learned, and isolating a particular skill to share it with another AI may not work well, as, e.g., it tends to use words in different ways and thereby generalizes differently from what the provided context contains. (It would still likely work much better/faster than whatever humans do to share knowledge/skills with each other, though)
Perhaps some in-between thing that’s neither on weight level nor in plain language, like some form of persistent memory of embeddings or so. No idea what that might look like in practice, but the blurry image I have of it still looks like it might make it difficult to extract some isolated thing out of it without corrupting it beyond usefulness.
I think it still seems very likely that AIs will be much better than humans at all of this, in many relevant ways, so I agree with your point directionally. I don’t want to rule out that “sharing years of experience in milliseconds” does turn out true. Just wanted to point out that to me, it’s not at all obvious that this will happen, and solving certain problems on the way to AGI may come at the expense of the feasibility of instant skill sharing between AI instances.
For the videos I mentioned, my p(at least some phrases for this came out of an LLM) range from maybe 80% (SeaGate) to 97% (Mo Bitar). So I definitely see a chance I may be wrong about one of these samples. But I’d be very surprised if I’m wrong about the general trend and if actually several of the cases I showed are fully human-written, after all.
What makes me confident is the density of these patterns, that most of them occur together in most cases, and that this seems, as far as I can tell, to be a pretty recent development. I’m interested in quantifying that pattern density, I’ll see when I find the time to do so.
Of course, one can find any single one of these patterns in writing from before 2022. But I’d assume that it’s very difficult to find text with such a density of all the specific patterns that LLMs show. It could of course be the case that this writing style is just a sort of “persona selection” that occurred during post-training and there really were people speaking like this online in the past, rather than LLMs having been the first entities to truly own that style. But even then: the ubiquity of this style nowadays seems way too high to me to be explained without LLMs being heavily involved in the process.
Interesting! If you come across any such examples, I’d be very curious to see them.
I guess it depends on the alternative. The writing of many inexperienced writers will surely get better in all kinds of ways. But the writing I would want to read almost certainly gets worse in ways that I care about. LLM writing to me almost always feels very thin and “style over substance” (and then even in a style I grew to dislike). Naturally, writing is a very high-dimensional thing, and “things different people value in writing” equally so. So there will be different answers for different writers and readers. To me, the negatives are:
It’s a less accurate representation of the author’s thinking (assuming thinking on side of the author took place)
It tends to be full of “empty sentences”, hedging, shallow examples
Writing all over the world gets heavily correlated
I subjectively find the style annoying
It potentially robs us of a useful signal of who expands actual effort in their work
The “Not X—Y” pattern in particular often seems quite useless. The “not”-part could usually just be omitted without making things worse. It’s rarely something that people would have thought that needs to be corrected. And in the rare cases that it is: why not write in a way to avoid such misconceptions to begin with instead of repeatedly creating and then correcting them? Occasionally, it can be useful to get people to a certain notion and then correct it, as a rhetorical or pedagogical move, but certainly not 5x within any given text.
I don’t get the downvotes, I think it’s an important question to ask when I arises.
To me, finding undisclosed LLM text just is a very strong indicator the person publishing it wants to get away with sloppy work and doesn’t respect (the time and epistemic environment of) their readers very much. Clearly this doesn’t apply to every case, it’s just by far the most common occurrence, so by posting LLM text, you’re moving into a different reference class. The most likely scenario behind someone posting LLM text, purely on very general priors, is that they wrote a short prompt and got a slop response. If the person invested more effort into the interaction and has reason to believe their LLM text is of high quality and worth reading, then they should point out why (which involves being transparent about LLM usage in the first place).
I think I’d still have this preference if there was only a single LLM in existence which was both aligned and superintelligent. Text is supposed to have effects on readers, so being transparent about the nature of a text just seems like the right move.