I attended an AI conference a while ago that was hosted in a historic railway museum. A practical railway museum, with lots of working machines, and you could see them at work, their plans, their components, how they were made, shipped, assembled.
It hit me really hard.
What humans had for those trains, that was understanding. They genuinely knew how they worked. Not just knew how to operate them. Not just had a vague understanding of the general principles behind it. They could build the whole thing, from scratch, by hand. They could explain what each tiny little part did. They could replace each part with something else serving the same function that they would themselves make. They could take the whole damn thing apart, and literally see each bit, move them by hand, watch them intersect. They could predict how the whole would behave, exactly.
It really struck me how much is required already just to do this for an old train. Just to get a wheel to turn smoothly under so much pressure. Just to weld it together so securely it wouldn’t blow up. Just to coordinate the different train times and movements. Just to calculate the needed fuel and material strength. It wasn’t something you could half ass based on having read a Wiki article; it needed exact precision and perfect understanding.
I got the impression that it must felt beautiful for the humans able to do that. They seemed so proud, so in control. They were close to these machines, knew them like their back hand. In a constant exchange with data, rationality, common sense. Back then, every time something wasn’t quite perfect, there was an explanation to be found, and they could each find it. Everything around them was understood and made sense.
I was passing between the AI talks, and the trains… and really understood, for the first time, what is meant with sufficiently advanced technology being indistinguishable from magic.
We don’t understand these AI systems.
We no longer can. Not as a collective, most certainly not as an individual.
We learn how to add bits that make them better. How to act when things go wrong. How to get them to reliably do the very things we like. That alone already takes a bunch of knowledge, and so we treat people who have achieved it like they have understood the thing in question, rather than just having learned how to handle this complex madness their predecessors handed them.
But this is akin to praying to a deity, hoping to get the right words, and then chucking a mix of sacred materials on the grounds in order to make plants grow, without having any understanding of the role of bacteria and fungi and fertiliser, and then being praised because you have memorised magic materials that so happen to contain more nitrogen—which in turn is something people don’t understand, they just see that the plants grow, and are satisfied.
Akin to an alien handing you a magic device, and you learning how to operate the buttons to get it to give you candy, and maybe even realising how to push more power into the system to get more candy more quickly. And then people praise you, because you have optimised the candy maker you do not understand.
I went home, and walked through my house, through my things, and asked myself… if these devices broke, and I had to replace a component from scratch—not just identify a broken one, reorder it, and insert it, but know enough about how it works to be able to build a replacement component from scratch, without browsing the internet… how many of these devices could I actually fix?
If you try, by hand, to build something as simple as a stable shelter, or a smoothly turning wheel, even things you thought were utterly trivial turn out to be bloody hard, and to have a lot of detail you need to actually properly understand to make something good. The ugly plastic containers I keep my rice in, from Ikea—those have a designer name written on them. And first, I laughed, it is just a container, how much thought can go into that, who “designs” such a thing? And then I noticed the feet that allow it to stand stable, even if there are grains of rice on the floor of my cupboard. The grooves that allow you fingers to get under the lid. The rubber so it makes a proper seal against food moths. It is just a bloody container, and yet getting that right required proper thought. How much would go into an actually complicated device?
I was sorta confident I could fix my toaster, for scenarios like replacing wires with other high resistance metal components, and some simpler problems in my washing machine and dishwasher, the kind that require visible tubes, or maybe soldering a missing connection. But for things beyond that… the components have become too tiny for me to work with my hands, sometimes too tiny to see. They require ingredients that I have no way of acquiring short of buying more machines and disassembling them; I have zero idea how to mine them, or to extract the ore, and even if I did, I would not be able to make the components out of it anyhow, they have become so fine. And I do not understand how they work. When I select a dishwasher program, how does that control heat and duration and spin, exactly? I don’t mean a vague answer based on programming a pentabug on the general principles of how this could be done. That horrid shell of understanding is what I got used to confusing with understanding—as though having a vague memory of a motor blueprint in school would allow you to actually make a motor on the first go. No, how does it actually work? What does that button do? What does opening the door do? If I got zero advice, could I reinvent it from scratch? How long would it take me until I had a safe, effective dishwasher? And then I look from there, to my bloody laptop, and it is a leap to the sky. I look from my laptop to the LLMs that I can run on it. Then to the LLMs that are being run on better devices… and the depth of my ignorance makes me choke. Understanding LLMs feels like primitive people trying to predict the weather and harvests through observing patterns, and then invoking an angry thunder God. Like cargo cult people building something that looks like a plane because they have observed a relation to travel. It is horrifying.
If right now, we had an apocalypse, where knowledge and manufacturing ability and human population was lost, and me and a few hundred survivors—let’s say really smart, really well educated, really hard-working survivors—were left with a bunch of laptops and antibiotic pills and air planes… I highly, highly doubt we’d learn how to fix, let alone replace and reproduce, the things we had inherited in time before they broke down and/or we died. We would sit there with a bunch of magic tools, rapidly diminishing. Our kids would be handed things they could not understand, could not make, could not fix, that would fail them, one by one. If they had the insane luck to have access to a remaining LLM… they would be begging the LLM to tell them what to do to save things, and one day, when the LLM stopped working, they would despair, as one after another, their devices lost their glow. - These things I am using, they are not my things, the way things I understand are my things. They are things I inherited, and their continued working is fragile.
LOL! Yes, we are not in the world of mechanical or electro-mechanical devices anymore, are we?
And yet I don’t think things are hopeless. Understanding LLMs is certainly no worse than understanding brains. After all, we can manipulate and inspect LLMs in a way we cannot manipulate and inspect brains. And I think we’ve made progress understanding brains. Back in the old days people used to believe in neurons that were facetiously called “grandmother cells.” The idea is that the was one neuron that recognized your grandmother, another one that recognized your dog Spot, yet another one for your Barbie doll, and so forth. I think the field has pretty much gotten over that fantasy to the idea of collective representation. Each neuron participates in recognizing many different things and each thing is recognized by a collectivity of neurons. Just how that might work, well, we’re working on it. I hear things might be like that inside LLMs as well.
How to act when things go wrong. (...) That alone already takes a bunch of knowledge, and so we treat people who have achieved it like they have understood the thing in question, rather than just having learned how to handle it.
I have felt the difference recently, twice. First, my old computer broke. Something with hard disk, not sure what exactly, but it wouldn’t boot up anymore. So I took the hard disk to a repair shop, and asked them to try salvage whatever is possible and copy it to an external disk.
I am not an expert on these things, but I expected them to do something like connect the disk by a cable to their computer, and run some Linux commands that would read the data from the disk and store it somewhere else. But instead, the guys there just tried to boot up the computer, and yes they confirmed that it wouldn’t work, and… they couldn’t do anything with it. After asking a bit more, my impression was that all they know to do is, basically, reinstall Windows and run some diagnostic and antivirus programs.
Second story, I bought a new smartphone. Then I realized that my contacts and SMS messages from the old one didn’t transfer, because they were stored in the phone or the SIM card, rather than in Google cloud. (I got a new SIM card for the new phone, because it required different size.) There was a smartphone repair shop nearby and I was lazy, so I thought “let’s just pay those guys to extract the messages and contacts from the old phone, and send them to me by an e-mail or something” (I wanted to have a backup on my computer, not just to transfer them to the new phone).
Again, I expected those guys to just connect something to my old phone, or put the SIM card in some machine, and extract the data. And again, it turned out that there was nothing they could do about it. After asking a bit more, I concluded that their business model is basically just replacing broken glass on the phones, or sending them to an authorized service provider if something more serious happens.
The thing that made me angry was realizing that despite what I perceived as deep incompetence, their business models actually make sense. In the spirit of the “80:20” rule, yes, 80% of problems average people have with computers do not require more than reinstalling Windows or maybe just changing some configuration, and 80% of problems average people have with smartphones are broken or scratched glass or battery dead, or something else that requires replacing a piece of hardware. Probably much more than 80%.
So yes, you can teach a bunch of guys how to reinstall Windows or replace a broken glass, and have a profitable repair shop. That is the economically rational thing to do. And yet I miss the old-style repairmen, most of them people from the older generation, who actually understood the things they worked with.
So, let me tell you a story about how I ‘fixed’ my first computer. This was the Ancient Days and that first machine was a Northstar Horizon, based on the S-100 bus and the Zilog Z80 microprocessor. You could take the lid off of the machine and see the circuit boards. Here’s a description from the Wikipedia article:
The computer consists of a thick aluminium chassis separated into left and right compartments with a plywood cover which sat on the top and draped over the left and right sides. (It is one of only a handful of computers to be sold in a wooden cabinet. Later versions featured an all-metal case which met safety standards.[5]) The rear section of the compartment on the right held a linear power supply, including a large transformer and power capacitors, comprising much of the bulk and weight of the system. The empty section in front of the power supply normally housed one or two floppy disk drives, placed on their side so the slots were vertical. The compartment on the left held the S-100 motherboard, rotated so the slots ran left-right. Although a few logic circuits were on the motherboard, primarily for I/O functions, both the processor and the memory resided in separate daughterboards.
The manual that came with the computer had circuit diagrams for the boards.
Now, I knew little or nothing about such things. But my good friend, Rich Fritzon, he lived and breathed computers. He knew a thing or two. So, once I got the machine I turned it over to Rich and he wrote some software for it. The most important piece was a WYSWYG text editor that took advantage of the special associative memory board from Syd Lamb’s company, the name of which escapes me.
Anyhow, I had this beast with me when I spent the summer of 1981 on a NASA project. One day the display went all wonky; the images just didn’t make sense. Well, I knew that the CPU board had a synch (synchronization) chip and, well, those wonky images looked like something that would happen if signals weren’t properly synchronized. I mean, I didn’t actually KNOW anything, I was just guessing based on bits and scraps of things I’d heard and read. Based on this guess I removed the motherboard, located the sync chip in the corresponding diagram, removed the synch chip and reseated it, and then put the board back into the machine. When I turned it on, voilà! problem solved. The display was back.
That’s the first and last time I ever fixed one of my machines. That sort of thing would be utterly impossible with today’s machines.
I attended an AI conference a while ago that was hosted in a historic railway museum. A practical railway museum, with lots of working machines, and you could see them at work, their plans, their components, how they were made, shipped, assembled.
It hit me really hard.
What humans had for those trains, that was understanding. They genuinely knew how they worked. Not just knew how to operate them. Not just had a vague understanding of the general principles behind it. They could build the whole thing, from scratch, by hand. They could explain what each tiny little part did. They could replace each part with something else serving the same function that they would themselves make. They could take the whole damn thing apart, and literally see each bit, move them by hand, watch them intersect. They could predict how the whole would behave, exactly.
It really struck me how much is required already just to do this for an old train. Just to get a wheel to turn smoothly under so much pressure. Just to weld it together so securely it wouldn’t blow up. Just to coordinate the different train times and movements. Just to calculate the needed fuel and material strength. It wasn’t something you could half ass based on having read a Wiki article; it needed exact precision and perfect understanding.
I got the impression that it must felt beautiful for the humans able to do that. They seemed so proud, so in control. They were close to these machines, knew them like their back hand. In a constant exchange with data, rationality, common sense. Back then, every time something wasn’t quite perfect, there was an explanation to be found, and they could each find it. Everything around them was understood and made sense.
I was passing between the AI talks, and the trains… and really understood, for the first time, what is meant with sufficiently advanced technology being indistinguishable from magic.
We don’t understand these AI systems.
We no longer can. Not as a collective, most certainly not as an individual.
We learn how to add bits that make them better. How to act when things go wrong. How to get them to reliably do the very things we like. That alone already takes a bunch of knowledge, and so we treat people who have achieved it like they have understood the thing in question, rather than just having learned how to handle this complex madness their predecessors handed them.
But this is akin to praying to a deity, hoping to get the right words, and then chucking a mix of sacred materials on the grounds in order to make plants grow, without having any understanding of the role of bacteria and fungi and fertiliser, and then being praised because you have memorised magic materials that so happen to contain more nitrogen—which in turn is something people don’t understand, they just see that the plants grow, and are satisfied.
Akin to an alien handing you a magic device, and you learning how to operate the buttons to get it to give you candy, and maybe even realising how to push more power into the system to get more candy more quickly. And then people praise you, because you have optimised the candy maker you do not understand.
I went home, and walked through my house, through my things, and asked myself… if these devices broke, and I had to replace a component from scratch—not just identify a broken one, reorder it, and insert it, but know enough about how it works to be able to build a replacement component from scratch, without browsing the internet… how many of these devices could I actually fix?
If you try, by hand, to build something as simple as a stable shelter, or a smoothly turning wheel, even things you thought were utterly trivial turn out to be bloody hard, and to have a lot of detail you need to actually properly understand to make something good. The ugly plastic containers I keep my rice in, from Ikea—those have a designer name written on them. And first, I laughed, it is just a container, how much thought can go into that, who “designs” such a thing? And then I noticed the feet that allow it to stand stable, even if there are grains of rice on the floor of my cupboard. The grooves that allow you fingers to get under the lid. The rubber so it makes a proper seal against food moths. It is just a bloody container, and yet getting that right required proper thought. How much would go into an actually complicated device?
I was sorta confident I could fix my toaster, for scenarios like replacing wires with other high resistance metal components, and some simpler problems in my washing machine and dishwasher, the kind that require visible tubes, or maybe soldering a missing connection. But for things beyond that… the components have become too tiny for me to work with my hands, sometimes too tiny to see. They require ingredients that I have no way of acquiring short of buying more machines and disassembling them; I have zero idea how to mine them, or to extract the ore, and even if I did, I would not be able to make the components out of it anyhow, they have become so fine. And I do not understand how they work. When I select a dishwasher program, how does that control heat and duration and spin, exactly? I don’t mean a vague answer based on programming a pentabug on the general principles of how this could be done. That horrid shell of understanding is what I got used to confusing with understanding—as though having a vague memory of a motor blueprint in school would allow you to actually make a motor on the first go. No, how does it actually work? What does that button do? What does opening the door do? If I got zero advice, could I reinvent it from scratch? How long would it take me until I had a safe, effective dishwasher? And then I look from there, to my bloody laptop, and it is a leap to the sky. I look from my laptop to the LLMs that I can run on it. Then to the LLMs that are being run on better devices… and the depth of my ignorance makes me choke. Understanding LLMs feels like primitive people trying to predict the weather and harvests through observing patterns, and then invoking an angry thunder God. Like cargo cult people building something that looks like a plane because they have observed a relation to travel. It is horrifying.
If right now, we had an apocalypse, where knowledge and manufacturing ability and human population was lost, and me and a few hundred survivors—let’s say really smart, really well educated, really hard-working survivors—were left with a bunch of laptops and antibiotic pills and air planes… I highly, highly doubt we’d learn how to fix, let alone replace and reproduce, the things we had inherited in time before they broke down and/or we died. We would sit there with a bunch of magic tools, rapidly diminishing. Our kids would be handed things they could not understand, could not make, could not fix, that would fail them, one by one. If they had the insane luck to have access to a remaining LLM… they would be begging the LLM to tell them what to do to save things, and one day, when the LLM stopped working, they would despair, as one after another, their devices lost their glow. - These things I am using, they are not my things, the way things I understand are my things. They are things I inherited, and their continued working is fragile.
LOL! Yes, we are not in the world of mechanical or electro-mechanical devices anymore, are we?
And yet I don’t think things are hopeless. Understanding LLMs is certainly no worse than understanding brains. After all, we can manipulate and inspect LLMs in a way we cannot manipulate and inspect brains. And I think we’ve made progress understanding brains. Back in the old days people used to believe in neurons that were facetiously called “grandmother cells.” The idea is that the was one neuron that recognized your grandmother, another one that recognized your dog Spot, yet another one for your Barbie doll, and so forth. I think the field has pretty much gotten over that fantasy to the idea of collective representation. Each neuron participates in recognizing many different things and each thing is recognized by a collectivity of neurons. Just how that might work, well, we’re working on it. I hear things might be like that inside LLMs as well.
_____________
PS. I just looked at your profile and noticed that you have ADHD. Some years ago I took a look at the technical (and not so technical) literature on the subject and wrote up some notes: Music and the Prevention and Amelioration of ADHD: A Theoretical Perspective.
I have felt the difference recently, twice. First, my old computer broke. Something with hard disk, not sure what exactly, but it wouldn’t boot up anymore. So I took the hard disk to a repair shop, and asked them to try salvage whatever is possible and copy it to an external disk.
I am not an expert on these things, but I expected them to do something like connect the disk by a cable to their computer, and run some Linux commands that would read the data from the disk and store it somewhere else. But instead, the guys there just tried to boot up the computer, and yes they confirmed that it wouldn’t work, and… they couldn’t do anything with it. After asking a bit more, my impression was that all they know to do is, basically, reinstall Windows and run some diagnostic and antivirus programs.
Second story, I bought a new smartphone. Then I realized that my contacts and SMS messages from the old one didn’t transfer, because they were stored in the phone or the SIM card, rather than in Google cloud. (I got a new SIM card for the new phone, because it required different size.) There was a smartphone repair shop nearby and I was lazy, so I thought “let’s just pay those guys to extract the messages and contacts from the old phone, and send them to me by an e-mail or something” (I wanted to have a backup on my computer, not just to transfer them to the new phone).
Again, I expected those guys to just connect something to my old phone, or put the SIM card in some machine, and extract the data. And again, it turned out that there was nothing they could do about it. After asking a bit more, I concluded that their business model is basically just replacing broken glass on the phones, or sending them to an authorized service provider if something more serious happens.
The thing that made me angry was realizing that despite what I perceived as deep incompetence, their business models actually make sense. In the spirit of the “80:20” rule, yes, 80% of problems average people have with computers do not require more than reinstalling Windows or maybe just changing some configuration, and 80% of problems average people have with smartphones are broken or scratched glass or battery dead, or something else that requires replacing a piece of hardware. Probably much more than 80%.
So yes, you can teach a bunch of guys how to reinstall Windows or replace a broken glass, and have a profitable repair shop. That is the economically rational thing to do. And yet I miss the old-style repairmen, most of them people from the older generation, who actually understood the things they worked with.
So, let me tell you a story about how I ‘fixed’ my first computer. This was the Ancient Days and that first machine was a Northstar Horizon, based on the S-100 bus and the Zilog Z80 microprocessor. You could take the lid off of the machine and see the circuit boards. Here’s a description from the Wikipedia article:
The manual that came with the computer had circuit diagrams for the boards.
Now, I knew little or nothing about such things. But my good friend, Rich Fritzon, he lived and breathed computers. He knew a thing or two. So, once I got the machine I turned it over to Rich and he wrote some software for it. The most important piece was a WYSWYG text editor that took advantage of the special associative memory board from Syd Lamb’s company, the name of which escapes me.
Anyhow, I had this beast with me when I spent the summer of 1981 on a NASA project. One day the display went all wonky; the images just didn’t make sense. Well, I knew that the CPU board had a synch (synchronization) chip and, well, those wonky images looked like something that would happen if signals weren’t properly synchronized. I mean, I didn’t actually KNOW anything, I was just guessing based on bits and scraps of things I’d heard and read. Based on this guess I removed the motherboard, located the sync chip in the corresponding diagram, removed the synch chip and reseated it, and then put the board back into the machine. When I turned it on, voilà! problem solved. The display was back.
That’s the first and last time I ever fixed one of my machines. That sort of thing would be utterly impossible with today’s machines.