I just asked Gemini 2.5 Pro to explain how to tie shoelaces to someone who has never done that before, a task probably works in its favor because it is so common, plenty of descriptions exist and most people can perform it with little cognitive effort within few seconds every day. It took about 1.5 letter-sized pages of text and still missed a little bit of detail but I think a humanoid robot could follow it and get to the right result. I imagine many tasks of machinists and craftsmen are more complex but simply don’t exist in writing, so I agree that lack of data is the obvious problem. (Cooking would be another field where AI models today may perform above their baseline.)
Also, professions like law and obviously math and software have invented their own languages that only vaguely resemble everyday-language. If we had the goal of capturing the tacit knowledge of craftsmen, like this last surviving stucco worker, we would probably also first invent a more precise language to do that.
I can see this being automated given the visual capabilities in the latest models along with a healthy dose of input from existing practitioners. Do detailed teardowns of different products across many different industries, with images of each component and subassembly along with detailed descriptions of those images (what the parts are, what they’re made of, how they were made, what purpose they serve as a whole and what purpose various features are serving). This could then start to create the textual training data to then allow the models to generate such information themselves in the opposite direction. And in fact this closely resembles how mechanical engineers often build up experience (along with making things, building them, and seeing why they don’t work like they thought they would).
I just asked Gemini 2.5 Pro to explain how to tie shoelaces to someone who has never done that before, a task probably works in its favor because it is so common, plenty of descriptions exist and most people can perform it with little cognitive effort within few seconds every day. It took about 1.5 letter-sized pages of text and still missed a little bit of detail but I think a humanoid robot could follow it and get to the right result. I imagine many tasks of machinists and craftsmen are more complex but simply don’t exist in writing, so I agree that lack of data is the obvious problem. (Cooking would be another field where AI models today may perform above their baseline.)
Also, professions like law and obviously math and software have invented their own languages that only vaguely resemble everyday-language. If we had the goal of capturing the tacit knowledge of craftsmen, like this last surviving stucco worker, we would probably also first invent a more precise language to do that.
Out of curiosity, can you share a link to Gemini 2.5 Pro’s response?
I can see this being automated given the visual capabilities in the latest models along with a healthy dose of input from existing practitioners. Do detailed teardowns of different products across many different industries, with images of each component and subassembly along with detailed descriptions of those images (what the parts are, what they’re made of, how they were made, what purpose they serve as a whole and what purpose various features are serving). This could then start to create the textual training data to then allow the models to generate such information themselves in the opposite direction. And in fact this closely resembles how mechanical engineers often build up experience (along with making things, building them, and seeing why they don’t work like they thought they would).