Also, I wonder to what extent our own “thinking” is based on concepts we ourselves understand. I’d bet I don’t really understand what concepts most of my own thinking processes use.
Like: what are the exact concepts I use when I throw a ball? Is there a term for velocity, gravity constant or air friction, or is it just some completely “alien” computation which is “inlined” and “tree-shaked” of any unneeded abstractions, which just sends motor outputs given the target position?
Or: what concepts do I use to know what word to place at this place in this sentence? Do I use concepts like “subject”, or “verb” or “sentiment”, or rather just go with the flow subconsciously, having just a vague idea of the direction I am going with this argument?
Or: what concepts do I really use when deciding to rotate the steering wheel 2 degrees to the right when driving a car through a forest road’s gentle turn? Do I think about “angles”, “asphalt”, “trees”, “centrifugal force”, “tire friction”, or rather just try to push the future into the direction where the road ahead looks more straight to me and somehow I just know that this steering wheel is “straightening” the image I see?
Or: how exactly do I solve (not: verify an already written proof) a math problem? How does the solution pop into my mind? Is there some systematic search over all possible terms and derivations, or rather some giant hash-map-like interconnected “related tricks and transformations I seen before” which get proposed?
I think my point is that we should not conflate the way we actually solve problems (subconsciously?), with the way we talk (consciously) about solutions we’ve already found when trying to verify them ourselves (the inner monologue) or convey them to another person. First of all, the Release binary and Debug binaries can differ (it’s completely different experience to ride a bike for a first time, than an on 2000th attempt). Second, the on-the-wire format and the data structure before serialization can be very different (the way I explain how to solve an equation to my kid is not exactly how I solve it).
I think, that training a separate AI to interpret for us the inner workings of another AI is risky, the same way a Public Relations department or a lawyer doesn’t necessarily give you the honest picture of what the client is really up to.
Also, I there’s much talk about distinction between system 1 and 2, or subconsciousness and consciousness, etc.
But, do we really treat seriously the implication of all that: the concepts our conscious part of mind uses to “explain” the subconscious actions have almost nothing to do with how it actually happened. If we force the AI to use these concepts it will either lie to us (“Your honor, as we shall soon see the defendant wanted to..”) , or be crippled (have you tried to drive a car using just the concepts from physics text book?). But even in the later case it looks like a lie to me, because even if the AI is really using the concepts it claims/seems/reported to be using, there’s still the mismatch in myself: I think I now understand that the AI works just like me, while in the reality I work completely differently than I thought. How bad that is depends on problem domain, IMHO. This might be pretty good if the AI is trying to solve a problem like “how to throw a ball” and a program using physic equations is actually also a good way of doing it. But once we get to more complicated stuff like operating a autonomous drone on the battlefield or governing country’s budget I think there’s a risk because we don’t really know how we ourselves make these kind of decisions.
Also, I wonder to what extent our own “thinking” is based on concepts we ourselves understand. I’d bet I don’t really understand what concepts most of my own thinking processes use.
Like: what are the exact concepts I use when I throw a ball? Is there a term for velocity, gravity constant or air friction, or is it just some completely “alien” computation which is “inlined” and “tree-shaked” of any unneeded abstractions, which just sends motor outputs given the target position?
Or: what concepts do I use to know what word to place at this place in this sentence? Do I use concepts like “subject”, or “verb” or “sentiment”, or rather just go with the flow subconsciously, having just a vague idea of the direction I am going with this argument?
Or: what concepts do I really use when deciding to rotate the steering wheel 2 degrees to the right when driving a car through a forest road’s gentle turn? Do I think about “angles”, “asphalt”, “trees”, “centrifugal force”, “tire friction”, or rather just try to push the future into the direction where the road ahead looks more straight to me and somehow I just know that this steering wheel is “straightening” the image I see?
Or: how exactly do I solve (not: verify an already written proof) a math problem? How does the solution pop into my mind? Is there some systematic search over all possible terms and derivations, or rather some giant hash-map-like interconnected “related tricks and transformations I seen before” which get proposed?
I think my point is that we should not conflate the way we actually solve problems (subconsciously?), with the way we talk (consciously) about solutions we’ve already found when trying to verify them ourselves (the inner monologue) or convey them to another person. First of all, the Release binary and Debug binaries can differ (it’s completely different experience to ride a bike for a first time, than an on 2000th attempt). Second, the on-the-wire format and the data structure before serialization can be very different (the way I explain how to solve an equation to my kid is not exactly how I solve it).
I think, that training a separate AI to interpret for us the inner workings of another AI is risky, the same way a Public Relations department or a lawyer doesn’t necessarily give you the honest picture of what the client is really up to.
Also, I there’s much talk about distinction between system 1 and 2, or subconsciousness and consciousness, etc.
But, do we really treat seriously the implication of all that: the concepts our conscious part of mind uses to “explain” the subconscious actions have almost nothing to do with how it actually happened. If we force the AI to use these concepts it will either lie to us (“Your honor, as we shall soon see the defendant wanted to..”) , or be crippled (have you tried to drive a car using just the concepts from physics text book?). But even in the later case it looks like a lie to me, because even if the AI is really using the concepts it claims/seems/reported to be using, there’s still the mismatch in myself: I think I now understand that the AI works just like me, while in the reality I work completely differently than I thought. How bad that is depends on problem domain, IMHO. This might be pretty good if the AI is trying to solve a problem like “how to throw a ball” and a program using physic equations is actually also a good way of doing it. But once we get to more complicated stuff like operating a autonomous drone on the battlefield or governing country’s budget I think there’s a risk because we don’t really know how we ourselves make these kind of decisions.