might be true if you just care about input and output behaviour
Yes, that is the assumption for “some computable function” or “black box which takes in strings and spits out other strings.”
I’m not sure your example (of an AI with a much wider range of possible input/output pairs than the lookup table) fits this underlying distinction. If the input/output sets are truly identical (or even identical for all tests you can think of), then we’re back to the “why do we care” question.
hmm, we seem to be talking past each other a bit. I think my main point in response is something like this:
In non-trivial settings, (some but not all) structural differences between programs lead to differences in input/output behaviour, even if there is a large domain for which they are behaviourally equivalent.
But that sentence lacks a lot of nuance! I’ll try to break it down a bit more to find if/where we disagree (so apologies if a lot of this is re-hashing).
I agree that if two programs produce the same input output behaviour for literally every conceivable input then there is not much of an interesting difference between them and you can just view one as a compression of the other.
As I said in the post, I consider a program to be ‘actually calculating the function’, if it is a finite program which will return f(x) for every possible input x.
If we have a finite length lookup table, it can only output f(x) for a finite number of inputs.
If that finite number of inputs is less than the total number of possible inputs, this means that there is at least one input (call it x_0) for which a lookup table will not output f(x_0).
I’ve left unspecified what it will do if you query the lookup table with input x_0. Maybe it doesn’t return anything, maybe it outputs an error message, maybe it blows up. The point is that whatever it does, by definition it doesn’t return f(x_0).
Maybe the number of possible inputs to f is finite and the lookup table is large enough to accommodate them all. In this case, the lookup table would be a ‘finite program which will return f(x) for every possible input x’ so I would be happy to say that there’s not much of a distinction between the lookup table and a different method of computing the function. (Of course, there is a trivial difference in the sense that they are different algorithms, but its not the one we are concerned with here).
However, this requires that the size of the program is on the order of (or larger than) the number of possible inputs it might face. I think that in most interesting cases this is not true.
By ‘interesting cases’, I mean things like humans who do not contain an internal representation of every possible sensory input, or LLMs where the number of possible sentences you could input is larger than the model itself.
This means that in ‘interesting cases’, a lookup table will, at some point, give a different output to a program which is directly calculating a function.
So structural difference (of the ‘lookup table or not’ kind) between two finite programs will imply behavioural difference in some domain, even if there is a large domain for which the two programs are behaviourally equivalent (for an environment where the number of possible inputs is larger than the size of the programs).
As I see it, this is the motivation behind the Agent-like structure problem. If you know that a program has agent-like structure, this can help you predict its behaviour in domains where you haven’t seen it perform.
Or, conversely, if you know that selecting for certain behaviours is likely to lead to agent-like structure you can avoid selecting for those behaviours (even if, within a certain domain, those behaviours are benign) because in other domains agent-like behaviour is dangerous.
Of course, there are far more types of programs than just ‘lookup table’ or ‘not lookup table’. Lookup tables are just one obvious way for which a program might exhibit certain behaviour in a finite domain which doesn’t extend to larger domains.
In non-trivial settings, (some but not all) structural differences between programs lead to differences in input/output behaviour, even if there is a large domain for which they are behaviourally equivalent.
I think this is a crux (of why we’re talking past each other; I don’t actually know if we have a substantive disagreement). The post was about detecting “smaller than a lookup table would support” implementations, which implied that the input/output functionally-identical-as-tested were actually tested in the broadest possible domain. I fully agree that “tested” and “potential” input/output pairs are not the same sets, but I assert that, in a black-box situation, it CAN be tested in a very broad set of inputs, so the distinction usually won’t matter. That said, nobody has built a pure lookup table anywhere near as complete as it would take to matter (unless the universe or my experience is simulated that way, but I’ll never know).
My narrower but stronger point is that “lookup table vs algorithm” is almost never as important as “what specific algorithm” for any question we want to predict about the black box. Oh, and almost all real-world programs are a mix of algorithm and lookup.
Yes, that is the assumption for “some computable function” or “black box which takes in strings and spits out other strings.”
I’m not sure your example (of an AI with a much wider range of possible input/output pairs than the lookup table) fits this underlying distinction. If the input/output sets are truly identical (or even identical for all tests you can think of), then we’re back to the “why do we care” question.
hmm, we seem to be talking past each other a bit. I think my main point in response is something like this:
In non-trivial settings, (some but not all) structural differences between programs lead to differences in input/output behaviour, even if there is a large domain for which they are behaviourally equivalent.
But that sentence lacks a lot of nuance! I’ll try to break it down a bit more to find if/where we disagree (so apologies if a lot of this is re-hashing).
I agree that if two programs produce the same input output behaviour for literally every conceivable input then there is not much of an interesting difference between them and you can just view one as a compression of the other.
As I said in the post, I consider a program to be ‘actually calculating the function’, if it is a finite program which will return f(x) for every possible input x.
If we have a finite length lookup table, it can only output f(x) for a finite number of inputs.
If that finite number of inputs is less than the total number of possible inputs, this means that there is at least one input (call it x_0) for which a lookup table will not output f(x_0).
I’ve left unspecified what it will do if you query the lookup table with input x_0. Maybe it doesn’t return anything, maybe it outputs an error message, maybe it blows up. The point is that whatever it does, by definition it doesn’t return f(x_0).
Maybe the number of possible inputs to f is finite and the lookup table is large enough to accommodate them all. In this case, the lookup table would be a ‘finite program which will return f(x) for every possible input x’ so I would be happy to say that there’s not much of a distinction between the lookup table and a different method of computing the function. (Of course, there is a trivial difference in the sense that they are different algorithms, but its not the one we are concerned with here).
However, this requires that the size of the program is on the order of (or larger than) the number of possible inputs it might face. I think that in most interesting cases this is not true.
By ‘interesting cases’, I mean things like humans who do not contain an internal representation of every possible sensory input, or LLMs where the number of possible sentences you could input is larger than the model itself.
This means that in ‘interesting cases’, a lookup table will, at some point, give a different output to a program which is directly calculating a function.
So structural difference (of the ‘lookup table or not’ kind) between two finite programs will imply behavioural difference in some domain, even if there is a large domain for which the two programs are behaviourally equivalent (for an environment where the number of possible inputs is larger than the size of the programs).
As I see it, this is the motivation behind the Agent-like structure problem. If you know that a program has agent-like structure, this can help you predict its behaviour in domains where you haven’t seen it perform.
Or, conversely, if you know that selecting for certain behaviours is likely to lead to agent-like structure you can avoid selecting for those behaviours (even if, within a certain domain, those behaviours are benign) because in other domains agent-like behaviour is dangerous.
Of course, there are far more types of programs than just ‘lookup table’ or ‘not lookup table’. Lookup tables are just one obvious way for which a program might exhibit certain behaviour in a finite domain which doesn’t extend to larger domains.
I think this is a crux (of why we’re talking past each other; I don’t actually know if we have a substantive disagreement). The post was about detecting “smaller than a lookup table would support” implementations, which implied that the input/output functionally-identical-as-tested were actually tested in the broadest possible domain. I fully agree that “tested” and “potential” input/output pairs are not the same sets, but I assert that, in a black-box situation, it CAN be tested in a very broad set of inputs, so the distinction usually won’t matter. That said, nobody has built a pure lookup table anywhere near as complete as it would take to matter (unless the universe or my experience is simulated that way, but I’ll never know).
My narrower but stronger point is that “lookup table vs algorithm” is almost never as important as “what specific algorithm” for any question we want to predict about the black box. Oh, and almost all real-world programs are a mix of algorithm and lookup.
Cool, that all sounds fair to me. I don’t think we have any substantive disagreements.