Most discussion of AI memory treats limited context as an engineering problem. Make the context window larger, add retrieval, connect the system to external databases, and eventually the machine will have everything relevant available to it.
I think this misses a deeper problem. Information does not arrive tagged with its relevance. A finite intelligence must decide what to retrieve, what to attend to, what to compress and what to ignore. More information therefore enlarges the selection problem rather than eliminating it.
I develop the argument as a sequence: finite intelligence requires exclusion; exclusion requires compression; compression introduces bias; and bias cannot be removed without replacing it with another criterion for relevance. The implication is that an omniscient, neutral AI oracle may be conceptually incoherent rather than merely technologically distant.
I would be particularly interested in criticism of that final step.
The Oracle That Forgot the Milk: Why Every Finite Intelligence Must Exclude
Link post
Most discussion of AI memory treats limited context as an engineering problem. Make the context window larger, add retrieval, connect the system to external databases, and eventually the machine will have everything relevant available to it.
I think this misses a deeper problem. Information does not arrive tagged with its relevance. A finite intelligence must decide what to retrieve, what to attend to, what to compress and what to ignore. More information therefore enlarges the selection problem rather than eliminating it.
I develop the argument as a sequence: finite intelligence requires exclusion; exclusion requires compression; compression introduces bias; and bias cannot be removed without replacing it with another criterion for relevance. The implication is that an omniscient, neutral AI oracle may be conceptually incoherent rather than merely technologically distant.
I would be particularly interested in criticism of that final step.