Task: Computing the relevance of a paper to solving a problem
Context: A researcher is looking at existing literature trying to solve some problem. However, it is not obvious what to search for. There might exist research whose relevance can only be determined once they read it and think about it. For this task a simple keyword-matching search engine isn’t useful.
Input type: A problem statement and an URL to a website or PDF. Optionally, text with reasoning about the problem and what has been tried before. The URLs in the training set can be papers that have been found useful in the past, but ideally they would be found by interacting with the system. When queried it evaluates as many pages in its crawling corpus as possible, perhaps only the top k (for a very big k) results given by a regular search engine.
Output type: An explanation (maybe optional, considering most pages are irrelevant) of why the document might or might not be useful for solving the problem, in addition to a numerical score. The score can be given in some scale of usefulness following some consistent guidelines. But ideally it would be the percentile of usefulness assuming it falls among the n pages the researcher can look at when interacting with the system.
Info constraints: If training with documents that have been useful in the past, the program can’t be aware of any work that has been the result of that.
Instances: (they should be collected by interacting with the system, the usefulness percentiles are made up)
Instance 1:
Input:
Problem: Ontological translation. Given two different ontologies about the same domain and some representation in one ontology, compute the corresponding representation (if any) on the other ontology.
Reasoning: This paper deals with translating between an AI’s old wrong ontology, represented as a POMDP-like system and a new bigger and better ontology. It presents a simple heuristic algorithm for doing so based on bisimulation. It has shortcomings, such as using monolithic states or being plausibly susceptible to steganography, but it seems like a promising starting point.
Reasoning: This paper describes a method for automatically computing an image in some distribution corresponding to a given image in another distribution, say: photos of apples to photos of oranges or zebras to horses. Cycle consistency might be useful in general. However, the loss:
Doesn’t guarantee that we will actually end up with the corresponding element, only something that seems to be from that distribution. While it works in this case, it is probably because doing it correctly happens to be simple, there is no guarantee that it will generalize to more complex situations.
Reasoning: This paper presents an algorithm for translation sentences between two languages in an unsupervised fashion. It is based on encoding sentences of both languages in the same latent space, forcing those two distributions to be indistinguishable using an adversarial loss and useful for performing two different tasks. While it works for this use case, it uses text translation specific tricks such as initializing the translator to word-by-word translation obtained using another method or adding “noise” by dropping and swapping words. It also doesn’t seem like it could scale well, especially if the sizes of the ontologies are very different.
Predicted usefulness percentile: 45%
Simply searching for “ontological translation” on Google Scholar gives terrible results.
Task: Computing the relevance of a paper to solving a problem
Context: A researcher is looking at existing literature trying to solve some problem. However, it is not obvious what to search for. There might exist research whose relevance can only be determined once they read it and think about it. For this task a simple keyword-matching search engine isn’t useful.
Input type: A problem statement and an URL to a website or PDF. Optionally, text with reasoning about the problem and what has been tried before. The URLs in the training set can be papers that have been found useful in the past, but ideally they would be found by interacting with the system. When queried it evaluates as many pages in its crawling corpus as possible, perhaps only the top k (for a very big k) results given by a regular search engine.
Output type: An explanation (maybe optional, considering most pages are irrelevant) of why the document might or might not be useful for solving the problem, in addition to a numerical score. The score can be given in some scale of usefulness following some consistent guidelines. But ideally it would be the percentile of usefulness assuming it falls among the n pages the researcher can look at when interacting with the system.
Info constraints: If training with documents that have been useful in the past, the program can’t be aware of any work that has been the result of that.
Instances: (they should be collected by interacting with the system, the usefulness percentiles are made up)
Instance 1:
Input:
Problem: Ontological translation. Given two different ontologies about the same domain and some representation in one ontology, compute the corresponding representation (if any) on the other ontology.
URL: Ontological Crisis in Artificial Agents’ Value Systems
Output:
Reasoning: This paper deals with translating between an AI’s old wrong ontology, represented as a POMDP-like system and a new bigger and better ontology. It presents a simple heuristic algorithm for doing so based on bisimulation. It has shortcomings, such as using monolithic states or being plausibly susceptible to steganography, but it seems like a promising starting point.
Predicted usefulness percentile: 80%
Instance 2:
Input:
Problem: Ontological translation.
URL: An ontological approach to translation and untranslatability
Output:
Reasoning: This is a paper about linguistics that deals with language translation. It contains no insights about general ontological translation.
Predicted usefulness percentile: <1%
Instance 3:
Input:
Problem: Ontological translation.
URL: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Output:
Reasoning: This paper describes a method for automatically computing an image in some distribution corresponding to a given image in another distribution, say: photos of apples to photos of oranges or zebras to horses. Cycle consistency might be useful in general. However, the loss:
Doesn’t guarantee that we will actually end up with the corresponding element, only something that seems to be from that distribution. While it works in this case, it is probably because doing it correctly happens to be simple, there is no guarantee that it will generalize to more complex situations.
Seems unusually susceptible to steganography.
Has the regular problems associated with GANs.
Predicted usefulness percentile: 63%
Instance 4:
Input:
Problem: Ontological translation.
URL: Unsupervised Machine Translation Using Monolingual Corpora Only
Output:
Reasoning: This paper presents an algorithm for translation sentences between two languages in an unsupervised fashion. It is based on encoding sentences of both languages in the same latent space, forcing those two distributions to be indistinguishable using an adversarial loss and useful for performing two different tasks. While it works for this use case, it uses text translation specific tricks such as initializing the translator to word-by-word translation obtained using another method or adding “noise” by dropping and swapping words. It also doesn’t seem like it could scale well, especially if the sizes of the ontologies are very different.
Predicted usefulness percentile: 45%
Simply searching for “ontological translation” on Google Scholar gives terrible results.
I might add more and better instances later.