I think you can extend the Appendix A construction to the original problem just by discarding any tokens with label >= (i.e. embedding and of that token applies a strong negative bias to all neurons, so that if that token is seen in either position, all neurons are inactive). This means you can only learn facts (or about 1⁄4 of the total facts), but I think this l represents an improvement over your current hand-coded weights, from eyeballing Figure 5.
This definitely falls squarely within the “probably not what models learn”, though—but I figure I’d point it out as a simple extension.
Edit: Actually, I think you can do one better and learn facts—continue to ignore any tokens with label in embedding , but there’s no need to ignore those tokens in embedding .
But also, this is probably not actually the correct way to look at things, because this assumes you’re trying to learn all facts—which would leave you with a particularly poor 50% or 25% accuracy. The problem is—this construction doesn’t account for how “which facts to learn” are not chosen.
I think you can extend the Appendix A construction to the original problem just by discarding any tokens with label >=(i.e. embeddingandof that token applies a strong negative bias to all neurons, so that if that token is seen in either position, all neurons are inactive). This means you can only learnfacts (or about 1⁄4 of the totalfacts), but I think this l represents an improvement over your current hand-coded weights, from eyeballing Figure 5.This definitely falls squarely within the “probably not what models learn”, though—but I figure I’d point it out as a simple extension.Edit:
Actually, I think you can do one better and learnfacts—continue to ignore any tokens with labelin embedding, but there’s no need to ignore those tokens in embedding.But also, this is probably not actually the correct way to look at things, because this assumes you’re trying to learn all