In one of Rohin’s comments he describes a ‘next level up’ of difficulty for the model.
″...you can reduce the variance by sampling continuations for a given prompt rather than sampling sequences unprompted...”
So basically, predicting the next n tokens where n > 1. Two? Three? Sentence completion multiple choice? Not sure how best to implement.
I’d be excited to see this next level turned into a web app somehow! I suspect we might find the LLMs do worse than n==1, but still surprisingly well.
Edit: played the second game w the probabilities and found it a fun challenge. Beating the 2-layer is easy, not thinking too hard, but couldn’t catch that darn 12 layer!
In one of Rohin’s comments he describes a ‘next level up’ of difficulty for the model.
″...you can reduce the variance by sampling continuations for a given prompt rather than sampling sequences unprompted...” So basically, predicting the next n tokens where n > 1. Two? Three? Sentence completion multiple choice? Not sure how best to implement. I’d be excited to see this next level turned into a web app somehow! I suspect we might find the LLMs do worse than n==1, but still surprisingly well.
Edit: played the second game w the probabilities and found it a fun challenge. Beating the 2-layer is easy, not thinking too hard, but couldn’t catch that darn 12 layer!