We train the Alpaca model on 52K instruction-following demonstrations
generated in the style of self-instruct using text-davinci-003.
And so it begins, LLM-generated datasets useful for training LLMs, that wouldn’t be found in the wild and would’ve been (prohibitively) expensive to purposefully generate with human labor. Hopefully the currently human-generated datasets used in SSL pre-training, the backbone of samulacrum alignment, won’t be mostly replaced by synthetic datasets that drift away from humanity.
And so it begins, LLM-generated datasets useful for training LLMs, that wouldn’t be found in the wild and would’ve been (prohibitively) expensive to purposefully generate with human labor. Hopefully the currently human-generated datasets used in SSL pre-training, the backbone of samulacrum alignment, won’t be mostly replaced by synthetic datasets that drift away from humanity.