I’m confused about what this research direction could concretely look like, can you give some examples? Do you mostly mean stuff like “get high quality transcripts of humans doing moral reasoning about problems they’ve never considered before and SFT on that”, or something altogether different?
I won’t share all details, but the plan is not “get high quality transcripts of humans doing moral reasoning about problems they’ve never considered before and SFT on that”.
Those high quality transcripts are certainly useful, especially for stage 3, “Balance and analyse the concepts to reach a verdict”. But when humans recognise that a situation is out of distribution, or see which features are missing from the first-pass assessment, they don’t operate via linguistically thinking through the situation (see e.g. Klein’s work and other related work on human expertise).
There are some skills—most skills—that can’t be learnt just from books. For instance social and business skills. However, it’s often useful to learn these skills with books, which is why non-useless business and social skill books exist.
So I see those high quality transcripts as complements to the value generalisation process. They will guide an agent that has strong generalisation abilities, allowing them to generalise faster and better.
I’m confused about what this research direction could concretely look like, can you give some examples? Do you mostly mean stuff like “get high quality transcripts of humans doing moral reasoning about problems they’ve never considered before and SFT on that”, or something altogether different?
I won’t share all details, but the plan is not “get high quality transcripts of humans doing moral reasoning about problems they’ve never considered before and SFT on that”.
Those high quality transcripts are certainly useful, especially for stage 3, “Balance and analyse the concepts to reach a verdict”. But when humans recognise that a situation is out of distribution, or see which features are missing from the first-pass assessment, they don’t operate via linguistically thinking through the situation (see e.g. Klein’s work and other related work on human expertise).
There are some skills—most skills—that can’t be learnt just from books. For instance social and business skills. However, it’s often useful to learn these skills with books, which is why non-useless business and social skill books exist.
So I see those high quality transcripts as complements to the value generalisation process. They will guide an agent that has strong generalisation abilities, allowing them to generalise faster and better.