Excellent summary, Harrison! I especially enjoyed your use of pillar diagrams to break up streams of text. In general, I found your post very approachable and readable.
As for Pillar 2: I find the description of goals as “generalised concepts” still pretty confusing after reading your summary. I don’t think this example of a generalised concept counts as a goal: “things that are perfectly round are objects called spheres; 6-sided boxes are objects called cubes”. This statement is a fact, but a goal is a normative preference about the world (cf. the is-ought distinction). Also, I think the ‘coherence’ trait could do with slightly more deconfusion—you could phrase it as “goals are internally consistent and stable over time”.
I think the most tenuous pillar in Ngo’s argument is Pillar 2: that AI will be agentic with large-scale goals. It’s plausible that the economic incentives for developing a CEO-style AI with advanced planning capabilities will not be as strong as stated. I agree that there is a strong economic incentive for CEO-style AI which can improve business decision-making. However, I’m not convinced that creating an agentic AI with large-scale goals is the best way to do this. We don’t have enough information about which kinds of AI are most cost-effective for doing business decision-making. For example, the AI field may develop viable models that don’t display these pesky agentic tendencies. (On the other hand, it does seem plausible that an agentic AI with large-scale goals is a very parsimonious/natural/easily-found-by-SGD model for such business decision-making tasks.)
Excellent summary, Harrison! I especially enjoyed your use of pillar diagrams to break up streams of text. In general, I found your post very approachable and readable.
As for Pillar 2: I find the description of goals as “generalised concepts” still pretty confusing after reading your summary. I don’t think this example of a generalised concept counts as a goal: “things that are perfectly round are objects called spheres; 6-sided boxes are objects called cubes”. This statement is a fact, but a goal is a normative preference about the world (cf. the is-ought distinction).
Also, I think the ‘coherence’ trait could do with slightly more deconfusion—you could phrase it as “goals are internally consistent and stable over time”.
I think the most tenuous pillar in Ngo’s argument is Pillar 2: that AI will be agentic with large-scale goals. It’s plausible that the economic incentives for developing a CEO-style AI with advanced planning capabilities will not be as strong as stated. I agree that there is a strong economic incentive for CEO-style AI which can improve business decision-making. However, I’m not convinced that creating an agentic AI with large-scale goals is the best way to do this. We don’t have enough information about which kinds of AI are most cost-effective for doing business decision-making. For example, the AI field may develop viable models that don’t display these pesky agentic tendencies.
(On the other hand, it does seem plausible that an agentic AI with large-scale goals is a very parsimonious/natural/easily-found-by-SGD model for such business decision-making tasks.)