It would certainly be valuable to have AIs that are more respected than Wikipedia as a source of knowledge.
I have some concerns about making AIs highly strategic. I see some risk that strategic abilities will be the last step in the development of AI that is powerful enough to take over the world. Therefore, pushing AI intellectuals to be strategic may speed up that risk.
I suggest aiming for AI intellectuals that are a bit more passive, but still authoritative enough to replace academia as the leading validators of knowledge.
Maybe we could call this something like “Strategic Determinism”
I think one more precise claim I could understand might be: 1. The main bottleneck to AI advancement is “strategic thinking” 2. There’s a decent amount of uncertainty on when or if “strategic thinking” will be “solved” 3. Human actions might have a lot of influence over (2). Depending on what choices humans make, strategic thinking might be solved sooner or much later. 4. Shortly after “strategic thinking” is solved, we gain a lot of certainty on what future trajectory will be like. As in, the fate of humanity is sort of set by this point, and further human actions won’t be able to change it much. 5. “Strategic thinking” will lead to a very large improvement in potential capabilities. One main reason is that it would lead to recursive self-improvement. If there is one firm that has sole access to an LLM with “strategic thinking”, it is likely to develop a decisive strategic advantage.
I think personally, such a view seems too clean to me. 1. I expect that there will be a lot of time where LLMs get better at different aspects of strategic thinking, and this helps to limited extents. 2. I expect that better strategy will have limited gains in LLM capabilities, for some time. The strategy might suggest better LLM improvement directions, but these ideas won’t actually help that much. Maybe a firm with a 10% better strategist would be able to improve it’s effectiveness by 5% per year or something. 3. I think there are could be a bunch of worlds where we have “idiot savants” who are amazing at some narrow kinds of tasks (coding, finance), but have poor epistemics in many ways we really care about. These will make tons of money, despite being very stupid in important ways. 4. I expect that many of the important gains that would come from “great strategy” would be received in other ways, like narrow RL. A highly optimized-with-RL coding system wouldn’t benefit that much with certain “strategy” benefits. 5. A lot of the challenges for things like “making a big codebase” aren’t to do with “being a great strategist”, but more with narrower problems like “how to store a bunch of context in memory” or “basic reasoning processes for architecture decisions specifically”
I didn’t spend much time on the limitations of such intellectuals. For the use cases I’m imagining, it’s fairly fine for them to be slow, fairly expensive (maybe it would cost $10/hr to chat with them), and not very great at any specific discipline. Maybe you could spend $10 to $100 and get the equivalent of one Scott Alexander essay, on any topic he could write about, for example.
I think that such a system could be pretty useful in certain AI agents, but I wouldn’t expect it to be a silver bullet. I’m really unsure if it’s the “missing link.”
I expect that a lot of these systems would be somewhat ignored when it comes to using them to give humans a lot of high-level advice, similar to how prediction markets or econ experts get ignored.
It’s tricky to understand the overlap between high-level reasoning as part of an AI coding tool-chain (where such systems would have clear economic value), and such reasoning in big-picture decision-making (where we might expect some of this to be ignored for a while). Maybe I’d expect that the narrow uses might be done equally well using more domain-specific optimizations. Like, reinforement learning on large codebases already does decently well on a lot of the “high-level strategy” necessary (though it doesn’t think of it this way), and doesn’t need some specialized “strategy” component.
I expect that over time we’ll develop better notions about how to split up and categorize the skills that make up strategic work. I suspect some things will have a good risk-reward tradeoff and some won’t.
I expect that people in the rationality community over-weight the importance of, well, rationality.
I suggest aiming for AI intellectuals that are a bit more passive, but still authoritative enough to replace academia as the leading validators of knowledge.
My main point with this topic is that I think our community should be taking this topic seriously, and that I expect there’s a lot of good work that could be done that’s tractable, valuable, and safe. I’m much less sure about exactly what that work is, and I definitely recommend that work here really try to maximize the reward/risk ratio.
Some quick heuristics that I assume would be good are: - Having AIs be more correct about epistemics and moral reasoning on major global topics generally seems good. Ideally there are ways of getting that that don’t require huge generic LLM gains. - We could aim for expensive and slow systems. - There might not be a need to publicize such work much outside of our community. (This is often hard to do anyway). - There’s a lot of work that would be good for people we generally trust, and alienate most others (or be less useful for other use cases). I think our community focuses much more on truth-seeking, Bayesian analysis, forecasting, etc. - Try to quickly get the best available reasoning systems we might have access to, to be used to guide strategy on AI safety. In theory, this cluster can be ahead-of-the-curve. - Great epistemic AI systems don’t need much agency or power. We can heavily restrict them to be tool AIS. - Obviously, if things seriously get powerful, there are a lot of various techniques that could be done (control, evals, etc) to move slowly and lean on the safe side.
It would certainly be valuable to have AIs that are more respected than Wikipedia as a source of knowledge.
I have some concerns about making AIs highly strategic. I see some risk that strategic abilities will be the last step in the development of AI that is powerful enough to take over the world. Therefore, pushing AI intellectuals to be strategic may speed up that risk.
I suggest aiming for AI intellectuals that are a bit more passive, but still authoritative enough to replace academia as the leading validators of knowledge.
“I see some risk that strategic abilities will be the last step in the development of AI that is powerful enough to take over the world.”
Just fyi—I feel like this is similar to what others have said. Most recently, benwr had a post here: https://www.lesswrong.com/posts/5rMwWzRdWFtRdHeuE/not-all-capabilities-will-be-created-equal-focus-on?commentId=uGHZBZQvhzmFTrypr#uGHZBZQvhzmFTrypr
Maybe we could call this something like “Strategic Determinism”
I think one more precise claim I could understand might be:
1. The main bottleneck to AI advancement is “strategic thinking”
2. There’s a decent amount of uncertainty on when or if “strategic thinking” will be “solved”
3. Human actions might have a lot of influence over (2). Depending on what choices humans make, strategic thinking might be solved sooner or much later.
4. Shortly after “strategic thinking” is solved, we gain a lot of certainty on what future trajectory will be like. As in, the fate of humanity is sort of set by this point, and further human actions won’t be able to change it much.
5. “Strategic thinking” will lead to a very large improvement in potential capabilities. One main reason is that it would lead to recursive self-improvement. If there is one firm that has sole access to an LLM with “strategic thinking”, it is likely to develop a decisive strategic advantage.
I think personally, such a view seems too clean to me.
1. I expect that there will be a lot of time where LLMs get better at different aspects of strategic thinking, and this helps to limited extents.
2. I expect that better strategy will have limited gains in LLM capabilities, for some time. The strategy might suggest better LLM improvement directions, but these ideas won’t actually help that much. Maybe a firm with a 10% better strategist would be able to improve it’s effectiveness by 5% per year or something.
3. I think there are could be a bunch of worlds where we have “idiot savants” who are amazing at some narrow kinds of tasks (coding, finance), but have poor epistemics in many ways we really care about. These will make tons of money, despite being very stupid in important ways.
4. I expect that many of the important gains that would come from “great strategy” would be received in other ways, like narrow RL. A highly optimized-with-RL coding system wouldn’t benefit that much with certain “strategy” benefits.
5. A lot of the challenges for things like “making a big codebase” aren’t to do with “being a great strategist”, but more with narrower problems like “how to store a bunch of context in memory” or “basic reasoning processes for architecture decisions specifically”
Alexander Gordon-Brown challenged me on a similar question here:
https://www.facebook.com/ozzie.gooen/posts/pfbid02iTmn6SGxm4QCw7Esufq42vfuyah4LCVLbxywAPwKCXHUxdNPJZScGmuBpg3krmM3l
One thing I wrote there:
I expect that over time we’ll develop better notions about how to split up and categorize the skills that make up strategic work. I suspect some things will have a good risk-reward tradeoff and some won’t.
I expect that people in the rationality community over-weight the importance of, well, rationality.
My main point with this topic is that I think our community should be taking this topic seriously, and that I expect there’s a lot of good work that could be done that’s tractable, valuable, and safe. I’m much less sure about exactly what that work is, and I definitely recommend that work here really try to maximize the reward/risk ratio.
Some quick heuristics that I assume would be good are:
- Having AIs be more correct about epistemics and moral reasoning on major global topics generally seems good. Ideally there are ways of getting that that don’t require huge generic LLM gains.
- We could aim for expensive and slow systems.
- There might not be a need to publicize such work much outside of our community. (This is often hard to do anyway).
- There’s a lot of work that would be good for people we generally trust, and alienate most others (or be less useful for other use cases). I think our community focuses much more on truth-seeking, Bayesian analysis, forecasting, etc.
- Try to quickly get the best available reasoning systems we might have access to, to be used to guide strategy on AI safety. In theory, this cluster can be ahead-of-the-curve.
- Great epistemic AI systems don’t need much agency or power. We can heavily restrict them to be tool AIS.
- Obviously, if things seriously get powerful, there are a lot of various techniques that could be done (control, evals, etc) to move slowly and lean on the safe side.
I’d lastly flag that I sort of addressed this basic claim in “Misconceptions 3 and 4” in this piece.