Donated $600 for now. My current (noncommital) guess is that I will continue supporting your work with small donations over the next few months if you remain somewhat funding-constrained.
Tejas
Karma: 16
I’ve been very impressed by Lightcone’s work recently. How valuable would you say small donations (say, on the order of ~$1k) are on the current margin?
Consider applying to a career transition grant.
That analogy is helpful, thanks. I think I still feel like there’s a gap between developing a good theory of decision-making and putting that theory into immediate practice. For instance, immediately after developing expected utility theory, I could imagine there’s a processing gap among someone who isn’t used to using it to make everyday decisions; I feel a meaningful amount of doubt that von Neumann was much less likely to read books while driving after internalizing expected utility theory (obviously, maybe his preference for doing this was just that strong, but let’s suppose it was in fact irrational of him!). I’d guess this would especially be true with high-pressure decision-making.
I don’t have a good picture of what solving agent foundations would look like, but I could also imagine a lot of theoretical problems being solved in ways that are not immediately translatable into decision-making. Expected utility theory has the advantage of feeling “simple,” in some way, to translate into a real-world decision. Do you think it would be comparably easy to intuit this kind of solution? (Genuinely unfamiliar here, and am open to believing that it would be!)
I’m persuaded, however, that if solving agent foundations would lead to a theory that powerful, it would cause better decision-making. I’m still intuitively skeptical that it would be better enough; I’d guess I would have to try to visualize object-level solutions better to get some intuition for why.
(sorry for the nitpick but I think you meant to say EDT with updatefulness double counts?)
I think their plan is more robust than you expect, because “solving agent foundations” is not like “writing down some code”, it’s more like “totally refactoring your ontology for how rational agents make decisions”. From an outside view, having the best understanding of how it’s rational to make decisions does seem like it’ll very plausibly help you not make obvious mistakes. And from my inside view, I actually think that the solution to agent foundations is specifically action-guiding on questions like “when and how should I try to accumulate power?” (I expect leading agent foundations researchers like Scott Garrabrant and Andrew Critch to agree with this claim.)
This seems exaggerated to me. I’m only somewhat familiar with the agent foundations agenda, but I think there are a bunch of ways people’s decision-making is bounded in practice (e.g., biology and cognitive limits, the difficulties of operating in a high-stakes environment) that understanding the mathematics of rational agency (in the context of trying to build friendly AI) doesn’t translate to, on a meta-level, making much better decisions about practical questions like when to start trying to build it.
I don’t have a view on the broader discussion about MIRI.
(Note: I edited this comment to change “better” to “much better,” as I imagine it would in fact cause people to make better decisions.)
This is the best initial attempt I’ve seen at answering this question (or something like it, anyway). I don’t think it has a clear answer to many of the thorny cases though.
I agree with that!
The idea that there are no more good ideas and it’s all just scaling from here
Fwiw, this is not how I understand his take. I think he’s saying AI progress will be bottlenecked by compute (and human expert data), which I interpret to mean that the elasticity of substitution between compute/data and ideas isn’t high enough. In fact, I think this his view includes compute to run experiments to do AI research.
I feel like there’s some kind of disanalogy between the prime factorization case (where the goal itself is well-defined (the job is to “find the prime factorization, where N has a process-independent prime factorization”), and hence processes can be refined to become better processes in reaching the goal) and normative goals under the broadly-Humean framework (where the output of the procedure is the goal, and there’s no further goal to reach).
That said, I think there’s something in the vicinity of this that feels right to me. Maybe it has to do with the fact that I don’t know what the process I would actually endorse is, and in some sense, picking a process is a form of attempted discovery (about who I am/the kind of person I am and will be) rather than a choice I already have in hand.
It seems to me that:
If you had a sudden influx of 100m high-skilled immigrants (for simplicity, just assuming this is ordinary very high-skilled immigrants, and let’s say they don’t have above-replacement fertility), the long-run growth rate wouldn’t actually change much. Besides the “declining research productivity” issue common to semi-endogenous models, I’d guess this would also be pretty bottlenecked by energy, housing/zoning laws, etc. A population level effect, which doesn’t substantially change the population growth rate, seems really unlikely to me to produce significantly accelerated growth for longer than 10-20 years. I’d expect, setting aside institutional issues like political backlash, the per-capita GDP growth rate to look something like: (i) For the first 2-5 years, absorption costs bottleneck growth heavily, so growth bumps up to ~2.5-4% a year but not more. (ii) After that, you get a ~10-20 year period of elevated growth, say 4-6% a year in per capita terms. (iii) Eventually, growth gradually moves back down to ~2% a year, gradually slowing down year after year. (iv) After the arriving cohort retires, growth falls below trend for around ~a decade. (v) Finally, growth returns to the ~1.8-2% long-run trend.
AI could in principle be much more transformative than this, because the initial “level effect” would trigger a further increase in the number of “digital workers,” in turn causing another productivity boost which once more is reinvested in creating even more digital workers, and so on. (This is assuming other bottlenecks don’t bind. I think it’s pretty plausible they don’t end up stopping the substantial growth boost, to be honest! My current best guess is that AGI, in many reasonable operationalizations of the term, would in fact cause explosive economic growth.)
As a moral anti-realist, I’m sympathetic to something sort of like Humean constructivism, although with a substantial component of “self creation”/understanding that, at the bottom, it’s still on me. In that case, though, I kind of think the values I end up with upon reflection — if the reflection happens in a way I endorse — are what I’d consider my “true values.” This also means that, if the reflection process is underspecified, I get to specify the idealization process I’d like, as it is a process of making, and discovering, myself according to the filters I consider valuable.
To be sure, I take pretty seriously that the reflection process of society wouldn’t necessarily be either the reflection process I would prefer nor lead to the outcomes I’d consider valuable. I also think I’d have to think pretty hard about what the process looks like for me; so I agree with a lot of your comment.
Some combination of:
general work to promote better epistemics and integrity among people working on AI safety (e.g., through maintaining LessWrong),
specific work to help maintain useful AI safety infrastructure (e.g., helping with design and increasing traffic to very valuable research projects, helping build good funding systems for AI safety that I trust),
and more recent specific work to capitalize on recent attention on AI safety (e.g., collcongress.ai).