ryan_b
Karma: 1,123
NewTop
Fortunately, because APM factories will likely be both technologically complex and optimized for use in fabricating specific materials rather than every material, the military risk of providing factories designed with food and infrastructure in mind is relatively low.
While I agree with this as written, I expect the difference is smaller than one would expect.
Given the choice between an opponent with APM armaments and a standard infrastructure, or an opponent with standard armaments and an APM infrastructure, the latter is a greater military threat.
It is trivially easy to militarize any given civilian tech: most military vehicles are armed and up-armored civilian ones anyway; armed drones initially were literally and exactly unarmed drones with missiles bolted to them; the opponents history’s foremost military power have been fighting these last 17 years have been doing it with old gear, gear provided to them by outsiders, and armed civilian gear.
While APM-hybrid equipment may be inferior to APM-optimized equipment, it will probably still be superior to standard equipment.
In the current security environment it is common to provide more advanced armaments to less advanced belligerents, and the nature of APM seems like it would make reverse engineering more difficult which suggests to me providing such armaments is an even safer bet than usual.
The role of logistics (which is to say, infrastructure) in modern military affairs is widely underappreciated, and I have no reason to suspect Drexler of any particular expertise in this area. In sum, it looks to me like the security impacts have been underrated in their complexity. Although I note this effectively pushes more strongly in the direction of your sense of being uncertain if the risks are worth the gains.
I should have made this specific, but I had not considered using such a thing for producing writing for other people’s consumption. What I wanted from Grammarly was this:
1) The latest grammar analysis.
2) The instant feedback.
With this, I envisioned two probable uses:
A) Writing your own thoughts down as notes. Thinkerly catches possible errors. This improves stream-of-consciousness writing as a tool for training better thinking, because the feedback loop is much tighter than with the draft-revision format to which we are usually constrained.
B) Looking critically at something from somewhere else. This seems like it would be more useful on the margins, because it is very easy even for skilled thinkers to accidentally rely on a few suspect thoughts.
I can’t see any way for it to drop in to writing workflow the same way as spellcheckers do now, because I don’t see how it could make good suggestions about replacements the way spellcheckers do. Even if there are signatures of poor thinking, that doesn’t mean there is a corresponding correct thought the way there is with spelling.
I wonder how much of the effectiveness of small groups is due to being able to see information people don’t intend to communicate. Under a lot of pressure you get to see the other person under pressure, and how they react, and whether they act differently afterward.
If their behavior changes, it would be reasonable to infer some kind of gut-level-crux was at work. This information is not otherwise available.
I had this thought too. It seems like there is a significant difference between ‘these resources are here if you want them’ and ‘doing these things is a part of your job’, though. Even just switching from opt-in to opt-out is a pretty big impact in lots of contexts, and I expect strategic pressure to participate to be more impactful still.
Honored to hear from you!
The intuition I have is that in a research context striking out isn’t just overhead but a positive contribution; all the other people working on the problem can now see that is not the answer. We can also look at why it wasn’t the answer, which is a source of new information. Therefore the next guy is more likely to get a hit.
It seems like everyone treats this kind of thing as trivial—it’s why we have scientific journals after all—but what I don’t see is much articulation of what value comes from where, and how to keep it. It looks to me like Xerox PARC did an amazing job of capitalizing on all of the information valuable to progress, and I suspect that’s because it was captured in the environment.
As a specific example, you have mentioned elsewhere that peer review didn’t make sense for PARC. Clearly eliminating the bureaucracy was a factor, but I suspect it is more important that what was happening instead did a superior job of delivering the same value that peer review is meant to. The team-of-peers has knowledge of the environment, familiarity with the previous work, contact with the generative process for an idea, and they can provide a new perspective on most any element of each other’s work at any time. Regular peer review is a static and passive check of correctness; because the PARC example was active and dynamic I want to call it “peer stabilization”.
I guess what I am gesturing at is the group is the unit of action. I suspect that if we want to do great things, or even just good things consistently, we need to build the context for the group. Then if it is made up of amazing people it will do amazing things.
Maybe we can disentangle the context from the vision, or the how from the why. Then we could move building powerful contexts into technical execution territory, waiting only for an appropriate vision or need to motivate them. I bet if I could break all this down into “value-added” language businesses and governments would be more willing to give it a shot.
Where athletic coaches have drills ready, for research I feel like it would be more like a procedure for identifying and rectifying a mistake. I strongly suspect that this falls under the heading of “things good researchers do anyway”, for example:
1. When checking a conclusion, notice that one element of the arguments is too weak
2. That element is too weak because it lies outside the researcher’s core of expertise and so the implications were unclear to them
3. The researcher seeks out a colleague who has better expertise so as to understand the implications better
The thing is I expect a very large difference between this being something a researcher may or may not do on their own, versus something that will happen because it is the group expectation and everything is organized to make it as easy as possible. The more reliable this kind of supporting infrastructure is, the more we could extend it down below the level of genius (to turn mediocre researchers into competent ones, say).
Well this seems to argue strongly for reviewing old technology proposals at some time interval. I wonder if we could develop some kind of test to apply that would make it easy to gauge whether an in-depth investigation makes sense?
If we could develop such a test, would it be worthwhile to apply it to new ideas immediately as a way to determine their viability and as an indicator of what the criteria would be for coming back to it?
Empathy; I wish it was going differently for you.
That being said there’s some interesting stuff here that I would like to hear more about.
What does influence on the social environment look like to you?
I notice you don’t talk at all about the outcomes of the volunteering projects you did. What did you think of them, apart from the effect on status?
Does it seem to you like the EA volunteer efforts are organized to allow for the flakiness you describe, or does it seem like they are being impacted negatively?
Why don’t we treat geniuses like professional athletes?
Thinkerly: Grammarly for writing good thoughts
Strong upvote for addressing what I feel is a neglected subject.
It feels like it would be helpful to state explicitly that working towards AI alignment and working against the development of misaligned AIs are not necessarily the same. In the casual discussions on the subject we usually refer to a military or multinational corporation as candidates who would build an AI lab, drive towards AGI, and then unleash the poorly-aligned result. The policy/strategy question goes directly to their behavior.
It seems like the time we have available to get this right is heavily influenced by how these other actors make decisions, and currently there is no particular pressure on them to make good ones. I’d like to toss a few other potential benefits of a strategy/policy echelon:
1. It would serve as a contact surface for people who are already in strategy and policy to engage with AI safety. Currently they have to use the same personal-interest method as the rest of us.
2. Aside from the institutional examples Richard provided, I point to Jean Monnet and the formation of the precursors to the European Union. Individual people are in a position to have very large influence if they have a framework ready when the opportunity presents itself.
3. Consider the risk of being unprepared if AI risk should come to the forefront of public consciousness and the government decides to act. The converse of Ben’s example where politicians abandon projects when the public loses interest is that a public outrage can drive the government into hasty action. For example, if the Russians/Chinese deploy a next-generation narrow AI in their weapons systems, or an American military AI test goes badly wrong, or if there are casualties from a commercial implementation of narrow AI the government may move to regulate AI research and funding, and there is no reason to suspect that law would be any better than the computer crime laws we have currently. I would go as far as to say that AI is the best candidate for a new Sputnik Moment, which seems like it would drive the incentives heavily in a direction we do not want.
I’m curious about the limits of the discretization strategy. How does this interact with communicating a complex conclusion? How does it sustain a narrative?
From reading regular marketing copy it feels like the strategy is very self-reinforcing; once adopted, there isn’t even any point in reading the whole thing because it communicates little-to-nothing. So these days it is more like not reading regular marketing copy; when I see name-dropping and bold words I bail unless I have a pressing reason to continue.
I am stealing this term because it does a good job of articulating specific effects and communicating magnitude at the same time.
How do you disentangle something like this from something that keeps the relevant expertise but heavily impacts scale, like a drop in cost below some critical threshold?
This is an interesting point, but I strongly suspect the motivation is completely different here. Scott’s goal is to allow people to progress from the beginning to the end in digestible chunks, so as to avoid blockers. By contrast, the marketing discretization strategy assumes most of the audience won’t read the whole thing, so they try to make the parts more or less independent so different levels of attention still get some kind of message.
Is it safe for me to assume that all the relevant sub-analogies within the dialogue have been linked already? By that I mean, wherever the specific example is an analogy for something actually from MIRI, that information is linked?
This made me want a State of MIRI Research post for the layperson, in the style of “the way hard problems are handled” instead of situating projects in their respective fields or within the objective of alignment. I feel like this post crystallized the meta-research level in a way I have not encountered before, and that might be an important contribution in its own right.
Agreed, but what it accomplishes is helping to make total distribution of risk have the shape he is advocating. When you said ‘taken literally’ I inferred that to mean we should try and make our total risk distribution look like a lopsided barbell, and it seems like the division between investing in a specific asset class and the multidimensional risk you describe dissolves under that construction.
I would expect that people with enough money to invest in Treasury bonds and startups almost all have life insurance, so they have already taken the same hedging step with the same goal as the low-savings couple. This makes it seem like they might have the same risk strategy, but execution of strategies takes place over time and it just unfolds more slowly because they have less income.
But having articulated it in that way, it occurs to me that there is a very big difference between deploying the barbell strategy in each risk dimension, and deploying the barbell strategy for total risk. That seems to shift the most important task from distinguishing risks within a dimension to comparing risk between dimensions, because investing a lot of thought in managing a dimension with very low risk-mass doesn’t affect total risk very much. Further, some dimensions may be all exposure-to-cost and little to no exposure-to-benefit, as with disability/death.
I like the insight that there is also risk affecting the dimension itself, as in the freezing of assets example. Logically there must also be the possibility of new risk dimensions appearing, which a little reflection suggests is obvious at least in the sense of our learning about them even if they were there all along, like germ theory. That makes me wonder about using risk management techniques to detect unknown dimensions.
Edit: I’m not driving at anything in particular here, this is all just a bit stream of consciousness that I wanted to note.
I have two opposing suspicions about this.
The first is that I expect the Western countries are much more familiar with the concept of psychological tests and games. I therefore suspect that they are more likely to be thinking that this is a game, and it has a score, and obviously you should try to win.
The second is based on my experiences in Baghdad; when we walked around the city we noticed mounds of trash everywhere, and so we expected the homes to be in bad shape inside. This was a mistake; the inside of a Baghdadi home was always immaculate. The conclusion I drew was that they just weren’t drawing their sense of obligation from geography; there was no sense of neighborhood, only of family and tribe. I therefore expect that at least the people from Muscat and Riyadh don’t react much to any kind of signals from strangers, because they didn’t have any expectations in the first place. This is often what people are measuring when they say ‘norms of civic cooperation and rule of law’.
It feels like buying life insurance accomplishes the ‘minimize exposure to large downsides’ objective he discusses. What’s the difference, utility-wise, between life insurance that pays out to their spouse/children in the event of their death and a stock that pays off the same amount at the same probability as death?
It seems like these two things should be comparable, but that the life insurance investment is preferable because it mitigates the problem of the children not being provided for (no exposure to large downside) whereas the stock does not (exposure to large upside, still have exposure to large downside).
It occurs to me a lot of these problems effectively stem from the fact that there is a fundamental bias towards large-downside risks; we can be killed or crippled, but there is no symmetric likelihood of living forever or becoming superhuman (for now).
I don’t see why the insight fails to apply along a continuum. If you do not have access to Treasury-bond equivalents in the risk you are trying to manage, take whatever the stablest option is for most, and then adjust your exposure-to-large-gains appropriately.
Perhaps this would be something like maintain relationships with your diverse friend group as the safe investment, and then do something for fun once a month where you might make friends with a billionaire as the exposure-to-large-upside investment.
I have always considered this a unilateral good of the proposal, for employers and consumers.
Employing people is hard, and people who just don’t show up or walk off halfway through the shift without saying a word is a perpetual problem, even after sorting through hundreds of resumes. If all the people who don’t want to work at all drop out, that means fewer resumes to sort through and probably also means higher quality applicants. It seems like an information advantage to businesses, which is a big deal as information is the hardest part about business.
From the consumer side, interacting with people who really don’t want to be there really sucks. I don’t like patronizing places that treat people poorly, and there is always the problem of the indifferent or negligent employee who screws up my order or damages my merchandise or whatever. I would prefer working with lower-stress, less-desperate people whenever possible.