(Another non-native having a go at it...)
When your advice both ways seems fine,
Calibrate, then make it rhyme.
alkexr
Karma: 270
You can tell a drawing from a painting
Simulacrum Levels of Trust
You’re saying that these hypothetical elites are hypercompetent to such a hollywoodical degree that normal human constraints that apply to everyone else don’t apply to them, because of “out of distribution” reasons. It seems to me that “out of distribution” here stands as a synonym of magic.
You’re saying that these hypothetical elites are controlling the world thanks to their hypercompetence, but are completely oblivious to the idea that they themselves could lose control to an AI that they know to be hypercompetent relative to them.
It seems to me that lie detection technology makes the scenario you’re worried about even less likely? It would be enough for just a single individual from the hypothetical hypercompetent elites to testify under lie detection that they indeed honestly believe that AI poses a risk to them.
It’s worth pointing out I suppose that the military industrial complex is still strongly interested in the world not being destroyed, no matter how cartoonishly evil you may believe they are otherwise, unless they are a Cthulhu cult or something. They could still stumble into such a terrible outcome via arms race dynamics, but if hypercompetence means anything, it’s something that makes such accidents less, not more, likely.
My arguments end here. From this point on, I just want to talk about… smell. Because I smell anxiety.
Your framing isn’t “here is what I think is most likely the truth”. Your framing is “here is something potentially very dangerous that we don’t know and can’t possibly ever really know”.
Also, you explicitly, “secretly” ask for downvotes. Why? Is something terrible going to happen if people read this? It’s just a blogpost. No, it’s not going to accidentally push all of history off course down into a chasm.
Asking for downvotes also happens to be a good preemptive explanation of negative reception. Just to be clear, I downvoted not because I was asked to. I downvoted because of poor epistemic standards.
Do note that I’m aware that very limited information is available to me. I don’t know anything about you. I’m just trying to make sense of the little I see, and the little I see strongly pattern matches with anxiety. This is not any sort of an argument, of course, and there isn’t necessarily anything wrong with that, but I feel it’s still worth bringing up.
(Meta: writing this in separate comment to enable voting / agreement / discussion separately)
If you want to make the case for tactical nuclear deployment not happening (which I hope is the likely outcome), I want to see a model of how you see things developing differently
I’ll list a few possible timelines. I don’t think any of these is particularly likely, but they are plausible, and together with many other similar courses of events they account for significant chunks of probability mass.
Discontinuity in power in Russia.
Internal turmoil or collapse in Russia (e.g. regions start declaring independence). It becomes clear that nuclear weapons won’t save Russia.
Abrupt cut in western support to Ukraine, including ammunition (e.g. due to another big war). Putin thinks he can win without nuclear weapons.
Russian army starts being competent. Putin thinks he can win without nuclear weapons.
Conflict freezes over winter, then turns into boiling-the-frog: events happening too slowly to trigger nuclear response. Over the years defeat slowly becomes an accepted fact in Russia.
On Nord Stream sabotage:
Looks like sabotage. Accidents very rarely look like this. (Very high confidence.)
Probably by state actors. It seems like a task that requires significant resources and planning. Also there was plenty of military presence in the area, it’s just hard to believe that non-state actors could perform something like this unnoticed. (Medium confidence.)
It wasn’t an ally of Germany. There is always a chance that you get caught / leave evidence, and after attacking the critical infrastructure of an ally no one will have a reason to trust you. None of the allies of Germany stand to gain anything that is anywhere near comparable to that risk (that I can think of). (High confidence.)
Given geographic reach, the countries that could perform something like this are probably those around the Baltic sea, plus USA, UK, France. (Medium confidence.)
That leaves us with Russia and Germany. I don’t see what Germany could gain from this. I don’t see what Russia could gain from this either, but then Russia has developed a habit of doing things despite having nothing to gain from them. Also, I see some reasons why Russia could think this is a good idea (implicitly threatening the West by demonstrating willingness to use grey-zone warfare against their critical infrastructure, to try to get them to back down).
So possibly Russia. (Low confidence.)
Epistemic status: proof by lack of imagination.
Your is correlated with , and that’s cheating for all practical purposes. The premise of Goodhart’s law is that you can’t measure your true goal well. That’s why you need a proxy in the first place.
If you select a proxy at random with the only condition that it’s correlated with your true goal in the domain of your past experiences, Goodhart’s law claims that it will almost certainly not be correlated near the optimum. Emphasis on “only condition”. If you specify further conditions, like, say, that your proxy is your true goal, then, well, you will get a different probability distribution.
My gut feeling is that attracting more attention to a metric, no matter how good, will inevitably Goodhart it.
That is a good gut feeling to have, and Goodhart certainly does need to be invoked in the discussion. But the proposal is about using a different metric with a (perhaps) higher level of attention directed towards it, not just directing more attention to the same metric. Different metrics create different incentive landscapes to optimizers (LessWrongers, in this case), and not all incentive landscapes are equal relative to the goal of a Good LessWrong Community (whatever that means).
I am not sure what problem you are trying to solve, and whether your cure will not be worse than the disease.
This last sentence comes across as particularly low-effort, given that the post lists 10 dimensions along which it claims karma has problems, and then evaluates the proposed system relative to karma along those same dimensions.
The way this topic appears to me is that there are different tasks or considerations that require different levels of conscientiousness for the optimal solution. In this frame, one should just always apply the appropriate level of conscientiousness in every context, and the trait conscientiousness is just a bias people have in one direction or the other that one should try to eliminate.
This frame is useful, because it opens up the possibility to do things like “assess required conscientiousness for task”, “become aware of bias”, “reduce bias”, etc. But it may also be wrong in an important way. It’s somewhere between difficult to impossible to tell how much conscientiousness is required in any particular case, what’s more, even what constitutes an optimal solution may not be obvious. In this frame, trait conscientiousness is not bias, but diversity that nature threw against the problem to do selection with.
I have trouble understanding why, in this case, everyone would need to have a consistently high or consistently low level of it across a wide range of contexts; and why, for example, one can’t just try a range of approaches of varying consientiousness levels and learn to optimize from the experience. It isn’t necessary in any of the examples above to have a person involved with consistently low levels of it, just a person who in that particular case takes the low-conscientiousness approach. This way we could still fall back on the interpretation as a bias and blame nature for just being inefficient doing the evolution biologically instead of memetically.
Some frames worth considering:
Strong Prune, weak Babble among LessWrongers
Conversation failing to evolve past the low-hanging fruit
People being reluctant to express thoughts that might make their account look stupid in a way that’s visible to the entire internet
Everyone can participate, and as the number of people involved in a conversation increases it becomes more and more difficult to track all positions
Even lurkers like me can attempt to participate, and it’s costly in terms of conversational effort to figure out what background knowledge someone is missing
Most topics that appear on LessWrong are suited for mental masturbation only, they offer no obvious course of action through which people can decide to care about said topics
There is way, way too much content (heck, I’ve only skimmed through the comments under this post)
Long-running conversations don’t tend to happen; therefore there is little incentive in delving deep into one topic, so people (well, me at least) end up engaging with more topics, but in a shallow manner; which in turn creates conditions where long-running conversations are less likely
Due to the way the platform is designed, the only real way to maintain a long-running conversation between persons A and B is if the pattern of response is ABABABAB..., so anyone not being confident at any point is a single point of failure
I also have a suggestion. After the discussion here inevitably fades, you could write another post in which you summarize the main takeaways and steelman a position or two that look valuable at that time. That might generate further discussion. Repeat. This way you could attempt to keep the flame of the conversation alive. But if you end up doing this, make sure to give the process a name, so that people realize that this is a Thing and that they are able to participate in this-Thing-specific ways.
Absence of evidence of X is evidence of absence of X.
A claim about the absence of evidence of X is evidence of:
the speaker’s belief of the listeners’ belief in X,
absence of evidence of NOT X,
the speaker’s intention of changing the listeners’ belief in X.
No paradox to resolve here.
Yes, I’m aware of all that, and I agree with your premises, but your argument doesn’t prove what you think it does. Let’s try to reductio it ad absurdum, and turn the same argument against the possibility of fast technological or scientific feedback cycles.
If you live in a technologically backwards society (think bronze age), you can’t become more advanced technologically yourself, because you’ll starve spending your time trying to do science. The technology of society (including agriculture, communication, tools, etc.) needs to progress as a whole. If you live in a scientifically backwards society, you can’t have more accurate beliefs, because you’ll be burned at the stake by all the people believing in nonsense. Therefore, science and technology can only progress as fast as the majority can adopt it.
And all of the above is true, actually, up to a certain point in history. But once the scientific understanding of society advances to the point where it understands that science is a thing and has a basic understanding of how science works, it can basically create a mesa-feedback-loop. Similarly, once you have technologies like writing and free market capitalism, suddenly it’s possible to set up a tech company, sell something worthwhile and in exchange not starve.
And that’s the frame for my original comment. I didn’t mean to imply that a fast moral feedback loop would involve a single person going on some meditation retreat that is somehow a clever feedback loop in disguise and then come back more moral or whatnot. I think it is possible that there is some innovation, moral or social or otherwise (e.g. a common understanding of common knowledge), that would enable the creation of fast moral and social feedback loops.
So the question, again: what are the necessary conditions for such a feedback loop? Are they present? What would it look like? How would you recognize it if it was happening right in front of you?
(EDIT: spelling)
You might have gone too far with speculation—your theory can be tested. If your model was true, I would expect a correlation between, say, the ability to learn ball sports and the ability to solve mathematical problems. It is not immediately obvious how to run such an experiment, though.
A Telepathic Exam about AI and Consequentialism
I immediately recognize the pattern that’s being playing out in this post and in the comments. I’ve seen it so many times, in so many forms.
Some people know the “game” and the “not-game”, because they learned the lesson the hard way. They nod along, because to them it’s obvious.
Some people only know the “game”. They think the argument is about “game” vs “game-but-with-some-quirks”, and object because those quirks don’t seem important.
Some people only know the “not-game”. They think the argument is about “not-game” vs “not-game-but-with-some-quirks”, and object because those quirks don’t seem important.
And these latter two groups find each other, and the “gamers” assume that everyone is a “gamer”, the “non-gamers” assume that everyone is a “non-gamer”, and they mostly agree in their objections to the original argument, even though in reality they are completely talking past each other. Worse, they don’t even know what the original argument is about.
Other. People. Are. Different.
Modeling them as mostly-you-but-with-a-few-quirks is going to lead you to wrong conclusions.
Lockdown incentivized politicians to establish positions on a lockdown, which has led to people having strong opinions about it. Even assuming no damage from further polarization, we have a roughly 50% chance of having an anti-lockdown government when the next pandemic hits, with a 10% chance of this new incentive being the deciding factor in not enacting a lockdown (or failing to implement it). Even if we assume that only 10% of the effects of this polarization is the result of the lockdown actually happening, with a 1% yearly chance of a pandemic more dangerous in expectation than the current one (~100M dead), we have ~1M QALYs lost, extrapolated worldwide over the next 10 years (while this effect is most pronounced).
Note: This is just a quick check to see that the effect is at least plausibly on an order of magnitude worth taking into consideration. I’m only somewhat confident that the effect isn’t in the opposite direction. I’m only commenting (as opposed to answering) because primarily I expect weak points in my general process of speculation pointed out, not because I believe this to be well-informed enough to be useful.
Thus I claim we don’t know whether people see dreams.
That’s a pretty bold claim just a few sentences after claiming to have aphantasia.
Some of my dreams have no visuals at all, just a vague awareness of the setting and plot points. Others are as vivid and detailed as waking experience (or even more, honestly), at least as far as vision is concerned. Dreams can fall anywhere on a spectrum between these extremes, and sometimes they can even be a mixture (e.g. a visual experience of the place and an awareness of characters in that place that don’t appear visually).
Yes, people do see dreams. I’m fairly certain I can tell the difference.
It seems pretty likely that moral and social progress are just inherently harder problems, given that you can’t [...] have fast feedback cycles from reality (like you do when trying to make scientific, technological and industrial progress).
We can’t? Have we tried? Have you tried? Is there some law of physics I’m missing? What would a real, genuine attempt to do just that even look like? Would you recognize it if it was done right in front of you?
These rituals are inefficient in cases where there is mutual trust between all participants. But sticking to formality is a great Schelling fence against those trying to gain an advantage by exploiting unwitting bureaucrats.
The above narratives seem to be extremely focused into a tiny part of narrative-space, and it’s actually a fairly good representation of what makes LessWrong a memetic tribe. I will try to give some examples of narratives that are… fundamentally different, from the outside view; or weird and stupid, from the inside view. (I’ll also try to do some translation between conceptual frameworks.) Some of these narratives you already know—just look around the political spectrum, and notice what narratives people live in. There are aslo some narratives I find better than useless:
Karma. Terrible parents will likely have children who can’t reach their full potential and can’t help the world, and who will themselves go on becoming terrible parents. Those who were abused by the powerful will go on abusing their power wherever and whenever they have any. Etc. Your role is to “neutralize the karma”, to break the part of the cycle that operates through you: don’t become a terrible parent yourself, don’t abuse your power, etc. even though you were on the recieving end.
The world is on the verge of collapse because the power of humanity through technology has risen faster than our wisdom to handle it. You have to seek wisdom, not more power.
The world is run by institutions that are run by unconscious people (i.e. people who aren’t fully aware of how their contribution as a cog to a complex machine affects the world). Most problems in the world are caused by the ignorant operation of these institutions. You have to elevate people’s consciousness to solve this problem.
Humans and humanity is evolving through stages of development (according to something like integral theory). Your role is to reach the higher stages of development in your life, and help your environment to do likewise.
History is just life unfolding. Your job isn’t to plan the whole process, just as the job of a single neuron isn’t to do the whole computation. The best thing you can do is just to live in alignment with your true self, and let life unfold as it has to, whatever the consequences (just as a neuron doing anything other than firing according to its “programming” is simply adding noise to the system).
Profit (Moloch) has overtaken culture (i.e. the way people’s minds are programmed). The purpose of profit (i.e. the utility function of Moloch that can be reconstructed from its actions) isn’t human well-being or survival of civilization, so the actions of people (which is a manifestation of the culture) won’t move the world toward these goals. Your role is to raise awareness, and to help reclaim culture from the hands of profit, and put a human in the driver’s seat again (i.e. realign the optimization process by which culture is generated so that the resulting culture is going to be aligned with human values).
Western civilization is at the end of its lifecycle. This civilization has to collapse, to make way for a new one that relates to this civilization in the same way the western civilization relates to the fallen Rome. Your role isn’t to prevent the collapse, but to start creating the first building blocks which will form the basis for the new civilization.
The world is on the brink of a context switch (i.e. the world will move to a formerly inaccessible region of phase space—or has already done so). Your models of the world are optimized to the current context, and therefore they are going to be useless in the new context (no training data in that region of the phase space). So you can’t comprehend the future by trying to think in terms of models, instead you have to reconnect with the process that generated those models. Your role is to be prepared for the context switch to try to mess things up as little as possible, though some of it is inevitable.
Reality (i.e. the “linear mapping” you use to project the world’s phase space to a lower dimensional conceptual space through your perception and sensemaking) is an illusion (i.e. has in its Kernel everything that actually matters). Your role is to realize that (and after that your role will be clear to you).
The world is too complex for any individual to understand. Your role is to be part of a collective sensemaking through openness and dialog that has the potential to collectively understand the world and provide actionable information. (In other words, there is no narrative simple enough for you to understand but complex enough to tackle the world’s challenges.)
The grand narrative you have to live your life by changes depending on your circumstances, just like it depends on where you are whether you have to turn left or right. Your role is to learn to understand and evaluate the utility of narratives, and to increase your capacity to do so.
This list is by no means comprehensive, but this is taking way too much time, so I’ll stop now, lest it should become a babble challenge.