It’s not just the legal system or software which is filled with loopholes (exploits), it’s actually everything. Life is like a game of chess, and the more intelligent an agent is, the more options it has to choose between each turn.
However, it isn’t a lack of intelligence which has prevented mass-exploitation so far, it’s actually a human thing clustering with taste/good faith/morality/cooperation/irrationality
Businessmen and everyone who considers it rational to optimize gains (this is the definition of a rational agent), have little of this.
Amazon pressures employees into working themselves to death because that’s what’s optimal for their profits. Even if it only lowered their profits by 0.1% to prevent this, they’d not do it. This is the very nature of optimization, which is about extracting as much value from things as possible.
As society is getting more “scientific” (rational, practical, whatever), it’s getting less moral, which means the tendency for cooperation decreases. Since the world has come to this, it’s ridiculous to think you can fix anything. The only means is power, and the most powerful player is usually not interested in making things better for less powerful players.
Human beings used to, quite intentionally, ignore the best choices available to them. They blinded themselves to them, throwing them into a category called “psychopathic behavior” and pruning the thought paths leading there entirely. This is how Moloch was kept at bay. And if this counter-force was religion, then yes, religious people were irrational. But this Chesterton’s Fence saved us from something worse.
Midwits feel so smart when they game the system and use strategies that normal people cannot see, but they’re slowly creating a world in which the most psychopathic players have an advantage. And as somebody who has a solid security mindset, and who is above midwit-level intelligence, I want to remind them that there’s 1000 ways to destroy them which they have zero defenses against. Harming me is quite easy as well, which is why I can only appeal to the good will of others.
This means that we now live in a world where you cannot become stronger. There’s too many dimensions to work with. Imagine you go to the gym for 10 years in a row, only to then get poisoned. Most of you will know about mirror bacteria. But I want to warn that there’s likely 1000 similar dangers just waiting to be discovered, in the same way that there’s 1000 zero-days in all common software and hardware waiting to be discovered. These attacks exist in different dimensions, and I think it’s impossible to protect oneself from them all simultaneously.
The phrase “This is why we can’t have nice things” as a response to people abusing something made available to them, describes the mechanism of a problem I think is becoming increasingly common. We won’t be able to have nice things anymore, as creating “nice things” is a move which you can only take if the move of the other players is not to immediately exploit the nice thing and ruin it for everyone. In other words, nice things require irrational behavior (for unexploited value being left alone).
“Good things” may still be worth it when the exploitative response is rare, but when the media wants a good thing gone, they exaggerate examples of the exploitative response in order to vaccinate society of the good thing (examples: Nuclear power discouraged by the Chernobyl incident, gun ownership discouraged by school shootings, Encryption discouraged by CSAM). Because people are biased towards reducing negatives over increasing positives, the absolute welfare of society falls as a result. I personally believe that the highest positive is limited by the highest tolerated negative. In either case, a good society requires vulnerabilities which are left unexploited despite players ability to exploit them. You cannot force good behavior because the mechanisms of force and the mechanisms of good behavior conflict. You cannot design your way around it because the theoretically optimal design of anything is neural and thus not biased towards good things. And the arms race between closing loop holes and discovering new ones seem to go nowhere at best, so I’d like a solution which is entirely different.
If we perceived more value in the sight of unextracted value than in extracting it, we’d be alright, but this requires having good taste and valuing more personal/subjective/human things, for the lack of better wording. This is certainly not what companies are about
A useful conceptual dimension in this kind of exploration is “legibility”. You can only optimize what you can measure. As human culture has gotten more focused on efficiency, it’s been forced to concentrate on measurable outcomes (because you can’t define “efficiency” using unmeasured values).
The protection of slack (inefficient space for un-measured values) is critical for my flourishing, at least. And it’s exactly counter to what most rationalists and EAs prefer to do: measure and optimize.
Slack comes from surplus. Many of the people I know who created illegible environments where people could explore and slow down, spent “winnings” from other domains to create those spaces. And to get those winnings, they usually had to be legible, to create, or to take from others (unfortunately). Everyone competes and builds and takes, then someone accumulates resources. Their team gets some slack, a pizza party, and a few trust fund kids. People trust each other, because they can and because they have enough. The circle of life.
Institutional measurement and optimization is one of those things which which is easy to critique but hard to suggest concrete improvements to. For example, what should EA do? Not measure, strive, or optimize? No, they must. They haven’t won. And if they’re going to measure, better to measure well. Measurement is the act of making something legible that was illegible before, to count something in the tally which you were blind to previously. The core of the complaint about legibility is that there are parts of us which aren’t legible but which we care about nonetheless. If only we could put their worth on the page too. The thing we all hate is when folks measure what they can, and assume the rest is irrelevant.
The classic way out of a competition or low trust environment is through, to create and win, and build a new environment. What that means for the world I suppose is that it’s important to fight hard for groups who will spend their winnings on something you like.
I feel very uneasy about all this, but it feels like the “god of everything else” analogue to OP’s moloch.
I mostly agree with both of you, but I’ve known about Moloch, slack and legibility for a while now, and wrote my comment in spite of that, so to speak.
The problem with building your own environment is that you’re still inside the old environment. It’s difficult to escape from society and live in the wild because there’s so little wild anymore. You can’t buy your own island and make your own rules on there. Anything you can create, really, is built with bricks manufactured by other people, and they make sure that their bricks have microphones and backdoors, and that you’re only really paying for a subscription to the bricks.
Life likes flexibility, but I think we’re building brittle systems inside which little movement is tolerated.
Measurement is the act of making something legible that was illegible before, to count something in the tally which you were blind to previously
What if they’re not discovering something old, but rather creating something new? Lets say somebody scammed me, but I was happy about the trade. It would only be the moment somebody else tells me “you got scammed” that I’d feel hurt about it. And even if I didn’t get scammed, I’d still suffer if somebody told me I had been. If it’s not things which harm us, but the idea of them, which suggests that interpreting something as bad makes it so. So actions that you’re doing now, which are currently completely harmless, might be interpreted as harmful in the future by one of the systems you’re embedded within.
And what if you love your son with a magnitude of 514 and your daughter with a magnitude of 410? Would you even want to know?
Finally, Revenge porn, doxing, and looking into peoples bedroom windows are all illegal, but they’re just the transfering of true information. This suggests that unlimited collection of information is not a strictly good thing, and this is still not a very common realization. By the way, the arguments I use here are very different from my previous comment, so reading one of them will not help much in understanding the other
It’s not just the legal system or software which is filled with loopholes (exploits), it’s actually everything. Life is like a game of chess, and the more intelligent an agent is, the more options it has to choose between each turn.
However, it isn’t a lack of intelligence which has prevented mass-exploitation so far, it’s actually a human thing clustering with taste/good faith/morality/cooperation/irrationality
Businessmen and everyone who considers it rational to optimize gains (this is the definition of a rational agent), have little of this.
Amazon pressures employees into working themselves to death because that’s what’s optimal for their profits. Even if it only lowered their profits by 0.1% to prevent this, they’d not do it. This is the very nature of optimization, which is about extracting as much value from things as possible.
As society is getting more “scientific” (rational, practical, whatever), it’s getting less moral, which means the tendency for cooperation decreases. Since the world has come to this, it’s ridiculous to think you can fix anything. The only means is power, and the most powerful player is usually not interested in making things better for less powerful players.
Human beings used to, quite intentionally, ignore the best choices available to them. They blinded themselves to them, throwing them into a category called “psychopathic behavior” and pruning the thought paths leading there entirely. This is how Moloch was kept at bay. And if this counter-force was religion, then yes, religious people were irrational. But this Chesterton’s Fence saved us from something worse.
Midwits feel so smart when they game the system and use strategies that normal people cannot see, but they’re slowly creating a world in which the most psychopathic players have an advantage. And as somebody who has a solid security mindset, and who is above midwit-level intelligence, I want to remind them that there’s 1000 ways to destroy them which they have zero defenses against. Harming me is quite easy as well, which is why I can only appeal to the good will of others.
This means that we now live in a world where you cannot become stronger. There’s too many dimensions to work with. Imagine you go to the gym for 10 years in a row, only to then get poisoned. Most of you will know about mirror bacteria. But I want to warn that there’s likely 1000 similar dangers just waiting to be discovered, in the same way that there’s 1000 zero-days in all common software and hardware waiting to be discovered. These attacks exist in different dimensions, and I think it’s impossible to protect oneself from them all simultaneously.
The phrase “This is why we can’t have nice things” as a response to people abusing something made available to them, describes the mechanism of a problem I think is becoming increasingly common. We won’t be able to have nice things anymore, as creating “nice things” is a move which you can only take if the move of the other players is not to immediately exploit the nice thing and ruin it for everyone. In other words, nice things require irrational behavior (for unexploited value being left alone).
“Good things” may still be worth it when the exploitative response is rare, but when the media wants a good thing gone, they exaggerate examples of the exploitative response in order to vaccinate society of the good thing (examples: Nuclear power discouraged by the Chernobyl incident, gun ownership discouraged by school shootings, Encryption discouraged by CSAM). Because people are biased towards reducing negatives over increasing positives, the absolute welfare of society falls as a result. I personally believe that the highest positive is limited by the highest tolerated negative. In either case, a good society requires vulnerabilities which are left unexploited despite players ability to exploit them. You cannot force good behavior because the mechanisms of force and the mechanisms of good behavior conflict. You cannot design your way around it because the theoretically optimal design of anything is neural and thus not biased towards good things. And the arms race between closing loop holes and discovering new ones seem to go nowhere at best, so I’d like a solution which is entirely different.
If we perceived more value in the sight of unextracted value than in extracting it, we’d be alright, but this requires having good taste and valuing more personal/subjective/human things, for the lack of better wording. This is certainly not what companies are about
A useful conceptual dimension in this kind of exploration is “legibility”. You can only optimize what you can measure. As human culture has gotten more focused on efficiency, it’s been forced to concentrate on measurable outcomes (because you can’t define “efficiency” using unmeasured values).
The protection of slack (inefficient space for un-measured values) is critical for my flourishing, at least. And it’s exactly counter to what most rationalists and EAs prefer to do: measure and optimize.
Slack comes from surplus. Many of the people I know who created illegible environments where people could explore and slow down, spent “winnings” from other domains to create those spaces. And to get those winnings, they usually had to be legible, to create, or to take from others (unfortunately). Everyone competes and builds and takes, then someone accumulates resources. Their team gets some slack, a pizza party, and a few trust fund kids. People trust each other, because they can and because they have enough. The circle of life.
Institutional measurement and optimization is one of those things which which is easy to critique but hard to suggest concrete improvements to. For example, what should EA do? Not measure, strive, or optimize? No, they must. They haven’t won. And if they’re going to measure, better to measure well. Measurement is the act of making something legible that was illegible before, to count something in the tally which you were blind to previously. The core of the complaint about legibility is that there are parts of us which aren’t legible but which we care about nonetheless. If only we could put their worth on the page too. The thing we all hate is when folks measure what they can, and assume the rest is irrelevant.
The classic way out of a competition or low trust environment is through, to create and win, and build a new environment. What that means for the world I suppose is that it’s important to fight hard for groups who will spend their winnings on something you like.
I feel very uneasy about all this, but it feels like the “god of everything else” analogue to OP’s moloch.
I mostly agree with both of you, but I’ve known about Moloch, slack and legibility for a while now, and wrote my comment in spite of that, so to speak.
The problem with building your own environment is that you’re still inside the old environment. It’s difficult to escape from society and live in the wild because there’s so little wild anymore. You can’t buy your own island and make your own rules on there. Anything you can create, really, is built with bricks manufactured by other people, and they make sure that their bricks have microphones and backdoors, and that you’re only really paying for a subscription to the bricks.
Life likes flexibility, but I think we’re building brittle systems inside which little movement is tolerated.
What if they’re not discovering something old, but rather creating something new? Lets say somebody scammed me, but I was happy about the trade. It would only be the moment somebody else tells me “you got scammed” that I’d feel hurt about it. And even if I didn’t get scammed, I’d still suffer if somebody told me I had been. If it’s not things which harm us, but the idea of them, which suggests that interpreting something as bad makes it so. So actions that you’re doing now, which are currently completely harmless, might be interpreted as harmful in the future by one of the systems you’re embedded within.
And what if you love your son with a magnitude of 514 and your daughter with a magnitude of 410? Would you even want to know?
Finally, Revenge porn, doxing, and looking into peoples bedroom windows are all illegal, but they’re just the transfering of true information. This suggests that unlimited collection of information is not a strictly good thing, and this is still not a very common realization. By the way, the arguments I use here are very different from my previous comment, so reading one of them will not help much in understanding the other