I didn’t think “we” were operating on that model.
I think it’s actually quite hard to have everyone in an organization trust everyone else in an organization, or to only hire people who would be trusted by everyone in the organization. So you might want to have some sort of tiered system, where (perhaps) the researchers all trust each other, but only trust the engineers they work with, and don’t trust any of the ops staff, and this lets you only need one researcher to trust an engineer to hire them.
[On net I think the balance is probably still in favor of “internal transparency, gated primarily by time and interests instead of security clearance”, but it’s less obvious than it originally seems.]
But he does not see, or will not admit, that a return to ‘free’ competition means for the great mass of people a tyranny probably worse, because more irresponsible, than that of the State. The trouble with competitions is that somebody wins them.
This bit feels like it’s missing something, which I don’t know that Hayek ever stated fully, which I think Rand stated a little more clearly. I think there are two main parts of it.
First, there’s a way of organizing that humans have had for a long time, with toiling peasants and idle rich. There’s a new way of organizing, that roughly dates to the Industrial Revolution (but not quite), where there are toiling peasants, idle rich, and organizing rich.
That is, sometimes people get rich by finding a bunch of peasants and somehow managing to get a percentage of their output (see The Emperor for a view of this), and this relationship is roughly parasitic—the main value the nobility provide is keeping other nobility from taxing the peasants instead.
There’s a different thing that people can do, tho, which is doing useful thinking or providing a useful service, and that makes for the organizing rich. The frame that sees Sam Walton as just another duke who someone managed to entitle himself is missing the ways in which a world with Walmart is richer than a world without Walmart. This relationship really does seem more mutualistic.
This sort of generates idle rich. I don’t think many of the Waltons since Sam have done all that much that’s interesting, but they’re much less obviously parasitic. (If they just lent out money and lived off the interest, that’s much more of a voluntary exchange than the medieval baron taxing his parents.)
Second, I think the “more irresponsible” thing is fake, for basically the reason discussed in the earlier paragraphs, and expanded on in the public choice literature. Governments could act in ways that put the public interest first, but empirically they act in a way that benefits the governing cartel (in a way that is worse for socialism than alternatives, because of the ‘tyrannical minority’ bit). Governments could set food subsidies in ways that make people who eat the food healthier (which is what you might expect voters would want), but at least in the US they set it in ways that make people who eat the food less healthy, but which benefit politically connected farmers.
Like, at the beginning of the pandemic, Gates was talking about building more vaccine factories at the same time that the federal government was punishing people for doing unlicensed tests that helped alert people to community spread inside the US; that makes it really difficult for me to take seriously the bit where monopolistic billionaires are ‘more irresponsible.’
[Of course, I’m writing in America in the 2020s, where most of the billionaires on my mind are tech billionaires; Orwell was writing in England in the 1940s, where many of the equivalent rich people would have been clearly in the ‘parasitic nobility’ camp, or directly descendant from them. But I think the thing about industrial fortunes being good, and often reinvested in progress, still could have been obvious then.]
I wanted to note that I think this comment both a) raises a good point (should Leverage pay restitution to people that were hurt by it? Why and how much?) and b) does so in a way that I think is hostile and assumes way more buy-in than it has (or would need to get support for its proposal).
First, I think most observers are still in “figuring out what’s happened” mode. Was what happened with Zoe unusually bad or typical, predictable or a surprise? I think it makes sense to hear more stories before jumping to judgment, because the underlying issue isn’t that urgent and the more context, the wiser a decision we can make.
Second, I think a series of leading questions asked to specific people in public looks more like norm enforcement than it does like curious information-gathering, and I think the natural response is suspicion and defensiveness. [I think we should go past the defensiveness and steelman.]
Third, I do think that it makes sense for people to make things right with money when possible; I think that this should be proportional to damages done and expectations of care, rather than just ‘who has the money.’ Suppose, pulling these numbers out of a hat, the total damage done to Leverage employees (as estimated by them) was $1M and the total value of Geoff’s tokens are $10M; the presumption that the tokens should all go to the victims (i.e. that the value of his tokens is equal to the amount of damage done) seems about as detached from reality to me as the assumption that the correct amount of restitution is 0. On a related note, some large amount of the Leverage experience appears to have been self-experimentation; I think the amount we should expect Geoff to be responsible should take into account how much responsibility the participants thought they were taking for themselves (while not just assuming that they were making an informed call and their initial estimate should be our final one).
An interesting side note is that Hayek, one of the earliest people to diagnose what was wrong with socialism (according to me, at least), wrote The Road to Serfdom while Orwell was still alive (before he wrote his major books, in fact), and he reviewed it (for what was, at the time, a non-partisan London newspaper).
[I had historically respected Orwell a lot, then read The Road to Serfdom, found it compelling, and thought “oh, I wish that Orwell had read this”, and then discovered that he had, and I didn’t much like what he made of it.]
Sapiens was previously discussed on LW here back in 2015.
Note that Musk parted ways with OpenAI back in 2018, in part because of a conflict of interest (between Tesla and OpenAI).
Civilization (the video game series) is 30 years old, and they posted a trailer to Youtube.
I found it… sort of disgustingly delusional? Like, I like the Civilization series a lot; I’ve put thousands of hours into the games. But:
Your deeds are legendary, and across the world, throughout the internet, everyone shall hear of you. … You are more than a turn taker; you shape the world.
I think there’s something good about strategy games in helping you see through different lenses, develop mental flexibility and systems thinking, and learn to see like a state, and so on. But this is not that! This is saying that, by playing a game of Civilization, you actually become a world leader (or, at least, a famous gamer)? What a pathetic lie.
speaks about how rationality could help us all build a better democracy—an ironic defense, given what he had just told us about how the mythological mindset can interact with our politics.
Tho this feels to me like it needs to grapple with The Myth of the Rational Voter. That is, Caplan claims voters are ‘rationally irrational’, where they correctly determine that voting calls for the mythological mindset instead of the reality mindset.
In order for people to vote in reality mindset, something needs to be structurally different, because if you just get people to drop the mythological mindset, they’ll probably rationally decide not to vote (because the expected benefit of their vote, under most reality-based analyses, will be less than the cost of voting).
[I am optimistic about some ways to make voting more conducive to reality mindset, but I think it doesn’t look very much like “more informed voters”. Also, I think most “well, educate people more” approaches look like “replace mythology A with mythology B”, which I’m in favor of!]
Compare that to the sequence on quantum mechanics here which forcefully argued for the deterministic many worlds interpretation.
Ok, but why does this matter?
IMO, the point of the QM sequence is “no, really, nowhere is safe; the call is coming from inside the house.” There’s a big difference between ‘rationalists’ who see irrationality as something to fight ‘out there’, and ‘rationalists’ who see irrationality as something to fight ‘in yourself’. Seeing irrationality in a field considered by many to be the peak of human intellect is a sobering observation.
But… you have to go through the details, and you have to be right. I don’t know that I would put it in my intro rationality book (for example, it’s not in Thinking and Deciding, which was my old go-to recommendation for a textbook on rationality).
But I have never seen an article pulled completely before.
It happened before, but it’s quite rare. Normally when I’ve done it, I’ve left a note in an Open Thread, such as this case where I moved to drafts a post that was talking about an ongoing legal case (now concluded). I think that’s the last one I did, and it was four years ago? But there are other mods as well.
Overall Kraken damage is substantially higher on a 4-gun ship than a 2-gun ship.
This seems reversed to me.
Also, I was under the impression that cryonics was a business with significant returns to scale—two facilities storing 100 bodies each is much more expensive than one facility storing 200 bodies, which makes ‘market share’ more important than it normally is.
Previously discussed on LW here.
There’s a paired optimization problem, where you assign everyone to a room, and the constraint that this assignment be ‘envy-free’; that is, no one looks at someone else’s assignment/rent combo and says “I’d rather have that than my setup!”. There was a calculator that I can’t easily find now which tried to find the centroid of the envy-free region.
There are other approaches that work differently; this one, for example, tries to split surplus evenly between the participants, and shows the comparison to other options.
Do you “manage the news” by refusing to read the morning’s newspaper, or by scribbling over the front page “Favored Candidate Wins Decisively!”? No: if you’re rational, your credence in the loss is still 70%.
I feel like the “No; if you’re rational” bit is missing some of the intuition against EDT. Physical humans do refuse to read the morning’s newspaper, or delay opening letters, or similar things, I think because of something EDT-ish ‘close to the wire’. (I think this is what’s up with ugh fields.)
I think there’s something here—conservation of expected evidence and related—that means that a sophisticated EDT won’t fall prey to those traps. But this feels sort of like the defense whereby a sophisticated EDT doesn’t fall prey to typical counterexamples because if you’re doing the expectation correctly, you’re taking into account causation, at which point we’re not really talking about EDT anymore. I do think it’s sensible to include proper probabilistic reasoning in EDT, but sometimes feels off about hiding this detail behind the word “rational.”
One frame I have for ‘maximizing altruism’ is that it’s something like a liquid: it’s responsive to its surroundings, taking on their shape, flowing to the lowest point available. It rapidly conforms to new surroundings if there are changes; turn a bottle on its side and the liquid inside will rapidly resettle into the new best configuration.
This has both upsides and downsides: the flexibility and ability to do rapid shifts mean that as new concerns become the most prominent, they can be rapidly addressed. The near-continuous nature of liquids means that as you get more and more maximizing altruist capacity, you can smoothly increase the ‘shoreline’.
Many other approaches seem solid instead of liquid, in a way that promotes robustness and specialization (while being less flexible and responsive). If the only important resources are fungible commodities, then the liquid model seems optimal; if it turns out that the skills and resources you need for tackling one challenge are different than the skills and resources needed for tackling another, or if switching costs dominate the relative differences between projects. Reality has a surprising amount of detail, and it takes time and effort to build up the ability to handle that detail effectively.
I think there’s something important here for the broader EA/rationalist sphere, tho I haven’t crystallized it well yet. It’s something like—the ‘maximizing altruism’ thing, which I think of as being the heart of EA, is important but also a ‘sometimes food’ in some ways; it is pretty good for thinking about how to allocate money (with some caveats) but is much less good for thinking about how to allocate human effort. It makes sense for generalists, but actually that’s not what most people are or should be. This isn’t to say we should abandon maximizing altruism, or all of its precursors, but… somehow build a thing that both makes good use of that, and good use of less redirectable resources.
[Note: I use Copilot and like it. The ‘aha’ moment for me was when I needed to calculate the intersection of two lines, a thing that I would normally just copy/paste from Stack Overflow, and instead Copilot wrote the function for me. Of course I then wrote tests and it passed the tests, which seemed like an altogether better workflow.]
Language models are good enough at generating code to make the very engineers building such models slightly more productive
How much of this is ‘quality of code’ vs. ‘quality of data’? I would naively expect that the sort of algorithmic improvements generated from OpenAI engineers using Copilot/Codex/etc. are relatively low-impact compared to the sort of benefits you get from adding your company’s codebase to the corpus (or whatever is actually the appropriate version of that). I’m somewhat pessimistic about the benefits of adding Copilot-generated code to the corpus as a method of improving Copilot.
Thanks for sharing negative results!
If I’m understanding you correctly, the structure looks something like this:
We have a toy environment where human preferences are both exactly specified and consequential.
We want to learn how hard it is to discover the human preference function, and whether it is ‘learned by default’ in an RL agent that’s operating in the world and just paying attention to consequences.
One possible way to check whether it’s ‘learned by default’ is to compare the performance of a predictor trained just on environmental data, a predictor trained just on the RL agent’s internal state, and a predictor extracted from the RL agent.
The relative performance of those predictors should give you a sense of whether the environment or the agent’s internal state give you a clearer signal of the human’s preferences.
It seems to me like there should be some environments where the human preference function is ‘too easy’ to learn on environmental data (naively, the “too many apples” case should qualify?) and cases where it’s ‘too hard’ (like ‘judge how sublime this haiku is’, where the RL agent will also probably be confused), and then there’s some goldilocks zone where the environmental predictor struggles to capture the nuance and the RL agent has managed to capture the nuance (and so the human preferences can be easily exported from the RL agent).
Does this frame line up with yours? If so, what are the features of the environments that you investigated that made you think they were in the goldilocks zone? (Or what other features would you look for in other environments if you had to continue this research?)
IMO there’s a big difference between “obviously material progress is good” and “obviously some progress is good”—it could be that after a careful consideration of the evidence, it turns out that the thing we need to do is focus on spiritual progress and all become monks (or w/e) and then progress can be measured in how rapidly that transition happens.
[Like, in one era the accumulation of slaves would have been a sign of progress, and now we view it as a sign of regress.]
There’s a second point that you might be making, that it’s weird to have a ‘theory of progress’ if your forecasts show the world getting worse, even if we do our best. (For example, suppose there was a massive volcanic eruption and so we knew volcanic winter was coming.) But I think even then it’s important to figure out what ways we can improve in and make those changes, even if the background is decay instead of progress.