What made the ancient Greeks so generative? It seems they founded the Western philosophical and scientific traditions, but what led to their innovativeness?
Hypothesis: they weren’t any more innovative than anyone else; their language and culture just got transmitted over a much larger area. In the wake of Alexander’s conquests, Greek became the working language of government bureaucrats across most of the Middle East. It quickly turned into a status symbol and a credential, sort of like a college degree today. Speaking, reading and writing Greek automatically put you into the educated social class. A working knowledge of Greek culture followed along naturally with the language.
Without Alexander’s conquests, I very much doubt that Aristotle’s name would be at all known today.
A general rule that I try to follow is “never write something which someone else has already written better”. Rather than give a numerical scale, I’ll list a few distinct ways that pieces can satisfy this rule. I’ll give examples of each from my own writing, with the caveat that they are not necessarily very good pieces—just examples of my own reasoning regarding their novelty.
Novel theory/observation/interpretation (or at least novel as far as I know). Ex.: Computational limits of empire
Carving up known ideas differently, e.g. by pointing to a particular sub-idea or creating a handle. Ex.: The broken chain problem
Taking a well-known idea from one area, and applying it in another area. Ex.: Competitive markets as distributed backprop
Analyzing something which has been analyzed many times before, but usually incorrectly, so it’s hard to sift signal from noise. In this case, the novelty/value is in making the analysis visibly correct. Ex.: Accounting for college costs
Translating some piece of metic knowledge into a more transmissible model/technique. Ex.: How to find unknown unknowns
Accessible explanations of technical ideas. Ex.: Queuing theory without math
Arguing that some known idea/problem is more important than people seem to have noticed. Ex.: Coordination economy
Note that these are ways that a piece of writing can be novel, not guarantees that nobody has ever written the same thing before.
Side note: if using a numerical scale, I worry about confusing novelty with importance—the example scale in the OP seems to mix the two. Perhaps a better approach would be to give handles for several different ways things can be novel, and then use those as tags?
Perhaps the difference is what you’re imagining as “under the hood”. Nobody wants to think about the axiom of choice when solving a differential equation.
Possible point of confusion: equilibrium does not imply static equilibrium.
If a firm can’t find someone to maintain their COBOL accounting software, and decides to scrap the old mainframe and have someone write a new piece of software with similar functionality but on modern infra, then that’s functionally the same as replacement due to depreciation.
If that sort of thing happens regularly, then we have a dynamic equilibrium. As an analogy, consider the human body: all of our red blood cells are replaced every couple of months, yet the total number of red blood cells is in equilibrium. Replacement balances removal. Most cell types in the human body are regularly replaced this way, at varying timescales.
That’s the sort of equilibrium we’re talking about here. It’s not that the same software sticks around needing maintenance forever; it’s that software is constantly repaired or replaced, but mostly provides the same functionality.
Yeah, in retrospect I should have been more clear about that. Thanks for drawing attention to it, other people probably interpreted it the same way you did.
Sounds like you’re mostly talking about ops, which is a different beast.
An example from my previous job, to illustrate the sort of things I’m talking about: we had a mortgage app, so we called a credit report api, an api to get house data from an address, and an api to pull current pricing from the mortgage securities market (there were others, but those three were the most important). Within a six-month span, the first two apis made various small breaking changes to the format returned, and the third was shut down altogether and had to be switched to a new service.
(We also had the whole backend setup on Kubernetes, and breaking changes there were pretty rare. But as long as the infrastructure is working, it’s tangential to most engineers’ day-to-day work; the bulk of the code is not infrastructure/process related. Though I suppose we did have a slew of new bugs every time anything in the stack “upgraded” to a new version.)
Good points, this gets more into the details of the relevant models. The short answer is that capital equilibrates on a faster timescale than growth happens.
About a year ago I did some research into where most capital investments in the US end up—i.e. what the major capital sinks are. The major items are:
infrastructure: power grid, roads, railroads, data transmission, pipelines, etc.
buildings (both residential and commercial)
Most of the things on that list need constant repair/replacement, and aren’t expanding much over time. The power grid, roads and other infrastructure (excluding data transmission) currently grow at a similar rate to the population, whereas they need repair/replacement at a much faster rate—so most of the invested capital goes to repair/replacement. Same for oil wells: shale deposits (which absorbed massive capital investments over the past decade) see well production drop off sharply after about two years. After that, they get replaced by new wells nearby. Vehicles follow a similar story to infrastructure: total number of vehicles grows at a similar rate to the population, but they wear out much faster than a human lifetime, so most vehicle purchases are replacements of old vehicles.
Now, this doesn’t mean that people “prioritize maintenance above new stuff”; replacement of an old capital asset serves the same economic role as repairing the old one. But it does mean that capital mostly goes to repair/replace rather than growth.
Since capital equilibrates on a faster timescale than growth, growth must be driven by other factors—notably innovation and population growth. In the context of a software company, population growth (i.e. growing engineer headcount) is the big one. Few companies can constantly add new features without either adding new engineers or abandoning old products/features. (To the extent that companies abandon old products/features in order to develop new ones, that would be economic innovation, at least if there’s net gain.)
Yeah, I used to have conversations like this all the time with my boss. Practically anything in an economics textbook can be applied to management in general, and software engineering in particular, with a little thought. Price theory and coordination games were most relevant to my previous job (at a mortgage-tech startup).
Yeah, sorry, I should have been more clear there. The mechanisms which trade off immune strength against other things are the underlying cause. Testosterone levels in males are a good example—higher testosterone increases attractiveness, physical strength, and spatial-visual reasoning, but it’s an immune suppressor.
We cannot “prove” that something is physically impossible, only that it is impossible under some model of physics. Normally that distinction would be entirely irrelevant, but when dealing with a superintelligent AI, it’s quite likely to understand the physics better than we do. For all we know, it may turn out that Alcubierre drives are possible, and if so then the AI could definitely break out that way and would have an incentive to do so.
I agree that the AI is not really boxed here; it’s the “myopia” that makes the difference. But one of two things should generally be true:
The AI doesn’t want to get out of the box, in which case the box doesn’t need to be secure in the first place.
The AI cannot get out of the box, in which case the AI doesn’t need to be safe (but also won’t be very useful).
This case seems like the former, so long as hacking the human is easier than getting out of the box. But that means we don’t need to make the box perfect anyway.
Can you expand a bit on why a commitment to give a high reward won’t save us? Is it a matter of the AI seeking more certainty, or is there some other issue?
This is pretty thorough from an engineering standpoint. I still would not at all trust such a box to hold a superintelligent AI. The unknown unknowns remain too big a factor. Just because an unknown information channel does not have any humans listening to it directly, does not mean that it can’t be used to push around matter, or couple to some other channel, or otherwise leave an opening.
Also, the human and their clothes are definitely not the only potentially-useful tools in the box: there’s also the computer, the computer’s power supply, whatever’s cooling the room, whatever’s keeping the air fresh, whatever’s used for input, and whatever’s used for output. If any of those things were outside the room, then they would offer a side channel for information to escape. They can be buried in concrete, but they’ll still need to be within the vacuums and whatnot.
In general, trying to outsmart a superintelligent AI is a Bad Idea, and building an AI-proof box definitely falls in that category. If an AI is not safe with arbitrarily large amounts of power, then it is not safe, full stop.
So I think this proposal is a specific instance of a general principle: wireheading acts as a safety valve. As long as the AI is prone to wireheading, and you leave it some path to wireheading, the AI will never bother with anything more difficult than that path. In this particular case, it will never do anything harder than hacking its human operator, and that’s where most of the safety benefit comes from.
This kind of setup creates a direct trade-off between safety and capability: the harder it is to hack the human operator, the more likely that some other escape channel is easier—manipulating electrical circuitry to push atoms around, or some weird thing like that. The AI will become an existential threat if you send in an operator who really doesn’t want to give it 100% reward, and make that operator hard to hack—then the AI has a strong incentive to gain whatever power it needs. (I definitely do not trust a door with a kill switch to keep a superintelligent AI in a box.)
My intuition says that nothing else in the proposal actually matters for safety, except the wireheading safety valve aspect. In particular, I think just giving the AI direct exploration abilities won’t mess up safety, as long as the wireheading path is present and there’s not “99.99% certainty is better than 99.98%“-type failure mode.
The reward belongs to a finite set of rationals between 0 and 1.
Once the AI becomes sufficiently powerful, it is definitely going to hack the operator any time it expects the operator to give a reward less than 1. So the operator’s input is really binary, at least once the AI has learned an accurate model. Given that, why allow non-binary rewards at all? Is it just supposed to provide faster learning early on?
Along similar lines: once the AI has learned an accurate model, why would we expect it to ever provide anything useful at all, rather than just hacking the operator all day? Do we think that hacking the human is likely to be harder than obtaining perfect rewards every time without hacking the human? Seems like that would depend very heavily on the problem at hand, and on the operator’s feedback strategy.
To put it differently: this setup will not provide a solution to any problem which is more difficult than hacking the human operator.
Two minor comments:
It would be nice for other people to be able to throw more into the bounty pool somehow.
A shorter-term, smaller-scale thing-to-try on the bounty front might be to make karma transferable, and let people create karma bounties. That would avoid having to deal with money.
Totally on board with that. The important point is eliminating risk-management overhead by (usually) only having to reward someone who contributes value, in hindsight.
I posted a handful of example questions I have been/would be interested in on Raemon’s bounty question. I think these examples address several of the challenges in section 4:
All of them are questions which I’d expect many people on lesswrong to be independently interested in, and which would make great blog-post-material. So the bounties wouldn’t be the sole incentive; even those who don’t get the bounty are likely to derive some value from the exercise.
There’s not really a trust/credentials problem; it would be easy for me to tell whether a given response had been competently executed. If I needed to hire someone in advance of seeing their answer, then there would be a trust problem, but the best-answer-gets-paid format mostly solves that. Even if there are zero competent responses, I’ve bought useful information: I’ve learned that the question is tougher than I thought. Also, they’re all the sort of project that I expect a smart, generally-educated non-expert to be able to execute.
They are all motivated by deeper/vaguer questions, but answers to the questions as stated have enough value to justify themselves. Directly answering the deeper questions would not be the objective of any of them.
They’re all sufficiently clean-cut that I wouldn’t expect much feedback to be necessary mid-effort.
I see that second point as the biggest advantage of bounties over a marketplace: just paying for the best answer means I don’t need to go to the effort of finding someone who’s competent and motivated and so forth. I don’t need to babysit someone while they work on the problem, to make sure we’re always on the same page. I don’t need to establish careful criteria for what counts as “finished”. I can just throw out my question, declare a bounty, and move on. That’s a much lower-effort investment on the asker’s side than a marketplace.
In short, with a bounty system, competition between answerers solves most of the trust problems which would otherwise require lots of pre-screening and detailed contract specifications.
Bounties will also likely need to be higher to compensate for answer-side risk, but that’s a very worthwhile tradeoff for those of us who have some money and don’t want to deal with hiring and contracts and other forms of baby-sitting.
I would like more concrete examples of nontrivial questions people might be interested in. Too much of this conversation is too abstract, and I worry people are imagining different things.
Toward that end, here are a few research projects I’ve either taken on or considered, which I would have been happy to outsource and which seem like a good fit for the format:
Go through the data on spending by US colleges. Look at how much is actually charged per student (including a comparison of sticker price to actual tuition), how much is spent per student, and where all the money is spent. Graph how these have changed over time, to figure out exactly which expenditures account for the rapid growth of college cost. Where is all the extra money going? (I’ve done this one; results here.)
Go through the data on aggregate financial assets held, and on real capital assets held by private citizens/public companies/the state (i.e. patents, equipment, property, buildings, etc) to find out where money invested ultimately ends up. What are the main capital sinks in the US economy? Where do marginal capital investments go? (I’ve also done this one, but haven’t gotten around to writing it up.)
Go through the genes of JCVI’s minimal cell, and write up an accessible explanation of the (known) functionality of all of its genes (grouping them into pathways/systems as needed). The idea is to give someone with minimal bio background a comprehensive knowledge of everything needed for bare-minimum life. Some of this will have to be speculative, since not all gene functions are known, but a closed list of known-unknowns sure beats unknown-unknowns.
Something like Laura Deming’s longevity FAQ, but focused on the macro rather than micro side of what’s known—i.e. (macroscopic) physiology of vascular calcification and heart disease, alzheimers, cancer, and maybe a bit on statistical models of old-age survival rates. In general, there seems to be lots of research on the micro side, lots known on the macro side, but few-if-any well-understood mechanistic links from tone to the other; so understanding both sides in depth is likely to have value.
An accessible explanation of Cox’ Theorem, especially what each piece means. The tough part: include a few examples in which a non-obvious interpretation of a system as a probabilistic model is directly derived via Cox’ Theorem. I have tried to write this at least four separate times, and the examples part in particular seems like a great exercise for people interested in embedded agency.
Reading this, I figured you were talking about local-descent-type optimization algorithms, i.e. gradient descent and variants.
From that perspective, there’s two really important pieces missing from these analogies:
The mountaineers can presumably backtrack, at least as long as the water remains low enough
The mountaineers can presumably communicate
With backtracking, even a lone mountaineer can do better sometimes (and never any worse) by continuing to explore after reaching the top of a hill—as long as he keeps an eye on the water, and makes sure he has time to get back. In an algorithmic context, this just means keeping track of the best point seen, while continuing to explore.
With backtracking and communication, the mountaineers can each go explore independently, then all come back and compare notes (again keeping track of water etc), all go to the highest point found, and maybe even repeat that process. In an algorithmic context, this just means spinning off some extra threads, then taking the best result found by any of them.
In an evolutionary context, those pieces are probably not so relevant.