(...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties.
- Saharon Shelah
As a true-born Dutchman I endorse Crocker’s rules.
For my most of my writing see my short-forms (new shortform, old shortform)
Twitter: @FellowHominid
Personal website: https://sites.google.com/view/afdago/home
Novel Science is Inherently Illegible
Legibility, transparency, and open science are generally considered positive attributes, while opacity, elitism, and obscurantism are viewed as negative. However, increased legibility in science is not always beneficial and can often be detrimental.
Scientific management, with some exceptions, likely underperforms compared to simpler heuristics such as giving money to smart people or implementing grant lotteries. Scientific legibility suffers from the classic “Seeing like a State” problems. It constrains endeavors to the least informed stakeholder, hinders exploration, inevitably biases research to be simple and myopic, and exposes researchers to constant political tug-of-war between different interest groups poisoning objectivity.
I think the above would be considered relatively uncontroversial in EA circles. But I posit there is something deeper going on:
Novel research is inherently illegible. If it were legible, someone else would have already pursued it. As science advances her concepts become increasingly counterintuitive and further from common sense. Most of the legible low-hanging fruit has already been picked, and novel research requires venturing higher into the tree, pursuing illegible paths with indirect and hard-to-foresee impacts.
Thank you, Evan, for living the Virture of Scholarship. Your work is appreciated.
I’m a little skeptical of your contention that all these properties are more-or-less independent. Rather there is a strong feeling that all/most of these properties are downstream of a core of agentic behaviour that is inherent to the notion of true general intelligence. I view the fact that LLMs are not agentic as further evidence that it’s a conceptual error to classify them as true general intelligences, not as evidence that ai risk is low. It’s a bit like if in the 1800s somebody says flying machines will be dominant weapons of war in the future and get rebutted by ‘hot gas balloons are only used for reconnaissance in war, they aren’t very lethal. Flying machines won’t be a decisive military technology ’
I don’t know Nate’s views exactly but I would imagine he would hold a similar view (do correct me if I’m wrong ). In any case, I imagine you are quite familiar with the my position here.
I’d be curious to hear more about where you’re coming from.
Encrypted Batteries
(I thank Dmitry Vaintrob for the idea of encrypted batteries. Thanks to Adam Scholl for the alignment angle. Thanks to the Computational Mechanics at the receent compMech conference. )
There are no Atoms in the Void just Bits in the Description. Given the right string a Maxwell Demon transducer can extract energy from a heatbath.
Imagine a pseudorandom heatbath + nano-Demon. It looks like a heatbath from the outside but secretly there is a private key string that when fed to the nano-Demon allows it to extra lots of energy from the heatbath.
P.S. Beyond the current ken of humanity lies a generalized concept of free energy that describes the generic potential ability or power of an agent to achieve goals. Money, the golden calf of Baal is one of its many avatars. Could there be ways to encrypt generalized free energy batteries to constraint the user to only see this power for good? It would be like money that could be only spent on good things.
This seems noncrazy on reflection.
10 million dollars will probably have very small impact on Terry Tao’s decision to work on the problem.
OTOH, setting up an open invitation for all world-class mathematicians/physicists/theoretical computer science to work on AGI safety through some sort of sabbatical system may be very impactful.
Many academics, especially in theoretical areas where funding for even the very best can be scarce, would jump at the opportunity of a no-strings-attached sabbatical. The no-strings-attached is crucial to my mind. Despite LW/Rationalist dogma equating IQ with weirdo-points, the vast majority of brilliant (mathematical) minds are fairly conventional—see Tao, Euler, Gauss.
EA cause area?
I know ‘Theorems as Definitions’ as ‘French definitions’.
In algebraic geometry the main objects of study are algebraic varieties and more generally schemes. Although the proper definition of schemes is famously involved one can best think of them as the space of solutions of polynomials equations (‘zeroes of polynomials’). The natural topology on this space (the Zariski topology) is quite weird—it is generically non-Hausdorff and can even have non-closed points. For instance, the ‘right way’ of saying a scheme is Hausdorff is not asking for the underlying topological space to be Hausdorff [which only gives very trivial examples] but to ask for the diagonal map to be a closed subscheme.
This is an example of your abstract-concrete cycle: for ordinary topological spaces Hausdorff and closed diagonal are equivalent, but the latter can be defined more broadly for schemes.
Grothendieck is famous for his many French definitions. Many of the constructions he proposed were inspired by constructions and theorems in classicial differential and complex analytic geometry but then ported to the algebraic and arithmetic realm.
Grothendieck predecessors noted that many features of algebraic varieties resemble those of more traditionally geometric spaces. I.e. Andre Weil, in his Weil conjectures observed that the number of solutions of a variety over finite fields (i.e. modulo p) is closely related to the number of higher-dimensional holes (‘Betti numbers’) of the complex analytic manifold associated to the set of equations of the variety. This geometric shadow was a guiding motif for Grothendieck’s work. He realized that the Zariski topology did not properly capture the ‘inherent’ geometric structure of algebraic equations.
The naive solution to this conundrum would be to look for a topology on the sets of solutions of polynomials that is Hausdorff to replace the Zariski topology. But this is the wrong move—there is no such topology. Instead Grothendieck noted that the properties that are required these geometric analogues are only a subset of the properies of a topological space. Roughly speaking, for (co)homological reasoning to go through one only needs to known when a collection of opens ‘cover’ another open. This is an instance of your ‘Abstract-Concrete-Cycle’.
By pursuing this analogy He generalized the notion of topological space to Grothendieck topology (which leads to fancy things like sites and topoi). This leads to the famous etale topology and etale cohomology which eventually lead to the proof of the Weil conjectures.
An early triumph of the Grothendieckian way of thinking was his French definition of the etale fundamental group. Naively trying to define a fundamental group for an algebraic variety fails because there aren’t enough loops, hence the usual fundamental group is almost always trivial. This is Zariski topology weirdness.
Instead, Grothendieck noted that there is a classical theorem that the fundamental group can also be seen as an automorphism group of locally trivial topological covers. He then identified a property ‘etaleness’ (following a suggestion by Serre) that is the analogy of locally trivial topological covers for algebraic varieties and schemes. The étale fundamental group is then simply defined in terms of automorphisms of these etale covers. This may strike one as just a mummer’s trick—but it’s actually profound because it turns out Galois theory becomes a special case of the theory of etale fundamental groups (‘Grothendieck-Galois theory’).
It is a truth universally acknowledged that any article about the hidden nature of reality will not garner close to the number of comments as one about your sexy sexy homies. As the inaugural and possibly lone commenter, let me say I’m delighted to see finally this long-awaited post. I look forward to do a deep dive soon.
strawman MIRI: alignment is difficult because AI won’t be able to answer common-sense morality questions
“a child is drowning in a pool nearby. you just bought a new suit. do you save the child?”
actual MIRI: almost by definition a superintelligent AI will know what humans want and value. It just won’t necessarily care. The ‘value pointing’ problem isn’t about pointing to human values in its belief but in its own preferences.
There are several subtleties: belief is selected by reality (having wrong beliefs is punished) and highly constrained, preferences are highly unconstrained (this is a more subtle version of the orthogonality thesis). human value is complex and hard to specify—in particular hitting it by pointing approximately at it (‘in preference space’) is highly unlikely to hit it (and because there is no ‘correction from reality’ like in belief).
strawman Barnett: MIRI believes strawman MIRI and gpt-4 can answer common-sense morality questions so it update.
actual Barnett: i understand the argument that there is a difference between making AI know human values versus caring about those values. I’m arguing that the human value function is in fact not that hard to specify. approximate human utility function is relatively simple and a gpt-4 knows it.
(which is still distinct from saying gpt-4 or some AI will care about it. but at least it belies the claim that human values are hugely complex).
I’d be worried about evaporative cooling. It seems that the net result of this would be that labs would be almost completely devoid of people earnest about safety.
I agree with you government pathways to impact are most plausible and until recently undervalued. I also agree with you there are weird competitive pressures at labs.
It’s fascinating how Gary Marcus has become one of the most prominent advocates of AI safety, and particularly what he call long-term safety, despite being wrong on almost every prediction he has made to date.
I read a tweet that said something to the effect that GOFAI researchers remain the best ai safety researchers since nothing they did worked out.
Why don’t animals have guns?
Or why didn’t evolution evolve the Hydralisk?
Evolution has found (sometimes multiple times) the camera, general intelligence, nanotech, electronavigation, aerial endurance better than any drone, robots more flexible than any human-made drone, highly efficient photosynthesis, etc.
First of all let’s answer another question: why didn’t evolution evolve the wheel like the alien wheeled elephants in His Dark Materials?
Is it biologically impossible to evolve?
Well, technically, the flagella of various bacteria is a proper wheel.
No the likely answer is that wheels are great when you have roads and suck when you don’t. Roads are build by ants to some degree but on the whole probably don’t make sense for an animal-intelligence species.
Aren’t there animals that use projectiles?
Hold up. Is it actually true that there is not a single animal with a gun, harpoon or other projectile weapon?
Porcupines have quils, some snakes spit venom, a type of fish spits water as a projectile to kick insects of leaves than eats insects. Bombadier beetles can produce an explosive chemical mixture. Skunks use some other chemicals. Some snails shoot harpoons from very close range. There is a crustacean that can snap its claw so quickly it creates a shockwave stunning fish. Octopi use ink. Goliath birdeater spider shoot hair. Electric eels shoot electricity etc.
Maybe there isn’t an incentive gradient? The problem with this argument is that the same argument can be made for lots and lots of abilities that animals have developed, often multiple times. Flight, camera, a nervous system.
But flight has an intermediate form: glider monkeys, flying squirrels, flying fish.
Except, I think there are lots of intermediate forms for guns & harpoons too:
There are animals with quills. It’s only a small number of steps from having quils that you release when attack to actively shooting and aiming these quils. Why didn’t Evolution evolve Hydralisks? For many other examples—see the list above.
In a Galaxy far far away
I think it is plausible that the reason animals don’t have guns is simply an accident. Somewhere in the vast expanses of space circling a dim sun-like star the water-bearing planet Hiram Maxim is teeming with life. Nothing like an intelligent species has yet evolved yet it’s many lifeforms sport a wide variety of highly effective projectile weapons. Indeed, the majority of larger lifeforms have some form of projective weapon as a result of the evolutionary arms race. The savannahs sport gazelle-like herbivores evading sniper-gun equppied predators.
Some many parsecs away is the planet Big Bertha, a world is embroilled in permanent biological trench warfare. More than 95% percent of the biomass of animals larger than a mouse is taken up by members of just 4 geni of eusocial gun-equipped species or their domesticastes. Yet the individual intelligence of members of these species doesn’t exceed that of a cat.
The largest of the four geni builds massive dams like beavers, practices husbandry of various domesticated species, agriculture and engages in massive warfare against rival colonies using projectile harpoons that grow from their limbs. Yet all of this is biological, not technological: the behaviours and abilites are evolved rather than learned. There is not a single species whose intelligence rivals that of a Great ape, either individually or collectively.