The Soviet Union did violate the Biological Weapons Convention, which seems like an example of “an important, binding, ratified arms treaty.” It did not lead to nuclear war.
Jeffrey Heninger
In practice, smoothness interacts with measurement: we can usually measure the higher-order bits without measuring lower-order bits, but we can’t easily measure the lower-order bits without the higher-order bits. Imagine, for instance, trying to design a thermometer which measures the fifth bit of temperature but not the four highest-order bits. Probably we’d build a thermometer which measured them all, and then threw away the first four bits! Fundamentally, it’s because of the informational asymmetry: higher-order bits affect everything, but lower-order bits mostly don’t affect higher-order bits much, so long as our functions are smooth. So, measurement in general will favor higher-order bits.
There are examples of measuring lower-order bits without measuring higher-order bits. If something is valuable to measure, there’s a good chance that someone has figured out a way to measure it. Here is the most common example of this that I am familiar with:
When dealing with lasers, it is often useful to pass the laser through a beam splitter, so part of the beam travels along one path and part of the beam travels along a different path. These two beams are often brought back together later. The combination might have either constructive or destructive interference. It has constructive interference if the difference in path lengths is an integer multiple of the wavelength, and destructive interference if the difference in path length is a half integer multiple of the wavelength. This allows you to measure changes in differences in path lengths, without knowing how many wavelengths either path length is.
One place this is used is in LIGO. LIGO is an interferometer with two multiple kilometer long arms. It measures extremely small ( $ 10^{-19} $ m) changes in the difference between the two arm lengths caused by passing gravitational waves.
Unfortunately, decisions about units are made by a bunch of unaccountable bureaucrats. They would rather define the second in terms that only the techno-aristocracy can understand instead of using a definition that everyone can understand. It’s time to turn control over our systems of measurement back to the people !
#DemocratizeUnits
This is more volunteer-based than I was expecting. I would have guessed that Solstice had a lot of creative work, the choir, and day-of work done by volunteers, but that the organizers and most of the performers were paid (perhaps below market rates). As it is, it is probably more volunteer-based than most Christmas programs.
I’ll edit the original post to say that this suggestion is already being followed.
The impression of incuriosity is probably just because I collapsed my thoughts into a few bullet points.
The causal link between human intelligence and neurons is not just because they’re both complicated. My thought process here is something more like:
All instances of human intelligence we are familiar with are associated with a brain.
Brains are built out of neurons.
Neurons’ dynamics looks very different from the dynamics of bits.
Maybe these differences are important for some of the things brains can do.
It feels pretty plausible that the underlying architecture of brains is important for at least some of the things brains can do. Maybe we will see multiple realizability where similar intelligence can be either built on a brain or on a computer. But we have not (yet?) seen that, even for extremely simple brains.
I think both that we do not know how to build a superintelligence and that if we knew how to model neurons, silicon chips would run it extremely slowly. Both things are missing.
Brains do these kinds of things because they run algorithms designed to do these kinds of things.
If by ‘algorithm’, you mean thing-that-does-a-thing, then I think I agree. If by ‘algorithm’, you mean thing-that-can-be-implemented-in-python, then I disagree.
Perhaps a good analogy comes from quantum computing.* Shor’s algorithm is not implementable on a classical computer. It can be approximated by a classical computer, at very high cost. Qubits are not bits, or combinations of bits. They have different underlying dynamics, which makes quantum computers importantly distinct from classical computers.
The claim is that the brain is also built out of things which are dynamically distinct from bits. ‘Chaos’ here is being used in the modern technical sense, not in the ancient Greek sense to mean ‘formless matter’. Low dimensional chaotic systems can be approximated on a classical computer, although this gets harder as the dimensionality increases. Maybe this grounds out in some simple mesoscopic classical system, which can be easily modeled with bits, but it seems likely to me that it grounds out in a quantum system, which cannot.
* I’m not an expert in quantum computer, so I’m not super confident in this analogy.
The London subway was private and returned enough profit to slowly expand while it was coal powered. Once it electrified, it became more profitable and expanded quickly.
The Baltimore tunnel was and is part of an intercity line that is mostly above ground. It was technologically similar to London, but operationally very different.
OpenAI has to face off against giants like Google and Facebook, as well as other startups like Anthropic. There are dozens of other organizations in this space, although most are not as competitive as these.
Commonwealth Fusion has to face off against giants like ITER (funding maybe $22B, maybe $65B, estimates vary) and the China National Nuclear Corporation (building CFETR at ?? cost, while a much smaller experiment in China cost ~$1B), as well as other startups like Helion. The Fusion Industry Association has 37 members, which are all private companies trying to get fusion.
There’s probably currently more private investment in AI, and more public investment in fusion. Many of the investments are not publicly available, so a direct comparison between the entire fields is difficult. I choose to focus on two startups with available data that seem to be leading in their respective fields.
Helion has raised a similar amount of capital as Commonwealth: $2.2B. Helion also has hundreds of employees: their LinkedIn puts them in the 201-500 employees category. It was founded in 2013, so it is a bit older than CFS or OpenAI.
My general sense is that there’s more confidence in the plasma physics community that CFS will succeed than that Helion will succeed.
SPARC is a tokamak, and tokamaks have been extensively studied. SPARC is basically JET with a stronger magnetic field, and JET has been operational since the 1980s and has achieved Q=0.67. It’s only a small extrapolation to say that SPARC can get Q>1. Getting to Q~10 involves more extrapolating of the empirical scaling laws and trusting numerical simulations, but these are scaling laws and simulations that much of the plasma physics community has been working on for decades.
Helion uses a different design. This design has been tested much less, and far fewer plasma physicists have worked on it. They also haven’t published as much data from their most recent experiment: last time I checked, they had published the temperature they had reached on Trenta, but not the density or confinement time. Maybe the unpublished results are really good, and suggest that the scaling that has worked so far will continue to work for Polaris, but maybe they’re not. It’s plausible that Polaris will get Q>1 when it is built (planned for 2024), but I’m not as confident about it.
Also, Helion uses D-He3 rather than D-T. This means that they produce far fewer and less energetic neutrons, but it means that their targets for temperature and density / confinement time are an order of magnitude higher. Even if you think D-He3 is better in the long term (and it’s not clear that it is), using D-T for initial experiments is easier.
It seems to me that governments now believe that AI will be significant, but not extremely advantageous.
I don’t think that many policy makers believe that AI could cause GDP growth of 20+% within 10 years. Maybe they think that powerful AI would add 1% to GDP growth rates, which is definitely worth caring about. It wouldn’t be enough for any country which developed it to become the most powerful country in the world within a few decades, and would be an incentive in line with some other technologies that have been rejected.
The UK has AI as one of their “priority areas of focus”, along with quantum technologies, engineering biology, semiconductors and future telecoms in their International Technology Strategy. In the UK’s overall strategy document, ‘AI’ is mentioned 15 times, compared to ‘cyber’ (45 times), ‘nuclear’ (43), ‘energy’ (37), ‘climate’ (30), ‘space’ (17), ‘health’ (15), ‘food’ (8), ‘quantum’ (7), ‘green’ (6), and ‘biology’ (5). AI is becoming part of countries’ strategies, but I don’t think it’s at the forefront. The UK government is more involved in AI policy than most governments.
The original version of the song reads to me as being deist or pantheist. You could replace ‘God’ with ‘Nature’ and the meaning would be almost the same. My view of Divinely Guided Evolution has a personal God fiddling with random mutations and randomly determined external factors to create the things He wants.
It is definitely anti-Young-Earth-Creationism, but it is also dismissive of the Bible. Even if you don’t think that Genesis 1 should be treated as a chronology, I think that you should take the Bible seriously. Its commentary on what it means to be human is important.
Staggering the gathering in time also works. Many churches repeat their Christmas service multiple times over the course of the day, to allow more people to come than can fit in the building.
The Lord of the Rings tells us that the hobbit’s simple notion of goodness is more effective at resisting the influence of a hostile artificial intelligence than the more complicated ethical systems of the Wise.
The miscellaneous quotes at the end are not directly connected to the thesis statement.
It seems like your comment is saying something like:
These restrictions are more relevant to an Oracle than to other kinds of AI.
Adding a compass is unlikely to also make the bird disoriented when exposed to a weak magnetic field which oscillates at the right frequency. Which means that the emulated bird will not behave like the real bird in this scenario.
You could add this phenomenon in by hand. Attach some detector to your compass and have it turn off the compass when these fields are measured.
More generally, adding in these features ad hoc will likely work for the things that you know about ahead of time, but is very unlikely to work like the bird outside of its training distribution. If you have a model of the bird that includes the relevant physics for this phenomenon, it is much more likely to work outside of its training distribution.
I’m currently leaning towards
kings and commonwealths and all
I thought about including valuation in the table as well, but decided against it:
I’m not sure how accurate startup valuations are. It make be less clear how to interpret what the funding received means, but the number is easier to measure accurately.
These are young companies, so the timing of the valuation matters a lot. OpenAI’s valuation is recent, or 8 years after the company was founded. Commonwealth Fusion’s valuation is from 2 years ago, or 4 years after the company was founded. If each had multiple valuations, then I would have made a graph like Figure 1 for this.
I don’t believe that “current AI is at human intelligence in most areas”. I think that it is superhuman in a few areas, within the human range in some areas, and subhuman in many areas—especially areas where the things you’re trying to do are not well specified tasks.
I’m not sure how to weight people who think most about how to build AGI vs more general AI researchers (median says HLAI in 2059, p(Doom) 5-10%) vs forecasters more generally. There’s a difference in how much people have thought about it, but also selection bias: most people who are skeptical of AGI soon are likely not going to work in alignment circles or an AGI lab. The relevant reference class is not the Wright Brothers, since hindsight tells us that they were the ones who succeeded. One relevant reference class is the Society for the Encouragement of Aerial Locomotion by means of Heavier-than-Air Machines, founded in 1863, although I don’t know what their predictions were. It might also make sense to include many groups of futurists focusing on many potential technologies, rather than just on one technology that we know worked out.
I chose the start date of 1866 because that is the first time the New York Senate appointed a committee to study rapid transit in New York, which concluded that New York would be best served by an underground railroad. It’s also the start date that Katz uses.
The technology was available. London opened its first subway line in 1863. There is a 1.4 mi railroad tunnel from 1873 in Baltimore that is still in active use today. These early tunnels used steam engines. This did cause ventilation challenges, but they were resolvable. The other reasonable pre-electricity option would be to have stationary steam engines at a few places, open to the air, that pulled cables that pulled the trains. There were also some suggestions of dubious power mechanisms, like the one you described here. None of the options were as good as electric trains, but some of them could have been made to work.
This is not a global technological overhang, because there continued to be urban railroad innovation in other cities. It would only be overhang for New York City. This is a more restrictive definition of overhang than I used in my previous post, but it might still be interesting to see what happened with local overhang.
This kind of situation is dealt with in Quine’s Two Dogmas of Empiricism, especially the last section, “Empiricism Without the Dogmas.” This is a short (~10k words), straightforward, and influential work in the philosophy of science, so it is really worth reading the original.
Quine describes science as a network of beliefs about the world. Experimental measurements form a kind of “boundary conditions” for the beliefs. Since belief space is larger than the space of experiments which have been performed, the boundary conditions meaningfully constrain but do not fully determine the network.
Some beliefs are closer to the core of the network: changing them would require changing lots of other beliefs. Some beliefs are closer to the periphery: changing them would change your beliefs about a few contingent facts about the world, but not much else.
In this example, the belief in Newton’s laws are much closer to the core than the belief in the stability of this particular pendulum.[1]
When an experiment disagrees with our expectations, it is not obvious where the change should be made. It could be made close to the edges, or it could imply that something is wrong with the core. It is often reasonable for science (as a social institution) to prefer changes made in the periphery over changes made in the core. But this is not always the implication the experiment makes.
A particular example that I am fond of involves the perihelion drifts of Uranus and Mercury. By the early 1800s, there was good evidence that the orbits of both planets were different from what Newtonian mechanics predicted. Both problems would be resolved by the mid 1900s, but the resolutions were very different. The unexpected perihelion drift of Uranus was explained by the existence of another planet in our solar system: Neptune. The number of planets in our solar system is a periphery belief: changing it does not require many other beliefs to change. People then expected that Mercury’s unexpected perihelion drift would have a similar cause: a yet undiscovered planet close to the sun, which they named Vulcan. This was wrong.[2] Instead, the explanation was the Newtonian mechanics was wrong and had to be replaced by general relativity. Even though the evidence in both cases was the same, they implied that there should be changes made at different places in the web of beliefs.
Also, figuring things out in hindsight is totally allowed in science. Many of our best predictions are actually postdictions. Predictions are more impressive, but postdictions are evidence too.
The biggest problem these students have is being too committed to not using hindsight.
I would say that this planet was not discovered, except apparently in 1859 a French physician / amateur astronomer named Lescarbault observed a black dot transiting the sun which looked like a planet with an orbital period of 19 days.
I would say that this observation was not replicated, except it was. Including by professional astronomers (Watson & Swift) who had previously discovered multiple asteroids and comets. It was not consistently replicated, and photographs of solar eclipses in 1901, 1905, and 1908 did not show it.
What should we make of these observations?
There’s always recourse to extremely small changes right next to the empirical boundary conditions. Maybe Lescarbault, Watson, Swift, & others were mistaken about what they saw. Or maybe they were lying. Or maybe you shouldn’t even believe my claim that they said this.
These sorts of dismissals might feel nasty, but they are an integral part of science. Some experiments are just wrong. Maybe you figure out why (this particular piece of equipment wasn’t working right), and maybe you don’t. Figuring out what evidence should be dismissed, what evidence requires significant but not surprising changes, and what evidence requires you to completely overhaul your belief system is a major challenge in science. Empiricism itself does not solve the problem because, as Quine points out, the web of beliefs is underdetermined by the totality of measured data.