The Soviet Union did violate the Biological Weapons Convention, which seems like an example of “an important, binding, ratified arms treaty.” It did not lead to nuclear war.
Jeffrey Heninger
Karma: 823
In practice, smoothness interacts with measurement: we can usually measure the higher-order bits without measuring lower-order bits, but we can’t easily measure the lower-order bits without the higher-order bits. Imagine, for instance, trying to design a thermometer which measures the fifth bit of temperature but not the four highest-order bits. Probably we’d build a thermometer which measured them all, and then threw away the first four bits! Fundamentally, it’s because of the informational asymmetry: higher-order bits affect everything, but lower-order bits mostly don’t affect higher-order bits much, so long as our functions are smooth. So, measurement in general will favor higher-order bits.
There are examples of measuring lower-order bits without measuring higher-order bits. If something is valuable to measure, there’s a good chance that someone has figured out a way to measure it. Here is the most common example of this that I am familiar with:
When dealing with lasers, it is often useful to pass the laser through a beam splitter, so part of the beam travels along one path and part of the beam travels along a different path. These two beams are often brought back together later. The combination might have either constructive or destructive interference. It has constructive interference if the difference in path lengths is an integer multiple of the wavelength, and destructive interference if the difference in path length is a half integer multiple of the wavelength. This allows you to measure changes in differences in path lengths, without knowing how many wavelengths either path length is.
One place this is used is in LIGO. LIGO is an interferometer with two multiple kilometer long arms. It measures extremely small ( $ 10^{-19} $ m) changes in the difference between the two arm lengths caused by passing gravitational waves.
Unfortunately, decisions about units are made by a bunch of unaccountable bureaucrats. They would rather define the second in terms that only the techno-aristocracy can understand instead of using a definition that everyone can understand. It’s time to turn control over our systems of measurement back to the people !
#DemocratizeUnits
I think you should ask the author of the song if it’s referring to someone using powerful AI to do something transformative to the sun.
This is extremely obvious to me. The song is opposed to how the sun currently is, calling it “wasteful” and “distasteful”—the second word is a quote from a fictional character, but the first is not. It later talks about when “the sun’s a battery,” so something about the sun is going to change. I really don’t know what “some big old computer” could be referring to if not powerful AI.
Thank you for responding! I am being very critical, both in foundational and nitpicky ways. This can be annoying and make people want to circle the wagons. But you and the other organizers are engaging constructively, which is great.
The distinction between Solstice representing a single coherent worldview vs. a series of reflections also came up in comments on a draft. In particular, the Spinozism of Songs Stay Sung feels a lot weirder if it is taken as the response to the darkness, which I initially did, rather than one response to the darkness.Nevertheless, including something in Solstice solidly establishes it as a normal / acceptable belief for rationalists: within the local Overton Window. You might not be explicitly telling people that they ought to believe something, but you are telling that it is acceptable for high status people in their community to believe it. I am concerned that some of these beliefs are even treated as acceptable.
Take Great Transhumanist Future. It has “a coder” dismantling the sun “in another twenty years with some big old computer.” This is a call to accelerate AI development, and use it for extremely transformative actions. Some of the organizers believe that this is the sort of thing that will literally kill everyone. Even if it goes well, it would make life as it currently exists on the surface of the Earth impossible. Life could still continue in other ways, but some of us might want to still live here in 20 years.[1] I don’t think that reckless AI accelerationism should be treated as locally acceptable.
The line in Brighter Than Today points in the same way. It’s not only anti-religious. It is also disparaging towards people who warn about the destructive potential of a new technology. Is that an attitude we want to establish as normal?If the main problem with changing the songs is in making them scan and rhyme, then I can probably just pay that cost. This isn’t a thing I’m particularly skilled at, but there are people who are who are adjacent to the community. I’m happy to ask them to rewrite a few lines, if the new versions will plausibly be used.
If the main problem with changing the songs is that many people in this community want to sing about AI accelerationism and want the songs to be anti-religious, then I stand by my criticisms.
- ^
Is this action unilateral? Unclear. There might be a global consensus building phase, or a period of reflection. They aren’t mentioned in the song. These processes can’t take very long given the timelines.
- ^
This is more volunteer-based than I was expecting. I would have guessed that Solstice had a lot of creative work, the choir, and day-of work done by volunteers, but that the organizers and most of the performers were paid (perhaps below market rates). As it is, it is probably more volunteer-based than most Christmas programs.
I’ll edit the original post to say that this suggestion is already being followed.
The impression of incuriosity is probably just because I collapsed my thoughts into a few bullet points.
The causal link between human intelligence and neurons is not just because they’re both complicated. My thought process here is something more like:
All instances of human intelligence we are familiar with are associated with a brain.
Brains are built out of neurons.
Neurons’ dynamics looks very different from the dynamics of bits.
Maybe these differences are important for some of the things brains can do.
It feels pretty plausible that the underlying architecture of brains is important for at least some of the things brains can do. Maybe we will see multiple realizability where similar intelligence can be either built on a brain or on a computer. But we have not (yet?) seen that, even for extremely simple brains.
I think both that we do not know how to build a superintelligence and that if we knew how to model neurons, silicon chips would run it extremely slowly. Both things are missing.
Brains do these kinds of things because they run algorithms designed to do these kinds of things.
If by ‘algorithm’, you mean thing-that-does-a-thing, then I think I agree. If by ‘algorithm’, you mean thing-that-can-be-implemented-in-python, then I disagree.
Perhaps a good analogy comes from quantum computing.* Shor’s algorithm is not implementable on a classical computer. It can be approximated by a classical computer, at very high cost. Qubits are not bits, or combinations of bits. They have different underlying dynamics, which makes quantum computers importantly distinct from classical computers.
The claim is that the brain is also built out of things which are dynamically distinct from bits. ‘Chaos’ here is being used in the modern technical sense, not in the ancient Greek sense to mean ‘formless matter’. Low dimensional chaotic systems can be approximated on a classical computer, although this gets harder as the dimensionality increases. Maybe this grounds out in some simple mesoscopic classical system, which can be easily modeled with bits, but it seems likely to me that it grounds out in a quantum system, which cannot.
* I’m not an expert in quantum computer, so I’m not super confident in this analogy.
The London subway was private and returned enough profit to slowly expand while it was coal powered. Once it electrified, it became more profitable and expanded quickly.
The Baltimore tunnel was and is part of an intercity line that is mostly above ground. It was technologically similar to London, but operationally very different.
OpenAI has to face off against giants like Google and Facebook, as well as other startups like Anthropic. There are dozens of other organizations in this space, although most are not as competitive as these.
Commonwealth Fusion has to face off against giants like ITER (funding maybe $22B, maybe $65B, estimates vary) and the China National Nuclear Corporation (building CFETR at ?? cost, while a much smaller experiment in China cost ~$1B), as well as other startups like Helion. The Fusion Industry Association has 37 members, which are all private companies trying to get fusion.
There’s probably currently more private investment in AI, and more public investment in fusion. Many of the investments are not publicly available, so a direct comparison between the entire fields is difficult. I choose to focus on two startups with available data that seem to be leading in their respective fields.
Helion has raised a similar amount of capital as Commonwealth: $2.2B. Helion also has hundreds of employees: their LinkedIn puts them in the 201-500 employees category. It was founded in 2013, so it is a bit older than CFS or OpenAI.
My general sense is that there’s more confidence in the plasma physics community that CFS will succeed than that Helion will succeed.
SPARC is a tokamak, and tokamaks have been extensively studied. SPARC is basically JET with a stronger magnetic field, and JET has been operational since the 1980s and has achieved Q=0.67. It’s only a small extrapolation to say that SPARC can get Q>1. Getting to Q~10 involves more extrapolating of the empirical scaling laws and trusting numerical simulations, but these are scaling laws and simulations that much of the plasma physics community has been working on for decades.
Helion uses a different design. This design has been tested much less, and far fewer plasma physicists have worked on it. They also haven’t published as much data from their most recent experiment: last time I checked, they had published the temperature they had reached on Trenta, but not the density or confinement time. Maybe the unpublished results are really good, and suggest that the scaling that has worked so far will continue to work for Polaris, but maybe they’re not. It’s plausible that Polaris will get Q>1 when it is built (planned for 2024), but I’m not as confident about it.
Also, Helion uses D-He3 rather than D-T. This means that they produce far fewer and less energetic neutrons, but it means that their targets for temperature and density / confinement time are an order of magnitude higher. Even if you think D-He3 is better in the long term (and it’s not clear that it is), using D-T for initial experiments is easier.
It seems to me that governments now believe that AI will be significant, but not extremely advantageous.
I don’t think that many policy makers believe that AI could cause GDP growth of 20+% within 10 years. Maybe they think that powerful AI would add 1% to GDP growth rates, which is definitely worth caring about. It wouldn’t be enough for any country which developed it to become the most powerful country in the world within a few decades, and would be an incentive in line with some other technologies that have been rejected.
The UK has AI as one of their “priority areas of focus”, along with quantum technologies, engineering biology, semiconductors and future telecoms in their International Technology Strategy. In the UK’s overall strategy document, ‘AI’ is mentioned 15 times, compared to ‘cyber’ (45 times), ‘nuclear’ (43), ‘energy’ (37), ‘climate’ (30), ‘space’ (17), ‘health’ (15), ‘food’ (8), ‘quantum’ (7), ‘green’ (6), and ‘biology’ (5). AI is becoming part of countries’ strategies, but I don’t think it’s at the forefront. The UK government is more involved in AI policy than most governments.
The original version of the song reads to me as being deist or pantheist. You could replace ‘God’ with ‘Nature’ and the meaning would be almost the same. My view of Divinely Guided Evolution has a personal God fiddling with random mutations and randomly determined external factors to create the things He wants.
It is definitely anti-Young-Earth-Creationism, but it is also dismissive of the Bible. Even if you don’t think that Genesis 1 should be treated as a chronology, I think that you should take the Bible seriously. Its commentary on what it means to be human is important.
Staggering the gathering in time also works. Many churches repeat their Christmas service multiple times over the course of the day, to allow more people to come than can fit in the building.
The Lord of the Rings tells us that the hobbit’s simple notion of goodness is more effective at resisting the influence of a hostile artificial intelligence than the more complicated ethical systems of the Wise.
The miscellaneous quotes at the end are not directly connected to the thesis statement.
It seems like your comment is saying something like:
These restrictions are more relevant to an Oracle than to other kinds of AI.
Adding a compass is unlikely to also make the bird disoriented when exposed to a weak magnetic field which oscillates at the right frequency. Which means that the emulated bird will not behave like the real bird in this scenario.
You could add this phenomenon in by hand. Attach some detector to your compass and have it turn off the compass when these fields are measured.
More generally, adding in these features ad hoc will likely work for the things that you know about ahead of time, but is very unlikely to work like the bird outside of its training distribution. If you have a model of the bird that includes the relevant physics for this phenomenon, it is much more likely to work outside of its training distribution.
I’m currently leaning towards
kings and commonwealths and all
I thought about including valuation in the table as well, but decided against it:
I’m not sure how accurate startup valuations are. It make be less clear how to interpret what the funding received means, but the number is easier to measure accurately.
These are young companies, so the timing of the valuation matters a lot. OpenAI’s valuation is recent, or 8 years after the company was founded. Commonwealth Fusion’s valuation is from 2 years ago, or 4 years after the company was founded. If each had multiple valuations, then I would have made a graph like Figure 1 for this.
This kind of situation is dealt with in Quine’s Two Dogmas of Empiricism, especially the last section, “Empiricism Without the Dogmas.” This is a short (~10k words), straightforward, and influential work in the philosophy of science, so it is really worth reading the original.
Quine describes science as a network of beliefs about the world. Experimental measurements form a kind of “boundary conditions” for the beliefs. Since belief space is larger than the space of experiments which have been performed, the boundary conditions meaningfully constrain but do not fully determine the network.
Some beliefs are closer to the core of the network: changing them would require changing lots of other beliefs. Some beliefs are closer to the periphery: changing them would change your beliefs about a few contingent facts about the world, but not much else.
In this example, the belief in Newton’s laws are much closer to the core than the belief in the stability of this particular pendulum.[1]
When an experiment disagrees with our expectations, it is not obvious where the change should be made. It could be made close to the edges, or it could imply that something is wrong with the core. It is often reasonable for science (as a social institution) to prefer changes made in the periphery over changes made in the core. But this is not always the implication the experiment makes.
A particular example that I am fond of involves the perihelion drifts of Uranus and Mercury. By the early 1800s, there was good evidence that the orbits of both planets were different from what Newtonian mechanics predicted. Both problems would be resolved by the mid 1900s, but the resolutions were very different. The unexpected perihelion drift of Uranus was explained by the existence of another planet in our solar system: Neptune. The number of planets in our solar system is a periphery belief: changing it does not require many other beliefs to change. People then expected that Mercury’s unexpected perihelion drift would have a similar cause: a yet undiscovered planet close to the sun, which they named Vulcan. This was wrong.[2] Instead, the explanation was the Newtonian mechanics was wrong and had to be replaced by general relativity. Even though the evidence in both cases was the same, they implied that there should be changes made at different places in the web of beliefs.
Also, figuring things out in hindsight is totally allowed in science. Many of our best predictions are actually postdictions. Predictions are more impressive, but postdictions are evidence too.
The biggest problem these students have is being too committed to not using hindsight.
I would say that this planet was not discovered, except apparently in 1859 a French physician / amateur astronomer named Lescarbault observed a black dot transiting the sun which looked like a planet with an orbital period of 19 days.
I would say that this observation was not replicated, except it was. Including by professional astronomers (Watson & Swift) who had previously discovered multiple asteroids and comets. It was not consistently replicated, and photographs of solar eclipses in 1901, 1905, and 1908 did not show it.
What should we make of these observations?
There’s always recourse to extremely small changes right next to the empirical boundary conditions. Maybe Lescarbault, Watson, Swift, & others were mistaken about what they saw. Or maybe they were lying. Or maybe you shouldn’t even believe my claim that they said this.
These sorts of dismissals might feel nasty, but they are an integral part of science. Some experiments are just wrong. Maybe you figure out why (this particular piece of equipment wasn’t working right), and maybe you don’t. Figuring out what evidence should be dismissed, what evidence requires significant but not surprising changes, and what evidence requires you to completely overhaul your belief system is a major challenge in science. Empiricism itself does not solve the problem because, as Quine points out, the web of beliefs is underdetermined by the totality of measured data.