How much progress actually happens in theoretical physics?
I frequently hear people make the claim that progress in theoretically physics is stalled, partly because all the focus is on String theory and String theory doesn’t seem to pan out into real advances.
Believing it fits my existing biases, but I notice that I lack the physics understanding to really know whether or not there’s progress. What do you think?
As Adam Scherlis implies, the standard model turns out to be very effective at all the scales we can reach. There are a handful of phenomena that go beyond it—neutrino masses, “dark matter”, “dark energy”—but they are weak effects that offer scanty clues as to what exactly is behind them.
On the theoretical side, we actually have more models of possible new physics than ever before in history, the result of 50 years of work since the standard model came together. A lot of that is part of a synthesis that includes the string theory paradigm, but there are also very large numbers of theoretical ideas that are alternatives to string theory or independent of string theory. So if a decisive new phenomenon shows up, or if someone has a radical insight on how to interpret the scanty empirical clues we do have, we actually have more theories and models than ever before, that might be capable of explaining it.
The idea that progress is stalled because everyone is hypnotized by string theory, I think is simply false, and I say that despite having studied alternative theories of physics, much much more than the typical person who knows some string theory. I think this complaint mostly comes from people who don’t like string theory (Peter Woit) or who have an alternative theory they think has been neglected (Eric Weinstein). String theory did achieve a kind of hegemony within elite academia, but this was well-deserved, and meanwhile many competing research programs have had a foothold in academia too, to say nothing of the hundreds of physicists worldwide who have a personal theory that they write about, when they aren’t doing other things like teaching.
Most likely there are lost opportunities during that 50 years (like everyone else, I have my own ideas about neglected directions of research), but “do less string theory” is no guarantee that they would have been picked up. There are even those who would argue that there should have been more string theory of a certain kind (Lubos Motl used to say that field-theoretic phenomenologists should pay more attention to string theory, as a constraint and a guide in their model-building, and “stringking42069″ says that the senior figures of string theory are holding the subject back by favoring work on their own little bandwagons, rather than bold and creative work on new directions and big unsolved problems).
Are you saying that progress in physics hasn’t stalled or that string theory isn’t to blame?
I’m not Mitchell, but I think I agree with him here enough to guess: He probably means to say that production of new plausible theories has increased, production of experimentally verified theories has stalled, and the latter is not string theory’s fault.
(And of course this whole discussion, including your question, is interpreting “physics” to means “fundamental physics”, since theoretical and empirical work on e.g. condensed matter physics has been doing just fine.)
String theorist here. Progress has absolutely not stalled. The last big theoretical breakthrough was the AdS/CFT correspondence ( https://arxiv.org/abs/hep-th/9711200 ). This was our Transformers moment. This paper revolutioned the field, has now 20k+ citations, the most for any physics theory paper. It did completely solve the problem of quantum gravity for AdS spacetimes (negative curvature). There are too many things that have followed from this but I can give a few tidbits. For example it was understood that quantum black holes are the most chaotic objects in nature (https://arxiv.org/abs/1306.0622 https://arxiv.org/abs/1503.01409). A deep relationship between quantum gravity and quantum information theory was found (e.g, https://arxiv.org/abs/hep-th/0603001 https://arxiv.org/abs/1306.0533 https://arxiv.org/abs/1411.7041 ) which ultimately led recently to a solution of the black hole information paradox 50+ years after posed by Hawking! (https://arxiv.org/abs/2006.06872). There are caveats ofc, our universe is a dS spacetime (positive curvature) so current work is trying to extend AdS/CFT to cosmological spacetimes like our own.
The main reason string theory has bad press in the public is due to the fact that most top scientits don’t have time to do public outreach (and their time is probably spent better elsewhere). So the only “physicists” who do public outreach are the ones who have nothing better to do… they are often bitter or disgrnuntled people whose subfield is becoming irrelevant and who have fallen behind. So they maintain blogs or publish books to feel more relevant saying things like “not even wrong” “physics is dead” “lost in math” etc. (there are exceptions ofc such as Brian Greene, Max Tegmark etc.).
The paper you linked about the last big breakthrough seems to be from 1997, so roughly 28 years ago. What do you consider to be the biggest breakthrough since then?
I might sound a bit daft here, but do theoretical physicists actually understand what they’re talking about? My main concern when trying to learn is it feels like every term is defined with ten other terms, and when you finally get to the root of it the foundations seem pretty shaky. For example, the spin-statistics theorem says particles with half-integer spins are fermions, and full-integer spins are bosons, and is proved starting from a few key postulates:
Positive energy
Unique vacuum state
Lorentz invariance
“Locality/Causality” (all fields commute or anti-commute).
The fractional quantum Hall effect breaks Lorentz invariance (1+1D universe instead of Lorentz’ 3+1D), which is why we see anyons, so obviously the spin-statistics theorem doesn’t always hold. However, the fourth postulate shows up everywhere in theoretical physics and the only justification really given is that, “all the particles we see seem to commute or anti-commute”… which is the entire point the spin-statistics theorem is trying to prove.
Physics is incredibly precise (only pure mathematics is more rigorous) and everything does indeed follows from a small number of basic principles. These principles are not sacrosent though, more like useful assumptions, and in quantum gravity we don’t expect locality to strictly holds. The solution of the black hole information paradox mentioned above is precisely showing that the late Hawking radiation (which is very far from the black hole) is the same quantum system as the black hole interior. This is a manifestion of quantum entanglement which is realized in gravity through a connected geometry (wormhole). So the solution of the Hawking information paradox is that the information thrown into a black hole is not lost after evaporation, it has escaped into Hawking radiation through tiny quantum wormholes. These non-local effects only happen at the Planck scale or in comlplicated systems such as old black holes.
About spin-statistics, the modern understanding is that is just follows from the representation theory of the Lorentz group. The existence of fermions is due to the fact that SO(3,1) is not simply connected so what gets represented is its universal cover (a double cover) so there exists fermionic representations with anti-commuting properties. Their properties are fixed from the requirement that they must form consistent unitary representations (in order to have a quantum theory giving positive probabilities). As you mention, in lower dimensions, anyons exist, and this is due to the fact that the lower-dimensional Lorentz groups have different properties, their universal cover can be an infinite cover so there are anyonic representations too.
I think you unfortunately can’t really verify the recent epistemic health of theoretical physics, without knowing much theoretical physics, by tracing theorems back to axioms. This is impossible to do even in math (can I, as a relative layperson, formalize and check the recent Langlands Program breakthrough in LEAN?) and physics is even less based on axioms than math is.
(“Even less” bc even math is not really based on mutually-agreed-upon axioms in a naive sense, cf. Proofs and Refutations or the endless squabbling over foundations.)
Possibly you can’t externally verify the epistemic health of theoretical physics at all, post-70s, given the “out of low hanging empirical fruit” issue and the level of prerequisites needed to remotely begin to learn anything beyond QFT.
Speaking as a (former) theoretical physicist: trust us. We know what we’re talking about ;)
The main thing I’d add is that string theory is not the problem. All the experimental low hanging fruit was picked decades ago. There are very general considerations that suggest that any theory of quantum gravity, stringy or otherwise, will only really be testable at the Planck scale. What this means in practice is that theoretical high-energy physics doesn’t get to check its answers anymore.
I think there’s still progress, and still hope for new empirical results (especially in astrophysics and cosmology), but it’s much harder without a barrage of unexplained observations.
This doesn’t seem to address the question, which was why do people believe there is a physics slow-down in the first place.
Isn’t the answer that the low hanging fruit of explaining unexplained observations has been picked?
That’s an inference, presumably Adam believes that for object-level reasons, which could be supported by eg looking at the age at which physicists make major advancements[1] and the size of those advancements.
Edit: But also this wouldn’t show whether or not theoretical physics is actually in a rut, to someone who doesn’t know what the field looks like now.
Adjusted for similar but known to be fast moving fields like AI or biology to normalize for facts like eg the academic job market just being worse now than previously.
I am not going to spend more than a few minutes here or there to give “speaking as a physicist” takes on random LW posts; I think convincing people that my views are correct in full detail would require teaching them the same things that convinced me of those views, which includes e.g. multiple years of study of QFT.
Instead, I tend to summarize what I think and invite people to ask specific questions about e.g. “why do you believe X” if they want to go further down the tree or test my beliefs more aggressively.
“That doesn’t answer the question because I am not convinced by everything you said” is not really a helpful way to do that imo.
To spell out my views: there has been a bit of a real slow-down in theoretical physics, because exploring the tree of possible theories without experiment as a pruning mechanism is slower than if you do get to prune. I think the theory slowdown also looks worse to outsiders than it is, because the ongoing progress that does happen is also harder to explain due to increasing mathematical sophistication and a lack of experimental correlates to point to. This makes e.g. string theory very hard to defend to laypeople without saying “sorry, go learn the theory first”.
This is downstream of a more severe slowdown in unexplained empirical results, which results from (imo) pretty general considerations of precision and energy scales, per the modern understanding of renormalization, which suggest that “low-hanging fruit gets picked and it becomes extremely expensive to find new fruit” is a priori pretty much how you should expect experimental physics to work. And indeed this seems to have happened in the mid 20th century, when lots of money got spent on experimental physics and the remaining fruit now hangs very high indeed.
And then there’s the 90s/2000s LHC supersymmetry hype problem, which is a whole nother (related) story.
A lot of the effect is picking high-hanging fruit.
Like, go to phys rev D now. There’s clearly a lot of hard work still going on. But that hard work seems to be getting less result, because they’re doing things like carefully calculating the trailing-order terms of the muon’s magnetic moment to get a change many decimal places down. (It turns out that this might be important for studying physics beyond the Standard Model. So this is good and useful work, definitely not being literally stalled.)
Another chunk of the effect is that you generally don’t know what’s important now. In hindsight you can look back and see all these important bits of progress woven into a sensible narrative. But research that’s being done right now hasn’t had time to earn its place in such a narrative. Especially if you’re an outside observer who has to get the narrative of research third-hand.
First, you should distinguish theoretical physics from fundamental physics. For example, there is a theory of superconductivity, but it doesn’t apply to high temperature superconductors, so there is room for more theories. The high status of fundamental physics is probably bad for theoretical physics, drawing people away from a huge range of open problems, but this isn’t the fault of string theory. String theory compounded this by producing the illusion of opportunity, but it is hard to say how much is each problem.
Second, even if string theory is wrong, it can be useful. AdS is obviously wrong, but its ease of study can be useful. People go back and forth between general arguments and specific arguments. General arguments about what any theory of quantum gravity must look like are good. Checking that they can be made rigorous in specific theories like string theory is also good. Having more specific theories would be better.
Since physics separated from natural philosophy in the times of Newton, it has almost always[1] progressed when new experimental data uncovered deficiencies in then-current understanding of the universe. During the Cold War unprecedentedly large amount of money were invested into experimental physics, and by the late 20th century all reasonably low hanging fruits have been picked (in the meantime the experiments have got absurdly expensive and difficult). I have also wrote on the topic at https://www.lesswrong.com/posts/CCnycGceT4HyDKDzK/a-history-of-the-future-2025-2040?commentId=KtusJZLAFDt4PW65R and the thread below, check it out.
As of the string theory in particular, it represents just one significant school of thought very popular in the US but other theories share the same problem of lacking the experimental data to test against.
Also, the body of knowledge in physics has become so large that local progress made here and there is not really visible in the grand scheme of things anymore even if it’s worth a Nobel Prize (while during the Second Industrial Revolution one discovery could, figuratively speaking, establish a new branch of science)
Two notable exceptions that, IMHO, kind of support the rule are Maxwell’s Equations and the General Relativity
Yes diminishing returns. For a purely theoretical field say pure mathematics this is even clearer. Take a fixed number of humans with a fixed intelligence (both average and outliers) then let mathematics advance. It will advance to the point that there is a vanishingly small number of people who can even understand the state of the art, with them then making little to no further progress. All else being equal the promising ideas are explored first etc.
More people just pushes you along the asymptote faster. If we took things even further, some knowledge would not even be live at one time. For example if there happened to be a few genius in a sub-field and it was in fashion, then humanities understanding of that sub field would advance, and be written down. However in 50 years or so no-one would actually understand the work, and it would require an unusual genius to be born to even take humanity knowledge to the edge of what is written down.
Given theoretical physics has such low useful bandwidth coming from experiments (Higgs is real, what else, countless null results upholding the standard model don’t help) then the same situation applies. We won’t solve quantum gravity this side of the Singularity or massive human enhancement.
In many cases, there are diminishing returns to a given scientific paradigm. The fact that you observe a field getting diminishing returns doesn’t mean that there isn’t a paradigm that the field could adopt that would allow for returns to flow again. Paradigm change is about pursuing ideas that people in the old paradigm don’t find promising.
Just adding more smart people who follow a hegemonic paradigm doesn’t automatically get you paradigm shifts that unlock new returns. If string theory stiffles progress, it would look from the inside like there are diminishing returns to theoretical physics.
How is that an argument against the meta prediction that you should expect diminishing returns in any field of a largely theoretical nature? In tech, you get one or both of better tools enabling more progress, or many results to guide you. Tech builds on itself in a way theoretical fields dont.
In theoretical fields the proof of a maths theorem doesn’t make everyone in the world a better mathematician or make everyone learn faster. In analogy, that kind of thing does happen in tech. A better computer chip does enable everyone using a computer to be more productive and makes chip design faster and more efficient.
If string theory stifled progress, then it would have been disrupted long before now. Even if only 20% of the smartest people weren’t working on string theory then over say 30 years they would have made more progress than the 80% because they were using their skills more efficiently. You see this constantly with startups—small groups disrupt big groups. The fact it hasn’t happened with physics suggests the problem is more fundamental than just choosing the wrong theory.
I think the problem is that you ignore the idea that science works via paradigms. Even if there’s a possible paradigm besides string theory that would produce more progress, there are a lot of different things that people who aren’t working on string theory could work on. Most of them won’t lead anywhere.
If a new paradigm could be found that has more potential, that paradigm would have new low hanging fruit.
However, researchers that would write papers about that low hanging fruit, might have trouble getting published in journals of the old paradigm because they are solving problems of interest to the new paradigm and not problems of interest of the old paradigm. Getting funding to work on problems of a new paradigm is also harder.
OK I will try to take that idea where as far as it can go. Lets assume that a lot of progress is stalled by the difficulty in overturning paradigms. For that to be the complete reason, the difficulty in overturning paradigms would have to not increase as the knowledge and maturity of the field increased. Otherwise that difficulty would still be side effect of the level of advancement in the field and just another argument in favor of diminishing returns as a field advanced.
Some fields its easy to overturn paradigms if there a very simple public metric like high jump with the Fosbury flop—it was immediately clear it was superior. If a field doesn’t have such a metric then its harder. Also if there is existing results that must be achieved by a new paradigm before its even clear that it is superior. An example of this is potentially AI where a new architecture may be superior on small data sets, but not so on large sets and if this happens a lot, then its hard to know you have an improvement without actually running costly tests. Additionally if the politics of the field is setup to make it hard for new paradigms to flourish then that is also a source of difficulty.
So for physics? Its clear there are a lot of existing results that need to be reproduced for a theory of quantum gravity or similar to be taken seriously, which both makes it difficult to get traction, but also gives external credibility. For example, today a theory has to be compatible with General Relativity and give Newtonian physics at low speed/energy etc. In Einstein’s day, the barrier would seem to have been lower.
So is there a way for a rational community to overcome these issues? As people note there are many ideas floating around for new paradigms but no easy way to find the promising ideas. That seems more like a structural problem than something politics could fix. The community would perhaps have to come up with a list of qualities or successful calc that a new paradigm could be judged on short of trying to reproduce all of physics? Anyway its not clear to me that there is large progress that could be made in that area.
So yes even if the slowdown is because we don’t have the best paradigm there needs to be an efficient way to find such paradigms that doesn’t get more inefficient as the field advances to count against the general diminishing returns argument.
In many cases, new paradigms care about different metrics than older paradigms. In the beginning, successful new paradigms usually don’t fulfill the qualities that heterodox researchers in the field are looking to. You might want to read Thomas Khun’s “The Structure of Scientific Revolutions”.
There are ways you can make it easier to overturn paradigms. You can change the way research is funded. You could change the grant-making processes in a way that makes it easier for very smart young people to pursue research agendas that heterodox old people don’t find interesting.
There’s the Max Planck quote of “Science advances one funeral at a time”. In the last decades, old researchers got more power over which research gets conducted than back in 1950 when Planck wrote his autobiography.
Sure but does this actually apply to physics? Can anyone suggest different metrics, or is there broad agreement with everyone what the major physics problems are, because it seems like there is. E.g. the non conventional people don’t say dark energy isn’t important, they have different explanations for what it is. Everyone agrees the nature and origin of time/entropy etc is important.
Can you give examples of very smart young physicists complaining they are pushed into old ways of thinking? Are you prepared to give an justify a % difference that such things would make?
For example the comment by “Mitchell_Porter”
No-one has pushed back against that here. I see your position as a theory that we need to gather evidence for and against and decide on a field by field basis. In this field I only see data against that position.
There are people like Sabine Hossenfelder who think that there are experiments we could run that we currently don’t run, because they don’t fit into the string physics paradigm.
If you look back through history, in most times you would not see data that shows that a specific non-hegemonic scientific paradigm is better than the hegemonic paradigm. We usually only have clear data that a different paradigm is better when the old paradigm stops being hegemonic.
Ok can we put some rationality to this. Your prior seems to be that when a field is stuck, it is almost entirely because of politics, hegemon etc. So how do we update that prior?
What would you expect to see in a field that is not stuck bc of inertia etc. You would still get people like Hossenfelder saying we are doing things wrong, and such people would get followers.
You suggested metrics before but havn’t provided any that I can see.
Evidence I take into account:
#1 There is not a significant group of young capable researchers saying we are taking things in the wrong direction, but a smaller number of older ones. Unless you are going to go so far as to say they are afraid to speak out, even anon, then that to me is evidence against your position. There are two capable experts on this blog from what I can see, one enthusiastic about string theory, and another who has investigated other theories in detail. Both disagree with your claim.
#2. There is not broad disagreement about what the major issues in physics are, so unlikely to be disagreements on metrics. You mentioned this as mattering, and if it does, I count this as evidence against your position.
Can you point to evidence that actually supports your prior? I can only see that which opposes it or is neutral. (In all fields no matter how things are progressing you get some people who think the establishment is doing it all wrong and have their own ideas. That can’t count as evidence in favor for a specific field unless it is happening to a greater degree)
I don’t disagree with your position in general. In fact it was clear to me that AI before neural networks was stuck with GOFAI and I believed that NN were clearly the way to go. I followed Hinton et all before they became famous. I saw the paradigm thing play out in front of my eyes there. Physics seems different.
You made statements that you know why the field is stuck. That both contains an assertion of the field being stuck and you knowing the explanation is due to diminishing returns. I made no claim that I know the answer to either of those questions.
If you think that theoretical physics is going in the wrong direction and are unlikely going to get to research in a way you think will make progress, there are good reasons for not being not making your PHD in theoretical physics. The strongest disbelievers filter themselves out.
But even among the people who are actually in the field, I don’t see a good reason why you would publically see signs.
You might ask PHD students: “If you wouldn’t need to seek grants and would get a lab with 5 million dollar per year, would you pursue the same research agendas as you are currently do or would you pursue research for which you wouldn’t get grants in the current academic environment?”
It might be interesting to ask that question for researchers in every field and see the responses, but unfortunately I don’t know of any source that asks such a question in a good way.
It’s certainly possible for their to be fields where it can be obvious to outsiders that a particular paradigm changing approach will work, but that’s not necessary for there to be superior paradigms that could be persued.
This ignores the possibility of advances in the teaching of math (or physics, or any other discipline). If improved teaching methods lower the level of intelligence required to reach a given level of knowledge, then a field can advance considerably.
Not to mention that the human population has been growing, and average intelligence has been increasing.
Finally, there’s specialization. It doesn’t take much intelligence to know everything that was known about genetics when Darwin was alive, but probably nobody is smart enough to know everything that was known about it in 2000. But there have still been make advances since then thanks to people specialized in subfields like DNA sequencing.
OK advances in teaching the highest level of physics/math needed for string theory is a big IF. Do you have evidence that is actually happening? I know of two people who tried to learn it and just found it too hard, don’t think a better teacher or materials would have helped. The evidence is mixed but personal accounts certainly suggest that only a very small number of people could get to such a high level and improved teaching probably wouldn’t help much. When we are talking about such extreme skills, people have their plateau or maximum ability level which is mostly fixed.
The human population growing just pushes us along the asymptote faster, rather than changing it.
To me the data shows that there has been no reliable increase in intelligence in the last ~30 years https://news.northwestern.edu/stories/2023/03/americans-iq-scores-are-lower-in-some-areas-higher-in-one/ Once again it needs to be at the very top level to matter.
Any specialization advantage is already tapped out with string theory and similar. My worldview definitely does not apply to advances in a field like biology as there is lots of experimental data, the tools are improving etc. I would expect advances there without any obvious plateau yet.
I don’t think pure mathematics make a good parallel. There are still discoveries made by single mathematicians or very small research groups, but this haven’t really been the case in physics since about mid-20th century, when the US and USSR invested lots of money in modern large-scale research done by huge groups
Maths doesn’t make an exact parallel but certainly fits in with my worldview. Lets say you view advanced physics as essentially a subfield of maths which is not that much of an exaggeration given how mathematical string theory etc is. If a sub field gets a lot of attention like physics has then it gets pushed along the diminishing returns curve faster. That means such single person discoveries would be much harder in physics than a given field of mathematics. The surface area of all mathematics is greater than that of just mathematical physics so that is exactly what you would predict. Individual genius mathematicians can take a field that has been given less attention—the distance from beginner to state of the art is less than that for physics. They can then advance the state of the art.
I have a second-handed source hearing this view from a theoretical physics 4th phd student at Stanford—he believes less breakthroughs nowadays as the field becomes more and more established, and this was exactly why he was a bit discouraged/sad. Not sure if things has changed, and that may or may not be his personal view.