DavidHolmes
p.s. and this now exists: https://mathforaisafety.org; “A starting point for mathematicians who want to engage with AI safety.”
Right, September was lifetimes ago! I also have the impression it makes a big difference what kind of x-risk you talk about; bio-terrorism is for example relatively easy to give a convincing explanation for.
Thanks, this motivated me to search around a bit more, the situation is better than last time I looked; I’ve tarted reading Lionel Levine’s recent paper [https://github.com/lionellevine/MAIS/blob/main/papers/P1/MAIS-P1.pdf].
My sense is many of these problems are either ill-defined or too hard to be tractable.
As a working mathematician who occasionally tries to think about some of this stuff, that’s also my feeling.
Context: I’m a mathematics professor in the Netherlands, and one of the coauthors of the Leiden Declaration on Artificial Intelligence and Mathematics. I think this article is was a nice summary of the current situation. My impression from talking to my colleagues is not that there is a lack of concern; people have not necessarily thought through the details, but the idea of existential risk from AI is taken seriously. What is missing is any clear idea of what to do about it; people feel powerless. Near the end of your essay you wrote:
“Many people say that alignment is primarily a mathematics problem, so I think we have a lot to offer.”
I’m curious if you can make that more concrete? I expect that if someone stated a reasonably concrete mathematical conjecture and said “solving this would really help with existential risk” there would be plenty of people happy to devote effort to it. But my own (perhaps very wrong) impression of the field it that the questions are more like “come up with a solution to inner alignment” which is too vague for most mathematicians to make a start on. There are some very mathematical alignment programmes out there (e.g. that of Vanessa Kosoy); do you think that having lots of people pile on a programme like that would help?
Somehow it would be great to have something like the Langland’s Programme for AI alignment; it can be hard and incompletely specified, but it gives a convincing direction, is concrete enough to get started on, and it’s clear that if you succeed you have really made progress on the problem. At the moment that seems to me to be missing, but I’d love to be corrected.
Thank you for sharing this interesting work! I will be very interested to see how far you can push this to non-repetition-based complexity measures in future.
I’d be interested to know what kind of results your LZP algorithm gives for text prediction? I’m assuming “not great”, but to me it would be an interesting point of comparison to the algorithm you are (I guess) eventually working towards.
A couple of tiny comments: for me it would have been helpful to briefly recall what
I agree with you to some extent; in the end a false statement is a false statement, whether it came from an LLM or a bad use of google (or anywhere else). But I think there are a lot of people who over-estimate the reliability of the LLMS they are using in their writing, so that the overall effect is more confident wrong claims than we had pre-LLM-use (there’s a reason the term “AI-slop” exists despite the fact that humans can also produce nonsense). I am generally in favour of policies that nudge authors towards extra checking in case of heavy LLM use.
Do you still stand by this comment in the light of the comment of Jeffrey Heninger on the Solar Storms post saying that he showed it to an expert and “The plasma physics in this post is mostly wrong.”? I think I was the first person to call into question whether the post was basically correct. I hesitated to do so because I knew I might be wrong and there was a risk of causing a pile-on. But in the light of the comment I mentioned above, I am inclined to think I made the right call?
For me, knowing when I am reading “ text written by a human, which includes facts, arguments, examples, etc, which were researched/discovered/developed with LLM assistance” is in fact way more important than knowing whether or not the actual words of the text were written by an LLM. This site is called LessWrong, and LLMs are not yet good at being it.
Perhaps a policy that facts which have been produced by an LLM and not independently verified should be flagged as such?
Thanks! For me that helps a lot. I do really appreciate the effort you have put into this, and I don’t want to suggest that no-one is allowed to talk about anything without becoming/consulting experts. At the same time I definitely agree with Gwern that in the age of LLM writing, it is more important than ever to be really clear about the epistemic status of our work.
Maybe this is a communication issue? The style of writing comes across as rather authoritative, the way you write gave me the impression that you are an expert on this topic. The only red flag that I found in the text was the “research by Claude” thanks at the end. Personally I would have appreciated a disclaimer near the start of the article. Saying that epistimics were discussed on a twitter thread not linked from the article is not helpful to me. I do not have a twitter account, so I’m afraid I’ve still not read it.
I feel like a terrible person for writing this, so apologies in advance. But when I read “Thanks to … Opus 4.6 for a lot the research”, and then in the comments people are pointing out what seem to be multiple factual errors, I can’t help but wonder whether this is all true? More precisely, it’s not clear to me how much I should update in the direction of any of the claims made in this post. Could you tell us a bit more about what fact-checking happened?
I love the idea of this! But it worries me a bit that when I look through the ones under “mathematics” the ordering seems pretty erratic. I’m a professional mathematician and managing editor for a good mathematics journal, so this should be the field I know best, and my doubts here make me question the rest.
It’s awkward for me to criticise the ranking of specific papers publicly, but to give one example the paper “Progress in the mirror symmetry program?: a criterion for the rationality of cubic fourfolds” seems vastly under-rated on the “big if true” axis relative to many other works (I think the p(generalises) for that paper is fair).
On the other hand, mathematics has a reputation for being hard for outsiders to evaluate; I’m curious of what people think of the rankings in other fields?
Hmm, so I’m very wary of defending tropical geometry when I know so little about it; if anyone more informed is reading please jump in! But until then, I’ll have a go.
tropical geometry might be relevant ML, for the simple reason that the functions coming up in ML with ReLU activation are PL
I’m not sure I agree with this argument.
Hmm, even for a very small value of `might’? I’m not saying that someone who wants to contribute to ML needs to seriously consider learning some tropical geometry, just that if one already knows tropical geometry it’s not a crazy idea to poke around a bit and see if there are applications.
The use of PL functions is by no means central to ML theory, and is an incidental aspect of early algorithms.
I agree this is an important point. I don’t actually have a good idea what activation functions people use in practise these days. Thinking about asymptotic linearity makes me think about the various papers appearing using polynomial activation functions. Do you have an opinion on this? For people in algebraic geometry it’s appealing as it generates lots of AG problems (maybe v hard), but I don’t have a good feeling as to whether it’s got anything much to do with `real life’ ML. I can link to some of the papers I’m thinking of if that’s helpful, or maybe you are already a bit familiar.
I don’t see why one wouldn’t just use ordinary currents here (currents on a PL manifold can be made sense of after smoothing, or in a distribution-valued sense, etc.).
I think you’re right; this paper just came to mind because I was reading it recently.
whether tropical geometry has ever been useful (either in proving something or at least in reconceptualizing something in an interesting way) in linear programming.
A little googling suggests there are some applications. This paper seems to give an application of tropical geometry to complexity of linear programming: https://inria.hal.science/hal-03505719/document and this list of conference abstracts seems to give other applications: https://him-application.uni-bonn.de/fileadmin/him/Workshops/TP3_21_WS1_Abstracts.pdf Whether they are ‘convincing’ I leave up to you.
1 Algebraic geometry in general (including tropical geometry) isn’t good at dealing with deep compositions of functions, and especially approximate compositions.
Fair, though one might also see that as an interesting challenge. I don’t have a feeling as to whether this is for really fundamental reasons, or people haven’t tried so hard yet.
2 [….] I simply can’t think of any behavior that is at all meaningful from an AG-like perspective where the questions of fan combinatorics and degrees of polynomials are replaced by questions of approximate equality.
There are plenty of cases where “high degree” is enough (Falting’s Theorem is the first thing that comes to mind, but there are lots). But I agree that “degree approximately 5″ feels quite unnatural.
Hi Dmitry,
To me it seems not unreasonable to think that some ideas from tropical geometry might be relevant ML, for the simple reason that the functions coming up in ML with ReLU activation are PL, and people in tropical geometry have thought seriously about PL functions. Of course this does not guarantee that there is anything useful to be said!
One possible example that comes to mind in the context of your post here is the concept of polyhedral currents. As I understand it, here the notion of “density of polygons’ is used as a kind of proxy for the derivative of a PL function? But I think the theory of polyhedral currents gives a much more general theory of differentiation of PL functions. Very naively, rather than just recording the locus where the function fails to be linear, one also records how much the derivative changes when crossing the walls. I learnt about this from a paper of Mihatsch: https://arxiv.org/pdf/2107.12067 but I’m certain there are older references.
I’m really a log person, I don’t know the tropical world very well; sorry if what I write does not make sense!
Get that agreement in writing.
I’m not sure that would be particularly reassuring to me (writing as one of the contributors). First, how would one check that the agreement had been adhered to (maybe it’s possible, I don’t know)? Second, people in my experience often don’t notice they are training on data (as mentioned in a post above by ozziegooen).
I think this is a key point. Even the best possible curriculum, if it has to work for all students at the same rate, is not going to work well. What I really want (both for my past-self as a student, and my present self as a teacher of university mathematics) is to be able to tailor the learning rate to individual students and individual topics (for student me, this would have meant ‘go very fast for geometry and rather slowly for combinatorics’). And while we’re at it, can we also customise the learning styles (some students like to read, some like to sit in class, some to work in groups, etc)?
This is technologically more feasible than it was a decade ago, but seems far from common.
Thanks Charlie.
Just to be double-sure, the second process was choosing the weight in a ball (so total L2 norm of weights was ⇐ 1), rather than on a sphere (total norm == 1), right?
Yes, exactly (though for some constant , which may not be , but turn out not to matter).
Is initializing weights that way actually a thing people do?
Not sure (I would like to know). But what I had in mind was initialising a network with small weights, then doing a random walk (‘undirected SGD’), and then looking at the resulting distribution. Of course this will be more complicated than the distributions I use above, but I think the shape may depend quite a bit on the details of the SGD. For example, I suspect that the result of something like adaptive gradient descent may tend towards more spherical distributions, but I haven’t thought about this carefully.
If training large neural networks only moves the parameters a small distance (citation needed), do you still think there’s something interesting to say about the effect of training in this lens of looking at the density of nonlinearities?
I hope so! I would want to understand what norm the movements are ‘small’ in (L2, L, …).
LayerNorm looks interesting, I’ll take a look.
Hi alkjash,
I’ve been trying to think more about what might help mathematicians make more progress on problems related to AI safety. AI labs (among others) publish and commission lists of open problems as benchmarks, which gather significant attention and work. What if they produced similar lists of safety-related mathematical challenges? Either in the form of a benchmark, or just a list of problems that they find important?