I am a longtime LessWrong and SSC reader who finally got around to starting a blog. I would love to hear feedback from you! https://harsimony.wordpress.com/
harsimony
Cool project! I was thinking along similar lines when I recently went through a bunch of his posts to find interesting ideas and collected them:
https://harsimony.wordpress.com/2021/10/20/robin-hansons-other-ideas/
Whoever decides to work on this project up might be able to use this as a starting point.
Some other order-of-magnitude estimates on available data, assuming words roughly equal tokens:
Wikipedia: 4B English words, according to this page.
Library of Congress: from this footnote a assume there are at most 100 million books worth of text in the LoC and from this page assume that books are 100k words, giving 10T words at most.
Constant writing: I estimate that a typical person writes at most 1000 words per day, with maybe 100 million people writing this amount of English on the internet. Over the last 10 years, these writers would have produced 370T words.
Research papers: this page estimates ~4m papers are published each year, at 10k words per paper with 100 years of research this amounts to 4T words total.
So it looks like 10T words is an optimistic order-of-magnitude estimate of the total amount of data available.
I assume the importance of a large quantity of clean text data will lead to the construction of a text database of ~1T tokens and that this database (or models trained on it) will eventually be open-sourced.
From there, it seems like really digging in to the sources of irreducible error will be necessary for further scaling. I would guess that a small part of this is “method error” (training details, context window, etc.) but that a significant fraction comes from intrinsic text entropy. Some entropy has to be present, or else text would have no information value.
I would guess that this irreducible error can probably be broken down into:
-
Uncertainty about the specific type of text the model is trying to predict (e.g. it needs some data to figure out that it’s supposed to write in modern English, then more data to learn that the writing is flamboyant/emotional, then more to learn that there is a narrative structure, then more to determine that it is a work of fiction etc.). The model will always need some data to specify which text-generating sub-model to use. This error can be reduced with better prompts (though not completely eliminated)
-
Uncertainty about location within the text. For example, even if the model had memorized a specific play by Shakespeare, if you asked it to do next-word prediction on a random paragraph from the text, it would have trouble predicting the first few words simply because it hasn’t determined which paragraph it has been given. This error should go away when looking at next-word prediction after the model has been fed enough data. Better prompts and a larger context window should help.
-
Uncertainty inherent to the text. This related to the actual information content of the text, and should be irreducible. I’m not sure about the relative size of this uncertainty compared to the other ones, but this paper suggests an entropy of ~10 bits/word in English (which seems high?). I don’t know how entropy translates into training loss for these models. Memorization of key facts (or database access) can reduce the average information content of a text.
EDIT: also note that going from 10T to 100T tokens would only reduce the loss by 0.045, so it may not be worthwhile to increase dataset size beyond the 10T order-of-magnitude.
-
Great post!!
I think the section “Perhaps we don’t want AGI” is the best argument against these extrapolations holding in the near-future. I think data limitations, practical benefits of small models, and profit-following will lead to small/specialized models in the near future.
https://www.lesswrong.com/posts/8e3676AovRbGHLi27/why-i-m-optimistic-about-near-term-ai-risk
Could someone help me collect the relevant literature here?
I think the complete class theorems are relevant: https://www.lesswrong.com/posts/sZuw6SGfmZHvcAAEP/complete-class-consequentialist-foundations
The Non-Existence of Representative Agents: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3302656
Representative Agents: https://en.wikipedia.org/wiki/Representative_agent
John Wentworth on Subagents: https://www.lesswrong.com/posts/3xF66BNSC5caZuKyC/why-subagents
Is it known whether predictive coding is easier to train than backprop? Local learning rules seem like they would be more parallelizable.
Every Model Learned by Gradient Descent Is Approximately a Kernel Machine seems relevant to this discussion:
We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel.
I wrote about the literature around the evolution of virulence previously which people might find useful for finding references on the topic.
In short, it is complicated:
-
zoonotic diseases (like COVID) don’t follow a consistent pattern early in their evolution because they are so far removed from being adapted to humans as a niche.
-
As long as there is a correlation between deadly-ness and ability to reproduce, diseases will face evolutionary pressure to become more/less deadly depending on the circumstances. This is strongly influenced by transmission mode (but isn’t the only factor):
a. Disease transmitted by close person-to-person contact → disease can’t make host too sick, or host won’t come into contact with others → disease evolves to cause milder infection.
Example: HIV has relatively few effects on hosts for years before culminating in AIDS.
b. Disease transmission doesn’t require person-to-person contact → disease can optimize other features of transmission such as producing many copies and using host resources → disease evolves to cause severe infection.
Example: malaria is spread via mosquitos and doesn’t require person-to-person contact in order to spread, which might explain why it is such a debilitating disease.
-
Do you have a good system for saving and prioritizing the ideas you have?
Building a habit of noticing a new idea and writing it down (even if it’s silly) has increased my overall output dramatically.
I personally don’t expect very high efficacy, and I do expect that Loyal will sell the drug for the next 4.5 years. However, as long as Loyal is clear about the nature of the approval of the drug, I think this is basically fine. People should be allowed to, at their own expense, give their pets experimental treatments that won’t hurt them and might help them. They should also be able to do the same for themselves, but that’s a fight for another day.
Agreed! Beyond potentially developing a drug, think Loyal’s strategy has the potential to change regulations around longevity drugs, raise profits for new trials, and bring attention/capital to the longevity space. I don’t see many downside risks here unless the drug turns out to be unsafe.
Great idea, thanks for posting this!
I wrote a post on how to have productive disagreements with loved ones:
https://harsimony.wordpress.com/2022/06/21/winning-arguments-with-loved-ones/
Here is the subsection on analyzing disagreements:
Because arguments are emotional, it can be helpful to try to dispassionately assess the situation with your partner and get to the root of the problem.
The first step is to break the disagreement down into isolated chunks. Identify the handful of differences you are having, and deal with them as independently as possible. If you notice discussion of one problem bleeding into another, try to refocus on the subject at hand.
For each problem, try to set aside your ego and emotions; what’s the real issue? Ask fair questions of your partner and hear them out. At the end of your discussion, you should be able to articulate your partner’s point in a way they would find satisfactory. A surprising number of arguments can be resolved simply by giving both partners a fair hearing.
It can be helpful to try to come up with neutral examples related to your problem. Flip the roles that you and your partner play in the example and be honest about how it would change things. It also can be helpful to consider things from a third-person perspective [4].
At the end of all of this, you should come to an agreement about the things you will do differently in the future. Once again, it doesn’t have to be perfect, it just has to be something you can build upon.
Two arguments I would add:
Conflict has direct costs/risks, a fight between AI and humanity would make both materially worse off
Because of comparative advantage, cooperation between AI and humanity can produce gains for both groups. Cooperation can be a Pareto improvement.
Alignment applies to everyone, and we should be willing to make a symmetric commitment to a superintelligence. We should grant them rights, commit to their preservation, respect it’s preferences, be generally cooperative and avoid using threats, among other things.
It may make sense to commit to a counterfactual contract that we expect an AI to agree to (conditional on being created) and then intentionally (carefully) create the AI.
Thanks for posting this, I like reading lists of people’s ideas.
Posts I’m most interested in reading:
Aluminum fuel cells
Re: nation-states House-elves
Nuclear strategy
Contemporary Nuclear Strategy
Corona Discharge. Million volt? I love the smell of Ozone in the morning.
A few fundamentally different AI architectures. Consequentialist. Stateless. Complexity-minimizing.
Natural Amphetamines and Algernon’s Law
Non-hostile superintelligence. Gears level AGI.
How a superintelligence can easily take over the world. No need for nanotech and von Neumann machines.
Laser power transfer, near infrared
Premises for realistic hard scifi. Fire control systems. AI more than robotics. No FTL. Use wheels. No hovercars. No artifical gravity.
What could an AGI invent that would change the world? Broadest impact (or benefit) for humanity.
Anything on prediction markets
Task: Identify How to Divide Up a Research Task Among Different Experts
Context: Many important projects are multi-disciplinary. Research can be accelerated by dividing tasks between individuals who specialize in different areas. However, it can be difficult to divide a project cleanly into sub-components and decide who is best suited to each part.
Input type: A problem statement, a list of people on a research team, and their skills.
Output type: A the list of team members each with an associated sub-task relevant to the overall goal.
Instance 1
Input:
Problem: Research how policy advocacy can improve the welfare of egg-laying hens
Team:
Alice, a legislative assistant in congress
Bob, a historian knowledgeable about the history of civil rights movements
Charlie, a poultry veterinarian
Output:
Alice: Research previous animal welfare legislation to create a blueprint for legislation which can pass in congress.
Bob: Identify the basis of a successful grassroots advocacy for animal welfare and determine what legislation would be popular with voters.
Charlie: Determine how conditions for egg-laying hens could most be improved.
Instance 2
Input:
Problem: Create a scalable technology to reduce surface transmission of pathogens.
Team:
Dina, an epidemiologist
Evan, a chemist
Fran, a biologist
Output:
Dina: Identify which surfaces contribute most to surface transmission of diseases.
Evan: Produce a set of antimicrobial compounds which can be added to a surface
Fran: Test the set of antimicrobial compounds produced by Evan for safety and effectiveness.
Thanks for the clarification! Do you know if either condition is associated with abnormal levels of IGF-1 or other growth hormones?
I agree that the difference between disease-treating interventions (that happen to extend life) versus longevity interventions is murky.
For example, would young people taking statins to prevent heart disease be a longevity intervention?
https://johnmandrola.substack.com/p/why-i-changed-my-mind-about-preventing
See this post arguing that rapamycin is not a longevity drug:
https://nintil.com/rapamycin-not-aging
Broadly, I’m not too concerned with what we classify a drug as as long as its safe, effective, well-understood, and gets approved by regulatory authorities.
I like this intuition and it would be interesting to formalize the optimal charitable portfolio in a more general sense.
I talked about a toy model of hits-based giving which has a similar property (the funder spends on projects proportional to their expected value rather than on the best projects):
https://ea.greaterwrong.com/posts/eGhhcH6FB2Zw77dTG/a-model-of-hits-based-giving
Updated version here: https://harsimony.wordpress.com/2022/03/24/a-model-of-hits-based-giving/
Yeah I think a lot of it will have to be resolved at a more “local” level.
For example, for people in a star system, it might make more sense to define all land with respect to individual planets (“Bob owns 1 acre on Mars’ north pole”, “Alice owns all of L4″ etc.) and forbid people from owning stationary pieces of space. I don’t have the details of this fleshed out, but it seems like within a star system, its possible to come up with a sensible set of rules and have the edge cases hashed out by local courts.
For the specific problem of predicting planetary orbits, if we can predict 1000 years in the future, it seems like the time-path of land ownership could be updated automatically every 100 years or so, so I don’t expect there to be huge surprises there.
For taxation across star systems, I’m having trouble thinking of a case where there might be ownership ambiguity given how far apart they are. For example, even when the Milky Way and Andromeda galaxies collide, its unlikely that any stars will collide. Once again, this seems like something that can be solved by local agreements where owners of conflicting claims renegotiate them as needed.
Setting aside how important timelines are for strategy, the fact that P(doom) combines several questions together is a good point. Another way to decompose P(doom) is:
-
How likely are we to survive if we do nothing about the risk? Or perhaps: How likely are we to survive if we do alignment research at the current pace?
-
How much can we really reduce the risk with sustained effort? How immutable is the overall risk?
Though people probably mean different things by P(doom) and seems worthwhile to disentangle them.
Talking about our reasoning for our personal estimates of p(doom) is useful if and only if it helps sway some potential safety researchers into working on safety …
Good point, P(doom) also serves a promotional role, in that it illustrates the size of the problem to others and potentially gets more people to work on alignment.
-
You may also want to check out John Wentworth’s natural abstraction hypothesis work:
I mostly agree with points 1 and 2. Many interacting AI’s are important to consider, and I think individual AI’s will have an incentive to multiply. And I agree that people will continue to push the capability frontier beyond what is directly useful, potentially giving AI dangerous abilities.
I think we differ on the timeline of those changes, not whether those changes are possible or important. This is the question I’m trying to highlight.
Longer timelines for dangerous AI don’t mean that we shouldn’t prepare for things going wrong; these problems will still need to be solved eventually. Ideally, some researchers would act as if timelines are really short and “roll the dice” on alignment research that could pan out in a few years.
But I’m arguing that the bulk of the field should feel safe investing in infrastructure, movement building, and projects over a 10-20 year timeframe. People considering pursuing AI safety should assume they will have enough time to make an impact. Finding ways to extend this horizon further into the future is valuable because it gives the field more time to grow.
Note: I’m not affiliated with Loyal or any other longevity organization, I’m going off the same outside information as the author.
I think there’s a substantial chance that this criticism is misguided. A couple points:
The term “efficacy nod” is a little confusing, the FDA term is “reasonable expectation of effectiveness”, which makes more sense to me, it sounds like the drug has enough promise that the FDA thinks its worth continuing testing. They may not have actual effectiveness data yet, just evidence that it’s safe and a reasonable explanation for why it might work.
I don’t know what the standard practices are for releasing trial data, especially for an initial trial like this. Are we sure this isn’t standard practice? Even if it isn’t, I don’t think this is sufficient to assume that Loyal is being disingenuous.
Take the outside view here, both Loyal and the FDA have veterinarians who seem to think that the drug is promising.
I also think there’s a reasonable argument to be made for an IGF-1 inhibitor in large-breed dogs. Large breed dogs often die of heart disease which is often due to dilated cardiomyopathy (heart becomes enlarged and can’t pump blood effectively). This enlargement can come from hypertrophic cardiomyopathy (overgrowth of the heart muscle). I don’t know if it’s known why large breed dogs have hypertrophic cardiomyopathy, but maybe IGF-1 makes the heart muscle grow over a dog’s lifetime which would suggest that an IGF-1 inhibitor is worth trying. It’s also suggestive that diabetes is a risk-factor for cardiomegaly (enlarged heart).
With this in mind, we can answer the next point:
So my theory says that high IGF-1 over a lifetime progressively increases the size of the heart muscle until you get dilated cardiomyopathy. Stopping IGF-1 even in middle age might help. We can falsify this theory by checking if large breed dogs show heart enlargement over their lifetime (instead of growth stopping after puberty like it should). Why would heart muscle keep growing while nothing else does? I’m not sure.
Now we can turn to the question: do large breed dogs actually have elevated IGF-1?
Looking at your first figure, the answer seems to be yes! There’s a straightforward correlation between bodyweight and IGF-1 concentration, the slope would likely be higher without the 3 outliers on the right. Notice also that the sample doesn’t have many large breed dogs (great danes weigh 110-180 lbs). I would guess that those 3 dogs are large breed dogs, and they do in fact have IGF-1 levels higher than most of the dogs in the sample.
Now lets turn to the second plot, we see that IGF-1 concentration decreases with age. Remember that there is survivorship bias at higher ages, large breed dogs with higher IGF-1 will die at around 72 months while chihuahuas will live over 150 months. Declining IGF-1 with age is exactly what we should see if we expected IGF-1 to correlate with longevity! The plot supports the theory that IGF-1 is important for aging, you can’t cherry-pick outliers and ignore the overall relationship in the plot.
I’m no expert, but I think there’s interest in IGF-1 inhibitors for longevity. To quote Sarah Constantin:
I would guess this is one of the reasons Loyal had an interest in IGF-1 inhibitors from the outset.
The dose makes the poison! Every drug has negative effects at a high enough dose, the trials will determine if these actually arise at the dose they are using.
I’m no expert, but this evidence doesn’t seem sufficient to stop research on this drug. Will it prove safe or effective? Will it also benefit human health? I have no idea, but unless we discover that the drug is hurting patients, I think its fine for Loyal to carry on.