It probably doesn’t matter, but I wonder why you used the name “Sam” and then referred to this person as “she”. The name “Sam” is much more common for men than for women. So this kicks the text a bit “out of distribution”, which might affect things. In the worst case, the model might think that “Sam” and “she” refer to different people.
Radford Neal
Karma: 728
There are in fact many universities that have both “research faculty” and “teaching faculty”. Being research faculty has higher prestige, but nowadays it can be the case that teaching faculty have almost the same job security as research faculty. (This is for permanent teaching faculty, sessional instructors have very low job security.)
In my experience, the teaching faculty often do have a greater enthusiasm for teaching than most research faculty, and also often get better student evaluations. I think it’s generally a good idea to have such teaching faculty.
However, my experience has been that there are some attitudinal differences that indicate that letting the teaching faculty have full control of the teaching aspect of the university’s mission isn’t a good idea.
One such is a tendency for teaching faculty to start to see the smooth running of the undergraduate program as an end in itself. Research faculty are more likely to have an ideological commitment to the advancement of knowledge, even if promoting that is not as convenient.
A couple anecdotes (from my being research faculty at a highly-rated university):
At one point, there was a surge in enrollment in CS. Students enrolled in CS programs found it hard to take all the courses they needed, since they were full. This led some teaching faculty to propose that CS courses (after first year) no longer be open to students in any other department, seeing as such students don’t need CS courses to fulfill their degree requirements. Seems logical: students need to smoothly check off degree requirements and graduate. The little matter that knowledge of CS is crucial to cutting-edge research in many important fields like biology and physics seemed less important...
Another time, I somewhat unusually taught an undergrad course a bit outside my area, which I didn’t teach again the next year. I put all the assignments I gave out, with solutions, on my web page. The teaching faculty instructor the next year asked me to take this down, worrying that students might find answers to future assigned questions on my web page. I pointed out that these were all my own original questions, not from the textbook, and asked whether he also wanted the library to remove from circulation all the books on this topic…
Also, some textbooks written by teaching faculty seem more oriented towards moving students through standard material than teaching them what is actually important.
Nevertheless, it is true that many research faculty are not very good at teaching, and often not much interested either. A comment I once got on a course evaluation was “there’s nothing stupid about this course”. I wonder what other experiences this student had had that made that notable!
These ideas weren’t unfamiliar to Hinton. For example, see the following paper on “Holographic Reduced Representations” by a PhD student of his from 1991: https://www.ijcai.org/Proceedings/91-1/Papers/006.pdf
The logic seems to be:
If we do a 1750 year simulation assuming yearly fresh water additions 80 times the current greenland ice melt rate, we see AMOC collapse.
Before this simulated collapse, the value of something that we think could be an indicator changes.
That indicator has already changed.
So collapse of the AMOC is imminent.
Regarding (1), I think one can assume that if there was any way of getting their simulation engine to produce an AMOC collapse in less than 1750 years, they would have showed that. So, to produce any sort of alarming result, they have to admit that their simulation is flawed, so they can say that collapse might in reality occur much sooner. But then, if the simulation is so flawed, why would one think that the simulation’s indicator has any meaning?
They do claim that the indicator isn’t affected by the simulation’s flaws, but without having detailed knowledge to assess this myself, I don’t see any strong reason to believe them. It seems very much like a paper that sets out to show what they want to show.
From the paper:
Under increasing freshwater forcing, we find a gradual decrease (Fig. 1A) in the AMOC strength (see Materials and Methods). Natural variability dominates the AMOC strength in the first 400 years; however, after model year 800, a clear negative trend appears because of the increasing freshwater forcing. Then, after 1750 years of model integration, we find an abrupt AMOC collapse
Given that the current inter-glacial period would be expect to last only on the order of some thousands of years more, this collapse in 1750 years seems a bit academic.
I don’t get it.
Apparently, the idea is that this sort of game tells us something useful about AI safety.
But I don’t get it.
You obviously knew that you were not unleashing a probably-malign superintelligence on the the world by letting Ra out. So how does your letting Ra out in this game say anything about how you would behave if you did think that (at least initially)?
So I don’t get it.
And if this does say something useful about AI safety, why is it against the rules to tell us how Ra won?
I don’t get it.
Interesting. I hadn’t heard of the Child Born on Tuesday Problem. I think it’s actually quite relevant to Sleeping Beauty, but I won’t go into that here...
Both problems (your 1 and 2) aren’t well-defined, however. The problem is that in real life we do not magically acquire knowledge that the world is in some subset of states, with the single exception of the state of our direct sense perceptions. One could decide to assume a uniform distribution over possible ways in which the information we are supposedly given actually arrives by way of sense perceptions, but uniform distributions are rather arbitrary (and will often depend on arbitrary aspects of how the problem is formulated).
Here’s a boys/girls puzzle I came up with to illustrate the issue:
A couple you’ve just met invite you over to dinner, saying “come by around 5pm, and we can talk for a while before our three kids come home from school at 6pm”.
You arrive at the appointed time, and are invited into the house. Walking down the hall, your host points to three closed doors and says, “those are the kids’ bedrooms”. You stumble a bit when passing one of these doors, and accidentally push the door open. There you see a dresser with a jewelry box, and a bed on which a dress has been laid out. “Ah”, you think to yourself, “I see that at least one of their three kids is a girl”.
Your hosts sit you down in the kitchen, and leave you there while they go off to get goodies from the stores in the basement. While they’re away, you notice a letter from the principal of the local school tacked up on the refrigerator. “Dear Parent”, it begins, “Each year at this time, I write to all parents, such as yourself, who have a boy or boys in the school, asking you to volunteer your time to help the boys’ hockey team...” “Umm”, you think, “I see that they have at least one boy as well”.
That, of course, leaves only two possibilities: Either they have two boys and one girl, or two girls and one boy. What are the probabilities of these two possibilities?
The symmetrical summaries of what is learned are intentionally misleading (it’s supposed to be a puzzle, after all). The way in which you learned they have at least one girl is not the same as the way you learned that they have at least one boy. And that matters.
You may think the difference between “the card is an Ace” and “JeffJo says the card is an Ace” is just a quibble. But this is actually a very common source of error.
Consider the infamous “Linda” problem, in which researchers claim that most people are irrational because they think “Linda is a bank teller” is less likely than “Linda is a bank teller and active in the feminist movement”. When you think most people are this blatantly wrong, you maybe need to consider that you might be the one who’s confused...
Actually, there is no answer to the problem as stated. The reason is that the evidence I (who drew the card) have is not “the card is an Ace”, but rather “JeffJo said the card is an Ace”. Even if I believe that JeffJo never lies, this is not enough to produce a probability for the card being the Ace of Spades. I would need to also consider my prior probability that JeffJo would say this conditional on it being the Ace of Space, the Ace of Hearts, the Ace of Diamonds, or the Ace of Clubs. Perhaps I believe the JeffJo would never say the card is an Ace if it is a Space. In that case, the right answer is 0.
However, I agree that a “reward structure” is not required, unless possible rewards are somehow related to my beliefs about what JeffJo might do.
For example, I can assess my probability that the store down the street has ice cream sundaes for sale when I want one, and decide that the probability is 3⁄4. If I then change my mind and decide that I don’t want an ice cream sundae after all, that should not change my probability that one is available.
Except, you know, that’s exactly what I do with Full Non-indexical Conditioning, but you don’t like the answer.
Philosophy is full of issues where lots of people think they’re just doing the “obvious thing”, except these people come to different conclusions.
The wording may be bad, but I think the second interpretation is what is intended. Otherwise the discussion often seen of “How might your beliefs change if after awakening you were told it is Monday?” would make no sense, since your actual first awakening is always on Monday (though you may experience what feels like a first awakening on Tuesday).
I’m not sure what you’re saying here.
Certainly an objective outside observer who is somehow allowed to ask the question, “Has somebody received a green ball?” and receives the answer “yes” has learned nothing, since that was guaranteed to be the case from the beginning. And if this outside observer were somehow allowed to override the participants’ decisions, and wished to act in their interest, this outside observer would enforce that they do not take the bet.
But the problem setup does not include such an outside objective observer with power to override the participants’ decisions. The actual decisions are all made by individual participants. So where do the differing perspectives come from?
Perhaps of relevance (or perhaps not): If an objective outside observer is allowed to ask the question, “Has somebody with blonde hair, six-foot-two-inches tall, with a mole on the left cheek, barefoot, wearing a red shirt and blue jeans, with a ring on their left hand, and a bruise on their right thumb received a green ball?”, which description they know fits exactly one participant, and receives the answer “yes”, the correct action for this outside observer, if they wish to act in the interests of the participants, is to enforce that the bet is taken.
Yes, this is the right view.
In real life we never know for sure that coin tosses are independent and unbiased. If we flip a coin 50 times and get 50 heads, we are not actually surprised at the level of an event with 1 in 2 to the −50 probability (about 1 in 10 to the −15). We are instead surprised at the level of our subjective probability that the coin is grossly biased (for example, it might have a head on both sides), which is likely much greater than that.
But in any case, it is not rare for rare events to occur, for the simple reason that the total probability of a set of mutually-exclusive rare events need not be low. That is the case with 50 coin tosses that we do assume are unbiased and independent. Any given result is very rare, but of course the total probability for all possible results is one. There’s nothing puzzling about this.
Trying to avoid rare events by choosing a restrictive sigma algebra is not a viable approach. In the sigma algebra for 50 coin tosses, we would surely want to include events for “1st toss is a head”, “2nd toss is a head”, …, “50th toss is a head”, which are all not rare, and are the sort of event one might want to refer to in practice. But sigma algebras are closed under complement and intersection, so if these events are in the sigma algebra, then so are all the events like “1st toss is a head, 2nd toss is a tail, 3rd toss is a head, …, 50th toss is a tail”, which all have probability 1 in 20 to the −50.
Well, for starters, I’m not sure that Ape in the coat disagrees with my statements above. The disagreement may lie elsewhere, in some idea that it’s not the probability of the urn with 18 green balls being chosen that is relevant, but something else that I’m not clear on. If so, it would be helpful if Ape in the coat would confirm agreement with my statement above, so we could progress onwards to the actual disagreement.
If Ape in the coat does disagree with my statement above, then I really do think that that is in the same category as people who think the “Twin Paradox” disproves special relativity, or that quantum mechanics can’t possibly be true because it’s too weird. And not in the sense of thinking that these well-established physical theories might break down in some extreme situation not yet tested experimentally—the probability calculation above is of a completely mundane sort entirely analogous to numerous practical applications of probability theory. Denying it is like saying that electrical engineers don’t understand how resistors work, or that civil engineers are wrong about how to calculate stresses in bridges.
But then… Why are you expecting point (2) to follow?
I’m not clear on your step (1). The new taxes would decrease the amount of money people have to spend, but this would be exactly balanced by an increase in money available to spend due to people no longer using their money to buy government bonds. The people with more money to spend may not be the same as the people with less money to spend, so there could be second-order effects if this shifts which goods or services money gets spent on, but how seems hard to predict. There is also the issue that some government bonds are bought by foreigners—but then, foreigners can buy exported goods too...
… priors are not actually necessary when working with Bayesian updates specifically. You can just work entirely with likelihood ratios...
I think that here you’re missing the most important use of priors.
Your prior probabilities for various models may not be too important, partly because it’s very easy to look at the likelihood ratios for models and see what influence those priors have on the final posterior probabilities of the various models.
The much more important, and difficult, issue is what priors to use on parameters within each model.
Almost all models are not going to fix every aspect of reality that could affect what you observe. So there are unknowns within each model. Some unknown parameters may be common to all models; some may be unique to a particular model (making no sense in the context of a different model). For parameters of both types, you need to specify prior distributions in order to be able to compute the probability of the observations given the model, and hence the model likelihood ratios.
Here’s a made-up example (about a subject of which I know nothing, so it may be laughably unrealistic). Suppose you have three models about how US intelligence agencies are trying to influence AI development. M0 is that these agencies are not doing anything to influence AI development. M1 is that they are trying to speed it up. M2 is that they are trying to slow it down. Your observations are about how fast AI development is proceeding at some organizations such as OpenAI and Meta.
For all three models, there are common unknown parameters describing how fast AI progresses at an average organization without intelligence agency intervention, and how much variation there is between organizations in their rate of progress. For M1 and M2, there are also parameters describing how much the agencies can influence progress (eg, via secret subsidies, or covert cyber attacks on AI compute infrastructure), and how much variation there is in the agencies’ ability to influence different organizations.
Suppose you see that AI progress at OpenAI is swift, but progress at Meta is slow. How does that affect the likelihood ratios among M0, M1, and M2?
It depends on your priors for the unknown model parameters. If you think it unlikely that such large variation in progress would happen with no intelligence agency intervention, but that there could easily be large variation in how much these agencies can affect development at different organizations, then you should update to giving higher probability to M1 or M2, and lower probability to M0. If you also thought the slow progress at Meta was normal, you should furthermore update to giving M1 higher probability relative to M2, explaining the fast progress at OpenAI by assistance from the agencies. On the other hand, if you think that large variation in progress at different organizations is likely even without intelligence agency intervention, then your observations don’t tell you much about whether M0, M1, or M2 is true.
Actually, of course, you are uncertain about all these parameters, so you have prior distributions for them rather than definite beliefs, with the likelihoods for M0, M1, and M2 being obtained by integrating over these priors. These likelihoods can be very sensitive to what your priors for these model parameters are, in ways that may not be obvious.
I think these examples may not illustrate what you intend. They seem to me like examples of governments justifying policies based on second-order effects, while actually doing things for their first-order effects.
Taxing addictive substances like tobacco and alcohol makes sense from a government’s perspective precisely because they have low elasticity of demand (ie, the taxes won’t reduce consumption much). A special tax on something that people will readily stop consuming when the price rises won’t raise much money. Also, taxing items with low elasticity of demand is more “economically efficient”, in the technical sense that what is consumed doesn’t change much, with the tax being close to a pure transfer of wealth. (See also gasoline taxes.)
Government spending is often corrupt, sometimes in the legal sense, and more often in the political sense of rewarding supporters for no good policy reason. This corruption is more easily justified when mumbo-jumbo economic beliefs say it’s for the common good.
The first-order effect of mandatory education is that young people are confined to school buildings during the day, not that they learn anything inherently valuable. This seems like it’s the primary intended effect. The idea that government schooling is better for economic growth than whatever non-mandatory activities kids/parents would otherwise choose seems dubious, though of course it’s a good talking point when justifying the policy.
So I guess it depends on what you mean by “people support”. These second-order justifications presumably appeal to some people, or they wouldn’t be worthwhile propaganda. But I’m not convinced that they are the reasons more powerful people support these policies.
If forced to choose between “prices increased a lot, and are still increasing a lot” and “prices increased a lot, but have now stabilized”, the correct answer is the first. A more accurate answer would be “prices increased a lot, and are still increasing, but at a bit slower pace”, but it wasn’t an option.
Good to know that most people in the US are clear on this, even if your president isn’t.
Your post reads a bit strangely.
At first, I thought you were arguing that AGI might be used by some extremists to wipe out most of humanity for some evil and/or stupid reason. Which does seem like a real risk.
Then you went on to point out that someone who thought that was likely might wipe out most of humanity (not including themselves) as a simple survival strategy, since otherwise someone else will wipe them out (along with most other people). As you note, this requires a high level of unconcern for normal moral considerations, which one would think very few people would countenance.
Now comes the strange part… You argue that actually maybe many people would be willing to wipe out most of humanity to save themselves, because… wiping out most of humanity sounds like a pretty good idea!
I’m glad that in the end you seem to still oppose wiping out most of humanity, but I think you have some factual misconceptions about this, and correcting them is a necessary first step to thinking of how to address the problem.
Concerning climate change, you write: “In the absence of any significant technological developments, sober current trajectory predictions seem to me to range from ‘human extinction’ to ‘catastrophic, but survivable’”.
No. Those are not “sober” predictions. They are alarmist claptrap with no scientific basis. You have been lied to. Without getting into details, you might want to contemplate that global temperatures were probably higher than today during the “Holocene Climatic Optimum” around 8000 years ago. That was the time when civilization developed. And temperatures were significantly higher in the previous interglacial, around 120,000 years ago. And the reference point for supposedly-disastrous global warming to come is “pre-industrial” time, which was in the “little ice age”, when low temperatures were causing significant hardship. Now, I know that the standard alarmist response is that it’s the rate of change that matters. But things changed pretty quickly at the end of the last ice age, so this is hardly unprecedented. And you shouldn’t believe the claims made about rates of change in any case—actual science on this question has stagnated for decades, with remarkably little progress being made on reducing the large uncertainty about how much warming CO2 actually causes.
Next, you say that the modern economy is relatively humane “under conditions of growth, which, under current conditions, depends on a growing population and rising consumption. Under stagnant or deflationary conditions it can be expected to become more cutthroat, violent, undemocratic and unjust.”
Certainly, history teaches that a social turn towards violence is quite possible. We haven’t transcended human nature. But the idea that continual growth is needed to keep the economy from deteriorating just has no basis in fact. Capitalist economies can operate perfectly fine without growth. Of course, there’s no guarantee that the economy will be allowed to operate fine. There have been many disastrous economic policies in the past. Again, human nature is still with us, and is complicated. Nobody knows whether social degeneration into poverty and tyranny is more likely with growth or without growth.
Finally, the idea that a world with a small population will be some sort of utopia is also quite disconnected from reality. That wasn’t the way things were historically. And even if it was, it woudn’t be stable, since population will grow if there’s plenty of food, no disease, no violence, etc.
So, I think your first step should be to realize that wiping out most of humanity would not be a good thing. At all. That should make it a lot easier to convince other people not to do it.