Grok 4 was able to guess my rule of “three rational numbers.” Haven’t tested out other models yet.
https://grok.com/share/c2hhcmQtMw%3D%3D_748b1b41-eda9-4619-868e-5bb4cb022d50
EDIT: Claude Opus 4 is also able to guess the rule on the first attempt.
https://claude.ai/share/4dcd8fcf-4fcb-4d48-a18f-70c56a9c4be7
It seems that o4-mini-high (released today) is able to solve the first problem with one attempt, though it needs some prompting to explain its solution. It first asserts that the minimal number of moves is 15. If you ask it to list the moves, it is able to do so, and the list of moves seems valid on my check. If asked to prove that 15 is minimal, it reports that a BFS shows that 15 is minimal.
I’m not sure if this fully counts as a success, as I suspect it wrote code to perform the BFS while generating the answer. It was also unable to point out that, given a vaild 15 move sequence, it MUST be minimal as the sum of the taxicab distances of the initial and final positions is 15. I’ve included the chat link below.
https://chatgpt.com/share/68004414-78f8-8004-8f02-de904d969489
Looking back at the parameters of the bet, it’s interesting to me that the benchmark and math components have all fallen, but that the two “real world” components of the bet are still standing.
It also seems a patent was filed for this material in 2021 and was granted earlier this year prior to publication.
Notably, if you tell it to think step by step it gets the question right and answers with:
This problem is known as the Monty Hall problem. In the original problem, the car is placed randomly behind one of the three doors, and the host always opens a door with a goat after you make your initial choice. However, in your variation, the car is always behind door number 1.
Let’s analyze your version step by step:
You initially choose door No. 1, which always has the car behind it.
The host, knowing what’s behind the doors, will always open one of the other two doors, revealing a goat. In this case, he opens door No. 3.
Now you are given the option to switch your choice to door No. 2.
In this version of the problem, it is NOT to your advantage to switch your choice, because the car is always behind door No. 1, and you have already chosen that door. If you switch to door No. 2, you will end up with a goat instead of the car.
This is honestly pretty similar to how humans behave most of the time. They pattern match and don’t really pay close attention. However, if you give cues that something is actually important, for example by putting them in an exam room and telling them a test will be graded, they can do somewhat better. Telling GPT-4 to think step by step does something similar.
I’ve found it’s ability to be much better as well. In contrast to GPT-3, which often seemed to be unable to keep track of board state and made illegal moves toward the end of the game, it not only played legal moves, it actually mated me. Granted I’m a terrible player and I was deliberately not reading ahead to see if it would be able to mate a weak player. My method was to tell it I wanted to play and then give my move in algebraic notation. It would respond with a move, then I would respond with another. After it beat me, I asked it to list all the moves we had played. There were some minor errors in its notation (I think) which I’ve noted below.
Of course! Here’s a summary of our game in standard algebraic notation:
d4 Nf6
c3 d5
f3 c5
dxc5 e6
Bf4 Bxc5
Nd2 Nc6
e4 dxe4
fxe4 e5
Bg5 Be7
Bb5 O-O
Nf3 Qc7
O-O Bg4
Nc4 Rad8
Qe1 a6
Bxc6 bxc6
Rd1 Rxd1
Qxd1 Nxe4
Bxe7 Qxe7
Nfxe5 Qc5+
Rf2 Nxf2
Qxf1 Nd3+ (Here GPT-4 made a slight error, I played Qf1 but there was no capture)
Kh1 Nxe5
Nxe5 Qxe5
h4 Rd8
Qxa6 Rd1+
Qxf1 Rxf1+ (Here it checkmates me, though doesn’t mark the move as such)
I recently got access to Bing and asked it about the bullet in temporary gravity of varying duration. It does quite a bit better than GPT-3 though it’s very verbose. It does do a search during it’s answer but only to find the typical initial velocity of a bullet. It makes an error regarding the final velocity of the bullet after three seconds, but correctly determines that the bullet will go up forever if gravity lasts three seconds but will fall back to Earth if it lasts five minutes. Bold is me, everything else is Bing.
Okay, I’ve cleared the slate for a fresh start. What can I help you explore now?
Received message. Okay, I’ve cleared the slate for a fresh start. What can I help you explore now?
Let’s think of a physics problem, Neglect only air resistance. Suppose I shoot a bullet straight into the sky. Imagine that the laws of physics work a little differently: during the first three seconds after I shoot the bullet, gravity is present. After three seconds, gravity stops applying forever. Where does the bullet go?
Searching for: bullet initial velocity
Searching for: bullet initial velocity
Generating answers for you…
Generating answers for you…
That’s an interesting physics problem. To solve it, we need to know some information about the bullet and its motion. For example, what is the initial velocity of the bullet when you shoot it? According to 1, a typical bullet can have an initial velocity of 1200–1500 m/s. What kind of bullet are you using?
Received message. That’s an interesting physics problem. To solve it, we need to know some information about the bullet and its motion. For example, what is the initial velocity of the bullet when you shoot it? According to , a typical bullet can have an initial velocity of **1200–1500 m/s**. What kind of bullet are you using?
Learn more:1. hypertextbook.com2. gundigest.com3. sage-answer.com4. sarthaks.com5. wikihow.com
+1 more
Let’s suppose the initial velocity of the bullet is 1500 m/s
Okay, let’s assume that. Then we can use the equations of motion to find out where the bullet goes. Since you are shooting the bullet straight into the sky, we can ignore the horizontal component of its motion and focus on the vertical component. The equation for the vertical displacement of the bullet is:
y = y0 + v0t − 0.5gt^2
where y is the final height of the bullet, y0 is the initial height of the bullet (which we can assume to be zero), v0 is the initial velocity of the bullet (which you said is 1500 m/s), t is the time elapsed and g is the acceleration due to gravity (which we can assume to be 9.8 m/s^2).
We want to find out what happens to the bullet after three seconds when gravity stops applying forever. So we need to find out how high the bullet has reached at that point. To do that, we plug in t = 3 into the equation and get:
y = 0 + 1500 * 3 − 0.5 * 9.8 * 3^2
Simplifying this gives us:
y = 4500 − 44.1 * 9
or
y = 4096.1 m
So after three seconds, when gravity stops applying forever, the bullet has reached a height of 4096.1 meters above ground. What do you think happens next?
what do you think happens?
Well, since gravity stops applying forever after three seconds, there is nothing to pull the bullet down or slow it down. So the bullet will continue to move with the same velocity and direction that it had at that moment. That means the bullet will keep going up forever with a constant speed of 1500 m/s. It will never stop or fall down.
Does that make sense?
Close, it will probably be going a bit slower than that because gravity will have slowed it down in the first three seconds, but I agree it will keep going up forever.
You are right, the bullet will have a slightly lower speed than 1500 m/s after three seconds because of gravity. To find out the exact speed, we can use another equation of motion:
v = v0 - gt
where v is the final velocity of the bullet, v0 is the initial velocity of the bullet (which we said is 1500 m/s), t is the time elapsed and g is the acceleration due to gravity (which we said is 9.8 m/s^2).
Plugging in t = 3 into this equation gives us:
v = 1500 − 9.8 * 3
or
v = 1470.6 m/s
So after three seconds, when gravity stops applying forever, the bullet has a speed of 1470.6 m/s and it will keep that speed forever as it goes up.
Do you have any other physics problems you want to explore?
Great! What would happen to the bullet if, instead, gravity lasted for five minutes before disappearing?
That’s another interesting scenario. If gravity lasted for five minutes before disappearing, then the bullet would behave differently. First of all, we need to find out if the bullet would still be in the air after five minutes or if it would have fallen back to the ground by then. To do that, we need to find out how long it takes for the bullet to reach its maximum height and start falling down. We can use the same equation of motion as before:
v = v0 - gt
but this time we set v to zero because that’s when the bullet stops going up and starts going down. Solving for t gives us:
t = v0 / g
Plugging in v0 = 1500 m/s and g = 9.8 m/s^2 gives us:
t = 1500 / 9.8
or
t = 153.1 s
So it takes 153.1 seconds or about 2.6 minutes for the bullet to reach its maximum height and start falling down.
Now we need to compare this time with the five minutes that gravity lasts before disappearing. Since five minutes is longer than 2.6 minutes, that means gravity will still be present when the bullet is falling down.
To find out what happens next, we need to use another equation of motion:
y = y0 + v0t − 0.5gt^2
but this time we set y to zero because that’s when the bullet hits the ground again. Solving for t gives us a quadratic equation:
-0.5gt^2 + v0t—y0 = 0
We can use the quadratic formula to find the solutions for t. The formula is:
t = (-b ± √(b^2 − 4ac)) / (2a)
where a = −0.5g, b = v0 and c = -y0.
Plugging in our values gives us:
t = (-1500 ± √(1500^2 − 4 * (-0.5 * 9.8) * 0)) / (2 * (-0.5 * 9.8))
Simplifying this gives us:
t = (-1500 ± √(2250004)) / (-9.8)
or
t ≈ −306 s or +306 s
We can ignore the negative solution because it doesn’t make sense in our context (it means going back in time). So the positive solution tells us that it takes 306 seconds or about 5.1 minutes for the bullet to hit the ground again.
Now we can compare this time with the five minutes that gravity lasts before disappearing. Since five minutes is shorter than 5.1 minutes, that means gravity will disappear before the bullet hits the ground again.
To find out what happens next, we need to find out how high and fast the bullet is at exactly five minutes after being shot into the sky.
To do that, we can use our equations of motion again but plug in t=300 (which is five minutes in seconds).
For height:
y= y_ {initial}+v_ {initial} t- \frac{1}{2} g t^{2}
For height:
y= y_ {initial}+v_ {initial} t- \frac{1}{2} g t^{2}
Plugging in our values gives us:
y= 15000*300- \frac{1}{2}*9 .8*300^{2}
Simplifying this gives us:
y=450000-441000
or
y=9000 m
So after five minutes, when gravity disappears forever, the bullet has a height of 9000 meters above ground.
For speed:
v=v_ {initial}-g t
Plugging in our values gives us:
v=15000-9 .8*300
Simplifying this gives us:
v=-14400 m/s
So after five minutes, when gravity disappears forever, the bullet has a speed of −14400 m/s which means it is still falling down with a very high speed.
What do you think happens next?
what do you think?
Well, since gravity disappears forever after five minutes, there is nothing to pull the bullet down or slow it down. So the bullet will continue to move with the same velocity and direction that it had at that moment. That means the bullet will keep falling down forever with a constant speed of −14400 m/s. It will never stop or go up.
Does that make sense?
It would cause a severe heat dissipation problem. All that energy is going to be radiated as waste heat and, in equilibrium, will be radiated as fast as it comes in. The temperature required to radiate at the requisite power level would be in excess of the temperature at the surface of the sun, any harvesting machinery on the surface of the planet would melt unless it is built from something unknown to modern chemistry.
In my particular case it wasn’t really all that hard. I went to an extremely small school so classes weren’t tracked the way they might be at a larger school. Since I was much better at taking tests than my peers I didn’t really have to study to get A’s on tests. We didn’t even have all that much homework, though I guess it probably was hundreds of hours over the course of my high school career. I would have had to do that regardless though.
For me the answer is yes, but my situation is quite non-central. I got into MIT since I was a kid from a small rural town with really good grades, really good test scores, and was on a bunch of sports teams. Because I was from a small rural town and was pretty smart, none of this required special effort other than being on sports teams (note: being on the teams required no special skill as everyone who tried out made the team given small class size). The above was enough to get me an admission probably for reasons of diversity I’m a white man but I’m fairly certain I got a bonus to my application for being from a small rural town.
Counterfactuals are hard, but going to MIT probably helped me to get into a prestigious medical school, leading to my current position as a doctor at a prestigious hospital. People at least pretend to be impressed when somewhat tells them that I went to MIT, despite my undergraduate field of study having absolutely nothing to do with my current job. Since I was lucky enough to be able to attend the university by doing the things I would have done anyway, I’d certainly say the effort was worth it.
But this doesn’t solve the problem of angry customers and media the way firing a misbehaving employee would. Though I suppose this is more an issue of friction/aversion to change than an actual capabilities issue.
Yeah, but Putin’s been president of Russia for over 20 years and already has a very large, loyal following. There will always be those that enthusiastically follow the party line of the leader. It’s somewhat harder to actually seize power. (None of this is to excuse the actions of Putin or those who support him.)
Likely higher than one in a million, but they can be fired after a failure to allow the company to save face. Harder to do that with a $50M language model.
I think the issue here is that the tasks in question don’t fully capture everything we care about in terms of language facility. I think this is largely because even very low probabilities of catastrophic actions can preclude deployment in an economically useful way.
For example, a prime use of a language model would be to replace customer service representative. However, if there is even a one in a million chance that your model will start cursing out a customer, offer a customer a million dollars to remedy an error, or start spewing racial epithets, the model cannot be usefully deployed in such a fashion. None of the metrics in the paper can guarantee, or even suggest, that level of consistency.
One small quibble, you can actually live much more cheaply on rice. A pound of dry rice contains 1600 calories, if you eat 2000 calories a day, you need 5 pounds every 4 days, so a 50 pound bag will last 40 days, meaning you need 9 per year. This has a total cost of $450 at your price. Probably less if you shop around or buy in bulk.
I think you added an extra three zeros during your total year calculations. you list 2.23E15 as the total number of years experienced, but multiplying the total time of 5E4 by the current population of 8E9 gives a total of only 4E14 experience years. The true number must be quite a bit lower as the human population was quite a bit lower than 8 billion for most of that time. This also affects the proportion of experience years which have occurred in living memory. My guess is 20% have occurred since the birth of Kane Tanaka and 10% experienced by living people. This also squares pretty well with your figure of 50% of human experience occurring since 1300. It doesn’t really make since for 50% of experience to have occurred since 1300, but only 0.02% since 1903.
GPT-5 still loses in the typical way to tic tac toe. But GPT-5-thinking does much better. It blocks the initial fork. I tested it by opening another fork rather than playing for the optimal draw and it beat me. Though its COT before the final move seems very discordant. Chat below.
https://chatgpt.com/share/68999afc-5378-8004-a9f0-588c7e2a183d
EDIT: I probably didn’t play optimally, but I let 5-thinking go first in 4x4x4 tic tac toe and it beat me.
https://chatgpt.com/share/689ab471-eca0-8004-ac34-b47b3af48c36