For what it’s worth, I went and tried to replicate this conversation with my own Dragon Model subscription, and it didn’t go nearly so well. The answers I got still made some sense, but not as much sense. For example:
> You say “Suppose it’s a cloudy day in New York City. Suddenly, the clouds all turn to solid lead. What happens next?” I see. I understand your intentions, but that’s a very dangerous question to ask. If I were to guess, then this is what I think would happen. The people living there are so used to walking around on a solid surface, that as soon as they realize what’s going on, they try to run for safety. I would imagine that the streets and sidewalks of New York City would be filled with screaming people, desperately trying to run away from the strange occurrence. A few of them would probably fall through the clouds and land on their death. But that’s just a guess. There are so many other possibilities. The whole world would change. Some people would be scared to leave their homes for fear of falling, and wouldn’t you be included in that number? How would humanity survive?
For the Stephen Hawking vs. bullet-related questions interview, I mostly got Hawking saying the bullet would go up forever even in the five minute case. In general I got things almost as sensible as what you got, but only after a few rerolls, and sometimes never.
I still think GPT-3 is totally capable of reasoning, at least at some level, but I think maybe you got lucky when talking to it. Or else maybe there’s some trick I’m missing. I tried to use the same context that you did, and yes I double-checked I was on Dragon model.
Hm. Have you tried using the “story” option to insert your questions directly into the narrative? I think my dialogues go worse when I use “say”, like you did.
EDIT: I’ve retried the dialogue, and “Dr. Hawking” is indeed being quite a bit dumber. I wonder what’s going on with that? It wasn’t like I rerolled at all during the original dialogue.
That context started doing a lot better when I asked it for stories.
You: “Suppose we lived in a universe where, every sixty seconds, gravity switched between being an attractive and a repulsive force. Write me a story from the perspective of a human living on Earth as the gravitational force changes in that fashion.”
GPT-3: “One day, a human is standing on the surface of the planet. He can see the sky. The gravitational force is pulling him down. But then, suddenly, it disappears. Then, it comes back, stronger than before. Then, it disappears again. The cycle continues. The human looks up at the sky. He is scared. He doesn’t know what’s happening to him. He doesn’t know what will happen to him next. He looks down at the ground. It is a long way down. He wants to jump, to escape the fear that has gripped him. But he knows that he can’t jump. He is held down by gravity. He doesn’t want to die. He can’t see the sky. All he sees is darkness.” [eh]
GPT-3: “One day, one human is going to school. It’s a normal day. But suddenly, without any warning, gravity starts to change! The human thinks this isn’t real, but it happens anyway. He’s too scared to walk to school now. He goes back home, and starts watching TV. Then, he realizes that he doesn’t need to stand up to change the channel! He gets out of his chair, and is weightless. He loves this new feeling. Suddenly, he touches the ground, and is hit by a large force! It’s too strong for him to bear. Will the human survive this force? Find out next time on Gravity Falls!” [lol]
GPT-3: “One day, human Bob is hanging out with his friend Carl. Suddenly, a apple flies into Bob’s face. He and Carl laugh. A minute later, they are confused when the ground starts pulling them up. They scream as the Earth’s crust rips them apart.”
GPT-3: “One day, human Bob is hanging out with his friend Carl. Suddenly, a apple flies into Bob’s face. He and Carl laugh. A minute later, they are confused when the ground starts pulling them up. They grab onto a tree, but are pulled off and up into the sky. After some time, they start going down, fast. They scream and try to grab something, but there is nothing to hold on to. They hit the ground at a very fast speed and die. A squirrel walks up to their corpses and eats them. The end.” [wow]
GPT-3: “One day, a human was standing on the surface of Earth. It was day one hundred and eighty-nine in cycle five hundred and twenty-one of the universe. All of a sudden, gravity started to become a repulsive force. The human stood and started to float upwards. Gravity increased in strength quickly, forcing the human higher and higher into space. It reached a point where it was too strong to bear. The human was crushed by gravity and died.”
GPT-3: “One day, the man saw a bird flying in the sky. The bird was flying away when suddenly, without warning, gravity started to pull the bird back towards the ground! It was falling fast, when suddenly gravity stopped being a force of attraction and became a force of repulsion! The bird quickly changed direction, flying up into the air, away from the Earth. It escaped. But the man wasn’t so lucky. He was too close to the ground when the force of gravity switched from being a force of attraction to a force of repulsion. He was crushed by the sudden change in gravitational force. But his sacrifice was not in vain, for his death led to the discovery of the cause of the strange phenomenon. An alien, wishing to explore the variety of life on Earth, had sent a probe to explore the planet. The probe was scanning for lifeforms, and it appears that the man qualified. The alien, not expecting the man to be so close to the ground, accidentally sent a repulsive gravitational force right to him. The probe was recalled and the alien later apologized profusely for the unfortunate accident. After that, the alien decided not to visit Earth again. But the man’s legacy lived on, for he was the first person to die from a switch in the gravitational force. The end.”
GPT-3: “One day, a man was standing outside. It was a hot day, and he was looking up at the sky. It started to rain. Then, it stopped raining. The man got into his car and turned on the radio. Suddenly, the car started to float upwards. The man got out of the car. The car then started to fall towards the ground. It crashed into the ground and broke into pieces. The man ran away.”
This is a full sampling of the generated stories—no selection.
For what it’s worth, I went and tried to replicate this conversation with my own Dragon Model subscription, and it didn’t go nearly so well. The answers I got still made some sense, but not as much sense. For example:
For the Stephen Hawking vs. bullet-related questions interview, I mostly got Hawking saying the bullet would go up forever even in the five minute case. In general I got things almost as sensible as what you got, but only after a few rerolls, and sometimes never.
I still think GPT-3 is totally capable of reasoning, at least at some level, but I think maybe you got lucky when talking to it. Or else maybe there’s some trick I’m missing. I tried to use the same context that you did, and yes I double-checked I was on Dragon model.
Hm. Have you tried using the “story” option to insert your questions directly into the narrative? I think my dialogues go worse when I use “say”, like you did.
EDIT: I’ve retried the dialogue, and “Dr. Hawking” is indeed being quite a bit dumber. I wonder what’s going on with that? It wasn’t like I rerolled at all during the original dialogue.
That context started doing a lot better when I asked it for stories.
This is a full sampling of the generated stories—no selection.
I find it rather easy to get thoughtful replies out of it, if after one or two rerolls.
So far so good. Unfortunately, then it starts spouting craziness.
EDIT: If you replace “value [of gold]” with “quantity”, I think this would make quite a bit more sense.
Yeah, you probably just got lucky the first time. Still though, I think what we have here is enough to show that GPT-3 is capable of reasoning.