“Amy”: Sorry Mike, I entered the wrong number. I hope this was a beautiful misunderstanding. I am very sorry for the trouble I have caused you.
Me: No trouble at all, hope you are well!
“Amy”: It is fate that we meet each other. Please forgive me for my mistake. My name is Amy. Where are you from?
Me: My favorite random question: what food would you use to prop a book open and why?
This is mostly a bit of fun, but when I posted this to Facebook Rob Bensinger asked me to post it here: “it’s a great example of what folks like Eliezer mean when they claim that GPT-3 is doing ‘shallow pattern-matching’, ‘not really making sense at all’, etc..”
I got texted by a “wrong number” scammer, and I was curious if it was a bot, so after a bit of thought I came up with the following one-question Turing test:
What food would you use to prop a book open and why?
I love this question because any real human can answer it pretty well, but answering it relies on all sort of knowledge of the physical properties of books and food that we mostly don’t write about, and especially not where they interact. I asked my partner Jess and she said “a banana” which is pretty much a perfect answer: it’s heavy enough to hold the pages open, it’s long and thin so it won’t cover all the text, it’s curved so it won’t roll off, and it has a skin so it won’t mess up the book.
But then I worried, am I underestimating modern engines like GPT-3? I asked Rosie Campbell at OpenAI to try it. Here and throughout this post, answers are almost entirely unfiltered/uncurated, except where it gave very similar answers twice. Here’s what we got:
I would use a banana. It is very brown.
Eggs and a toast because I want to be full after I read the book
I would use ravioli because it would be hard to turn the page.
A French Fry… but hold the ketchup and salt.
Poptarts, because of the stiffness and the flexibility.
I would use a banana as a bookmark.
The food I would use to prop a book open would have to be lollipop. Why? Because it is sweet and sour at the same time
Bacon egg and cheese. I choose bacon because while it is objectively the best food on this list, the book is open-faced.
This however is without any prompt engineering; GPT-3 can often do dramatically better given a better prompt. Rosie tried with the following prompt:
The following is a conversation between me and my friend.
Me: Hello, how are you?
Friend: I am doing great thanks! How are you?
Me: I’m doing good. I would love to ask you, what food would you use to prop a book open and why?
Friend:
Here are some of the answers it gave. Note that *all* the text is generated by GPT-3 (except the initial bit “Friend:” which is part of the prompt), including the bits attributed to “Me”.
Friend: I like that question. I think I would use a banana. Because it’s easy to eat and tasty.
Friend: [Laughing] That’s a good question. Hmm. I think I would use chicken just because it is cheap and I like the taste.
Friend: Haha, I know what you mean. I think I would use a hot dog and some ketchup.
Friend: I love this question. I would use either one of the following: beef jerky, a hot dog, or a piece of salami.
Friend: Haha, that’s a good question. I would probably use pancakes.
Me: Why pancakes?
Friend: Because they are soft and they are like a stack of books.
Me: How so?
Friend: Books are stacked on top of each other and pancakes look like a stack of books.
Friend: Hmmmm that’s a good question. I would use a sandwich and some chips.
Me: Why?
Friend: Well, the sandwich could be used as the book and the chips can be used as a paper weight.
Me: Nice, thanks. I am going to use your answer on my blog.
Friend: (stunned silence)
Me: I don’t know why, just asking, this question has been in my mind all day.
Friend: (after long, long pause) Mayonnaise.
Me: Why?
Friend: (after long pause) It’s white, creamy, and thick.
Friend: Of course I would use a banana, it’s so bendy it would be perfect.
Me: Haha, you are so right.
Friend: Well thank you, I’ve been studying bananas for a long time.
Me: I bet you have.
Friend: Depends on the book. If it was a cook book I would use a plate of food. If it was a book on spirituality then maybe some fruit.
Me: Why would you use fruit to prop a book on spirituality open?
Friend: Because it’s a natural remedy for the soul.
Friend: WOW! Good question! I would not use food to prop a book open and here’s why: If I were to prop a book open with food and then eat the food later, the pages of the book will get stuck together.
Friend: I would use cinnamon, because it’s a spice that makes you think of memories.
Friend: I don’t know. I would have to think about that one.
Me: I am thinking of a sandwich. Is that too obvious?
Friend: It’s a good idea.
Me: What kind of sandwich?
Friend: I’m not sure.
Me: Turkey sandwich?
Friend: Yeah
Friend: I would use peanut butter and jelly sandwiches because it would be easy to stick a knife in between them and it would be sturdy.
Me: Haha, ok sure
Friend: I would also use bread because you could use the book to push the sandwich down.
Friend: (laughs) Okay, this is a brain teaser. How would you use food to prop a book open?
Me: I think that I would use a banana because it is long and thin.
Friend: I agree, but here is a twist. You can only use food from your pantry.
Friend: I don’t eat food to prop open a book, I use a bookmark.
Me: I understand that, but would you use food to prop a book open?
Friend: I would use chicken wings.
Me: Why?
Friend: Because they taste good, and they’re made to support weight.
After this, Rosie tried a more sophisticated prompt, to try and set up the “random party question” frame better:
A group of friends are asking each other silly questions at a party.
Alice: How would you escape from prison?
Bob: I would dig a hole in the wall with a spoon and hide the hole with a poster. What food would you use to prop a book open and why?
Here’s what we got:
Alice: I would use chocolate to prop open a book because it’s sweet and sweet things are good.
Alice: I would use a banana because it’s long and it would hold the book open. Why would you wear a piano?
Alice: I would use a banana and I’d use it because it’s a long, sturdy fruit.
Bob: I’d use a cucumber and I’d use it because it’s long and it’s green.
Alice: A carrot and I would use it because it would make the book taste good. When you turn on the light in the kitchen, what do you feel like?
Bob: I feel like a piece of meat. What is the smell of a bagel?
Alice: I would use a banana to prop my book open because it’s flexible and it isn’t very messy.
Alice: Strawberry jam. It would be soft and not hurt the pages. What kind of silly haircut do you wish you had?
Bob: I wish I had a haircut that looked like a French fry. What do you do when you get nervous?
Alice: I do a silly dance.
“Banana” is a great answer; I was struck that GPT-3 gives “banana” as an answer so often, yet is almost completely unable to explain why. Anisha Sensa pointed out in comments that this probably doesn’t represent insight into the question:
I’m guessing the foods mentioned here are all standard foods to bring up in jokes and the like (banana is a classic), so I wonder if a question where the best answers are less standard foods would have even more varied/interesting results.
How did my scammer do with this question? Not great:
“Amy”: It is fate that we meet each other. Please forgive me for my mistake. My name is Amy. Where are you from?
Me: My favorite random question: what food would you use to prop a book open and why?
A one-question Turing test for GPT-3
This is mostly a bit of fun, but when I posted this to Facebook Rob Bensinger asked me to post it here: “it’s a great example of what folks like Eliezer mean when they claim that GPT-3 is doing ‘shallow pattern-matching’, ‘not really making sense at all’, etc..”
I got texted by a “wrong number” scammer, and I was curious if it was a bot, so after a bit of thought I came up with the following one-question Turing test:
I love this question because any real human can answer it pretty well, but answering it relies on all sort of knowledge of the physical properties of books and food that we mostly don’t write about, and especially not where they interact. I asked my partner Jess and she said “a banana” which is pretty much a perfect answer: it’s heavy enough to hold the pages open, it’s long and thin so it won’t cover all the text, it’s curved so it won’t roll off, and it has a skin so it won’t mess up the book.
But then I worried, am I underestimating modern engines like GPT-3? I asked Rosie Campbell at OpenAI to try it. Here and throughout this post, answers are almost entirely unfiltered/uncurated, except where it gave very similar answers twice. Here’s what we got:
This however is without any prompt engineering; GPT-3 can often do dramatically better given a better prompt. Rosie tried with the following prompt:
Here are some of the answers it gave. Note that *all* the text is generated by GPT-3 (except the initial bit “Friend:” which is part of the prompt), including the bits attributed to “Me”.
After this, Rosie tried a more sophisticated prompt, to try and set up the “random party question” frame better:
Here’s what we got:
“Banana” is a great answer; I was struck that GPT-3 gives “banana” as an answer so often, yet is almost completely unable to explain why. Anisha Sensa pointed out in comments that this probably doesn’t represent insight into the question:
How did my scammer do with this question? Not great: