Interesting! Yes, I am using ChatGPT with GPT-4. It printed out the code, then *told me that it ran it*, then printed out a correct answer. I didn’t think to fact-check it; instead I assumed the OpenAI has been adding some impressive/scary new features.
Oh this is funny. It told me that it ran the code and got the answer [64, 91, 39, 47]. I checked that these satisfied the problem. But I didn’t check (until reviewing other comments) whether that’s actually what the code outputted. It’s not. Technically the code actually doesn’t output anything, it saves the result to a variable instead. And if I print that variable, it found [64, 6, 96, 75].
Lesson 1: I was not careful enough in checking its output, even when I thought I was being careful.
Lesson 2: It is indeed not running code, even if it tells me it is.
Possibly it somehow got lucky with its pattern-matching heuristics. The start of B is 64, 6, 9, 91, 59, 47… And 59 is similar to 39, which occurs a few numbers later in the list. So it’s a successful subset which is not too many permutations and substitutions away from the original list.
This makes sense. Another thought I had was that I picked 50 numbers 1-100. It could’ve just gotten lucky. If I were to do this again, I’d do 1-1000 to decrease the odds of this.
Are you using chat-GPT-4? I thought it can’t run code?
Interesting! Yes, I am using ChatGPT with GPT-4. It printed out the code, then *told me that it ran it*, then printed out a correct answer. I didn’t think to fact-check it; instead I assumed the OpenAI has been adding some impressive/scary new features.
Oh this is funny. It told me that it ran the code and got the answer [64, 91, 39, 47]. I checked that these satisfied the problem. But I didn’t check (until reviewing other comments) whether that’s actually what the code outputted. It’s not. Technically the code actually doesn’t output anything, it saves the result to a variable instead. And if I print that variable, it found [64, 6, 96, 75].
Lesson 1: I was not careful enough in checking its output, even when I thought I was being careful.
Lesson 2: It is indeed not running code, even if it tells me it is.
I am confused how it got the answer correct without running code?
Possibly it somehow got lucky with its pattern-matching heuristics. The start of B is 64, 6, 9, 91, 59, 47… And 59 is similar to 39, which occurs a few numbers later in the list. So it’s a successful subset which is not too many permutations and substitutions away from the original list.
This makes sense. Another thought I had was that I picked 50 numbers 1-100. It could’ve just gotten lucky. If I were to do this again, I’d do 1-1000 to decrease the odds of this.