I’ve tried it. Here are some examples. I didn’t save the original prompts and answers.
Use the elliptic functions provided by Matlab to calculate the length of an elliptic arc. (I already knew that there is a closed-form solution to this in terms of elliptic functions. Everyone writing an introduction to elliptic functions mentions this, but I have never seen anyone give an actual formula. The task is complicated by the existence of multiple conventions for defining these functions.)
It gave some Matlab code using elliptic functions, but it was simply wrong. With some effort of my own, I eventually worked out a correct formula, and verified that it agreed with numerical integration.
Devise a function that maps [0,1] onto [1,∞], is strictly increasing and differentiable, and which takes a parameter specifying how late and sharp its divergence to ∞ should be. Then program it in Matlab.
It produced a function that missed several of the requested properties. It also programmed it in Matlab, but given that the function was wrong, I didn’t bother to see if the programming was right.
I chose this problem because I’d recently needed to do that myself. It took no longer to do it right myself than to ask the LLM and determine whether it got it right — which it didn’t, so that would have been wasted time.
I asked it how to modify an iOS app to respond to the user’s dark/light mode setting.
The answer it gave me I could have looked up on Google just as quickly, and Google’s answer had the advantage of going directly to Apple’s documentation and a WWDC presentation, sources of ground truth rather than the ungrounded vagaries of a chatbot which were not even worth reading.
Score: 0⁄3. This is typical of the results I see from LLMs on every task they are applied to, whether mine or other people’s. When I have a question, I want an answer that rings like a bell, not an LLM’s leaden clunk.
You can test the latter hypothesis by trying it (more) :)
I’ve tried it. Here are some examples. I didn’t save the original prompts and answers.
Use the elliptic functions provided by Matlab to calculate the length of an elliptic arc. (I already knew that there is a closed-form solution to this in terms of elliptic functions. Everyone writing an introduction to elliptic functions mentions this, but I have never seen anyone give an actual formula. The task is complicated by the existence of multiple conventions for defining these functions.)
It gave some Matlab code using elliptic functions, but it was simply wrong. With some effort of my own, I eventually worked out a correct formula, and verified that it agreed with numerical integration.
Devise a function that maps [0,1] onto [1,∞], is strictly increasing and differentiable, and which takes a parameter specifying how late and sharp its divergence to ∞ should be. Then program it in Matlab.
It produced a function that missed several of the requested properties. It also programmed it in Matlab, but given that the function was wrong, I didn’t bother to see if the programming was right.
I chose this problem because I’d recently needed to do that myself. It took no longer to do it right myself than to ask the LLM and determine whether it got it right — which it didn’t, so that would have been wasted time.
I asked it how to modify an iOS app to respond to the user’s dark/light mode setting.
The answer it gave me I could have looked up on Google just as quickly, and Google’s answer had the advantage of going directly to Apple’s documentation and a WWDC presentation, sources of ground truth rather than the ungrounded vagaries of a chatbot which were not even worth reading.
Score: 0⁄3. This is typical of the results I see from LLMs on every task they are applied to, whether mine or other people’s. When I have a question, I want an answer that rings like a bell, not an LLM’s leaden clunk.