Certainly, someone will for sure ask it to produce the text that maximizes the stock price of their company, then the superLLM will pass that prompt through its model, and output the most likely continuation of that request, which is not at all text that actually maximizes the stock price. Because out of all instances of text containing “Please maximize my stock price” over the internet, there are no examples of superintelligent outputs to that request. It’s more likely to consider that request as part of a story prompt, or output something like “I don’t know how to do that”, even if it did internally know how to do that.
I want to note that if we assume it’s merely a superintelligent predictor, trained on all available data in the world, but only able to complete patterns super-well, it’s still extremely useful for predicting stock prices. This is in itself an incredibly profitable ability, and can also be leveraged to “output text that maximizes stock price” without too much difficulty:
Have the system output some text periodically.
Interleave the company stock prices between text blocks.
Generate a large number of samples for each new prediction, and keep the text blobs for which further completions predict high stock prices down the line. (This can be done automatically—no human review, just look at the predicted price.)
Not saying this is a great technique in real life, just saying that if we assume “really great predictor” and go from there, this will eventually start working well, as the system notices the influence of its text blobs on the subsequent stock prices.
My answer is that that would happen by default, and then some clever human would figure out a way to prompt engineer the system/slightly reconfigure it so that it did what it really knew how to do.
Certainly, someone will for sure ask it to produce the text that maximizes the stock price of their company, then the superLLM will pass that prompt through its model, and output the most likely continuation of that request, which is not at all text that actually maximizes the stock price. Because out of all instances of text containing “Please maximize my stock price” over the internet, there are no examples of superintelligent outputs to that request. It’s more likely to consider that request as part of a story prompt, or output something like “I don’t know how to do that”, even if it did internally know how to do that.
I want to note that if we assume it’s merely a superintelligent predictor, trained on all available data in the world, but only able to complete patterns super-well, it’s still extremely useful for predicting stock prices. This is in itself an incredibly profitable ability, and can also be leveraged to “output text that maximizes stock price” without too much difficulty:
Have the system output some text periodically.
Interleave the company stock prices between text blocks.
Generate a large number of samples for each new prediction, and keep the text blobs for which further completions predict high stock prices down the line. (This can be done automatically—no human review, just look at the predicted price.)
Not saying this is a great technique in real life, just saying that if we assume “really great predictor” and go from there, this will eventually start working well, as the system notices the influence of its text blobs on the subsequent stock prices.
Misread your comment.
My answer is that that would happen by default, and then some clever human would figure out a way to prompt engineer the system/slightly reconfigure it so that it did what it really knew how to do.