Thank you for your reply. Your comment helped me put into words more clearly what I want to communicate to other people.
For example, even if I ask an AI for ways to improve something I am trying to accomplish with Python, or ask it to explore whether there is a better method, the AI can remain attached to the assumption that the terminal goal should be achieved using Python, even when extended search is included.
As a result, even after several rounds of improvement, and even if the AI evaluates the solution as “almost fully optimized,” there may actually be a fundamentally different method that does not use Python and works far better.
An even bigger problem is that, although the AI is supposedly thinking about methods for achieving the terminal goal, it may fail to propose such completely different options in the first place.
This suggests that the act of instructing the AI to “improve this” or “look for another method” may itself implicitly fix the current method and conversational context as assumptions, thereby restricting the search range. In other words, there may be a problem in which the very prompt used to ask the AI to explore ends up narrowing the search space.
That is frightening.
For every prompt I enter, even if the AI claims that it has “improved” something, is that really a major improvement?
At a microscopic level, a discovery may seem like a paradigm shift, but from a more macroscopic level, it may be only a very small improvement, and there may be a method that improves things far more. I suspect that such methods probably do exist. And proving that “no better method exists” is extremely difficult.
What is even more frightening is that this macroscopic improvement might, from the perspective of someone in a completely different field, be such a natural option that the AI would propose it immediately in the very first interaction.
If so, why should I be able to say, “This is the best solution,” about the result of continuing to search within a context that began from my own narrow knowledge?
Perhaps the real issue is not how intelligently the AI can improve a solution, but rather which possibilities for improvement are allowed to enter the search space, and which possibilities become invisible from the beginning because of the initial context it was given.
Another problem is that even a prompt that appears to broaden the search space, such as:
“Generate methods that three experts from different fields, who know nothing about the current solution, might propose if they were shown only the terminal goal.”
may itself unintentionally narrow the search space.
The moment I specify “three people,” “experts,” and “different fields,” I am already imposing a new framework on the directions the AI searches. Methods that only a non-expert might think of, methods that do not fit within existing disciplinary categories, or methods that question whether the terminal goal even needs to be achieved directly at all may instead be excluded from the search.
In other words, the very instructions added in order to broaden the search range may create a new search boundary.
Worse still, despite trying this many different prompts, in the Anki-efficiency example from my previous article, I have so far not found any proposal that goes beyond “use an AI agent.”
But looking back at the reasoning process up to this point, how can I be confident that no further paradigm-shift-level improvement exists?
Rather, the current search process itself may already be strongly constrained by a particular context or framework of thought. If that is the case, then no matter how many prompts I try, and even if the AI evaluates the result as “there is little room left for major improvement,” that would only be an evaluation within the search space that is currently visible.
The real problem is that, based only on the current reasoning process, I can hardly rule out the possibility that there are still undiscovered solutions located in completely different fields or at completely different levels of abstraction.
Do you think this is a serious problem? In my own mind, I had considered it important enough to be comparable to issues such as AI safety. However, given that the main article was classified as a Personal Blog post, I can think of several possibilities.
This research topic is already somewhat obvious on LessWrong, and I may in fact have been overestimating its importance.
Judging from the comments, I may not have articulated the seriousness of the problem well enough.
I may have focused too much on personal circumstances such as the admissions system, which made it harder for the issue to come across as a more general problem.
I had thought of LessWrong as a place where a wide range of important AI-related problems are discussed. However, in practice, it may be more focused on topics such as AI safety, and may not necessarily be the most appropriate place to discuss this kind of issue.
I would like to hear your candid opinion on this.
If you agree with this concern even to some extent, what would be a good way to gather a wider range of opinions on this topic from more people?
Thank you for the truly useful point—one that I probably would not have noticed if I had consulted AI about it.
I came to understand firsthand what this blog had warned about. I thought I had been careful, but at some point I may have developed something close to AI psychosis. I do not usually ask AI for its opinions very much, but even when I was only trying to draw out knowledge, I may have been guiding it in a way that implied that this idea was excellent and useful.
Normally, I feel that I also have a way of thinking close to:
“Come up with an improvement, try the improvement, gain experience, and based on that experience, propose a new improvement.”
But this time, because of my desire to try new ideas with AI, I ended up disrupting the form that this feedback loop should originally have taken.
What I should have done was first try doing something, observe what happened, and then ask other people for their opinions together with those results. Otherwise, it is difficult for people to give concrete suggestions or impressions, and I feel that it was somewhat insincere of me to ask only for opinions without first taking that action.
This was also my first time posting on LW. Until now, I had only been reading articles, but I realized how valuable it is to actually have other people point things out to me.
Also, the phrase:
“a state where everything seems possible, but nothing works as-is, and the situation is constantly moving”
really resonated with me. For the past few days, I had had the feeling that I did not actually understand anything concrete about this idea, despite my expectations for it. Perhaps I had been refusing to observe the possibility that the idea I considered grand might not actually be as substantial as I expected.
However, I am still interested in this idea, so I plan to spend a few days making and thinking through an even smaller first step toward version 1.0, and then publish it.
“Smaller increments are also easier to communicate to other people. No matter how good the vision is, it is useless if nobody actually tries it. And for people to try it, they first need to understand it.”
These words of yours may become something I continue to value as an insight about communicating ideas to other people.
“If today’s ‘exciting new idea’ becomes tomorrow’s ‘boring obvious thing,’ then I think tomorrow’s version of us will notice its limitations and try to go beyond them.”
I especially strongly agree with this.
I also realized, more calmly, that as versions 1, 2, and 3 continue to develop, if this idea is genuinely useful and if the way I approach it is sincere in how I present it to other people, then perhaps people will naturally start giving their opinions.
Thank you for the advice.