I think the collection part of GTD addresses exactly this problem. There’s two part:
You want to free your brain by writing what you want to do
You want to stop feeling like you forgot writing something down
The way proposed by GTD is to collect EVERYTHING. The goal is really to not have any commitment or desire stored internally, but collect everything outside of your brain. This solves the first problem if you give enough details, and the second problem when your brain learns that it can always find what it needs from your notes.
Anecdotically, it works for me.