1. A major question here is LLMs. Scattershot thoughts:
Any notebase-refactoring tool would be much more useful if it’s able to understand semantics, instead of operating off of coarse measures like blind text-clustering algorithms. LLMs seem like the tool for the job.
LLMs are reportedly pretty good at understanding research papers and at reading comprehension in general, and those skillsets have a lot of overlap with this. They may do a good job here.
LLMs have pretty bad research taste, and research taste is tightly entangled with the whole “choice of correct abstractions/ontologies” thing we want to get right here. They may not do a good job here.
Plenty of research ideas are fragile Butterfly Ideas, so the process of refactoring your thinking on a topic often has to be “intimate”. Injecting the opinions of some external person-like entity into it may be lethal. LLM contributions, if there are any, would likely need to be heavily sanitized/de-personalized/abstracted over.
Plenty of research ideas are dream ideas/empty ideas. Having an LLM with proper context correctly shoot holes in them may save you months of wasted time.
LLMs are sycophantic and sometimes good at persuasion, so they may encourage your delusions about your dream/empty ideas. Or just inject subtle misunderstandings into your ontology in an attempt to avoid upsetting you.
Overall, I expect consideration (1) dominates here, and mitigating (2)-(6) would require a fair bit of tinkering.
An approach that might work here is to define a bunch of useful high-level functions for manipulating natural-language notes (such as “redefine(X, Y) means ‘redefine X in terms of Y’”), describe those to an LLM, then build an interface where interactions with the LLM are conducted exclusively through this abstraction layer. As in, there’s a literal “redefine” button, you select notes X and Y, press the button, and get the redefined notes, without having to talk to the LLM at all.
Ideally, the LLM is also fine-tuned to do the task correctly, minimizing how much personal flourish it injects and enhancing its attention to detail.
Some illustrative examples of such functions:
“Redefine the set of notes N in terms of X.”
“Fetch the list of phenomena in the set of notes N that {contradict}/{are unexplained given} the model described in the notes M.”
“Check whether X and Y are isomorphic in the context of N.”
“Find all information in the set of notes N relevant to the thought X.”
“Check whether assertion X contradicts/breaks anything in the set of notes N.”
“Merge notes X and Y, distilling the information redundant between them into a new node and creating 0-2 new nodes recording information unique to each.”
2. “Mundane” UI ideas:
I think a 2D representation along the lines of Obsidian’s canvas (the canvas, not the graph view) is a good start. By contrast, 1D (a text document) or “1.5D” (set of text documents separated into folders, creating a “tree-like” format) are too impoverished to help with reasoning about any nontrivial structure at all, and 3D (let alone higher) is hard to parse because we process visual information in 2D.
Some sort of native multi-level hierarchical representation would be nice. “These two concepts often interact/appear together” (a “horizontal” connection) and “these low-level concepts are how this higher-level concept is implemented” (a “vertical” connection) are qualitatively different types of connections – yet a “flat” graph representation forces you to mix them up.
I. e., we prospectively want a “2.5D” representation: a partially ordered set of arbitrary 2D graphs, corresponding to systems living on different levels of abstraction. Equipped with native functionality for flexibly editing it.
History-keeping would be useful. Preferably with a good visualizer regarding how the notebase was transformed from A to B, with an easy ability to (partially) revert the changes.
None of those seem like high-enough-impact improvements, though.
I quite like some of your ideas about how to design note-taking software and I think I have a partial answer to the issues you point to:
Make the repository of notes more semantic than usual, halfway between pure text notes and a structured database. If the notes exist in a graph database then it becomes possible to write database queries (probably with something like SPARQL).
Allow the user to store queries within notes so the query is run whenever the note is opened. Maybe allow for a graph visualization of the query. This way the semantic structure becomes a part of the normal way to interact with the “hub” notes.
The ontology used on a topic is defined by the structure of the links. If the user wants a new ontology they just create it on top of the old one and let the old one rot. No need to erase it and it does not distract from the new one (maybe there is some name reuse in the link/attribute names, but that’s trivial to fix).
Use LLMs to correctly annotate notes with the semantic information that populates the database (links, tags, some attributes, …) This way they don’t need taste to know how to edit, they just follow the pattern that comes with a given ontology. Perhaps this requires the user to create templates.
Provide a good CLI (Command Line Interface) for the software. This way the user can write scripts to update their notes using any language. It becomes easy to apply the new ontology to the old notes, possibly fully automagically but most of the time there is probably a bit of manual work left afterward. That’s fine.
(unrelated to your post) let the GUI be a VScode extension so the user can use whatever other existing extension they like, this way there is no need to also design a good text editor.
It’s quite barebones, and just more of a silly sketch of a project rather than serious project, but a lot of these ideas are what I’m thinking about within https://github.com/Jadael/Library-of-Aletheia—some way to systematically and reliably apply library science style ministerial operations, sometimes LLM powered and sometimes traditional scripts, to a collection of information, all in an Obsidian-canvas-esque “web of knowledge” framing and a visual UI.
I stopped working it on it because the LLM extension I was using in Godot wasn’t being updated, and couldn’t go above a hardcoded context window of 512 which limited the ability to do anything interesting, but perhaps it’s possible to update it t just use Ollama or something nowadays 🤔
1. A major question here is LLMs. Scattershot thoughts:
Any notebase-refactoring tool would be much more useful if it’s able to understand semantics, instead of operating off of coarse measures like blind text-clustering algorithms. LLMs seem like the tool for the job.
LLMs are reportedly pretty good at understanding research papers and at reading comprehension in general, and those skillsets have a lot of overlap with this. They may do a good job here.
LLMs have pretty bad research taste, and research taste is tightly entangled with the whole “choice of correct abstractions/ontologies” thing we want to get right here. They may not do a good job here.
Plenty of research ideas are fragile Butterfly Ideas, so the process of refactoring your thinking on a topic often has to be “intimate”. Injecting the opinions of some external person-like entity into it may be lethal. LLM contributions, if there are any, would likely need to be heavily sanitized/de-personalized/abstracted over.
Plenty of research ideas are dream ideas/empty ideas. Having an LLM with proper context correctly shoot holes in them may save you months of wasted time.
LLMs are sycophantic and sometimes good at persuasion, so they may encourage your delusions about your dream/empty ideas. Or just inject subtle misunderstandings into your ontology in an attempt to avoid upsetting you.
Overall, I expect consideration (1) dominates here, and mitigating (2)-(6) would require a fair bit of tinkering.
An approach that might work here is to define a bunch of useful high-level functions for manipulating natural-language notes (such as “redefine(X, Y) means ‘redefine X in terms of Y’”), describe those to an LLM, then build an interface where interactions with the LLM are conducted exclusively through this abstraction layer. As in, there’s a literal “redefine” button, you select notes X and Y, press the button, and get the redefined notes, without having to talk to the LLM at all.
Ideally, the LLM is also fine-tuned to do the task correctly, minimizing how much personal flourish it injects and enhancing its attention to detail.
Some illustrative examples of such functions:
“Redefine the set of notes N in terms of X.”
“Fetch the list of phenomena in the set of notes N that {contradict}/{are unexplained given} the model described in the notes M.”
“Check whether X and Y are isomorphic in the context of N.”
“Find all information in the set of notes N relevant to the thought X.”
“Check whether assertion X contradicts/breaks anything in the set of notes N.”
“Merge notes X and Y, distilling the information redundant between them into a new node and creating 0-2 new nodes recording information unique to each.”
2. “Mundane” UI ideas:
I think a 2D representation along the lines of Obsidian’s canvas (the canvas, not the graph view) is a good start. By contrast, 1D (a text document) or “1.5D” (set of text documents separated into folders, creating a “tree-like” format) are too impoverished to help with reasoning about any nontrivial structure at all, and 3D (let alone higher) is hard to parse because we process visual information in 2D.
Some sort of native multi-level hierarchical representation would be nice. “These two concepts often interact/appear together” (a “horizontal” connection) and “these low-level concepts are how this higher-level concept is implemented” (a “vertical” connection) are qualitatively different types of connections – yet a “flat” graph representation forces you to mix them up.
I. e., we prospectively want a “2.5D” representation: a partially ordered set of arbitrary 2D graphs, corresponding to systems living on different levels of abstraction. Equipped with native functionality for flexibly editing it.
History-keeping would be useful. Preferably with a good visualizer regarding how the notebase was transformed from A to B, with an easy ability to (partially) revert the changes.
None of those seem like high-enough-impact improvements, though.
I quite like some of your ideas about how to design note-taking software and I think I have a partial answer to the issues you point to:
Make the repository of notes more semantic than usual, halfway between pure text notes and a structured database. If the notes exist in a graph database then it becomes possible to write database queries (probably with something like SPARQL).
Allow the user to store queries within notes so the query is run whenever the note is opened. Maybe allow for a graph visualization of the query. This way the semantic structure becomes a part of the normal way to interact with the “hub” notes.
The ontology used on a topic is defined by the structure of the links. If the user wants a new ontology they just create it on top of the old one and let the old one rot. No need to erase it and it does not distract from the new one (maybe there is some name reuse in the link/attribute names, but that’s trivial to fix).
Use LLMs to correctly annotate notes with the semantic information that populates the database (links, tags, some attributes, …) This way they don’t need taste to know how to edit, they just follow the pattern that comes with a given ontology. Perhaps this requires the user to create templates.
Provide a good CLI (Command Line Interface) for the software. This way the user can write scripts to update their notes using any language. It becomes easy to apply the new ontology to the old notes, possibly fully automagically but most of the time there is probably a bit of manual work left afterward. That’s fine.
(unrelated to your post) let the GUI be a VScode extension so the user can use whatever other existing extension they like, this way there is no need to also design a good text editor.
I sent you a DM to talk more about this.
It’s quite barebones, and just more of a silly sketch of a project rather than serious project, but a lot of these ideas are what I’m thinking about within https://github.com/Jadael/Library-of-Aletheia—some way to systematically and reliably apply library science style ministerial operations, sometimes LLM powered and sometimes traditional scripts, to a collection of information, all in an Obsidian-canvas-esque “web of knowledge” framing and a visual UI.
I stopped working it on it because the LLM extension I was using in Godot wasn’t being updated, and couldn’t go above a hardcoded context window of 512 which limited the ability to do anything interesting, but perhaps it’s possible to update it t just use Ollama or something nowadays 🤔