Introducing Familiar, a quantified reasoning assistant (feedback sought!)
tl;dr: I’m making a thing that uses probabilistic graphical models to assist in drawing inferences from personal data. You should check it out, and share with me your wisdom/user experience.
I had this not-completely-original idea that there should be some kind of tool for easily performing statistical inference on Quantified Self-style data.
There are a lot of QS apps out there, but for the most part they seem to be designed for 1. a single domain and/or 2. recording things primarily to combat akrasia or (more often) sating curiosity/as a lifestyle accessory, rather than actively helping you discover correlations or determine causality between things-you-do and things-you-care-about. Quantified Mind stands out as a counterexample, but I can’t come up with many others in that vein.
There are also commercial products and programming languages that allow one to use machine learning to perform inference on data, but they mostly seem to be proprietary and expensive software aimed at businesses, or free but intended to be used by scientists, engineers, etc.; nothing I’ve yet to find is really suitable for an individual without a background in statistics/machine learning who just wants to learn what they can by smashing together their Moodscope and their FitBit.
In our era of FOSS, APIs, QS, and ML, this seems like a seriously lacking state of affairs. Hence, Familiar.
Currently, it consists of a command line interface for storing variable definitions and data in a local database without too much fuss, building a naive Bayes classifier on those variables, and finding maximum likelihood estimates given the state of one variable for the states of all the other variables. This is unsophisticated and not extremely user-friendly, but those things will change in the near future. In the case where I keep working on this for a very long time, I want to automate away as much recording as possible (including things like mood and productivity), record everything with the highest reasonable time resolution, plug into every other app out there that might provide useful data, use more complex machine learning algorithms to identify causality and generate suggestions for personal experimentation, and generally have a piece of software that knows you so well it can help you think more like an ideal Bayesian reasoner and thereby assist you in living your life (thus the name). Manfred Macx’s glasses from Accelerando have something like this inside them, and I want it too.
Anyway, back to the present. You can help me by answering whichever of these questions applies to you the most:
Would you use something like this at all, or do you think the potential for extracting useful information out of messy personal data is too low?
If you might use something like this, but don’t want to use Familiar in its current state, what do you think is the most important factor? e.g. “no GUI”, “not a web app”, “too manual”, “doesn’t connect to other stuff yet”, etc.
If you’re brave enough to start using this now or even look at the source code, what mistakes am I making? There are countless ways this could be easier to use, more helpful, faster, more readable, and otherwise better, and you can tell me what those ways are.
Oh my god if we can get this working with org-mode and habitrpg it will be the ultimate trifecta. And I’ve already got the first two (here).
Seriously this could be amazing. Org-mode and habitrpg are great, but they don’t really solve the problem of what to do next. But with this, you get the data collection power of org mode with the motivational power of habitrpg—then Familiar comes in, looks at your history (clock data, tags, agendas, all of the org mode stuff will be a huge pool of information that it can interact with easily because emacs) and does its thing.
It could tell habitrpg to give you more or less experience for things that are correlated with some emotion you’ve tagged an org mode item with. Or habits that are correlated with less clocked time on certain tasks. If you can tag it org mode you can track it with familiar, and familiar will then controls how habitrpg calculates your experience. Eventually you won’t have that nagging feeling in the back of your head that says “Wow, I’m really just defining my own rewards and difficulty levels, how is this going to actually help me if I can just cheat at any moment?”—Maybe you can still cheat yourself, but Familiar will tell you exactly the extent of your bullshit. It basically solves the biggest problem of gamification! You’ll have to actually fight for your rewards, since Familiar won’t let you get away with getting tons of experience for tasks that are not correlated with anything useful. Sure it won’t be perfectly automated, but it will be close enough.
It could sort your agenda by what you actually might get done vs shit that you keep there because you feel bad about not doing it—and org mode already has a priority system. It could tell you what habits (org-mode has these too) are useful and what you should get rid of.
It could work with magit to get detailed statistics about your commit history and programming patterns.
Or make it work with org-drill to analyze your spaced repetition activity! Imagine, you could have an org-drill file associated with a class you are taking and use it to compare test grades and homework scores and the clocking data from homework tasks. Maybe there is a correlation between certain failing flashcards and your recent test score. Maybe you are spending too much time on SRS review when it’s not really helping. These are things that we usually suspect but won’t act on, and I think seeing some hard numbers, even if they aren’t completely right, will be incredibly liberating. You don’t have to waste cognitive resources worrying about your studying habits or wondering if you are actually stupid, because familiar will tell you! Maybe it could even suggest flashcards at some point, based on commit history or wikipedia reading or google searches.
Maybe some of this is a little far fetched but god would it be fun to dig into.
My sentiment exactly! This seems like the sort of thing that would enable people to seriously improve their lives in a lot of different ways, but there are also many more ways to use it that probably wouldn’t help much or at all. That’s why I’m trying to focus on ease of use—the more people out there experimenting with measuring different things in combination, the sooner everyone gets to benefit from those methods and combinations of measurements that actually do help.
Have you taken a look into SamIam? It has a decent GUI and might, while not being the tool that you intend to build yourself, inspire you.
Just reading its documentation has been very helpful so far. Thank you!
I would immediately download this iff it had a GUI.
Yes, I’m pretty sure this is currently the most significant hurdle to getting people to want to use it (this isn’t the ’80s!). Adding some kind of browser-based GUI is my current task.
So immediately download it!
That’s precisely the reason why I gave up and am building my own Bayesian classifier to do … almost exactly what this post’s project will do. Only, mine is meant for strictly personal use and is related to hashing out a diagnosis with the help of a doctor.
Researchers in pharmacogenetics have recently discovered that they can get important cues about drug side effects using people’s search histories—you might find for instance, that people who have either of two drugs in their search history have a certain frequency of searching for terms like ‘drowsiness’ or ‘diabetes’ but people who have both search for these terms with a greatly increased frequency.
Quantified life?
“The unquantified life is not worth living”—pseudo-Socrates
“The value of an unquantified life cannot be determined exactly.”
Firstly I do have a QS background and study bioinformatics.
Machine learning usually requires a lot of data. For QS person you usually only have one person so your data is limited. If you throw a complex machine learning algorithm against the data, you are likely to overfit.
If you analyse your own data in QS you know something about the data and that’s not in the numbers. You might know that your mood on a particular day dropped because you ended a relationship but you don’t have a quantified variable that tracks the event of a relationship ending.
Hooray, people with credentials! Thank you for sharing your knowledge.
This was my most convincing reason to try to bother implementing the statistical guts myself in the first place; it was pretty easy to put together a little naive Bayes classifier that calculates the maximum likelihood estimate for all your other variables/predicates given the value of one variable/predicate intended to be minimized or maximized, and I’m pretty sure it works mostly correctly, and I’m pretty sure the additional return from using virtually any of the existing more sophisticated ML algorithms that I’ve yet to hear of won’t be nearly as high as the initial return from being able to answer the question “if this is the case, what other stuff was most likely the case?”. I’m starting to get the suspicion that the next most useful modeling-related task may be to focus on generating lots and lots of different compound predicates given all your raw variables in some way that doesn’t reek of unhelpful combinatorial explosion, then calculating the maximum likelihood and probability of that value for all of them using my existing dumb little classifier, which isn’t something I can recall seeing any work on. If that reminds you of something then I will desperately consume whatever resources or names of algorithms that come to your mind.
In the near term, very low-frequency events that have a very clear impact on other variables might be usefully grouped under the raw variable “significant and obvious other things that aren’t worthy of their own variable in my opinion”, and many of the more useful terminally-valued predicates could imaginably require that variable to take a certain value/range of values. Maybe “rejection by people currently in my social circle” with a holistic, multi-valued rating could be its own variable, if that happens unfortunately often for some people. I don’t expect to automate measuring something like this any time soon, but it is undoubtedly important to know about in figuring out optimal conditions for “normal days”, which makes me think the manual data entry part will be unavoidable if you want this to be particularly useful.
In the more distant future, I see no reason why such a thing couldn’t be learned into a variable that actually does an okay job of carving the important parts of your personal reality at the joints. If you have a software system that knows relationships are important to people, knows which of your relationships are important to you, knows who you were talking to, and knows the valence, arousal, duration, frequency, etc. of your interaction with that person over time, then, yes, something like “ended a relationship today” probably could be inferred. It doesn’t sound trivial but it sounds absolutely plausible, given sufficient effort.
It’s possible to track 1000 different variables with your model. If you do so, you will however get a lot of false positives.
I think about QS data like it gives you more than your five senses. In the end you still partly rely on your own ability of pattern matching. Graphs of data just gives you additional input to understand what’s going on that you can’t see or hear.
I plan on addressing false positives with a combination of sanity-checking/care-checking (“no, drinking tea probably doesn’t force me to sleep for exactly 6.5 hours the following night” or “so what if reading non-fiction makes me ravenous for spaghetti?”), and suggesting highest-information-content experimentation when neither of those applies (hopefully one would collect more data to test a hypothesis rather than immediately accept the program’s output in most cases). In this specific case, the raw conversation and bodily state data would probably not be nodes in the larger model—only the inferred “thing that really matters”, social life, would. Having constant feedback from the “expert”, who can choose which raw or derived variables to include in the model and which correlations don’t actually matter, seems to change the false positive problem.
Or, I mean, just use Facebook and other social media activity to identify the formation, strengthening, and slow or abrupt end of friendships and relationships. Many of us do basically already live in that world.
This seems very cool; commenting to encourage you, and so I remember to come back and look at this when I have more time to think about it.
Hi.
Based on your github description: is this just a program that does parameter and structure learning in graphical models? Why not simply use one of the large set of existing programs that do this?
All the ones I’ve found so far have some combination of the following traits:
Proprietary and expensive
Steep learning curve/assumes the user will be a technical professional
Not designed to use data from other services
I think a program that avoids all of those things simultaneously will be able to fill a useful niche for many people. If you know of a free program that an intelligent layperson could learn to use with minimal effort, and that’s made to collect data from a variety of existing Quantified Self apps and devices (or at least be easily amenable to such extension), I’ll happily stop development and use that instead.
There are lots of open source research programs for graphical model stuff.
I think it is a better (if less glamorous) use of time to put a user friendly UI around one of these, and write glue code to use data formats you want than to re-implement the statistical guts from scratch. Academics have lots of incentive to have very smart “guts,” but not a lot of incentive to interoperate with data sources and have user friendly UIs.
That seems extremely sane in retrospect; thankfully I haven’t done very much work on the statistics side, and sunk costs are therefore comfortably low if I were to take that path. I’m also not worried about glamour at all in that case, since I’m the one doing the work of “make it something people will actually want to use and that is useful”. There are indeed many of them—do you have a specific recommendation?
Hello, thanks for your reply.
I am mostly on the supply side of the aforementioned “statistical guts,” so I am not sure I am the best person to ask. The two big questions (which I am sure are obvious to you as well) are (a) language used, and (b) what functionality do you want in the “guts”? For instance Kevin Murphy’s BNT has a lot of stuff, but is written in Matlab and not R. The fact that lots of people have to pay for Matlab will hinder adoption. SAMIAM is not open source, I think.
@jamesf—which one of the programs will you be picking?
I’m researching that right now. Recommendations are welcome.
I’ve just started playing with it now so these are just initial thoughts:
1) Command Line History would be really nice: Messing up a command (new-prd $var1 (fn ….) and having to retype from scratch is a pain. Although if you get a nice GUI it won’t matter;
2) (Never mind; I can’t reproduce it now) (After I did (change-time (days −1)); (data ..) nothing was returned from (entered) _ ;
3) Allowing data to be entered via a source file might be nice, but I suppose a script would work ok too. Perhaps I should write a Perl script to convert CSV into Familiar data?
For #1, just use readline/editlne. Don’t try to implement your own.
Although personally I’d prefer a nice shell-scripting interface. (Note: I have not actually tried the program or checked if it has one.)
It can take command line arguments and therefore be put into a shell script, if that’s what you mean. Here’s an example:
It looks like getting editline to talk with my Clojure code would be somewhat non-trivial. Using the Clojure REPL is probably the better alternative for serious use until I get a GUI working.
Reading CSVs is now on the todo list.
Get rlwrap. Wrap the program in a script that runs ‘rlwrap program’. All the benefits of readline with no code. You’re welcome. ;-)
I’m hesitant to use software licensed under the GPL in my own program, as convenient as that sounds. I’d like to release it under a (even) less restrictive license.
More questions!
Is there a way to get at the data this produces, if I want to manually edit it, or use it with a different program? (Or rather, of course there’s a way—how would you recommend I do it?)
How many data points do you need before (correlations) starts giving you results?
It’s an H2 database saved inside /data in your Familiar directory. You can make SQL queries into it with other programs. Exporting to JSON or CSV or something will happen eventually.
Two, technically, I suppose, but I’d probably collect data for a couple of months before I started seriously interpreting correlations involving variables with a resolution of one day. This will be a topic in the more extended documentation.
Ah. Well, (correlations enough-sleep true), for example, just gives me “That didn’t work”—what am I doing wrong?
I’m not sure. If you’re comfortable sharing your data, PM me a link to the contents of your /data folder.
I’d rather not, but I reproduced the problem with some sample data here.
It’s just six days of “example1-6”, and two predicates. example1 is a {0,1,2}, example2-4 are non-negative, example5-6 are boolean; example7 is defined as “example4 on day-1″, example8 is defined as “example3 >= 20”. I’ve filled them in so that example8 generally implies example1=2, and example6 implies example1=2, with one exception as noise (with example6 && example8 and example1=1).
Generates the same error.
Also, the default time is a good four hours off (presumably because of timezones?) Is there a way to change that?
Can you use percentage variables?
(And while we’re at it, is there are more comprehensive readme anywhere?)
You want
if the active time is on the wrong day. The active time being wrong but on the correct day doesn’t matter yet since only day-resolution variables are supported. A time zone setting will be added along with variables of arbitrary time resolution.
For percentage variables, use
as the validator for a variable. (I will add “percent” as a built-in validator.)
Besides the built-in documentation for all the API functions, the readme on GitHub is the most comprehensive existing documentation. Of course I intend to fix that eventually, probably along with the next release which is going to be the one where there is a GUI.
Thank you for all your feedback!
Yeah, this is what I need. I’m planning to do most of my data-input at 2am-ish, and it’d be nice to not have to remember to mentally subtract a day from the existing data.
And thanks for making this!
Is there a way to modify a variable once you’ve created it, or delete it? In particular, I’d like to change the units and the value type.
There is, though it’s not implemented as a neat API function (yet), so if you’re using the official release from when this was first posted you can’t do it. It looks like this:
This will break preexisting data if the prior validator wasn’t a strict subset of the new validator. Converting variables into a new representation sanely and easily is something I plan to add in.