I’m the creator of FSRS, an open-source spaced repetition algorithm which has been used in Anki, and my success using Anki in high school sparked my deep interest in spaced repetition algorithms.
Technical details
I improved my grades dramatically after using Anki for 1.5 years in high school, which helped me gain admission to a top-tier university in China. This experience inspired my research into spaced repetition scheduling. For more on my research as an undergraduate, see: How did I publish a paper in ACMKDD as an undergraduate? | by Jarrett Ye | Medium
But then, one commenter dismissed it as one of those ‘Things that sound cool on paper and then nobody actually implements them’. That comment really rubbed me the wrong way. So, determined to prove them wrong, I decided to implement the FSRS algorithm within Anki.
Technical details
At that point, I hadn’t used Anki in a while. In the meantime, its codebase had been rewritten in Rust, and its developers had introduced support for custom scheduling via JavaScript. As I was completely unfamiliar with Rust at the time, I opted to implement FSRS in Anki using its JavaScript-based custom scheduling script feature.
2022-08-30
I quickly encountered my first obstacle: custom scheduling didn’t support storing memory states directly in the cards, which is essential for implementing FSRS. I reported this issue on the Anki forum, and Anki’s lead developer, Dae, implemented the necessary functionality in Anki 2.1.55.
I quickly finished implementing a simplified version of the algorithm from my paper and released the scheduler’s code as open-source on GitHub. Following this, the Redditor who had initially dismissed it actually took back his words. Funnily enough, he went on to become one of the most active contributors within the FSRS community.
FSRS must learn an individual’s memory patterns from review history. I couldn’t run the optimizer inside Anki’s JavaScript scheduler or as an add-on, so I used Google Colab to host the machine-learning code. The FSRS optimizer and scheduler code were released on GitHub as FSRS v1.
2022-09-21
I built a Python-based FSRS simulator in Colab to test scheduling. This allowed me to see how the optimized FSRS would actually schedule reviews.
Technical details
Since I’m more comfortable with Python, I wrote a simulator notebook in Google Colab.
I refined the model, adding more parameters and using my paper’s post-lapse stability formula. Conveniently, this update aligned with the release of the Anki 2.1.55 Beta. This beta enabled storing custom data on cards through the custom scheduling script feature.
FSRS v1 used SuperMemo’s PLS formula, which didn’t fit my data well. I ported my paper’s PLS formula, added more parameters for initial stability and difficulty, and implemented difficulty mean reversion to avoid “ease hell,” increasing the total number of parameters from 7 to 14. Anki 2.1.55 Beta enabled storing custom data.
I created an add-on to read full review logs and accurately recalculate memory states.
Technical details
The script couldn’t access a card’s full history, so converting SM-2 data into FSRS state was approximate. Also, updating parameters led to compounding errors. I built the FSRS Helper add-on to parse logs, recompute memory states with current parameters, and adjust intervals.
Parsing JavaScript code from Python proved to be a major headache. I eventually settled on using regular expressions to directly extract the parameters from the custom scheduling script. The problem was that in FSRS v2, parameters were grouped based on the memory formulas they belonged to, which made regex matching quite complex. Therefore, I decided to store all parameters in a single flat array. While refactoring the code for this new parameter structure, I also took the opportunity to redesign the difficulty calculation within FSRS, drawing inspiration from SM-18′s difficulty formula.
After FSRS v3 launched, increased feedback led me to focus on implementing feature requests and fixing bugs. During this phase, I added the ‘suggested retention’ feature, designed to minimize review workload. It employs a simplified version of the SSP-MMC optimization method from my paper.
My experience with SuperMemo highlighted the value of its Advance and Postpone features. FSRS provided the capability to intelligently prioritize which specific cards would benefit most from being reviewed early or delayed. Consequently, I incorporated these two features into the FSRS Helper add-on.
Some users complained about significant fluctuations in their daily review workload, while others wanted to reduce their reviews on weekends. Although add-ons addressing these issues already existed, they often took a long time to take effect. FSRS, however, has the capability to modify card due dates and intervals in bulk during rescheduling. Acting on requests from several users, I integrated both ‘load balance’ and ‘free days’ features into the FSRS Helper add-on. The former helps to smooth out the daily review load, while the latter allows users to have fewer reviews scheduled on specific days of the week.
As positive feedback within the community grew, more and more Anki users started using FSRS. Consequently, Anki’s developer, Dae, began considering integrating FSRS directly into Anki. For me, this was undoubtedly the most exciting news, as it meant the most popular open-source spaced repetition software would potentially use the algorithm I had researched and developed. This also motivated me to plan further improvements for FSRS.
Introducing the calibration graph acted as a catalyst for community-driven improvements to the FSRS algorithm. From that point forward, several active contributors, along with myself, have put forward and tested dozens of improvement ideas.
Meanwhile, some users complained that FSRS was showing siblings closer to each other. I implemented the Disperse Siblings feature in the FSRS Helper add-on.
Hundreds of rounds of debate ensued among several dedicated users online, eventually resulting in some key ideas that significantly improved FSRS.
2023-06-09
I refactored the optimizer into a standalone Python package, added detailed evaluation, and introduced mini-batch support to speed up training by ~10×.
To aid community debugging and idea validation, I added detailed model evaluation. With contributor help, we also refactored the optimizer into a standalone, encapsulated Python package, greatly simplifying maintenance and development. Later, to boost optimization speed, I added mini-batch support, cutting training time by about 10x.
2023-07-13 (FSRS v4)
I released FSRS v4 with a power forgetting curve, improved formulas for calculating difficulty and memory stability, and with outlier filtering.
The FSRS difficulty calculation formula is quite simple, so we all thought there was obvious room for improvement there. However, most attempts failed.
I split the optimizer into its own repo and started defining a standard review-log format for broader adoption.
Technical details
To streamline development and maintenance, I extracted the optimizer code from the fsrs4anki repository into a dedicated one — fsrs-optimizer. Alongside this, I initiated the process of defining a standard format for spaced repetition review logs. The intention behind this standardization effort is to enable various SRS applications to adopt FSRS and leverage their respective user data for algorithm optimization.
My friend (Asuka Minato) and I began developing a Rust version of the optimizer. He had a strong foundation in Rust but lacked machine learning knowledge, while I had the ML background but didn’t know Rust. It seemed like a perfect match, so we decided to team up and develop a Rust version of the FSRS optimizer, specifically to prepare for the eventual integration of FSRS into Anki.
Technical details
Initially, we attempted using the tch crate. However, its dependency on libtorch resulted in a compiled file of around 200MB – nearly half the size of Anki itself – which was clearly unacceptable. This setback almost led us to abandon the Rust approach altogether. Following that, Minato recommended tinygrad to me. Since it doesn’t rely on torch, it seemed promising for potential use within Anki. But after persistent efforts, I found its performance was too poor and it was plagued by numerous bugs, forcing me to abandon that path as well.
After this, Minato stepped in again to help evaluate different crates. He explored dfdx, candle, and burn. Ultimately, burn turned out to be the most user-friendly and suitable for our needs. And so, the development of FSRS-rs officially got underway.
By the way, GPT-4 was incredibly useful for writing code back then. It allowed me, someone who knew absolutely no Rust, to use it to translate Python code into Rust. I also started learning Rust during this process, and Minato taught me quite a bit too. I estimate that about 60% of the initial FSRS-rs code was AI-generated.
2023-08-23
I found that the calibration graph could be gamed. This meant that metrics based solely on the calibration graph could be misleading. Log loss became the preferred gold standard metric.
I created a benchmark suite using 66 volunteer collections to evaluate FSRS and future models.
Technical details
During the FSRS v4 improvement process, we had already picked much of the ‘low-hanging fruit’, making further advancements increasingly difficult. Additionally, the dataset used for evaluating models at that time came only from a few active contributors, which made it hard to reliably validate smaller improvements. After discussing this with community members, I started working on creating a benchmark. The goal was to evaluate FSRS v4 and future improvements using the larger set of review data I had previously collected (which consisted of 66 collections at the time).
This benchmark also helped me align FSRS-rs with the FSRS-Optimizer, so that both produce near-identical results.
2023-09-08
After fixing some issues, FSRS-rs achieved full optimizer functionality and integration into Anki began.
Technical details
After several weeks of development, I encountered several bugs, which turned out to be upstream issues originating from the burn library. Following collaboration with burn’s developers to troubleshoot the problems, and thanks to patches submitted by community members, FSRS-rs finally implemented the optimizer functionality. The integration into Anki then began. Throughout this process, Anki’s developer, Dae, provided substantial help, for which I am incredibly grateful. Moreover, the timing perfectly coincided with FSRS’s first anniversary, and I shared this progress on r/Anki: In the 1st anniversary of FSRS, I want to share some progress of recent works. : r/Anki
I cannot summarize them here, but the key result was changing the forgetting curve’s shape to make it flatter.
2023-11-01
Anki 23.10 was released, marking the first official version with FSRS built-in. This means the number of users utilizing the FSRS algorithm is expected to grow rapidly. It also significantly increased FSRS’s visibility among developers, leading to the gradual emergence of FSRS algorithm libraries implemented in additional programming languages, and adoption by a growing number of other spaced repetition software.
I’m very grateful to Dae. Under Anki’s privacy policy allowing research use of review data, he provided raw data from 20,000 user collections containing a staggering 1.4 billion review logs – the largest dataset of its kind in the spaced repetition field.
Technical details
I converted all 20,000 files into the .csv format. With this dataset, I updated the SRS Benchmark and expanded it to include comparisons with additional models, aiming to uncover potential areas for improving FSRS.
My research on short-term review effects revealed a key finding: when users review a new card multiple times on the day it’s first learned, the sequence of ratings significantly influences the card’s initial stability. This insight subsequently led to the approach in FSRS-5 of using same-day reviews to update stability.
To make metrics intuitive and harder to cheat, I redesigned RMSE(bins).
Technical details
As I mentioned before, the calibration graph is hackable, so we primarily used log loss as the main metric. However, log loss is not intuitive. So I redesigned RMSE(bins) based on a new binning method which is hard to cheat and easy to read.
After researching short-term memory models for several months, I gave up. The key lesson learned from trying to predict short-term memory with FSRS was that the working mechanisms of short-term and long-term memory are quite different. Ultimately, I adopted a simplified approach: using short-term reviews to refine predictions related to long-term memory.
Technical details
The outcomes of the short-term reviews themselves were not used for model optimization. In other words, I included the logs of short-term reviews in the time-series features but excluded them from the labels used for training. Furthermore, because there was no dedicated short-term memory model, I also ignored the specific time intervals of these short-term reviews. This simplified solution resulted in a slight reduction in FSRS’s prediction error for long-term retention. But it was still not worth a major version update.
I modeled initial difficulty as an exponential function of initial rating, slightly improving the accuracy of FSRS.
Technical details
While analyzing the distribution of FSRS parameters, I noticed that the initial stability corresponding to the ‘easy’ button was very high. Specifically, the difference (or gap) between the initial stability for ‘easy’ and ‘good’ was much larger than the difference between the stability for ‘good’ and ‘hard’. The same pattern held for the gap between initial stability for ‘hard’ and ‘again’. This led me to hypothesize that initial difficulty might follow a similar pattern. Consequently, I conducted an experiment where I modeled initial difficulty as an exponential function of the initial rating. The results indeed showed a slight reduction in FSRS’s error.
I updated the simulator to approximate short-term reviews by averaging counts and ratings per day.
Technical details
Updating the FSRS simulator to account for FSRS-5′s consideration of short-term reviews presented a challenge. The existing simulator functioned on a day-by-day basis and, lacking a short-term memory model, couldn’t simulate the nuances of multiple reviews within the same day. My solution was a simplification: instead of simulating each short-term review individually, I decided to represent them collectively. This involved calculating the average count and average rating of a user’s typical short-term reviews (calculated per learning step) and treating that aggregate as a single event in the simulation. This effectively bypassed the need for a major simulator overhaul. To perform the analysis required to obtain these average figures, I set up a dedicated repository:
I released FSRS 5, adding short-term review effects and improved initial difficulty, cutting prediction error by ~4%.
Technical details
After completing all the necessary corresponding updates and benchmarks, I released FSRS-5. This version accounted for the effects of short-term reviews and included an improved initial difficulty calculation, adding two parameters in total. Compared to FSRS-4.5, it reduced prediction error by approximately 4%. At the time, I believed this would be the final version of FSRS. This was because community members and I had subsequently attempted dozens more improvement methods, all of which, without exception, proved ineffective.
As discussions about FSRS grew more frequent, the FSRS Megathread was created on the Anki Discord server to provide a centralized place for these conversations. This has attracted more contributors and generated more ideas for improving FSRS.
A contributor refactored the Rust simulator, boosting speed by ~8 times.
Technical details
Originally, the FSRS-rs simulator closely mirrored its Python counterpart. But there was a key difference: the Python version utilized Numpy for efficient parallel processing optimized at a daily granularity, an optimization missing in the Rust implementation. Thanks to contributions from a community member, the FSRS-rs simulator was then refactored to operate at the card level granularity. I made sure during the refactor that this change didn’t alter the simulation outcomes compared to the day-level approach. The end result of this refactoring was a significant performance boost, speeding up simulations by almost 8 times.
I implemented damping on difficulty updates, making difficulty approach its maximum value more slowly. It unexpectedly reduced error by ~1%.
Technical details
An FSRS user observed that many of their cards were rapidly reaching the maximum difficulty value of 10. This significantly reduced the difficulty metric’s ability to differentiate between cards, offering poor granularity for sorting them. Consequently, they proposed adding damping to the difficulty update process, such that the magnitude of the update decreases as the difficulty (D) approaches 10.
Benchmarking conducted by our community members revealed that this approach surprisingly reduced prediction error by about 1%, without introducing any additional parameters. However, while implementing this method, I encountered an issue: the damping effect was bidirectional. This meant that as D neared 10, both increases and decreases in difficulty would be dampened (reduced in magnitude). This created a situation potentially analogous to ‘ease hell’, where difficulty could get stuck at high values. Yet, when I implemented unidirectional damping (only slowing down increases but not decreases), the improvement in metrics disappeared.
This led me to reconsider: perhaps ‘ease hell’ isn’t actually the problem it’s often made out to be. Most attempts to specifically eliminate it seemed to negatively impact the metrics. Ultimately, despite the potential drawback of bidirectional damping, I decided to implement that version in FSRS-5 due to the positive benchmark results.
With Dae’s help, we released a new Anki dataset. It contains 10,000 collections with note, deck, and preset IDs for more detailed analysis.
Technical details
The motivation for this came from my analysis of the 20k dataset, where I noticed that some users’ forgetting curves were not monotonic. These looked like the result of mixing curves from different learning materials and study options. To investigate this issue further, I needed to know which decks the different cards belonged to and whether those decks used different preset configurations. Ultimately, we added Note, Deck, and Preset IDs to the new dataset. This makes it possible to analyze things like the interactions between different cards originating from the same note, the effects of optimizing parameters separately for different decks, and more.
Due to the slow progress in developing a short-term memory model, I considered adding statistical analysis of short-term reviews to the FSRS Helper add-on. The goal is to help users quantify their own short-term memory and provide them with data they can use to adjust their learning steps.
I added recency weighting to the optimizer, penalizing FSRS more for bad predictions on newer, more recent reviews and penalizing it less for bad predictions on older reviews. This reduced prediction error by ~4.5%.
Technical details
Feeling fatigued by refining the model structure, I started revisiting past experiments. I rediscovered one where I had experimented with assigning different weights to review samples: ‣
This prompted me to reconsider focusing on the optimization process itself – potentially improving model performance without altering its architecture. The use of TimeSeriesSplit in the SRS Benchmark reminded me that users’ memory patterns can evolve over time (e.g., due to learning different materials or changing study habits). This led to the hypothesis: perhaps giving higher weight to more recent data could improve the model’s predictive performance on future reviews? Through discussions with Claude, I learned this approach is known as ‘Recency weighting’.
Following suggestions from community members, I then tested various weighting functions, ultimately achieving a reduction in prediction error of around 4.5%. Then I implemented it in FSRS-rs.
The History of FSRS for Anki
Link post
Background
I’m the creator of FSRS, an open-source spaced repetition algorithm which has been used in Anki, and my success using Anki in high school sparked my deep interest in spaced repetition algorithms.
Technical details
I improved my grades dramatically after using Anki for 1.5 years in high school, which helped me gain admission to a top-tier university in China. This experience inspired my research into spaced repetition scheduling. For more on my research as an undergraduate, see: How did I publish a paper in ACMKDD as an undergraduate? | by Jarrett Ye | Medium
My full paper is available here: A Stochastic Shortest Path Algorithm for Optimizing Spaced Repetition Scheduling | Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
I’m doing memory research at MaiMemo Inc.
2022
2022-08-19
Everything began with a post I made on Reddit. After my paper was accepted by ACM SIGKDD, I posted about it on the r/Anki:
A Stochastic Shortest Path Algorithm for Optimizing Spaced Repetition Scheduling | Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining : r/Anki
But then, one commenter dismissed it as one of those ‘Things that sound cool on paper and then nobody actually implements them’. That comment really rubbed me the wrong way. So, determined to prove them wrong, I decided to implement the FSRS algorithm within Anki.
Technical details
At that point, I hadn’t used Anki in a while. In the meantime, its codebase had been rewritten in Rust, and its developers had introduced support for custom scheduling via JavaScript. As I was completely unfamiliar with Rust at the time, I opted to implement FSRS in Anki using its JavaScript-based custom scheduling script feature.
2022-08-30
I quickly encountered my first obstacle: custom scheduling didn’t support storing memory states directly in the cards, which is essential for implementing FSRS. I reported this issue on the Anki forum, and Anki’s lead developer, Dae, implemented the necessary functionality in Anki 2.1.55.
Discussion: Some problems in implementing a state-of-the-art SRS scheduler on Anki—Anki / Scheduling—Anki Forums
2022-09-08
I quickly finished implementing a simplified version of the algorithm from my paper and released the scheduler’s code as open-source on GitHub. Following this, the Redditor who had initially dismissed it actually took back his words. Funnily enough, he went on to become one of the most active contributors within the FSRS community.
Implement a new spaced repetition algorithm based on anki custom scheduling. : r/Anki
2022-09-18 (FSRS v1)
I added an optimizer via Google Colab, creating the first usable FSRS version.
New progress in implementing the custom algorithm. : r/Anki
Technical details
FSRS must learn an individual’s memory patterns from review history. I couldn’t run the optimizer inside Anki’s JavaScript scheduler or as an add-on, so I used Google Colab to host the machine-learning code. The FSRS optimizer and scheduler code were released on GitHub as FSRS v1.
2022-09-21
I built a Python-based FSRS simulator in Colab to test scheduling. This allowed me to see how the optimized FSRS would actually schedule reviews.
Technical details
Since I’m more comfortable with Python, I wrote a simulator notebook in Google Colab.
Release v1.1.0: run notebook in local & format outputs & add intro of simulator (#5) · open-spaced-repetition/fsrs4anki
2022-09-28 (FSRS v2)
I refined the model, adding more parameters and using my paper’s post-lapse stability formula. Conveniently, this update aligned with the release of the Anki 2.1.55 Beta. This beta enabled storing custom data on cards through the custom scheduling script feature.
Anki 2.1.55 Beta is now available. : r/Anki
Technical details
FSRS v1 used SuperMemo’s PLS formula, which didn’t fit my data well. I ported my paper’s PLS formula, added more parameters for initial stability and difficulty, and implemented difficulty mean reversion to avoid “ease hell,” increasing the total number of parameters from 7 to 14. Anki 2.1.55 Beta enabled storing custom data.
Release v2.0.0 · open-spaced-repetition/fsrs4anki
2022-10-05 (FSRS v3 & Helper add-on)
I created an add-on to read full review logs and accurately recalculate memory states.
Technical details
The script couldn’t access a card’s full history, so converting SM-2 data into FSRS state was approximate. Also, updating parameters led to compounding errors. I built the FSRS Helper add-on to parse logs, recompute memory states with current parameters, and adjust intervals.
Parsing JavaScript code from Python proved to be a major headache. I eventually settled on using regular expressions to directly extract the parameters from the custom scheduling script. The problem was that in FSRS v2, parameters were grouped based on the memory formulas they belonged to, which made regex matching quite complex. Therefore, I decided to store all parameters in a single flat array. While refactoring the code for this new parameter structure, I also took the opportunity to redesign the difficulty calculation within FSRS, drawing inspiration from SM-18′s difficulty formula.
FSRS v3 had 13 parameters, while FSRS v2 had 14.
FSRS v3 release: Big update in FSRS4Anki v3.0.0 : r/Anki
Add-on: ⚙FSRS Helper (Postpone & Advance & Load Balance & Easy Days & Disperse Siblings) - AnkiWeb
2022-10-18
I started collecting review data for SRS research from volunteers.
Data collection form: Collect review data for SRS research.
2022-11-16
After FSRS v3 launched, increased feedback led me to focus on implementing feature requests and fixing bugs. During this phase, I added the ‘suggested retention’ feature, designed to minimize review workload. It employs a simplified version of the SSP-MMC optimization method from my paper.
New features of FSRS4Anki from v3.0.0 to v3.6.0 : r/Anki
Introduce recent changes of FSRS4Anki, and want to collect some feedback : r/Anki
2023
2023-01-28
My experience with SuperMemo highlighted the value of its Advance and Postpone features. FSRS provided the capability to intelligently prioritize which specific cards would benefit most from being reviewed early or delayed. Consequently, I incorporated these two features into the FSRS Helper add-on.
Let your review be freer: postpone & advance cards via FSRS4Anki Helper : r/Anki
2023-02-11
Some users complained about significant fluctuations in their daily review workload, while others wanted to reduce their reviews on weekends. Although add-ons addressing these issues already existed, they often took a long time to take effect. FSRS, however, has the capability to modify card due dates and intervals in bulk during rescheduling. Acting on requests from several users, I integrated both ‘load balance’ and ‘free days’ features into the FSRS Helper add-on. The former helps to smooth out the daily review load, while the latter allows users to have fewer reviews scheduled on specific days of the week.
Load Balance & Free Weekend have been implemented in the FSRS4Anki helper add-on! : r/Anki
2023-03-16
As positive feedback within the community grew, more and more Anki users started using FSRS. Consequently, Anki’s developer, Dae, began considering integrating FSRS directly into Anki. For me, this was undoubtedly the most exciting news, as it meant the most popular open-source spaced repetition software would potentially use the algorithm I had researched and developed. This also motivated me to plan further improvements for FSRS.
Integrate FSRS into Anki as an optional feature · Issue #2443 · ankitects/anki
2023-04-12
To identify FSRS’s weaknesses intuitively, I introduced the calibration graph into the optimizer.
Feat/Calibration graph by L-M-Sherlock · Pull Request #212 · open-spaced-repetition/fsrs4anki
2023-04-16
Introducing the calibration graph acted as a catalyst for community-driven improvements to the FSRS algorithm. From that point forward, several active contributors, along with myself, have put forward and tested dozens of improvement ideas.
Meanwhile, some users complained that FSRS was showing siblings closer to each other. I implemented the Disperse Siblings feature in the FSRS Helper add-on.
Calibration between actual retention and predicted retention is not great · Issue #215 · open-spaced-repetition/fsrs4anki
Feat/disperse siblings by L-M-Sherlock · Pull Request #61 · open-spaced-repetition/fsrs4anki-helper
2023-04-30
Remember the commenter I mentioned at the start? They sparked these incredible discussion threads.
[Feature Request] Sharing ideas for further improvement of the algorithm · Issue #239 · open-spaced-repetition/fsrs4anki
[Feature Request] Improving the algorithm, continuation · Issue #282 · open-spaced-repetition/fsrs4anki
Hundreds of rounds of debate ensued among several dedicated users online, eventually resulting in some key ideas that significantly improved FSRS.
2023-06-09
I refactored the optimizer into a standalone Python package, added detailed evaluation, and introduced mini-batch support to speed up training by ~10×.
Main updates of FSRS4Anki from v3.7.0 to v3.23.0 : r/Anki
Technical details
To aid community debugging and idea validation, I added detailed model evaluation. With contributor help, we also refactored the optimizer into a standalone, encapsulated Python package, greatly simplifying maintenance and development. Later, to boost optimization speed, I added mini-batch support, cutting training time by about 10x.
2023-07-13 (FSRS v4)
I released FSRS v4 with a power forgetting curve, improved formulas for calculating difficulty and memory stability, and with outlier filtering.
Technical details
Major changes:
Exponential → power-law forgetting curve
hard_penalty & easy_bonus parameters
Four separate initial stability parameters
Pre-training on first reviews
Outlier filter
Best-epoch parameter selection
Parameter count rose from 13 to 17.
Release v4.0.0 · open-spaced-repetition/fsrs4anki
2023-07-14
The FSRS difficulty calculation formula is quite simple, so we all thought there was obvious room for improvement there. However, most attempts failed.
[Enhancement] Improving the function for calculating difficulty · Issue #352 · open-spaced-repetition/fsrs4anki
2023-07-29 (FSRS-Optimizer)
I split the optimizer into its own repo and started defining a standard review-log format for broader adoption.
Technical details
To streamline development and maintenance, I extracted the optimizer code from the fsrs4anki repository into a dedicated one — fsrs-optimizer. Alongside this, I initiated the process of defining a standard format for spaced repetition review logs. The intention behind this standardization effort is to enable various SRS applications to adopt FSRS and leverage their respective user data for algorithm optimization.
FSRS-Optimizer on PyPI: FSRS-Optimizer·PyPI
2023-08-17 (FSRS-rs)
My friend (Asuka Minato) and I began developing a Rust version of the optimizer. He had a strong foundation in Rust but lacked machine learning knowledge, while I had the ML background but didn’t know Rust. It seemed like a perfect match, so we decided to team up and develop a Rust version of the FSRS optimizer, specifically to prepare for the eventual integration of FSRS into Anki.
Technical details
Initially, we attempted using the
tchcrate. However, its dependency onlibtorchresulted in a compiled file of around 200MB – nearly half the size of Anki itself – which was clearly unacceptable. This setback almost led us to abandon the Rust approach altogether. Following that, Minato recommendedtinygradto me. Since it doesn’t rely on torch, it seemed promising for potential use within Anki. But after persistent efforts, I found its performance was too poor and it was plagued by numerous bugs, forcing me to abandon that path as well.After this, Minato stepped in again to help evaluate different crates. He explored
dfdx,candle, andburn. Ultimately,burnturned out to be the most user-friendly and suitable for our needs. And so, the development of FSRS-rs officially got underway.WIP/rewrite FSRS in burn · open-spaced-repetition/fsrs-rs@a9cc7df
From Asuka Minato’s perspective: 陪伴是最长情的告白(contribute to anki)
By the way, GPT-4 was incredibly useful for writing code back then. It allowed me, someone who knew absolutely no Rust, to use it to translate Python code into Rust. I also started learning Rust during this process, and Minato taught me quite a bit too. I estimate that about 60% of the initial FSRS-rs code was AI-generated.
2023-08-23
I found that the calibration graph could be gamed. This meant that metrics based solely on the calibration graph could be misleading. Log loss became the preferred gold standard metric.
Calibration graph can be cheated by the algorithm which always predicts the average. · Issue #1 · open-spaced-repetition/spaced-repetition-algorithm-metric
2023-09-06 (SRS Benchmark)
I created a benchmark suite using 66 volunteer collections to evaluate FSRS and future models.
Technical details
During the FSRS v4 improvement process, we had already picked much of the ‘low-hanging fruit’, making further advancements increasingly difficult. Additionally, the dataset used for evaluating models at that time came only from a few active contributors, which made it hard to reliably validate smaller improvements. After discussing this with community members, I started working on creating a benchmark. The goal was to evaluate FSRS v4 and future improvements using the larger set of review data I had previously collected (which consisted of 66 collections at the time).
[Doc] Introduction for FSRS v4 · Issue #351 · open-spaced-repetition/fsrs4anki
The first commit of SRS Benchmark: build dataset from anki file · open-spaced-repetition/srs-benchmark@450ee90
This benchmark also helped me align FSRS-rs with the FSRS-Optimizer, so that both produce near-identical results.
2023-09-08
After fixing some issues, FSRS-rs achieved full optimizer functionality and integration into Anki began.
Technical details
After several weeks of development, I encountered several bugs, which turned out to be upstream issues originating from the
burnlibrary. Following collaboration withburn’s developers to troubleshoot the problems, and thanks to patches submitted by community members, FSRS-rs finally implemented the optimizer functionality. The integration into Anki then began. Throughout this process, Anki’s developer, Dae, provided substantial help, for which I am incredibly grateful. Moreover, the timing perfectly coincided with FSRS’s first anniversary, and I shared this progress on r/Anki: In the 1st anniversary of FSRS, I want to share some progress of recent works. : r/AnkiIntegrate the FSRS optimizer by dae · Pull Request #2633 · ankitects/anki
Integrate FSRS into Anki by dae · Pull Request #2654 · ankitects/anki
2023-09-14
Again, hundreds of rounds of debate ensued.
[Feature Request] Ideas to further improve the accuracy of the algorithm · Issue #461 · open-spaced-repetition/fsrs4anki
I cannot summarize them here, but the key result was changing the forgetting curve’s shape to make it flatter.
2023-11-01
Anki 23.10 was released, marking the first official version with FSRS built-in. This means the number of users utilizing the FSRS algorithm is expected to grow rapidly. It also significantly increased FSRS’s visibility among developers, leading to the gradual emergence of FSRS algorithm libraries implemented in additional programming languages, and adoption by a growing number of other spaced repetition software.
Release 23.10 · ankitects/anki
2023-11-22 (Dataset from Anki)
I’m very grateful to Dae. Under Anki’s privacy policy allowing research use of review data, he provided raw data from 20,000 user collections containing a staggering 1.4 billion review logs – the largest dataset of its kind in the spaced repetition field.
Technical details
I converted all 20,000 files into the
.csvformat. With this dataset, I updated the SRS Benchmark and expanded it to include comparisons with additional models, aiming to uncover potential areas for improving FSRS.[Feature Request] More accurate default parameters using Anki user’s data and help from Dae · Issue #493 · open-spaced-repetition/fsrs4anki
Updating the benchmark with new data · Issue #14 · open-spaced-repetition/srs-benchmark
Feat/re-benchmark with 20k collections by L-M-Sherlock · Pull Request #16 · open-spaced-repetition/srs-benchmark
2023-12-26 (FSRS 4.5)
Based on the earlier debates and analysis, the flatter forgetting curve idea was accepted, and I released FSRS-4.5 incorporating this change.
Feat/update to FSRS-4.5 by L-M-Sherlock · Pull Request #568 · open-spaced-repetition/fsrs4anki
2024
2024-01-06
My research on short-term review effects revealed a key finding: when users review a new card multiple times on the day it’s first learned, the sequence of ratings significantly influences the card’s initial stability. This insight subsequently led to the approach in FSRS-5 of using same-day reviews to update stability.
First day’s series of ratings may have significant impact on initial stability · Issue #2 · open-spaced-repetition/short-term-memory-research
2024-01-29
I released FSRS-rs v0.1.0 to crates.io.
Release v0.1.0 · open-spaced-repetition/fsrs-rs
fsrs—crates.io: Rust Package Registry
2024-02-23
With the release of AnkiDroid 2.17.0, native FSRS support was complete across all major platforms: desktop, iOS, and Android.
AnkiDroid Changelog
2024-02-24 (FSRS-Anki-20k)
To attract more researchers, I released the dataset of 20,000 Anki collections used for FSRS development, naming it FSRS-Anki-20k.
open-spaced-repetition/FSRS-Anki-20k · Datasets at Hugging Face
2024-03-01
To make metrics intuitive and harder to cheat, I redesigned RMSE(bins).
Technical details
As I mentioned before, the calibration graph is hackable, so we primarily used log loss as the main metric. However, log loss is not intuitive. So I redesigned RMSE(bins) based on a new binning method which is hard to cheat and easy to read.
Feat/RMSE based on R-Matrix by L-M-Sherlock · Pull Request #87 · open-spaced-repetition/fsrs-optimizer
The Metric · open-spaced-repetition/fsrs4anki Wiki
2024-04-06
After researching short-term memory models for several months, I gave up. The key lesson learned from trying to predict short-term memory with FSRS was that the working mechanisms of short-term and long-term memory are quite different. Ultimately, I adopted a simplified approach: using short-term reviews to refine predictions related to long-term memory.
Technical details
The outcomes of the short-term reviews themselves were not used for model optimization. In other words, I included the logs of short-term reviews in the time-series features but excluded them from the labels used for training. Furthermore, because there was no dedicated short-term memory model, I also ignored the specific time intervals of these short-term reviews. This simplified solution resulted in a slight reduction in FSRS’s prediction error for long-term retention. But it was still not worth a major version update.
Feat/FSRS-5 by L-M-Sherlock · Pull Request #114 · open-spaced-repetition/fsrs-optimizer
2024-05-17
I modeled initial difficulty as an exponential function of initial rating, slightly improving the accuracy of FSRS.
Technical details
While analyzing the distribution of FSRS parameters, I noticed that the initial stability corresponding to the ‘easy’ button was very high. Specifically, the difference (or gap) between the initial stability for ‘easy’ and ‘good’ was much larger than the difference between the stability for ‘good’ and ‘hard’. The same pattern held for the gap between initial stability for ‘hard’ and ‘again’. This led me to hypothesize that initial difficulty might follow a similar pattern. Consequently, I conducted an experiment where I modeled initial difficulty as an exponential function of the initial rating. The results indeed showed a slight reduction in FSRS’s error.
Feat/FSRS-5 by L-M-Sherlock · Pull Request #114 · open-spaced-repetition/fsrs-optimizer
2024-06-13
I updated the simulator to approximate short-term reviews by averaging counts and ratings per day.
Technical details
Updating the FSRS simulator to account for FSRS-5′s consideration of short-term reviews presented a challenge. The existing simulator functioned on a day-by-day basis and, lacking a short-term memory model, couldn’t simulate the nuances of multiple reviews within the same day. My solution was a simplification: instead of simulating each short-term review individually, I decided to represent them collectively. This involved calculating the average count and average rating of a user’s typical short-term reviews (calculated per learning step) and treating that aggregate as a single event in the simulation. This effectively bypassed the need for a major simulator overhaul. To perform the analysis required to obtain these average figures, I set up a dedicated repository:
open-spaced-repetition/Anki-button-usage: A preliminary analysis about the button usage in Anki dataset
Feat/FSRS-5 by L-M-Sherlock · Pull Request #114 · open-spaced-repetition/fsrs-optimizer
2024-07-10 (FSRS 5)
I released FSRS 5, adding short-term review effects and improved initial difficulty, cutting prediction error by ~4%.
Technical details
After completing all the necessary corresponding updates and benchmarks, I released FSRS-5. This version accounted for the effects of short-term reviews and included an improved initial difficulty calculation, adding two parameters in total. Compared to FSRS-4.5, it reduced prediction error by approximately 4%. At the time, I believed this would be the final version of FSRS. This was because community members and I had subsequently attempted dozens more improvement methods, all of which, without exception, proved ineffective.
Feat/FSRS-5 by L-M-Sherlock · Pull Request #98 · open-spaced-repetition/srs-benchmark
Feat/FSRS-5 by L-M-Sherlock · Pull Request #114 · open-spaced-repetition/fsrs-optimizer
Feat/FSRS-5 by L-M-Sherlock · Pull Request #197 · open-spaced-repetition/fsrs-rs
2024-09-07 (FSRS Megathread)
As discussions about FSRS grew more frequent, the FSRS Megathread was created on the Anki Discord server to provide a centralized place for these conversations. This has attracted more contributors and generated more ideas for improving FSRS.
https://discord.com/channels/368267295601983490/1282005522513530952
2024-10-11
A contributor refactored the Rust simulator, boosting speed by ~8 times.
Technical details
Originally, the FSRS-rs simulator closely mirrored its Python counterpart. But there was a key difference: the Python version utilized Numpy for efficient parallel processing optimized at a daily granularity, an optimization missing in the Rust implementation. Thanks to contributions from a community member, the FSRS-rs simulator was then refactored to operate at the card level granularity. I made sure during the refactor that this change didn’t alter the simulation outcomes compared to the day-level approach. The end result of this refactoring was a significant performance boost, speeding up simulations by almost 8 times.
Make simulate iterate by card instead of by day. by Luc-Mcgrady · Pull Request #235 · open-spaced-repetition/fsrs-rs
2024-10-17
I implemented damping on difficulty updates, making difficulty approach its maximum value more slowly. It unexpectedly reduced error by ~1%.
Technical details
An FSRS user observed that many of their cards were rapidly reaching the maximum difficulty value of 10. This significantly reduced the difficulty metric’s ability to differentiate between cards, offering poor granularity for sorting them. Consequently, they proposed adding damping to the difficulty update process, such that the magnitude of the update decreases as the difficulty (D) approaches 10.
Benchmarking conducted by our community members revealed that this approach surprisingly reduced prediction error by about 1%, without introducing any additional parameters. However, while implementing this method, I encountered an issue: the damping effect was bidirectional. This meant that as D neared 10, both increases and decreases in difficulty would be dampened (reduced in magnitude). This created a situation potentially analogous to ‘ease hell’, where difficulty could get stuck at high values. Yet, when I implemented unidirectional damping (only slowing down increases but not decreases), the improvement in metrics disappeared.
This led me to reconsider: perhaps ‘ease hell’ isn’t actually the problem it’s often made out to be. Most attempts to specifically eliminate it seemed to negatively impact the metrics. Ultimately, despite the potential drawback of bidirectional damping, I decided to implement that version in FSRS-5 due to the positive benchmark results.
Suggestion for Adjusting Difficulty Score to Use an Asymptote at 10 · Issue #697 · open-spaced-repetition/fsrs4anki
2024-11-05 (anki-revlogs-10k)
With Dae’s help, we released a new Anki dataset. It contains 10,000 collections with note, deck, and preset IDs for more detailed analysis.
Technical details
The motivation for this came from my analysis of the 20k dataset, where I noticed that some users’ forgetting curves were not monotonic. These looked like the result of mixing curves from different learning materials and study options. To investigate this issue further, I needed to know which decks the different cards belonged to and whether those decks used different preset configurations. Ultimately, we added Note, Deck, and Preset IDs to the new dataset. This makes it possible to analyze things like the interactions between different cards originating from the same note, the effects of optimizing parameters separately for different decks, and more.
open-spaced-repetition/anki-revlogs-10k · Datasets at Hugging Face
2024-11-10 (Steps Stats)
Due to the slow progress in developing a short-term memory model, I considered adding statistical analysis of short-term reviews to the FSRS Helper add-on. The goal is to help users quantify their own short-term memory and provide them with data they can use to adjust their learning steps.
Feat/step stats by L-M-Sherlock · Pull Request #487 · open-spaced-repetition/fsrs4anki-helper
New Feature: Quantify Your Short-Term Memory in Detail. : r/Anki
Recommended (re)learning steps powered by FSRS Helper : r/Anki
2024-12-30 (FSRS-5 recency)
I added recency weighting to the optimizer, penalizing FSRS more for bad predictions on newer, more recent reviews and penalizing it less for bad predictions on older reviews. This reduced prediction error by ~4.5%.
Technical details
Feeling fatigued by refining the model structure, I started revisiting past experiments. I rediscovered one where I had experimented with assigning different weights to review samples: ‣
This prompted me to reconsider focusing on the optimization process itself – potentially improving model performance without altering its architecture. The use of TimeSeriesSplit in the SRS Benchmark reminded me that users’ memory patterns can evolve over time (e.g., due to learning different materials or changing study habits). This led to the hypothesis: perhaps giving higher weight to more recent data could improve the model’s predictive performance on future reviews? Through discussions with Claude, I learned this approach is known as ‘Recency weighting’.
Consequently, I implemented this method in the optimizer and benchmarked it. The results indicated that this method reduced prediction error by another 3%: https://discord.com/channels/368267295601983490/1282005522513530952/1318519440647655445
Following suggestions from community members, I then tested various weighting functions, ultimately achieving a reduction in prediction error of around 4.5%. Then I implemented it in FSRS-rs.
Feat/support recency weighting by L-M-Sherlock · Pull Request #260 · open-spaced-repetition/fsrs-rs
2025
FSRS-6 is coming. To be continued.