Any psychological test of response time that uses only consumer PC hardware will have difficulty with timing resolution and calibration. Platforms like operating systems, libraries, and virtual machines, and competition by other processes for processor time, can add difficulty. Some operating systems have a scheduler resolution like 10ms on an unloaded system, and they put input events into a queue without recording when the events happened. Even if an operating system does not do this, a cross-platform media I/O platform like Flash (used by Cognitive Fun, with an 18ms frame delay) or SDL (used by PEBL) might. Computer displays refresh around every 18ms, and the top part of a display refreshes earlier than the bottom part. The display technology may add delays which depend on the starting and ending light levels. The synchronization of sound stimuli with other timings depends on the sound hardware, drivers, and libraries. Input device hardware and communication protocols can cause other delays; for example, USB mice and keyboards are by default polled by the OS only every 8ms (though this can be changed in software).
If the experiment software only knows the time of an event up to the start and end of an interval during which the software didn’t have CPU time or input poll results, then normal statistical methods would need that interval to be converted to a point estimate of the event time. This can cause bias. Intervals should be interpreted as a source of relative likelihoods, such as can be used in a maximum-likelihood estimator of the parameters of a distribution of possible values, or a Bayesian model. (Check out the posterior mean reconstructions on page 10!)
(My brother works partly for the company that sells equipment, training, and interpretation services for the Test of Variables of Attention, which may be the continuous performance task implemented with the most attention to timing resolution. For frequent measurement of mental performance, the TOVA itself might not be worth the price (including $15 per test) outside of clinical ADHD assessment or personalized dose-response measurement, unless maybe you get a discount for non-commercial use, but he says “all the TOVA folks are very open-source and science/philosophy friendly and will help you out” with advice on using a computer to test mental performance (even if you’re trying to build a DIY free competitor). He suggests dropping in on Steve Hughes at one of their training workshops. The next one is in Los Angeles on December 4.)
Any psychological test of response time that uses only consumer PC hardware will have difficulty with timing resolution and calibration. Platforms like operating systems, libraries, and virtual machines, and competition by other processes for processor time, can add difficulty. Some operating systems have a scheduler resolution like 10ms on an unloaded system, and they put input events into a queue without recording when the events happened. Even if an operating system does not do this, a cross-platform media I/O platform like Flash (used by Cognitive Fun, with an 18ms frame delay) or SDL (used by PEBL) might. Computer displays refresh around every 18ms, and the top part of a display refreshes earlier than the bottom part. The display technology may add delays which depend on the starting and ending light levels. The synchronization of sound stimuli with other timings depends on the sound hardware, drivers, and libraries. Input device hardware and communication protocols can cause other delays; for example, USB mice and keyboards are by default polled by the OS only every 8ms (though this can be changed in software).
Free libraries which are more careful about timing resolution are psychopy (which uses pygame for sound and pyglet for everything else) and pyepl. More information on response timing resolution is in the psychopy documentation or the sections on timing and CRT stimulus presentation in the documentation for the commercial neuroscience experiment software Presentation.
If the experiment software only knows the time of an event up to the start and end of an interval during which the software didn’t have CPU time or input poll results, then normal statistical methods would need that interval to be converted to a point estimate of the event time. This can cause bias. Intervals should be interpreted as a source of relative likelihoods, such as can be used in a maximum-likelihood estimator of the parameters of a distribution of possible values, or a Bayesian model. (Check out the posterior mean reconstructions on page 10!)
(My brother works partly for the company that sells equipment, training, and interpretation services for the Test of Variables of Attention, which may be the continuous performance task implemented with the most attention to timing resolution. For frequent measurement of mental performance, the TOVA itself might not be worth the price (including $15 per test) outside of clinical ADHD assessment or personalized dose-response measurement, unless maybe you get a discount for non-commercial use, but he says “all the TOVA folks are very open-source and science/philosophy friendly and will help you out” with advice on using a computer to test mental performance (even if you’re trying to build a DIY free competitor). He suggests dropping in on Steve Hughes at one of their training workshops. The next one is in Los Angeles on December 4.)