OK, then why was the test timed in the first place? You don’t care about speed, remember?
Someone who takes a very long time to get answers may be more likely to completely forget the information in the future, may be less able to use the information in real world situations, etc. even though he eventually managed to write the answer on the test. Someone who takes a long time to write the answer down because he has a disability may not have these problems.
So while we don’t care about speed by itself, we care about problems that are correlated with speed, and they may be less so for the disabled.
Someone who takes a very long time to get answers may be more likely to completely forget the information in the future, may be less able to use the information in real world situations, etc. even though he eventually managed to write the answer on the test. Someone who takes a long time to write the answer down because he has a disability may not have these problems.
So while we don’t care about speed by itself, we care about problems that are correlated with speed, and they may be less so for the disabled.