Even if the duel lasted only a few minutes and everything was over by, say, 12:10 or 12:20, that would mean Quirrel only waited six hours and 13-23 minutes, depending. Could even be deliberate—an attempt to throw suspicion off himself by making the timing not quite perfect.
If by “not quite perfect” you mean “suspiciously close after the point that made observation with a Time Turner impossible”, then yes, it’s not quite perfect.
If by “not quite perfect” you mean “suspiciously close after the point that made observation with a Time Turner impossible”, then yes, it’s not quite perfect.
One wonders whether it was possible to cast the Blood-Chilling Charm such that it would take seven and a half hours to get Draco to death’s door.