I find it hard to believe that users would notice latency under 1ms. Probably not even under 5ms.

Have you tried buffering for 5ms? Was result bad? 1 ms?