Just thinking out loud here…
You have a speed of 800 steps/sec (incorrectly displayed on the web app as mm/sec).
So that’s 4 revolutions/sec, or one revolution every 250ms.
The encoder is 360 lines per revolution, so that’s one pulse every 694us. But it’s a quadrature encoder so that’s one edge every 175us. The tolerance is +/-45 degrees so two edges could occur within 86us.
A quick glance at the Arduino code and you’re sampling the encoders every 200us? Something must be wrong with my maths (quite likely)… or you’re having some challenges with the encoders