Hey bud! I’ve got the same specs as you describe. I use a focusrite scarlett 2nd gen.
The minimum latency I play with is around 2 - 1.5 ms. Try decreasing the sample rate until you get a desired result ( lowering this off limits can lead to crackles in the output sound).
I’ve been playing live with this rig and I’ve got no issues yet!