Mantra buffer issue

Recently got Mantra and am on version 1.1.0.

In Ableton, if I simply turn it on then off, my latency goes in the toilet and I have to restart Live to get the latency back down even though Mantra had been disabled.

Is there any way around this or a known Mantra or Live issue?

Thanks.