Try manually selecting the X
I've experienced a few issues related to receiving / playing a text message while Nav is active. I personally don't use the Nav voice directions, but have found that playing a text message overrides and turns Nav voice back on. So I'm curious if your experience might be somehow related to this bug, if you happen to be using the text message function.