I'm pretty sure this is a bluetooth absolute volume issue. If you own an iOS phone you are screwed because Apple doesn't allow it to be disabled. If you own an Android phone just go into developer settings and disable absolute volume.
Personally, I hate absolute volume. It is supposed to simplify volume control but it always caused me far more headaches than it solved. I prefer having the phone's bluetooth volume independent from the bluetooth device. With it OFF your phone remembers the last volume it was set to for each individual device it connects to. For my cars that's always max output from the phone where I then control the volume I hear via the car's volume control. For headphones I prefer to have max volume on the headphones where I then control the volume I hear via the phone's volume.