...Can you drive on track mode all the time?...
I have no idea about racing and I've never been in on a "Performance" model but I am not sure what you are asking.
I assume as long as you can click on your 15" screen, you are on track mode. I don't imagine Tesla has geofenced so drivers can't use track mode outside of race tracks.
So, should the answer be "Yes, you can" but I guess what you mean is: If you can, then should you?
...Will it be more wear on the batteries and brakes?...
I assume yes because the focus of track mode is performance while energy savings and brake savings are secondary.
However, you still get the full regen even if you manually use brakes more often in this mode.
...What’s the downside of driving in Track Mode on my day to day daily drives?...
Like you mentioned above, energy consumption is devoted for performance. If it can boost power to your 2 motors, it would with no hesitation on energy savings.
Also, there would be less ABS interventions to give you more controls on braking.
Also, there would be less traction-control which gives you more slide actions that race drivers love.
Those are great for the track but I am not sure the police would like to see you sliding toward their patrol car