I have 5700 miles on my M3P in 2 months of ownership from our regular road trips. I knew the family had a limited threshold of discomfort before I started getting complaints, so I erred on the side of what I knew would be better overall (I.e both more comfortable and still provide more control during aggressive maneuvering), even if not as good as it could be from a racing perspective.
That, and
@MasterC17 mentioned in the big Road Course modification thread that he only 2 seconds on a >3 mile track going from stock suspension to Coilovers. I figure if I could split that difference I would be doing fine.
Okay, I think you'll be fine.
My best guess is your family actually would've been fine with the Sport kit too on the medium-to-softer damper settings, but surely the Comfort kit is more comfortable still and keeping the family happy with a shared family car is critical! Don't go too race car on a family car.
As a reference point, with my Redwood Öhlins "Performance Sport" kit - which I am guessing has similar spring rates as MPP's Sport kit - my wife really likes how it drives on a medium damper setting. After her first long drive with it, about 150 miles including lots of rural roads, twisties, good and bad pavement, etc she called it "smoother than [our Model S]." The complaints about me wasting time installing the suspension stopped after that first good drive with it.
With the dampers cranked up stiffer and riding as a passenger on extra rough city roads she was not so happy, but that's what adjustable dampers are for!
They make a real, useful difference as you've experienced. Before experiencing them I assumed I'd find one setting I liked and never adjust them again for street use, but actually I adjust them regularly depending on mood, who's driving, and who's going to be a passenger. Keeps the experience fresh and suited to whoever will be in the car.
So I think what I'm saying is you'll be happy with the Comforts, but if opportunity arises to swap for a Sport kit for a price that works for you, I think you should do it. (Assuming you're still doing autox or track days.)