If you can remove them and store them somewhere in your home, then later when you go to sell your car just pop out the all seasons and put in the factory-fresh carpet mats and your interior will look like new again. That's my suggestion, and the approach we have taken.
But more importantly: most/many all-weather floor mats are molded to fit the interior without the carpet mats, and won't give you a proper fit with the extra bulk of the carpet mats.