Exactly - and we both work from home full time so we do not bring the cars down low very often. Most charges are topping off from 60% or higher to 80%.Back to my own personal experience...I think 40A is certainly sufficient. Especially if, as you say, you are going to pretty much keep them plugged in all the time.
My regimen is to only plug in when the range falls below 100 miles (~30%) and charge to 80% (unless preparing for a trip). My current EVSE is only 30A, and that 50% charge usually completes in less than 6 hours.
If you are plugging in every day and replenishing your daily use, 40A charging will only take an hour or two. Definitely don't worry about "losing" the additional 8A unless your daily needs are extreme.