I certainly understand your concern. But actual failures, as in the car won't move, are rare. The milling sound tends to get louder with time. That does not mean that the DU is going to fail in a day, or a week, or in months.
You are right, the bearing wear(cause of the drone and milling sound), won't cause a failure right away, but the bearings will eventually seize or come apart. The reason that we haven't seen many failures, is because Tesla replaces the DU once it hits a certain noise threshold(either car or owner).
I have an alternative mechanical explanation which I consider more likely. In the interest of quietness the pinion and spur gears are helical. This provides for a quieter torque transmission mesh. One downside is that is it slightly less efficient, but the larger downside is that end thrust is generated on both gear shafts. The torque in the forward and reverse directions causes end thrust in opposite directions. Hence when traveling forward the end thrust is in one direction and when in regen or reverse the end thrust is in the opposite direction. I have listened to the "milling" noise recordings and to me it sounded like the thrust bearing was starting get rough and to come apart. The thrust bearing balls or rollers will eventually spall if sufficiently overloaded and the non-round results will only get worse with further use. Note that reverse and regen apply an end thrust in the opposite direction, unloading the failing thrust bearing and loading the opposite thrust bearing. Hard acceleration would also accelerate the failure mode and increase the sound level. My estimate is that this is what is actually happening.
There are classes of radial ball bearings that are designed to resist axial loads. These are often used in pairs on milling machines, among other things. Their generic name is "angular contact" ball bearings. The angular contact ball bearing is a combination of a conventional ball bearing and a thrust bearing. My explanation is still applicable since changing from forward to reverse alters which angular contact face that is being loaded. I suspect that Tesla didn't quite complete the design by using FEA (Finite Element Analysis) to analyze the end thrust caused by the helical gear pair. The drive motors generate quite large torques relative to their IC counterparts.
Your statement about contamination is correct. However, the eventual failure due to overstressing a bearing depends on the usage history. For all ball bearings the balls distort with loading; the actual distortion being a function of the bearing design. A simple first approximation would be to assume that the ball will become an oblate sphere with loading. As the ball rotates the oblate spherical shape does not rotate and as a result the ball heats up due to the continually changing forces distorting its initially spherical shape. If the temperature build up is less than the rate at which the heat increase can dissipate through the surrounding structure (the ball race) the temperature will stabilize. It the heat created cannot be dissipated fast enough the temperature continually increases and it can reach high enough values that the materials start to weaken. Since higher loading and higher RPM both contribute to the temperature rise the time integral of both of those heat sources determine the useful life of the ball bearing. Pre-stressed angular contact bearings are used in lots of machinery. If the amount of pre-stress is too high or the loading is too high or the RPM is too high, etc. the lifetime will be significantly impacted.
