There is indeed some evidence, though only indirect, that the car modulates the degree of braking based on the nearness of any following vehicle (which would after all be sensible). Also, the term "phantom braking" has no precise definition .. to some it means a very sudden hard stop (or near stop), while to other it means any slow-down by the car (even small) that appears to have no cause. As a result no-one really knows the actual incidence of "hard" PB events and thus the severity of the problem. Almost certainly there is some of the usual "internet magnification" going on as well, which makes it even harder to make any unbiased inferences. However, as you note, if PB is truly as bad as some claim, then where are all the accidents? Given the hunger in the press for anything sensational related to Tesla, you can be sure they would be covered.
There is also the issue of erroneous reporting. If we define a PB event as the car braking with no external cause, how certain can we be sure that any given braking event IS phantom? It's quite possible (perhaps even likely) that the car saw a real threat that the driver missed, which causes a correct response by the car to be categorized by the driver as PB, even though it wasn't.