Yes, that may well be part of it.
Discussed previously in more detail, but I've been convinced for coming up on 2 years now that the total amount of metadata one has for all tracks on the USB device (not the number of tracks or the size of all the files combined, but the physical length of filenames and associated metadata (title, artist names, art, etc.) the interface keeps track of, has a lot to do with it. ...and as you say, the more memory someone has consumed with Nav History, Phone Contact data, AP data flowing up/down, etc. plays a factor in this, since it all has to fit in the same 2GB of available CID memory. There is no place else for it all to go, and Tesla has infamously never documented an upper limit on any of the user-controlled data like every other mfgr I know does -- so it's anyone's guess what happens when memory becomes constrained or if Tesla has even chosen to deal with that possibility in their Infotainment code design. The rescans which happen to some people and not others under varying conditions, combined with tests I did some time ago varying what I expected would change memory usage across phone contacts, nav history and aggregate USB tracks metadata size, lead me to believe memory is part of the challenge. It's why my (automated) workaround is so complex building a Tesla-unique USB stick to reduce metadata while keeping the most important info I use most of the time, so I can have a few more tracks in my MS with me, before my rescan problem gets worse than it already is.
Alas. It would only take a couple engineers at a desk somewhere to resolve so many of the problems, then begin to slowly move Infotainment functionality up to match what the competition has had for years, and eventually to get back into a leadership position once again with useful "Infotainment Easter Eggs" -- aka features owners would really use and make their Tesla Ownership even better in daily use. Unlike building a new vehicle, there are no physical prototypes, capital investments, varying laws to work within, large amounts of personnel required on an ongoing-basis, or significant cost to deploy the changes once they become available. Sadly, it just takes a Tesla Exec to decide it's important enough to do.