Not true at all. While it's a big factor, it's far from the only determining factor.
Driver death rates
Case and point, smart cars (smallest and one of the lightest on the road) have less fatalities per crash than a Chevy Silverado full-sized pickup truck, even after normalizing for the number of seats available on both vehicles.
A heavy safe car will always be the best (Model S), but a well-designed small car (smart) will still trump out an unsafe larger car (Chevy Silverado).
Also OP, definitely glad to hear you're safe! I think safety is the number one reason I'm going for a Model 3 next. I'm normally the type who drives small (very small) cars, but the possibility of an accident makes me want to change that habit.