I never buy a vehicle without test driving first, and I think many people are like that too. I will sit in and drive a Model Y before I put any money at all down on it. If the deposit was refundable then sure it would be a no brainer, but it isn't.
Never be impatient when investing in something as pricey as an automobile. It pays to wait and be absolutely sure in my opinion.
That said, I went to my local Tesla store this past weekend and they expect to have a test drive MY by April / May timeframe, so maybe some of the early production MY's are being reserved for the stores?