According to the agreement in my order (I don't have the car yet), once the vehicle is physically delivered to you, the delivery fee isn't refundable. If you heard different things, it's possible that others have different delivery agreements.
Tesla has changed their delivery agreements many times over the years, so you'd just have read over yours to see. If you cancel before delivery, only the $100 deposit is lost.... and if it's canceled due to some craziness on Tesla's part, a firm call to a customer service person can sometimes get that refunded also... but it's less likely.
I put in my order on June 25.