The suspension height is forced lower at certain speeds from "High" and "Very High" - there is no way to disable this that I'm aware of. The settings will let you have it default to "Standard" or "Low" or atomically adjust between the "Standard" and "Low" heights based on speed.
Model X Owner Manual Page 147 said:
VERY HIGH. When set to Very High, the suspension automatically lowers to High when driving speed reaches 35 mph(56 km/h).
HIGH. When set to High, the suspension automatically lowers to Standard or Low(based on the Suspension Lowering setting described below) when drivingspeed reaches 45 mph (73 km/h).
I agree that an "off road" mode would be great to have it return to a higher setting when you slow back down.
My experience with offroading in my X is that it's very difficult on the car because the suspension isn't soft enough to absorb potholes well. Even at reasonably low speeds over potholes I'll occasionally hear a very audible clunk, which I assume is the suspension reaching it's limit when that corner of the car is not supporting the vehicle. The potholes I'm referring to aren't excessively large either, they are about the size you'd find in a long gravel driveway that a Model 3 could easily clear, but I have to go sub 10 mph, almost near 5mph, to fully avoid the clunk. I previously had a Ford Edge I'd take one the same roads and the suspension never limited out like the X's does. It could be the rigidity of the X that is making the difference here.
As far as power, control, and clearance I'm extremely impressed with it though. Hills, gravel, rocky lake beds, and snow have all been a breeze. The most concerning situation I've had mine was in loose, deep sand. It often detects a strong drag and places itself into trailer mode - whcih mostly changes Autopilot and sensor fucntionality more than anything critical to offroading. One time I was having fun with traction control off on a beach, came to a stop, attempted to accelerate gently and went down instead of forward. I didn't want to bottom out so reenabled traction control and got right back out.
I'll occasionally take the "road less traveled" just for fun but I don't do it as much as I'd like simply because of the stiffness of the suspension I feel it's a bit too hard on the vehicle.