I personally just set a charge time of 9:30pm and this works reliably for me, so unfortunately I cannot answer your question directly, but it does seem like you are possibly giving it two different instructions that maybe it is interpreting as conflicting (even though they probably never would).
I imagine that if you set a departure time of 4:30am it's going to start charging at the latest possible time so that it finishes up by 4:30am. Unless you are charging at 120V or have a ridiculously slow L2 charger, it should probably never commence charging prior to 4pm the day before. So have you tried to just leave the off peak setting alone?
Alternatively, just do as I do and set a start time that you are comfortable with. For example, I normally plug in when my car gets down to about 30%, and it takes between 5-6 hours on my EVSE to charge to my 80% set point. I don't particularly worry about having the car sit at 80%, so I just arbitrarily chose 9:30pm as my start time, but if you wanted to wind up hitting your target at 4:30am, if it was me with my settings, I would back up 6 hours from 4:30am and put in 10:30pm as my start time.