No, that's not how that works. They would need the transformer to get 120V.
Exactly. The RV is trying to provide all of its 120V loads using a hot and the neutral to complete the circuit. The J1772 supplying this whole situation through that adapter is only the two hot lines that are 240V apart from each other (or 208V), and there is no neutral provided there. And I would be horrified to think if whoever made that adapter tried to use a hot to ground as a legitimate current path for the circuits. That would be dangerous.
Yikes! I apologize - I didn't look at the J1772 spec so didn't realize the neutral doesn't go through to the car. You're absolutely right, that's terrifyingly dangerous.
That said, a typical RV 50a setup is much like a residential circuit panel; two split-phase hots, a neutral and ground. Why would they need a transformer to step down to 120V? They use residential-grade circuit breakers - taps into either of the split phases. You're right though, there's no neutral with the J1772 adapter!
Smaller RVs of course use the TT-30 connector which is 120V/30a. Single hot phase, plus a neutral and ground.