Only if you pretty much call Elon a liar (or at least an idealist) during the SuperCharger announcements.
Elon: "we want to generate more energy from the sun over the cause of the year than is used by Model S's that recharge at the station"
A SuperCharger that is drawing in more power than it distributes, by solar math, will have to be physically larger than the maximum peak usage of that SuperCharger. The moment lines start to form, that SuperCharger can't possibly draw in more energy than it uses - not even if you put it in the middle of Nevada.
Swapping doesn't actually change this, except if swapping is outright not being supported by Solar power. Knowing what you know of Elon, how likely do you think it would be for him to support a system that would rely on grid power?