Thanks for the replies, guys. I got out my voltmeter and did a bit more investigating today and here is what I found:
First, a bit of background. I have 200 amp service. The service comes up from underground to a small panel with a meter on the outside of the house. The outside panel has a couple of breakers in it for the heat pump and an irrigation well pump as well as a feed for a larger panel in the garage, which feeds the rest of the house. My HPWC is on it's own 100 amp breaker which is in the panel in the garage.
So, for testing I took my voltmeter out to the main panel on the outside of the house and got a voltage reading, then I started to charge the car. As usual, the voltage reading on the car showed a 7 volt drop as it ramped up from 0 amps to 80 amps. But the voltage reading on the panel only showed a 4 volt drop when the car had gotten up to 80 amps. I also did the same test on the panel inside the garage to see if the problem may be the feed between the panels, but I got the same results. Just a 4 volt drop at the garage panel when the car was charging at 80 amps.
So, it seems like the voltage drop may be split... 3 volts are being lost in the HPWC and 4 volts are being lost on the power company side.
Just for 'fun' I wanted to see what would happen if I tried drawing the max amount of power, so I left the car charging at 80 amps and also turned on my oven and my water heater. It's an on demand water heater which can draw a decent amount of power on it's own. Among the three devices and whatever else was on in the house I suspect I was drawing close to 140 amps. With all that on I saw a full 10V drop at both panels. Since I didn't see any difference between the two panels this tells me that the wire between the panels is OK and can handle the power draw, but the wire from the pole to the house likely cannot.
I can see the markings on the wire between the two panels and it shows that it has three AWG 4/0 conductors and one AWG 2/0 ground. The meter part of the panel is locked of course, but I could see a short section of the wires coming in from the pole in there. I didn't see any markings on them, but they are considerably smaller in diameter than the 4/0 wires going between the panels and I think they are even a bit smaller than the 2/0 ground. They definitely appear to be undersized to me.