Who told you that the wires from the breaker box are inadequate? Did somebody inspect your wiring, or was this speculation? If it was an inspection, was the person competent (like a licensed electrician)? I wouldn't take this diagnosis as certain unless it was an actual inspection by somebody competent; however, it is a plausible hypothesis, so I wouldn't dismiss it, either.
As others have suggested, the problem could also be a loose connection or even a defective NEMA 10-30 adapter or a defective portable connector. I don't know if Tesla service centers have NEMA 10-30 outlets for testing (I'm guessing not, since they aren't supposed to be installed new any more), but you could certainly take the portable connector to a service center for testing on a NEMA 14-50, which I'm pretty sure they all have.
You can try reducing the amperage when using the NEMA 10-30 outlet by using the car's screen. If the wiring under-sized, that might be enough to get it to work better; however, you'd basically be guesstimating a safe amperage, and it might be reset by some random event, so I wouldn't want to rely on this for the long term. Plus which, you'll be charging slower. If the in-the-wall wiring really is not up to the task, it's much safer in the long term to replace it. If you do that, then you might as well upgrade to a NEMA 14-50 outlet (with matching wiring and circuit breaker), which will enable you to charge at 32A using Tesla's Gen2 portable connector, or at 40A with a Gen1 connector or some third-party EVSEs. The cost of such an upgrade will vary depending on how long the run is from the breaker box.