Final update - I have mostly resolved this issue.
At the end of the day, I do think my toe links had a part in this, but I also replaced my front lower fore arms and aft arms (which made the biggest difference).
The inner mount on my left front aft arm had been gouged, so the bolt actually had some play in there. That meant that the control arm moved when I applied throttle. Changing this resolved a popping noise that I occasionally heard when shifting from drive into reverse, and kept it going straight under throttle.
I have NOT replaced my front upper control arms, sway bar end links, and rear lower control arms / integral links. They all seem to be in fine shape.