So I finally got around to swapping my lights around (I agree it didn't sound like it would make a difference, but I've got 5 minutes and nothing to lose, I figured), and it turned out to make a difference after all.
It didn't completely eliminate the problem, but it reduced it dramatically...