Thinking about this critically, this is actually a really good case study in human perception and context clues.
The first (and obvious) context clue in determining which light is ours is one being slightly less visible than the other. This context is probably what most perception networks should attempt to perceive first:
View attachment 850437
But failing that, and taking this edge case to the extreme, what happens when the lights appear as if they are equal in size? Without distance or shadows, it's literally impossible to know which light is ours in this case:
View attachment 850436
Oddly enough, I think the next best context clue that we all pick up on (but don't consciously realize we consider) is actually the orientation of the wire on which the lights are. Here's the lights plus the wire as they appeared in the video:
View attachment 850435
And here's the wire if you were to view it from the perspective of a driver in the perpendicular lane:
View attachment 850434
So we instinctually know that because the lights are designed to face two adjacent angles in the 4-way intersection, the light that faces the direction that the wire opens must belong to the other lane, and can be ignored.