"It won't ever work because it doesn't work today" is not rational, and "it won't work because see this one example of it not working" is pure FUD, like on the level of seat belts don't save lives because a seat belt killed someone in <insert your favorite specific example here>. At the end of the day, FSD is NOT a human, so it doesn't "think" like you, and you certainly don't think like it.
I'm not saying Tesla HW3 FSD v11 is going to work or even that they're even on the right track, but one thing going wrong now doesn't mean it always will in the future, and most of the things being perceived as latency lately aren't even latency. Latency is the time it takes to react to an input, not the time it takes to react to what you see. So, unless you wrote or reverse engineered the software, you generally don't know if something is latency. For instance, if BSA isn't an input, then the fact that it was triggered isn't relevant at all because FSD simply didn't react to it by design. Meanwhile, phantom slowing is clearly reacting to some input that isn't even included in the visualization. As such, we also know that one can't simply rely on the visualization to determine what FSD does and doesn't see as input. Given these points, the only thing we know is that something went wrong there. This can be described as a bug, and bugs can be squashed. In an even simpler example, people keep mentioning latency when it comes to stop signs and stop lights, but those also aren't necessarily always an input just because you can see them. This is why I hypothesized about planning areas/zones several pages back. When FSD is stopping at a rate that you find uncomfortable for something for which you could have stopped at a comfortable rate, it is easy to assume that the computer is slow or FSD "didn't see it" when in reality FSD could just as easily have been ignoring it by design (it could have started slowing down within microseconds of having the input, reacting multitudes of times faster than a human). OTOH, when FSD flat out runs a stop sign, all you know is something didn't work right. Regarding "latency": I should imagine is safe to assume that the HW3 computer can react faster than you to any given input (whether the input will occur and whether the reaction will be correct can remain up for debate) outside of a performance-impacting hardware failure, and as great as you may believe you are, you also don't react to things that you don't see or register (and you also won't react at your normal rates during a performance impacting physiological failure).