I think OCR for speed limit signs seems reasonable. (I'm pretty sure that Mobileye does that.)
That way you train the NN to recognize a speed limit sign, then pass it to OCR to determine the actual limit on the sign. (Otherwise, I think that you would have to train it on every possible version of every possible speed limit in both MPH and KMH.)