5 miles is about 25000 ft. The Concord flew at 60000 ft and was too loud, so I think we're talking more like 20 miles, which puts the launch/landing sites in pretty deep water for most places.
"The altitude at which the Concorde can go supersonic was limited by the regulations. For example, the aircraft was only allowed to go at 400 knots during the climb phase between 6,000 and 32,000 ft; also, the aircraft was allowed to go supersonic only after it has crossed the coastline.
As the air gets thinner during climb, the Mach number increases for same IAS, reaching around 0.93 at 25,000 ft and 0.95 at 28,000 ft. After this, the afterburners are engaged and autopilot is set to the cruising altitude (~ 60,000 ft). During this climb phase to the cruising altitude, the aircraft goes supersonic, settling at around Mach 2 at over 50,000 ft."
The problem with the Concorde was it was noisy as all get out at ground level going no faster than a Boeing or Airbus would at takeoff. It wasn't the sonic boom people complained about it was the god awfully loud engines. at barely missing the cars and buildings heights.
as you can see in this video you could stand or drive directly under the path of the concorde at takeoff, no way you'll ever do that for SpaceX.
SpaceX goes practically straight up. They don't cross parking lots, they don't cross highways, they don't cross apartment buildings. It's just not the same issue. They don't have to be further away because of how loud the concorde was, they are already further away at launch.
Guaranteed, you'll never see a spaceX rocket launching in between normal air traffic at JFK or Heathrow. It'll launch from somewhere less populated with less traffic and it won't set off car alarms for miles around.