According to Wikipedia, South Korea has 15 submarines and Japan has 17 submarines. All of these are attack boats. The US has about 50 SSNs, perhaps half of which are deployed in the Pacific, and an additional 4 SSGNs. If the US and allies combined have 50-60 modern diesel and nuclear attack subs, in addition to massive surface fleet and air support, I don't think the North Korean navy would last very long.
However, I do think the North could inflict plenty of damage on surface ships. My guess is that a sea battle would play out similar to the Falklands conflict. The US and allies would lose ships to the North, but ultimately win via a combination of greater numbers, better training, and superior equipment.
The North Koreans alternatively could avoid direct confrontation and target civilian shipping lanes in order to maximize economic pressure.
Used as a first strike weapon in peacetime, any submarine or small surface ship, as the USS Cole found out, can cause a lot of damage.
Same with commando units, very destructive in peacetime. Their effectiveness in wartime is diminished when they are being targeted.
The NK subs are loud and slow. They would not last long in wartime with modern anti-submarine warfare forces actively seeking them even in NK waters.
IMO, when that NK sub sank that SK warship in 2010, it should have been declared open season on all NK subs in international waters.
However, everything changes when we allow NK to deploy ballistic missile submarines.