The button is used for more than just opening the charge port. If the car is actively charging, the plug is locked to the car for safety, you can't just yank it out (to prevent electric arching). You need to press the button on the plug to interrupt the charging and unlock the plug from the car.
If you don't have the button on the plug, you need to either pull out your app and press Stop Charging or go inside the car to do the same thing to unlock the plug. That to me is a huge hassle for not having the button on the plug.