you should 100% always be cautious, but in my experience PB is a non-issue, i drive 2-3000 miles per month probably 60-70% on FSDb/AP and even back when i'd get fairly hard PB it was never breaking to a stop or anything crazy, at most 10mph or so, I think a lot of the alarm just stems from the "feeling" of how the car decelerates, can give you that roller coaster sinking feeling lol.
Thanks ! - that does put my mind at ease - Im pretty sure you're right stating that the PB is mostly feeling,rather than drastic slow down.