I put racks on my Y, expecting to take them off when not using them... but I've left them on : didn't really notice much efficiency hit. perhaps a bit of noise at fast HW (70+?).. but I spend very little time at that speed... So the risk and effort of taking off and putting on racks wasn't worth it..
But that was with bare rack bars. put anything on there and I'll be you start to notice.