If anyone asks or makes you feel uncomfortable, ask them if they're paying your medical bills?
Seriously, forget what anyone says or thinks, do what makes you comfortable and safe.
When I first started riding I thought, you can't really hurt yourself here riding XC.
2 concussions
3 fractured vertabrae
1 torn hamstring
1 dislocated rib
1 torn ulnar collateral ligament
1 AC joint rupture
And countless grazes bruises, cuts and sprains over the years.
This is all on XC stuff, and yes I can ride despite the above evidence to the contrary.