I've been DHing for about 25 years now and have taken the approach of wearing it when I think it's required. ie track that I find difficult or consider it a high risk on goes the body armour.
Pretty much any time I throw a leg over the DH bike though its full face helmet, goggles, gloves, knees...