Personally, I typically find that the horse rider is way more skittish than the horse itself. And the horse is also usually much prettier...and quite often more aware of its surroundings than the human on its back.
We tend to have a shout to make the horse aware that there are humans coming up behind then another shout to make the human aware that we've had the sense to tell the horse that we are there. This tends to surprise the human which (in turn) surprises the horse because they are in intimate contact with a startled human and it makes them skittish. So despite trying to do the right thing, the horse and its human are now skittish, the human is also grumpy and the horse is blithering into the scrub and we ride on by wondering why a) anyone would want to ride a horse if it just makes you agro, broke and looking only half as good as a horse and b) the horse is silly enough to put up with it.
Easily spooked horses are poorly trained horses...that's not the horses fault - it's because of the monkey on its back. There's few local riders who complain about the kangaroos because they jump across the race track and 'frighten' the horse...ever heard of training & familiarisation, oh horse riding monkey?
Frankly, a horse is usually much improved by not having a human on its back.