I just raced today and this is fresh in my mind cos I was thinking about and discussing exactly the same thing with a few other guys...
Skill (encompassing technique, fear/confidence, willingness to take risks, etc) is critical. It doesn't matter how fit you are if your technique can't physically get you through a corner/section fast. Hit a section (whether that's a single corner or a 20 second chunk of track) time and time again, see if you can hit it at whatever speed you're setting as your benchmark (in this case, your mate Mark). If you can't, practice the technique until you can.
However, if you're actually racing over more than about 30 seconds at a time, fitness and strength are huge factors. I don't know what you're like, but my fitness sucks, and it makes a significant difference to how fast you can ride when you're 2 minutes into a rough 4 minute track and your forearms feel so weak that you can barely hold onto the bars. You can see how big the difference is (esp if you have a Freelap system) by timing yourself between two points, say across 3 corners or whatever, near the end of the track. Time how fast you can do it all if you start just above the first point, so you can come into it flat out but not tired, then time how long it takes when you start from the top of the track, so that you're tired by the time you get there.
Chris Kovarik won the 2002 Fort William world cup by 14 seconds. When he was interviewed about it a few years later, he was asked what he thought the difference was back then, how he could win by so much when he was (at the time) having to fight tooth and nail just to get on the podium (this was when he'd just come back after breaking his leg really bad). His reply was that fitness was a huge factor - "normally you get halfway down the track and you're half f**ked". Being able to hit sections at the end of the track as hard as you would if they were at the start of the track certainly makes a difference.
However, fortunately for some of us lazy bastards, DH is far more about skill than it is about fitness. A very skillful but unfit rider will still typically put big margins into a very fit but unskilled rider.
Disclaimer: take this with a grain of salt, I'm not the one winning any races