Being a local I will give you the lowdown.
First of all, Phillip Island has a few tracks, you just gotta know where to find them. The best out of the few that are around in my opinion is the Cape Woolamai loop. Head all the way to the main surf beach where the lifesaving club rooms are, hit the beach, and head left along the coast for 1km until a set of stairs. Once at the top, the rest is self explanitory. Good fun, and I would be temped to say it will be one of the best views you will find in Australia from a trail, at least Victoria (not including the Alpine regions!)
Ride On Bikes in Cowes are a great shop. They may not cater specifically for MTB riders due to low MTB population but Ash, the owner and mechanic is a good bloke and knows his stuff.
Another option, although a further 20 minutes down the coast is Crossover Cycles in Wonthaggi. If you are down here for a few days there are some fun single track loops starting near the old State Coal Mine in Wonthaggi. Just head into Crossover and ask for directions.
For a nice fun ride with consideration of your other half, try mcoccia's idea.
The Gurdies, just before Grantville has plenty of trails to suit you XC riders out there, and a quite a few fun descents, you just have to push, no shuttles.
And s.dogg, if by the 'famous Inverloch DH track' you mean the walking track to the top of the cliffs starting at the end caravan park, stay off it. It is clearly signposted and fences and gated to restrict bikes from being on it. If this isn't the spot you are talking about, I think your high. There is barely a respectable hill in Invy let alone enough for a DH track.