The CBD riding is not so bad, especially when it blocks up a bit. The cars aren't going anywhere anyway so no one seems to mind you ducking and weaving a bit to get through. It is all part of the picture now.
It is on the Northside of the Harbour Bridge that the drivers scare me.
That being said, if you want to avoid the traffic go around it, I would make a bee line down to Darling Harbour, duck past the Ent. Cent., ride between the Chinese Gardens and the Convention Centre. Then go either left or right of the harbour.
Left:
Go past the Maritime Museum and over the Pyrmont Bridge, down the walkway/cycleway to the left, then sussex/Hickson, Taking the left off Hickson to Observatory Hill.
Right: Follow the water until you go past Squires, turn right up the ramps and up the road, and then left onto Hickson, and you are laughing.