There is ground on both sides. On the eastern side I've fished quite a bit down near Lamb Rocks (make sure you are familiar with the green zone down that way) and the "channel" between Lamb and Macleay. Across the top (Potts Point) is always worth a look provided the water isn't too clear and on the western side, Iron Stem and another rock/gravel bank further down the island are worth a look. I've never spent too much time with surface stuff - predominantly soft plastics and small (very small) shallow to medium divers. Take a good look at google earth and some charts before you go - there are some seriously substantial prop destroyers in these areas - which just happen to be where the fish hang out.