Hi Gareth,
We were down by the river a couple of weeks ago and the bandstand was open, there may be a gate across the entrance which may be locked sometimes. Sorry I can't be more specific on that one.
Nearby:
If you walk along the river away from the Old Dee Bridge there is the pedestrian bridge (featured on the most viewed on Flickr thread) and just by the end of that bridge as you walk up towards the city centre there is a gate into Grosvenor park. This has romanesque arches and stone works as well as gardens, paths, loads of trees (and a miniature railway). If you walk up the road instead of into the park and turn right at the top you'll get to the amphitheatre and a bit further round on the right is an old chapel and roman gardens.
Going the other way from the Bandstand you can walk along the riverside towards the Race course and get to the bottom of the Grosvenor Bridge and round onto the perimeter path of the the Race course, or follow the road up to the pedestrian crossing and walk round the Roman Walls that go all the way round the City Centre (takes an hour or so and some sections were closed for repairs quite recently).
There is also Dingle Bridge which a cast iron footbridge, if you're at the pedestrian crossing (google little roodee car park for reference) cross over to the race course side and walk away from the centre over the Grosvenor Bridge and about 20yds over the bridge its on your left but it can be quite dark in there especially if you go over it and down the steps to the right...
Another option is to go into the centre of town and up to the Eastgate clock...
All of that is within 10-15 mins walk each of the Bandstand...
Hope that helps!