Forward: Even though you're using a full frame body, I thought I'd mention my experiences with a crop body as well as other people might find them handy.
I use the 35mm f/2 when pottering about town. Great piece of glass but a bit out-dated now (it's a 1998 design, I think). It's sharp, even wide open, light, and a great focal length on a 1.6x crop body. It's not USM and focus can be slow and noisy but it's accurate the vast majority of the time. It's also an absolute bargain. Just wish Canon would update it with USM and perhaps a more sturdy body!
I have the 35mm f/1.4L on hire at the minute and it's a lot better than the f/2 version but it's also five or six times more expensive. I'd love to get one of my own but unfortunately really can't justify the cost right now. It'd be great as a walkabout lens. As long as you're well insured, anyhoo!
I also have the 50mm f/1.4 USM and, while it's fantastic, I find it a bit too long for street-style shots on a crop body. On your 5D MkII though, it'd probably be incredible. I've not used the f/1.2L but spoke about it at great length with a 5D MkI user (while getting major lens envy!) and she couldn't have given it more praise. The f/1.4 would be a great option if you can't stretch to the f/1.2L. It's light, quick to focus, sharp, has more than reasonable quality bokeh, good colour and contrast, and is pretty much a good all-rounder. A bit soft wide open (the Sigma version is better at f/1.4 apparently) but otherwise a great lens.
Hope this helps!
George.