28mm, 24mm and 20mm are all quite different to use, and 28mm is quite a lot wider than 35 (which isn't wide enough a lot of the time, as you found, because it's almost a second 'standard' lens). 28mm and 24 mm are both very useful focal lengths, but I'd suggest that 20mm is a bit more specialist, and once you got over the novelty it would be used less than 24mm.
I've just (as in earlier this week) picked up an older Sigma 21-35, and I'm very happy with it so far from a single photo-expedition in Oxford city. It's surprising how much the excitement of having the 21mm focal length available over-rides good sense, and how often photos taken super-wide would have benefited from a slightly longer focal length.

These lenses aren't expensive relative to what you're looking at, and it might be worth picking one up (or the slightly similar Cosina/vivitar/Tokina/Quantarray 19-35) to see which focal lengths work for you before investing more in a prime. Worth remembering too that super-wide lenses are more about 'getting inside' a scene than simply getting more into the picture.
Which focal length do I prefer? For a single prime length probably a well designed 28mm, because distortion is minimal and it's still pretty wide without making distant objects too tiny: 24mm is pretty darn good too, but needs you to get in closer to be most effective.