@antc I have the 300 f4, same as you - I'll probably be selling it soon

This is my first properly long lens and I've been lusting after it for a while.
As for wider lenses....
24mm is a darned good focal length and the 24-105 is a good lens.
There's a reason ultra-wide lenses are a bit specialist - they aren't for everyone.
The classic Canon ultra-wide lenses are rectilinear and stretch the corners producing a distinctive look, and I'm not a fan. Although some subjects work well (e.g. architecture), as soon as anything recognisable appears in the corners they often look uncomfortably odd (especially people in crowds). I've seen a way too many cracking landscape shots that the stretched corners irritate the heck out of me.
Learn to embrace the 24mm focal length
I do have a wider lens - the 8-15mm fisheye. I use it a lot at 15mm where it has 180 deg coverage corner to corner.
Superb for capturing the whole scene when isolating part of it just isn't going to work; it's also my main lens for cave photography where getting far enough away from the subject is a real problem.
It's still an ultra-wide, so all the problems of keeping the camera level to avoid crazy perspective still apply.
But at least I get the choice to go with the composition at the time I took the shot, or de-fish the images. I only de-fish occasionally and far less than I would have predicted before getting the lens.