I only own 2 lenses, and they're primes. Generally take one out at a time. It's restrictive, but I've done this for two years and I don't even know what to do with a zoom anymore!
You don't really miss any shots with a prime. You just have to take different ones..