It depends on the focal length as well, a 500mm f4 would be a 'fast' lens, but anything below 300mm at f4 wouldn't be thought of as particularly fast, in my opinion.
Add prime vs zoom into the mix, a zoom would get a bit of slack when thinking in fast terms...a 24-70mm 2.8 is a fast lens, but a 50mm 2.8 wouldn't be regarded as particularly fast.