I shoot a lot of sports (rugby mostly) and tend to switch the IS off. Not sure if it slows down the speed of the AF, but I have other reasons to keep it off.
1. The shutterspeeds I use are fast enough that IS is not really needed
2. Although newer lenses may be better, I experienced in the past unsharp photos when following play where it seemed that the IS was contra-productive.
This was one of the reasons I read about in the past not to use IS. I think that newer lenses perform better in this respect, so it becomes less relevant.
3. If I don't use IS (because I do not need it) there is a smaller chance on wearing the IS out. I have an old EF 400mm f2.8 L IS lens which is no longer serviced. The more I can do to prevent components to wear down/ out the smaller my risk.
I say it is safe to have IS off. I would only turn it on if you feel it makes your shots better or if it helps following play due to a less stuttering image in the viewfinder (you need to engage IS with the half shutter press than offcourse). The latter being a reason I have read on the internet as a case to keep IS on.