There's a useful model of driving behaviour [1] which breaks down the task into three different cognitive levels:
- Strategic - route planning, etc.
- Tactical - perceiving and reacting to the environment, e.g. keeping enough distance to other cars
- Operational - steering, accelerating, braking, changing gear, etc.
All three levels can and do operate simultaneously, but you only have a finite amount of cognitive resources available, and if demands on one level are too high then something on another level might give. For example if you're (level 2) paying attention to another car that is near you but driving erratically, you might (level 1) miss your turning. Or conversely, if you're (level 1) worrying about whether you've taken a wrong turning, and what you need to do to get back on the right route, you might (level 2) not be as aware of what's going on around you as you would be if you could give it your full attention. Similarly, if actually (level 3) driving the car, including changing gear, takes too much effort then again your (level 2) awareness of hazards may be reduced.
There have been studies [2] demonstrating that elderly drivers make more errors - including errors with the potential to cause accidents - when driving manual cars than when driving automatic cars. The explanation is that elderly people tend to have experienced declines in both cognitive resources and motor skills, and the attention required to change gear manually affects their ability to do other necessary things such as judge speeds and distances. A classic example of this situation is when you're trying to turn right at a T-junction on a slight upward slope.
I agree with you that for most people, most of the time, changing gear manually isn't a big deal. But there are definitely situations where it contributes to making things slightly less safe. Yes, it's mainly elderly drivers for whom this is an issue, but let's face it, we all want to be elderly some day. (Because the alternative really sucks.)
[1] A critical review of driver behavior models, by John A Michon, 1985 -
http://jamichon.nl/jam_writings/1985_criticial_view.pdf
[2] For example, this Swedish / Australian study:
http://casr.adelaide.edu.au/rsr/RSR2011/2APaper%20130%20Falkmer.pdf
and this Dutch study:
http://www.hfes-europe.org/wp-content/uploads/2014/06/Piersma.pdf