It is probably running at USB1 speed by the sounds of it. USB2 is 10x faster than USB1. If 2 USB devices are connected, they default to the slowest speed of either device, if they are both USB2, then fine, but if either is USB1, then they will transfer at the slower speed.
Most of the internal card readers I've used have been quite slow. I think it is just how the PC manufacturer has designed the card reader and connections on the motherboard.
You may get faster transfer speeds with an external card reader, but again it could depend on how the PC manufacturer has implemented their USB ports on your PC.