Likewise I find myself feeling similar - if a random stranger on the street wanted to start taking photos of my nephews and/or nieces while they were in my care I would expect at the very least for them to ask me very, very nicely indeed and expect to be able to see the photos once they'd taken them. Whether this is right or wrong it is how I feel, and I'd venture that plenty more feel that way, regardless of how they think they "should" feel.
I think this is fair enough TBH. Everyone's different and their reaction to things is going to be different (subjectivity). I don't think I would take immediate offence to the situation you describe. I'd "be aware" of going on, but unless I think their activities are unhealthily focused on my kids alone then I'm not going to take objection i.e. there's a big difference between a bloke taking photos out and about that happens to have my kid in it and one that's just fixated on taking photos of my kid.
Even then, my reaction wouldn't be to storm up and shout "put that camera down you p***!", I'd hope to ask politely - though sternly - "why such an interest?" and unless I'm satisfied I'll ask them to stop. In most cases, I would expect the photographer to be as reasonable and respect my wishes. Of course you can go sighting law, public place, etc but I think decency (which is really at the heart of the issue and where things have all gone awry) is to respect another's, reasonable, wishes - so allow the tog reasonable latitude to carry out their activities, and expect a reasonable response when I think he's overstepped the mark. Most people just can't be decent to people anymore.
A big part of the problem as I see it is there is a demonisation of photographers around children as being paedophiles and as such they become the face of an evil that happens behind closed doors. I'm not even sure where that's come from. But I'd be willing to bet that in 99.9999+% cases the photographer is acting entirely innocently (certainly with respect to this particular law) and so it's incredibly sad that everyone is tarred with that brush, particularly where no actual evidence to the contrary has been presented (i.e. that a large portion of photographers down the park, swimming pool, beach, etc are there for unhealthy reasons). And society is the poorer for it.
Something that might be a bit unsettling and that I can't quite square in my head is if a paedophile with a camera is acting entirely normally as compared to a "regular" photographer, is it actually a crime? Obviously what they do with the images "may" become a crime, but is it actually a crime (serious question)? As I say I have a horrible head/heart logic/emotion on this one. On the one hand the act of taking the photograph has no "victim" - the subject matter may be entirely oblivious. On the other, nobody wants to think of someone like that having pictures of our children :shudder:.
Not a nice subject obviously, made worse if we're all branded as one.
(ps I'm willing to bet that as my child grows older and I have more encounters my stance is likely to harden in that over-protective parent way - I'd hope not and that and I can maintain a sensible head, but it's a crazy world we live in)