Yes, I'm 100% certain they were wasps, I had loads of them in the house and the loft. If they stayed out of the house and stuck to the nest I would have left them alone.
The nest was here when I moved in 8 years ago, no idea how old it is. Clearly it wasn't reused by former occupants but it was reused by wasps.
I assumed that whoever found the hole in the chimney came across the old nest and decided to set to work. The nest in the loft shows signs that it has been extended.
The nest is in the loft space but accessed though a hole in the mortar where the lead flashing is on the chimney stack