Well the only person who can judge if he was unfairly treated is the OP, judging by the fact he has spoke to his MP, I no doubt assume he was troubled with this.
From my point of view, reading what he has said, he joked with them and he was in no way accosted, which insinuates a level of...