If you are really worried, get yourself a little aluminium card case like this one;
it's not a wall impenetrable to magnetic fields, but will give a bit of distance between the magnet and the card, they aren't powerful magenets on those enclosures anyway. Whatever you do though, don't put a...