Can't help with flights but if you went as a foot passenger from Aberdeen to Lerwick on the ferry it would cost less than £40, which must be a lot less than the cost of a flight. As already mentioned if you book early you can get cheap rail tickets. Or combine a flight to Aberdeen with the ferry.
We are in Shetland now and the ferry crossing was a little over 12 hours, but as the ferry left at 7pm quite a bit of that time can be spent asleep. Arriving in the early morning gives an extra day and similarly on the way back as the ferry also leaves Lerwick at 7pm.
Have to admit the sleeping was a bit difficult but that was probably down to us. This is our first visit so a bit of an adventure. We booked a couple of the recliner chairs but in future would probably sleep on one of the large number of settees and bench seats around the ship as quite a few were doing.
Dave
PS Shetland is spectacular.