TBH, if you're within an easy walk of the beach, you don't need a pool - most of the beaches where there are any facilities (on Crete, at least) have fresh water showers for you to rinse off under and it's far nicer swimming in salt water than a pool anyway!
Just had a quick look on Google Earth to see if there were any pools in the Elafonisi area and there don't seem to be. It could well be just a bit TOO far off the beaten track for you, not much choice as far as eateries goes but there is some self catering accommodation down there. It does get very busy on the beach during the day, with coach trips from many of the resorts all over the island taking people down there but after the coaches and busses leave, it's as quiet as you could want!
Another idea could be somewhere like Loutro, a tiny village with no road access, just a few ferries. Quite a few places to eat and it's only a short boat ride along to the bottom of the Samaria Gorge. Thinking about it, Agia Roumelli is pretty quiet and basically exists purely to service the people who have walked down the Gorge and who almost all leave on the ferry. Never stayed in AR, although we have visited a few times either on the ferry from Hora Sfakion (another contender for a quiet place to stay) or after doing the gorge walk (10 miles or so on what feels like railway ballast!) We did spend a night in Loutro once though - very pretty but not really enough to do for a few days' stay. You might consider Agia Gallini too, loads of places to stay and to eat, good beaches, boat trips, tourist and "local" shops.
On Crete, I would definitely have a car. The driving can be interesting but in almost 20 years of at least 1 visit (and often 2) per year, we've not had a problem and only seen the aftermath of a couple of incidents (although we've seen a few shells at the bottom of long drops...) I can give a number for the place we rent ours from but can't guarantee the same deals we get! The busses are fairly frequent and do seem to get to most places but can be very crowded. Plenty of coach trips from many resorts/centres too but the costs soon add up to the cost of car hire!