I personally wouldn't swap from m4/3 to APS-C as there's not enough difference and you'd just be throwing money away imo if it's purely for IQ reasons. I've cropped pretty heavy with M4/3 and got decent results. IF you're worried about reach you can buy a used olympus 40-150mm f4-5.6 R (80-300mm eq) for under £100. It's really lightweight (to the point it feels a bit cheap) but IQ is very good. You can stick it in a coat pocket and barely know it's there.
Moving from m4/3 to FF I believe there is enough of a difference in IQ to justify the swap, however you then have to question whether your budget can stretch to something that you want and not just making do, plus the system is bigger and heavier.
What you have to ask yourself is whether
you are going to see enough of a difference because at the end of the day it doesn't matter what others think.
Here's some comparisons from myself. Not exact comparisons but close.
m4/3
P8246787 by
Toby Gunnee, on Flickr
FF
A9_05337-Edit by
Toby Gunnee, on Flickr
m4/3
P1250064-HDR by
Toby Gunnee, on Flickr
FF
DSC_5098 by
Toby Gunnee, on Flickr
m4/3
P2130021 by
Toby Gunnee, on Flickr
FF
A9_04359 by
Toby Gunnee, on Flickr