I had both systems for a while, XP1 plus 18, 35 and 60mm lenses, OMD plus 12, 45 and 75mm lenses. I have no doubt that both cameras can produce lovely shots and follow the work of Damien Lovegrove (Fuji) and Damian McGillicuddy (OMD), two top pros using these cameras.
The huge difference for me was in the mode of operation. I find the Fuji's a bit more old fashioned and basic. I use the electronic viewfinder now almost all the time so the rangefinder element isn't relevant for me. All I do is set the iso and aperture then ride the exposure compensation dial. I shoot in Raw so few of the other settings are relevant.
The olympus is the other end of the spectrum. While you could shoot in the same simple way, the OMD has so many adjustments you can make, different ways you can shoot (I did like taking shots by touching the rear screen), fancy off-camera flash command systems etc etc. I found using the OMD like using a tiny DSLR and the Fuji like shooting with my old film cameras.
Ergonomics are a lot different. The OMD is too small for me and I needed to add the full grip to get close to something comfortable. The Fuji is big and heavy in comparison and sat better in my hands.
Shots do look different but it isn't clear to me whether one is better than the other. I happen to like the Fuji rendering but others don't. I am a sucker for a shallow depth of field shot so the Fuji's larger sensor helps. Once the 58/1.2 comes out I will have a hard time resisting.
If you really use the OMD's fast AF and IS then I don't think the Fuji is for you, at least not with primes as they lack IS. AF is still sluggish after the firmware changes - perfectly reasonable and not something I worry about, but the AF on the OMD is exceptional.
The Fuji's are well worth a play - I ended up getting one despite going into the shop to get a Canon 5DIII - but the difference in the quality of the photos isn't likely to be huge IMO.