Lightroom 5 export speeds

JonathanRyan

JonathanRyan

Suspended / Banned
Messages
10,765
Name
Jonathan
Edit My Images
Yes
Just running my first really major export on LR5.

I set it going, went off for some lunch and a bit of Dexter, came back and it's still running :thinking:

It seems to be taking about 15 seconds per picture. That seems awfully slow. About half the files are D800 raws and a little more than half are D3/D3S raws.

CPU load looks like this
cpu.jpg


Running on an iMac 3.4 Ghz i7 with 32 GB of ram and 2 GB graphics card.

Really, does that speed look right to you?
 
I've seen a few people commenting on LR5's desire to chew up memory Jonathan, especially when exporting.

It's a real pain because I'd love to upgrade! :(
 
It's not the memory Mark - the iMac has shed loads and LR can't even begin to stress it.

It just plain doesn't utilise the processors properly. To run at full speed you'd expect to see pretty much no black on that graph - it seems to be using about half the processor power. Nothing else is in short supply and in fact, even on v5, 3 years after this was first spotted it's still quicker to split big exports in two and run them in parallel. That's pretty shocking.

I did a quick survey on FB and it's the same old story. Some people reporting slow imports, some slow exports, some slow performance in general and some say it's great. There appears to be absolutely no way to predict how LR will run on a given system.

Really v5 should have been the "stop adding new stuff and make it run properly" edition.
 
I noticed this on my i7 as well. It utilises 60% of the processor and running 2 jobs makes it utilise 100%.

When I had i5, LR was using 100% of processor. so my guess is that Adobe optimised the program to use 4 cores.
 
JonathanRyan said:
Really v5 should have been the "stop adding new stuff and make it run properly" edition.
Click to expand...

Agreed completely. V4 is pretty damned good but it slugs at times, even on an i7 with quads and an SSD. :(
 
I built a Windows system based on an Core i7 - 920 (2,67 GHz) four years ago. I have 16 GB of memory, and a 512 GB SSD (Samsung 840). And an ATI graphics card bought last year, I forget which, but it's one of the faster ones.

Unusual for me, I used to build a new system with every new processor generation, but LR and all other application software works ok for me on that system. When in LR I switch from default image size to full size in the Library or Developer module (clicking on the image), the first time around it takes a few seconds with the D800 images. All processing work is fast. Exporting and importing I'd say takes half the time Jonathan estimates on his Mac.

Jonathan, the fact that you see some black areas does not indicate that the processors are not used properly. It only indicates that they are not used fully.

You probably know that a process or thread has a priority assigned, and application program threads or processes should never acquire a priority so high that they stop critical system functions. I'm not sure about the Mac's arbitration strategy, but I think if LR leaves this to the operating system (as it should), then the OS will not attribute a lot more time to one process than what you are seeing in the CPU load chart.

You could try to increase LR's priority - then the OS would give it more CPU time. But you run the risk of catching out critical system functions such as disk access or mouse/keyboard etc. (leaving you without the option to interfere with the process).

I actually find that some Adobe applications grab too much system ressources sometimes, leaving the user unable to do anything else while they are doing certain things. Of course, high volume disk access is one of these things which usually grabs a lot of system ressources and you probably don't want to interrupt it, as any error there is critical.

I think if Jonathan does not have an SSD yet, that would probably be the most effective single upgrade he could make. And it should be one of those which can write a lot of data without slowing down, and it should probably better not be one of the self-compressing ones, because while these may be fast writing RAW data, they're probably not fast writing jpgs or any other precompressed format.

You might also check whether there are any background processes occupying processor time - because I guess your chart above shows the overall CPU usage, and not just the usage LR causes?

You could also try to increase LR's priority, but I don't know how to do that on a Mac.

Personally I would suspect disk access to be the bottleneck first, before looking at the CPU.

Edit - one more thing. I have no idea whether on a Mac this could happen, but I had an issue with extremely slow imports via USB3. I found out that my virus scanner slowed things down drastically. As I said, I don't know whether this could happen on a Mac or in your configuration. But I exempted all locations LR reads from or writes to, and the LR program directory and applications from virus scans. Sped up my imports dramatically, to where they had been before the problem arose.
 
Last edited:
Thanks Achim - some interesting stuff.

I'm using an iMac so I have no opportunity to add an SSD :( However, it has an Apple fusion drive which bench tests way faster than any SSD I've used. I suspect the point at which it bogs down is with writing many large files (so, exactly what I'm doing).

I'll look into using "renice" to set LR's priority but I don't like to tinker with Mountain Lion too much. FWIW for most of the export I left the computer idle (I watched Dexter on a real TV) there wasn't even an email daemon running so it should have been able to use what it wanted.

I don't use a virus scanner ;)

The key thing for me is that it's well known that if you split the export in 2 and run them in parallel then it's faster. That surely shows there is a lack of optimisation somewhere (since LR could just optimise the process by doing the splitting itself).

This is starting to bug me - especially as Hugh's i5 seems faster than my i7. When I get some time, I think I'll make a test LR catalogue and make it available online. Then people can see what results they get on the same data.
 
Does your i7 maybe gets throttled down because it's hot?

The i5, if the clock speed is high, can be as fast as an i7 for certain applications. I'm not fully up to date WRT this anymore, but the i5 often also has 4 cores, right? And just a tad less L3 cache.

Of course, and I'm only guessing, but those valleys in your CPU charts, couldn't they be the CPU waiting for (disk-) I/O?

Because while the SSD in the Fusion drive is not full, the storage system will be blazingly fast, but once the SSD gets fuller, I guess the OS will start writing more to the hard disk directly, in an attempt to keep application software or other things you use often on the SSD, before it finally yields to the onslaught of data and decides to delete them in favour of the currently written image data. You depend a bit on the OS's strategy there. Can you check how full your SSD is when your exports are slow?

If your Mac has Thunderbolt, I think you could consider an external SSD. I think Thunderbolt has 10Gbit/s? If this is true, IMO it would be fast enough to fully use the potential of an external SSD (SSDs having something around 500 MByte/s). But please check these numbers, I'm not knowledgeable enough to be a reliable source WRT to these speeds.

You mention the name 'Hugh' without further reference, which makes me wonder whether you know more about me than I know about you :) (i.e. we have a common friend and I'm not aware of it) - is that Hugh you are referring to currently battling with developing medium format films shot with an analog 45 year old Yashica?
 
AchimT said:
You mention the name 'Hugh' without further reference, which makes me wonder whether you know more about me than I know about you :) (i.e. we have a common friend and I'm not aware of it) - is that Hugh you are referring to currently battling with developing medium format films shot with an analog 45 year old Yashica?
Click to expand...

Ah, no :) I was referring to Hugh aka Boyfallendown who often posts here. I thought he'd replied to this thread but it turns out he's only replied to me on FB.

The other stuff all makes sense - though it's a bit over my head. Although i used to know a lot about computers, I'm now at the stage where all I need to know is "give Apple more money and they will give you a faster machine" :)

I can't find a decent external TB caddy that would allow daisy chaining but if I can I'll try an external SSD. As far as I understand it, SSDs are bad at writing lots of big files though so that may bottle neck.

Really what I need to do is finish this work I have in the edit queue and then do some proper testing. If parallel exports really do run faster than one monolithic one then it can't be CPU throttling or disk i/o.
 
@AchimT
Quote
Edit - one more thing. I have no idea whether on a Mac this could happen, but I had an issue with extremely slow imports via USB3. I found out that my virus scanner slowed things down drastically. As I said, I don't know whether this could happen on a Mac or in your configuration. But I exempted all locations LR reads from or writes to, and the LR program directory and applications from virus scans. Sped up my imports dramatically, to where they had been before the problem arose.


Can I do this with Microsoft Security Essentials and if so, how?
Thanks
JohnyT
 
Hi John, you can exempt file types and drives with MSE as well.

Provided you have Windows 7, open MSE in your System Tray, and select the 'Settings' tab. On the left side you see a list with settings, there are three relevant ones for this:

Excluded Files and Locations
Excluded Filetypes
Excluded Processes

I selected Excluded Files and Locations, and added the directories I wanted to exclude (like my USB3 Card Reader, the directory where my images are stored, the directory where my PP software is located, and also temp folders that that software might use, and Lightroom's catalog folder for example). These paths can be added manually (written into that 'File locations' box, then clicking on Add), or you can browse to them using the Browse button, and then use the Add button to add them to the list.

Then I saved the changes

Then I selected 'Excluded Filetypes' and excluded .NEF, .JPG, .JPEG, .TIFF. I forget whether I entered more file types, it's been a while.

Then I saved the changes

Then I selected 'Excluded Processes' and browsed to the .exe files in my PP software's folder, and added them one by one. I also added the exes of the Nik filters and another filter software I use.

Then I saved the changes

This is it. If it doesn't help, you have to think about which other processes, folders or file types might be the culprit.

Be aware, if you exclude the right file types, processes and directories, MSE will stop checking these files and locations. Your system is no longer protected in those locations, and if you get an infected file in those locations, the scanner will not detect it, and your system becomes vulnerable in those places and to those files or filetypes.

So it might be a good idea to check those files unless files added there come from your own camera and you are sure they are not infected. And maybe also an occasional manual scan of those locations may make sense.
 
CPU utilisation seems about right... It takes around 5 seconds per D800 RAW here. That's just down to the speed of machine though, and a slower i7 with fewer cores will take longer, so there's not really much wrong with 10-15 secs per RAW.

...but CPU load is what I'd expect.

Here's mine doing a batch of D800 RAWs to 16bit TIFFs.

PeOyBI2.jpg
 
Last edited:
You must log in or register to reply here.
Back
Top