Finally, a decision...it's DNG

  • Thread starter Thread starter Deleted member 68495
  • Start date Start date
D

Deleted member 68495

Guest
I have procrastinated and prevaricated for what seems like years (and probably is) about whether to go DNG and dump all the problems with sidecar files and the like. I have harboured a rsistance to moving away from Nikon NEF format as I felt like I was being 'disloyal' or leaving myself open to lost photographs. In the end I decided that since once I had finished editing a picture I always converted it to high-quality JPG anyway -- which is readable everywhere -- I would not lose my best pictures if something went wrong with support for DNG or the OEM RAW format.

I am running the converter as I write this and while I haven't burnt all my boats at present -- as the converter program makes a DNG file while leaving the original RAW intact -- any future RAW photographs will be converted to DNG on import so there will be no RAW file.

I am 60 now so I don't have to rely on support for too much longer. If Adobe manage to look after my pictures for the next twenty years (there's optimism), it'll do for me; descendants will only be interested in the jpegs anyway.
 
Don't understand what the need to convert is? You have raw convertors, plenty of programs support Canon and Nikon raw format, as does the manufacturers apps and I can't see support dropping for them, plus you already have jpeg. More importantly would be your backup solution rather than the format.
 
Byker28i said:
Don't understand what the need to convert is? You have raw convertors, plenty of programs support Canon and Nikon raw format, as does the manufacturers apps and I can't see support dropping for them, plus you already have jpeg. More importantly would be your backup solution rather than the format.
Click to expand...
What he said.

I think the OP has found an answer to a non-existent question. What's more I'm not even sure it's the right answer. Surely there's more chance that RAW converters will still be around than that Adobe will (whilst it's unlikely either will disappear)?
 
What's wrong with having both? I copy the CR2 files to a external drive then import them as DNG files into LR from there. Disk space is relatively cheap.
 
I don't think anyone is saying it's wrong having both, more a point being made that they feel the extra process of conversion to what is a third format when both existing formats are well catered for is simply not required.
 
Converting to DNG is sometimes not a choice. Eg. You buy a newer camera and your post processing software no longer supports the latest version of camera RAW. In that case conversion to DNG is not an option but I do concur with Phil and Byker, it's hard to justify the extra work.
 
But the OP has said that from now on all files will be converted on import so he will not have the original raw, just the dng raw. What I am saying is have both.
 
I have toyed around with the idea of using dng's for a few years. 3 reasons why I haven't:-
  • It adds an unnecessary step to the workflow.
  • Increased back up bandwidth. An update to the dng triggers an update to the whole file. Using raw + sidecar files, only the sidecar files need to be backed up. Using raw + lightroom catalogue, only the catalogue needs to be backed up. This can make a massive difference, especially as I back everything up online. If I choose to change my metadata tags (e.g. change my email address, or change the format of peoples name from first, last to last, first) this will mean backing up everything again.
  • It solves a problem that I do not perceive as being a real threat (the future inaccessibility of proprietary raw files.) The fact os that there are masses of third party RAW converters currently that have successfully reverse-engineered current raw file encoding. Why this will cease to be the case is just scare-mongering.
  • You do lose some data when converting to dng. When I shot nikon, nikon view showed the selected focus point of nef files, but not dng files.
  • DNG's were not as universally supported as common raw file formats. This was the case a few years ago when I found that some raw converters didn't read dng's, but may have improved since then.
 
I have just thought of an ulterior motive that adobe may have.

If they can get the world using dng, they can probably get rid of quite a few jobs in their raw conversion department....
 
Rapscallion said:
Using raw + lightroom catalogue, only the catalogue needs to be backed up. This can make a massive difference, especially as I back everything up online. If I choose to change my metadata tags (e.g. change my email address, or change the format of peoples name from first, last to last, first) this will mean backing up everything again.
Click to expand...

This - exactly this. Raw plus metadata, any changes means the backup of changes only.
 
A plus point for DNGs is that LR can run faster when doing certain things if you use them. But if you don't want to use them, and I can understand why some wouldn't, then that's fine.
 
Theres been talk of a new version of DNG, how compatable things will be in the future might be an issue Personally I don't see any advantage from changing one raw format for what is essentially another with less support
If it works for you fine, but it's not for me.
 
I'll be staying with raw. File formats don't just become extinct overnight. Even if it does, it will supported for long enough to convert, so just wait until it's necessary.

I recently found some old Kodak Photo CDs from the 90s. It's been a defunct format for almost a decade, and I still managed to find a program to convert them to 16bit TIFFs.

Plus.. as others have said: There's just as much chance of DNG becoming obsolete as raw. "raw" isn't one file format anyway... there are loads of them. If anything I reckon DNG will disappear first.

I hate to say it, but I think the OP is wasting a great deal of time.
 
Last edited:
My problem is I keep losing track of sidecar files and the plus point for me in DNG is there aren't any.
 
Just wondering what operation you are doing to lose track of your sidecar files?

Have you tried not using sidecar files and just use the lightroom catalogue?
 
Bythesea said:
A plus point for DNGs is that LR can run faster when doing certain things if you use them. But if you don't want to use them, and I can understand why some wouldn't, then that's fine.
Click to expand...

Such as what? Import perhaps, but you've the pre processing of conversion that takes time? Import and render, if you've lots to do then walk away and do something else whilst they are importing. Don't import connected to the camera, use fast CF or SD cards and a fast card reader.

I use Raptor hybrid drives. Have this years Raw on one, Lightroom cache and catalogue on another. Previous years raws are moved to 3Tb sata drives from within lightroom so the latest (this year) are always on the fastest drive. OS and applications are on a SSD. Apart from MS Office, lightroom and photshop, there's no other apps installed.

ShinySideUp said:
My problem is I keep losing track of sidecar files and the plus point for me in DNG is there aren't any.
Click to expand...

How do you lose track of your 'sidecar' files within lightroom? I've never had any issues?

Lightroom keeps the original file and automatically writes adjustments and settings metadata to the catalog. You can also instruct Lightroom to write the changes to XMP. In order for changes made in Lightroom to be recognized by other applications, metadata must be written to XMP.

Choose Edit > Catalog Settings (Windows) or Lightroom > Catalog Settings (Mac OS).
Click the Metadata tab, and then do either of the following:

To write adjustments and settings metadata to XMP, select Automatically Write Changes Into XMP.

Schedule a regular backup within lightroom - I use a week, then have a background job that backs up the important files, robocop, syncbackpro (i'm currently on Windows 7) to copy the changes to external drives.
 
Byker28i said:
Such as what? Import perhaps, but you've the pre processing of conversion that takes time? Import and render, if you've lots to do then walk away and do something else whilst they are importing. Don't import connected to the camera, use fast CF or SD cards and a fast card reader.
Click to expand...
I never import from the camera, I use a USB 3.0 card reader so have no issues with that. It takes time but I usually let that run while I go off and do something else. The speed increase is in the rendering of individual images once they are imported, as long as you have selected the "Embed fast load data" option before importing.
 
I tend to lose sidecar files because I'm a bit sloppy when it comes to file movements. I have been known to move pictures to another directory (sorry, folder) outside Lightroom and forget the XMP files so when I re-locate the file there are no adjustments to the picture, it is just as it was taken which is when I start looking for the XMP file.

I suppose I should start at the beginning and learn exactly how Lightroom does it's cataloging, etc. I know there is help but TBH I'd rather read about it than sit through endless help-videos looking for one small piece of information.
 
Another reason to avoid sidecar files is something I heard recently. The story went (and I heard it from a person that I believe) that someone who wasn't particularly IT confident, noticed that they had a lot of duplicate files. So they deleted all the big versions to save space... :eek:
 
Gazamonk said:
Converting to DNG is sometimes not a choice. Eg. You buy a newer camera and your post processing software no longer supports the latest version of camera RAW. In that case conversion to DNG is not an option but I do concur with Phil and Byker, it's hard to justify the extra work.
Click to expand...
Surely DNG is a backwards compatible format so a DNG converted from a RAW from a newly released camera would still work with an older version of photoshop even if that version does not work with the unconverted RAW? I have got an old copy of Lightroom (v3) and that will not support RAW files from many new cameras but once converted to DNG, LR3 will then read them.
 
ShinySideUp said:
I tend to lose sidecar files because I'm a bit sloppy when it comes to file movements. I have been known to move pictures to another directory (sorry, folder) outside Lightroom and forget the XMP files so when I re-locate the file there are no adjustments to the picture, it is just as it was taken which is when I start looking for the XMP file.

I suppose I should start at the beginning and learn exactly how Lightroom does it's cataloging, etc. I know there is help but TBH I'd rather read about it than sit through endless help-videos looking for one small piece of information.
Click to expand...

Yeah really - Adobe have some great short help videos on how it all works, split down into the sections. You should always move files within lightroom, but even moving the original files isnt terrible (just can be time consuming) as you click and say find missing and point it at the new folder. I've never manually moved XMP files, no need.
 
Byker28i said:
Yeah really - Adobe have some great short help videos on how it all works, split down into the sections. You should always move files within lightroom, but even moving the original files isnt terrible (just can be time consuming) as you click and say find missing and point it at the new folder. I've never manually moved XMP files, no need.
Click to expand...
I would question why you would be moving the files, full stop.

I have created a folder to contain all my pictures, with a hierarchy yyyy/yyyy-mm-dd[shoot description]/FILENAME. Lightroom automatically sorts the photos on import into the correct folder. I never have to move anything.

Just wondering Martin, why are you moving files?
 
I have a around 130,000 images on my computer, stored on a number of drives. Some of these are significantly faster than the others.

I have a yearly folder structure 2012, 2013, 2014, 2015 etc as Lightroom inports into folders by date of image taken. I find this is a good storage system as an additional way to assist with finding images within lightroom.

The latest year, 2015, sits on a fast hybrid disk. My catalogue and lightroom cache sit on another and previous years work sit on slower, large sata drives. This gives best performance, but this does mean once a year I move the previous years files from the fast to slower drives. All this is easily done within lightroom.

For the categories, I use a heierarchy of categories and subcategories, i.e. Family, year, event, or for my degree work, OCA, course, part of the course, exercise. I also keyword on import.

So I can then search by date, category and event, metadata serach on keywords, lens etc. Makes it very easy to find things.
 
Rapscallion said:
I would question why you would be moving the files, full stop.

I have created a folder to contain all my pictures, with a hierarchy yyyy/yyyy-mm-dd[shoot description]/FILENAME. Lightroom automatically sorts the photos on import into the correct folder. I never have to move anything.

Just wondering Martin, why are you moving files?
Click to expand...


I keep changing my mind as to how to file my pictures, videos and scans; getting rid of XMP files means one more thing I don't have to be concerned about.
 
Well, I have taken all your comments on board, the pros, the cons, the in-betweens and have changed my mind. It seems what I really need to is to get my file system organised then the existence of XMP files will be inconsequential.

One last question regarding Lightroom. Why does it create so many folders? It seems to create a folder for every single photograph. I only noticed this when I did a backup using Microsoft SyncToy (excellent backup software by the way) for the first time after installing Lightroom and found it was backing up more folders than I could shake a stick at.
 
ShinySideUp said:
Well, I have taken all your comments on board, the pros, the cons, the in-betweens and have changed my mind. It seems what I really need to is to get my file system organised then the existence of XMP files will be inconsequential.

One last question regarding Lightroom. Why does it create so many folders? It seems to create a folder for every single photograph.
Click to expand...

???

Not sure how you're managing to get all these problems. It creates nothing here. There is a Lightroom folder, and within that is my catalogue, a catalogue previews folder, a settings folder, and a couple of folders for plugins I've installed. Lightroom does nothing to my images folder... adds nothing, creates nothing, moves nothing.

View attachment 36342

That's it. There are a few other folders hidden away in C:\users\David\AppData etc... but you can essentially forget about those as their hidden system files.

There is also another folder where the application is actually installed... in my case C:\\Program Files\Adobe\Adobe Photoshop Lightroom

View attachment 36344

But those are the actual program files... nothing is updated in there except settings or new develop presets, plug-ins etc.



Your lightroom catalogue just links to your files... that's all. It should be making no changes to them, or adding folders to your images folder unless you have specifically asked it to add sidecar XMP files, and even then they should be just sitting there along with the images, not in separate folders.

Set Lightroom to use the catalogue.. not XMP files. If you need to export your settings for an image to another program you can export them for that image only as a XMP. No need to have sidecar files for all your images with LR.



I imported images once and once only when I first started using Lightroom. From then on, after shooting, I don't import from Lightroom. I just drag the raw files into the appropriate folder in Windows (or create one in Windows if none already exists for new projects)... then in Lightroom, right click that folder and synchronise it.

View attachment 36343

Lightroom just then updates the catalogue. No folder created by Lightroom here.
 
Last edited:
I've just had a search through my LR folders and find that all that multitude of folders is in the folder 'lightroom catalog previews' and indeed there are many, many folders labeled from A to F each one having fifty to a hundred sub-folders: SyncToy backs all of these up. I suppose i could exclude them from the backup but why are they there?
 
ShinySideUp said:
I've just had a search through my LR folders and find that all that multitude of folders is in the folder 'lightroom catalog previews' and indeed there are many, many folders labeled from A to F each one having fifty to a hundred sub-folders: SyncToy backs all of these up. I suppose i could exclude them from the backup but why are they there?
Click to expand...

That's normal. They're the previews generated when you view a photo. They're actually only temporary files for full size previews, and 1:1 previews are set to delete after 24 hours unless you've changed it. Smaller previews for thumbnails live in their too. You can safely exclude them from back up. If you need to restore from a back up without them, the first time you look at images in LR, they may take a little longer to display is all, as the previews will just rebuild.

As you described it, it sounded as if your actual image folder was getting filled up with stuff. All previews should be in c:\\lightroom\lightroom 'x' [catalogue_name].lrdata
 
Last edited:
Rapscallion said:
I have just thought of an ulterior motive that adobe may have.

If they can get the world using dng, they can probably get rid of quite a few jobs in their raw conversion department....
Click to expand...
Adobe have openly stated that their main reason for promoting dng is to reduce the amount of work that they (Adobe) have to do writing software to support all the different raw formats.
I find this difficult to understand, since all the basic algorithms have been written and any new raw files from existing manufacturers are just tweaks to the basic file format.

ShinySideUp said:
My problem is I keep losing track of sidecar files and the plus point for me in DNG is there aren't any.
Click to expand...
I puzzled over this statement until I read your next post.

ShinySideUp said:
I tend to lose sidecar files because I'm a bit sloppy when it comes to file movements. I have been known to move pictures to another directory (sorry, folder) outside Lightroom and forget the XMP files so when I re-locate the file there are no adjustments to the picture, it is just as it was taken which is when I start looking for the XMP file.

I suppose I should start at the beginning and learn exactly how Lightroom does it's cataloging, etc. I know there is help but TBH I'd rather read about it than sit through endless help-videos looking for one small piece of information.
Click to expand...
Any file movement should always be done from within Lightroom and then all the changes are known to Lightroom.
This is one of the basic rules of using Lightroom.
If you're "a bit sloppy when it comes to file movements" then you should learn NOT to be, and understand file management within Lightroom.
There are loads of tutorials about this (Lightroom file management), so there's no excuse in not understanding it.
Xmp files contain a lot more than just the adjustment information, and if you are not titling and keywording all your image files then you are wasting a big part of Lightroom's file management features.

The only time I do anything to my files outside Lightroom is when I make backups to external (USB) drives, but this is not "moving" the file in the computer.
For backups I use Microsoft "Sync Toy" to make regular incremental backups to external USB drives.
When the image file is copied, the xmp sidecar file is automatically copied alongside it, so I finish up with a backup of both - simple to do and nothing is lost.

I'd agree that by using dng you embed the xmp information within a single image file, but other than that, dng seems like an unnecessary extra step to me, and I'd always choose to keep the original raw file, in case an improved raw processor became available, and I may want to revisit and re-process some of my old files.
I know that with certain raw files (Canon CR2, I'm not sure of others) it is possible to embed the xmp inside the image file, but so far manufacturers have not endorsed such practices.
 
Last edited:
Brian G said:
Any file movement should always be done from within Lightroom and then all the changes are known to Lightroom.
Click to expand...

I never do this. I move files around in Windows. Why would I use Lightroom as a file management system... it sucks. I move files, then re-sync folders in LR. Works perfectly, and I've been doing this for years. Then again.... I don't use sidecar files. I've no idea why anyone would use sidecar files in Lightroom.
 
Pookeyhead said:
I never do this. I move files around in Windows. Why would I use Lightroom as a file management system... it sucks. I move files, then re-sync folders in LR. Works perfectly, and I've been doing this for years. Then again.... I don't use sidecar files. I've no idea why anyone would use sidecar files in Lightroom.
Click to expand...

That may work for you Dave, but for me that is an unnecessary additional step. That said, I rarely move files around after the initial import.
 
Pookeyhead said:
I never do this. I move files around in Windows. Why would I use Lightroom as a file management system... it sucks. I move files, then re-sync folders in LR. Works perfectly, and I've been doing this for years. Then again.... I don't use sidecar files. I've no idea why anyone would use sidecar files in Lightroom.
Click to expand...
Sorry if it wasn't clear, but my implication was that you move your image files within Lightroom - NOT that you use LR as a general file manager, of which Windows is obviously a lot more effective.
Adobe recommend that you always move your catalog files within Lightroom, so the program is always aware of where your files are located, otherwise you have to go through the process of re-Syncing your catalog.
I too rarely move my image files after the initital import.
 
Brian G said:
Sorry if it wasn't clear, but my implication was that you move your image files within Lightroom - NOT that you use LR as a general file manager, of which Windows is obviously a lot more effective.
Adobe recommend that you always move your catalog files within Lightroom, so the program is always aware of where your files are located, otherwise you have to go through the process of re-Syncing your catalog.
I too rarely move my image files after the initital import.
Click to expand...

You don't need to re-sync the entire catalogue... just the folders in which file movement has taken place. It's for this reason I never "import" images via Lightroom either. I just copy raw files from the cards, straight into the relevant image folder, then re-sync that folder. Just works for me. My point is that there is no "danger" in moving raw files around outside of lightroom. DNG's are problematic because of sidecar files though... another reason not to use DNGs so far as I'm concerned.
 
Brian G said:
Tutorial: Why you should move your files within Lightroom.
Click to expand...


That's not showing anything not already discussed.

I do not move files in Lightroom.. nor am I going to start doing so. I manage all files in Windows... because I find it easier. After being sorted, moved, or re-arranged, I will re-sync. It works. Simple as.

You don't have to OBEY the lady in that tutorial you know. That's not the only way to manage your files. I find it easier to manage files in Windows, and re-sync, than I do managing files in Lightroom... which is a actually a pain in the ass. As a file management system... Lightroom blows.
 
This has been a very useful discussion. Upon reflection I might actually stick with the original raw files and start using the side cars.

It tend to perform incremental backups and I don't want the full raw file backed up several times. It helped me realise what the issues is.

But yes I find it much less risky to move my folders around within lightroom opposed to messing about with then outside the awareness of the catalog.
 
You must log in or register to reply here.
Back
Top