Photo storage and management

A

alchemistkevin

Suspended / Banned
Messages
146
Name
Kevin
Edit My Images
Yes
I've been growing my collection of digital images over the years and now have tens of thousands of images to store, back up and manage.

In the past I used to store images under folders named after events or geographic location of the type of object. It's easy for your first car album to be named car but then car 5 and car 10 doesn't make much sense. From this I reverted to using brand names but they don't make much sense when you've bought your third merc... So it was manufacturer name follow by a year or a month or names of people in the folder followed by the event. However, I soon started having hundreds of folders within the main storage area.

I switched this to a date based system:
The current structure I have is:

Year 1
->Month 1
- - ->Day1
- - ->Day31

->Month 2
- - ->Day1
- - ->Day31

->Month 3
- - ->Day1
- - ->Day31
...

Year 2
->Month 1
- - ->Day1
- - ->Day31

->Month 2
- - ->Day1
- - ->Day31

...

I have also had multiple cameras and have to create multiple folders under dates when shooting with multiple cameras on the same day.

The obvious problem with the current approach is: It keeps files under a directory limited in number and easy to locate once you know the date but it's not fun when you want to find images of your car or your kid.

For storage: I use a USB drive/remote disk - sometimes have a NAS disk connected and always have a copy sync with one of the disks in my desktop.

For backups: I use a batch file on windows platform to synchronize disks with changes and rsync scripts on linux that do me well and I'm not bothered setting up a more complicated RAID setup.


So, what do you use?

Do you have a folder based approach or do you use a piece of software that manages this for you? I know the big studios have some sort of a database system but I keep my images on NAS/remote disks accessible via Windows and Linux platforms and would like to keep it this way. I'll buy a mac if I have to and there's no comparable software on the Windows platform but I'd rather have a simple setup and be application & platform independent.
 
Last edited:
folder structure wise, i just dump RAW into folders labelled with name and date then import to Lightroom and let that deal with organisation.

harware wise to save me typing it out again im going to quote myself from another thread..

neil_g said:
depends on your budget really..

personally for a business id suggest a scalable storage system, something like a synology DS1511+ that is expandable from 5 to 15 disks via addon units and run it in RAID5. but that aint cheap.

i run the DS1010+ which is the previous model and is expandable to 10 disks, they might prove cheaper if still available to buy. in mine i have 5x 2TB drives in RAID5 (giving 7.15TB). i have my RAW files on this, the LR cat is on my computers internal hard drive and backs up to the NAS. on the rear of the synology units are USB ports which you can connect USB devices such as external drives, printers and UPS. (there is also an eSATA but using this will take the port needed for an extention unit).

in one of my USB ports is an IcyBox RAID0 enclosure holding 2x 2TB (total 3.6TB) drives which the NAS backs up to overnight. i can expand this as my data grows by adding another unit or replacing this one. (in RAID0 if one drive fails both drives data is lost, however the chance of that happening plus losing the NAS is unlikely).

i then burn RAW plus LR cat to Blu-Ray as it is more reliable/resiliant media compared to DVD and remove them from site (this is where your cloud backup would kick in i imagine).

i would also concider hooking your storage solution up to a UPS, there is no point having all this storage for it to get wiped out in a power cut/surge. "UPS Trader" on eBay is a good starting point for refurbs. I run an APC SmartUPS 1500 and it easily runs my i7 desktop, 2 screens, NAS, external drive, speakers, network switch, printer.. litterally one light is lit on the load guage :lol:


722Q3091 by neilgates, on Flickr


Dusty 3.6TB Backup. by neilgates, on Flickr

any use?
Click to expand...

taken from here
 
Last edited:
Personally, I just use a directory structure of date (American format) and a brief description. ie, \photos\2011 09 11 Three Sisters Longton

I find it quite easy to find what I'm looking for.

Everything is backed up using Synctoy onto external drives, spare desktop and a copy at work! If I lose any images it'll be through sheer stupidity :D
 
neil_g said:
folder structure wise, i just dump RAW into folders labelled with name and date then import to Lightroom and let that deal with organisation.

harware wise to save me typing it out again im going to quote myself from another thread..



taken from here
Click to expand...

Very impressive Neil. Ive got a synology DS410 with 2x1Tb and 2x2Tb drives. Also have quite a number of internal and external drives of varying capacities (providing backup to the NAS). My backup consists of copying the data and waiting an age as I use wireless and it's quite slow.

You seem to have quite a set up there - without going off topic could I PM you for some info on your setup and backup options?

Cheers
Jim
 
EOS_JD said:
Very impressive Neil. Ive got a synology DS410 with 2x1Tb and 2x2Tb drives. Also have quite a number of internal and external drives of varying capacities (providing backup to the NAS). My backup consists of copying the data and waiting an age as I use wireless and it's quite slow.

You seem to have quite a set up there - without going off topic could I PM you for some info on your setup and backup options?

Cheers
Jim
Click to expand...

sure, though it could be relvent to this thread too as it underpins the hole storage system topic :)
 
Very Similar to Neil.

PC Running Lightroom.
RAW files sat on a volume on my Synology DS710+ NAS
Lightroom catalog lives on local C Drive, but backed up to NAS upon exit from LR.
Overnight scheduled backup of RAW & Catalog to 2.5" drive attached to NAS.

And occasionally, I also export the the RAW's to 2048 x 2048 jpg's and import into Picasa and upload to Google. (can take a while to upload, but nice to have instant access to the photo's from my mobile etc and a "house burn down" solution) lol.
 
EOS_JD I wouldn't mind you posting your discussion on this thread as it is an integral topic of this discussion :)

Thanks for sharing - any more for any more please? :)
 
Similar to something I posted yesterday, but havent really had a definitive answer yet...

If you've got a Lightroom catalog full of your year's worth of images, all the RAW files sat on a drive somewhere, and then you want to do further edits to a specific photo that Lightroom cannot do - therefore you require another external editor (photoshop for me) - is there any way to pass files from LR to PS and back again in Jpeg format?

Im aware you can do this in TIFF files... but 60+mb TIFF files + 30MB RAW files will soon eat up my hard drives lol...

Advice for how to do this would be fantastic!
 
futureal33 said:
Similar to something I posted yesterday, but havent really had a definitive answer yet...

If you've got a Lightroom catalog full of your year's worth of images, all the RAW files sat on a drive somewhere, and then you want to do further edits to a specific photo that Lightroom cannot do - therefore you require another external editor (photoshop for me) - is there any way to pass files from LR to PS and back again in Jpeg format?

Im aware you can do this in TIFF files... but 60+mb TIFF files + 30MB RAW files will soon eat up my hard drives lol...

Advice for how to do this would be fantastic!
Click to expand...

dont think you can do JPG? i know you can do PSD though if that is any use.
 
OK
I'm not sure where to start but really my question revolves around how to back up a system and how best to make use of my NAS. It's very slow when used wirelessly (would it be better wired to the PC)? they are in different rooms but thinking of drilling a hole to allow the cable to pass through.

I'm pretty PC literate however when it comes to networking I'm totally lost.

My system is a bit of a mish mash and am thinking of consolidating the drives a bit.

When I back up it's really me just copying folders and to be honest with moving stuff around I'm finding I have different copies of stuff. I've used SyncToy but not sure that is the best way to do it!

Currently I have the following.

Internal
1x300Gb (windows drive)
1x250Gb
2x200Gb
1x1Tb RAWs backed up to the NAS

External = Mostly with RAWs backed up to the 2Tb NAS
2x500Gb
1x400Gb
1x1Tb
1x200Gb

Synology NAS
2x1Tb (Raid1)
2x2Tb (Raid1)
 
Last edited:
personally id drill the hole and run cat5. id also get the relevent network adapters and network switch to run 1gbps if not in place already. wireless is too slow and flakey for my liking. in areas where drilling is not acceptable then the powerline adapters are worth a look.

i used to use allway sync to backup my data, ive also used robocopy with sucess at work. so synctoy is perfectly acceptable. you also have built in backup util in some versions of windows, thatll do the job too although very basic.

there is certainly scope to consolidate your drives a bit, 2tb units are about £70 a pop now so shouldnt break the bank too much either.
 
neil_g said:
personally id drill the hole and run cat5. id also get the relevent network adapters and network switch to run 1gbps if not in place already. wireless is too slow and flakey for my liking. in areas where drilling is not acceptable then the powerline adapters are worth a look.

i used to use allway sync to backup my data, ive also used robocopy with sucess at work. so synctoy is perfectly acceptable. you also have built in backup util in some versions of windows, thatll do the job too although very basic.

there is certainly scope to consolidate your drives a bit, 2tb units are about £70 a pop now so shouldnt break the bank too much either.
Click to expand...

I take it the cat5 is the ethernet cable I got supplied with Sky's Sagem Router?

"network adapters and network switch to run 1gbps"

have no idea what that is? What does that do?

What do you use as a backup solution? I have some backup software but not sure if it's better than just syncing the data?

Thanks for the assistance - I'll have more questions when I get home probably :)
 
Last edited:
Synology NAS's have a backup option/scheduling in the management software. I just schedule mine to go from the NAS (chosen volume) onto a USB attached drive on a daily basis, so no need to leave PC's on with backup software etc.

If you want to know more about NAS in general, I wrote this guide over on AVForums sometime ago.

http://www.avforums.com/forums/netw...rotection-performance-configuration-more.html

Bits are a little out of date as technology moves on quickly, but most is still relevant, especially the part about Performance etc.
 
EOS_JD said:
I take it the cat5 is the ethernet cable I got supplied with Sky's Sagem Router?
Click to expand...

yeah cat5 is a type of ethernet cable.

"network adapters and network switch to run 1gbps"

have no idea what that is? What does that do?
Click to expand...

the network card in your computer will be rated to a max of 10mbps, 100mbps or 1000mbps depending on how old it is.

most routers wont be 1000mbps, only 100mbps therefore a separate network box called a switch may be required.

What do you use as a backup solution? I have some backup software but not sure if it's better than just syncing the data?
Click to expand...

the synology has a built in backup that lets you take data off of the share(s) on the NAS and backup to an attached USB/eSATA drive or another NAS.

as long as the data is copied be it via a sync utility or a backup program it doesnt matter too much. however its probably worth turning off the option that deletes files from the destination drive when they are deleted from the source so you dont lose the files.
 
neil_g said:
yeah cat5 is a type of ethernet cable.



the network card in your computer will be rated to a max of 10mbps, 100mbps or 1000mbps depending on how old it is.

most routers wont be 1000mbps, only 100mbps therefore a separate network box called a switch may be required.



the synology has a built in backup that lets you take data off of the share(s) on the NAS and backup to an attached USB/eSATA drive or another NAS.

as long as the data is copied be it via a sync utility or a backup program it doesnt matter too much. however its probably worth turning off the option that deletes files from the destination drive when they are deleted from the source so you dont lose the files.
Click to expand...

Very usdeful

Only issue I have is that I currently have quite a lot of data on the NAS (more than 2Tb over the two RAID volumes) so guuess I'd need a big drive connected by USB?

Regards the motherboard I have a dual gigabit lan controller. A marvel one and a realtek one (both standard on my Asus P5K Premium board - What do I need to do to connect the NAS to that? Just plug in the CAT5 cable?

Also how do I know whether the cable I use is cat5 or not?

I believe the Sagem 2504 speed is 10/100. So what would I need regards a switch? Sky don't allow the use of other routers so would that affect me?

Thanks for the help I'm understandig some of it :)
 
Zarch said:
Synology NAS's have a backup option/scheduling in the management software. I just schedule mine to go from the NAS (chosen volume) onto a USB attached drive on a daily basis, so no need to leave PC's on with backup software etc.

If you want to know more about NAS in general, I wrote this guide over on AVForums sometime ago.

http://www.avforums.com/forums/netw...rotection-performance-configuration-more.html

Bits are a little out of date as technology moves on quickly, but most is still relevant, especially the part about Performance etc.
Click to expand...

Mick very interesting and useful.

I didn't realise how poorly a NAS would run over wireless. I get about 2Mb/s MAX and mostly a lot less!!!

So happy to invest a little more to get the speed up. Can attach a USB drive to the NAS for back up too. Might get a couple of 2Tb drives and if I can, I'll go to RAID5.


Will wait and see what the speed is like once I have it connected properly :)

THanks
Jim
 
Jim,

To get your home networked performing well, you really need it wired or to a lesser extent use homeplugs.

Wireless is only any good for browsing (imho) :)

The central part to a good system is a Gigabit switch like this. (only £30)
http://www.ebuyer.com/64190-netgear-gs605-5-port-gigabit-switch-gs605uk

So you leave your internet router as is, but connect it into this switch, so you are extending your "network" over onto this switch. You then plug all our PCs and NAS devices that are capable of Gigabit into this new switch. This then means that your PC will talk to your NAS at Gigabit speed as they are both going through the Gigabit switch.

I imagine you are currently plugging everything into your internet router, which is more than likely only 100Mbit..... so all traffic is throttled at this speed..... not mention the use of wireless <yuk> lol

So for £30 and the cost of a few Cat5e (or cat 6) network cables you should see an improvement.

You can get a variety of lengths and colours of cables from eBuyer for as low as 69p too.
http://www.ebuyer.com/store/Peripherals/cat/Cables-and-Tools/subcat/Network-CAT5,-CAT6

Hope this helps.
PS. Any existing networking cable you have 'should' have cat 5/cat 5e or cat6 written on the outside of the cable..... and most cables that come with devices like routers and NAS 'should' be at least cat 5.
 
Last edited:
EOS_JD said:
Very usdeful

Only issue I have is that I currently have quite a lot of data on the NAS (more than 2Tb over the two RAID volumes) so guuess I'd need a big drive connected by USB?

Regards the motherboard I have a dual gigabit lan controller. A marvel one and a realtek one (both standard on my Asus P5K Premium board - What do I need to do to connect the NAS to that? Just plug in the CAT5 cable?

Also how do I know whether the cable I use is cat5 or not?

I believe the Sagem 2504 speed is 10/100. So what would I need regards a switch? Sky don't allow the use of other routers so would that affect me?

Thanks for the help I'm understandig some of it :)
Click to expand...

pretty much what mick said, just need a 1gbps switch and plug one port into your rouyter then the rest of your devices into the switch.

any of your onboard network ports should do as long as theyre 1gbps, theyll be much and much the same.

i think im right in saying you have 2 USB on the back of your synology so you can split your backup between 2 large (2tb) drives. or i found more space for your money would be the icybox with 2x 2tb in RAID0 (RAID0 offers no protection from hard drive failure but the chances of that failing at the same time as my RAID5 NAS is minimal).
 
Zarch said:
Jim,

To get your home networked performing well, you really need it wired or to a lesser extent use homeplugs.

Wireless is only any good for browsing (imho) :)

The central part to a good system is a Gigabit switch like this. (only £30)
http://www.ebuyer.com/64190-netgear-gs605-5-port-gigabit-switch-gs605uk

So you leave your internet router as is, but connect it into this switch, so you are extending your "network" over onto this switch. You then plug all our PCs and NAS devices that are capable of Gigabit into this new switch. This then means that your PC will talk to your NAS at Gigabit speed as they are both going through the Gigabit switch.

I imagine you are currently plugging everything into your internet router, which is more than likely only 100Mbit..... so all traffic is throttled at this speed..... not mention the use of wireless <yuk> lol

So for £30 and the cost of a few Cat5e (or cat 6) network cables you should see an improvement.

You can get a variety of lengths and colours of cables from eBuyer for as low as 69p too.
http://www.ebuyer.com/store/Peripherals/cat/Cables-and-Tools/subcat/Network-CAT5,-CAT6

Hope this helps.
PS. Any existing networking cable you have 'should' have cat 5/cat 5e or cat6 written on the outside of the cable..... and most cables that come with devices like routers and NAS 'should' be at least cat 5.
Click to expand...

I'm starting to understand and that looks perfect! The only thing plugged into the router at the moment is the NAS. Everything else on my home network is wireless - 2 laptops, 1 PC and an XBox (as well as a few phones).

Thanks for the assistance. Shopping basket filled again :)
 
neil_g said:
pretty much what mick said, just need a 1gbps switch and plug one port into your rouyter then the rest of your devices into the switch.

any of your onboard network ports should do as long as theyre 1gbps, theyll be much and much the same.

i think im right in saying you have 2 USB on the back of your synology so you can split your backup between 2 large (2tb) drives. or i found more space for your money would be the icybox with 2x 2tb in RAID0 (RAID0 offers no protection from hard drive failure but the chances of that failing at the same time as my RAID5 NAS is minimal).
Click to expand...

Yes 2x USB ports on the back and I have used a 1Tb drive in them - Forgot about that. Speed seemed better with the drive attached that way. Will look at my drives and start transferring the data around so I can set that up,

Thanks again for the help. You've both made it so much easier to understand.

Cheers
Jim
 
You must log in or register to reply here.
Back
Top