New (imported) Capture One catalogue just over 10% of previous size!

[ChrisR]

A little while back I had a few problems with my C1Pro library catalogue: things missing appearing to be in the wrong place, and so on. I did a couple of repairs, but that left most issues un-fixed, so in the end I decided to make a new C1Pro [catalogue] and import my old [catalogue]. I have a bit under 42,000 images; it turns out that 3 were in the [catalogue] (they have since been deleted), and the remainder were referenced (ie the originals stay where they were in the file store). I was staggered to find out that the old [catalogue] was 42GB, while the [catalogue] library is just 4.5GB!

I'm a little concerned about this size drop, and at a loss to explain it. I'm worried that there might be (surely, must be) some significant information loss here However, I have yet to discover what it could be.

I didn't make any changes to C1Pro settings. I did wonder whether the image preview size had changed for some reason, but it is set at 2048 (the selected screen resolution for my M1 MBP is 2056x1329).

Obviously I'd quite like to get rid of the old [catalogue] and get 40GB of disk space back ("get rid" here means moving to a USB drive).

Anyone any idea what's going on?

EDIT, TLDR: It appears that previews are not rebuilt until the appropriate folder is visited, which accounts for the huge reduction.

 

[ecoleman]

History?
Cache?
Previews?

 

[myotis]

A little while back I had a few problems with my C1Pro library: things missing appearing to be in the wrong place, and so on. I did a couple of repairs, but that left most issues un-fixed, so in the end I decided to make a new C1Pro library and import my old library. I have a bit under 42,000 images; it turns out that 3 were in the library (they have since been deleted), and the remainder were referenced (ie the originals stay where they were in the file store). I was staggered to find out that the old library was 42GB, while the new library is just 4.5GB!

I'm a little concerned about this size drop, and at a loss to explain it. I'm worried that there might be (surely, must be) some significant information loss here However, I have yet to discover what it could be.

I didn't make any changes to C1Pro settings. I did wonder whether the image preview size had changed for some reason, but it is set at 2048 (the selected screen resolution for my M1 MBP is 2056x1329).

Obviously I'd quite like to get rid of the old library and get 40GB of disk space back ("get rid" here means moving to a USB drive).

Anyone any idea what's going on?

My catalogue with 50,000 images is 52gb, so although I can't help you, that does sound like a big drop.

Have you waited long enough for all the previews to be rebuilt?

 

[droj]

Could it be that importing the old library didn't import its associated previews? I don't think the previews are (re-)built until you open (in C1) any given image source folder for the first time, upon which you may see the little wheel spinning top right as they are built .... which can take a while ...

 

[ChrisR]

History?
Cache?
Previews?

Not sure I know what History is in this context. If it's a cache it was, um, extremely large.

Previews seems like the obvious, but...

Could it be that importing the old library didn't import its associated previews? I don't think the previews are (re-)built until you open (in C1) any given image source folder for the first time, upon which you may see the little wheel spinning top right as they are built .... which can take a while ...

I've just opened a random project that I've not looked at for some years, and the previews came up instantly!

My catalogue with 50,000 images is 52gb, so although I can't help you, that does sound like a big drop.

Have you waited long enough for all the previews to be rebuilt?

I imported the old catalogue on 1st May, so I think I've waited long enough. The size quoted above was today's figure.

EDIT: just estimated 42000 images at the resolution above, 3 bytes per pixel and I get about 66 MByte, so that seems the wrong order of magnitude! EDIT again: my calculation here is clearly rubbish! See below...

 

[myotis]

Not sure I know what History is in this context. If it's a cache it was, um, extremely large.

Previews seems like the obvious, but...


I've just opened a random project that I've not looked at for some years, and the previews came up instantly!


I imported the old catalogue on 1st May, so I think I've waited long enough. The size quoted above was today's figure.

EDIT: just estimated 42000 images at the resolution above, 3 bytes per pixel and I get about 66 MByte, so that seems the wrong order of magnitude!

Using my catalogue size of about 50gb for 50,000 files and your original catalogue size of 42gb for 42,000 files, I think we hazard a fairly good guess at the bytes per file

But it doesn't help with your original question, and I haven't got any other ideas :-(

 

[the_accidental]

You’re on a Mac, with APFS, could it be c1 correctly implements duplication rather than copies to not take up additional space? See here for an explanation https://hypercritical.co/hyperspace/#how-it-works (for a different app but it explains the file system feature well).

However I think that should only affect free space reporting not individual file/folder size. So I’ll guess previews but they build quickly on a m1?

but it is set at 2048 (the selected screen resolution for my M1 MBP is 2056x1329).

As an aside your screen resolution is higher than that, that’s the display size in points, it has more pixels which will be used to render text/pictures and so on. Check the apple tech specs (different for 14 vs 16 mbp).

42000 images at the resolution above, 3 bytes per pixel and I get about 66 MByte,

They’ll be stored compressed and probably not 8bit rgb. But if they were 3 bytes a pixel, 2056x1329 (say), that’s 320Gb for 42000 images. But they’ll be compressed.

 

[ecoleman]

Not sure I know what History is in this context. If it's a cache it was, um, extremely large.

I don't use Capture One so don't know what is stored in the catalog hence my question marks.

Lightroom stores an edit history which has to be stored somewhere. Perhaps C1 does the same.

 

[ChrisR]

OK, thanks to @the_accidental for demonstrating my arithmetic above was clearly rubbish, apologies for that!

I have just done a couple of tests. Similar to what was mentioned above, I opened a project that I had not looked at since making the new catalogue. Previews came up instantly (maybe I saw the slightest spinning wheel for a fraction of a second?). Then I went into finder, found another folder that I'd not looked at, and changed its name. In C1Pro I looked at that project... and there were no previews! Instead there was a blank rectangle with a large question mark and some text saying the files didn't exist or were corrupt. This was quite different from what I've seen before in the "mature" catalogue if a file went offline, when the previews would be shown with a small question mark at the top. Went back to the finder, reverted the folder name, quickly flipped to C1Pro and I could see the mini-preview images in the film strip appearing in front of my eyes.

I guess the speed of the M1 made those previews come up really fast (and the source files are JPEGs from scans, so much easier than RAW files).

So, I think it's reasonable to assume that the large reduction in size, as @droj suggested, is because previews for un-visited projects have not been rebuilt, and so the library will increase in size as more and more previews are rebuilt.

This does save disk space, but also of course negates one of the claimed benefits of C1Pro's approach: we can continue to adjust images based on the previews even if the source images have gone off-line. I haven't yet found a way to cause all previews to be rebuilt.

BTW C1Pro backups store images metadata (adjustments) and a few other bits and bobs, but not previews (@ecoleman I think this is the edit history you're referring to). I just checked and the last one of the old catalogue before importing was 229 MB; the current one is 224 MB. So no major change there (no idea why it has slightly dropped).

 

[myotis]

OK, thanks to @the_accidental for demonstrating my arithmetic above was clearly rubbish, apologies for that!

I have just done a couple of tests. Similar to what was mentioned above, I opened a project that I had not looked at since making the new catalogue. Previews came up instantly (maybe I saw the slightest spinning wheel for a fraction of a second?). Then I went into finder, found another folder that I'd not looked at, and changed its name. In C1Pro I looked at that project... and there were no previews! Instead there was a blank rectangle with a large question mark and some text saying the files didn't exist or were corrupt. This was quite different from what I've seen before in the "mature" catalogue if a file went offline, when the previews would be shown with a small question mark at the top. Went back to the finder, reverted the folder name, quickly flipped to C1Pro and I could see the mini-preview images in the film strip appearing in front of my eyes.

I guess the speed of the M1 made those previews come up really fast (and the source files are JPEGs from scans, so much easier than RAW files).

So, I think it's reasonable to assume that the large reduction in size, as @droj suggested, is because previews for un-visited projects have not been rebuilt, and so the library will increase in size as more and more previews are rebuilt.

This does save disk space, but also of course negates one of the claimed benefits of C1Pro's approach: we can continue to adjust images based on the previews even if the source images have gone off-line. I haven't yet found a way to cause all previews to be rebuilt.

BTW C1Pro backups store images metadata (adjustments) and a few other bits and bobs, but not previews (@ecoleman I think this is the edit history you're referring to). I just checked and the last one of the old catalogue before importing was 229 MB; the current one is 224 MB. So no major change there (no idea why it has slightly dropped).

If you select all files cmd+A and right click on any one of them you will see a rebuild previews option.

If the old catalogue was built before C1 16.3, you should still see a reduction in catalogue size, as they reduced the size of the preview files by up to 41% with that release

I;m not sure I follow your point about working offline the way that C1 works offline is because the previews are stored in the catalogue, but your catalogue hasn't built th previews.

With me it seems to do strange things, when I make major changes, and it builds previews everytime I relaunch C1, even though it had already suggested all the previews had been built

I've never imported an entire catalogue, so not sure what should happen, as the previews should already exist.. Maybe, assuming the older catalogue predates 16.3, and you are using a newer version, its to do with the change in how peviews are being built. Still doesn't sound right though.

 

[ChrisR]

If you select all files cmd+A and right click on any one of them you will see a rebuild previews option.

Thanks for this. Did Select All as you suggest, and then re-generated previews. Spinning wheels for a couple of hours; my catalogue is now 22 GB in size. Old catalogue(s) copied to offline disk, backup in progress, after which I'll remove the old catalogue from the main hard drive.

If the old catalogue was built before C1 16.3, you should still see a reduction in catalogue size, as they reduced the size of the preview files by up to 41% with that release

I'm still on C1 16.2.6.7, from when I bought C1Pro 23.

I've never imported an entire catalogue, so not sure what should happen, as the previews should already exist.. Maybe, assuming the older catalogue predates 16.3, and you are using a newer version, its to do with the change in how previews are being built. Still doesn't sound right though.

I think it must simply import the catalogue without the previews.

Thanks for your help everyone. Your suggestions have been extremely useful.

 

[ChrisR]

BTW I have been blithely clicking "yes" when C1Pro asks if I'd like to backup the catalogue on exit. The largest backup I could find was around 220 MB. However, taken together, all my backups since 2019 amount to over 20GB! Probably time for a bit of weeding there...

 

[myotis]

Thanks for this. Did Select All as you suggest, and then re-generated previews. Spinning wheels for a couple of hours; my catalogue is now 22 GB in size. Old catalogue(s) copied to offline disk, backup in progress, after which I'll remove the old catalogue from the main hard drive.

That is still a good reduction in size, it makes my database sound big, but I'm sure it was over 80gb at one time. Maybe I need to do a preview rebuild, I do it reulalry and I'm pretty sure I did it after 16.3 came out. Maybe need to look at this.

 

[myotis]

BTW I have been blithely clicking "yes" when C1Pro asks if I'd like to backup the catalogue on exit. The largest backup I could find was around 220 MB. However, taken together, all my backups since 2019 amount to over 20GB! Probably time for a bit of weeding there...

I use Hazel to auto delete backups older that a certain age. Of hand I can't remember what that age is, but it keeps it under control for me.