Why so noisey?

[EOS_JD]

I think lighting is the key factor Jim but I could be wrong. I know very little with respect to mathematical equations,sensors etc. Here is one from a while ago shot @ -5/3 EV. About 2/3 minutes of my very basic PP skills applied, does it look that bad :shrug: Would I do anything different given the opportunity again more as likely I would do, as I am not a stickler for it has to be taken a certain way, my settings could be completely different


Magpie by Richard Venn, on Flickr

Totally agree. In bright light with a scene like this tyhere won't be much difference. Try 2/3rd under in a dark church with a B&G. THen you will see the noise - so yes it's also about the light and the scene. THe OPs image showed a lot of dark pixels

Like I say I have in image 3 stops under that you can't see much noise in.

 

[EOS_JD]

"Correct" exposure based on ISO is a myth.
A sensor only has one native sensitivity.

The problem is you never see an unadulterated raw file, you would be shocked it you could.

You have?

When you set an exposure on a camera that has a native ISO of 100 and set the speed as ISO 200. It simply underexposes by one stop and adds the ISO information to the raw file. ( in the case of a Peg it does the same but processes it, to recover the stop under exposure.

When you later open the raw file in your raw processor It reads the ISO setting and automatically makes the required exposure adjustment.
( different raw processors give slightly different results and different levels of noise.)

However the camera may or may not do additional noise reduction procedures to reduce known noise issues, but it can do nothing to reduce the basic random shot noise (which is an Photon issue related to the actual exposure)

This is all true what ever the ISO set.
Noise is directly related to photons, exposure.

The ISO you set is used by the firmware or software to compensate for the under exposure of the sensor.

Can you show where you get this info as personally I think it's wrong. Changing the ISO changes other things too -If it didn't matter we could all just shoot everything at ISO100 (or 200 depending on your camera)!

 

[Terrywoodenpic]

You have?



Can you show where you get this info as personally I think it's wrong. Changing the ISO changes other things too -If it didn't matter we could all just shoot everything at ISO100 (or 200 depending on your camera)!

The main source of my information is from articles by Professor Bob Newman.
And follow up reading.

Your conclusion is not far from the truth, as far as the sensor is concerned.

But the firmware does need the ISO setting information as it has no other way of knowing the extent of, or the noise characteristics of the underexposure set that it needs to reduce. Different amounts of underexposure need different treatments.
ISO is a convenient measure that people feel comfortable with. However a range from A to Z would work just as well, It is simply a scale, as on an exposure meter (which also does nothing except set a scale)

 

[Johnytuono]

It does if you look at the full size image on Flickr

Ah yes now I see it on the full size image...colour noise....thanks for that. Would still like to see the unaltered photo though just to see how much it was 'brought up'.
JohnyT

 

[HoppyUK]

Terry, we know it is far better to expose at say ISO800 than it is to shoot at ISO100 and attempt to restore brightness in post processing. But since the light/photons falling on the sensor is the same in both cases, then one way or another the same amount of amplification/gain is required either way. So why the difference in quality?

Is there a layman's analogy in hi-fi and vinyl albums? In those days, there was a pre-amp that took the tiny signal from the stylus/cartridge and turned it efficiently into something manageable for the main amp to build further into high quality sound. The reason being, no main amp could take the tiny cartridge output directly and make it properly audible without making a mess along the way.

Is that kind of what's happening here with ISO?

 

[Terrywoodenpic]

Terry, we know it is far better to expose at say ISO800 than it is to shoot at ISO100 and attempt to restore brightness in post processing. But since the light/photons falling on the sensor is the same in both cases, then one way or another the same amount of amplification/gain is required either way. So why the difference in quality?

Is there a layman's analogy in hi-fi and vinyl albums? In those days, there was a pre-amp that took the tiny signal from the stylus/cartridge and turned it efficiently into something manageable for the main amp to build further into high quality sound. The reason being, no main amp could take the tiny cartridge output directly and make it properly audible without making a mess along the way.

Is that kind of what's happening here with ISO?

On a CCD sensor there is a single amplifier that amplifies the whole signal. A CMOS sensor has an individual pixel sensor amplifiers on the sensor itself.

All subsequent processing is done by the processors firmware. Such as processing for ISO. (some times this is changed in a firmware up date to improve noise and processing).
You do not have the equivalent of a knob and amplifier to adjust gain as on a sound amplifier. The signal after AD conversion stays as it is. You only need a readable signal in a digital sense, it does not need to be a powerful one.

Image processing is a purely digital manipulation.

This is a dissertation I found on the net covering noise in CMOS sensors. You will note that nowhere does it mention ISO, It is simply not a factor.
http://www-isl.stanford.edu/~abbas/group/papers_and_pub/hui_thesis.pdf

Most camera/photography based articles talk about almost nothing else than ISO/noise.

I can not think of any other field where the science and the practical application have lead to such opposite thinking.

You asked why the in camera ISO processing seems so much better than you can do on a computer.? I would suggest that if you had the algorithms that the camera uses you would be able to match the results. Perhaps that is exacty what we will do one day.

 

[gad-westy]

On a CCD sensor there is a single amplifier that amplifies the whole signal. A CMOS sensor has an individual pixel sensor amplifiers on the sensor itself.

All subsequent processing is done by the processors firmware. Such as processing for ISO. (some times this is changed in a firmware up date to improve noise and processing).
You do not have the equivalent of a knob and amplifier to adjust gain as on a sound amplifier. The signal after AD conversion stays as it is. You only need a readable signal in a digital sense, it does not need to be a powerful one.

Image processing is a purely digital manipulation.

This is a dissertation I found on the net covering noise in CMOS sensors. You will note that nowhere does it mention ISO, It is simply not a factor.
http://www-isl.stanford.edu/~abbas/group/papers_and_pub/hui_thesis.pdf

Most camera/photography based articles talk about almost nothing else than ISO/noise.

I can not think of any other field where the science and the practical application have lead to such opposite thinking.

You asked why the in camera ISO processing seems so much better than you can do on a computer.? I would suggest that if you had the algorithms that the camera uses you would be able to match the results. Perhaps that is exacty what we will do one day.

It's an interesting topic this one. I'm sure I'm not the only that blindly (or otherwise) follows the philosophy of exposing to the right. I usually achieve this by upping the ISO as chances are if you're up at this level, you'll already be compromised on shutter speed and aperture. A little while ago there was quite a lengthy discussion that steered in the direction of this. I think it was Phil Young that tried taking a few shots, one correctly exposed at a high iso (maybe 6400) and another badly under exposed at a lower iso then pulled back in PP. I don't remember seeing any significant difference between the two. It's made me question the whole 'ETTR' thing ever since though I've never got around to doing my own experiment.

 

[Terrywoodenpic]

It's an interesting topic this one. I'm sure I'm not the only that blindly (or otherwise) follows the philosophy of exposing to the right. I usually achieve this by upping the ISO as chances are if you're up at this level, you'll already be compromised on shutter speed and aperture. A little while ago there was quite a lengthy discussion that steered in the direction of this. I think it was Phil Young that tried taking a few shots, one correctly exposed at a high iso (maybe 6400) and another badly under exposed at a lower iso then pulled back in PP. I don't remember seeing any significant difference between the two. It's made me question the whole 'ETTR' thing ever since though I've never got around to doing my own experiment.

Modern sensors seem to have excellent headroom. and shots exposed between 100 and 800 ISO show little difference in quality.
However Dynamic range and well capacity are strongly linked.
Think of a sensor as a container for photons. In dark situations there are few photons to collect, but the system still generates noise of various sorts. The signal to noise ratio in this situation favours noise.
At the other end of the scale when the containers are full to overflowing we say the image is clipped and we lose highlight detail. But at the same time the point just below that, has an excellent signal but in fact greater noise, as noise increases with signal... however the signal to noise ratio is now in favour of the signal, so we experience little or no noise.

As the scientists find ways to increase the dynamic range of sensors, so it will become easier to extract high quality over a wider range.
However our cameras will never be able to generate high quality in low light levels, where the incoming photons are few and far between.
For that you need a different technology as used in astronomy.

Our sensor already count photons over an amazingly wide range, from deep moonlight to the brightest sunlight. But to do so we have to restrict the dynamic range falling on the sensor to match its inbuilt range, by adjusting the shutter speed and aperture.

To get the best out of an image we want to put the brightest point Just before the clipping point. Hence the instruction to expose to the right.

When you do so, you get the widest possible tonal range with the least possible noise.
(But you should also use the lowest ISO conditions will allow)

 

[Rob Thomson]

Terry, those are really interesting posts. Thanks very much.

 

[HoppyUK]

It's an interesting topic this one. I'm sure I'm not the only that blindly (or otherwise) follows the philosophy of exposing to the right. I usually achieve this by upping the ISO as chances are if you're up at this level, you'll already be compromised on shutter speed and aperture. A little while ago there was quite a lengthy discussion that steered in the direction of this. I think it was Phil Young that tried taking a few shots, one correctly exposed at a high iso (maybe 6400) and another badly under exposed at a lower iso then pulled back in PP. I don't remember seeing any significant difference between the two. It's made me question the whole 'ETTR' thing ever since though I've never got around to doing my own experiment.

Similar test here, with lots of dark shadow detail, from Raw files.

First crop correctly exposed, ISO800 1/250sec at f/11, the second one at ISO100 with same camera settings, so under-exposed by 3 stops, then the brightness recovered in Lightroom. No other adjustments applied, camera Canon 7D.

Clearly the correctly exposed shot is far better than the under-exposed and recovered one, but with both shots the sensor has received exactly the same amount of light, and therefore both images have received the same amount of 'gain' somewhere along the line to achieve the same brightness.

So what is happening in-camera that is so much more efficient than post-processing?


Full image

Crop from centre, ISO800, correct exposure

ISO100, under-exposed -3 stops, then pushed +3 stops to ISO800 equivalent in Lightroom

And just for good measure, same shot correctly exposed at ISO100 - 1/30sec at f/11

 

[EOS_JD]

It's possible the sensor "gain" is much purer in-camera.

 

[HoppyUK]

It's possible the sensor "gain" is much purer in-camera.

That was what I was alluding to in my post #45 above.

But what, exactly, is going on? I struggled manfully with the link Terry posted previously to a PhD thesis, but if somebody can tell me in English what it says there, I'd be grateful though it's now 13 years old in a fast changing era when it was probably out of date before the ink was dry.

 

[EOS_JD]

The designers at Canon/Nikon will know

 

[LCPete]

Terry will be able to explain it way better but I think.its because the signal is amplified in camera before its had any processing done to it
The raw file that's exported from the camera has had processing done to it and altering the "gain " afterwards in the computer will never be as good

 

[Cagey75]

Never copped this one, but without reading through I'm going to say you were well under exposed taking this. Doesn't look like you tried recovering too much in post, but if it was even darker than this to begin, recover would have added noise. Not that I can see much tbh.

 

[Terrywoodenpic]

Similar test here, with lots of dark shadow detail, from Raw files.

First crop correctly exposed, ISO800 1/250sec at f/11, the second one at ISO100 with same camera settings, so under-exposed by 3 stops, then the brightness recovered in Lightroom. No other adjustments applied, camera Canon 7D.

Clearly the correctly exposed shot is far better than the under-exposed and recovered one, but with both shots the sensor has received exactly the same amount of light, and therefore both images have received the same amount of 'gain' somewhere along the line to achieve the same brightness.

So what is happening in-camera that is so much more efficient than post-processing?

I suspect is because the in camera processing is using Algorithms specific to the noise characteristics of that camera. Where as the Raw processor must use a generic one.

It does not mean the raw processors we buy are in any way bad, it is just that they can not be as fine tuned to a particular cames sensor, as an individual camera maker can.

The same sensors are often used in many cameras, however the results are rarely identical.

In your example the Iso 100 raw file recieved the normal 100 ISO noise correction so did nothing about the unexpected noise.
In your Iso 800 shot all the considerable experience of the Firmware designers were bought into action.

The correctly exposed shot might have even taken a bit more exposure to the right before all detail was lost in the snow, this could have opened out the tones in the leaves even further.

 

[HoppyUK]

I suspect is because the in camera processing is using Algorithms specific to the noise characteristics of that camera. Where as the Raw processor must use a generic one.

It does not mean the raw processors we buy are in any way bad, it is just that they can not be as fine tuned to a particular cames sensor, as an individual camera maker can.

The same sensors are often used in many cameras, however the results are rarely identical.

Sorry Terry, but I don't buy that. You seem to be saying, I think, that if I'd post-processed in Canon's own DPP software the pushed result would have been the same as the correctly exposed one. I don't have DPP loaded on this computer so I can't do that comparison, but I (and the rest of the world) would be amazed if the result was any different. That just doesn't happen with any post processing regime. I didn't do the comparison above to reveal anything new or surprising, just to give a working example of how pushing in post is not the same, or anywhere near as good, as setting the correct ISO in-camera. Nothing new there.

It's more fundamental than the software used, something happening at the analogue-to-digital conversion and gain applied at sensor level, at source. I'd just like to know exactly what and why it's better that way.

Yes Lightroom is generic software, but the only significant aspect of that is the de-mosaicing algorithm, and since 99% of current cameras use a Bayer pattern RGB filter array, that's not a factor here. Only Fuji's X-Trans and Sigma's Foveon are different in that respect and I acknowldge that LR currently has problems with X-Trans as I did that comparison myself last week.

PS Just seen your edit. Exposure was pretty full - one third of a stop more and the snow was close to clipping (blinkies flashing). I could probably have given it a bit more and got away with it, but that's not really the point - the point is, the exposure settings were exactly the same but the result is not.

 

[Suvv]

I thought that the difference was because the ISO setting applies amplification to the analogue signal before it is converted to digital.
The digital conversion will lose data, think about all the discussions re: 12 bit and 8 bit, the analogue signal contains all the data much more than even raw, no losses due to sampling intervals, no losses due to rounding errors etc.

 

[HoppyUK]

I thought that the difference was because the ISO setting applies amplification to the analogue signal before it is converted to digital. The digital conversion will lose data, think about all the discussions re: 12 bit and 8 bit, the analogue signal contains all the data much more than even raw, no losses due to sampling intervals, no losses due to rounding errors etc.

That, I think, is probably the top and bottom of it.

 

[Terrywoodenpic]

Sorry Terry, but I don't buy that. You seem to be saying, I think, that if I'd post-processed in Canon's own DPP software the pushed result would have been the same as the correctly exposed one. I don't have DPP loaded on this computer so I can't do that comparison, but I (and the rest of the world) would be amazed if the result was any different. That just doesn't happen with any post processing regime. I didn't do the comparison above to reveal anything new or surprising, just to give a working example of how pushing in post is not the same, or anywhere near as good, as setting the correct ISO in-camera. Nothing new there.

It's more fundamental than the software used, something happening at the analogue-to-digital conversion and gain applied at sensor level, at source. I'd just like to know exactly what and why it's better that way.

Yes Lightroom is generic software, but the only significant aspect of that is the de-mosaicing algorithm, and since 99% of current cameras use a Bayer pattern RGB filter array, that's not a factor here. Only Fuji's X-Trans and Sigma's Foveon are different in that respect and I acknowldge that LR currently has problems with X-Trans as I did that comparison myself last week.

PS Just seen your edit. Exposure was pretty full - one third of a stop more and the snow was close to clipping (blinkies flashing). I could probably have given it a bit more and got away with it, but that's not really the point - the point is, the exposure settings were exactly the same but the result is not.

Canons Software would not have helped.
The in camera firmware does the processing before the raw file is saved... raw is only raw to a point. It also tells your computer raw processor what ISO was set through the EXIF data and your this is set by your processor as a first step when you open the file... you then go on from there.

None of the clever stuff would have been done in camera or on your computer To compensate for your under exposure because as far as it is aware the shot was taken at ISO 100..

 

[Terrywoodenpic]

I thought that the difference was because the ISO setting applies amplification to the analogue signal before it is converted to digital.
The digital conversion will lose data, think about all the discussions re: 12 bit and 8 bit, the analogue signal contains all the data much more than even raw, no losses due to sampling intervals, no losses due to rounding errors etc.

There is some amplification before the digital conversion to make the signal readable,(on a CMOS sensor each individual pixel sized sensor output is amplified by its own coupled amplifier on the chip.) but not to set ISO, this does not need further amplification, only digital manipulation of the data.

The data is by then just 1 and 0's so amplification would do nothing.

 

[EOS_JD]

If camera companies provided images that were clean from any noise - How many more cameras would they sell?

 

[ernesto]

If camera companies provided images that were clean from any noise - How many more cameras would they sell?

36,753 per year

 

[James J]

There are the odd occasions when underexposing is required, mainly for effect, BUT if noise is important then its far better to expose to the right and even over expose a smidgen but to underexpose and try to pull back in PP will just cause trouble.

Pretty much what I was going to say. This is a good case for watching the histogram and shifting the peek to the right as its definitely far left at the moment (without even checking it). Shadow noise can be present even at base ISO if the image isn't exposed properly, so recovering none existent detail (due to dynamic range being the limiting factor, amongst other things) in the shadows creates a shedload of noise.

 

[Suvv]

There is some amplification before the digital conversion to make the signal readable,(on a CMOS sensor each individual pixel sized sensor output is amplified by its own coupled amplifier on the chip.) but not to set ISO, this does not need further amplification, only digital manipulation of the data.

The data is by then just 1 and 0's so amplification would do nothing.

Everything reliable that I have read states that normally changes to the ISO are achieved by amplification of the analogue signal, however some higher ISO settings are achieved by manipulating the digital data.
If changing ISO was achieved by solely digital signal processing then it could be achieved easily in PP programs and you could change from 100 to 3200 or higher in PP with no problem, but exposure compensation is normally only OK for 2 stops.

 

[HoppyUK]

Canons Software would not have helped.
The in camera firmware does the processing before the raw file is saved... raw is only raw to a point. It also tells your computer raw processor what ISO was set through the EXIF data and your this is set by your processor as a first step when you open the file... you then go on from there.

None of the clever stuff would have been done in camera or on your computer To compensate for your under exposure because as far as it is aware the shot was taken at ISO 100..

Agreed. I thought that you were implying something else earlier. Apologies.

 

[Terrywoodenpic]

Everything reliable that I have read states that normally changes to the ISO are achieved by amplification of the analogue signal, however some higher ISO settings are achieved by manipulating the digital data.
If changing ISO was achieved by solely digital signal processing then it could be achieved easily in PP programs and you could change from 100 to 3200 or higher in PP with no problem, but exposure compensation is normally only OK for 2 stops.

There is a great deal of unreliable information repeated out there, even on "Good" sites.

Think about it... when You have amplified the analogue signal sufficiently to be converted in to digital, what would further amplification actually do? a digital number can simply be multiplied, or otherwise manipulated.