Histogram Myth explicated: “you’ll lose half your tonal values for each one full stop of under-exposure”

Histogram Myth explicated:
By “Mike” Michael L. Baird mike {at] mikebaird d o t com

19 Feb 2012: Join the related discussion at naturescapes.net (ref. this https://photomorrobay.wordpress.com/ )

Many photography instructors and experts in the field are innocently and incorrectly making inaccurate claims, such as those three below, about how to interpret a histogram as seen on your camera’s LCD.  The representation that the “tonal x-axis value” is somehow logarithmic in scale, as suggested by bringing “stops” or “amount of light collected” into the discussion, is the cause of confusion.  By the very definition of what the histogram is, we should know that it is just a linear scale.  We can clearly see that this is the case just by watching the histogram shift on the camera LCD while stopping down.  I’ve known otherwise good photographers to give up trying to understand things in this field because they can’t comprehend what’s being thrown at them when histograms are being discussed by the experts… all the experts couldn’t be wrong could they?  Well, in this case, yes.  Those of you previously driven to insanity can now come back to school.

Claim: A 12 bit image is capable of recording 4,096 (2^12) discrete tonal values. One would think that therefore each F/Stop of the 5 stop range would be able to record some 850 (4096 / 5) of these steps. But, alas, this is not the case. The way that it really works is that the first (brightest) stop’s worth of data contains 2048 of these steps — fully half of those available… This realization carries with it a number of important lessons, the most important of them being that if you do not use the right-hand fifth of the histogram for recording some of your image you are in fact wasting fully half of the available encoding levels of your camera. [Michael Reichmann] [repeated by Ron Day]
Wrong: losing half of your light by stopping down one stop, doesn’t mean you will lose half your tonal values.

Claim: “You’ll lose half your tonal values for each one full stop of under-exposure.”  Wrong, losing half of your light by stopping down one stop, doesn’t mean you will lose half your tonal values.

Claim: “Fully half of the tonal values are in the brightest fifth of a histogram.  So, if you don’t have at least some pixels heading out into that rightmost fifth of the histogram, you’re wasting half of the potential tonal information that your camera can capture!

Claim: “If we array the stops of dynamic range along our monochromatic or luminance histogram, we’ll notice that each stop (from left to right) contains two times more information than the previous stop.  And notice also that most of the possible color values are in the brightest areas. That is, our camera can capture only a relatively few dark tonal values and lots of bright tonal values.” [Greg Basco]

Illustration: An example of the myth being propagated.

Histogram Stop Myth Illustrated - this is not a correct interpretation.

Histogram Stop Myth Illustrated - this is not a correct interpretation

Illustration is courtesy of, and many thanks to, Costa Rica photographer Greg Basco who was great in helping me to finally understand this issue as he tolerated my barrage of challenges with grace and patience.

In a sense this whole argument is silly, because if you define a histogram to be a plot of brightness or “tonal” values ranging from 1 to N, and the horizontal axis is treated as and shown as linear, N/2 being in the center, etc., then saying that “there are more tonal values in one part of the histogram than another” is like saying there are more years in the last 1/5th of your life than the first 1/5th.

I’ve seen this myth repeated over and over for years throughout the literature.  This stop metaphor applied to the interpretation of histograms is misleading and inaccurate, and may teach photographers poor technique.  The resultant implied prescription to not unnecessarily under-expose, because you’ll lose up to half your tonal values in just a one stop change, is quantitatively false, and may encourage photographers to overexpose their images in avoiding loss of tonal information that is in fact not as at-risk as implied.

By stopping down one stop, you are indeed capturing only one-half the number of photons, but you are not losing one-half of your measured and perceptible tonal values.  There is no argument that we should almost never intentionally over-underexpose an image if we can keep all pixel values within the histogram and not blown out to the right (at least if the contrast range of the scene you are shooting is less than the dynamic range of the camera’s sensor, and you are sure you are not blowing out any of the brightest pixels… not so easily done or verifiable in practice) .  Collecting more light will increase the signal and information in the scene, and minimize noise (although at the trade-off of tolerating a longer shutter speed or wider aperture – but that’s getting us off the point at hand).

<<< below paragraph added for clarity 14 Feb 2012>>
Understanding the definition of Lightness is essential to understanding the histogram on your camera and as seen in Photoshop, etc., and thus accepting the fact that lightness is not doubled or halved when one aperture stop adjustment is made, doubling or halving the amount of light impinging on the sensor.  Wiki says “… scientists… converged on a roughly cube-root curve, consistent with the Stevens power law for brightness perception, reflecting the fact that lightness is proportional to the number of nerve impulses per nerve fiber per unit time…  at first glance, you might approximate the lightness function by a cube root, an approximation that is found in much of the technical literature. However, the linear segment near black is significant… an 18% grey card, having a reflectance of 0.18, has lightness very close to 50. It is called ‘mid grey’ because its lightness is midway between black and white.”

To learn more about the 18% grey card measure see http://en.wikipedia.org/wiki/Middle_gray “… middle gray or middle grey is a tone that is perceptually about half way between black and white on a lightness scale;[1] in photography, it is typically defined as 18% reflectance in visible light.[2]

The following chart can be thought of, in the first order of interpretation, as the related plot on the horizontal axis of the amount of light impinging on the sensor (reflectance), and the vertical axis representing the perceived lightness or brightness in the human visual perception system.  “Yn is the Y tristimulus value of a ‘specified white object.'”
You can pretty much see this curve in action if you just open a Canon CR2 RAW file in Canon’s DPP Digital Photo Professional – clicking the “linear” check box under the “RAW” tab to see what the sensor has recorded, and then un-checking “linear” and see how the transformation/relationship curve below changes the RAW reflectance values into perceived brightness or tonal values.

http://en.wikipedia.org/wiki/Lightness_%28color%29


Observe that the lightness is 50% for a luminance of around 18% relative to the reference white.

A histogram is, by definition, a plot of frequency-of-occurrence of perceived brightness [~Lightness] or tonal values of pixels, and you can see for yourself, say using live view (in a camera that has say just 5-stops of dynamic range) that as you stop down your camera, for each stop, the histogram shifts smoothly and linearly to the left by one-fifth of the horizontal-axis tonal value range.
That’s by definition of what the histogram does – it plots the frequency of occurrence of tonal values or perceived brightness of the pixels in an image.

 <<< below paragraph added for clarity 12 Jan 2012>> The key to understanding is to first define what the histogram is supposed to represent, including the terms used in the definition.  So.. if the definition of the histogram in this context is “the frequency of occurrence of tonal values plotted linearly” then any claim that the horizontal-axis is not linear would immediately seem silly on the surface, as it indeed is.  And you can really clearly see in Liveview that the histogram as implemented by camera manufacturers, just scoots along in a nice linear fashion, as the lens is stopped down.  The histogram isn’t an abstract concept or metaphor, it’s a simple data visualization tool.  Each pixel in an image has a tonal value as perceived by a human in say a grey scale chart test pattern with 256 gradations from left-to-right.  The values range from say 1 or black at the left, to say 256 or white at the right.   If the brightest pixels are 256, and the darkest are 1, then 128 represents the tonal value of the center pixels of the test pattern.  There are by definition 256 tonal values.  To state that fully half of these values are found in the upper 1/5th or 1/8th portion of the plot (depending on your dynamic range model) is silly.  So once again, “losing half of your light by stopping down one stop, doesn’t mean you will lose half your tonal values.”  <<< this  paragraph added for clarity 12 Jan 2012>>

The dynamic range of the camera is totally accommodated and compensated for by this point in time in the production of a linear brightness scale (~luminosity) seen in the histogram.  It is unimportant, for purposes of this argument, where and how this conversion takes place (the more usual place for this to occur is in RAW conversion).

It is easy to see how the “stops” myth started, because people did recognize that a lot more photons were being measured and recorded in the brighter regions of an image because of the non-linear response of the human visual perception system to reflected light.

The myth arises primarily because (1) people try to make photography more complicated than it is, and (2) it isn’t appreciated that human perceived brightness is a “logarithmic” function of the number of photons detected and recorded by a sensor.  Doubling the number of photons hitting pixels (by changing aperture by one stop) does not double the perceived brightness.  Doubling the intensity (photon density) of a point of light does not double our perception of it’s brightness (this is a biological phenomenon).  And similarly, by design, the conversion of voltages read out from a sensor chip eventually takes this relationship between reflectance in visible light and the perception of brightness or lightness, into consideration.  If shooting JPGs in-camera, or in the rendering of JPG preview files for RAW images, for LCD viewing on the camera, then RAW conversion is done for you, in-camera, using the reflectance/brightness relationship being discussed.  When shooting RAW files for your workflow, this conversion is usually done during RAW conversion in software on a computer, as part of your workflow.  (But, (and we are getting off track her) in theory, it would certainly be possible to do some of the transformation and processing in-camera (this would depend on the camera manufacturer’s implementation, especially if not using the only real RAW standard, DNG.  Most RAW and all DNG images would be expected to record more reflectance values in the brighter regions.  But remember, reflectance isn’t brightness!).

In the end, the histograms we see on the camera LCD and in Lightroom or Photoshop are just plots of the perceived brightness vs. frequency-of-occurrence of pixels.
Because photon-density vs. brightness is logarithmic and not linear in human perception, doubling the amount of light collected (or doubling the number of photons counted or equivalently the signal voltage created) does not double the perceived brightness (tonal level) quantized by the camera’s electronics and written into RAW or jpg values.   By definition, our camera and Photoshop histograms display the distribution of pixels by perceived brightness.

Laboratory assignment.  Prove this yourself.

To show this empirically, I downloaded a 10-tonal-level grey scale chart from the internet,  which has ten vertical intensity bands, then opened it in Photoshop, viewed Histogram, and selected regions over each pattern.
The histogram looks like a flat or evenly pulsed line, since the grey-levels are distributed evenly and the frequency of occurrence of pixels in each of these tonal value zones is equal in number to any other because of the design of the test pattern.
Photoshop quantizes the histogram view into 256 columns/containers for display purposes.
The RBG tonal values will be linearly different from sample pattern column to sample pattern column by about 28.4 on this projection to 256 plotting points (256/9), or 11% (100/9)
I measured (in 8-bits/channel RGB mode, these mean values)
0, 28, 56, 85, 113, 141, 170, 198, 226, 255
and converted to Grey-levels and measured in these percentages:
0, 15, 28, 41, 53, 64, 75, 84, 92, 100

Another example using a 21-tonal level test pattern
yields the same, as similarly expected, these linear values…
0, 13, 26, 38, 51, 64, 77, 89, 102, 115, 128, 140, 153, 166, 178, 191, 204, 217, 229, 242, 255  (~13 difference per step or 256/20=12.8)

So again, the top 1/5th values, in these cases,
[226, 255] and
[204, 217, 229, 242, 255]
range across 20% of the tonal range, not the 50% as claimed.

QED

15 Feb 2012 here are some notes I’ve included in private responses that may be useful to preserve.

…the number of buckets or zones used and the crossing of those zones is also a bit of a red herring in that it detracts from the basic problem that people want to interpret the horizontal axis, regardless of the number of buckets used, as something other than what it is.
Always reason by extremes to find the truth.  Imagine an image with quantization to 256 brightness levels (each pixel value is a number between 1 and 256) and plot those in a histogram with 256 buckets.  Then you see the histogram for what it is.
I’m convinced the only way people will understand the histogram is to make up a small say 5×5 image and write brightness numbers in by hand and then make the plot themselves.
Here is another exercise.
Ask if you took a photo of a random pattern, what should the histogram look like?
A: a straight line.  If “all the tonal values were in the upper brightness levels” as mis-stated it would rise to the right.
Take a photo of a test pattern going from black to white left to right.
The histogram is a straight line.
Cut the strip in half.
The histogram simply stops half way over.
If you want to make the argument that losing that last stop of information over the first stop is more important because the number of photons being sacrificed is so much more, then you had better find a species whose neurons respond linearly to light stimuli.
You won’t find in Wikipedia the myth propagated, because it is a myth.
Go through some thought exercises. If I shine a flashlight on a piece of paper and call that brightness X, will adding a second flashlight (doubling the amount of light) would that double the perceived brightness to 2X?  No. Why not?  Read my blog above again.
And finally just ask why would anyone believe the “overlay” interpretation of a histogram when up to that point histograms made perfect sense.  My conclusion is that those who add the myth layer never understood the basic histogram to start… And that includes a lot of heroes in our hobby/business.  They can send a salesman to the moon but a salesman can’t design the rocket to get there.
************

I basically agree with   on exposing to the right hereClick Here: Why Shooting to the Right Gives You Better Final Images “

and he does not make the myth mistake https://photomorrobay.wordpress.com/ of stating that losing one stop loses you half your tonal values.

Where he is wrong, or at least he is chasing a red-herring, is in stating that the sensor is most efficient in the bright areas.
more efficient at capturing the light at the brighter end of the exposure and less efficient at capturing the darker end”

It’s an irrelevant statement in any case, as we are not talking about efficiency, but in exposing to the right we are collecting more photons on ever pixel, and thus have a stronger signal and less noise, at the possible expense of tolerating a longer shutter speed or opening the aperture and losing depth-of-field.
********************
19 Feb 2012.  Thanks to Pat Brown in private communications for pointing out via an experiment she recently conducted that RAW files do slightly increase in size with increased length of exposure, but not enough to suggest that tonal values are halved with one stop changes.. Here is her report:
I think an experiment I did proves the linear nature of the histogram. I photographed a gray stucco wall with these constants.ISO 100
11
Focal length 50 mm (which would be 80 mm full frame)
Distance from wall 1.1 m (3’10”)
TripodI varied the exposure in 1 stop increments for a total of 8 exposures. Here are the shutter speeds and the sizes of the files for each exposure. These exposures were chosen to to produce data on my histogram from the far left to the far right.1/500 sec    18.55 MB
1/250 sec    19.18 MB
1/125 sec    20.18 MB
1/60  sec     21.85 MB
1/30  sec     24.58 MB
1/15  sec     27.17 MB
1/8    sec     30.51 MB
1/4    sec     33.72 MBSo the size of the files I’m assuming is more or less a linear progression (I have not plotted it). If the myth were true, based on a far right value of 33.72 MB, the file size on the far left at 1/500 sec should be 2.11 MB, not 18.55 MB. This is probably easier to understand if you can see the histograms. I hope you are able to see the two PowerPoint slides that I have attached. I’ll send it as an online Office document, which Hotmail says everyone is supposed to be able to view. Let me know if you can’t see it.  Pat
Pat has a file to share with you on SkyDrive. To view it, click the link below.
Histogram Experiment.ppt
I conclude that the slight change in file size could easily be due not only to the fact that a wider range of data is being recorded off the sensor (since an under-exposured image would not have any representative pixels in the upper ranges), and thus the space required to store that data would be larger, but further that more raw data could well be quantized and recorded in the brighter rather than darker regions of the image, as RAW captures sensor “reflectance” data that only during RAW conversion is scaled into a “lightness” scale.
1/500 sec    18.55 MB
1/250 sec    19.18 MB
1/125 sec    20.18 MB
1/60  sec     21.85 MB
1/30  sec     24.58 MB
1/15  sec     27.17 MB
1/8    sec     30.51 MB
1/4    sec     33.72 MB
********************
The online discussion is yielding some useful observations:
19 Feb 2012: Join the related discussion at naturescapes.net (ref. this https://photomorrobay.wordpress.com/ )
As Ari Hazeghi ahazeghi on Mon Feb 20, 2012 1:16 pm said at http://www.naturescapes.net/phpBB3/viewtopic.php?f=2&t=212875

If you have Canon DPP Digital Photo Professional you can see what the linear sensor output looks like (for CR2 files at least).  This is an exercise well worth doing.
“… open a CR2 file and then click linear on the RAW tab. This is what the sensor has recorded. Now un-check linear and observe how the tone curve changes. You can also click on the RGB tab to see the RGB histogram… I wouldn’t say ‘brightest fifth of the histogram holds 50% of all tonal information…’ that’s not true because as I mentioned histogram is a result of the tone curve applied to RAW data and it is variable. The real RAW histogram is what is shown in the RAW tab in DPP and it is quite different from the RGB histogram you see in PS or LR… And yes, the ‘number of levels’ argument is just nonsense, the ADC in the camera does not care about exposure. The RAW is always 14Bits.”
********************
Personal note on why the heck should you listen to Mike Baird?
In the past I did hard core image processing science (I have a PhD in Computer Science).
http://bit.ly/MyGoogleScholarCitations lists some of my referred papers and patents in the field of computer vision, imaging, and robotics.
We had to define our terms and support our conclusions with valid sources and everything we published went through a gruesome a referee process.

Resume Bio
“Mike” Michael L. Baird is an enthusiast photographer doing nature, wildlife, people, and surf photography along the California Central Coast. He is a CA State Park volunteer docent, and retired Silicon Valley Internet entrepreneur, now living in Morro Bay, CA, where he encourages collaboration via http://photomorrobay.com http://bairdphotos.com http://flickr.bairdphotos.com and http:// morro-bay.com
Mike has an MBA, and a PhD in Computer Science, and was the first VP of Engineering of the $7 billion ask.com search engine company.
He is author of the 20-year best-selling entrepreneurship book entitled “Engineering Your Start-up: A Guide for the High-Tech Entrepreneur” http:// eysu.org.
Contact Mike at mike [at} mikebaird d o t com. (805) 704-2064.

http://bit.ly/MyGoogleScholarCitations lists some of my image-processing related patents and referred papers.

285×285 heqdshot at http://www.flickr.com/photos/mikebaird/6556151331/

**********
Advertisements

14 thoughts on “Histogram Myth explicated: “you’ll lose half your tonal values for each one full stop of under-exposure”

  1. Mike, are you speaking about “exposing right” when generating RAW images, so that you maximize the dynamic range (particularly in shadows) of your camera? I’ve never heard it expressed the way you say here, but I do “expose right” and find it really effective at minimizing noise and preserving detail.

    Here’s an article that explain what I believe to be true. Can you shoot any holes in the position?
    http://www.luminous-landscape.com/tutorials/optimizing_exposure.shtml

    • Yes, sorry to say, http://www.luminous-landscape.com/tutorials/optimizing_exposure.shtml to http://www.luminous-landscape.com/tutorials/expose-right.shtml (relevant section pasted below***) is one of those myth originators without references. Looked at as a system, no one should be trying to second guess what transformations are going on in terms of the linearity or logarithmic response of silicon sensors to photons, nor the transformation curves of A/D converters utilized and the algorithms underlying them. Those are internal implementation considerations that can confuse one studying the system’s output. The values written into JPG files by the system are merely integers representing brightness (at best a logarithmic relationship to quantity of photons). (And relying on the mystery of RAW takes you down a false branch for sure. Most RAW data formats are not disclosed; in any case, before RAW can be rendered in any visible form, whether it’s done in camera or later using software in a computer, the integers produced are no different – they just have more bit depth and/or the user gets to decide things the camera otherwise decides in compressing the image and making JPGs, like setting the white point temperature, max and min intensity values to collapse, sharpening, saturation level, etc… all the things you can adjust in Adobe Camera RAW).
      If these integers run from 1-256, there are 256 tonal values, each being one equal amount brighter than the previous on a linear scale. That is to say 256 is twice the brightness of 128. 128 is twice the brightness of 64, etc. Halving the amount of light or photons exposed to the sensor, by stopping down one stop, does not then make values 129-256 have no members as suggested repeatedly in the above myth.

      *** “This analogue voltage is then converted into digital format within the camera by an IC called an Analogue to Digital converter. … Data in the raw file is linear. There has been no tone curve applied. This is done later in the raw processor.”
      “… CCD and CMOS chips are linear devices. And, of course, each F/Stop records half of the light of the previous one, and therefore half the remaining data space available. This little table tells the tale.Within the first F/Stop, which contains the Brightest Tones
      2048 levels available Within the second F/Stop, which contains Bright Tones
      1024 levels available Within the third F/Stop, which contains the Mid-Tones
      512 levels available Within the fourth F/Stop, which contains Dark Tones
      256 levels available Within the fifth F/Stop, which contains the Darkest Tones
      128 levels available Within the fifth F/Stop, which contains the Darkest Tones”
      The above jut isn’t true.

      And, of COURSE we should all expose as far to the right as possible without blowing out pixels, in order to maximize the amount of information captured.
      There is no argument on this.

      • The Luminous Landscape take on ETTR has been around for awhile and as I understand it came about from a conversation that Michael had with Thomas Knoll, one of the creators of Photoshop.

  2. A couple of things that I know from personal experience about ETTR (expose to the right) is that it tends to avoid under-exposure, which reduces the tendency to add noise at higher ISOs when I try to raise the brightness level in RAW conversion. If I’ve slightly over exposed by ETTR and then normalize the brightness level in RAW converion by bringing down the level, no noise seems to be added.

    So, if I haven’t blown out any highlights or colors, then there’s no harm in ETTR for RAW shooting, is there? I like the “insurance” of having an extra stop or so in the shadow areas vs. the risk of under exposing those areas a stop or so and seeing lots of noise when I try to pull those areas up for RAW conversion.

    Isn’t the reason that exposure in dark areas is critical is that the signal-to-noise ratio is so low for the darker portions of digital images? If so, and I may be wrong, then it makes sense to me to give yourself a cushion whenever possible. I think you agree with me on this point in your last paragraph of your response above.

    • I totally agree ETTR
      is good practice.
      My only beef is the statement that is not true

      “You’ll lose half your tonal values for each one full stop of under-exposure.”

      • I wonder if that’s more of a regional way to describe ETTR. I never heard explained that way until I saw your post.

        Anyway, thanks for the clarification.

  3. Thank you, I’ve just been searching for info about this topic for a while and yours is the greatest I have discovered so far. However, what in regards to the conclusion? Are you certain concerning the supply?

  4. Pingback: Expose to the Right (ETTR) - Straight Up Outdoor Photography

  5. Hi, nice work, my only request is that the latter half could be better facilitated with some diagrams. Sadly most of your diagrams are to demonstrate what you are arguing against, not what you are arguing for which may end up leaving the wrong message in the mind of the reader.

    I wonder where you stand on the an old SuperCCD by Fuji which in my dissections has a range up to register values of 0011 1101 1111 1101 which gives about 2 stops of headroom as I make it out.

    • Pellicle, thanks for the suggestions. I probably could have best made my write up about 1/10th its length, because the whole argument is based on a general misunderstanding of what “it is.” Regarding specific hardware or bits/level of digitization, I consider that a red herring that would take one further off track. It is hard to argue, analogously, that there are more years in the last 1/5th of your life than the first 1/5th. QED

      • Hi

        I have done tests with the SuperCCD (my Aunt is a wedding photographer and it tore her away from negatives) and there is definately more headroom. Bottom end remains the same but it has WAY more bits (because it has differently tuned receptors for that).

        Anyway … great post and best wishes

  6. Pingback: Essentials of Digital Photography « M&M's Musings

  7. Pingback: Working with light | Expressing Your Vision

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s