Histogram Myth explicated: “you’ll lose half your tonal values for each one full stop of under-exposure”

Histogram Myth explicated:
By “Mike” Michael L. Baird mike {at] mikebaird d o t com

19 Feb 2012: Join the related discussion at naturescapes.net (ref. this https://photomorrobay.wordpress.com/ )

Many photography instructors and experts in the field are innocently and incorrectly making inaccurate claims, such as those three below, about how to interpret a histogram as seen on your camera’s LCD.  The representation that the “tonal x-axis value” is somehow logarithmic in scale, as suggested by bringing “stops” or “amount of light collected” into the discussion, is the cause of confusion.  By the very definition of what the histogram is, we should know that it is just a linear scale.  We can clearly see that this is the case just by watching the histogram shift on the camera LCD while stopping down.  I’ve known otherwise good photographers to give up trying to understand things in this field because they can’t comprehend what’s being thrown at them when histograms are being discussed by the experts… all the experts couldn’t be wrong could they?  Well, in this case, yes.  Those of you previously driven to insanity can now come back to school.

Claim: A 12 bit image is capable of recording 4,096 (2^12) discrete tonal values. One would think that therefore each F/Stop of the 5 stop range would be able to record some 850 (4096 / 5) of these steps. But, alas, this is not the case. The way that it really works is that the first (brightest) stop’s worth of data contains 2048 of these steps — fully half of those available… This realization carries with it a number of important lessons, the most important of them being that if you do not use the right-hand fifth of the histogram for recording some of your image you are in fact wasting fully half of the available encoding levels of your camera. [Michael Reichmann] [repeated by Ron Day]
Wrong: losing half of your light by stopping down one stop, doesn’t mean you will lose half your tonal values.

Claim: “You’ll lose half your tonal values for each one full stop of under-exposure.”  Wrong, losing half of your light by stopping down one stop, doesn’t mean you will lose half your tonal values.

Claim: “Fully half of the tonal values are in the brightest fifth of a histogram.  So, if you don’t have at least some pixels heading out into that rightmost fifth of the histogram, you’re wasting half of the potential tonal information that your camera can capture!

Claim: “If we array the stops of dynamic range along our monochromatic or luminance histogram, we’ll notice that each stop (from left to right) contains two times more information than the previous stop.  And notice also that most of the possible color values are in the brightest areas. That is, our camera can capture only a relatively few dark tonal values and lots of bright tonal values.” [Greg Basco]

Illustration: An example of the myth being propagated.

Histogram Stop Myth Illustrated - this is not a correct interpretation.

Histogram Stop Myth Illustrated - this is not a correct interpretation

Illustration is courtesy of, and many thanks to, Costa Rica photographer Greg Basco who was great in helping me to finally understand this issue as he tolerated my barrage of challenges with grace and patience.

In a sense this whole argument is silly, because if you define a histogram to be a plot of brightness or “tonal” values ranging from 1 to N, and the horizontal axis is treated as and shown as linear, N/2 being in the center, etc., then saying that “there are more tonal values in one part of the histogram than another” is like saying there are more years in the last 1/5th of your life than the first 1/5th.

I’ve seen this myth repeated over and over for years throughout the literature.  This stop metaphor applied to the interpretation of histograms is misleading and inaccurate, and may teach photographers poor technique.  The resultant implied prescription to not unnecessarily under-expose, because you’ll lose up to half your tonal values in just a one stop change, is quantitatively false, and may encourage photographers to overexpose their images in avoiding loss of tonal information that is in fact not as at-risk as implied.

By stopping down one stop, you are indeed capturing only one-half the number of photons, but you are not losing one-half of your measured and perceptible tonal values.  There is no argument that we should almost never intentionally over-underexpose an image if we can keep all pixel values within the histogram and not blown out to the right (at least if the contrast range of the scene you are shooting is less than the dynamic range of the camera’s sensor, and you are sure you are not blowing out any of the brightest pixels… not so easily done or verifiable in practice) .  Collecting more light will increase the signal and information in the scene, and minimize noise (although at the trade-off of tolerating a longer shutter speed or wider aperture – but that’s getting us off the point at hand).

<<< below paragraph added for clarity 14 Feb 2012>>
Understanding the definition of Lightness is essential to understanding the histogram on your camera and as seen in Photoshop, etc., and thus accepting the fact that lightness is not doubled or halved when one aperture stop adjustment is made, doubling or halving the amount of light impinging on the sensor.  Wiki says “… scientists… converged on a roughly cube-root curve, consistent with the Stevens power law for brightness perception, reflecting the fact that lightness is proportional to the number of nerve impulses per nerve fiber per unit time…  at first glance, you might approximate the lightness function by a cube root, an approximation that is found in much of the technical literature. However, the linear segment near black is significant… an 18% grey card, having a reflectance of 0.18, has lightness very close to 50. It is called ‘mid grey’ because its lightness is midway between black and white.”

To learn more about the 18% grey card measure see http://en.wikipedia.org/wiki/Middle_gray “… middle gray or middle grey is a tone that is perceptually about half way between black and white on a lightness scale;[1] in photography, it is typically defined as 18% reflectance in visible light.[2]

The following chart can be thought of, in the first order of interpretation, as the related plot on the horizontal axis of the amount of light impinging on the sensor (reflectance), and the vertical axis representing the perceived lightness or brightness in the human visual perception system.  “Yn is the Y tristimulus value of a ‘specified white object.'”
You can pretty much see this curve in action if you just open a Canon CR2 RAW file in Canon’s DPP Digital Photo Professional – clicking the “linear” check box under the “RAW” tab to see what the sensor has recorded, and then un-checking “linear” and see how the transformation/relationship curve below changes the RAW reflectance values into perceived brightness or tonal values.


Observe that the lightness is 50% for a luminance of around 18% relative to the reference white.

A histogram is, by definition, a plot of frequency-of-occurrence of perceived brightness [~Lightness] or tonal values of pixels, and you can see for yourself, say using live view (in a camera that has say just 5-stops of dynamic range) that as you stop down your camera, for each stop, the histogram shifts smoothly and linearly to the left by one-fifth of the horizontal-axis tonal value range.
That’s by definition of what the histogram does – it plots the frequency of occurrence of tonal values or perceived brightness of the pixels in an image.

 <<< below paragraph added for clarity 12 Jan 2012>> The key to understanding is to first define what the histogram is supposed to represent, including the terms used in the definition.  So.. if the definition of the histogram in this context is “the frequency of occurrence of tonal values plotted linearly” then any claim that the horizontal-axis is not linear would immediately seem silly on the surface, as it indeed is.  And you can really clearly see in Liveview that the histogram as implemented by camera manufacturers, just scoots along in a nice linear fashion, as the lens is stopped down.  The histogram isn’t an abstract concept or metaphor, it’s a simple data visualization tool.  Each pixel in an image has a tonal value as perceived by a human in say a grey scale chart test pattern with 256 gradations from left-to-right.  The values range from say 1 or black at the left, to say 256 or white at the right.   If the brightest pixels are 256, and the darkest are 1, then 128 represents the tonal value of the center pixels of the test pattern.  There are by definition 256 tonal values.  To state that fully half of these values are found in the upper 1/5th or 1/8th portion of the plot (depending on your dynamic range model) is silly.  So once again, “losing half of your light by stopping down one stop, doesn’t mean you will lose half your tonal values.”  <<< this  paragraph added for clarity 12 Jan 2012>>

The dynamic range of the camera is totally accommodated and compensated for by this point in time in the production of a linear brightness scale (~luminosity) seen in the histogram.  It is unimportant, for purposes of this argument, where and how this conversion takes place (the more usual place for this to occur is in RAW conversion).

It is easy to see how the “stops” myth started, because people did recognize that a lot more photons were being measured and recorded in the brighter regions of an image because of the non-linear response of the human visual perception system to reflected light.

The myth arises primarily because (1) people try to make photography more complicated than it is, and (2) it isn’t appreciated that human perceived brightness is a “logarithmic” function of the number of photons detected and recorded by a sensor.  Doubling the number of photons hitting pixels (by changing aperture by one stop) does not double the perceived brightness.  Doubling the intensity (photon density) of a point of light does not double our perception of it’s brightness (this is a biological phenomenon).  And similarly, by design, the conversion of voltages read out from a sensor chip eventually takes this relationship between reflectance in visible light and the perception of brightness or lightness, into consideration.  If shooting JPGs in-camera, or in the rendering of JPG preview files for RAW images, for LCD viewing on the camera, then RAW conversion is done for you, in-camera, using the reflectance/brightness relationship being discussed.  When shooting RAW files for your workflow, this conversion is usually done during RAW conversion in software on a computer, as part of your workflow.  (But, (and we are getting off track her) in theory, it would certainly be possible to do some of the transformation and processing in-camera (this would depend on the camera manufacturer’s implementation, especially if not using the only real RAW standard, DNG.  Most RAW and all DNG images would be expected to record more reflectance values in the brighter regions.  But remember, reflectance isn’t brightness!).

In the end, the histograms we see on the camera LCD and in Lightroom or Photoshop are just plots of the perceived brightness vs. frequency-of-occurrence of pixels.
Because photon-density vs. brightness is logarithmic and not linear in human perception, doubling the amount of light collected (or doubling the number of photons counted or equivalently the signal voltage created) does not double the perceived brightness (tonal level) quantized by the camera’s electronics and written into RAW or jpg values.   By definition, our camera and Photoshop histograms display the distribution of pixels by perceived brightness.

Laboratory assignment.  Prove this yourself.

To show this empirically, I downloaded a 10-tonal-level grey scale chart from the internet,  which has ten vertical intensity bands, then opened it in Photoshop, viewed Histogram, and selected regions over each pattern.
The histogram looks like a flat or evenly pulsed line, since the grey-levels are distributed evenly and the frequency of occurrence of pixels in each of these tonal value zones is equal in number to any other because of the design of the test pattern.
Photoshop quantizes the histogram view into 256 columns/containers for display purposes.
The RBG tonal values will be linearly different from sample pattern column to sample pattern column by about 28.4 on this projection to 256 plotting points (256/9), or 11% (100/9)
I measured (in 8-bits/channel RGB mode, these mean values)
0, 28, 56, 85, 113, 141, 170, 198, 226, 255
and converted to Grey-levels and measured in these percentages:
0, 15, 28, 41, 53, 64, 75, 84, 92, 100

Another example using a 21-tonal level test pattern
yields the same, as similarly expected, these linear values…
0, 13, 26, 38, 51, 64, 77, 89, 102, 115, 128, 140, 153, 166, 178, 191, 204, 217, 229, 242, 255  (~13 difference per step or 256/20=12.8)

So again, the top 1/5th values, in these cases,
[226, 255] and
[204, 217, 229, 242, 255]
range across 20% of the tonal range, not the 50% as claimed.


15 Feb 2012 here are some notes I’ve included in private responses that may be useful to preserve.

…the number of buckets or zones used and the crossing of those zones is also a bit of a red herring in that it detracts from the basic problem that people want to interpret the horizontal axis, regardless of the number of buckets used, as something other than what it is.
Always reason by extremes to find the truth.  Imagine an image with quantization to 256 brightness levels (each pixel value is a number between 1 and 256) and plot those in a histogram with 256 buckets.  Then you see the histogram for what it is.
I’m convinced the only way people will understand the histogram is to make up a small say 5×5 image and write brightness numbers in by hand and then make the plot themselves.
Here is another exercise.
Ask if you took a photo of a random pattern, what should the histogram look like?
A: a straight line.  If “all the tonal values were in the upper brightness levels” as mis-stated it would rise to the right.
Take a photo of a test pattern going from black to white left to right.
The histogram is a straight line.
Cut the strip in half.
The histogram simply stops half way over.
If you want to make the argument that losing that last stop of information over the first stop is more important because the number of photons being sacrificed is so much more, then you had better find a species whose neurons respond linearly to light stimuli.
You won’t find in Wikipedia the myth propagated, because it is a myth.
Go through some thought exercises. If I shine a flashlight on a piece of paper and call that brightness X, will adding a second flashlight (doubling the amount of light) would that double the perceived brightness to 2X?  No. Why not?  Read my blog above again.
And finally just ask why would anyone believe the “overlay” interpretation of a histogram when up to that point histograms made perfect sense.  My conclusion is that those who add the myth layer never understood the basic histogram to start… And that includes a lot of heroes in our hobby/business.  They can send a salesman to the moon but a salesman can’t design the rocket to get there.

I basically agree with   on exposing to the right hereClick Here: Why Shooting to the Right Gives You Better Final Images “

and he does not make the myth mistake https://photomorrobay.wordpress.com/ of stating that losing one stop loses you half your tonal values.

Where he is wrong, or at least he is chasing a red-herring, is in stating that the sensor is most efficient in the bright areas.
more efficient at capturing the light at the brighter end of the exposure and less efficient at capturing the darker end”

It’s an irrelevant statement in any case, as we are not talking about efficiency, but in exposing to the right we are collecting more photons on ever pixel, and thus have a stronger signal and less noise, at the possible expense of tolerating a longer shutter speed or opening the aperture and losing depth-of-field.
19 Feb 2012.  Thanks to Pat Brown in private communications for pointing out via an experiment she recently conducted that RAW files do slightly increase in size with increased length of exposure, but not enough to suggest that tonal values are halved with one stop changes.. Here is her report:
I think an experiment I did proves the linear nature of the histogram. I photographed a gray stucco wall with these constants.ISO 100
Focal length 50 mm (which would be 80 mm full frame)
Distance from wall 1.1 m (3’10”)
TripodI varied the exposure in 1 stop increments for a total of 8 exposures. Here are the shutter speeds and the sizes of the files for each exposure. These exposures were chosen to to produce data on my histogram from the far left to the far right.1/500 sec    18.55 MB
1/250 sec    19.18 MB
1/125 sec    20.18 MB
1/60  sec     21.85 MB
1/30  sec     24.58 MB
1/15  sec     27.17 MB
1/8    sec     30.51 MB
1/4    sec     33.72 MBSo the size of the files I’m assuming is more or less a linear progression (I have not plotted it). If the myth were true, based on a far right value of 33.72 MB, the file size on the far left at 1/500 sec should be 2.11 MB, not 18.55 MB. This is probably easier to understand if you can see the histograms. I hope you are able to see the two PowerPoint slides that I have attached. I’ll send it as an online Office document, which Hotmail says everyone is supposed to be able to view. Let me know if you can’t see it.  Pat
Pat has a file to share with you on SkyDrive. To view it, click the link below.
Histogram Experiment.ppt
I conclude that the slight change in file size could easily be due not only to the fact that a wider range of data is being recorded off the sensor (since an under-exposured image would not have any representative pixels in the upper ranges), and thus the space required to store that data would be larger, but further that more raw data could well be quantized and recorded in the brighter rather than darker regions of the image, as RAW captures sensor “reflectance” data that only during RAW conversion is scaled into a “lightness” scale.
1/500 sec    18.55 MB
1/250 sec    19.18 MB
1/125 sec    20.18 MB
1/60  sec     21.85 MB
1/30  sec     24.58 MB
1/15  sec     27.17 MB
1/8    sec     30.51 MB
1/4    sec     33.72 MB
The online discussion is yielding some useful observations:
19 Feb 2012: Join the related discussion at naturescapes.net (ref. this https://photomorrobay.wordpress.com/ )
As Ari Hazeghi ahazeghi on Mon Feb 20, 2012 1:16 pm said at http://www.naturescapes.net/phpBB3/viewtopic.php?f=2&t=212875

If you have Canon DPP Digital Photo Professional you can see what the linear sensor output looks like (for CR2 files at least).  This is an exercise well worth doing.
“… open a CR2 file and then click linear on the RAW tab. This is what the sensor has recorded. Now un-check linear and observe how the tone curve changes. You can also click on the RGB tab to see the RGB histogram… I wouldn’t say ‘brightest fifth of the histogram holds 50% of all tonal information…’ that’s not true because as I mentioned histogram is a result of the tone curve applied to RAW data and it is variable. The real RAW histogram is what is shown in the RAW tab in DPP and it is quite different from the RGB histogram you see in PS or LR… And yes, the ‘number of levels’ argument is just nonsense, the ADC in the camera does not care about exposure. The RAW is always 14Bits.”
Personal note on why the heck should you listen to Mike Baird?
In the past I did hard core image processing science (I have a PhD in Computer Science).
http://bit.ly/MyGoogleScholarCitations lists some of my referred papers and patents in the field of computer vision, imaging, and robotics.
We had to define our terms and support our conclusions with valid sources and everything we published went through a gruesome a referee process.

Resume Bio
“Mike” Michael L. Baird is an enthusiast photographer doing nature, wildlife, people, and surf photography along the California Central Coast. He is a CA State Park volunteer docent, and retired Silicon Valley Internet entrepreneur, now living in Morro Bay, CA, where he encourages collaboration via http://photomorrobay.com http://bairdphotos.com http://flickr.bairdphotos.com and http:// morro-bay.com
Mike has an MBA, and a PhD in Computer Science, and was the first VP of Engineering of the $7 billion ask.com search engine company.
He is author of the 20-year best-selling entrepreneurship book entitled “Engineering Your Start-up: A Guide for the High-Tech Entrepreneur” http:// eysu.org.
Contact Mike at mike [at} mikebaird d o t com. (805) 704-2064.

http://bit.ly/MyGoogleScholarCitations lists some of my image-processing related patents and referred papers.

285×285 heqdshot at http://www.flickr.com/photos/mikebaird/6556151331/


Photomorrobay tidbits by “Mike” Michael L. Baird

Photomorrobay tidbits by “Mike” Michael L. Baird mike [at} mikebaird d o t com

This blog is a container for various thoughts, hints, suggestions, discussions, etc. regarding photography subjects of interest to the photomorrobay.com community including the photomorrobay Yahoo! Group and photomorrobay Flickr Group members.