Raw is a Rip Off!! Why Do DSLRs have Low Dynamic Range Raw? Log Gamma, CR2

It makes ZERO sense to me that the raw data from the sensor would not be stored as the log of the actual value.   That way the full spectrum from brightest to darkest would be recorded with an even number of bits instead of half the bits being used just for the brightest "stop" on a 10 or 11 stop range.

Well... guess what?   Arri and Red, Sony and even Canon use log encoding for their Cinema Raw formats.
"UPDATE: Correction/Clarification. OK, there is room for more confusion as I have been reminded that ArriRaw uses Log encoding as does RedCode. It is also likely that Sony’s raw uses data reduction for the higher stops (as Sony’s raw is ACES compliant it possibly uses data rounding for the higher stops). ArriRaw uses log encoding for the raw data to minimise data wastage, but the data is still unencoded data, it has not been encoded into RGB or YCbCr and it does not have a white balance or have gain applied, all of this is added in post. Sony’s S-Log, S-Log2, Arri’s LogC,  Canon’s C-Log as well as Cineon are all encoded and processed RGB or YCbCr video with a set white balance and with a Log gamma curve applied."

How about Canon?  Their current still EOS raw format is CR2.

 I found one old paper on digital photography - back before it was common.   It was discussing this whole brightness scale problem and basically said it was impractical to record the data linearly because you would need 12 -14 bits to adequately describe a decent grayscale pixel.
At that time, a 1 MB image was pushing things.  So you need to use a log gamma curve.
(gamma is a function - or curve - or transform - depending on your jargon preference, that takes the raw info on pixel brightness as stored in the image file and converts it to the brightness that ought to be visible in the output - like a monitor.   Best to Google or look on Wikipedia for a fuller definition). 

Well, now Canon uses 14 bits in their new cameras! (formerly 12).   So we have fulfilled the destiny of dumb encoding that even an ancient digital imaging paper saw as impractical.   Just another example of huge tech advances allowing implementations to be inefficient and inferior.

Log gamma:
Using Log gamma means that if the darkest black is 1 and your whitest white is 10 stops higher - so 1024 - you could record the black as 0 (2 to the power zero is 1) and 1024 could be recorded simply as 10 (2 to the power 10).   So now your ten stop scale is written as 1 - 10 instead of 1-1024.  But more importantly, instead of the brightest of brights - the 10th stop - taking up a full half of that range (513-1024), the different tones are evenly allocated.   The brightest get 9-10 and the darkest 0-1      If you use the same number of bits, you're going to record a lot more detailed tone info at the dark shadowy end. 

Side Note: NOT Really Raw.
One thing for sure, the Canon CR2 format is not "raw" in the sense of just being able to read a sensor value off the file.  It's compressed in "lossless JPEG" format (whatever that is).
I think it's  kind of TIFF like format.   Not really relevant to my point, but interesting....

Canon Knows Log Gamma!
The "Cinema" C300 (pictured above - kind of a DSLR in a video camera body) has a Log gamma compressed format - where the data was on a log scale and then compressed.    That claims a 12 stop range - vs the max of 11 typically claimed for full frame I think (10 on aps-c like mine.  Canon 7d dynamic range). 


Apparently, pro VIDEO cameras have had a log compressed format for a few years (Sony, Arri, Red, and Canon too).   The claim was that you got the same processing flexibility in post (or better?) than you get with Raw with much smaller files.  "Raw" is a linear encoding of the brightness striking a pixel.  Way more bits used to describe the bright part of the spectrum than darks.   One of the things that actually distinguished RED 1 in 2006 was its 12 bit log raw format (that required acceleration to edit directly).

The Video guys seem way more on top of this stuff and more sophisticated about it than the photo guys.

Canon (and other) still cameras (except maybe Red Cameras used for still work?) use linear gamma curves - so they record far less dark data.

I think this article from Canon Cinema itself makes it about as clear as I'm going to find:

It explains that Raw still images are very big - 25MB+ (and 125MB once opened in Photoshop), and just aren't very practical for Cinema.

Red's first codec was the big innovator.
Now all the majors have "compressed" raw video formats or close to it using log gamma curves.

Postscript:   I looked up this stuff because I read about a technique for DSLR Raw shooting called "expose to the right" or ETTR in the blogs.   It advocates increasing exposure just to the point where you start to clip whites.  That's often higher exposure than the typical 12-18% gray that is the standard for exposure meter proper exposure.   The idea is that it reduces the portion of the available Raw encoding range dedicated to the brights by pushing it as high as possible.  That leaves more of the range for midtones and darks.    You would then bring the exposure down to something that looks good when processing the raw image.   This seems crazy to me, but if raw still formats are using linear gamma, then maybe it does make sense.

DSLR Exposure: ETTR - "Expose to the Right"

EXPOSURE:
I read an article yesterday advocating what the author called "Expose to the Right" exposure settings for digital cameras.
(By the way, almost everyone who tries to justify this technique points to that article or basically quotes it.   At least from my Googling).

Normally and exposure meter recommends settings that average the scene to about 12-18% reflective gray.
That's supposed to be based on typical scenes or whatever.  It's ancient lore.   There are corrections for bright backgrounds and whatnot and cameras are pretty good about it these days.

For shooting on Film the conventional wisdom was that it's best to use the lowest exposure you can get away with and still get adequate shadow detail (expose to the left - if you think of dark on the left and bright on the right).  That makes some sense because the more you expose a negative, the more material is washed away in the developer - till it's totally transparent.   You can compensate for a dark negative by printing it lighter.  If you overexpose, and there's nothing left on the film, you can't print it darker.

For digital cameras things are different:  The argument for "expose to the right" (optimal exposure is just at the threshold where you lose too much detail in the bright areas) is interesting.
Human light perception is logarithmic.    Twice as bright seems just like a linear increment.  Every time you double the light, it appears to be just a little brighter by the same perceived amount.      Camera lenses are calibrated that way (one F-stop - opening the iris - doubles the light).  And shutter speed dials are the same.  1/125, 1/250, 1/500.
Doubling is the natural increment.

But camera sensors don't work that way.   Pretty sensibly , doubling the light produces double the electrical response.     The "raw" format image encodes those electrical responses just as they come.     A typical digital camera sensor can record a 10 "stop" range ( darkest x 2 x 2 x2.... 2   brightest- ten doublings from darkest to brightest).

But (the argument says), it records that range linearly - so the final brightest area takes half the scale available!      Most sensors record each of the R G & B pixels with 12 bits.
That's 4096 levels of brightness.    2048 of them are used for just the brightest - final doubling - area of the image.
1024 for the one before that.     So you only have 1024 shades for the whole rest of the image brightness range - which usually includes more important stuff.

SO:  The idea is that by pushing all of the brights farther to the right (overexposing till you start to lose detail), you will have more bits on the scale for encoding the rest of the image.    You just lower the exposure back down in Photoshop.   The brights will still have more than their share of bits so you will be able to recover what looks washed out in the unprocessed overexposed image.

(There's a lot of debate about this idea.   I don't understand why cameras wouldn't just apply a log transform to the signal before encoding the data.  I wouldn't have thought that was a big deal)  I tried ETTR yesterday.  Hard to tell for sure, but I think I got more noise in the mid tones.   But I can't say for sure.  And I don't know why I'd get that result.

One writer who seemed technically knowledgeable (here's his PDF) argued that Expose to the Right (ETTR in the blogs) does make sense, but not because of the argument above.
It works because more is gained by reducing noise in shadow areas - where there isn't enough light for a clear signal than is lost by brightening the highest highlights.

It's not about the encoding, but about the sensor itself and how well it gives a clean image at low light levels.   It does better the more light it gets.  Clearly true for small sensors.  My iPhone is AWESOME if there's bright even light.   It's a POS in low light.  So this argument makes more sense to me.  You can't fix a sensor that just needs more photons to get a good signal, but you could easily fix simple numerical encoding of a scale.