Tones and Dynamic Range. Why You Should Shoot RAW

To help me keep on writing this blog, please consider buying your Adobe Photoshop Lightroom license at the best price on Amazon using the link below.


I'm writing this post to provide a small introduction about how a digital camera sensor reads and stores data, how human eyes perceives this data and the impact that all of this has on your decisions as a photographer. There's some mathematics in it that I'm not willing to avoid: although photographers should not worry about mathematics and such "technicalities", it's important to understand what's going on inside your camera, first, and in post-processing, later, if you want to get the better out of your images.

How The Sensor Reads and Stores Data

The first thing that's important to understand is that your camera sensor is linear. That's the easiest part of all: the sensor reads light intensity, samples it and stores its value. It's plain old sampling going on here. Cameras typically use three channels of a fixed bit-height to store light intensities of the three primary colors. The bit width of the channel depends on the camera and the wider the channel, more information can potentially be stored into it.

Photographers usually think in terms of f-stops or zones. Changing the exposure by an f-stop means doubling the light that reaches the sensor, if the compensation is positive, or halving it if the compensation is negative. Photographers use f-stops because the eye is a logarithmic sensor: in its working range, it is sensible to relative differences in light intensities, regardless its the absolute value.

Since sensors are linear, how is a specific zone's data distributed into the RAW file? If we think in binary representation, doubling the light intensity (stepping up 1 f-stop) means shifting a sensor reading (for each pixel) to the left (we're using 8 bits as an example):

7 6 5 4 3 2 1 0    7 6 5 4 3 2 1 0
--------------- -> ---------------
f 0 0 0 0 a b c    0 0 0 0 a b c 0

On the other hand, halving the light intensity (stepping down 1 f-stop):

7 6 5 4 3 2 1 0    7 6 5 4 3 2 1 0

--------------- -> ---------------

0 0 0 0 0 a b c    0 0 0 0 0 0 a b

Let's now think in terms of zones. Basically, an n-bit channel can store data of up to n zones and zones don't use the same number of bits to be stored (and thus they cannot store the same level of detail). The highest zone, in this case the 1st, is 7 bit wide, the 2nd is 6 bit wide and so on, up to the 8th, which is 0-bit wide.

If you're proficient in Mathematics, this is in fact pretty obvious and derives from the very nature of the binary representation: adding 1 bit to the representation means doubling the range of values you can express with those bits. But stepping up 1-stop from a zone to another precisely means doubling the light intensity. That's why the maximum number of zones you can store in an n-bit number is n, and each zone will have half of the bits of the previous one (from the lightest to the darkest), to store its information.

You know understand why a camera such as a Nikon D5100, that produces 14 bit RAW files, may have a dynamic range of about 13 EV.

How Many Levels Can Be Stored In Each Zone?

As we've seen, the number of bits available to store the data of each zone decrease to a half from one zone to the next one (from the lightest to the darkest). Since the number of distinct unsigned integer values that you can store with an n-bits representation is 2^(n), it comes out that each in each zone you will be able to store an exponentially decreasing number of levels. For an 8-bit file and a 14-bit file, you will have the following:

Zone  | Levels | Levels |
------+--------+--------+
    1 |    128 |   8142 |
    2 |     64 |   4096 |
    3 |     32 |   2048 |
    4 |     16 |   1024 |
    5 |      8 |    512 |
    6 |      4 |    256 |
    7 |      2 |    128 |
    8 |      1 |     64 |
    9 |      - |     32 |
   10 |      - |     16 |
   11 |      - |      8 |
   12 |      - |      4 |
   13 |      - |      2 |
   14 |      - |      1 |
------+--------+--------+

Awful numbers in the 8-bit case, aren't they? If you've heard about the zone systems, you're probably expecting at least 8 zones in your shots. If you're starting to worry about them, and thinking something like: "Am I saving my images in 8-bits JPG files?", then: yes. There's plenty of reasons to worry about this. But still wait some minutes and read on.

Human Vision

The human eye response to light intensity is logarithmic, under all our practical purposes, amongst the visible spectrum. We have to factor this into the equations to correct the estimations we did in the previous section.

The correction we've got to apply is well known, and you probably heard about it: it's called gamma correction. I won't go into details in this post, the linked Wikipedia article is pretty well done and useful for our introductory purposes. However, it's important to stress out that gamma correction partially compensates for the great (exponential) unbalance in the number of levels we can store for each zone.

The gamma corrected intensities are calculated with:

v_o = v_i ^ gamma

where the gamma exponent is a number that commonly takes the value 2.2. The monitor you're using is applying a gamma correction right now, as well as whichever photographic software you have.

Since we're interested in the zones we perceive (v_o in the previous equation), we should apply the inverse transformation to the values we've calculated and update our estimates accordingly.

Let's then assign some values to the zones so that we can gamma correct them. We will use the following (rounded at 2 decimal digits):

Zone  | Value  | Gamma     |

      |        | Corrected |

------+--------+-----------+

    1 |   8142 |     60.09 |

    2 |   4096 |     43.85 |

    3 |   2048 |     32.00 |

    4 |   1024 |     23.35 |

    5 |    512 |     17.04 |

    6 |    256 |     12.43 |

    7 |    128 |      9.07 |

    8 |     64 |      6.62 |

    9 |     32 |      4.83 |

   10 |     16 |      3.52 |

   11 |      8 |      2.57 |

   12 |      4 |      1.88 |

   13 |      2 |      1.37 |

   14 |      1 |      1    |

------+--------+-----------+

To determine how the corrected values distribute into an n-bit channel, we apply a linear transformation, to "stretch" them into the desired interval. For an 8-bit channel the scale factor is 255/60.09 while for a 14-bit channel the scale will be 16383/60.09. Results are:

Zone  | Maximum | Maximum |

      | Zone    | Zone    |

      | Value   | Value   |

------+---------+---------+

    1 |     255 |   16383 |

    2 |     186 |   11955 |

    3 |     136 |    8724 |

    4 |      99 |    6366 |

    5 |      72 |    4646 |

    6 |      53 |    3390 |

    7 |      39 |    2474 |

    8 |      28 |    1805 |

    9 |      21 |    1317 |

   10 |      15 |     961 |

   11 |      11 |     702 |

   12 |       8 |     512 |

   13 |       6 |     374 |

   14 |       4 |     273 |

------+---------+---------+

It's pretty evident that the width of the zones is more balanced than it was in the non gamma corrected case. The last step is to calculate the number of levels per zone, subtracting from the maximum value of a zone the maximum value of the next zone:

Zone  | Levels | Levels |

------+--------+--------+

    1 |     69 |   4428 |

    2 |     50 |   3231 |

    3 |     37 |   2358 |

    4 |     27 |   1721 |

    5 |     20 |   1256 |

    6 |     14 |    916 |

    7 |     10 |    669 |

    8 |      8 |    488 |

    9 |      6 |    356 |

   10 |      4 |    260 |

   11 |      3 |    190 |

   12 |      2 |    138 |

   13 |      2 |    101 |

   14 |      1 |     74 |

------+--------+--------+

8-bit images aren't that bad, in fact, but aren't that good, either.

It's clear that using RAW is a huge improvement. Also, when converting from RAW to another format, you should try and avoid 8-bit formats such as JPEG, unless you're willing to lose all that information. You should try and stick with 16-bit image file formats, although few programs can use them. Notably, Photoshop Elements is able to open them but not manipulate them. It's a good selling point for Photoshop, if you're a professional.

How Bad Is Using 8 Bits Image Files?

To fully understand how bad 8-bits images can be, it's necessary to understand how sensible human eyes are to light intensity. It turns out that this question is answered by the Weber-Fechner Law. This law states that human eyes are sensible enough to distinguish a difference in light intensity of about 1%.

How many such levels are in a zone? To calculate it, using the definition of a zone (an interval in which the intensity of light doubles), you must find the number x such that:

(1.01)^x = 2

That number is approximately 70. There are 70 levels in a zone that human eyes can distinguish. Let's observe once more the zone levels in the 8 bit and 14 bit case. We notice that 8 bit images provide a good level of detail only in the brightest zone. In the darkest ones artifacts such as "banding" will easily occur. A 14-bit file, on the other hand, will provide good zone level details up to the darkest zone. That's one more reason why you should always use RAW when shooting and post processing your images. If you use narrowest channels, artifacts will soon pop up.

Useful Tips for Shooting

Now that we've learnt how zones are stored into our image files and the level of details we can expect from each one of them, we can draw some conclusions that may help us shoot the perfect photo.

Although gamma correction introduces some balance into the number of levels that can be stored for each zone, it's clear that brightest zones can use much more information than darkest one. Under this aspect, digital sensors like a bit of overexposure. To capture the highest level of details, you should guarantee that the zones you're interested are exposed to the right. If you slightly overexpose, provided you don't burn out any channel, you can lower the exposition in post-production while retaining the maximum number of details.

Even with dynamic ranges as wide as 14 EV, you must be careful not to burn a channel out. When a channel has filled up, you start clipping information and the first side effects will be: a partial loss of color saturation, a color drift and, finally, the saturation of all the channels to a pure white.

Many RAW files will let you recover 1 f-stop of light, but if you clip the white, as well as the black, you will be losing information. That's why on camera histograms are a good tool to check your exposure and ensure you're using the channels efficiently.

Do These Recommendations Apply Only to Pro?

No. I'm not a pro, either. However, it's pretty easy to see how artifacts will quickly appear in a typical 8-bit image with relatively light post-processing. For example, bands quickly appear in the dark areas and information is not sufficient to modify the exposure of a shot even for only 1 f-stop.

If you can shoot RAW, do it. And if you can distribute and store 16-bit images, do it as well. Today, very few people is going to run out of storage space for his photographies and, at least, you should be aware that you're going to lose a great deal of information with an apparently innocuous transformation (8-bits images can use more than 16 millions colors, but now you know that this is insufficient in many cases).

How To Disable Time Machine Local Backups (and Relinquish Space in the MobileBackups Folder)

This morning, after importing a bunch of RAW files from my Nikon camera, I was startled (once more) by the lack of responsiveness of my MacBook Pro running OS X Lion.

Since the update, in fact, I'm growing pretty disappointed with Lion. It never roared just like the good ole Snow Leopard. The Lion cub has only shyly purred and whined, so far.

One of the things that bothers me most is Spotlight performance. Assuming you noticed it, have you ever wondered what the mdworker process is doing when it jeopardizes your Mac resources? I have: staring impotently at the screen, listening to the spinning rust crackling, powerlessly witnessing how the operating system is waiting to complete who-knows-what.

This morning, I opened Lightroom, started importing some RAW files, and started working on them. Well, trying to work on them while Lion was performing its... "housekeeping".

Frustrated, I expressed my anger on Google+. A friend of mine, Joaquin, pointed out both the cause and a workaround of the new Lion's behaviour.

It turns out that Lion is performing periodic local backups that will be flushed upon your time machine disk when connected. Thinking about versions, aren't you? Anyway, that's also why the new /.MobileBackups folder into your OS X hard disk is eating up disk space. Never noticed purple and white snapshot lines in Lion's Time Machine user interface? White ones are local backups, purple ones are "regular"time machine backups on an external disk. Well, local backups are kept for 8 days and they'll be taken only if you've got enough space on your disk. However, if you, like me, manage huge quantities of big files (such as a camera RAW files), this feature can start eating up an enormous amount of space.

If you, like me, feel like turning off this amazing feature you can use the following command:

$ sudo tmutil disablelocal

Should you change your mind about it, you can always turn it on again using:

$ sudo tmutil enablelocal

These are just the kind of options that should be accessible in the user interface.

Hope this helps you.

Java Generics Tutorial - Part IV - Wildcards in Method Signatures and Bounded Type Variables

Part I - The Basics
Part II - Subtyping
Part III - Wildcards
Part IV - Bounded Type Variables

Long time no hear. In the previous parts of this blog post we learned what generics classes and methods are, how they behave in subtyping relations and how wildcards can be used to provide covariant and contravariant subtyping to generic types.
In this part of this series we will learn what bounded type variables are and the flexibility they provide.

Wildcards in Method Signatures

As seen in Part II of this series, in Java (as in many other typed languages), the Substitution principle stands: a subtype can be assigned to a reference of any of its supertypes.

This applies during the assignment of whichever reference, that is, even when passing parameters to a function or storing its result. One of the advantages of this principle, then, is that when defining class hierarchies, "general purpose" methods can be written to handle entire sub-hierarchies, regardless of the class of the specific object instances time being handled. In the Fruit class hierarchy we've used so far, a function that accepts a Fruit as a parameter will accept any of its subtypes (such as Apple or Strawberry).

As seen in the previous post, wildcards restore covariant and contravariant subtyping for generic types: using wildcards, then, let the developer write functions that can take advantage of the benefits presented so far.

If, for example, a developer wanted to define a method eat that accepted a List of whichever fruit, it could use the following signature:

void eat(List<? extends Fruit> fruits);

Since a List of whichever subtype of the class Fruit is a subtype of List<? extends Fruit>, the previous method will accept any such list as a parameter. Note that, as explained in the previous section, the Get and Put Principle (or the PECS Rule) will allow you to retrieve objects from such list and assign them to a Fruit reference.

On the other hand, if you wanted to put instances on the list passed as a parameter, you should use the ? super wildcard:

void store(List<? super Fruit> container);

This way, a List of whichever supertype of Fruit could be passed in to the store function and you could safely put whichever Fruit subtype into it.

Bounded Type Variables

The flexibility of generics is greater than this, though. Type variables can be bounded, pretty much in the same way wildcards can be (as we've seen in Part II). However, type variables cannot be bounded with super, but only with extends. Look at the following signature:

public static <T extends I<T>> void name(Collection<T> t);

It takes a collections of objects whose type is bounded: it must satisfy the T extends I<T> condition. Using bounded type variables may not seem more powerful than wildcards at first, but we'll detail the differences in a moment.

Let's suppose some, but not all, fruits in your hierarchy can be juicy as in:

public interface Juicy<T> {
    Juice<T> squeeze();
}

Juicy fruits will implement this interface and publish the squeeze method.

Now, you write a library method that takes a bunch of fruits and squeezes them all. The first signature you could write might be:

<T> List<Juice<T>> squeeze(List<Juicy<T>> fruits);

Using bounded type variables, you would write the following (which, indeed, has got the same erasure of the previous method):

<T extends Juicy<T>> List<Juice<T>> squeeze(List<T> fruits);

So far, so good. But limited. We could use the very same arguments used in the same posts and discover that the squeeze method is not going to work, for example, with a list of red oranges when:

class Orange extends Fruit implements Juicy<Orange>;
class RedOrange extends Orange;

Since we've already learned about the PECS principle, we're going to change the method with:

<T extends Juicy<? super T>> List<Juice<? super T>> squeezeSuperExtends(List<? extends T> fruits);

This method accepts a list of objects whose type extends Juicy<? super T>, that is, in other words, that there must exist a type S such that T extends Juicy<S> and S super T.

Recursive Bounds

Maybe you feel like relaxing the T extends Juicy<? super T> bound. This kind of bound is called recursive bound because the bound that the type T must satisfy depends on T. You can use recursive bounds when needed and also mix-and-match them with other kinds of bounds.

Thus you can, for example, write generic methods with such bounds:

<A extends B<A,C>, C extends D<T>>

Please remember that these examples are only given to illustrate what generics can do. Bounds you're going to use always depend on the constraints you're putting into your type hierarchy.

Using Multiple Type Variables

Let's suppose you want to relax the recursive bound we put on the last version of the squeeze method. Let's then suppose that a type T might extend Juicy<S> although T itself does not extends S. The method signature could be:

<T extends Juicy<S>, S> List<Juice<S>> squeezeSuperExtendsWithFruit(List<? extends T> fruits);

This signature has pretty much equivalent to the previous one (since we're only using T in the method arguments) but has got one slight advantage: since we've declared the generic type S, the method can return List<Juice<S> instead of List<? super T>, which can be useful in some situations, since the compiler will help you identify which type S is according to the method arguments you've passed. Since you're returning a list, chances are you want your caller to be able to get something from it and, as you've learned in the previous part, you can only get Object instances from a list such as List<? super T>. 


You can obviously add more bounds to S, if you need them, such as:


<T extends Juicy<S>, S extends Fruit> List<Juice<S>> squeezeSuperExtendsWithFruit(List<? extends T> fruits);

Multiple Bounds

What if you want to apply multiple bounds on the same type variable? It turns out that you can only write a bound per generic type variable. The following bounds are thus illegal:

<T extends A, T extends B> // illegal

The compiler will fail with a message such as:

T is already defined in...

Multiple bounds must be expressed with a different syntax, which turns out to be a pretty familiar notation:

<T extends A & B>

The previous bounds means that T extends both A and B. Please take into account that, according to the Java Language Secification, Chapter 4.4, states that a bound is either:

• A type variable.
• A class.
• An interface type followed by further interface types.

This means that multiple bounds can only be expressed using interface types. There's no way of using type variables in a multiple bound and the compiler will fail with a message such as:

A type variable may not be followed by other bounds.

This is not always clear in the documentation I've read.

Multiple Icons of the Same Application in Launchpad

One of the new features introduced by OS X Lion is the Launchpad. The launchpad gives you access to the installed applications in your Mac from a window that is reminiscent of iOS devices. In fact, it superimposes on your current desktop window, with a fancy blur effect, and a list of application icons is presented in a rectangular grid. You can even make folder in it by dragging an icon over one other. It seems an iPhone screen on steroids.

In my humble opinion, it's awful.

Anyway, one day, using a wrong gesture, the Launchpad screen was launched and I noticed that the Skype icon was presented twice. Strange, thought I. I tried to remove one of them, but they could not be deleted. I tried dragging one into the trash. It didn't work. To make the story short, I did not lose time investigating this weird Launchpad behavior: it's the kind of window I prefer not to see at all.

Some weeks after Skype informs me that an update is available: I accept to install it immediately and wait for the installer to finish. Curiosity killed the cat: I opened the Launchpad and guess what? Another brand-new Skype icon had just been added to the long row of Skype icons already present in the Launchpad window.

A quick glance confirmed what I already knew: there was just one Skype application installed in my Mac and no other link to it was present whatsoever that might have gotten Launchpad confused. I completely removed Skype from the system and reinstalled it from scratch. No way.

It was just time to act and stop the uncontrollable icon proliferation. Since every user can customize Launchpad, I thought that a good place to start was the ~/Library folder. I grep-ped every file into it looking for the "Skype" string (the icon title in the Launchpad) and a file caught my attention:

~/Library/Application Support/Dock/FABCDB4F-FBDE-42E4-A14F-80FA7476989C.db

An SQLite 3 database file (as confirmed by the file command)! It looks like the potential offender, doesn't it? I then backed up the file, removed it, logged out and logged in again. I started the Launchpad and bingo! Launchpad had just regenerated the database file and now only one Skype icon was shown in the Launchpad window.

As I said, I'm not going to use again. But at least I can rest assured that icons are gone away. Hope this can help you guys if you run into similar troubles with Launchpad.

Weird Auto ISO Behaviour on Nikon D5100 With a Hot-Shoe Mounted Flash

If you want to help me keep on writing this blog, please buy your new Nikon camera at the best price on Amazon using the link below.


ISO sensitivity is one of the three parameters, together with aperture and shutter speed, used to determine the exposure of a shot. Raising the ISO sensitivity lets the sensor react more quickly to light, thus allowing for smaller apertures or faster shutter speeds. The ISO sensitivity scale is linear: using an ISO 200 setting, for example, will have the sensor react with twice the speed than with an ISO 100. If other parameters are kept fixed, raising ISO sensitivity from 100 to 200 corresponds to a light increment of 1 stop.

The possibility of choosing an appropriate ISO sensitivity with the flick of a switch gives photographers a degree of freedom: you can maintain a fixed exposure level changing the ISO sensitivity and compensating with a corresponding aperture change or shutter speed change. It's so handy that I assigned the Fn button of my camera to ISO sensitivity, so that I can change it with just one click.

Auto ISO

To make photographers' life easier, many digital cameras offer an Auto ISO mode: the camera will automatically raise the ISO sensitivity in insufficient light conditions. Auto ISO on recent Nikon cameras works as follow:

• The currently (manually) selected ISO sensitivity is treated as a minimum.
• You choose a maximum ISO sensitivity.
• You choose a minimum shutter speed.
• When light condition is such that a shutter speed slower than the selected minimum is required, the camera will automatically increase the ISO sensitivity.
• When the maximum ISO sensitivity is reached, the camera won't increase it further and will fall back to changing other parameters, depending on the mode you're shooting.

The good thing of this algorithm is its predictability: I turn Auto ISO on often and also made it part of a custom menu for easier access (although I'd really like to be able to assign a button to it).

Auto ISO with a Hot-Shoe Mounted Flash

The behaviour of the Auto ISO algorithm is consistent when using the camera pop-up flash in slow mode as well. Unfortunately, things are weirder when using a hot-shoe mounted flash, such as an SB-400. In this case, the camera increases the ISO sensitivity up to four times the value currently selected and won't raise it any more, even if you selected a greater maximum ISO.

Let's suppose you chose a value of 1600 ISO as maximum sensitivity and the current sensitivity is set to 100. When using a hot-shoe mounted flash, the camera will progressively increase the sensitivity up to four time the selected ISO, in this case 400, and will not increase it any more.

As far as I know, this behaviour is not documented in the camera manual and a quick Google search confirms this behaviour is known on other Nikon cameras as well.

Once you learn it, it's something you can live with. In fact, I often manually increase the ISO sensitivity so as to increase the maximum sensitivity the camera will choose. Since I'm not often using sensitivities as high as 6400 ISO, that's just a couple of button clicks away.

However, since it's undocumented behaviour, I do consider that overriding the it Auto ISO settings is no good. After all we shoot manually, although with the help of partially automated task such as this, because we're supposed to know what we're doing.

Could I quickly switch Auto ISO on and off, I'd surely shoot with manual ISO when using an external flash. Unfortunately, the Nikon D5100 won't allow you to do this easily, the quickest way being customizing your own menu; but to be fair, the reduced control customization capabilities is by far the only complaint I have about the D5100.

Update: D5100 Firmware v. 1.01

As described in a later post, the 1.01 firmware seems to fix this undocumented "feature": the Auto ISO behaviour is the same when using either the pop-up flash or a hot-shoe mounted flash.