Using XMP Metadata Effectively with Lightroom

Yesterday I had an interesting discussion about the upcoming Adobe Lightroom 4 with a reader of this blog. One of the questions that came up was: "What are XMP files?"

That's an interesting question about a subject that's often overlooked. Many Lightroom users, in fact, need not concern about this very often. However, they're a great tool you should know about since they can dramatically improve your user experience and also show the way to a more proficient use of Lightroom (as well as any other programs understanding this kind of files).

What Is XMP?

XMP stands for eXtensible Metadata Platform, a standard proposed by Adobe Systems, Inc. to define, create, store and process extensible file metadata.

Many file formats already contemplate some mechanisms to store metadata, even forms of extensible metadata. The advantage of being able to embed XMP data into a file is a huge leap towards standardization, since you will be able to take advantage of it in different applications. Also, compliant applications will simply ignore the content they can't handle.

At times, however, embedding XMP data into an existing file isn't an option: that is the case, for example, when dealing with proprietary RAW file formats (such as Nikon's NEF). You can obviously save XMP data for such a file too, but you'll have to use a so-called sidecar file, that is, a distinct XMP file provided alongside the original file. This solution is more cumbersome and error prone, but some people will argue that it has its own advantages such as leaving the original file unchanged, thus avoiding potential file corruption.

Why Is This Important to Me as a Photographer?

For all your practical purposes, Lightroom is able to save all the information provided by the user during the development process, including Adobe Camera RAW settings, in an XMP-compliant way, either embedded (when the file format allows it) or in a sidecar file.

If your workflow is performed in just one application and one machine, you won't probably benefit so much from this feature. However, as soon as you need to move your images to another application, you will probably face the problem of preserving adjustments (which can be thought of as metadata themselves).

Even a simple scenario such as test driving another program (such as the upcoming Lightroom 4) or moving the files to another machine will require you to migrate metadata as well. If your application isn't storing metadata in the source file itself, or if you choose not to do so, you will need to export it and make it available to the target application.

Migrating Images And Metadata

The simplest scenario is migrating an entire Lightroom catalog from a Lightroom instance to another, as in the case of a Lightroom upgrade. Since Lightroom stores the information of all your photos into the catalog itself, which act as a centralized database, it's sufficient to perform a copy of it and simply import it into another Lightroom instance. This operation is straightforward and no further option will be required for you to preserve all your metadata.

Even if you need to migrate only a subset of your catalog to another Lightroom instance there's an easy way to go. Select the images you want to migrate and choose the File/Export as Catalog menu item. Lightroom will generate a brand new catalog for you to migrate:

Export as Catalog Dialog Box

Things get slightly more complicated when you need to move images to a different application and metadata cannot be embedded into the original file. In this case, you have to tell Lightroom to generate the relevant XMP sidecar files for the images to be migrated and move them alongside the originals. You've got two choices:

• Have Lightroom alway save an XMP sidecar file for every photo.
• Have Lightroom generate on the fly XMP sidecar files only for the photos you need to migrate.

Unless your requirements justify the choice, I would always choose the second option since the first will introduce a significant, and often unnecessary, overhead.

To have Lightroom always generate XMP sidecar files, you need to select the Automatically write changes into XMP option in the Metadata tab of the Catalog Settings window:

Catalog Settings Window

Since I always leave this option off, as you can see in the previous screenshot, Lightroom is warning me that "changes made in Lightroom will not automatically be visible in other applications". If you select it, Lightroom will generate a sidecar file for every photo you process and keep it in sync.

To generate an XMP sidecar file on the fly, all you need to do is using the Metadata/Save Metadata to Files menu item (or use the Command-S shortcut if you're on a Mac).

Either way, Lightroom will generate the relevant XMP files alongside your original images, as you can see in the following screenshot:

XMP Files Together With Their Originals

No matter how you decide to generate them, you can now take your images and use them into another application. All the metadata that the target application is able to understand will be processed and the corresponding changes reapplied.

This mechanism is interesting also in the case you're often working with more than one computer. You could keep the catalog in sync, as explained in another post, or just keep XMP files in sync. Once you move an XMP file, just put it alongside the original and have your application read the metadata from it. In the case of Lightroom, all you have to do is using the Metadata/Read Metadata from Files command.

You could argue XMP metadata is easier to backup than an entire Lightroom catalog. Well, I bet it is. However, you Lightroom catalog is more than that and chances are you're backing it up regularly as well, thus defeating this potential advantage. 

Is There a Simpler Way?

This procedure is pretty simple and many XMP compliant applications (such as Lightroom, Camera RAW and Photoshop) will do the heavy lifting for you. Provided the XMP file is distributed with the original, they won't even ask you anything: they'll just read the metadata and apply the relevant settings.

As explained in the previous sections, you rely on XMP files in two cases:

• You cannot modify the original image and embed metadata into it.
• You don't want to modify the original image.

If you want to preserve your original files, there's no other option but using sidecar XMP files. If you cannot modify the original image (probably because it's written in a proprietary RAW format) but would willingly have Lightroom modify it if it could, you should consider converting your images to the DNG format (either during the import phase or afterwards with a simple export operation).

DNG stands for Digital Negative and it's a file format from Adobe, Inc. whose purpose, amongst others, is standardizing a RAW format and enable multi-vendor interoperability. However, DNG is much more than that:

• It's based on TIFF.
• It's both a container of RAW images and processed images.
• Metadata can be embedded in a variety of formats, including XMP and EXIF.

Some camera manufacturers already support on-camera DNG. If your camera does not, you can use a converter (such as Lightroom) to generate a DNG file from a proprietary RAW file. You can even choose to embed the original RAW file into the DNG, which is something I would recommend you do since sometimes some proprietary RAW metadata might be lost during the conversion.

Since XMP metadata can be embedded into a DNG file and Lightroom will keep it in sync, you will need neither no sidecar XMP file to migrate your photos. To some extent, you won't even need to rely on the very Lightroom catalog itself.

If you want to help me keep on writing this blog, buy your Adobe Photoshop licenses at the best price on Amazon using the links below.

Fixing Mac OS X Lion's SSH UTF-8 Issues

After switching from Mac OS X v. 10.6 (Snow Leopard) to OS X v. 10.7 (Lion), I began experiencing some weird locale-related problems when using OS X as an SSH client. The issues I experienced include, amongst others:

• Having the terminal complain about invalid locale settings:

$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory 
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE=UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8" 
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8" 
LC_TELEPHONE="en_US.UTF-8" 
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

• Being unable to correctly type some characters (such as accented characters).
• Having programs that tunnel their traffic over SSH, such as Subversion, complain about invalid locale settings during their operations:

$ svn co svn+ssh://user@repository/path
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is UTF-8
svnserve: warning: please check that your locale name is correct
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is UTF-8
svnserve: warning: please check that your locale name is correct


In fact, if you look for example at the output of the locale command seen previously, you can see how the LC_CTYPE variable is set UTF-8 while all other LC_* variables (excluding the empty LC_ALL), are set to en_US.UTF-8 (which are correct according to that server settings).

What is the SSH client shipped with Lion doing? Apparently sending incorrect values to the server. It's not just a nuisance having to fix them after logging in, provided you're able to do so: this behaviour is breaking other programs tunnelling their traffic over SSH, such as Subversion, that aren't establishing an interactive SSH tunnel for the user.

Fortunately, the fix is quick: just tell the SSH client not to send any LC_* variable:

• Open the SSH client configuration file (/etc/ssh_config).
• Comment out the line that instructs the SSH client to send those variables (SendEnv LANG LC_*) from the general host configuration section (Host *). You should end up with the following fragment:

Host *

# SendEnv LANG LC_*

The configuration change will be immediately applied to any SSH session opened from now on.

Hope this helps.

Book Review: JIRA Development Cookbook, by Jobin Kuruvilla

I just finished reading Jobin Kuruvilla's JIRA Development Cookbook and I'm pretty pleased with it. In my opinion, that's a book worth buying if you want to get serious about JIRA customization and plugin development. I admit there's plenty of information in the Internet about the subject, especially from Atlassian itself, and possibly no question of yours would remain unanswered. However, having a good book like this on your desk and using it as sort of a reference guide is a very good idea.

The Book's Structure

If you read about my first impressions, you'd realize that reading it took me more than a month. Beware: I read it in its entirety because I wanted to review it but this book, as many technical books out there, is not meant to be read from beginning to end as you'd do with a novel. Instead, you should read it diagonally to be aware of what it offers and to be able to quickly get the information you need.

The book is organized into 11 chapters, each of which is a walkthrough of the addressed subject. The typical chapter structure is "programmer friendly": it starts with a high level overview of the subject, analyzing JIRA architecture and internals when deemed necessary, after which it goes on exploring the subject making extensive use of task-lists and code examples. If you already master a subject, more often than not you will end up quickly scrolling the task list and jumping directly into the code. In the following screenshots you can see some excerpts of the book to familiarize with its structure.

A "Getting Ready" Task List

A Summary of What You're Going To Do

Explanatory Section

In Depth Discussion

The first two chapters, Plugin Development Process and Understanding Plugin Framework, offer an overview of the common tasks and procedures you'll undergo during your JIRA plugin development, including the installation of the development environment, the development, debugging and testing phases of your code and the deployment of your deliverable into a JIRA installation.

The third and the fourth chapters, Working with Custom Fields and Programming Workflows, explore some of the most important JIRA customizations you'll deal with when writing your plugins or even just extending the basic platform's functionality: custom fields and workflows.

The fifth chapter, Gadgets and Reporting in JIRA, is dedicated to gadget development and the extension of JIRA's reporting capabilities. The title is somewhat misleading, since this chapter also somewhat deals with JIRA's integration with other systems (at least from a gadget's standpoint), as a consumer of RESTful web services. Another chapter is entirely dedicated to this subject (Chapter 9) and the inclusion of this material in Chapter 5 could pass unnoticed, if not somewhat misleading.

The sixth chapter, The Power of JIRA Searching, examines the techniques you can use to deal with and extend the JIRA query language, JQL, and how to take advantage of it in your plugins.

The seventh chapter, Programming Issues, describes how issues can be managed programmatically. Issue management is extensively covered, discussing subjects such as issue creation and lifecycle management, issue operations, time tracking, issue linking and validation.

The eighth chapter, Customizing the UI, explores how the developer can customize the user interface of JIRA, including customizing and extending the default JIRA pages or creating your own web sections and items.

The ninth chapter, Remote Access to JIRA, is devoted to JIRA integration with external systems using SOAP web services, RESTful web services and, to a lesser extent, JAX-RPC web services. The first part of the chapter introduces the JIRA web service API and how to build a SOAP client. The second part describes how you develop your own web services and deploy them as part of your JIRA installation.

The tenth chapter, Dealing with a Database, instructs the developer in the available tools and how to use them to interact with the database JIRA is connected to. The chapter teaches developers how to programmatically retrieve important JIRA data from the database, such as issue, field and workflow information, and how to extend its schema to hold plugin-specific data.

The eleventh and last chapter, Useful Recipes, gathers interesting information and additional code samples to fulfill common tasks not strictly related with the content of the other chapters.

A Book for Programmers

I defined this book as "programmer friendly" because source code examples are often used as the mean to exemplify how JIRA works internally. Explanations are indeed placed after task lists and code samples, instead of before. In fact, to cite the very author, the book seems to be organized as a cookbook: every section is a recipe with ingredients (prerequisites and task lists), a quick procedure (the how-tos) and, only then, in depth discussions and explanations of the JIRA internals. I'm not a big fan of this kind of content organization but I do admit that, in such a case and for such a subject, it's an effective mean to organize and clarify the content, undoubtedly at the advantage of the reader himself.

Likewise, the book covers a great portion of a plugin development workflow. It's not, and probably cannot be, exhaustive, but rest assured that it will effectively guide you through your development of most, if not all, your plugins, since the vast majority of the APIs and the technologies that you're going to need are covered.

The contents are clearly exposed and are well detailed and I think it's a valuable addition to the official JIRA documentation that's worth having if you're developing on JIRA as a platform. Some subjects that are somewhat less covered in the official JIRA documentation (at least so far), such as UI customization and JQL function customizations, are extensively detailed and are, on their own, a valid reason to own this book.

The only thing I'm missing is a better typesetting: a book should be judged by its contents, and this is what I'm doing, but it's important to recognize that an effective presentation is a key factor to take into account. As a reader, I find the quality of the book typesetting (title formats, paragraph styles, etc.) to stand below the quality of its contents. Of course, it's better like this than the other way round, but I'd appreciate some redesign should this book be revised in the future.

Conclusion

I felt compelled to write a review of this book because I think it deserves it, it's a must-have for a JIRA plugin programmer. Before writing this, I thought I'd write a more in-depth review. Surprisingly, I discovered that enumerating each and every subject this book covers was not necessary: it's a homogeneous, extensive and well organized book.

From a developer standpoint, it's an effective and efficient mean to learn the subject and get the job done, much better than any tutorial I've seen so far. Together with the official Atlassian JIRA documentation, this book is pretty much all you need.

Title: JIRA Development Cookbook
Author: Jobin Kuruvilla

Pages: 476

Publisher: Packt Publishing

ISBN-10: 1849681805

ISBN-13: 978-1849681803

Adobe Photoshop Lightroom: How I Share Catalogs Across Different Devices In a Networked Environment

During my workflow as an amateur photographer I'm using Adobe Lightroom and Photoshop on Mac OS X. As many of you already know, Lightroom does not allow you to use a catalog on a network share. That doesn't mean, however, that all your hopes of easily replicating it across different devices are lost. This is what I'm doing and to replicate it you will need:

• An NFS server: there are many operating systems that can easily be configured as NFS servers. The quickest way to go it's probably GNU/Linux. Personally, I opt for Solaris and I'll tell you why in a few minutes.
• The ability to mount NFS file system from your client devices: this is easily accomplished in every operating system supported by Lightroom.
• A way to easily synchronize your catalog across the different client devices you will use. I use rsync.

NFS Server

My photos are stored on remote storage, published using the NFS protocol, and accessible from any client I need to work from. The NFS server in my home setup is not only the place that publishes my data: it's also the place that stores it. Personal data is valuable. That's why I'm replicating it in a configuration of hard disk drives with a high level of redundancy. Even so, really important things are backed up regularly. Do not use non-redundant storage for long term storage of your data. Stay away from those cheap "storage servers" built into a single disk drive. No matter how good the brand is or how much it costs, any drive will fail: you just don't know when it's going to happen.

The first difficulty to overcome when dealing with personal storage is: easy of use. I bet many of us would not adopt a solution, no matter how good, that badly impairs our user experience. Fortunately, though, it's very easy to build a relatively cheap home storage solution that fulfills all our needs. However, I'm not going to explain you how: just search the Internet and you'll discover plenty of different options that are available to you.

My home storage solution is built upon Solaris. Explaining why and all the benefits I enjoy would be too extensive: I'll just say I love the ZFS file system. ZFS and Solaris are the stone that killed so many birds that I don't imagine living without it.

First of all, it's ridiculously simple to export a ZFS file system using the NFS protocol. In some cases I export other kinds of target, such as iSCSI, which I need to have Time Machine working onto remote storage. If you're interested, you can check an old post about it.

NFS Client

Once you've installed your favorite NFS server of choice, it's time to use it. Windows ships with good and native NFS support at least since Vista and if you're using ZFS you will also be able to access ZFS snapshots from the Windows UI. UNIX and UNIX-like operating systems, such as Mac OS X and GNU/Linux, usually have good NFS support as well.

If you're using Mac OS X, mounting an NFS file system is a no brainer. Just open the Finder, select Go/Connect to Server… and just write the address of your NFS server.

Finder - Connect to Server...

OS X will mount the NFS share and it will appear as a new network drive:

Finder - Mounted NFS Share Mounted Displayed as a Network Drive

When organizing photos in Lightroom, just store them into this drive. As far as it concerns Lightroom, there's absolutely no difference when working on photos on your local hard disk or with photos on a network share such as this. In the next picture you can see the Folders panel in Lightroom Library module showing how I'm storing my photos on that NFS share.

Lightroom - Folders Panel

Synchronizing the Catalogs

Now everything but one thing, the catalog itself, is stored on the network in a location accessible from all your clients. The last problem is synchronizing the catalog across devices.

Of course, I'm using an NFS file system to store a copy of the catalogs too. The only thing we need to do is:

• Synchronize it on the NFS share when leaving a device.
• Synchronize it back from the NFS share when using another device.

This sounds difficult but it absolutely is not. Moreover, always performing a synchronization of your catalog is a good practice that will protect you from losing it because of missing or improper backup policies.

You might be wondering that the synchronization of a big catalog could take a lot of time. Well, it certainly could, if done naïvely with the inappropriate tools. A tool such rsync, for example, will help keeping synchronization times to a minimum taking care of just synchronizing the differences between the two catalog copies.

Another good news for Mac OS X user is the following: Mac OS X ships with rsync, you won't need to install anything on your Mac OS X devices.

Synchronizing an entire directory tree with rsync is "trivial", but it requires some command line proficiency:

$ rsync -az --inplace --delete [local-catalog] [remote-catalog-parent]

rsync has got plenty of options: the previous command is a good starting point to begin synchronizing your catalogs while you learn using rsync.

To synchronize the catalog back to another device, just change the source with the target:

$ rsync -az --inplace --delete [remote-catalog] [local-catalog-parent]

In my case, for example, I'd run the following line when synchronizing a catalog to the NFS server:

$ rsync -az --inplace --delete /Users/enricomariacrisostomo/Pictures/Lightroom /Volumes/enrico/lightroom-backup

On the other hand, when I need to synchronize a local device against the remote catalog, I just run the following:

$ rsync -az --inplace --delete /Volumes/enrico/lightroom-backup/Lightroom /Users/enricomariacrisostomo/Pictures

A Word of Warning: do not use trailing slashes in the previous commands unless you know what you're doing. If both source and target are directories, and you're not using trailing slashes, rsync will synchronize the source directory into a subdirectory of the target directory. In the previous example, rsync is creating a Lightroom directory inside lightroom-backup.

To speed things up, just create shell scripts with your synchronization commands and run them when you need them.

These commands may sound scary if you're not accustomed to using the terminal. However, I think the benefit is worth the trouble. If you don't want to fiddle with the terminal, ask a friend of yours to write the scripts for you. Also, you may find decent rsync GUIs out there, should you prefer using one.

Just to give you an idea of how efficient rsync is, here's an example: I just imported 20 RAW files into a catalog of mine and worked on them. This catalog is managing more than 10.000 images and is made up of more than 45.000 files. Here's the tail of the rsync output log of the last synchronization:

sent 34421438 bytes received 231055 bytes 2772199.44 bytes/sec
total size is 16648567921 speedup is 480.44

Pretty fast, isn't it? When you get used to it, you can't live without it.

Further Steps

Depending on how your catalog is configured, you may want to synchronize your Lightroom presets as well. Please check this post for further information. The technique to synchronize them will be exactly the same technique we used for catalogs.

If you want to help me keep on writing this blog, buy your Adobe Photoshop licenses at the best price on Amazon using the links below.

Misty Sunset on a Mancha Plain

A couple of weeks ago I was on a AVE (a Spanish high-speed train) coming back home from Málaga. The trip lasts two hours and a half and I was pretty tired and half asleep when the sunset caught my attention. The train had just passed Ciudad Real, a city in Castilla-La Mancha, and found itself in the middle of a plain. A few kilometers away, just in front of some small hills, there was some mist that could be easily confused with a cloud. The atmosphere was incredibly clean and since the sun had just set behind the hills there was a nice luminance gradient in the sky: from a deep blue (not really caught in the shot because it's landscape oriented) to a very bright yellow and orange shades. That was really worth a shot.

The train was moving at almost 300 km/h (186 mph) and the lighting in it was pretty strong. However, I came out with a decent shot I'm sharing with you. When I took it I had a black and white picture in mind. However, the colors were beautiful and I'm going to share both versions with you.

Misty Sunset on a Mancha Plain - Color

Misty Sunset on a Mancha Plain - Duotone Sepia

If you want to see them bigger, the images in this blog are linked with the original pictures in Flickr.