Thursday, February 23, 2012

Using XMP Metadata Effectively with Lightroom

Yesterday I had an interesting discussion about the upcoming Adobe Lightroom 4 with a reader of this blog. One of the questions that came up was: "What are XMP files?"

That's an interesting question about a subject that's often overlooked. Many Lightroom users, in fact, need not concern about this very often. However, they're a great tool you should know about since they can dramatically improve your user experience and also show the way to a more proficient use of Lightroom (as well as any other programs understanding this kind of files).

What Is XMP?

XMP stands for eXtensible Metadata Platform, a standard proposed by Adobe Systems, Inc. to define, create, store and process extensible file metadata.

Many file formats already contemplate some mechanisms to store metadata, even forms of extensible metadata. The advantage of being able to embed XMP data into a file is a huge leap towards standardization, since you will be able to take advantage of it in different applications. Also, compliant applications will simply ignore the content they can't handle.

At times, however, embedding XMP data into an existing file isn't an option: that is the case, for example, when dealing with proprietary RAW file formats (such as Nikon's NEF). You can obviously save XMP data for such a file too, but you'll have to use a so-called sidecar file, that is, a distinct XMP file provided alongside the original file. This solution is more cumbersome and error prone, but some people will argue that it has its own advantages such as leaving the original file unchanged, thus avoiding potential file corruption.

Why Is This Important to Me as a Photographer?

For all your practical purposes, Lightroom is able to save all the information provided by the user during the development process, including Adobe Camera RAW settings, in an XMP-compliant way, either embedded (when the file format allows it) or in a sidecar file.

If your workflow is performed in just one application and one machine, you won't probably benefit so much from this feature. However, as soon as you need to move your images to another application, you will probably face the problem of preserving adjustments (which can be thought of as metadata themselves).

Even a simple scenario such as test driving another program (such as the upcoming Lightroom 4) or moving the files to another machine will require you to migrate metadata as well. If your application isn't storing metadata in the source file itself, or if you choose not to do so, you will need to export it and make it available to the target application.

Migrating Images And Metadata

The simplest scenario is migrating an entire Lightroom catalog from a Lightroom instance to another, as in the case of a Lightroom upgrade. Since Lightroom stores the information of all your photos into the catalog itself, which act as a centralized database, it's sufficient to perform a copy of it and simply import it into another Lightroom instance. This operation is straightforward and no further option will be required for you to preserve all your metadata.

Even if you need to migrate only a subset of your catalog to another Lightroom instance there's an easy way to go. Select the images you want to migrate and choose the File/Export as Catalog menu item. Lightroom will generate a brand new catalog for you to migrate:

Export as Catalog Dialog Box

Things get slightly more complicated when you need to move images to a different application and metadata cannot be embedded into the original file. In this case, you have to tell Lightroom to generate the relevant XMP sidecar files for the images to be migrated and move them alongside the originals. You've got two choices:
  • Have Lightroom alway save an XMP sidecar file for every photo.
  • Have Lightroom generate on the fly XMP sidecar files only for the photos you need to migrate.

Unless your requirements justify the choice, I would always choose the second option since the first will introduce a significant, and often unnecessary, overhead.

To have Lightroom always generate XMP sidecar files, you need to select the Automatically write changes into XMP option in the Metadata tab of the Catalog Settings window:

Catalog Settings Window

Since I always leave this option off, as you can see in the previous screenshot, Lightroom is warning me that "changes made in Lightroom will not automatically be visible in other applications". If you select it, Lightroom will generate a sidecar file for every photo you process and keep it in sync.

To generate an XMP sidecar file on the fly, all you need to do is using the Metadata/Save Metadata to Files menu item (or use the Command-S shortcut if you're on a Mac).

Either way, Lightroom will generate the relevant XMP files alongside your original images, as you can see in the following screenshot:

XMP Files Together With Their Originals

No matter how you decide to generate them, you can now take your images and use them into another application. All the metadata that the target application is able to understand will be processed and the corresponding changes reapplied.

This mechanism is interesting also in the case you're often working with more than one computer. You could keep the catalog in sync, as explained in another post, or just keep XMP files in sync. Once you move an XMP file, just put it alongside the original and have your application read the metadata from it. In the case of Lightroom, all you have to do is using the Metadata/Read Metadata from Files command.

You could argue XMP metadata is easier to backup than an entire Lightroom catalog. Well, I bet it is. However, you Lightroom catalog is more than that and chances are you're backing it up regularly as well, thus defeating this potential advantage. 

Is There a Simpler Way?

This procedure is pretty simple and many XMP compliant applications (such as Lightroom, Camera RAW and Photoshop) will do the heavy lifting for you. Provided the XMP file is distributed with the original, they won't even ask you anything: they'll just read the metadata and apply the relevant settings.

As explained in the previous sections, you rely on XMP files in two cases:
  • You cannot modify the original image and embed metadata into it.
  • You don't want to modify the original image.

If you want to preserve your original files, there's no other option but using sidecar XMP files. If you cannot modify the original image (probably because it's written in a proprietary RAW format) but would willingly have Lightroom modify it if it could, you should consider converting your images to the DNG format (either during the import phase or afterwards with a simple export operation).

DNG stands for Digital Negative and it's a file format from Adobe, Inc. whose purpose, amongst others, is standardizing a RAW format and enable multi-vendor interoperability. However, DNG is much more than that:
  • It's based on TIFF.
  • It's both a container of RAW images and processed images.
  • Metadata can be embedded in a variety of formats, including XMP and EXIF.

Some camera manufacturers already support on-camera DNG. If your camera does not, you can use a converter (such as Lightroom) to generate a DNG file from a proprietary RAW file. You can even choose to embed the original RAW file into the DNG, which is something I would recommend you do since sometimes some proprietary RAW metadata might be lost during the conversion.

Since XMP metadata can be embedded into a DNG file and Lightroom will keep it in sync, you will need neither no sidecar XMP file to migrate your photos. To some extent, you won't even need to rely on the very Lightroom catalog itself.

If you want to help me keep on writing this blog, buy your Adobe Photoshop licenses at the best price on Amazon using the links below.

Sunday, February 5, 2012

Fixing Mac OS X Lion's SSH UTF-8 Issues

After switching from Mac OS X v. 10.6 (Snow Leopard) to OS X v. 10.7 (Lion), I began experiencing some weird locale-related problems when using OS X as an SSH client. The issues I experienced include, amongst others:
  • Having the terminal complain about invalid locale settings:

$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory 

  • Being unable to correctly type some characters (such as accented characters).
  • Having programs that tunnel their traffic over SSH, such as Subversion, complain about invalid locale settings during their operations:

$ svn co svn+ssh://user@repository/path
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is UTF-8
svnserve: warning: please check that your locale name is correct
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is UTF-8
svnserve: warning: please check that your locale name is correct

In fact, if you look for example at the output of the locale command seen previously, you can see how the LC_CTYPE variable is set UTF-8 while all other LC_* variables (excluding the empty LC_ALL), are set to en_US.UTF-8 (which are correct according to that server settings).

What is the SSH client shipped with Lion doing? Apparently sending incorrect values to the server. It's not just a nuisance having to fix them after logging in, provided you're able to do so: this behaviour is breaking other programs tunnelling their traffic over SSH, such as Subversion, that aren't establishing an interactive SSH tunnel for the user.

Fortunately, the fix is quick: just tell the SSH client not to send any LC_* variable:
  • Open the SSH client configuration file (/etc/ssh_config).
  • Comment out the line that instructs the SSH client to send those variables (SendEnv LANG LC_*) from the general host configuration section (Host *). You should end up with the following fragment:

Host *
# SendEnv LANG LC_*

The configuration change will be immediately applied to any SSH session opened from now on.

Hope this helps.

Thursday, February 2, 2012

Book Review: JIRA Development Cookbook, by Jobin Kuruvilla

I just finished reading Jobin Kuruvilla's JIRA Development Cookbook and I'm pretty pleased with it. In my opinion, that's a book worth buying if you want to get serious about JIRA customization and plugin development. I admit there's plenty of information in the Internet about the subject, especially from Atlassian itself, and possibly no question of yours would remain unanswered. However, having a good book like this on your desk and using it as sort of a reference guide is a very good idea.

The Book's Structure

If you read about my first impressions, you'd realize that reading it took me more than a month. Beware: I read it in its entirety because I wanted to review it but this book, as many technical books out there, is not meant to be read from beginning to end as you'd do with a novel. Instead, you should read it diagonally to be aware of what it offers and to be able to quickly get the information you need.

The book is organized into 11 chapters, each of which is a walkthrough of the addressed subject. The typical chapter structure is "programmer friendly": it starts with a high level overview of the subject, analyzing JIRA architecture and internals when deemed necessary, after which it goes on exploring the subject making extensive use of task-lists and code examples. If you already master a subject, more often than not you will end up quickly scrolling the task list and jumping directly into the code. In the following screenshots you can see some excerpts of the book to familiarize with its structure.

A "Getting Ready" Task List

A Summary of What You're Going To Do

Explanatory Section

In Depth Discussion

The first two chapters, Plugin Development Process and Understanding Plugin Framework, offer an overview of the common tasks and procedures you'll undergo during your JIRA plugin development, including the installation of the development environment, the development, debugging and testing phases of your code and the deployment of your deliverable into a JIRA installation.

The third and the fourth chapters, Working with Custom Fields and Programming Workflows, explore some of the most important JIRA customizations you'll deal with when writing your plugins or even just extending the basic platform's functionality: custom fields and workflows.

The fifth chapter, Gadgets and Reporting in JIRA, is dedicated to gadget development and the extension of JIRA's reporting capabilities. The title is somewhat misleading, since this chapter also somewhat deals with JIRA's integration with other systems (at least from a gadget's standpoint), as a consumer of RESTful web services. Another chapter is entirely dedicated to this subject (Chapter 9) and the inclusion of this material in Chapter 5 could pass unnoticed, if not somewhat misleading.

The sixth chapter, The Power of JIRA Searching, examines the techniques you can use to deal with and extend the JIRA query language, JQL, and how to take advantage of it in your plugins.

The seventh chapter, Programming Issues, describes how issues can be managed programmatically. Issue management is extensively covered, discussing subjects such as issue creation and lifecycle management, issue operations, time tracking, issue linking and validation.

The eighth chapter, Customizing the UI, explores how the developer can customize the user interface of JIRA, including customizing and extending the default JIRA pages or creating your own web sections and items.

The ninth chapter, Remote Access to JIRA, is devoted to JIRA integration with external systems using SOAP web services, RESTful web services and, to a lesser extent, JAX-RPC web services. The first part of the chapter introduces the JIRA web service API and how to build a SOAP client. The second part describes how you develop your own web services and deploy them as part of your JIRA installation.

The tenth chapter, Dealing with a Database, instructs the developer in the available tools and how to use them to interact with the database JIRA is connected to. The chapter teaches developers how to programmatically retrieve important JIRA data from the database, such as issue, field and workflow information, and how to extend its schema to hold plugin-specific data.

The eleventh and last chapter, Useful Recipes, gathers interesting information and additional code samples to fulfill common tasks not strictly related with the content of the other chapters.

A Book for Programmers

I defined this book as "programmer friendly" because source code examples are often used as the mean to exemplify how JIRA works internally. Explanations are indeed placed after task lists and code samples, instead of before. In fact, to cite the very author, the book seems to be organized as a cookbook: every section is a recipe with ingredients (prerequisites and task lists), a quick procedure (the how-tos) and, only then, in depth discussions and explanations of the JIRA internals. I'm not a big fan of this kind of content organization but I do admit that, in such a case and for such a subject, it's an effective mean to organize and clarify the content, undoubtedly at the advantage of the reader himself.

Likewise, the book covers a great portion of a plugin development workflow. It's not, and probably cannot be, exhaustive, but rest assured that it will effectively guide you through your development of most, if not all, your plugins, since the vast majority of the APIs and the technologies that you're going to need are covered.

The contents are clearly exposed and are well detailed and I think it's a valuable addition to the official JIRA documentation that's worth having if you're developing on JIRA as a platform. Some subjects that are somewhat less covered in the official JIRA documentation (at least so far), such as UI customization and JQL function customizations, are extensively detailed and are, on their own, a valid reason to own this book.

The only thing I'm missing is a better typesetting: a book should be judged by its contents, and this is what I'm doing, but it's important to recognize that an effective presentation is a key factor to take into account. As a reader, I find the quality of the book typesetting (title formats, paragraph styles, etc.) to stand below the quality of its contents. Of course, it's better like this than the other way round, but I'd appreciate some redesign should this book be revised in the future.


I felt compelled to write a review of this book because I think it deserves it, it's a must-have for a JIRA plugin programmer. Before writing this, I thought I'd write a more in-depth review. Surprisingly, I discovered that enumerating each and every subject this book covers was not necessary: it's a homogeneous, extensive and well organized book.

From a developer standpoint, it's an effective and efficient mean to learn the subject and get the job done, much better than any tutorial I've seen so far. Together with the official Atlassian JIRA documentation, this book is pretty much all you need.

Title: JIRA Development Cookbook
Author: Jobin Kuruvilla

Pages: 476
Publisher: Packt Publishing
ISBN-10: 1849681805
ISBN-13: 978-1849681803