An @ Symbol in OS X File Permissions and the Problems That Came

Today I was deploying some software updates in our Solaris 10 servers and, since such updates are distributed in files archived with the GNU tar utility (not shipped by default in Solaris 10), I had to convert them before moving them to our servers. I'm quite accustomed to such a procedure: that's why (amongst other more important reasons) I always have an OpenSolaris virtual machine running in my other desktops (thanks to VirtualBox).

Today, notwithstanding the fact the I had a gut feeling that I was not doing the correct thing, I decided to open a Mac OS X terminal and do the job from there. GNU tar and pax are there, aren't they? There should not be any problem, thought I.

Well, I should have paid attention to that gut feeling of mine: to spare the required OpenSolaris boot, I lost much more than what I had planned. I learned something new but... what was it worth? A bunch of non portable behaviours of Mac OS X? Just the confirmation I needed, once again (I realize I'm a stubborn fellow): if you can, do not rely on a Mac to do your UNIX-ish tasks. Chances are you're going to waste more than you believed.

This is what happened.

Making a pax File for a Solaris Host

This was meant to be a quick task. I copied the archive from a Solaris NFS mount where, some days ago, I had stored the file. I ran gunzip to decompress the file, ran tar to unpack the files, ran pax to store the resulting directory in a pax file and used rsync to quickly transfer the resulting archive back to the target Solaris machine:

$ gunzip [file.tar.gz]

$ tar xf [file.tar]

$ pax -w -f ./[file.pax] [dir]

$ rsync -vz --inline [file.pax] [myserver]

I then connected to the target server and used pax to explode the transferred archive. Since one of the things I feared was Mac OS X messing around with file permissions, I quickly checked the results on the Solaris side and everything looked fine. Well, sort of...

The next thing I usually do with such bundles is checking the differences between the running version and the soon-to-be-deployed one to be sure I won't miss any customization I made in the instance to be patched. It usually is another easy job since very few XML files are usually involved. Great was my surprise when diff began to write out hundreds of differences between one version and another.

Doh! It seemed as though somewhere in between somebody had decided to duplicate each file following a naming pattern; if the expected file was named x, I was seeing two files in Solaris: x and ._x. Where did these file came out from?

Back to the Mac

The original directory had been deleted just after producing the pax file so that the first I did was using pax to unpack the archive in the Mac. As I told you, everything looked fine. Except for an @ character that should not be there:

drwxr-xr-x@ [...snip...]

Maybe you don't know about it but that means that such a file has extended attributes set. Extended attributes are basically metadata: the file system does not interpret them and the ls command uses the @ symbol after the file permissions to inform you that they are there.

Strange Things Are Happening

There's nothing strange about extended attributes: it's very important to be aware of their existence to correctly handle them when, for example, you need to do a backup or move files from one file system to another (unless you don't mind losing them.) Solaris' pax and tar, for example, both support archival of extended attributes even if, by default, they ignore them. Since I know that my original archive had no extended attribute set in any of its file, I just ran a plain pax to build my archive. Maybe OS X pax defaults to a different behaviour so that I checked pax man page.

Doh! Oddly, OS X pax man page makes no mention at all of extended attributes, notwithstanding the fact that, undoubtedly, it was in fact managing some. Which? In OS X, just run xattr -l to find out.

Quarantine

A xattr told me that the com.apple.quarantine extended attribute was set in the culprit and a quick script confirmed that every file in that directory had that attribute set. According to Apple's documentation, that attribute is set by recent versions of OS X to mark the file as a potential threat. Do you remember when finder kindly asks you if you're really sure you want to open a downloaded file? The quarantine extended attribute is the mechanism OS X programs use to store that information.

How did it make it there?

But wait! I downloaded a tar.gz file and stored in an NFS 4 mounted file system for later processing. The question is:

• How did that attribute was stored in the NFS 4 file system where I stored the file?
• How did that attribute make it into every file stored into the pax archive?

As far as it concerns the first question, NFS 4 does support extended attributes. But again, should it be there, I would have known because I checked it directly in the Solaris host and, moreover, the box that acts as a NFS server runs some cron scripts that checks (and fixes to my taste) ACLs and extended attributes in some of the exported NFS file systems. It was very easy to find out that OS X, when storing such a file in a remote NFS share, instead of using NFS 4 extended attributes it stores them in a separate file named with the pattern mentioned above. I then discovered that the shared NFS file system in question is plagued by ._* files. Everywhere.

As far as it concerns the second question, well... I copied once again the file using the finder and performed the same operations I did before. The resulting files all had the quarantine extended attribute set. Who's the culprit, I do not know. I don't even care. The fact is that OS X is performing this operation with tools such as pax and tar without informing the user and, what's even worse, without even mentioning it in such commands' man pages.

Conclusion

The reasoning above explains why, even if Solaris pax (such as NFS) aren't managing any extended attribute, I did end up with duplicated files which were inserted into the pax file produced with Mac OS X. Probably they aren't going to hurt my deployment but surely they should not be there.

For the sake of curiosity, I checked that removing the ._* files in the NFS share made everything go back to normal.

Nevertheless, since I feel there are many OS X dark corners left for me to discover, I won't be making such an error again. Instead of waiting a couple of minutes for a virtual machine to bootstrap, I ended up wasting an entire hour for a (yet another...) bad documented and non-portable idiosyncrasy of an operating system that brings no added value with it.

Spain Wins the 2010 FIFA World Cup

As a permanent guest in this fantastic country, I cannot but congratulate the Spanish National Football Team for its performance during this World Championship. You really deserved it.

VirtualBox: Using Differencing Disks and Immutable Images

As I promised some time ago, I'm now writing an introductory blog post about a subject that some VirtualBox desktop users probably aren't aware of: differencing disks.

When you configure your guest's storage, you might have noticed a Differencing Disks checkbox in the configuration panel. If you tried to check it, chances are that nothing seems to happen. The VirtualBox GUI, indeed, provides no way to configure disks types and, consequently, not even differencing disks. The only clue that differencing disks are there can be seen if you made a snapshot of a virtual machine disk image. In that case, if you inspect the Storage settings of your virtual machine, you will notice that the attached disk is a "Normal (VDI)" disk type, as shown in the following screenshot:

If you now click the Differencing Disks checkbox, you will see that the attached disk actually is a differencing disk:

If you inspect the Hard Disk list box you will notice that what's really attached to your virtual machine is a differencing disk by UUID:

It won't be difficult to discover that UUID corresponds to the disk that stores the "differencing information" of the snapshot I took:

as can be seen in the file system:

There's a chance, then, that differencing disks are already part of your VirtualBox experience as a hidden "side effect" of taking virtual machine snapshots.

In this blog post we'll discover that immutable images (and differencing disks) are disk image types with its own advantages and we'll explore some use cases you can benefit from them.

Image Write Modes

Every virtual disk image you're using in your guests has a property callled image write mode. The mode can be:

• Normal.
• Write-through.
• Immutable.

Normal Images

The default normal images are what you're accustomed to when running your typical guest. The guest instance can read and write to the image without restrictions and the image can be snapshotted. Snapshot data are written to a differencing image that receives the differences to the original image (or to its parent snapshot.) Normal  images can only be attached to one running guest at a time.

Write-through Images

Write-through images, with respect to normal images, aren't affected by snapshots.

Immutable Images

An immutable images, as its name implies, does not permanently record changes: changes are available only while the machine is running. As soon as it's power cycled, changes are lost and the image its reverted to its original state. As in the case of snapshots, VirtualBox will use a differencing image to record changes to an immutable image. The differencing image is then reset to its original state at guest power cycle. The auto-reset feature of a differencing disk can also be turned off. Immutable images can be attached to as many running hosts as you wish: each guest they'll be attached to will create a new differencing image to store their changes.

Use Cases

Each image type has its own pros and cons:

• Normal images can be easily snapshotted but can't be used by more than a guest at a time.
• Write-through images aren't affected by snapshots: they're ideal to host critical data that won't be compromised by a snapshot restore.
• Immutable images won't ever be compromised: a power-cycle will restore them to their initial state. They can be used by multiple host at a time and are the quickest way to deploy multiple (initially) identical guest instances.

Immutable Image with a Differencing Disk

Using immutable images with differencing disks is one of the quickest way to deploy multiple instances of the "same" virtual machine. Since more than one virtual machine will be sharing the same underlying image you'll also spare a good amount of storage space.

I, for example, had to deploy more than ten instances of a proprietary database: all of the instances could share the same immutable image since all of the instance would use the very same operating system and the same initial data base configuration.

Creating A Virtual Machine

The first step to build your own immutable image is creating a virtual machine. This step is the same as usual:

• I created a virtual machine (with a normal image).
• Installed the operating system.
• Installed the data base.
• Finished to configure the environment with most of the configurations all instances would share.

Cloning the Disk

To clone your hard disk image you've got to use VirtualBox command line interface. Open your favorite shell and use the VBoxManage command to clone your hard disk:

$ VBoxManage clonehd <uid>|<file> <output>

You may check with either VirtualBox GUI, adding your newly created image or with the command line interface, as shown in the following images:

Marking the Image as Immutable

Once you're finished with your virtual machine you've got to mark the image as immutable. Once again, this step can only be performed using VirtualBox command line interface. Open your favorite shell and use the VBoxManage command to mark to change the image type:

$ VBoxManage modifyhd <file> \

  --type immutable

Both VirtualBox GUI and the VBoxManage command may be used to check an image type:

Attaching the Immutable Image to a Virtual Machine

Now you can attach your immutable image to a virtual machine. Every time you attach an immutable image to a virtual machine, VirtualBox will create a differencing disk for you that will store the changes recorded for a specific virtual machine.

As explained in the previous sections, the default differencing disks created by VirtualBox will be reset every time a virtual machine is stopped and restarted. Depending on your use case, you might want changes to be permanent.

Setting an Image Auto Reset Flag to Off

Every differencing disk has got an autoreset flag which, by default, is set to on. To set it to off you have to use the VBoxManage command:

$ VBoxManage modifyhd <uuid>|<filename> \

  --autoreset off

Your differencing disk won't "forget" any more the changes it records.

Conclusion

I hope that this quick walkthrough helps you discover some lesser known functionality of VirtualBox, either if you're using it as a desktop or as a server virtualization platform. Either if you're deploying many images or if you're simply installing a couple of virtual machines for your family, immutable images and differencing disk will help you spare a lot of storage space and, above all, a lot of your precious time.

Further Readings

If you're a Solaris user (at least Solaris 10, OpenSolaris, OpenIndiana, etc.) you will notice that the basic benefit of using immutable images can be achieved using native Solaris technologies such as ZFS snapshots and clones or ZFS deduplication. Using ZFS instead of VirtualBox immutable images is leaner and reduces VirtualBox administration times.

Although not related to VirtualBox, if you're interested at the advantages that the Solaris' ZFS file system can bring even in a deployment of a virtualized desktop environment, these are interesting reads:

• Backing Up ZFS File Systems: An introduction to some ZFS features such as snapshots, clones and ZFS streams.
• Sparing Disk Space With ZFS Clones While Setting Up a Developer Environment For Multiple Users: How ZFS snapshots and clones can help you spare space by avoiding unnecessary data duplication.

Letter From Apple: iPhone's Signal Strength Algorithm Is Flawed.

Yesterday Apple published an open letter regarding the problems that are affecting iPhone 4 and the supposed flaw in the antenna design. Just to quote the “interesting” part of it (trimming down iPhone eulogies and Apple’s declarations of love toward its newly born child):

We have discovered the cause of this dramatic drop in bars, and it is both simple and surprising.

Upon investigation, we were stunned to find that the formula we use to calculate how many bars of signal strength to display is totally wrong. Our formula, in many instances, mistakenly displays 2 more bars than it should for a given signal strength. For example, we sometimes display 4 bars when we should be displaying as few as 2 bars. Users observing a drop of several bars when they grip their iPhone in a certain way are most likely in an area with very weak signal strength, but they don’t know it because we are erroneously displaying 4 or 5 bars. Their big drop in bars is because their high bars were never real in the first place.

“Totally wrong”. I wonder what does that mean.

Anyway, Since I own an iPhone I always complained about dropped calls and pretty inaccurate signal strength estimations shown by the phone. Either I had at least 4 bars or I had none. Even at maximum strength, the phone was dropping some calls. Simple cross checks with other phones I own (connected to the same operator) made me suspect that the iPhone was making way too optimistic signal strength estimations.

I’m glad to hear that they finally found the “solution” to this problem although I believe that such a flaw shouldn't have been discovered years after the first iPhone release. Too few of us were complaining? Many of us were blinded by the supposedly Apple’s infallibility? Apple wasn’t listening?

I don’t know. After the next software update, although signal won’t improve, at least the iPhone won’t be lying to us any longer.