Installing macOS High Sierra from Scratch: Not So Good a User Experience

One week ago a MacBook Pro of a relative of mine stopped booting. The symptoms indicated a hard disk failure, and the diagnosis confirmed it. I ordered a replacement drive on Amazon and I decided to allocate 30 minutes of a bright Saturday morning to fix it. The procedure should have been very simple:

• Create a bootable image of the latest macOS (High Sierra).
• In the meantime, swap the hard disk in the MacBook Pro (a procedure that hardly takes more than 10 minutes on a MacBook with a serviceable hard disk drive such as this one).
• Boot from an external device.
• Launch the installation.
• Go out and run.

Unfortunately, it took at least twice the estimated time and three macOS installation attempts.

Attempt 1: Launching the macOS Installer

The MacBook Pro booted the base system correctly from the external device, but when I launched the installer, it failed with the following message: 

This copy of the Install macOS High Sierra.app application is damaged and can't be used to install macOS.

A quick search confirmed that this error frequently appears since macOS Sierra and the workarounds I found are quite imaginative. Apparently, many people thinks the installer is actually broken and are trying to download it again. Fortunately, it didn't take that much time to fix the problem. A post on the Apple website described the same problem (although on Sierra) and its solution (after several calls to Apple customer support, according to the author): deleting the ./Contents/SharedSupport/InstallInfo.plist from the bootable image should do the trick. And it did.

Attempt 2: Launching the macOS Installer

The second attempt looked promising: the installer now launched correctly. Unfortunately, it failed on the second screen with the following message:

The recovery server could not be contacted.

The first thing I do in these cases is open a terminal and perform basic checks. The second check I made caught the problem: the system was reporting January, 1st 2001 as the current date. Therefore, most probably, the HTTPS connection to the recovery server was failing because the certificate check was failing (chances are the validity period of that certificate appeared to be in the future). 

I fixed the date using the date command and restarted the macOS installer.

Attempt 3: Finding the Hard Disk Drive

The installer was now working, but it couldn't find any disk to install the system. I quitted the installer (for the third time) and opened Disk Utility. To my surprise, Disk Utility couldn't find any internal disk either. Again, a quick search confirmed the obvious: High Sierra's Disk Utility doesn't recognise unformatted internal drives. I opened the terminal again in order to quickly format the drive:

• First of all, confirm the disk is available and its device name:

$ diskutil list

/dev/disk0 (internal, physical):

[...snip...]

• Then, format it to make it available to Disk Utility:

$ diskutil eraseDisk JHFS+ NewDisk GPT disk0

Disk Utility could now see the device. I reformatted it using APFS, restarted the installer again and this time the I successfully made it to the end of the installation.

Final Thoughts

I never needed to reinstall OS X or macOS on any of the three computers I owned in the last 10 years: this was the first macOS installation from scratch I performed in a very long time. However, the three errors I experienced have three characteristics in common which make this experience unsettling:

• They are obscure and difficult to troubleshoot. I bet most non-tech-savvy user would have to ask for help.
• They have been known for years.
• They are, or seem to be, easily solvable. The date problem, for instance: why wouldn't an installer which requires a working Internet connection not synchronise the current time using NTP?

Come on Apple, you can do better than this.

A Shell Script to Delete Old Time Machine Backups and Free Disk Space

In another post I described the procedure to manually delete Time Machine snapshots and optionally shrink the disk image (in case a sparse bundle is being used) in order to free up disk space.

I have written a shell script, tm-cleanup.sh, to automate the backup deletion. The script accepts an optional argument to specify the maximum backup age in days: older backups are deleted (by default, backups older than 30 days are deleted).

The latest version of the script can be obtained by cloning its GitHub repository and, optionally, run its installation script to create symbolic links to it into /usr/local/bin:

$ sudo make install

fswatch 1.4.5 - Release Notes

fswatch 1.4.5 has been released.

New features and bug fixes added since v. 1.4.0 are:

• Add custom record formats.
• Localize fswatch and libfswatch using GNU gettext.
• Add Italian (it) localization.
• Add Spanish (es) localization.
• Fix Makefile.am because of broken link when DESTDIR installs are performed.
• Fix bug in fswatch-run wrapper script for ZSH which caused last argument not to be split when passed to xargs.
• Add batch marker feature to delimit the boundaries of a batch of events.
• Add Texinfo documentation.
• libfswatch API is now versioned.
• Improved Autoconf checks.
• The inotify monitor now waits for events and honours the latency settings.
• Automaticaly generate the ChangeLog using Git.
• Update autogen.sh to honour some commonly used environment variables.
• The inotify monitor now provides the same functionality provided by all the other monitors.
• Recursive directory monitoring is now implemented.
• Version and revision is now determined dynamically from Git by ancillary scripts invoked by the GNU Build System.
• fswatch does not compile on OS X < 10.9 because some required C++11 classes are not supported by the C++ runtime.
• fswatch does not compile on OS X < 10.9 because some required C++11 classes are not supported by the C++ runtime.
• The libfswatch library has been added with bindings for C and C++.
• fswatch let users specify the monitor to use by name.

Shrink Your Time Machine Backups and Free Disk Space

Update: In a newer post, I describe a shell script I published to automate Time Machine backup deletion.

Time Machine is a backup and restore tool from Apple which is very well integrated into OS X. In my personal opinion Time Machine is not yet awesome and its shortcomings often force me to use alternative methods when I need to backup files and folders with certain characteristics.

The most important use case in which I strongly discourage anybody to use Time Machine is the backup of frequently updated big files. Time Machine always copies the whole file because it is not able to transfer only the differences between the previous state of a file and the current one. This fact has many detrimental consequences:

• Backups are slow.
• Backups may take a huge amount of disk space.

When I say huge, I mean enormous. Think about a virtual machine disk stored as a file in your disk: if just one bit of that file changes (and you can bet it changes every time you use your virtual machine), Time Machine will perform a copy of the whole file the next time a backup is run.

Fortunately, there exist tools which are able to examine a file and transfer only the difference with a previous stored state, such as rsync, on which you can rely to build your custom backup policies.

Anyway, no matter how Time Machine has eaten up a lot of your disk space, it may come a moment when you really need to free some of it, possibly deleting old backups and shrinking a Time Machine sparse bundle disk image.

Deleting Old Backups

Old backups can be deleted in many ways, the simplest one being the following:

• Open the Time Machine application when the Finder application is in the foreground.
• Navigate to the backup to be deleted double clicking on the corresponding position of the ruler on the right side of the screen.
• Right click with the mouse on the empty space in the Finder.
• Select Delete Backup.

Time Machine will ask you to introduce the super user password and then it will delete the selected backup.

Unfortunately this method is very clumsy if you need to delete many backups.

Another way is using the tmutil command to perform the deletion. Not only it is simpler, but it will allow you to do it programmatically if you need to. To delete a backup using tmutil you must perform the following operations:

• Make sure your time machine backup disk is mounted (the simplest way to do it is opening the Time Machine application).
• The format of the backup folders created by Time Machine is YYYY-MM-DD-HHmmss. If you want to delete a specific backup, you can use the following command:

$ sudo tmutil delete /full/path/to/backup/Backups.backupdb/machine/backup-name

Time Machine currently stores its backups in a folder named after the backed up machine, into the Backups.backupdb folder in the backup disk. This means that, if your machine is called iMac and your backup disk is mounted on /Volumes/Time Machine Backups, then backups will be located in the following folder:

/Volumes/Time Machine Backups/Backups.backupdb/iMac

If you want to delete the 2014-02-02-123411 backup, you must run the following command:

$ sudo tmutil delete /Volumes/Time Machine Backups/Backups.backupdb/iMac/2014-02-02-123411

You can easily make a script, for example, to delete all the backups of a specific month. The following commands will delete all the backups created in January 2014 (lines were split with \):

$ sudo bash
Password:
$ for i in /Volumes/Time\ Machine\ Backups/Backups.backupdb/iMac/2014-01* ; \
  do \
    tmutil delete "$i" ; \
  done

Deleting Backups of Any Machine

The same commands can be used to delete backups performed by any machine, not only the machine you are executing them from. This is particularly useful if your time machine backup disk is located on a NAS (such as Apple's Time Capsule).

In this case, though, you will have to mount the disk image file created for every machine and named after it. Time machine creates sparse bundle disk images which can be mounted using the Disk Utility application:

• Open the Disk Utility application.
• Choose the File/Open Disk Image... menu item (⌥⌘O).
• Choose the disk image file corresponding to the backup disk of the machine whose backups you want to delete.

Disk Utility will mount the disk image and you can now use the commands shown above to delete the time machine backups stored in this disk image.

Freeing Space

Now that you are able to delete backups, you may notice that no disk space is relinquished if a sparse bundle disk image is being used as backup disk, which is the default behaviour when the backup disk is not a locally attached physical disk.

In this case, when Time Machine frees disk space, the space is freed inside the disk image, but unless the disk image itself frees it as well, the result is that no space is relinquished outside of it.

Fortunately, unallocated space can be freed from a sparse bundle using the following command (make sure the disk image is not mounted when executing this command):

$ sudo hdiutil compact /path/to/disk-image
Starting to compact…
Reclaiming free space…
.................................................................................................................................................................................................................
Finishing compaction…
Reclaimed 11.2 GB out of 118.7 GB possible.

As you can see, hdiutil reports the space which has been reclaimed and you should now see the specified amount of space as additional free space in the corresponding disk.

fsw: a File System Event Monitor for OS X and *BSD Systems

A very often demanded feature by many different types of users is the possibility of receiving an event when a change in a specific file or directory occurs. One of the most common way to consume such an event in a UNIX(-like) system would be:

• Having a command which lets you watch a set of files or directories outputting on standard output a detailed record for each event received.
• Piping such a process to a shell script that reacts accordingly.

Many UNIX and UNIX-like systems provide kernel facilities, subsystems and programs providing the possibility of receiving file system change events:

• Silicon Graphic's portable FAM.
• FreeBSD's kqueue, originated in FreeBSD and available on many *BSD systems.
• Linux inotify.
• Apple OS X FSEvents API.

However, unlike Linux (where inotifywatch is available), neither OS X nor kqueue-enabled *BSD systems provide a user-land program for users to take advantage of those APIs. That's why I decided to fill the gap and write a small C++ program: fsw, a shell-friendly file system event notifier for OS X and kqueue-enabled *BSD systems, hosted on GitHub.

fsw

fsw will let you subscribe to file system change events on multiple files or directories:

$ fsw file-0 ... file-n

When a file system event is received, fsw prints a complete record of the event such as

Sat Feb 15 00:26:30 2014 - /full/path/to/file:created renamed modified changeOwner isFile 

Sat Feb 15 00:26:31 2014 - /full/path/to/file:created renamed modified changeOwner xattrMod isFile

that include the following information:

• The timestamp when the event was received.
• The full path of the changed file or directory.
• The list of event flags, describing the type of change that occurred.

Watchers

fsw implements three kind of watchers:

• A watcher based on the File System Events API, available only on Apple OS X.
• A watcher based on kqueue, an event notification interface introduced in FreeBSD 4.1 and supported on most *BSD systems (including OS X).
• The poll watcher, a watcher which periodically stats the file system, saves file modification times in memory and manually calculates file system changes.

The limitations of fsw depend largely on the watcher being used:

• The FSEvents watcher, available only on Apple OS X, has no known limitations and scales very well with the number of files being observed.
• The kqueue watcher, available on any *BSD system featuring kqueue, requires a file descriptor to be opened for every file being watched. As a result, this watcher scales badly with the number of files being observed and may begin to misbehave as soon as the fsw process runs out of file descriptors. In this case, fsw dumps one error on standard error for every file that cannot be opened.
• The poll watcher, available on any platform, only relies on available CPU and memory to perform its task. The performance of this watcher degrades linearly with the number of files being watched.

Recommendations

• On OS X, use only the FSEvents watcher.
• If the number of files to observe is sufficiently small, use the kqueue watcher. Beware that on some systems the maximum number of file descriptors that can be opened by a process is set to a very low value (values as low as 256 are not uncommon), even if the operating system may allow a much larger value. In this case, check your OS documentation to raise this limit on either a per process or a system-wide basis.
• If feasible, watch directories instead of watching files.
• If none of the above applies, use the poll watcher. The authors' experience indicates that fsw requires approximately 150 MB or RAM memory to observe a hierarchy of 500.000 files with a minimum path length of 32 characters. A common bottleneck of the poll watcher is disk access, since stat()-ing a great number of files may take a huge amount of time. In this case, the latency should be set to a sufficiently large value in order to reduce the performance degradation that may result from frequent disk access.

Configuring the Latency

Events are passed to the watching process via a callback and every callback call pass an array of events. When creating the event stream, a latency is specified and it defines how many seconds d (a double value) elapse between a callback invocation and another. fsw lets users specify the desired latency using the -l/--latency options:

$ fsw -l 5 .

In this case, fsw receives change events every 5 seconds.

Other Options

fsw currently supports the following options:

• -f, --format-time: Print the event time using the specified format.
• -h, --help: Show this message.
• -k, --kqueue: Use the kqueue watcher.
• -l, --latency=DOUBLE: Set the latency.
• -n, --numeric: Print numeric event mask.
• -p, --poll: Use the poll watcher.
• -r, --recursive: Recurse subdirectories.
• -t, --time-format: Print the event time using the specified format.
• -u, --utc-time: Print the event time as UTC time.
• -v, --verbose: Verbose output.

Installing fsw

fsw is a C++ project using the GNU Build System and can be installed on any supported system (OS X, v. >= 10.6 or any *BSD system supported kqueue).

Regular users should download a release tarball, uncompress it, configure it and build it:

$ ./configure
$ make

If you're a developer and have the GNU Build System installed on your machine, you can clone the git repository to get the sources:

$ git clone https://github.com/emcrisostomo/fsw.git

Then, cd into fsw directory, bootstrap the GNU Build System, configure it and build it:

$ ./autogen.sh
$ ./configure
$ make

Finally, no matter you used a release tarball or the repository sources, you can install it system-wide to /usr/local by running the following command:

$ sudo make install

If you cannot install it on /usr/local or you just prefer to install it on a private directory, you can use the following command instead:

$ DESTDIR=/installation/path make install

Documentation

fsw ships with a detailed man page you can consulting using the following command:

$ man fsw

Getting Help or Giving Feedback

If you experience some issues or simply want to give your feedback, please contact me.

Lightroom Users Upgrading to Mountain Lion: Back Up Your Adobe Camera RAW Cache Directory

To improve Lightroom's performance, Camera RAW maintains a cache which speeds up some stages of an image processing. By default, the cache size is set to 1 GB, but you should increase it (as Adobe suggests) to store more image data in order to speed up preview generation of cached images. Since hard disk space is hardly an issue nowadays, I've set it up to 32 GB and disk usage is currently around 8 GB (with some thousands of images in my catalog). The beneficial effects of the cache are easily seen, that's why the cache directory is now included in my standard Lightroom backup.

Some days ago, I updated my Macs to Mountain Lion with the hope of benefitting from its performance improvements. The update process was easy and flawless but as soon as I opened my catalog I started suspecting something was wrong. A quick research confirmed my suspicions: Mountain Lion's update process had completely wiped away the ~/Library/Caches directory which, by default, contains the Adobe Camera RAW cache directory. 8 GB worth of data swept away without even asking: not good and not fair. If Apple wanted to clean its program caches, the installer could have limited to cleaning just those.

Fortunately, I could restore it from my latest backup and I was soon back to work. The bottom line is: if you're upgrading to Mountain Lion, you'd better backup your Adobe Camera RAW cache directory, unless you don't mind Lightroom recreating it from scratch.

If you still don't know your cache size (chances are it's still the default 1GB) or your cache location, you can check it in the Ligthroom preferences (File Handling pane):

Lightroom Preferences - File Handling

You should increase the cache size to increase the number of image data that can be stored: that depends on the number of RAW images in your catalog. In my system, I'm observing an average 1 MB worth of data per image. You can also change the cache location in case you prefer storing it closer to your catalogs for backups' and "visibility's" sake (by default the ~/Library folder is hidden in Finder).

High CPU and I/O activity? Disable Time Machine local backups

Some weeks ago I wrote a blog post about how mdworker can sometimes jeopardize your computer resources. Since then, I experienced an enhancement of my computer's performance. But it was not sufficient.

I'm telling nothing new: I have often read myself how disabling Time Machine local backups as well can have beneficial performance effects. The interesting part is how I realized.

I'm using OS X on two machines: a MacBook Pro and an iMac. The iMac was restored from a Time Machine backup of the MacBook Pro and:

• I'm using the same applications on them.
• I'm using the same applications data on them (synchronized using rsync).

This means that, at least as far as applications and the operating systems are concerned, they cannot have diverged so much over time.

Interestingly, though, the iMac performs much better than the iMac, even if the latter has got half the quantity of RAM the former has (8 GB).

I always had issues with Time Machine, mainly because I'm working on huge files and Time Machine does not perform incremental, or delta backups: it backs up the entire file each time it runs and detects a change. In fact, I spend most of the time using virtual machines (for development purposes), Lightroom catalogs and... Apple Mail: all the working files' size of these application is in the gigabyte order of magnitude).

What I hadn't realized (yet) is the huge performance impact of Time Machine local backups. On a daily basis I'm changing gigabytes of data on my hard disk and, as a consequence, Time Machine local backups were generating a storm of CPU and I/O activity. The impact was visible only on the MacBook Pro since Time Machine local backups are enable by default only on laptops. Once I disabled them, I had a huge performance improvement (and, as a consequence, freed up a great deal of space in my hard disk).

If you're running OS X Lion on a laptop, chances are local backups are enabled. If they are, you'll see the fancy .MobileBackups directory in your hard disk root. To disable local backups, just run the following command:

$ sudo tmutil disablelocal

After rebooting, OS X Lion will transition the .MobileBackups folder to .MobileBackups.trash and finally delete it. Depending on its actual size, it will take a certain amount of time. In my case, begin bigger than 100 GB, it took approximately five minutes.

Is mdworker Jeopardizing Your Resources? Run It at a Lower Priority

Since I've installed OS X Lion, I've been complaining about the performance degradation I was observing. More often than not, I find the guilty to be mdworker, the (in)famous Spotlight's indexer process, which may use lots of CPU cycles and generate lots of IOPs. Perhaps, unless you're using an SSD, the hard disk is going to be the bottleneck in many situations. Even if it's not, I've seen mdworker using an entire CPU core for entire seconds: it's a lot of computational time, and even a human being is going to notice the performance degradation of his machine when mdworker stubbornly keeps on using CPU time over and over again.

It's important to realize that the job of mdworker is observing changes in the file system and reflect those changes into Spotlight's content indexes. That's not the problem. The problem is that, by default, mdworker appears to be running with the same priority and I/O priority than most your user space processes. Depending on your computer usage pattern, you may or may not observe any performance degradation caused by mdworker: in fact, I bet most users won't notice it, but when you do, it's a real pain.

In my case, I notice a huge performance degradation when using programs that can generate massive changes in the file system in a short period of time, such as Adobe Lightroom or Apple Aperture. I found myself watching the Lightroom window freezing so often, waiting for mdworker to finish its job and relinquish some resources, that I decided to look for a solution to this problem.

Fortunately, there's a quick workaround that works pretty well and addresses the core issue with mdworker: its priority.

OS X Tiger (10.4) introduced a service management framework, launchd, that essentially replaced a bunch of legacy service handling daemons such as init, inetd, xinetd, and all the related scripts. launchd centralizes service management and, as such, it also centralizes service configuration using a property list files. If you're new to OS X development, property list essentially are XML files using a schema defined by Apple.

If you want to customize the behavior of an OS X service, you can just modify the corresponding configuration file without any deeper knowledge of the service behavior. You can read the launchd.plist official documentation for a list of valid configuration keys.

Although we're focusing on mdworker here, the concepts herein can be applied to any service you'd like to configure.

Modifying a Service Configuration File

Service configuration files are currently stored in two directories:

• /System/Library/LaunchAgents
• /System/Library/LaunchDaemons

First of all, you've got to locate the configuration file of the service you want to customize. Open the file in your favorite XML editor and carefully modify it. An invalid property list file will result in launchd rejecting the invalid configuration and the corresponding service failing to start.

With the release of XCode 4, the Property List Editor is no longer available as a separate application and all the editing is performed inside XCode. If you've got XCode, it's very easy to modify property list files using its GUI. If you haven't got XCode, it's a bit overkill to install such a huge application only to modify a property list file. I'd rather use an alternative XML editor and carefully review the changes before applying them.

In any case, I'm including my entire configuration file in this blog post so that you can just copy it over yours.

Reducing the mdworker Priority

The configuration file of mdworker (the Spotlight's indexer) is:

/System/Library/LaunchDaemons/com.apple.metadata.mds.plist

As we can see in launchd.plist documentation, we can take advantage of two configuration keys to lower the working priority of this process:

• LowPriorityIO
• Nice

For CPU-intensive processes, Nice can be the solution, since it specifies the service's scheduling priority. The higher the nice level, the lower its priority. Nice can take values in the [-20, 20] range.

Since mdworker is also a I/O-intensive process, we will use the LowPriorityIO configuration key as well. This configuration key takes a boolean value and, when set to true, it instructs the kernel to consider the process as low priority when performing file system I/O. That's just what we need to avoid mdworker jeopardizing our resources when using I/O intensive processes such as Adobe Lightroom.

The resulting configuration file is the following:


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>KeepAlive</key>
  <true/>
  <key>Label</key>
  <string>com.apple.metadata.mds</string>
  <key>ProgramArguments</key>
  <array>
    <string>/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Support/mds</string>
  </array>
  <key>MachServices</key>
  <dict>
    <key>com.apple.metadata.mds</key>
    <true/>
    <key>com.apple.metadata.mds.xpc</key>
    <true/>
    <key>com.apple.metadata.mds.xpcs</key>
    <true/>
  </dict>
  <key>SoftResourceLimits</key>
  <dict>
    <key>NumberOfFiles</key>
    <integer>2048</integer>
  </dict>
  <key>HardResourceLimits</key>
  <dict>
    <key>NumberOfFiles</key>
    <integer>2048</integer>
  </dict>
  <key>LowPriorityIO</key>
  <true/>
  <key>Nice</key>
  <integer>20</integer>
</dict>
</plist>

Here's the same file as seen in XCode:

XCode 4 - Property List Editor (Integrated)

Conclusion

The next time the service restarts the new configuration will take effect. The easiest way to do it is just reboot. mdworker will now work as usual, but won't steal your resources as aggressively as before.

You can check that mdworker is now running with a lower priority examining ps output (6th and 7th column):

$ ps -axl | grep mdworker

  UID   PID  PPID        F CPU PRI NI       SZ    RSS WCHAN     S             ADDR TTY           TIME CMD

  501   234   232    84004   0   4 17  2485976  25916 -      SN   ffffff800f0bbdc0 ??         3:56.98 /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker -s mdworker -c MDSImporterWorker -m com.apple.mdworker.pool.0
   89   881   879    84004   0   4 17  2479740   6420 -      SN   ffffff800fd84200 ??         0:00.14 /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker -s mdworker -c MDSImporterWorker -m com.apple.mdworker.pool.0

Readability: An Alternative to Safari Reader for Google Chrome Users

The title of this blog post is intentionally misleading: Readability, in fact, was the project that "inspired" the Safari Reader. Nevertheless, many OS X users start using Safari as their first browser and, when switching to other browsers such as Google Chrome, tend to thinking that Safari Reader was the first implementation of such an idea.

Google Chrome has been experiencing a rapid growth since its inception and I recognize it's a great browser that shines, above all, for its speed. However, I never really switched away from Safari for the same reason I haven't switched away from iChat, iCal or Mail, either. Which one? Because of its tight integration, even across multiple Apple devices.

However, running multiple browsers side by side poses no problems and it's pretty common nowadays (most web developers are compelled to do so while testing their applications). Despite being a strong Safari user, Chrome is the browser I use while accessing Google's own services (such as Google+, GMail, Calendar and Blogger).

When Safari Reader made its appearance, I wasn't really impressed and wondered whether I would really use it. I was wrong. I find it's very useful. Not only to actually read a cleaned-up version of a web page (many websites already provided a specific CSS for printing that you can use for this purpose). Safari Reader lets you convert its output to PDF (using the standard OS X print dialog) or send that PDF by email with just one click.

Safari Reader - Toolbar

When I started using Google Chrome, the Reader was the feature I missed the most.

Readability

Readability is a Chrome extension that provides Reader-like features to Google Chrome. In fact, as stated earlier, Readability is the project that inspired Safari Reader (sometimes, though, inspirers are more obscure that fine-tuned copies). To install readability, just open the Chrome Web Store, search for Readability, and install it:

Chrome Web Store - Readability Application

Once you install it, it will be immediately available without even restarting your browser.

You can access the Readability features by using its icon in the Chrome toolbar or using one of its keyboard shortcuts (more on this later).

Chrome Toolbar - Redability icon

Clicking the Readability icon will open its main menu and you soon realize that it offers more than the Safari Reader does: you can Read Now (the equivalent feature of Safari Reader), you can Read Later (Safari also offers this: it's called Reading List) and you can Send to Kindle.

Redability Menu

In this post we will only focus on the Read Now feature. Here's the same New York Times articles as shown by Readability:

Readability

In my opinion, apart from correctly displaying far more pictures than Safari Reader does (as well as some ad too), Readability produces better-looking previews of most web pages I visit.

The features that the "preview" window offers are superior to Reader's:

• You can print the article (and produce PDF files).
• You can tweak the page appearance.
• You can send the converted article by mail.
• You can share it on social networks (Twitter and Facebook are currently supported).
• You can produce a short URL to it.

Small Glitches on OS X

I don't know how many OS X are really affected by this issue (given their historic idiosyncrasy to using the keyboard :), but Latin languages speakers (such as Italian and Spanish) surely are.

First of all: the OS X keyboard shortcut to cycle between the windows of an application is ⌘-` (Command-`). Guess which is the default keyboard shortcut that Readability assigns to some of its functions? Yes, you're right. The ` character.

I really don't know what the developers were thinking about when they chose this character. They surely aren't Latin languages speakers, otherwise they would have devised how often this character is used. And they surely aren't Mac users either, otherwise they would have realized how annoying it is, too.

Don't worry, though. They're really easy to change. Just right-click over the Readability icon and choose Options. In the configuration window, just assign the keyboard shortcuts you like or do as I did: disable them all.

Readability - Keyboard Shortcut Configuration

Conclusion

This blog post is not about which browser you should use: it's your choice and, as I stated, you can run multiple ones side-by-side. However, many Safari users switching from Safari often look to alternatives to the Safari Reader as well.

If you're using Chrome, you're lucky because the Readability extension provides the same features provided by the Safari Reader (and even more). Moreover, in my opinion the layouts produced by Readability are superior and far better looking than those produced by Safari Reader. You'll be surprised.

Fixing Mac OS X Lion's SSH UTF-8 Issues

After switching from Mac OS X v. 10.6 (Snow Leopard) to OS X v. 10.7 (Lion), I began experiencing some weird locale-related problems when using OS X as an SSH client. The issues I experienced include, amongst others:

• Having the terminal complain about invalid locale settings:

$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory 
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE=UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8" 
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8" 
LC_TELEPHONE="en_US.UTF-8" 
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

• Being unable to correctly type some characters (such as accented characters).
• Having programs that tunnel their traffic over SSH, such as Subversion, complain about invalid locale settings during their operations:

$ svn co svn+ssh://user@repository/path
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is UTF-8
svnserve: warning: please check that your locale name is correct
svnserve: warning: cannot set LC_CTYPE locale
svnserve: warning: environment variable LC_CTYPE is UTF-8
svnserve: warning: please check that your locale name is correct


In fact, if you look for example at the output of the locale command seen previously, you can see how the LC_CTYPE variable is set UTF-8 while all other LC_* variables (excluding the empty LC_ALL), are set to en_US.UTF-8 (which are correct according to that server settings).

What is the SSH client shipped with Lion doing? Apparently sending incorrect values to the server. It's not just a nuisance having to fix them after logging in, provided you're able to do so: this behaviour is breaking other programs tunnelling their traffic over SSH, such as Subversion, that aren't establishing an interactive SSH tunnel for the user.

Fortunately, the fix is quick: just tell the SSH client not to send any LC_* variable:

• Open the SSH client configuration file (/etc/ssh_config).
• Comment out the line that instructs the SSH client to send those variables (SendEnv LANG LC_*) from the general host configuration section (Host *). You should end up with the following fragment:

Host *

# SendEnv LANG LC_*

The configuration change will be immediately applied to any SSH session opened from now on.

Hope this helps.

How To Disable Time Machine Local Backups (and Relinquish Space in the MobileBackups Folder)

This morning, after importing a bunch of RAW files from my Nikon camera, I was startled (once more) by the lack of responsiveness of my MacBook Pro running OS X Lion.

Since the update, in fact, I'm growing pretty disappointed with Lion. It never roared just like the good ole Snow Leopard. The Lion cub has only shyly purred and whined, so far.

One of the things that bothers me most is Spotlight performance. Assuming you noticed it, have you ever wondered what the mdworker process is doing when it jeopardizes your Mac resources? I have: staring impotently at the screen, listening to the spinning rust crackling, powerlessly witnessing how the operating system is waiting to complete who-knows-what.

This morning, I opened Lightroom, started importing some RAW files, and started working on them. Well, trying to work on them while Lion was performing its... "housekeeping".

Frustrated, I expressed my anger on Google+. A friend of mine, Joaquin, pointed out both the cause and a workaround of the new Lion's behaviour.

It turns out that Lion is performing periodic local backups that will be flushed upon your time machine disk when connected. Thinking about versions, aren't you? Anyway, that's also why the new /.MobileBackups folder into your OS X hard disk is eating up disk space. Never noticed purple and white snapshot lines in Lion's Time Machine user interface? White ones are local backups, purple ones are "regular"time machine backups on an external disk. Well, local backups are kept for 8 days and they'll be taken only if you've got enough space on your disk. However, if you, like me, manage huge quantities of big files (such as a camera RAW files), this feature can start eating up an enormous amount of space.

If you, like me, feel like turning off this amazing feature you can use the following command:

$ sudo tmutil disablelocal

Should you change your mind about it, you can always turn it on again using:

$ sudo tmutil enablelocal

These are just the kind of options that should be accessible in the user interface.

Hope this helps you.

Multiple Icons of the Same Application in Launchpad

One of the new features introduced by OS X Lion is the Launchpad. The launchpad gives you access to the installed applications in your Mac from a window that is reminiscent of iOS devices. In fact, it superimposes on your current desktop window, with a fancy blur effect, and a list of application icons is presented in a rectangular grid. You can even make folder in it by dragging an icon over one other. It seems an iPhone screen on steroids.

In my humble opinion, it's awful.

Anyway, one day, using a wrong gesture, the Launchpad screen was launched and I noticed that the Skype icon was presented twice. Strange, thought I. I tried to remove one of them, but they could not be deleted. I tried dragging one into the trash. It didn't work. To make the story short, I did not lose time investigating this weird Launchpad behavior: it's the kind of window I prefer not to see at all.

Some weeks after Skype informs me that an update is available: I accept to install it immediately and wait for the installer to finish. Curiosity killed the cat: I opened the Launchpad and guess what? Another brand-new Skype icon had just been added to the long row of Skype icons already present in the Launchpad window.

A quick glance confirmed what I already knew: there was just one Skype application installed in my Mac and no other link to it was present whatsoever that might have gotten Launchpad confused. I completely removed Skype from the system and reinstalled it from scratch. No way.

It was just time to act and stop the uncontrollable icon proliferation. Since every user can customize Launchpad, I thought that a good place to start was the ~/Library folder. I grep-ped every file into it looking for the "Skype" string (the icon title in the Launchpad) and a file caught my attention:

~/Library/Application Support/Dock/FABCDB4F-FBDE-42E4-A14F-80FA7476989C.db

An SQLite 3 database file (as confirmed by the file command)! It looks like the potential offender, doesn't it? I then backed up the file, removed it, logged out and logged in again. I started the Launchpad and bingo! Launchpad had just regenerated the database file and now only one Skype icon was shown in the Launchpad window.

As I said, I'm not going to use again. But at least I can rest assured that icons are gone away. Hope this can help you guys if you run into similar troubles with Launchpad.

A Shell Script to Find and Remove the BOM Marker

Edited:

As pointed out by Omri, the script is failing on OS X apparently because of an idiosyncrasy in Apple's sed implementation. I temporarily fixed the script switching from sed to perl on OS X: perl is also shipped by default on OS X so there shouldn't be any problem. However, on OS X this version of the script scans by default the entire file, and not only the first line as it does with other sed implementations.

Introduction

Have you ever seen this characters while dumping the contents of some of your text files, haven't you?



If you have, you found a BOM marker! The BOM marker is a Unicode character with code point U+FEFF that specifies the endianness of an Unicode text stream.

Since Unicode characters can be encoded as a multibyte sequence with a specific endianness, and since different architectures may adopt distinct endianness types, it's fundamental to signal the receiver about the endianness of the data stream being sent. Dealing with the BOM, then, it's part of the game.

If you want to know more about when to use the BOM you can start by reading this official Unicode FAQ.

This post has been modified to solve some problems and improve the script according to your comments:

• Solved some mktemp inconsistencies across UNIX flavours (such as Solaris and Mac OS X).
• Files can now be filtered by extension using the -e option, as suggested by Goldan.
• BOM can be removed throughout the file using the -a option, as suggested by Goldan.
• An arbitrary number of files can be safely passed as a parameter.
• The script behaves correctly even with filenames with whitespaces in it.

Safe Harbour Statements: I try to test the script in the greatest number of systems but I'm not guaranteeing that it is working correctly on your. I'll be glad if you give me your feedback: any suggestion or bug report will be appreciated.

UTF-8

UTF-8 is one of the most widely used Unicode characters encoding on software and protocols that have to deal with textual data stream. UTF-8 represents each Unicode character with a sequence of 1 to 4 octects. Each octect contains control bits that are used to identify the beginning and the length of an octect sequence. The Unicode code point is simply the concatenation of the non control bits in the sequence. One of the advantages of UTF-8 is that it retains backwards compatibility with ASCII in the ASCII [0-127] range since such characters are represented with the same octect in both encodings.

If you feel curious about how the UTF-8 encoding works, I've written an introductory post about it.

Common Problems

Because of its design, the UTF-8 encoding is not endianness-sensible and using the BOM with this encoding is discouraged by the Unicode standard. Unfortunately some common utilities, notably Microsoft Notepad, keep on adding a BOM in your UTF-8 files thus breaking those application that aren't prepared to deal with it.

Some programs could, for example, display the following characters at the beginning of your file:



A more serious problem is that a BOM will break a UNIX shell script interfering with the shebang (#!).

A Shell Scripts to Check for BOMs and Remove Them

The Byte Order Mark (BOM) is a Unicode character with code point U+FEFF. Its UTF-8 representation is the following sequence of 3 octects:

1110 1111 1011 1011 1011 1111
E    F    B    B    B    F
The quickest way I know of to process a text file and perform this operation is sed. The following syntax will instruct sed to remove the BOM from the first line of its input file:

sed '1 s/\xEF\xBB\xBF//' < input > output

A Warning for Solaris Users

I haven't found a way (yet) to correctly use a sed implementation bundled with Solaris 10 to perform this operation, neither using /usr/bin/sed nor /usr/xpg4/bin/sed. If you're a Solaris user, please consider installing GNU sed to use the following script.

The quickest way to install sed and a lot of fancy Solaris packages is using Blastwave or OpenCSW. I've also written a post about loopback-mounting Blastwave/OpenCSW installation directory in Solaris Zones to simplify Blastwave/OpenCSW software administration.

A Suggestion for Windows Users

If you want to execute this script in a Windows environment, you can install CygWin. The base install with bash and the core utilities will be sufficient for this script to work on your CygWin environment.

Source

This is the source code of a skeleton implementation of a bash shell script that will remove the BOM from its input files. The script support recursive scanning of directories to "clean" an entire file system tree and a flag (-x) to avoid descending in a filesystem mounted elsewhere. The script uses temporary files while doing the conversion and the original file will be overwritten only if the -d option is not specified.



#!/bin/bash


set -o nounset
set -o errexit


DELETE_ORIG=true
DELETE_FLAG=""
RECURSIVE=false
PROCESSALLFILE=false
PROCESSING_FILES=false
PROCESSALLFILE_FLAG=""
SED_EXEC=sed
USE_EXT=false
FILE_EXT=""
TMP_CMD="mktemp"
TMP_OPTS="--tmpdir="
XDEV=""
ISDARWIN=false


if [ $(uname) == "SunOS" ] ; then
  if [ -x /usr/gnu/bin/sed ] ; then
    echo "Using GNU sed..."
    SED_EXEC=/usr/gnu/bin/sed
  fi 
  TMP_OPTS="-p "
fi


if [ $(uname) == "Darwin" ] ; then
  TMP_OPTS="-t tmp"

  SED_EXEC="perl -pe"
  echo "Using perl..."
  ISDARWIN=true

fi


function usage() {
  echo "bom-remove [-adrx] [-s sed-name] [-e ext] files..."
  echo ""
  echo "  -a    Remove the BOM throughout the entire file."
  echo "  -e    Look only for files with the chosen extensions."
  echo "  -d    Do not overwrite original files and do not remove temp files."
  echo "  -r    Scan subdirectories."
  echo "  -s    Specify an alternate sed implementation."
  echo "  -x    Don't descend directories in other filesystems."
}


function checkExecutable() {
  if ( ! which "$1" > /dev/null 2>&1 ); then
    echo "Cannot find executable:" $1
    exit 4
  fi
}


function parseArgs() {
  while getopts "adfrs:e:x" flag
  do
    case $flag in
      a) PROCESSALLFILE=true ; PROCESSALLFILE_FLAG="-a" ;;
      r) RECURSIVE=true ;;
      f) PROCESSING_FILES=true ;;
      s) SED_EXEC=$OPTARG ;;
      e) USE_EXT=true ; FILE_EXT=$OPTARG ;;
      d) DELETE_ORIG=false ; DELETE_FLAG="-d" ;;
      x) XDEV="-xdev" ;;
      *) echo "Unknown parameter." ; usage ; exit 2 ;; 
    esac
  done


  shift $(($OPTIND - 1))

  if [ $# == 0 ] ; then

    usage;

    exit 2;

  fi



  # fixing darwin
  if [[ $ISDARWIN == true && $PROCESSALLFILE == false ]] ; then
    PROCESSALLFILE=true
    echo "Process all file is implicitly set on Darwin."
  fi

  FILES=("$@")


  if [ ! -n "$FILES" ]; then
    echo "No files specified. Exiting."
  fi


  if [ $RECURSIVE == true ]  && [ $PROCESSING_FILES == true ] ; then
    echo "Cannot use -r and -f at the same time."
    usage
    exit 1
  fi


  checkExecutable $SED_EXEC
  checkExecutable $TMP_CMD
}


function processFile() {
  if [ $(uname) == "Darwin" ] ; then
    TEMPFILENAME=$($TMP_CMD $TMP_OPTS)
  else
    TEMPFILENAME=$($TMP_CMD $TMP_OPTS"$(dirname "$1")")
  fi
  echo "Processing $1 using temp file $TEMPFILENAME"


  if [ $PROCESSALLFILE == false ] ; then 
    cat "$1" | $SED_EXEC '1 s/\xEF\xBB\xBF//' > "$TEMPFILENAME"
  else
    cat "$1" | $SED_EXEC 's/\xEF\xBB\xBF//g' > "$TEMPFILENAME"
  fi


  if [ $DELETE_ORIG == true ] ; then
    if [ ! -w "$1" ] ; then
      echo "$1 is not writable. Leaving tempfile."
    else
      echo "Removing temp file..."
      mv "$TEMPFILENAME" "$1"
    fi
  fi
}


function doJob() {
  # Check if the script has been called from the outside.
  if [ $PROCESSING_FILES == true ] ; then
    for i in $(seq 1 ${#FILES[@]})
    do
      echo ${FILES[$i-1]}
      processFile "${FILES[$i-1]}"
    done


  else
    # processing every file
for i in $(seq 1 ${#FILES[@]})
do
CURRFILE=${FILES[$i-1]}
      # checking if file or directory exist
      if [ ! -e "$CURRFILE" ] ; then echo "File not found: $CURRFILE. Skipping..." ; continue ; fi
      
      # if a paremeter is a directory, process it recursively if RECURSIVE is set
      if [ -d "$CURRFILE" ] ; then
        if [ $RECURSIVE == true ] ; then
          if [ $USE_EXT == true ] ; then
            find "$CURRFILE" $XDEV -type f -name "*.$FILE_EXT" -exec "$0" $DELETE_FLAG $PROCESSALLFILE_FLAG -f "{}" \;
          else
            find "$CURRFILE" $XDEV -type f -exec "$0" $DELETE_FLAG $PROCESSALLFILE_FLAG -f "{}" \;
          fi
        else
          echo "$CURRFILE is a directory. Skipping..."
        fi
      else
        processFile "$CURRFILE"
      fi
    done
  fi
}


parseArgs "$@"
doJob

Examples

Assuming the script is in your $PATH and it's called bom-remove, you can "clean" a bunch of files invoking it this way:

$ bom-remove file-to-clean ...

If you want to clean the files in an entire directory, you can use the following syntax:

$ bom-remove -r dir-to-clean ...

If your sed installation is not in your $PATH or you have to use an alternate version, you can invoke the script with the following syntax:

$ bom-remove -s path/to/sed file-to-clean ...

If you want to clean a directory in which other file systems might be mounted, you can use the -x option so that the script does not descend them:

$ bom-remove -xr dir-to-clean ...

Next Steps

The most effective way to fight the BOM is avoiding spreading it. Microsoft Notepad, if there's anybody out there using it, isn't the best tool to edit your UTF-8 files so, please, avoid it.

However, should your file system be affected by the BOM-desease, I hope this script will be a good starting point to build a BOM-cleaning solution for your site.

Enjoy!

An @ Symbol in OS X File Permissions and the Problems That Came

Today I was deploying some software updates in our Solaris 10 servers and, since such updates are distributed in files archived with the GNU tar utility (not shipped by default in Solaris 10), I had to convert them before moving them to our servers. I'm quite accustomed to such a procedure: that's why (amongst other more important reasons) I always have an OpenSolaris virtual machine running in my other desktops (thanks to VirtualBox).

Today, notwithstanding the fact the I had a gut feeling that I was not doing the correct thing, I decided to open a Mac OS X terminal and do the job from there. GNU tar and pax are there, aren't they? There should not be any problem, thought I.

Well, I should have paid attention to that gut feeling of mine: to spare the required OpenSolaris boot, I lost much more than what I had planned. I learned something new but... what was it worth? A bunch of non portable behaviours of Mac OS X? Just the confirmation I needed, once again (I realize I'm a stubborn fellow): if you can, do not rely on a Mac to do your UNIX-ish tasks. Chances are you're going to waste more than you believed.

This is what happened.

Making a pax File for a Solaris Host

This was meant to be a quick task. I copied the archive from a Solaris NFS mount where, some days ago, I had stored the file. I ran gunzip to decompress the file, ran tar to unpack the files, ran pax to store the resulting directory in a pax file and used rsync to quickly transfer the resulting archive back to the target Solaris machine:

$ gunzip [file.tar.gz]

$ tar xf [file.tar]

$ pax -w -f ./[file.pax] [dir]

$ rsync -vz --inline [file.pax] [myserver]

I then connected to the target server and used pax to explode the transferred archive. Since one of the things I feared was Mac OS X messing around with file permissions, I quickly checked the results on the Solaris side and everything looked fine. Well, sort of...

The next thing I usually do with such bundles is checking the differences between the running version and the soon-to-be-deployed one to be sure I won't miss any customization I made in the instance to be patched. It usually is another easy job since very few XML files are usually involved. Great was my surprise when diff began to write out hundreds of differences between one version and another.

Doh! It seemed as though somewhere in between somebody had decided to duplicate each file following a naming pattern; if the expected file was named x, I was seeing two files in Solaris: x and ._x. Where did these file came out from?

Back to the Mac

The original directory had been deleted just after producing the pax file so that the first I did was using pax to unpack the archive in the Mac. As I told you, everything looked fine. Except for an @ character that should not be there:

drwxr-xr-x@ [...snip...]

Maybe you don't know about it but that means that such a file has extended attributes set. Extended attributes are basically metadata: the file system does not interpret them and the ls command uses the @ symbol after the file permissions to inform you that they are there.

Strange Things Are Happening

There's nothing strange about extended attributes: it's very important to be aware of their existence to correctly handle them when, for example, you need to do a backup or move files from one file system to another (unless you don't mind losing them.) Solaris' pax and tar, for example, both support archival of extended attributes even if, by default, they ignore them. Since I know that my original archive had no extended attribute set in any of its file, I just ran a plain pax to build my archive. Maybe OS X pax defaults to a different behaviour so that I checked pax man page.

Doh! Oddly, OS X pax man page makes no mention at all of extended attributes, notwithstanding the fact that, undoubtedly, it was in fact managing some. Which? In OS X, just run xattr -l to find out.

Quarantine

A xattr told me that the com.apple.quarantine extended attribute was set in the culprit and a quick script confirmed that every file in that directory had that attribute set. According to Apple's documentation, that attribute is set by recent versions of OS X to mark the file as a potential threat. Do you remember when finder kindly asks you if you're really sure you want to open a downloaded file? The quarantine extended attribute is the mechanism OS X programs use to store that information.

How did it make it there?

But wait! I downloaded a tar.gz file and stored in an NFS 4 mounted file system for later processing. The question is:

• How did that attribute was stored in the NFS 4 file system where I stored the file?
• How did that attribute make it into every file stored into the pax archive?

As far as it concerns the first question, NFS 4 does support extended attributes. But again, should it be there, I would have known because I checked it directly in the Solaris host and, moreover, the box that acts as a NFS server runs some cron scripts that checks (and fixes to my taste) ACLs and extended attributes in some of the exported NFS file systems. It was very easy to find out that OS X, when storing such a file in a remote NFS share, instead of using NFS 4 extended attributes it stores them in a separate file named with the pattern mentioned above. I then discovered that the shared NFS file system in question is plagued by ._* files. Everywhere.

As far as it concerns the second question, well... I copied once again the file using the finder and performed the same operations I did before. The resulting files all had the quarantine extended attribute set. Who's the culprit, I do not know. I don't even care. The fact is that OS X is performing this operation with tools such as pax and tar without informing the user and, what's even worse, without even mentioning it in such commands' man pages.

Conclusion

The reasoning above explains why, even if Solaris pax (such as NFS) aren't managing any extended attribute, I did end up with duplicated files which were inserted into the pax file produced with Mac OS X. Probably they aren't going to hurt my deployment but surely they should not be there.

For the sake of curiosity, I checked that removing the ._* files in the NFS share made everything go back to normal.

Nevertheless, since I feel there are many OS X dark corners left for me to discover, I won't be making such an error again. Instead of waiting a couple of minutes for a virtual machine to bootstrap, I ended up wasting an entire hour for a (yet another...) bad documented and non-portable idiosyncrasy of an operating system that brings no added value with it.