Friday, June 20, 2014

Installing Logstash v 1.4 (and Greater) on FreeBSD

In a previous post I described how to install Logstash (v. 1.3 and previous) on FreeBSD and in this post I will describe how to install Logstash v. 1.4 and greater.

Until version 1.3 included, Logstash was distributed as a single JAR file, and when version 1.4 was released a new packaging style was introduced. As a consequence, new instructions are required to properly setup Logstash in FreeBSD and registering it as a service. Further information about the new    distribution layout can be found in the Logstash release notes.

As seen in the previous post, a Logstash FreeBSD port exists, but it is currently outdated since it bundles Logstash v. 1.2. But while this could be used as a starting point for JAR-based Logstash installations (as we have seen, the update process only required updating the Logstash JAR), this is not possible with the new Logstash distribution because the included rc script will fail to work.

Edit: I've managed to patch the broken Logstash dependency and I'm waiting for the pull request to be merged upstream. In the meantime you can use an updated FreeBSD port I've created to install Logstash on FreeBSD 10.

Edit: The Logstash port has been updated and I'm the new maintainer. Also, the pull request solving a JRuby bug on FreeBSD has been merged to upstream and will hopefully hit a Logstash release soon.

Skimming through the original post is recommended because it provides general information about Logstash and FreeBSD which is required to properly plan and execute a Logstash setup.

Prerequisites

The essential prerequisites required to execute Logstash are:
The former is required because LogStash is a JRuby application while the latter, although not technically a requirement, is the recommended output for Logstash.

Installing Java

To install OpenJDK on FreeBSD you can use pkg to install a ready-to-use binary package:

# pkg install openjdk

Currently, this command will install an instance of OpenJDK v. 7 in both FreeBSD 9 and 10. If you'd rather install a different version, you can search the available packages and pick the one you prefer (command output has been filtered for brevity):

# pkg search openjdk
openjdk-7.60.19,1
openjdk6-b31_3,1
openjdk8-8.5.13_7
# pkg install openjdk8-8.5.13_7

Installing ElasticSearch

Logstash includes an embedded ElasticSearch instance you can use for standalone installations (see my previous post for an introductory view on Logstash operation modes). The required configuration to bootstrap the embedded ElasticSearch instance and to have Logstash use it as its outputs is described in the following sections.

Although simpler from the standpoint of the configuration, Logstash installations using separate ElasticSearch instances are out of the scope of this post.

Installing Logstash

Logstash installation procedure is fairly simple since it is distributed as a tarball:
  • Download Logstash from the official website.
  • Extract the tarball in the designated installation directory (my personal suggestion is to avoid /usr/local because it is used by ports and to use /opt instead).

Creating an rc.d Script

An rc.d script is required in a BSD system to register a service, define its configuration and have the rc framework manage its lifetime. The following script can be used as is or as a starting point to customise your own. If used as is, be aware that the script uses the following default values:
  • Installation directory: ${logstash14_home="/opt/logstash-1.4.1"}
  • Configuration file path: ${logstash14_config="/usr/local/etc/${name}/${name}.conf"}
  • ElasticSearch data directory: ${logstash14_elastic_datadir="/var/db/logstash14"}
  • Java home: ${logstash14_java_home="/usr/local/openjdk6"}
You can override any of the supported configuration values in the /etc/rc.conf file. If, for example, you want to use an alternate Java home path, just add the following line to /etc/rc.conf setting the desired value:

logstash14_java_home="/usr/local/openjdk7"

Testing the Service

To test the Logstash service, the following command can be used:

# service logstash14 onestart

To stop it, use:

# service logstash14 onestop

To help troubleshooting any problem you might find you can enable the Logstash log, setting the logstash14_log variable to YES in the /etc/rc.conf file:

logstash14_log="YES"

The log file location is specified by the logstash14_log_file variable, whose default value is set by the service rc file (only the relevant lines are shown):

name=logstash14
logdir="/var/log"
: ${logstash14_log_file="${logdir}/${name}.log"}

The log file location can be overridden setting the logstash14_log_file variable in the /etc/rc.conf file.

Enabling the Service

Note that the rc script described above does not enable the Logstash service:

: ${logstash14_enable="NO"}

If everything works, you can enable the Logstash service just adding the following line to /etc/rc.conf:

logstash14_enable="YES"

8 comments:

Michael said...

Have you tried to get into touch with the maintainer of the FreeBSD logstash port regis.despres@gmail.com)? It would be kind of nice to have this back as an "pkg install" task.

Enrico Maria Crisostomo said...

Hi Michael,

In fact, yes, I did. And not only this time, but some months ago when I wanted to submit to his attention some improvements of the rc scripts. No luck the first time, and some luck this time: on June 18th he literally told me "1.4.1 will be PR this week.".

So far, it is not ready. Let's wait and see.

Cheers,
-- Enrico

Anonymous said...

Any word on updates to the ports for logstash?

Hopefully something will happen soon... it has been another 3 months!

Enrico Maria Crisostomo said...

I'm still waiting too. I'll get in touch once more with the port maintainer.

mmairs said...

It's in ports now. Not sure about usability/config yet...

Enrico Maria Crisostomo said...

Thanks mmairs: yes, it's in port and I'm the new maintainer, the port should be working just fine. And if you find any problem with it, just file a bug report to the FreeBSD ports mailing list.

Ole said...

Hi Michael

Thanks your for the great articles and for your work on the FreeBSD port.

I'm currius to know if there is a way on FreeBSD to start log stash with 5 workers (-w 5)?

Also is there a way to verify it?

I have tried editing the rc script and indserting the "-w 5" there but I'm not able to verify it. Logstash is started by daemon så it is not possible to se the command from "ps aux"

Cheers
Ole

Henning Kessler said...

Hello Michael

just had a test install of the most current version (1.5.1) which seams to be fixed to client mode. So there is no standalone option anymore? And when i start logstash with "service logstash onestart" or by enabling it rc.conf it doesn't seam to run as no pid file is created and i can't see any related processes in ps or top. In the logfile only one line is written complaining about running in embedded mode. Any help with this would be really great.

Regards

Henning