Quick and Dirty: Install and setup Elasticsearch, Logstash, and Kibana

Quick and Dirty: Install and setup Elasticsearch, Logstash, and Kibana

First you obviously need to download all of the packages. You can get them from HERE. Its also a given that you have Apache webserver installed and running.

For me I like to use the TGZ files. Im kinda funny about not getting files spread out all over the place and I like to keep them all together so I put all the files into /opt and exploded them there…

tar -zxvf *.tar

I setup my symlinks…

ln -s /opt/elasticsearch-1.1.1 ES
ln -s /opt/logstash-1.4.0 LG
ln -s /opt/kibana-3.0.1 KB

From here you pretty much are customizing your install if you already know what you are wanting to log and track. In my example I was just logging all the log files found in /var/*

First I configured a conf file for LG and ES. Here im just telling LG to log all the *log files it finds under /var/* Im doing this as root and this is just on a local vm so im not particularly worried about permissions or security. Just know that you dont want to do it this way in a prod environment.

CONF FILE:  logstash-apache.conf     I created this and put it in /opt/LG/bin/

input {
file {
path => "/var/*/*log"
start_position => beginning
filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
output {
elasticsearch {
host => localhost
stdout { codec => rubydebug }

For more info on that conf file check out the Logstash tutorial here.

Now you want to copy everything that’s in the /opt/KB/ dir into a dir in your webserver. For me…

cp /opt/KB/* /var/www/html/

You will then need to edit a file. Open “config.js” and find the line that says “elasticsearch:” This is the location of your webserver. Since mine is locally hosted on a VM, mine reads:

elasticsearch: “http://localhost.localdomain:9200”,

From here you are ready to start things up. First start Elasticsearch, then Logstash and then we will fire up Kibana.

/opt/LG/bin/logstash -f logstash-apache.conf

Then in your browser go to: http://localhost.localdomain:9200

You should now be looking at the default Kibana page. It will have some further info on getting started and how to change your default startup page in Kibana.



RUNIT and the ruby syntax error

Ive been working on writing my own cookbook to standup a fully ready to run ELK (Elasticsearch,Logstash,Kibana) server on CentOS. In doing this I have run into a few minor issues of compatibility here and there but nothing major. Once thing that I did find that was particularly troublesome was an error when I was trying to knife up a cookbook for “runit”. Its one of the dependancies for the chef-kibana cookbook and it relys on the “yum-epel” cookbook. I was able to knife up the yum-epel cookbook just fine but when I tried the runit cookbook I hot an error that said:

[seth@localhost cookbooks]$ knife cookbook upload runit yum-epel
Uploading runit [1.5.10]
FATAL: Cookbook file test/spec/libraries/provider_runit_service_spec.rb has a ruby syntax error:
FATAL: /home/seth/chef/chef-repo/.chef/../cookbooks/runit/test/spec/libraries/provider_runit_service_spec.rb:62: syntax error, unexpected '}', expecting tASSOC
FATAL: { provider.load_current_resource }.should raise_error
FATAL:                                                                          ^

My first mistake was installing and using RVM for Ruby management from way before I installed Chef. RVM is too big and robust and handles too many things to try and make it play nice and only worry about managing ruby for me. I uninstalled that and went with RBENV. This is the recommended manager of Ruby from Chef anyways. Once I got rbenv setup and installed I installed ruby 1.9.3p545. I tried again with the knife upload and I still got the same thing. I decided that maybe my version of Chef needed to be updated. I was on 11.2. I reinstalled Chef and tried the knife again. No luck. Same error…thanks for playing, try again.

Now I take to Google to see if this is something unique to me or have I found some bug somewhere. I came across this page and even though it was a year old it did help me out some. https://github.com/rcbops/chef-cookbooks/issues/352 The fix came to me when I re-downloaded the runit cookbook from https://github.com/hw-cookbooks/runit instead of using the cookbook I downloaded directly from Chef Community. For some reason the version was off by one iteration.

[seth@localhost cookbooks]$ knife cookbook upload runit
Uploading runit [1.5.11]
Uploaded 1 cookbook.

Success. Theres not really a better explanation as to what really was wrong here but I do feel it had more to do with the ruby installation than anything else. If you come across this and figure something out please feel free to leave a comment below.