Base install of Elasticsearch 5.5.1 for Ubuntu 16.04 in AWS EC2

Base install of Elasticsearch 5.5.1 for Ubuntu 16.04 in AWS EC2


CONNECT TO YOUR INSTANCE VIA SSH…

Laptop:$ ssh ubuntu@54.174.41.136

INSTALL JAVA/OPENJDK FIRST.

Find OpenJDK in apt…

$ sudo apt search openjdk

As of this writing OpenJDK9 doesnt work with ES…I installed 8…

$ sudo apt-get install openjdk-8-jdk
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
ca-certificates-java fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra hicolor-icon-theme
java-common libasound2 libasound2-data libasyncns0 libatk1.0-0 libatk1.0-data libavahi-client3 libavahi-common-data
libavahi-common3 libcairo2 libcups2 libdatrie1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libflac8
libfontconfig1 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa
libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8
libjpeg8 liblcms2-2 libllvm4.0 libnspr4 libnss3 libnss3-nssdb libogg0 libpango-1.0-0 libpangocairo-1.0-0
libpangoft2-1.0-0 libpciaccess0 libpcsclite1 libpixman-1-0 libpthread-stubs0-dev libpulse0 libsensors4 libsm-dev
libsm6 libsndfile1 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisenc2 libx11-dev libx11-doc
libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shm0
libxcb-sync1 libxcb1-dev libxcomposite1 libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxi6 libxinerama1
libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6 libxtst6 libxxf86vm1 openjdk-8-jdk-headless openjdk-8-jre
openjdk-8-jre-headless x11-common x11proto-core-dev x11proto-input-dev x11proto-kb-dev xorg-sgml-doctools xtrans-dev
Suggested packages:
default-jre libasound2-plugins alsa-utils cups-common librsvg2-common gvfs libice-doc liblcms2-utils pcscd
pulseaudio lm-sensors libsm-doc libxcb-doc libxt-doc openjdk-8-demo openjdk-8-source visualvm icedtea-8-plugin
openjdk-8-jre-jamvm libnss-mdns fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei fonts-wqy-zenhei
fonts-indic
The following NEW packages will be installed:
ca-certificates-java fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra hicolor-icon-theme
java-common libasound2 libasound2-data libasyncns0 libatk1.0-0 libatk1.0-data libavahi-client3 libavahi-common-data
libavahi-common3 libcairo2 libcups2 libdatrie1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libflac8
libfontconfig1 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa
libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8
libjpeg8 liblcms2-2 libllvm4.0 libnspr4 libnss3 libnss3-nssdb libogg0 libpango-1.0-0 libpangocairo-1.0-0
libpangoft2-1.0-0 libpciaccess0 libpcsclite1 libpixman-1-0 libpthread-stubs0-dev libpulse0 libsensors4 libsm-dev
libsm6 libsndfile1 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisenc2 libx11-dev libx11-doc
libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shm0
libxcb-sync1 libxcb1-dev libxcomposite1 libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxi6 libxinerama1
libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6 libxtst6 libxxf86vm1 openjdk-8-jdk openjdk-8-jdk-headless
openjdk-8-jre openjdk-8-jre-headless x11-common x11proto-core-dev x11proto-input-dev x11proto-kb-dev
xorg-sgml-doctools xtrans-dev
0 upgraded, 100 newly installed, 0 to remove and 25 not upgraded.
Need to get 66.6 MB of archives.
After this operation, 367 MB of additional disk space will be used.
Do you want to continue? [Y/n] y

Install Elastic’s GPG key…

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK

Make sure transport is installed/up to date…

$ sudo apt-get install apt-transport-https
Reading package lists… Done
Building dependency tree
Reading state information… Done
apt-transport-https is already the newest version (1.2.20).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Save the repo definition…

$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
deb https://artifacts.elastic.co/packages/5.x/apt stable main

Update and install…

$ sudo apt-get update && sudo apt-get install elasticsearch
Fetched 12.0 MB in 2s (5,955 kB/s)
Reading package lists… Done
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
Need to get 33.4 MB of archives.
After this operation, 37.3 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 elasticsearch all 5.5.1 [33.4 MB]
Fetched 33.4 MB in 0s (40.0 MB/s)
Selecting previously unselected package elasticsearch.
(Reading database … 51035 files and directories currently installed.)
Preparing to unpack …/elasticsearch_5.5.1_all.deb …
Creating elasticsearch group… OK
Creating elasticsearch user… OK
Unpacking elasticsearch (5.5.1) …
Processing triggers for systemd (229-4ubuntu19) …
Processing triggers for ureadahead (0.100.0-19) …
Setting up elasticsearch (5.5.1) …
Processing triggers for systemd (229-4ubuntu19) …
Processing triggers for ureadahead (0.100.0-19) …
$ ps -p 1
PID TTY TIME CMD
1 ? 00:00:02 systemd

Reload the daemon…

$ sudo /bin/systemctl daemon-reload

ES doesnt start up on boot by itself…lets change that…

$ sudo /bin/systemctl enable elasticsearch.service
Synchronizing state of elasticsearch.service with SysV init with /lib/systemd/systemd-sysv-install…
Executing /lib/systemd/systemd-sysv-install enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

Start ES…

$ sudo systemctl start elasticsearch.service

We’ve got logs…

$ sudo ls -la /var/log/elasticsearch/
total 12
drwxr-x— 2 elasticsearch elasticsearch 4096 Aug 15 14:55 .
drwxrwxr-x 8 root syslog 4096 Aug 15 14:54 ..
-rw-r–r– 1 elasticsearch elasticsearch 0 Aug 15 14:55 elasticsearch_deprecation.log
-rw-r–r– 1 elasticsearch elasticsearch 0 Aug 15 14:55 elasticsearch_index_indexing_slowlog.log
-rw-r–r– 1 elasticsearch elasticsearch 0 Aug 15 14:55 elasticsearch_index_search_slowlog.log
-rw-r–r– 1 elasticsearch elasticsearch 3552 Aug 15 14:56 elasticsearch.log

Lets look to see how we went with the startup…

$ sudo cat /var/log/elasticsearch/elasticsearch.log
[2017-08-15T14:55:56,662][INFO ][o.e.n.Node ] [] initializing …
[2017-08-15T14:55:56,730][INFO ][o.e.e.NodeEnvironment ] [aZ2tzij] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [27.6gb], net total_space [29gb], spins? [no], types [ext4]
[2017-08-15T14:55:56,731][INFO ][o.e.e.NodeEnvironment ] [aZ2tzij] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-08-15T14:55:56,732][INFO ][o.e.n.Node ] node name [aZ2tzij] derived from node ID [aZ2tzijmSg2Jixolu5X9Kw]; set [node.name] to override
[2017-08-15T14:55:56,732][INFO ][o.e.n.Node ] version[5.5.1], pid[17918], build[19c13d0/2017-07-18T20:44:24.823Z], OS[Linux/4.4.0-1022-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-08-15T14:55:56,732][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [aggs-matrix-stats]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [ingest-common]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-expression]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-groovy]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-mustache]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-painless]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [parent-join]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [percolator]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [reindex]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [transport-netty3]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [transport-netty4]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] no plugins loaded
[2017-08-15T14:55:59,268][INFO ][o.e.d.DiscoveryModule ] [aZ2tzij] using discovery type [zen]
[2017-08-15T14:55:59,763][INFO ][o.e.n.Node ] initialized
[2017-08-15T14:55:59,763][INFO ][o.e.n.Node ] [aZ2tzij] starting …
[2017-08-15T14:55:59,887][INFO ][o.e.t.TransportService ] [aZ2tzij] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-15T14:56:02,944][INFO ][o.e.c.s.ClusterService ] [aZ2tzij] new_master {aZ2tzij}{aZ2tzijmSg2Jixolu5X9Kw}{K5qXuQfDT7mvt_Dne2cwfA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-08-15T14:56:02,967][INFO ][o.e.g.GatewayService ] [aZ2tzij] recovered [0] indices into cluster_state
[2017-08-15T14:56:02,969][INFO ][o.e.h.n.Netty4HttpServerTransport] [aZ2tzij] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-15T14:56:02,970][INFO ][o.e.n.Node ] [aZ2tzij] started

Log looks good…Curl the server…

$ curl -XGET 'localhost:9200/?pretty'
{
“name” : “aZ2tzij”,
“cluster_name” : “elasticsearch”,
“cluster_uuid” : “bUCHwEKfTbKfBvNE_lOfVg”,
“version” : {
“number” : “5.5.1”,
“build_hash” : “19c13d0”,
“build_date” : “2017-07-18T20:44:24.823Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.0”
},
“tagline” : “You Know, for Search”
}

Sweet…we’re good to go. Next steps would be to set up your mapping and import some data assuming you dont want to tweak the configs. Configs will be in /etc/elasticsearch/elasticsearch.yml

Advertisements

Quick and Dirty: Install and setup Elasticsearch, Logstash, and Kibana

Quick and Dirty: Install and setup Elasticsearch, Logstash, and Kibana

First you obviously need to download all of the packages. You can get them from HERE. Its also a given that you have Apache webserver installed and running.

For me I like to use the TGZ files. Im kinda funny about not getting files spread out all over the place and I like to keep them all together so I put all the files into /opt and exploded them there…

tar -zxvf *.tar

I setup my symlinks…

ln -s /opt/elasticsearch-1.1.1 ES
ln -s /opt/logstash-1.4.0 LG
ln -s /opt/kibana-3.0.1 KB

From here you pretty much are customizing your install if you already know what you are wanting to log and track. In my example I was just logging all the log files found in /var/*

First I configured a conf file for LG and ES. Here im just telling LG to log all the *log files it finds under /var/* Im doing this as root and this is just on a local vm so im not particularly worried about permissions or security. Just know that you dont want to do it this way in a prod environment.

CONF FILE:  logstash-apache.conf     I created this and put it in /opt/LG/bin/

input {
file {
path => "/var/*/*log"
start_position => beginning
}
}
filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
host => localhost
}
stdout { codec => rubydebug }
}

For more info on that conf file check out the Logstash tutorial here.

Now you want to copy everything that’s in the /opt/KB/ dir into a dir in your webserver. For me…

cp /opt/KB/* /var/www/html/

You will then need to edit a file. Open “config.js” and find the line that says “elasticsearch:” This is the location of your webserver. Since mine is locally hosted on a VM, mine reads:

elasticsearch: “http://localhost.localdomain:9200”,

From here you are ready to start things up. First start Elasticsearch, then Logstash and then we will fire up Kibana.

/opt/ES/bin/elasticsearch
/opt/LG/bin/logstash -f logstash-apache.conf

Then in your browser go to: http://localhost.localdomain:9200

You should now be looking at the default Kibana page. It will have some further info on getting started and how to change your default startup page in Kibana.

 

RUNIT and the ruby syntax error

Ive been working on writing my own cookbook to standup a fully ready to run ELK (Elasticsearch,Logstash,Kibana) server on CentOS. In doing this I have run into a few minor issues of compatibility here and there but nothing major. Once thing that I did find that was particularly troublesome was an error when I was trying to knife up a cookbook for “runit”. Its one of the dependancies for the chef-kibana cookbook and it relys on the “yum-epel” cookbook. I was able to knife up the yum-epel cookbook just fine but when I tried the runit cookbook I hot an error that said:

[seth@localhost cookbooks]$ knife cookbook upload runit yum-epel
Uploading runit [1.5.10]
FATAL: Cookbook file test/spec/libraries/provider_runit_service_spec.rb has a ruby syntax error:
FATAL: /home/seth/chef/chef-repo/.chef/../cookbooks/runit/test/spec/libraries/provider_runit_service_spec.rb:62: syntax error, unexpected '}', expecting tASSOC
FATAL: { provider.load_current_resource }.should raise_error
FATAL:                                                                          ^

My first mistake was installing and using RVM for Ruby management from way before I installed Chef. RVM is too big and robust and handles too many things to try and make it play nice and only worry about managing ruby for me. I uninstalled that and went with RBENV. This is the recommended manager of Ruby from Chef anyways. Once I got rbenv setup and installed I installed ruby 1.9.3p545. I tried again with the knife upload and I still got the same thing. I decided that maybe my version of Chef needed to be updated. I was on 11.2. I reinstalled Chef and tried the knife again. No luck. Same error…thanks for playing, try again.

Now I take to Google to see if this is something unique to me or have I found some bug somewhere. I came across this page and even though it was a year old it did help me out some. https://github.com/rcbops/chef-cookbooks/issues/352 The fix came to me when I re-downloaded the runit cookbook from https://github.com/hw-cookbooks/runit instead of using the cookbook I downloaded directly from Chef Community. For some reason the version was off by one iteration.

[seth@localhost cookbooks]$ knife cookbook upload runit
Uploading runit [1.5.11]
Uploaded 1 cookbook.

Success. Theres not really a better explanation as to what really was wrong here but I do feel it had more to do with the ruby installation than anything else. If you come across this and figure something out please feel free to leave a comment below.