Base install of Elasticsearch 5.5.1 for Ubuntu 16.04 in AWS EC2

Base install of Elasticsearch 5.5.1 for Ubuntu 16.04 in AWS EC2


CONNECT TO YOUR INSTANCE VIA SSH…

Laptop:$ ssh ubuntu@54.174.41.136

INSTALL JAVA/OPENJDK FIRST.

Find OpenJDK in apt…

$ sudo apt search openjdk

As of this writing OpenJDK9 doesnt work with ES…I installed 8…

$ sudo apt-get install openjdk-8-jdk
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
ca-certificates-java fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra hicolor-icon-theme
java-common libasound2 libasound2-data libasyncns0 libatk1.0-0 libatk1.0-data libavahi-client3 libavahi-common-data
libavahi-common3 libcairo2 libcups2 libdatrie1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libflac8
libfontconfig1 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa
libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8
libjpeg8 liblcms2-2 libllvm4.0 libnspr4 libnss3 libnss3-nssdb libogg0 libpango-1.0-0 libpangocairo-1.0-0
libpangoft2-1.0-0 libpciaccess0 libpcsclite1 libpixman-1-0 libpthread-stubs0-dev libpulse0 libsensors4 libsm-dev
libsm6 libsndfile1 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisenc2 libx11-dev libx11-doc
libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shm0
libxcb-sync1 libxcb1-dev libxcomposite1 libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxi6 libxinerama1
libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6 libxtst6 libxxf86vm1 openjdk-8-jdk-headless openjdk-8-jre
openjdk-8-jre-headless x11-common x11proto-core-dev x11proto-input-dev x11proto-kb-dev xorg-sgml-doctools xtrans-dev
Suggested packages:
default-jre libasound2-plugins alsa-utils cups-common librsvg2-common gvfs libice-doc liblcms2-utils pcscd
pulseaudio lm-sensors libsm-doc libxcb-doc libxt-doc openjdk-8-demo openjdk-8-source visualvm icedtea-8-plugin
openjdk-8-jre-jamvm libnss-mdns fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei fonts-wqy-zenhei
fonts-indic
The following NEW packages will be installed:
ca-certificates-java fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra hicolor-icon-theme
java-common libasound2 libasound2-data libasyncns0 libatk1.0-0 libatk1.0-data libavahi-client3 libavahi-common-data
libavahi-common3 libcairo2 libcups2 libdatrie1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libflac8
libfontconfig1 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgif7 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa
libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8
libjpeg8 liblcms2-2 libllvm4.0 libnspr4 libnss3 libnss3-nssdb libogg0 libpango-1.0-0 libpangocairo-1.0-0
libpangoft2-1.0-0 libpciaccess0 libpcsclite1 libpixman-1-0 libpthread-stubs0-dev libpulse0 libsensors4 libsm-dev
libsm6 libsndfile1 libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libvorbis0a libvorbisenc2 libx11-dev libx11-doc
libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shm0
libxcb-sync1 libxcb1-dev libxcomposite1 libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxi6 libxinerama1
libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6 libxtst6 libxxf86vm1 openjdk-8-jdk openjdk-8-jdk-headless
openjdk-8-jre openjdk-8-jre-headless x11-common x11proto-core-dev x11proto-input-dev x11proto-kb-dev
xorg-sgml-doctools xtrans-dev
0 upgraded, 100 newly installed, 0 to remove and 25 not upgraded.
Need to get 66.6 MB of archives.
After this operation, 367 MB of additional disk space will be used.
Do you want to continue? [Y/n] y

Install Elastic’s GPG key…

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK

Make sure transport is installed/up to date…

$ sudo apt-get install apt-transport-https
Reading package lists… Done
Building dependency tree
Reading state information… Done
apt-transport-https is already the newest version (1.2.20).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Save the repo definition…

$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
deb https://artifacts.elastic.co/packages/5.x/apt stable main

Update and install…

$ sudo apt-get update && sudo apt-get install elasticsearch
Fetched 12.0 MB in 2s (5,955 kB/s)
Reading package lists… Done
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
Need to get 33.4 MB of archives.
After this operation, 37.3 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 elasticsearch all 5.5.1 [33.4 MB]
Fetched 33.4 MB in 0s (40.0 MB/s)
Selecting previously unselected package elasticsearch.
(Reading database … 51035 files and directories currently installed.)
Preparing to unpack …/elasticsearch_5.5.1_all.deb …
Creating elasticsearch group… OK
Creating elasticsearch user… OK
Unpacking elasticsearch (5.5.1) …
Processing triggers for systemd (229-4ubuntu19) …
Processing triggers for ureadahead (0.100.0-19) …
Setting up elasticsearch (5.5.1) …
Processing triggers for systemd (229-4ubuntu19) …
Processing triggers for ureadahead (0.100.0-19) …
$ ps -p 1
PID TTY TIME CMD
1 ? 00:00:02 systemd

Reload the daemon…

$ sudo /bin/systemctl daemon-reload

ES doesnt start up on boot by itself…lets change that…

$ sudo /bin/systemctl enable elasticsearch.service
Synchronizing state of elasticsearch.service with SysV init with /lib/systemd/systemd-sysv-install…
Executing /lib/systemd/systemd-sysv-install enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

Start ES…

$ sudo systemctl start elasticsearch.service

We’ve got logs…

$ sudo ls -la /var/log/elasticsearch/
total 12
drwxr-x— 2 elasticsearch elasticsearch 4096 Aug 15 14:55 .
drwxrwxr-x 8 root syslog 4096 Aug 15 14:54 ..
-rw-r–r– 1 elasticsearch elasticsearch 0 Aug 15 14:55 elasticsearch_deprecation.log
-rw-r–r– 1 elasticsearch elasticsearch 0 Aug 15 14:55 elasticsearch_index_indexing_slowlog.log
-rw-r–r– 1 elasticsearch elasticsearch 0 Aug 15 14:55 elasticsearch_index_search_slowlog.log
-rw-r–r– 1 elasticsearch elasticsearch 3552 Aug 15 14:56 elasticsearch.log

Lets look to see how we went with the startup…

$ sudo cat /var/log/elasticsearch/elasticsearch.log
[2017-08-15T14:55:56,662][INFO ][o.e.n.Node ] [] initializing …
[2017-08-15T14:55:56,730][INFO ][o.e.e.NodeEnvironment ] [aZ2tzij] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [27.6gb], net total_space [29gb], spins? [no], types [ext4]
[2017-08-15T14:55:56,731][INFO ][o.e.e.NodeEnvironment ] [aZ2tzij] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-08-15T14:55:56,732][INFO ][o.e.n.Node ] node name [aZ2tzij] derived from node ID [aZ2tzijmSg2Jixolu5X9Kw]; set [node.name] to override
[2017-08-15T14:55:56,732][INFO ][o.e.n.Node ] version[5.5.1], pid[17918], build[19c13d0/2017-07-18T20:44:24.823Z], OS[Linux/4.4.0-1022-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-08-15T14:55:56,732][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [aggs-matrix-stats]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [ingest-common]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-expression]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-groovy]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-mustache]
[2017-08-15T14:55:57,610][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [lang-painless]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [parent-join]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [percolator]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [reindex]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [transport-netty3]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] loaded module [transport-netty4]
[2017-08-15T14:55:57,611][INFO ][o.e.p.PluginsService ] [aZ2tzij] no plugins loaded
[2017-08-15T14:55:59,268][INFO ][o.e.d.DiscoveryModule ] [aZ2tzij] using discovery type [zen]
[2017-08-15T14:55:59,763][INFO ][o.e.n.Node ] initialized
[2017-08-15T14:55:59,763][INFO ][o.e.n.Node ] [aZ2tzij] starting …
[2017-08-15T14:55:59,887][INFO ][o.e.t.TransportService ] [aZ2tzij] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-08-15T14:56:02,944][INFO ][o.e.c.s.ClusterService ] [aZ2tzij] new_master {aZ2tzij}{aZ2tzijmSg2Jixolu5X9Kw}{K5qXuQfDT7mvt_Dne2cwfA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-08-15T14:56:02,967][INFO ][o.e.g.GatewayService ] [aZ2tzij] recovered [0] indices into cluster_state
[2017-08-15T14:56:02,969][INFO ][o.e.h.n.Netty4HttpServerTransport] [aZ2tzij] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-08-15T14:56:02,970][INFO ][o.e.n.Node ] [aZ2tzij] started

Log looks good…Curl the server…

$ curl -XGET 'localhost:9200/?pretty'
{
“name” : “aZ2tzij”,
“cluster_name” : “elasticsearch”,
“cluster_uuid” : “bUCHwEKfTbKfBvNE_lOfVg”,
“version” : {
“number” : “5.5.1”,
“build_hash” : “19c13d0”,
“build_date” : “2017-07-18T20:44:24.823Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.0”
},
“tagline” : “You Know, for Search”
}

Sweet…we’re good to go. Next steps would be to set up your mapping and import some data assuming you dont want to tweak the configs. Configs will be in /etc/elasticsearch/elasticsearch.yml

Advertisements