Learning to install Elastic Search on Linux (and failing at it)

6 min readAug 30, 2018

I wrote this blog post in Jan 2017, as I was transitioning to a new team. I learned a lot since then, especially that there are Docker Containers these days, but still I love my first steps with Linux and since I am closing my other blog, I want to report this post here.

On new ground

For over 20 years I have been working on the Windows platform, but now it is time for me to learn Linux. I want to be proficient and use this great OS that attracts millions of developers and community contributors.

I am kind of a n00b, so, to break the ice, I decided to install elasticsearch on a Linux VM.

VMing

I will start by creating a Linux instance in Azure, just because I have an MSDN subscription that allows me to use $150 Azure credits per month for development.

Creating the VM is suuuuper simple, but if you are in doubt, here is one of 100 videos on YouTube for that. Also, what version of Linux? Well I don’t know what is the difference, so I will use Ubuntu, just because…

The latest version available is 16.04. I like fresh, so here we go.

My first question was: after I created the Linux VM, how do I connect to it? Turns out you need to connect via an SSH session, you can install PuTTY on your Windows machine. I downloaded the msi from this link.

Then you just run PuTTY, enter your machine IP address and click Open. Voila’ you are connected to the Linux VM.

Installing Java

Ok, now I learned that elasticsearch needs Java, but which version? And how do I install it on Ubuntu, does it come with it? So I tried:

maurgi@eslinux:~$ java
The program ‘java’ can be found in the following packages:
* default-jre
* gcj-5-jre-headless
* openjdk-8-jre-headless
* gcj-4.8-jre-headless
* gcj-4.9-jre-headless
* openjdk-9-jre-headless
Try: sudo apt install <selected package>

That taught me that Java is no longer installed by default on Ubuntu 16.04. Now, which version?

Elasticsearch compatibility matrix says that OpenJDK should work.

Now, what is the default-jre? Hmmm default looks good on a system I don’t know…

Turns out this is the same as openjdk-8 (a bit confusing I got to say), so there I go:

maurgi@eslinux:~$ sudo apt install default-jre

But I got in my first error:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/n/nss/libnss3_3.23-0ubuntu0.16.04.1_amd64.deb 404 Not FoundE: Unable to fetch some archives, maybe run apt-get update or try with — fix-missing?

No sweat, let’s update apt-get, easy enough.

maurgi@eslinux:~$ apt-get update

Another error (cold sweat starting…):

E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

Uh, ok I got it, I need to be super user, and this is why people prefix their command with sumo, ehm sudoku.. er… SUDO!

maurgi@eslinux:~$ sudo apt-get update

Boom! We are in business!

Then, easy:

maurgi@eslinux:~$ sudo apt install default-jre

Installing Elasticsearch

Ok so now how do I install Elasticsearch?

Elastic guys come to help: Get Started with Elastic Search

But… I liked the sudo apt thing, can I do it again? It must be a package manager thing…

Installing Elasticsearch.

It took me a second to figure out that Ubuntu is a Debian, then I can use the procedure for Debian Package.

I also learned that the package distribution type apt-get vs yum is the main difference between Debian (Ubuntu) and RPM (RedHat) type distributions…

Using a package manager requires 4 commands right now — well actually 5 there is a cheat with an && that means and also do this… :)

wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -sudo apt-get install apt-transport-httpsecho “deb https://artifacts.elastic.co/packages/5.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.listsudo apt-get update && sudo apt-get install elasticsearch

Ok this installed Elasticsearch, but where???? bin? ~ (what is this ?)

Nah, it’s in /usr/share/elasticsearch …

Install Kibana and X-Pack

Kibana is the web ui of Elasticsearch, and I need that, so I follow the same process here.

But it turns out all I need is:

sudo apt-get install kibana

that’s my boy!

Then we need the monitoring part: X-Pack, here is simple simple:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack

and

sudo /usr/share/kibana/bin/kibana-plugin install x-pack

easy-peasy.

Starting Elasticsearch and Kibana

According to Get Started with Elastic Search this is as easy as :

/usr/share/elasticsearch/bin/elasticsearch

right?

WRONG!!

java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config

Looking on GitHub I found this thread: https://github.com/elastic/ansible-elasticsearch/issues/58

that mentions:

@SalahAdDin if you install via deb/rpm package, ES is meant to run via the initscript (e.g. service node1_elasticsearch start) and not manually. This is not only true for the install via ansible, but in general.

Ok.. ok… I did install with a package manager. so let me go back to the deb installation page…

sudo -i service elasticsearch start

Yay!

Then for Kibana, I did not even look at the documentation and I went:

sudo -i service kibana start

Done. King. I RULEZ!

Well… not THAT fast, how do I test if it’s working now?

Testing Elasticsearch and Kibana

According to the getting started documentation, you can simply open your browser and open Kibana at http://localhost:5601.

Wait, I am on a terminal window, there is no browser here…

Maybe I can use a command to test?

The Debian installation package says to use :

curl -XGET ‘localhost:9200/?pretty’

and I got a nice red x:

{
  “error” : {
    “root_cause” : [
      {
      “type” : “security_exception”,
      “reason” : “missing authentication token for REST request [/?pretty]”,
      “header” : {
        “WWW-Authenticate” : “Basic realm=\”security\” charset=\”UTF-8\””
      }
      }
    ],
    “type” : “security_exception”,
    “reason” : “missing authentication token for REST request [/?pretty]”,
    “header” : {
      “WWW-Authenticate” : “Basic realm=\”security\” charset=\”UTF-8\””
    }
  },
  “status” : 401
}

I installed the X-Pack and this adds security so I need username (elastic) and password (changeme) but how to do that from the command line???

Let’s try to access Kibana then, it is on port 5601:

curl -XGET ‘localhost:5601/?pretty’

nothing returned….

Not sure if this is good or not (probably good because a 404 Not Found would have been worse!).

Install X-Pack

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html

Using Scripts

Elastic has an Azure Resource Manager (ARM) package that all of this automatically :)

https://github.com/elastic/azure-marketplace

Opening the firewall

Damn I crashed and burned here!

I tried to use the Firewall on Ubuntu :

sudo ufw enable
sudo ufw allow 5601

and then tried to open http://<my linux box ip>:5601 but it does not work!

I did remember to do :

sudo ufw disable

And avoid having troubles connecting to the box

I think the reason is that Kibana needs to access Elasticsearch and that is by default localhost?

But actually not, looking at Fiddler, the request never returns :(

Well I found someone else that tried to do the same thing I did and had all the details (but also way overkill with reverse proxy and using an older version of ELK (2.0).

I also found another person doing the same thing, this one more relevant/similar, so let’s see….

Giving up (for now)

So I am giving up for the moment, but I will try again…

Well, at least I learned a lot about Linux, Ubuntu, the distribution packages and accessing a Linux VM hosted on Azure!

Until next time…

P.S.: I think I know what’s wrong

I think what I got wrong is opening the Firewall on Azure -

I realized that was closed after the fact and I did dispose of the VM so I cannot try that now-

If you are looking at doing the same thing, just go to the Firewall settings for the VM in Azure and open port 5601 -

Ciao.