ELK stack for beginners – Part 1

Posted by

ELK stands for Elasticsearch, Logstash and Kibana, now known as the Elastic Stack. It is a highly popular set of tools that allows the aggregation, analysis and searching of data we care about. You may for instance use it as a logging and monitoring solution for the applications you create but it can be used for much more than that.

Elasticsearch

Simply put, Elasticsearch is a open-source search engine. Not like Google, but it’s an engine that you can feed with large amounts of your own data. It uses JSON so this data you’re inputting would be in the form of JSON. Once the data is in, you can do things with it!

The most basic functions available can be divided into  indexing, searching and modifying the data. When we ‘index’ data we are inputting it into Elasticsearch. Then we can search the indexed data by creating custom queries based on the questions we are asking about that data. For example, if we were to index an array of 100 Customer JSON objects containing data such as a first name and a last name, we retrieve the list of ten most popular first names.

Elasticsearch is easy to get started with but will take time to master.

Logstash

Logstash is a processing pipeline. It will also take some form of data, possibly do something to it, then output that data somewhere else. In a simple case, a CSV file may be read containing records of Customers with a set of fields related to each customer. The read data may be filtered and modified (such as converting it into a JSON format), then the newly modified data is outputted or ‘stashed’ into another medium such as Elasticsearch. Of course, Elasticsearch will be pleased with the JSON format because that is what it works with.

Elasticsearch and Logstash go in hand-in-hand but there are plenty of sources Logstash can input/output into, for example what if you wanted to slap it into an Amazon S3 bucket instead? With the help of input and output plugins available for Logstash, this is possible.

Kibana

Finally we come to Kibana. Remember all that data you indexed into Elasticsearch? How are you going to know what is in that data and start asking questions about the bigger picture without visualising it in some way? You can’t inspect every individual piece of data with a calculator in your hand. This is where Kibana helps. It allows you to create a colourful yet powerful set of dashboard components such as graphs, histograms, pie charts, tables etc. Behind the scenes, Kibana is calling the functions available in Elasticsearch, receiving data responses, performing some statistics and displaying them to you.


In part 1 we will be doing the following:

    • Install Elasticsearch
    • Install Logstash
    • Install Kibana

NOTE:  I will be using the Mac OS. All components can be downloaded from the  http://www.elastic.co website.

Install Elasticsearch

    • Ensure you have at least Java 8 installed on your machine. You can install this using homebrew:

brew cask install java

  • Using the command line, change directory into the newly downloaded directory.
  • Enter the command ‘bin/elasticsearch’ and this will start the Elasticsearch cluster. This will take a few seconds to start. By default, the service should be running on http://localhost:9200 so head over to it in your browser and you should see a response from the cluster. This should verify that your Elasticsearch cluster is working.

Install Logstash

  • Go to https://www.elastic.co/downloads/logstash and download the TAR file.
  • Change into the new directory through your command line.
  • Create and save a logstash.conf file (with the following content for now just to get the service running, later we will update the configure the desired inputs/filters/outputs):
input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
  • It doesn’t matter where the file is saved, however the file location will need to be referenced when starting the Logstash service. To keep things simple, save the new file in the Logstash directory you changed directory into.
  • Once the file is created, run the following from within that same Logstash directory:
bin/logstash -f logstash.conf
  • You should see in the console that the Logstash service was successfully started.

Install Kibana

  • Go to https://www.elastic.co/downloads/kibana and download Kibana using the MAC link (or the link for whichever OS you are using).
  • Open the config/kibana.yml and change the elasticsearch.url to point to your Elasticsearch instance so Kibana knows where it will access Elasticsearch. You can do this by uncommenting the #elasticsearch.url: http://localhost:9200.
  • Now change directory into the downloaded Kibana folder and run ‘bin/kibana‘ in the command line. This will start the Kibana server. By default, the Kibana server will run on http://localhost:5601 so head over to it in your web browser.

You should now have both Elasticsearch and Kibana running in the console. Remember not to close the consoles or the services will no longer be running!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s