Knowledgebase

Introduction

Elastic Stack, also known as ELK, comprises three open-source programs: Elasticsearch, Logstash and Kibana. The stack is optimized for searching, analyzing, and visualization of large volumes of log data. The main components of the Elastic Stack are:

• Elasticsearch: This is the main component of the stack. It is a distributed RESTful search engine that stores and searches the text-based collected data.

• Logstash: This data processing component collects and parses the incoming data before sending it to Elasticsearch for storage.

• Kibana: This is the web interface dashboard used for searching and exploring the analyzed log data.

• Beats: This is a lightweight transport agent with plugins used to aggregate application data from different servers and applications and then send the data to either Logstash or Elasticsearch for processing.

This article describes how to install Elastic Stack on Ubuntu 20.04 server.

Prerequisites

• Deploy a fully updated RCS Ubuntu 20.04 Server.

• Create a non-root user with sudo access.

1. Install Java and Nginx

1. Update system packages.

   $ sudo apt update
   

2. Install required packages.

   $ sudo apt install wget curl gnupg2 -y
   

3. To run Elasticsearch, you require Java. Install Java.

   $ sudo apt install openjdk-11-jdk -y
   

4. Verify the installation.

   $ java -version
   

5. Kibana dashboard uses Nginx as a reverse proxy. Install Nginx webserver.

   $ sudo apt install nginx -y
   

2. Install and Configure Elasticsearch

1. Install required packages.

   $ sudo apt install apt-transport-https -y
   

2. Import the Elasticsearch PGP signing key.

   $ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
   

3. Add Elasticsearch APT repository.

   $ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee –a /etc/apt/sources.list.d/elastic-7.x.list
   

4. Update the system.

   $ sudo apt update
   

5. Install Elasticsearch.

   $ sudo apt install elasticsearch -y
   

6. Edit Elasticsearch configuration file.

   $ sudo nano /etc/elasticsearch/elasticsearch.yml
   

7. Uncomment the following lines.

   #network.host: 192.168.0.1
   
   #http.port: 9200
   

   Change value of network.host to localhost and add the following line in the Discovery section.

   discovery.type: single-node
   

   The final file should have the lines as follows:

   network.host: localhost
   
   http.port: 9200
   
   
   
   discovery.type: single-node
   

8. Save and close the file.

9. Reload the daemon.

   $ sudo systemctl daemon-reload
   

10. Start the Elasticsearch service.

    $ sudo systemctl start elasticsearch
    

11. Enable Elasticsearch service to start at system startup.

    $ sudo systemctl enable elasticsearch
    

12. Verify that Elasticsearch is running and listening on port 9200.

    $ curl -X GET "localhost:9200"
    

3. Install Logstash

1. Install Logstash.

   $ sudo apt install logstash -y
   

2. Start the Logstash service.

   $ sudo systemctl start logstash
   

3. Enable Logstash service to start at system startup.

   $ sudo systemctl enable logstash
   

4. Verify Logstash service status.

   $ sudo systemctl status logstash
   

4. Install and Configure Kibana

1. Install Kibana.

   $ sudo apt install kibana -y
   

2. Edit the Kibana configuration file.

   $ sudo nano /etc/kibana/kibana.yml
   

3. Uncomment and modify the following lines from:

   #server.port: 5601
   
   #server.host: "localhost"
   
   #elasticsearch.hosts: ["http://localhost:9200"]
   

   To:

   server.port: 5601
   
   server.host: "0.0.0.0"
   
   elasticsearch.hosts: ["http://localhost:9200"]
   

4. Save and close the file.

5. Start the Kibana service.

   $ sudo systemctl start kibana
   

6. Enable Kibana service to start at system startup.

   $ sudo systemctl enable kibana
   

7. Allow traffic on port 5601.

   $ sudo ufw allow 5601/tcp
   

5. Install and Configure Filebeat

1. Install Filebeat.

   $ sudo apt install filebeat -y
   

2. Edit the Filebeat configuration file.

   $ sudo nano /etc/filebeat/filebeat.yml
   

   Comment out the following lines:

   #output.elasticsearch:
   
   # Array of hosts to connect to.
   
   #hosts: ["localhost:9200"]
   

   Uncomment these lines in Logstash output section:

   output.logstash:
   
   hosts: ["localhost:5044"]
   

3. Save and exit the file.

4. Enable the Filebeat system module.

   $ sudo filebeat modules enable system
   

5. Load the index template.

   $ sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
   

6. Start the Filebeat service.

   $ sudo systemctl start filebeat
   

7. Enable Filebeat service to start at system startup.

   $ sudo systemctl enable filebeat
   

8. Verify that Filebeat is shipping log files to Logstash for processing.

   $ curl -XGET http://localhost:9200/_cat/indices?v
   

6. Access Kibana Web Interface

Open your web browser and access the Kibana web interface using the URL http://YourServerIP:5601. For example:

http://192.0.2.10:5601

Conclusion

You have successfully installed Elastic Stack on your server. You can now access the main dashboard via the Kibana web interface.

More Information

For more information, please see:

• Elasticsearch documentation.

• Logstash documentation.

• Kibana documentation.

• Filebeat documentation.