ELK Stack, also known as the Elastic Stack is a powerful and versatile open-source toolset that has revolutionized the way businesses manage and analyze their data. ELK Stack seamlessly integrates these three robust components to offer a comprehensive solution for searching, analyzing, and visualizing large volumes of data in real-time. So, buckle up, for a comprehensive overview of the ELK stack and its components, which will be a great starting point for beginners.

What Is Log Analytics?

Log analysis refers to the process of examining and interpreting log files generated by various systems, applications, or devices. It involves analyzing these logs to gain insights, identify patterns, detect anomalies, troubleshoot issues, and make informed decisions.

Here are some common benefits of log analytics: 

• Troubleshooting and issue resolution
• Performance monitoring and optimization
• Security analysis and threat detection
• Compliance and auditing
• Business insights and decision-making

ELK stack plays an important role in achieving the above.

What is ELK Tech Stack?

The ELK stack, which is an acronym for Elasticsearch, Logstash, and Kibana, forms a powerful combination for centralized logging, log analysis, and real-time data visualization. An extended and robust elastic stack, it also incorporates Beats and Xpack, augmenting its capabilities.

Developed by Elastic, these open-source tools are widely utilized to streamline log management and gain valuable insights from real-time data visualization. 

Let’s understand the components of the ELK Stack individually.

Elasticsearch

Elasticsearch is a distributed, real-time search and analytics engine. It stores and indexes large volumes of structured or unstructured data, making it highly scalable and efficient for searching, querying, and analyzing data in near real-time. 

Elasticsearch provides fast and flexible search capabilities, enabling users to perform complex searches across various fields and apply aggregations to explore and visualize data.

Logstash

Logstash is a data collection and processing tool. It’s basically a data processing pipeline that takes the data from multitude of sources and tosses it over to a visualization tool like Kibana or Elasticsearch. 

Logstash can also enrich data by applying filters, transformations, and enrichments before sending it to Elasticsearch. It allows ingestion, parsing, and transforming data from various sources and formats. 

Together, Logstash and Elasticsearch serve as the foundation for data processing and storage, seamlessly feeding valuable insights into Kibana's powerful visualization and analytics capabilities. 

Kibana

Let’s now look at where Kibana fits into the ELK Stack model. To be defined in a single line, Kibana is a dashboard for analyzing and visualizing data. 

Now that you’ve started receiving data from Elasticsearch, what should be your next step? This is where a data visualization tool like Kibana jumps in. You can analyze and visualize any datalog with Kibana. It provides a user-friendly interface to interact with the data stored in Elasticsearch.

Beats

Beats serves as lightweight data shipper that send various types of data from different sources to Elasticsearch or Logstash for processing and analysis. 

They flawlessly integrate with the ELK stack, enhancing its capabilities by facilitating the collection and transmission of data from diverse sources, such as system logs, network packets, metrics, and audit logs.

It simply sends over data to Logstash or Elasticsearch that can be installed over the servers. There are multiple types of beats that have different tasks.

Other tools offer similar functionalities to beats and might be better suited for specific use cases. For example, Fluentd, RSyslog, Splunk Universal Forwarder, Logagent, NXlog, Filestash, etc.

How Does the ELK Stack Work?

The components of the Elastic Stack – Beats, Elasticsearch, Kibana, and Logstash – collaborate seamlessly to ingest, process, store, and visualize data. Here's a simplified workflow illustrating the same:

Data Collection with Beats

• Beats collects data from various sources such as logs, metrics, or network packets. 
• Sends the collected data directly to Logstash.

Data Processing with Logstash

• Logstash receives data from Beats and applies filters, transformations, and enrichments to the data, ensuring its compatibility and consistency.
• Processed data can be sent to Elasticsearch or to other systems for further processing or storage.

Data Indexing and Storage with Elasticsearch

• Elasticsearch receives data from Logstash, indexes and stores the data in a distributed manner, ensuring high availability and scalability.
• Indexed data becomes searchable, enabling fast and efficient retrieval using Elasticsearch's powerful search capabilities.

Data Visualization with Kibana

• Kibana acts as the visualization layer. It connects to Elasticsearch and provides a user-friendly interface for data exploration. Users can create interactive dashboards, visualizations, and reports using a rich set of tools and templates.
• Kibana enables real-time monitoring, data discovery, and analysis, allowing users to gain valuable insights from the indexed data.

As the complexity of your application increases, you might end up using additional components to enhance the resiliency of your application, such as Kafka, RabbitMQ, and Redis, etc.

Now that you have a clear understanding of how the ELK Stack works, the next crucial step is to install and configure the stack appropriately.

ELK Stack Plugins & Integrations

The ELK stack can hold an unlimited number of integrations, as long as you have the resources to support them. You can use Elasticsearch API to create custom ELK Stack integrations apart from the existing plugins & integrations.

You can also integrate ELK Stack for collecting data from various data sources like Azure Monitor, Amazon Cloudwatch, Google Cloud Platform, Sumo Logic, etc. 

Check complete list of Elastic integrations & plugins.

How to Install ELK Stack?

A straightforward installation takes a few hours. Install the Elastic Stack products in the following order:

1. Elasticsearch
2. Kibana 
3. Logstash 
4. Beats

Why? 

Installing in this order ensures that the components each product depends on are in place.

Elastic Cloud provides a hosted service for ELK stack available on AWS, GCP, and Azure. You can sign up for a free trial of Elastic Cloud.

To install & manage Elasticsearch on your own, there are several options to run Elasticsearch on:

• Linux, MacOS, or Windows machines
• Docker container
• Elastic Cloud on Kubernetes (for Kubernetes environments)

You can download the latest version of each component from here.

ELK Stack Installation On Windows

Check a few requisites here to make sure you have everything in place before installing Elastic Stack on windows. Now follow the following steps:

Install Elasticsearch with .zip on windows 

To install Elasticsearch on Windows, you can utilize the Windows .zip archive. This package includes the elasticsearch-service.bat command, which facilitates setting up Elasticsearch as a service. Click here for a recent stable version of Elasticsearch.

• Directly download and install the .zip package for Elasticsearch 8.8.1 from this link here. It will create a folder named elasticsearch 8.8.1 in your downloads folder.

You may also check this page to download for different platforms.

• Now extract the zip file in your desirable directory. This will install Elasticsearch in your system. Let’s get it running now. 
• To run Elasticsearch from the command line, open the bin folder > navigate the elasticsearch batch file > open it.

• If you can see the password, username, and enrollment token, then your elastic server is up and running on your system.

Now we’ll configure the Kabana component. The enrollment token shall be used ahead. 

Install Kibana

• Directly download Kibana 8.8.1 zip for windows from here. This will create a folder kibana-8.8.1-windows-x86_64.

You may also check this page to download for different platforms.

• Now extract the zip file in your desirable directory. Kibana is now installed. Let’s get it running now. 
• Similar to what we did in Elasticsearch, open the bin folder > navigate to the batch Kibana file.

• Once you click the batch file it will run in cmd. The highlighted link will take you to configure Kibana. Simply copy the link and paste it on your browser.

• Once you open the link, you’ll be redirected to a 'Configure Elastic to get started' page. Here, you’ll need to enter the enrollment token from the terminal we saw before. Next hit the 'Configure Elastic' button.

• If you’ve pasted the correct token, your Elastic is likely to have been configured.

• Now you’ll need to enter the username and password. This was also prompted back when you got Elasticsearch running.

• Copy your username and password and paste it here now. Click on login and you’ll enter the Elastic UI.

• Now if you want to continue adding more integrations you can select the Add integrations option, else simply click on Explore on my own option.

You can now begin to explore the platform and take advantage of it.

Note:

• Use the elasticsearch-reset-password tool to change password.
• Use the elasticsearch-create-enrollment-token tool to create new enrollment tokens for Kibana or Elasticsearch nodes. Both are located in the Elasticsearch bin directory.

Install Logstash

Logstash can be installed from downloaded binary or packaged repositories. 

• Obtain the Logstash binaries from the official download page at https://www.elastic.co/downloads. Choose the appropriate Logstash installation file based on your host environment, which can be TAR.GZ, DEB, ZIP, or RPM format.

• Extract the contents of the downloaded file without installing Logstash into a directory path that includes colon (:) characters.
• The Free Logstash packages are available under the Elastic license including both open source and free commercial features. Download the oss package, which includes features available under the Apache 2.0 license.
• For supported Linux operating systems, you can conveniently use a package manager to install Logstash.

To stop Logstash, simply enter the CTRL+C command in the console where Logstash is running. This will gracefully terminate the Logstash process.

Install Beats

Depending on the type of beat you want to install you can refer to the official documentation here. 

We’ll quickly cover the filebeats installation in windows:

Download the Filebeat Windows zip file from the downloads page. To install:

• Extract the contents of the zip file into the C:\Program Files directory.
• Rename the extracted directory from "filebeat-<version>-windows" to "Filebeat".
• Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select "Run As Administrator").
• Change the directory to 'C:\Program Files\Filebeat' in the PowerShell prompt.
• Run the command '.\install-service-filebeat.ps1' to install Filebeat as a Windows service.

Note: If script execution is disabled on your system, use the following command:

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/f8d4daec0d30fc39e28e370698281b16.js</p>

Set the host and port for Filebeat to find the Elasticsearch installation, as well as configure the username and password for an authorized user. You’ll need to connect to Elastic stack. Follow these steps:

• Open the Filebeat configuration file, filebeat.yml, located in the Filebeat installation directory.
• Search for the Elasticsearch output section in the configuration file.
• Set the "hosts" parameter to the desired Elasticsearch host and port. For example:

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/a0927aad142a47ea77c0328f42982bc9.js</p>

• If your Elasticsearch cluster requires authentication, uncomment the "username" and "password" lines, and provide the appropriate credentials. For example:

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/f78c9ee1cc56ad96c3a008878d08360b.js</p>

• Save the changes to the configuration file.

If you intend to utilize the pre-built Kibana dashboards provided by Filebeat, you'll need to configure the Kibana endpoint. However, if your Kibana instance is running on the same host as Elasticsearch, you can skip this step.

You’ll have to now enable & configure data collection modules. 

To determine which modules you should enable, you can view the list of available modules by running the following command in windows:

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/e8f8fb475bdb1e97856f4dd4ddb363ff.js</p>

Note: By default, filesets within a module are disabled and you need to enable at least one fileset.

Load your assets. Ensure that the user specified in the filebeat.yml configuration file has the necessary authorization to set up Filebeat. To initiate the setup process, run the following command from the installation directory:

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/5faa2d73bb6ded65c7eeeba757851a44.js</p>

Before starting Filebeat, make sure to update the user credentials in the filebeat.yml configuration file. Specify a user who has the necessary authorization to publish events.

Finally, Filebeat will start streaming events to Elasticsearch, sending the collected data for indexing and analysis.

While this guide primarily focuses on the installation process of the ELK Stack on Windows, it's important to note that the ELK Stack is compatible with various operating systems. 

If you're seeking installation instructions for other operating systems such as Linux or macOS, refer to the official documentation by Elastic here.

Congratulations, you've successfully completed the ELK Stack installation! Now, you too can harness the potential of ELK Stack to optimize performance, enhance security, and make data-driven decisions across various use cases. 🧑‍💻

Query Example in Elasticsearch

Let's write a query in Elasticsearch to search an address that has either a lane or street in the name and has a balance between 20,000 to 30,000.

You can use a combination of the bool query, should clause for the name, and a range query for the balance. Assuming you have an index named "addresses" and the field names for the address and balance are "address_name" and "balance," respectively, the query would look like this:

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/05ac19d84736098863984601c2c0f246.js</p>

Now you have successfully ran your first query in Elasticsearch.

ELK Stack v/s Grafana

Kibana and Grafana are potent data visualization tools but have different origins and purposes. Let's find out how do they stack up against each other so you can figure which one fits your needs better.

• Kibana (the ‘K’ in ELK Stack) was built on top of the Elasticsearch stack, famous for log analysis and management. Grafana was created mainly for metrics monitoring, supporting visualization for time-series databases.
• The ELK stack is designed to be scalable, storing and analyzing large amounts of data. Grafana is not as scalable as the ELK stack, so it may not be a good choice for organizations that need to store and analyze large amounts of data.
• If you’re using Elasticsearch as your primary data source, Kibana might be the right choice for advanced query & analysis capabilities.
• Elasticsearch is a powerful search engine that can be used to search and filter data quickly and efficiently. Grafana does not have a built-in search engine, so you will need to use a third-party search engine to search your data.
• If you self host Grafana, it’s easier to maintain than Elasticsearch. Maintenance is an overhead with Elasticsearch.

Pricing Comparison:

• Both Grafana & Kibana start free. Moving up, Grafana offers cloud-based service with different plans & features. The cloud service has three plans: Free, Pro, and Advanced. The Free plan has limited usage and features, such as 50 GB of logs, 10k metrics, and 500 k6 virtual user hours per month. Pro plan costs $29 per month plus usage, and includes more usage and features, such as 100 GB of logs, 20k metrics, 1000 k6 virtual user hours per month, and one enterprise plugin. 
• ELK stack also offers a cloud-based service called Elastic Cloud, which has four plans: Standard, Gold, Platinum, and Enterprise. The Standard plan costs $95 per month. It includes core Elastic Stack features, security alerting, centralized ingest and agent management, malware prevention, host data collection, case management, APM observability apps, logging, metrics, enterprise search apps for websites, mobile apps, and workplace searches.

Conclusion

In conclusion, the ELK Stack proves to be an invaluable tool for organizations seeking powerful log analysis and real-time data visualization capabilities. So, let the ELK Stack guide your data journey, turning log chaos into valuable wisdom, because in the world of data, it's all about finding the ELK-usive truth. 🙃

‍