ELK Stack: A Complete Guide with Real-World Use Cases

Introduction

Modern applications generate vast amounts of logs, making log aggregation, analysis, and visualization critical for operational efficiency and security. The ELK Stack — comprising Elasticsearch, Logstash, and Kibana — is one of the most powerful solutions for centralized logging and observability.

In this guide, we’ll explore:

• What is ELK Stack and why use it?
• Setting up ELK Stack in a real-world scenario
• Configuring ingestion pipelines
• Advanced log analytics with Kibana
• Scaling and securing ELK
• Common troubleshooting tips
• FAQs

What is ELK Stack?

The ELK Stack consists of three major components:

1. Elasticsearch — A distributed search engine that stores and indexes logs.
2. Logstash — A data pipeline that ingests, processes, and forwards logs.
3. Kibana — A web UI to visualize logs and create dashboards.

Why Use ELK?

• Centralized Logging: Aggregate logs from multiple sources.
• Real-time Monitoring: Get instant visibility into system behavior.
• Advanced Search & Filtering: Quickly find relevant logs.
• Security & Compliance: Detect anomalies and prevent security threats.
• Integrates with Kubernetes, AWS, and cloud-native applications.

Setting Up ELK Stack on AWS (Real-World Example)

Step 1: Deploy Elasticsearch on AWS

Using AWS Managed Elasticsearch

aws es create-elasticsearch-domain --domain-name my-elk-domain \
    --elasticsearch-version 7.10 \
    --elasticsearch-cluster-config InstanceType=m5.large.elasticsearch,InstanceCount=2 \
    --ebs-options EBSEnabled=true,VolumeSize=50

For self-hosted deployment, you can use Docker:

docker run -d --name elasticsearch -p 9200:9200 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.0

Step 2: Deploy Logstash for Log Processing

Create a configuration file logstash.conf:

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:log}" }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

Run Logstash:

logstash -f logstash.conf

Step 3: Deploy Kibana for Log Visualization

docker run -d --name kibana -p 5601:5601 --link elasticsearch:elasticsearch docker.elastic.co/kibana/kibana:7.10.0

Access Kibana at http://localhost:5601

Advanced Use Cases

1. Integrating Kubernetes Logs with ELK

To collect logs from Kubernetes pods, deploy Filebeat:

kubectl apply -f https://raw.githubusercontent.com/elastic/beats/7.10/deploy/kubernetes/filebeat-kubernetes.yaml

Configure filebeat.yml to send logs to Elasticsearch:

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

2. Anomaly Detection with Machine Learning

Enable ML-based anomaly detection in Kibana to detect security threats:

POST _ml/anomaly_detectors/syslog_anomalies
{
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "function": "mean",
        "field_name": "response_time"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  }
}

3. Scaling ELK Stack

• Elasticsearch Scaling:

1. Use multi-node clusters (InstanceCount=3+)

2. Enable sharding and replication

• Logstash Scaling:

1. Use Kafka as a buffering layer

2. Deploy multiple Logstash instances behind a load balancer

• Kibana Scaling:

1. Deploy behind Nginx for multi-user access

Common Errors & Fixes

Error1:Max virtual memory areas vm.max_map_count [65530] is too low

Cause: Elasticsearch requires higher limits

Fix: Run sysctl -w vm.max_map_count=26214

Error2: Logstash logs not showing in ElasticsearchCause: Logstash pipeline misconfiguration

Fix: Check logstash.log for errors

Error3:Kibana not connecting to Elasticsearch

Cause: Elasticsearch not running or incorrect URL

Fix: Verify with curl http://localhost:9200

FAQs

1. What’s the difference between ELK and EFK?

ELK uses Logstash for ingestion, while EFK (Elasticsearch, Fluentd, Kibana) uses Fluentd, which is lighter and Kubernetes-friendly.

2. How much does ELK cost on AWS?

Costs depend on data volume. AWS OpenSearch (managed Elasticsearch) reduces operational overhead but incurs storage and compute costs.

3. Can I use ELK for real-time monitoring?

Yes, ELK supports near real-time monitoring with Kibana dashboards and Elasticsearch alerting.

4. How do I secure ELK?

• Enable TLS for communication between nodes.
• Use IAM roles for AWS Elasticsearch.
• Restrict access with RBAC in Kibana.

5. What’s the best alternative to ELK?

• Datadog, Splunk, Loki (for Kubernetes logs) if you need managed observability solutions.

Conclusion

ELK Stack provides a powerful and flexible way to collect, store, and analyze logs in real-time. Whether you are monitoring cloud-native apps, troubleshooting infrastructure, or ensuring security compliance, ELK helps uncover insights hidden in logs.

🚀 Next Steps: Deploy ELK in your environment, create dashboards, and start optimizing log analytics!

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!

Originally published on Medium.