19 October 2015

ELF on ELK on Docker


The ELF on ELK on Docker repository is available!

You can download it from Github: https://github.com/developerforce/elf_elk_docker.

What in the world is ELK? How does an ELF fit on top of an ELK? Who is this Docker I keep hearing about? Why do I feel like I've fallen down the on-premise rabbit hole of acronym based logging solutions??!!

Okay, lets back up a second. We're trying to solve the problem of creating insights on top of Event Log File (ELF) data.

ELF stands for Event Log Files. It's Salesforce's solution for providing an easy to download set of organization specific log files. Everything from page views to report downloads. You can't really swing a cat by it's tail (not that I really would try) without hitting a blog post on SalesforceHacker.com about Event Log Files. Event Monitoring is the packaging of Event Log Files.

Since we launched Event Log Files last November, I've talked with a lot of customers about how to derive insights and visualizations on top of the log data. One of the solutions I keep hearing about is the ELK stack.

ELK stands for Elasticsearch, Logstash, and Kibana. The ELK stack is an open-source, scalable log management stack that supports exploration, analysis, and visualization of log data.

It consists of three key solutions:
  1. Elasticsearch: A Lucene-based search server for storing log data.
  2. Logstash: ETL process for retrieving, transforming, and pushing logs into data warehouses.
  3. Kibana: Web GUI for exploring, analyzing, and visualizing log data in Elasticsearch.
ELK requires multiple installations and configurations on top of commoditized hardware or IaaS like AWS. To simplify the installation and deployment process, we use Docker.

Docker is an emerging open source solution for software containers. From the Docker website:
"Docker is an open platform for building, shipping and running distributed applications. It gives programmers, development teams and operations engineers the common toolbox they need to take advantage of the distributed and networked nature of modern applications."
With Docker, all the user needs to do to start working with ELF on ELK is:
  1. download the ELF on ELK from Github
  2. change the sfdc_elf.config file (add authorization credentials)
  3. run Docker from the terminal
The purpose of the plug-in is to reduce the time it takes integrating Event Log Files into ELK, not to provide out-of-the-box dashboards like this one that I quickly created:

As a result, once you start importing Event Log Files into ELK through this ETL plug-in, you'll still need to create the visualizations on top of the data. The advantage of Kibana is that it makes that part point-and-click easy.

Depending on how you configure Docker and ELK, you might want to expose your new dashboards onto to the corporate network. I found the following terminal command helps to enable access across the VPN:
VBoxManage controlvm "default" natpf1 "tcp-port8081,tcp,,8081,,8081";
ELF on ELK on Docker provides an on-premise, scalable solution for visualizing Event Monitoring data.

The ELF on ELK on Docker plug-in was created by the dynamic duo of Abhishek Sreenivasa and Mohammaed Islam.

Let us know what you think!

2 comments:

  1. Hi, thanks for that. Could you please provide the query you used for the dashboard above ?

    ReplyDelete
    Replies
    1. Check out the source repo for a screen shot showing the setup for kibana: https://github.com/developerforce/elf_elk_docker. thanks!

      Delete

Note: Only a member of this blog may post a comment.