19 October 2015

ELF on ELK on Docker


The ELF on ELK on Docker repository is available!

You can download it from Github: https://github.com/developerforce/elf_elk_docker.

What in the world is ELK? How does an ELF fit on top of an ELK? Who is this Docker I keep hearing about? Why do I feel like I've fallen down the on-premise rabbit hole of acronym based logging solutions??!!

Okay, lets back up a second. We're trying to solve the problem of creating insights on top of Event Log File (ELF) data.

ELF stands for Event Log Files. It's Salesforce's solution for providing an easy to download set of organization specific log files. Everything from page views to report downloads. You can't really swing a cat by it's tail (not that I really would try) without hitting a blog post on SalesforceHacker.com about Event Log Files. Event Monitoring is the packaging of Event Log Files.

Since we launched Event Log Files last November, I've talked with a lot of customers about how to derive insights and visualizations on top of the log data. One of the solutions I keep hearing about is the ELK stack.

ELK stands for Elasticsearch, Logstash, and Kibana. The ELK stack is an open-source, scalable log management stack that supports exploration, analysis, and visualization of log data.

It consists of three key solutions:
  1. Elasticsearch: A Lucene-based search server for storing log data.
  2. Logstash: ETL process for retrieving, transforming, and pushing logs into data warehouses.
  3. Kibana: Web GUI for exploring, analyzing, and visualizing log data in Elasticsearch.
ELK requires multiple installations and configurations on top of commoditized hardware or IaaS like AWS. To simplify the installation and deployment process, we use Docker.

Docker is an emerging open source solution for software containers. From the Docker website:
"Docker is an open platform for building, shipping and running distributed applications. It gives programmers, development teams and operations engineers the common toolbox they need to take advantage of the distributed and networked nature of modern applications."
With Docker, all the user needs to do to start working with ELF on ELK is:
  1. download the ELF on ELK from Github
  2. change the sfdc_elf.config file (add authorization credentials)
  3. run Docker from the terminal
The purpose of the plug-in is to reduce the time it takes integrating Event Log Files into ELK, not to provide out-of-the-box dashboards like this one that I quickly created:

As a result, once you start importing Event Log Files into ELK through this ETL plug-in, you'll still need to create the visualizations on top of the data. The advantage of Kibana is that it makes that part point-and-click easy.

Depending on how you configure Docker and ELK, you might want to expose your new dashboards onto to the corporate network. I found the following terminal command helps to enable access across the VPN:
VBoxManage controlvm "default" natpf1 "tcp-port8081,tcp,,8081,,8081";
ELF on ELK on Docker provides an on-premise, scalable solution for visualizing Event Monitoring data.

The ELF on ELK on Docker plug-in was created by the dynamic duo of Abhishek Sreenivasa and Mohammaed Islam.

Let us know what you think!

5 comments:

  1. Hi, thanks for that. Could you please provide the query you used for the dashboard above ?

    ReplyDelete
    Replies
    1. Check out the source repo for a screen shot showing the setup for kibana: https://github.com/developerforce/elf_elk_docker. thanks!

      Delete
  2. Great Article
    Cyber Security Projects

    projects for cse

    Networking Security Projects

    JavaScript Training in Chennai

    JavaScript Training in Chennai

    The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

    ReplyDelete
  3. Now there is a part of my job that you need to be impartial. That's part of being an analyst. I don't think I'm going to do anything that's probably out of the ordinary. For each year that Melbourne does not tunnel to the airport for example, it is another year of costs that WILL be added to the eventuality of going underground. There is a space issue around Newport so why not put the suburban lines underground and leave the country lines above ground. They do it in London and Sydney now I recall the layout of Central station.. (tags: Jordan Shoes For Sale Cheap, 2020 Jordan Release Dates, Ray Ban Sunglasses Outlet)

    Having emerged once again as a wealthy and powerful force in the New York financial community, Keene went back to investing in racing. In the 1890s, his Castleton Farm near Lexington, Ky., became one of the most important breeding operations in American history. Keene brought 40 mares from England for breeding and brought in James Rowe, Sr. (tags: Coach Outlet Store Online , Michael Kors Bags Sale, Coach Outlet Store, MK Outlet)

    National same store gross sales development for September was 1 Auckland moving firm to depend on. Similar retailer sales on Sundays. Suma specialize in vegetarian diets obtainable and They'll be served having a bowel motion. Const. Peter Galopoulos, Cet. Const.

    "In general I thought the guys played really well. Basics killed us there today, so we have to work on that. You can afford to have the ball bouncing off the stick and getting picked off by the others. Braving the shops to search for that elusive gift can be a chore and we often put it off. However, buying online also eliminates this problem. There is no need to spend money on petrol and parking to get into town to shop til you drop (tags: Cheap Real Yeezys, Discount Jordan Shoes Wholesale, Cheap Yeezys).

    ReplyDelete
  4. Learn how Docker helps developers bring their ideas to life by conquering the complexity of app development.... Docker

    ReplyDelete