23 March 2015

Event Monitoring Quick Start Guide or How to get from zero to dashboard in 10 minutes or less


So you are trying Event Monitoring out or perhaps you've purchased the add-on. Now what?

That was exactly what happened the other day when a customer asked me where the check box in setup was to get started with Event Monitoring.

Unfortunately, he stumbled upon two critical points when getting started with Event Monitoring:
  1. It's an API only feature. There is no check box in setup.
  2. It's just log data. What you do with that data is up to you whether you store it for a long time or you analyze it with a dashboard.
But it dawned on me that what the customer really needed was a quick start guide - from zero to dashboard in ten minutes or less.

Below is a set of steps to help get started quickly regardless of whether you are trying it out or you are ready to implement.

Steps to getting started


1. What should you expect to get for your edition?

  • Enterprise, Unlimited, Performance Edition: Login/Logout log lines for free with 1 day data retention
  • Enterprise, Unlimited, Performance Edition: full 28 log lines for add-on price with 30 day data retention
  • Developer Edition: full 28 log lines for free with 1 day data retention
That means that pretty much any organization that has access to our API has some form of Event Log Files already provisioned, even if it's only the Login and Logout log file types. Knowing what edition you have is important because it's what determines what you can access using the API. However, if you don't have a production organization and still want to try it out, sign up for Developer Edition which is free.

Troubleshooting tip: You won't see any Event Log File records for the first 24 hours from when it is provisioned. So if you don't see any records at first, try again tomorrow.

2.  What permissions do you need to access Event Log Files?

You need at least the following permissions:
  • API Enabled
  • View Event Log Files (However, any user with View All Data automatically gets access as well)
To add them, go to Setup > Administer > Manage Users > Permission Sets to create a new permission set with these two permissions.

To assign them to your user,  click on the Manage Assignments button in the permission set you just created and click Add Assignments to find your user and assign them to the permission set you just created. 

Troubleshooting tip: if you don't have the ability to create, edit, and assign a permission set, talk with a system administrator who does. 

3. Now you have access, but how do you actually access Event Log Files?

Remember, there is no check box in setup. You have to use the API.

I typically try new things out in the API using the workbench which works on any platform and provides access to a Swiss Army knife of great API features.


Once you login to workbench, the first thing to verify is that you have EventLogFile data. Go to queries > SOQL Query and from the object drop down, select EventLogFile.

To verify that you have some data, run the following query:

SELECT count() FROM EventLogFile

Troubleshooting tips:
  • if you can't login, you don't have API access and need to go back to step 2.
  • if you don't see EventLogFile from the object drop down, you don't have access and need to go back to step 2.
  • if you get 0 records returned from the query, you don't have any data yet and you should plan on trying again tomorrow.

4. Now that you have data, how do you view it?

While in workbench, go to utilities > REST Explorer and enter the following query into the text box:

/services/data/v33.0/query?q=SELECT+Id+,+EventType+,+LogDate+,+LogFileLength+,+LogFile+FROM+EventLogFile+ORDER+BY+CreatedDate+DESC+Nulls+Last


You should get some records back. Expand one of the records and click on the LogFile link attribute.


Copy everything in double quotes and pasted into a text editor like Notepad or Sublime. If you save that content with a '.csv' file extension, you now have a CSV file with your log data.

Troubleshooting tips:
  • if workbench times out, try a smaller file. Login as and Report Export tends to render in workbench. URI, Visualforce, Apex, and API tend to be too large.

5. Now you've seen the data, how do you download it when there's a lot of it?

That's where an automation script works great. I've written several blog posts to get started with writing an automation script:


You can also use a middleware provider like Cast Iron or Informatica. Just make sure they can deserialize base64 (Blob) content and/or handle CSV data.

Troubleshooting tips:
  • if you only have a Windows machine, I recommend using the python script since the other scripts are optimized for Linux and Mac operating systems.
  • if you run into some problems, comment on this blog post and I'll try to help out.

6. Now you have the data locally, but how do you make it look good?

To make it look good, you need a visualization layer. Event Monitoring doesn't come with one by default; however, there are a number of add-on tools that you can use. Below are some great ISVs (independent software vendors) who already build on top of Event Monitoring:
  • Salesforce Analytics Cloud and the Wave Platform
  • Splunk
  • New Relic
  • Fairwarning
  • ezCloudAudit
  • SkyHigh Networks
  • Cloudlock
That doesn't mean you can't use other tools like Tableau or Qlik.  As long as they can handle CSV data, they can visualize Event Monitoring data.

In case you don't have access to any of these, I recommend a free service like Plot.ly or Raw Designs.

In the Visualizing Identity Fraud Using Login History blog post, I discussed the Raw app from Density Designs.

To use Raw, just paste the CSV data you downloaded in step 4 and pick from one of their great visualizations like a Circular Dendrogram.


When you map your dimensions, pick USER_ID and URI to get a sense of who is download what reports.


And finally visualize your data.


Event Monitoring enables organizations to have self-service access to the app logs for a variety of use cases. What are you going to do with the data once you have it?

31 comments:

  1. Hi Adam, thanks for that, it was just what I was looking for. I hope you're on the SFDC payroll. :)

    ReplyDelete
    Replies
    1. Hi @Nedzer,

      I'm actually the salesforce product manager responsible for Event Monitoring and formerly responsible for Profiles and Permission Sets.

      Thanks for reading the blog!

      AT

      Delete
  2. Looking through several of the blog entries and online docs for the new Event Monitoring API, I keep seeing references to Insert, Update, Delete (equating to the C, U, and D of CRUD). I need to be able to monitor every user who views (Read from the CRUD model) a customer record (HIPAA, etc, doncha know). Assuming we are using the API, and, for example, that a user searched for customer Lastname=smith and opened Tom Smith, Dick Smith and Harry Smith's records, , can we get audit data showing that each of these customer records was viewed, and who by, and when? Once I have the record in my collection tool, I can run the various other ways of slicing this (what records was this clerk browsing through on his lunch break, how many clerks viewed the records for the celebrity of the moment, etc)
    Of course, a link to documentation that makes this clear would be terrific also.

    ReplyDelete
    Replies
    1. Hi @Liam65,

      Have you seen the APIEvent / Data Leakage Detection pilot that just launched recently:

      http://www.salesforcehacker.com/2015/07/who-stole-cookie-from-cookie-jar.html

      This is built using our 2nd generation technology that allows us to collect more granular information than what's in the 1st generation event log files. In this case, it allows us to gather customer data like SOQL query strings.

      Delete
    2. Missed that. Downloading the linked tip sheet etc.
      Thanks

      Delete
    3. Hi @Adam, @Liam65, Just wanted to understand if Data Leadage Detention Pilot is on top of Event Monitoring app. I after after the exact viewed by a user functionality but confused becuase I heard Event Monitoring gives this information in the logs. Am I missing something?

      Delete
    4. Last I checked, the data leakage pilot was not built into the wave app due to limitations of scale. The scale is an order of magnitude greater with data leakage.

      Delete
  3. Hi Adam, The only problem is the cost of event monitoring is so high that most customers won't be able to justify the price. Or do you think it is our sales guy. Any reason to justify the high price?

    -John

    ReplyDelete
  4. Hi Adam. Great post! Was wondering if you could help a non-technical person learn a bit more about what the logs mean. Is there a way to determine what the events themselves were?

    ReplyDelete
    Replies
    1. Hi Unknown, logs are kind of funny. They have amazing information but a lot of signal to noise ratio issues in general. They key for me is the identifying information - userid, time stamp, up address. This tells me who, what, when. What is harder. Each log will have info that helps understand what happened. If you can find similarities like an API post or a web page hit, rolling the results up into a report really help to reduce some of that noise and increase some of the signal.

      Delete
  5. Adam: Any hints (safe harbor) about W17? Will there be any UI components at all (ability to use SF dashboards, or hand over to WAVE directly), or are most customers satisfied with the API access, and analyzing elsewhere?

    ReplyDelete
  6. Hi Adam,
    Can Event Monitoring be used to track Delegated Admin activity while Logged In-AS? We are looking for a way to track what changes were made to records by the Delegated Admin.
    Thank you for your response.

    ReplyDelete
    Replies
    1. Yes! We added that log line specifically for that use case! Thanks!

      Delete
    2. Can we also track the old vs new value for changed fields?
      Would it be possible to get a sample log for the Login-AS functionality? Thank you.

      Delete
    3. Hi SFDCMaria, unlike field history tracking, we do not capture the state of data with event monitoring logs. However, we do track old and new values for login as related to setup changes only in the setup audit trail. This doesn't help with crud changes but does help with any metadata or org config changes.

      Delete
    4. Hi Adam,
      Could you please clarify what is the advantage of event monitoring vs setup audit trail? Setup changes can be found in audit trail. What additional info can be found in the event monitoring logs when it comes to Login-As?
      Thank you.

      Delete
    5. Two different features created at different times for different purposes. Setup audit trail is only limited to changes in setup which are often required for audits. Event monitoring is an extract and transformation of our app server logs which capture a significant amount more activity like each time a record is viewed.

      Delete
  7. Hi Adam, I am not able to create a lens for ApexCallout event type in Wave event monitoring app with pre built datasets. Is there a way to do that in Wave?

    ReplyDelete
  8. Hi Adam,I am not able to find Event Monitoring Wave App. Wave Analytics & Event Monitoring is already enabled.

    Can you please help me on this...

    ReplyDelete
    Replies
    1. If you DM me on twitter (atorman) or leave your email, I can have someone from the team help you troubleshoot.

      Delete
  9. How do you use Informatica to do the download for each event type seperately

    ReplyDelete
  10. hi Adam,

    Great Post!

    I was wondering how to use the event log file browser application, as per the trailhead?

    I cannot open the link they gave us, it is generating an error: https://salesforce-elf.herokuapp.com/event_log_files

    ReplyDelete
    Replies
    1. Did you try: https://salesforce-elf.herokuapp.com/

      Delete
  11. same result, it redirects to https://salesforce-elf.herokuapp.com/event_log_files

    the error is:

    We're sorry, but something went wrong.

    If you are the application owner check the logs for more information.

    ReplyDelete
    Replies
    1. Is it possible that your network blocks this ip or heroku apps? I’m still getting the live site at https://salesforce-elf.herokuapp.com/

      Delete
  12. Can you please list out a number of use cases for visualizing EventLogFile data? I am interested in creating dashboard(s) but I feel that there are probably a lot of use cases but I have no experience and so I am not taking full advantage of the data.

    ReplyDelete
  13. Hi Adam,

    Salesforce team informed us that Event Monitoring has been enabled now. How should i confirm that the feature is now available. Is there any new license added that i can see and verify or is there something in set up that says event monitoring for these event type is enabled for the ORG.

    Thanks
    Ashish Arora

    ReplyDelete
    Replies
    1. Hi Ashish, congratulations and thanks for getting event monitoring. Easiest way to see if you have it enabled is to look for the user permission on a profile or permission set. If you see view event log files, you should be good to go!

      Delete
  14. Thanks for the detailed post. Has anyone created Tableau dashboards to visualize the data? If so, can you please share?

    Thanks

    ReplyDelete
  15. Hi Adam. This is a great post. Very useful. Can you please help in answering below questions I have?
    1) My client wants the event log files to be published to Splunk in real time. Is this possible? I came to know that Real Time Events are in Pilot now but I couldn't find a lot of documentation around it. Any help here would be great.
    2) If the logs are to be sent to Splunk for Visualization , how will the external system know about Salesforce ID's? Like the userid or any object id's as response of SOQL query?
    3) Can I track the response I get when an external API call out is made? Basically my client wants to see 'what' information was sent and what information was retrieved from an external system when API calls are made.

    ReplyDelete
  16. Hey Adam, when i set up EventMonitoring App , I get this error.

    Your org does not currently meet minimum data requirements to proceed. Please fix the following issues before creating a 'Event Monitoring Analytics App' Application:
    In the 'ExtractContentObject' node, the 'ContentDocument' object doesn't exist or isn't accessible to the Integration User

    I know we have full license of it but this error is just going away. Any idea how to resolve it ? I can show you when you hv 15min to spare pls also

    ReplyDelete

Note: Only a member of this blog may post a comment.