10 March 2015

ElfPy - a Tasty Little Script for Downloading Event Log Files on Multiple Platforms



We launched Event Monitoring with Event Log Files last November. Since then, we've had a lot of customers sign up to download their log files.

But recently, one of those customers reached out with a problem. He was a Salesforce administrator on a Windows machine. That meant he had to write code or use one of my stock Bash shell scripts. Most administrators aren't used to writing code and Windows as a platform doesn't support Bash shell scripts without a lot of elbow grease, duct tape, and spit.

To fix this, I created a Python script that runs on multiple platforms including Windows, Linux, and Mac OSX. You can download the script from my Github repo.  You will need to install Python version 2.7.9 on your machine first; however, that's far easier than trying to get a Bash shell script working with Cygwin.

Why python? Because it's easy to learn, it's easy to read, it has incredible library support, and most importantly, it supports multiple platforms including some I haven't even heard of!

To run the script, open a terminal (or cmd on Windows) and type:

python elf.py

or if you have multiple versions of Python, including 2.7.9, installed already:

python2.7 elf.py

The script is very simple and takes four prompts:
  1. Username
  2. Password (hidden)
  3. Date range
  4. Download directory
Beyond that, it's as easy as running the script from a terminal or command prompt and you're ready to start downloading Event Log Files on Mac, Linux, or Windows.

30 comments:

  1. Hi Adam -- I don't see any other comments -- how can this be possible?!

    Thanks so much for this -- I'm having some difficulties in my first attempt to run, but will do some more diagnosis before bothering you further. It can't be because I downloaded the default, Python 2.7.10 instead of your cited 2.7.9, can it? Can the dot version be that different?

    If I've piqued your curiosity, here's the output I'm getting:

    C:\Users\David\Downloads>python elf.py
    Username:
    dbrenn23@optonline.net.legacy1
    Password:

    Using user inputed username: dbrenn23@optonline.net.legacy1
    Traceback (most recent call last):
    File "elf.py", line 236, in
    download_elf()
    File "elf.py", line 112, in download_elf
    access_token, instance_url = login()
    File "elf.py", line 94, in login
    res = urllib2.urlopen(req)
    File "C:\Python27\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
    File "C:\Python27\lib\urllib2.py", line 437, in open
    response = meth(req, response)
    File "C:\Python27\lib\urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
    File "C:\Python27\lib\urllib2.py", line 475, in error
    return self._call_chain(*args)
    File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
    File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    urllib2.HTTPError: HTTP Error 400: Bad Request

    C:\Users\David\Downloads>

    ReplyDelete
    Replies
    1. I plead Sunday-Morning-Before-9-itis. Log indicates that I didn't put in my API Token. . Tried appending token to password - first attempt a no-go, but may still be user/typo error. If not, good opp for me to learn more about Python by building a separate Token prompt into your script. I'll be back...

      Delete
  2. Hi Adam,

    Tried to download the logs using your script with Python 279. It will not log me into Salesforce neither do I see a rejection in the SF login history with my user. The error python throws is:

    Traceback (most recent call last):
    File "elf.py", line 236, in
    download_elf()
    File "elf.py", line 112, in download_elf
    access_token, instance_url = login()
    File "elf.py", line 94, in login
    res = urllib2.urlopen(req)
    File "C:\Localdata\Programma's\Python279\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
    File "C:\Localdata\Programma's\Python279\lib\urllib2.py", line 431, in open
    response = self._open(req, data)
    File "C:\Localdata\Programma's\Python279\lib\urllib2.py", line 449, in _open
    '_open', req)
    File "C:\Localdata\Programma's\Python279\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
    File "C:\Localdata\Programma's\Python279\lib\urllib2.py", line 1240, in https_open
    context=self._context)
    File "C:\Localdata\Programma's\Python279\lib\urllib2.py", line 1197, in do_open
    raise URLError(err)
    urllib2.URLError:

    Regards,
    Jos

    ReplyDelete
    Replies
    1. Hi Jos,

      Sorry you're encountering this error. I didn't put enough error handling in the original script. Couple of suggestions:
      1. Even though you're not getting a login history failure row, try logging in as an API user with a profile that has ip restrictions: 0.0.0.0 - 255.255.255.255. I wouldn't keep this setting permanent but it will tell us if it's a security token issue.
      2. Add print res and print req statements into the login block
      3. Implement try catch in login block

      If that doesn't work, we can set up some time to talk via goto or google hangout to review what's happening. Thanks!

      AT

      Delete
  3. Thanks for the post. No issues for me after I updated my PATH environment variable on my Windows machine. Can you offer any suggestions on how to automate running this script in windows? How can we store the password securely?

    ReplyDelete
  4. I don't know of any good job schedulers on Windows unfortunately. I have a sample script for Linux to hide a password but not sure how on Windows unfortunately. Hope someone in the community who knows Windows better might have a good answer.

    ReplyDelete
    Replies
    1. Thanks for your reply to that. I am "Unknown" .. I may switch gears then to Linux. Can you provide that script? And I guess i could schedule the script via cron, right?

      Delete
    2. I usually use crontab on Linux. It's pretty easy to find tutorials online for it. And it's easier than Launchd on Mac for scheduled jobs imho...

      Delete
    3. I scheduled it to run using the Task Scheduler. This link might help anyone who hasn't figured it out : http://desktop.arcgis.com/en/arcmap/10.3/analyze/executing-tools/scheduling-a-python-script-to-run-at-prescribed-times.htm

      Delete
  5. Thanks for writing this script. I notice that it downloads the login and logout activities for all users. Does it also download other activities like "Exported Reports"?

    ReplyDelete
    Replies
    1. It does but only in Developer Edition orgs or if you have purchased the full Event Monitoring product offering for your production org where we support 32 event types.

      Delete
  6. Hi Adam,

    Thanks for the script, it's perfect!
    But sometimes it downloads just first four event types and nothing more.
    That's what it downloaded yesterday when I selected last 4 days:

    2016-03-17-ApexCallout.csv
    2016-03-17-ApexExecution.csv
    2016-03-17-ApexSoap.csv
    2016-03-17-API.csv
    2016-03-17-AsyncReportRun.csv
    2016-03-18-ApexCallout.csv
    2016-03-18-ApexExecution.csv
    2016-03-18-ApexSoap.csv
    2016-03-18-API.csv
    2016-03-18-AsyncReportRun.csv
    2016-03-19-ApexCallout.csv
    2016-03-19-ApexExecution.csv
    2016-03-19-ApexSoap.csv
    2016-03-19-API.csv
    2016-03-19-AsyncReportRun.csv
    2016-03-20-ApexCallout.csv
    2016-03-20-ApexExecution.csv
    2016-03-20-ApexSoap.csv
    2016-03-20-API.csv
    2016-03-20-AsyncReportRun.csv

    Could you please point on the possible solution?
    Thank you!

    ReplyDelete
    Replies
    1. we only generate files for event types where there's activity. For instance, if you didn't run any reports for a day, we wouldn't generate a report file.

      Is it possible, you haven't done anything via the UI in the org for several days?

      Delete
    2. Hi Adam,

      Thanks for your reply. The main thing is that there were other event types, I'm sure because I've checked them through heroku event log browser.
      I absolutely have no clue why it happens :(

      Delete
  7. Hi Adam,
    I'm having trouble Finding my Client id and Client secret in SFDC. Also, for password, I' putting in password and token. Is that right ? I'm getting a 400 error.
    Thank in advance fro your reply.
    Regards,
    Bill

    ReplyDelete
  8. Hi Adam,
    I've got the client id and secret set up but am still getting this error:
    Traceback (most recent call last):
    File "D:\SFDCMonitor\elf.py", line 232, in
    download_elf()
    File "D:\SFDCMonitor\elf.py", line 124, in download_elf
    res = urllib2.urlopen(req)
    File "C:\Python27\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
    File "C:\Python27\lib\urllib2.py", line 437, in open
    response = meth(req, response)
    File "C:\Python27\lib\urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
    File "C:\Python27\lib\urllib2.py", line 475, in error
    return self._call_chain(*args)
    File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
    File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    HTTPError: HTTP Error 400: Bad Request

    ReplyDelete
    Replies
    1. Hi Bill,

      Thanks for posting. Did you resolve this? One potential reason may be because of the Security Token? Did you try adding that into the password flow for the user? Thanks!

      AT

      Delete
  9. Hi Adam,
    I am not able to download data for Event Login.
    When i have inserted username & password, it is giving me below error.
    Using user inputed username: acutedge@cssny.org
    check point
    Traceback (most recent call last):
    File "elf.py", line 239, in
    download_elf()
    File "elf.py", line 115, in download_elf
    access_token, instance_url = login()
    File "elf.py", line 97, in login
    res = urllib2.urlopen(req)
    File "C:\Python27\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
    File "C:\Python27\lib\urllib2.py", line 435, in open
    response = meth(req, response)
    File "C:\Python27\lib\urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
    File "C:\Python27\lib\urllib2.py", line 473, in error
    return self._call_chain(*args)
    File "C:\Python27\lib\urllib2.py", line 407, in _call_chain
    result = func(*args)
    File "C:\Python27\lib\urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    urllib2.HTTPError: HTTP Error 400: Bad Request


    Please give any solution.

    ReplyDelete
    Replies
    1. Hi Arvind - may need som emore information. From the traceback, all I know is that the login failed, not why. If you can't get this to work, you can also try the elf browser (https://trailhead.salesforce.com/modules/event_monitoring/units/event_monitoring_download) which bootstrap creates a bash file for the same purposes. thanks!

      Delete
    2. Any luck on solving for this? I'm getting the same error. when trying to download data.

      Delete
  10. Hey Adam, Don't the log files change with each release? Is there a good way to parse the file to insert into Postgre? If the log files are constantly changing, then the Postgre schema would be broken with every release. Any suggestions on getting the log file data to Wave?

    ReplyDelete
  11. There are two fields on event log file object that provide field name and data type. We created them so that you could introspect the schema in sandbox prior to a major release to check for changes.

    ReplyDelete
  12. Adam,

    I do not know scripting but trying to use this and getting error:
    Traceback (most recent call last):
    File "C:\Users\esssrv\elf.py", line 239, in
    download_elf()
    File "C:\Users\esssrv\elf.py", line 115, in download_elf
    access_token, instance_url = login()
    File "C:\Users\esssrv\elf.py", line 97, in login
    res = urllib2.urlopen(req)
    File "C:\Python27\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
    File "C:\Python27\lib\urllib2.py", line 435, in open
    response = meth(req, response)
    File "C:\Python27\lib\urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
    File "C:\Python27\lib\urllib2.py", line 473, in error
    return self._call_chain(*args)
    File "C:\Python27\lib\urllib2.py", line 407, in _call_chain
    result = func(*args)
    File "C:\Python27\lib\urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    urllib2.HTTPError: HTTP Error 400: Bad Request

    C:\Users\esssrv>



    ReplyDelete
    Replies
    1. Adam or anyone, is there a way we can select just select event types?

      Attempted to update the script, but had no luck only pulling 2 events types that my company would like to retain. Currently running into a memory error when pulling APEX Triggers that I am not able to get around.

      At the start of July the script was pulling multiple days worth of values before I would run into this error.

      Delete
  13. Hi Mike,

    I adjusted the script in a way that might help you.

    With these lines you can select timing and event type at prompt:

    if len(day) < 1:
    day = 'Last_n_Days:2'
    print 'Using default date range: ' + day + '\n'
    else:
    print 'Using user inputed date range: ' + day + '\n'

    et = raw_input('\nType: \n')
    # query Ids from Event Log File
    url = instance_url+'/services/data/v33.0/query?q=SELECT+Id+,+EventType+,+Logdate+From+EventLogFile+Where+LogDate+='+ day +'+and+EventType+='+ et

    Hope that helps.
    Jos

    ReplyDelete
  14. Hi Adam,

    I am also facing same issue

    File "elf.py", line 236, in
    download_elf()
    File "elf.py", line 112, in download_elf
    access_token, instance_url = login()
    File "elf.py", line 94, in login
    res = urllib2.urlopen(req)
    File "C:\Python27\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
    File "C:\Python27\lib\urllib2.py", line 437, in open
    response = meth(req, response)
    File "C:\Python27\lib\urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
    File "C:\Python27\lib\urllib2.py", line 475, in error
    return self._call_chain(*args)
    File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
    File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    urllib2.HTTPError: HTTP Error 400: Bad Request

    What i noticied is it's taking user name in my case & password even if i try to type it is taking blank

    ReplyDelete
    Replies
    1. I had problems with v 2.7 and the TLS upgrade in orgs. Have you tried upgrading to v 3.4 of python? It may help and only require minor refactor.

      Delete
  15. Hello Adam,

    I'm also getting the "HTTPError: HTTP Error 400: Bad Request" error. I have Python 2.7.9 installed.

    I tried attaching my two factor code to my password since my API logins require two factor, but it wasn't working. Perhaps I'm misunderstanding how the API logins work. Any ideas?

    Thanks for your help!

    ReplyDelete
  16. Hi

    You have to create a Connected App under Setup->Apps and copy the Consumer key and Consumer secret in lines 41 and 42 of the python script. You should then be able t run it fine

    ReplyDelete

Note: Only a member of this blog may post a comment.