24 March 2014

Using a bash shell script to solve basic administrative tasks

In my ideal world, there would be a simple yet powerful user interface for everything I needed to do.

However, while waiting for my grand utopian vision to come true, I'm still left finding anyway possible to solve the everyday problems that I have.

In particular, I have repeatable administrative tasks that I need to automate. For instance, the ability to mass export, dedupe, download, update, and delete data from my org.

So when there isn't a user interface for me to use, I resort to using the API and great tools like Workbench and Data Loader for all of my administrative needs. But even these API tools have their limitations, in particular automating tasks.

Recently, while trying to automate the download of very large CSV files, I turned to a  tool from a former lifetime, bash shell scripts. While not ideal compared to using user interfaces or API based tools, bash shell is exceedingly efficient when performing rudimentary, repetitive, and ridiculously simple tasks.

However, I ran into a problem trying to use a common bash app, curl, with the salesforce REST API. To use curl with the REST API, I need to use oauth. While oauth is incredibly powerful and secure, it turned out to be a little more difficult to use with curl than I originally expected.

With oauth, a user logs into salesforce from a client app, curl in this case. After logging in, the client app receives a security access token. With this access token, the app can continue to make API calls on behalf of the user. So to use the REST API, I needed an access token.

The problem was that oauth provides that access token in an embedded JSON response that has to be parsed.


While playing around with a variety of techniques to parse JSON, I found a great reference to pulling out the access token using regular expressions (regex).

if [[ "$response" =~ (\"access_token\"):\"(.+)\" ]]; then
    access_token="${BASH_REMATCH[2]}"
After that, including the access token in subsequent REST API calls was easy.
curl https://${instance}.salesforce.com/services/data/v29.0/query?q=Select+Id+From+Account -H "Authorization: Bearer ${access_token}" -H "X-PrettyPrint:1"

I created a github repo called curlREST to help bootstrap the creation of these bash shell scripts in the future. If you have the need to mass manage data easily, I definitely recommend checking it out. With very few changes, you can quickly automate the most simple administrative tasks.

Sometimes it's about using a powerful and simple user interface. But always, it's about solving the problems before us by any means possible.

2 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. Thanks, i build shell scripts using this method; those inturn runs in side Splunk to provide me cool adoption dashboards for SalesForce.
    its nice little project...:-)

    ReplyDelete