05 January 2016

Using Asynchronous SOQL with Event Monitoring

Working with BigData requires re-thinking how you work on the Salesforce platform. Since you can assume that you're working with billions of records instead of thousands or even millions, common tasks like aggregate functions (e.g. SELECT count() FROM LoginEvent)  or complex queries (e.g. SELECT username FROM ApiEvent WHERE Soql LIKE '%SSN__c%)) are difficult to perform quickly. By quickly, a SOQL query needs to typically complete within two minutes or it will timeout.

The advantage of working with BigData on the Salesforce platform is that use cases involving billions of records are now achievable for a variety of use cases including:

  • long term adoption metrics
  • audit
  • performance monitoring
  • archiving

As a result, it's better to think of BigData as a data lake, where massive amounts of data can be processed at scale to meet any of the above use cases.

In a more concrete case of user login events which we now store in the Salesforce equivalent of BigData called a BigObject, it becomes important to re-think how you can work with the data at scale. This is especially important since not all capabilities that we've come to expect on the Salesforce platform, like operational reports or workflow, may be possible with a BigObject. It's a trade off of scale for limited platform capabilities.

For instance, rather than querying a BigObject like LoginEvent using a convention like synchronous SOQL via the API for use in real-time applications, it's better to use a new convention called asynchronous SOQL, currently in pilot, for extracting select use cases with analytic apps.

Asynchronous SOQL is similar to the Bulk API in the way it's job based and invoked using the API. But instead of the full CRUD capabilities with Bulk API designed primarily to work with the data off-platform, asynchronous SOQL provides a query-and-insert capability to retrieve sets of data and insert it into a structured object. Currently, this means moving data between BigObjects and custom objects on the platform.

This doesn't just mean BigObject only data but really any use cases where there is a significant amount of data including (but not inclusive of):

  • LoginHistory
  • SetupAuditTrail
  • AccountHistory (or any other field history object including Field Audit Trail)
  • Profile or PermissionSet
  • AccountShare (or any other sharing object)

For instance, you may want to create a subset set of data hourly, daily, or weekly in order to have the full transactionality of the Salesforce platform at your fingertips. Using this design pattern, we can utilize schedule and batch Apex to migrate subsets of data from a BigObject into custom objects. For instance, the ability to extract last week's worth of LoginEvents in order to report on it.

As a result, it's now possible to report on it using operational reports:

or with Wave for advanced exploratory capabilities:

or utilize workflow for generating tasks and alerts:

Asynchronous SOQL provides a new way of interacting with large sets of data on the Salesforce platform. For Event Monitoring use cases, it brings us closer to the kinds of analytic apps that enable CISOs and VPs of IT to understand the state and health of their organization.

To try this out, contact your Account Executive to have the Asynch SOQL pilot enabled in your org. Then just download the sample code and configurations in my Github asyncSOQL repository to get started with Event Monitoring use cases like Login Forensics, Data Leakage, or Apex Limit Events.


  1. This is precisely what OLAP/Analytical reporting is for. Can't the use cases move there? Salesforce can feed off the aggregated data.

  2. Nowadays, every organization has understood the significance of having a decent researcher available to them, which has expanded the interest for qualified and well-prepared information researchers.ExcelR Data Science Courses

  3. I have read your Excellent Post.This is Great Job. i have enjoyed your reading your post first time.
    i wnat to say thanks for this post..Thank You Satta King
    Satta King

  4. I am impressed. I don't think Ive met anyone who knows as much about this subject as you do. You are truly well informed and very intelligent. To see How to Change Route IP Address? PLease check link 192.168.l.l

  5. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    Data Science Certification in Bangalore

  6. Such a very useful article. Very interesting to read this article. I would like to thank you for the efforts you had made for writing this awesome article.
    Data Science Course in Pune
    Data Science Training in Pune

  7. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own Blog Engine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it.
    Data Science Course in Bangalore

  8. Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained!
    Data Science Training in Bangalore

  9. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    Data Analytics Course in Pune
    Data Analytics Training in Pune

  10. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    Business Analytics Course in Pune
    Business Analytics Training in Pune

  11. Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

    Data Science In Banglore With Placements
    Data Science Course In Bangalore
    Data Science Training In Bangalore
    Best Data Science Courses In Bangalore
    Data Science Institute In Bangalore

    Thank you..

  12. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    data science training in guduvanchery

  13. Extremely good write-up, I enjoyed reading your blog post, very good share, It's new game sims 4 skill cheats , are you ready for the rock climbing readmil? Go to xbox one.

  14. Mindblowing blog appreciating your endless efforts in developing a truly transparent content. Which probably the best one to come across disclosing the content which people might not aware of it. Thanks for bringing out the amazing content and keep sharing more further.

    360DigiTMG PMP Certification Course

  15. Thanks for the lovely blog. It helped me a lot. I'm glad I found this blog. Thanks for sharing with us, I too am always learning something new from your post.

    360DigiTMG Data Science Courses

  16. I just found this blog and hope it continues. Keep up the great work, it's hard to find good ones. I added to my favorites. Thank you.

    360DigiTMG Data Science Certification

  17. Thanks for sharing the wonderful blog information was of great help keep up the good work.
    360DigiTMG Data Science Training in Hyderabad

  18. I really liked every part of the blog and information was new thanks for sharing.
    Data Analytics Certification Training

  19. Found your post interesting to read. I cant wait to see your post soon. Good Luck for the upcoming update. This article is really very interesting and effective, data sciecne course in hyderabad


  20. Data Science Course Training in Hyderabad

    Data Science Training in Hyderabad, Data Science Course in Hyderabad, Data Science Training, Data Science Course, Data Science Institute, Data Science Certification

    Data Science Course in Hyderabad

    Data Science Training in Hyderabad
    Data Science Course Training in Hyderabad

    Advanced-Data science training with Free Internship & 100% Placement Assistance

    About the Data Science course Training in Hyderabad

    Data is everywhere, which is growing exponentially globally, and this may still grow at an accelerating rate for the foreseeable future. Businesses generate massive amounts of data within the type of blogs, messages, transaction documents, mobile device data, social media, etc. By using this data effectively, a firm can create vital value and grow their economy by enhancing productivity, increasing efficiency, and delivering more value to consumers.

    Data Science helps in combining the disruption into categories and communicating their potential, which allows data and analytics leaders to drive better results. Top businesses thought there is a necessity to research the data for significant benefits. They use the insights from data for the advantage of users.

    Human deciding is becoming increasingly inadequate to pander to a never-ending expansion of the data . However, Data Science and Machine Learning are excelling in solving highly complex data-rich problems. To think beyond the human brain and maintain the balance with the knowledge that's evolved, disrupted, and being employed the sectors altogether, data scientists foster new methodologies. Data scientists must try 'big data expeditions' to explore the data for previously undiscovered value - the first common application of data science. Typical applications include marketing segmentation, advertising, tweaking dynamic pricing models, or banks finding risks and adjusting the financial risk models.

    What are the Tools utilized in Data Science?

  21. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have for sharing.

  22. Her blog has given us valuable information to work on. Every tip in your post is amazing. Thank you so much for sharing. Keep blogging.

    360DigiTMG Business Analytics Course in Bangalore

  23. It's very educational and well-written content for a change. It's good to see that some people still understand how to write a great article!

    360DigiTMG Data Analytics Course in Bangalore

  24. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.data science training in Hyderabad


  25. I'm really thankful that I read this. It's extremely valuable and quite informative and I truly learned a great deal from it.

    Data Science Training Institute in Bangalore

  26. Additionally, this is an excellent article which I truly like studying. It's not everyday I have the option to see something similar to this.
    Data Science Course In Bangalore With Placement

  27. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

  28. Now is the perfect time to plan for the future and now is the time to be happy. I have read this article and if I can I would like to suggest some cool tips or advice. Perhaps you could write future articles that reference this article. I want to know more!

    Business Analytics Course in Bangalore

  29. Writing in style and getting good compliments on the article is hard enough, to be honest, but you did it so calmly and with such a great feeling and got the job done. This item is owned with style and I give it a nice compliment. Better!

    Data Analytics Course in Bangalore

  30. I found Habit to be a transparent site, a social hub that is a conglomerate of buyers and sellers willing to offer digital advice online at a decent cost.

    Artificial Intelligence Course in Bangalore

  31. Very informative article with valuable information found resourceful thanks for sharing waiting for next blog.
    Ethical Hacking Course in Bangalore


  32. Top quality blog with very informative information found very useful thanks for sharing.
    Data Analytics Course Online

  33. Nice Information Your first-class knowledge of this great job can become a suitable foundation for these people. I did some research on the subject and found that almost everyone will agree with your blog.
    Cyber Security Course in Bangalore

  34. Writing in style and getting good compliments on the article is hard enough, to be honest, but you did it so calmly and with such a great feeling and got the job done. This item is owned with style and I give it a nice compliment. Better!
    Cyber Security Training in Bangalore

  35. I want to thank you for your efforts in writing this article. I look forward to the same best job from you in the future. PMP Certification in Hyderabad

  36. Really fine and interesting informative article. I used to be looking for this kind of advice and enjoyed looking over this one. Thank you for sharing. Learn Tableau Course in Bangalore

  37. It's good to visit your blog again, it's been months for me. Well, this article that I have been waiting for so long. I will need this post to complete my college homework, and it has the exact same topic with your article. Thanks, have a good game.

    Artificial Intelligence Course in Bangalore

  38. You know your projects stand out from the crowd. There is something special about them. I think they're all really cool!. PMP Certification in Hyderabad

  39. Absolutely an extraordinary informative article. Hats off to you! The details which you have furnished is quite valuable. Tableau Course in Bangalore

  40. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    Best Institute for Data Science in Hyderabad

  41. I really enjoy every part and have bookmarked you to see the new things you post. Well done for this excellent article. Please keep this work of the same quality.
    Artificial Intelligence course in Chennai

  42. Thanks for posting the best information and the blog is very important.artificial intelligence course in hyderabad