By Pankaj Mehra
We often need to process data in Salesforce using batches — sometimes at the time of migration, sometimes on a more regular basis. The most common problems we used to run into were that of performance and speed when it came to finishing the batches by a deadline.
I faced a challenge in one of my Salesforce orgs where I needed to process 650,000 records in production through batch class in just 10 days. The batch processing was somewhat heavy, as it included roughly 4 Web Service callouts, XML parsing, and record processing in Salesforce. The Web Service callouts involved document processing, which required moving the document from one server to another.
Not only was the process time-consuming, but I was only able to process 550 records per hour with a single batch running. Because we only have 10 days of downtime in production, a 30-day process just wasn’t acceptable. We needed a way to accomplish more within our time frame.
The solution we came up with was to run concurrent batches. We created a stateful batch to initiate concurrent batches. The stateful batch queries all the IDs to be processed and splits them into 5 sets of IDs (5 being the concurrent batch limit in Salesforce). Once we had sets of IDs, we initiated 5 concurrent batches from finish method.
Through this process, batch performance has increased by 400% and we can now process 2,800 records per hour. I’ve included a code snippet to further explain: