How we use the Salesforce APIs
Last updated
Last updated
Choosing the right API, especially for data retrieval, can be difficult. There are REST APIs, SOAP APIs, as well as two generations of Bulk APIs.
This page provides a summary of which APIs we use in different circumstances. There are some further edge cases we left out, but for the most part this should paint an accurate picture of how we interact with Salesforce.
For outbound syncs, we always use the Bulk v1 API. The reason for this is that it gives us access to the "Serial Mode" parameter, which can be used to prevent contention-related errors in busy orgs.
Salesforce can have many hundreds of objects, and often people want to just select all of them. However, querying hundreds of objects creates a big overhead.
So we first use the to get an estimated record count of all objects. If an object returns 0 records from this, we skip it. In most situations, this saves us from checking many hundreds of objects.
Then, if the stream is configured as Incremental and there is some stream state present, we run a SOQL query to determine the count of new changes. If that is 0, we skip it.
Then, we use the , provided that:
The object supports the Bulk APIs (not all do)
There are more than 2000 records to retrieve
We fall back to the regular REST for the remainder, to avoid the overhead of bulk jobs.
In most cases, we can use the nextRecordsUrl property to incrementally fetch the whole result set. However, there's one more scenario we have to cater for. The SOQL Query we submit contains all of the fields of the object, e.g. select Id,Name,..... from Account.
In some situations, this query can get so long that it causes Salesforce to throw an error. So we check the length, and if it's longer than 18,000 characters then we fall back to using the FIELDS(ALL) function. The caveat here is that you have to use LIMIT 200, and only offset by up to 2000. So we fetch 200 records at a time with a moving cursor value until we get no more results.