Gearset Deploy is designed to retrieve metadata from Salesforce organizations in order to compare them and identify the operations required to deploy changes between them. This information is obtained through the Salesforce Metadata API, which can be queried for specific object types or names and provides a .zip file containing the requested metadata.
However, when registering large organizations, it’s easy to run into its limits. Calls retrieving metadata can only request up to 10,000 files or 400MB at a time, and attempting to retrieve more than this results in failure. Tools designed primarily for Apex development, such as the Force.com IDE and MavensMate, are less likely to hit these limits because they’re geared towards working only with a subset of the objects in your organization. When comparing and deploying entire organizations, though, it becomes increasingly likely that the limits will be exceeded. If you’ve used the Force.com Migration Tool, which is a wrapper around the Metadata API, you may have come across these limitations yourself.
We’ve heard from some users that Gearset Deploy is unable to register their organizations because they can’t be retrieved in one request, so we’ve looked into ways to solve this. In some cases, there’s a simple workaround: if you only want to compare and deploy specific object types, you can select just those types in the “Customize comparison” dialog box, and registration should be successful as long as the files returned are within the limits imposed by the API.
Sometimes this isn’t sufficient, though, as you want to compare differences between entire organizations. The first approach we considered for this scenario was to issue a series of retrieve calls, making sure none of them had more than 10,000 entries. As we don’t have any prior knowledge of the structure of a particular Salesforce organization, this would mean enumerating all the containers that map to files (e.g. individual custom objects, Apex classes, and layouts) before we could work out what to request each time. Listing this metadata can only be done for three object types per API call, so we were conscious that performing this discovery for every organization we register would use more calls than we’d prefer, as well as potentially slowing down the registration process.
As a compromise between consuming additional API calls during registration and working with larger organizations, we’ve initially decided to start registering Salesforce organizations by retrieving the metadata in several batches, split up by object type. This should mean that even if you have 6,500 Apex classes and triggers, 2,000 custom objects, and 2,000 Visualforce pages, Gearset Deploy should still be able to register your organization.
Due to the way the Metadata API works, it isn’t possible to simply retrieve each object in isolation and end up with a complete set of all the metadata in an organization. Some components are only present in the .zip file that’s returned if their dependencies have also been requested in the same retrieval request. For example, this is the case for field-level security in profiles:
The content of a profile returned by Metadata API depends on the content requested in the RetrieveRequest message. For example, profiles only include field-level security for fields included in custom objects returned in the same RetrieveRequest as the profiles.
Requesting and deploying the correct profiles and permission sets for individual objects therefore requires following a number of rules which are not immediately obvious. There are also other object types that have similar dependencies, including layouts and custom object translations, which are incomplete if custom objects they’re dependent upon are not requested at the same time.
Given that we can’t know in advance what dependencies exist in a particular organization, and failure to request them could mean they’re omitted from comparisons and deployments, we’ve decided to always request certain object types (including profiles and permission sets) in every batch. This will mean that, as long as all the appropriate types are selected for the comparison, Gearset Deploy should be able to detect all relevant changes and deploy them to your target organization, with the tradeoff that more objects being requested brings us back towards the API limits.
We hope this solution for retrieving metadata will work for most organizations, but it might not work for everyone, and we’re still considering other aspects of working with large organizations. If you’re finding that Gearset Deploy isn’t comparing your organization successfully, or if you have any other thoughts, we’d love to hear from you at [email protected]