In any software development project, the riskiest point is always deployment time. It's often the first time that new features come into contact with the real production environment, and this means it's a time when a lot of the possible failure modes are concentrated.
With infrastructure deployments - such as the typical Salesforce deployment scenario - the stakes are higher still, as the users are dependent on the development team to fix all of the most critical problems that crop up before they start to interfere with their jobs.
Here are some things that can be done to manage deployments and mitigate the risks.
This is the art of making deployment a consideration throughout the development cycle. The simplest form of preparation is to track what changes are being made during development at the time when they’re being made, so that it's possible to check that the deployment plan includes all the necessary changes for production.
Deployment checklists can help too: during a deployment they can be essential for keeping a team on track. They should be created by the developers of each component while the component is being developed. This ensures that the developers are always thinking about how the deployment will happen, and it also means the checklists are written at the time when the developers have the best knowledge of what might fail.
Ideally, during a production deployment, nobody will be doing anything they haven't done before. Improvisation should only be required if something completely unexpected happens. The best way to get the necessary experience is to deploy as often as possible. Salesforce's Sandbox feature provides the means to do this: a sandbox can be regularly refreshed to the current state of the production environment and used as a practice target to make sure that everything will deploy successfully. It can also give developers experience of deploying their work while it’s in progress.
While it's important to try to avoid the need for improvisation, it's also important to find ways to practice it in case something does go wrong. The sandbox is again ideal for this: when trying out deployments, it's inevitable that mistakes will sometimes be made. It's easy to refresh a broken sandbox from production again, but recovering from a similar situation with a live deployment is likely to be much more work, so it makes sense to use the sandbox to learn how to recover from failures and to work out how to improve things. It might even make sense to deliberately cause problems when deploying to a sandbox, in order to ensure that the team can handle any unexpected problems that might occur in production.
Shipping more often
One way to reduce the risks associated with deployments is to keep them small: try to structure projects so that instead of ending with a single large deployment of many features, they involve many deployments of individual features. Problems with smaller deployments are usually easier to fix and affect fewer users, and it becomes possible to validate the development of a project as it progresses. Additionally, deploying more frequently to production increases the amount of practice and experience a team has with handling deployments.
When deploying to production, Salesforce has a 75% test coverage requirement. One problem with code coverage is that it's a poor measurement of how effective a particular test suite is, and scrambling to reach a particular level can result in poorly-designed tests written to meet the coverage requirement rather than because of their value as tests.
Every piece of code should be written with an idea of how it’s going to be tested. A popular way to do this is test-first development, where the public API is written first, followed by the tests that can verify if it's working, and finally the actual implementation. With good test cases, this method makes it easy to see when a feature is complete, as all the tests will pass.
Tests should focus on the functionality that's visible to the user (note that, with shared code, the 'user' might be considered to be other developers). If a piece of code can't be tested from a user-visible place, it’s likely to be redundant, meaning that it can (and probably should) be removed.
Programming to tests, if done correctly, should always result in high code coverage: if the coverage is low but the tests are well-chosen, it's a sign that the code being written is unnecessary. With an effective test-first approach, the 75% coverage is reached not by writing redundant tests (which decreases overall code quality) but instead by removing redundant code (which will always increase code quality).
Much more can be written on deployment in general, and deploying on Salesforce in particular. However, these core principles of preparation, practice, shipping more often, and testing can be key in transforming deployment from an occasional, scary and error-prone process into a reliable and routine feature of the life of a development team.