8 Best Practices to keep the Release Train moving


We have moved far from the traditional way of developing and releasing software. All those few weeks-to-months of developed code that gets tested for a few weeks after which it’s packaged for production is no longer exciting for engineering and release teams. A highly iterative and agiltraine development process has evolved as a norm for companies dealing with products of any size and complexity.

Today, we see Facebook built and deployed in less than half an hour; Flickr having 10+ deploys a day and companies thriving for a daily push. All these web companies share some common continuous delivery principles, which drive their passion for short and frequent releases.

Here are 8 best practices adopted by some of the world’s best release engineering teams. While each of these practices alone can be a book of explanation, I tried to capture some “valuable clips”. Jez Humble and David Farley’s book on Continuous Delivery explores some of the concepts in great detail.

1. Automate Everything – Build, Test and Deploy: You should release your product with the mere push of a button. Automation in all stages of building, testing and deploying allows for better control and feeds faster feedback channels. CI tools like Hudson, CruiseControl, Jenkins automatically build from your SCM for every check-in and also run automated tests on a grid.

2. Zero Downtime Trunk: All engineers commit code to a mainline or trunk frequently and every commit is validated through a set of commit stage tests. A quick feedback cycle is facilitated through these validations, which allows for spotting out any defective code. Trunk should be available for all teams – dev, test and ops ALL time.

3. Fail Fast Commit Tests: The “Fail Fast” mantra of the commit tests is to validate every commit to the system through a set of tests to spot issues early in the cycle. The commit tests execute in less than 10 minutes and include compilation, unit, and integration tests. Hence, designing the right suite of tests for the commit stage is very critical. All tests should PASS to pass the commit stage and meet thresholds on other key code metrics like coverage, complexity etc.  E.g. 60-70% of unit test coverage.

4. Concurrent Test Execution: Tests that validate the functional and non-functional behaviors of the application should be automated. Functional Test Automation should target creation of stateless tests to allow for parallel execution. Tests that carry state and depend on a pre-condition or other application states will consume time and may demand sequential execution. Hence, a test should create its own data that doesn’t conflict with other tests and tear down/clean up in the end. Tests should be written to run on multiple nodes concurrently and deliver quick results. A handful of exploratory tests and tests verifying usability and acceptance should still be conducted manually.

5. Short Cycle Time: Keep your cycle time to the minimum, an important metric to always keep an eye on. This is the time it takes for a single change to move from check-in to release. This means reduced Time to Market for products that value it as a key strategy. Culture plays an important role in achieving this, the DevOps movement has shown high success rate in continuous deployment.

6. Emergency Stop: Applications demanding high availability and quality cannot afford to fail or break. In case of any undesired behavior, the application is rolled back to a known-good state by maintaining versions of builds. However, use of this emergency stop is considered only a last resort.

7. Canary Releasing: Releasing new versions only to a subset of your users to get faster feedback on how the new features are being used. E.g: Flickr started as a gaming site but soon realized that people are using it for sharing photos. This canary releasing helps in A/B testing which is an experimental approach to comparing between two different versions of the same product.

8. Release Monitoring System: Create a suite of highly efficient tools for monitoring and analyzing the status of the builds in different stages, the quality checks it has passed and visualizing after pushing into production for any errors, resource utilization and other relevant metrics. Without creating this kind of a dashboard, continuous delivery will remain to be a dream. Remember, your release train needs to have an intelligent real time failure detector!