8 Best Practices to keep the Release Train moving

We have moved far from the traditional way of developing and releasing software. All those few weeks-to-months of developed code that gets tested for a few weeks after which it’s packaged for production is no longer exciting for engineering and release teams. A highly iterative and agiltraine development process has evolved as a norm for companies dealing with products of any size and complexity.

Today, we see Facebook built and deployed in less than half an hour; Flickr having 10+ deploys a day and companies thriving for a daily push. All these web companies share some common continuous delivery principles, which drive their passion for short and frequent releases.

Here are 8 best practices adopted by some of the world’s best release engineering teams. While each of these practices alone can be a book of explanation, I tried to capture some “valuable clips”. Jez Humble and David Farley’s book on Continuous Delivery explores some of the concepts in great detail.

1. Automate Everything – Build, Test and Deploy: You should release your product with the mere push of a button. Automation in all stages of building, testing and deploying allows for better control and feeds faster feedback channels. CI tools like Hudson, CruiseControl, Jenkins automatically build from your SCM for every check-in and also run automated tests on a grid.

2. Zero Downtime Trunk: All engineers commit code to a mainline or trunk frequently and every commit is validated through a set of commit stage tests. A quick feedback cycle is facilitated through these validations, which allows for spotting out any defective code. Trunk should be available for all teams – dev, test and ops ALL time.

3. Fail Fast Commit Tests: The “Fail Fast” mantra of the commit tests is to validate every commit to the system through a set of tests to spot issues early in the cycle. The commit tests execute in less than 10 minutes and include compilation, unit, and integration tests. Hence, designing the right suite of tests for the commit stage is very critical. All tests should PASS to pass the commit stage and meet thresholds on other key code metrics like coverage, complexity etc.  E.g. 60-70% of unit test coverage.

4. Concurrent Test Execution: Tests that validate the functional and non-functional behaviors of the application should be automated. Functional Test Automation should target creation of stateless tests to allow for parallel execution. Tests that carry state and depend on a pre-condition or other application states will consume time and may demand sequential execution. Hence, a test should create its own data that doesn’t conflict with other tests and tear down/clean up in the end. Tests should be written to run on multiple nodes concurrently and deliver quick results. A handful of exploratory tests and tests verifying usability and acceptance should still be conducted manually.

5. Short Cycle Time: Keep your cycle time to the minimum, an important metric to always keep an eye on. This is the time it takes for a single change to move from check-in to release. This means reduced Time to Market for products that value it as a key strategy. Culture plays an important role in achieving this, the DevOps movement has shown high success rate in continuous deployment.

6. Emergency Stop: Applications demanding high availability and quality cannot afford to fail or break. In case of any undesired behavior, the application is rolled back to a known-good state by maintaining versions of builds. However, use of this emergency stop is considered only a last resort.

7. Canary Releasing: Releasing new versions only to a subset of your users to get faster feedback on how the new features are being used. E.g: Flickr started as a gaming site but soon realized that people are using it for sharing photos. This canary releasing helps in A/B testing which is an experimental approach to comparing between two different versions of the same product.

8. Release Monitoring System: Create a suite of highly efficient tools for monitoring and analyzing the status of the builds in different stages, the quality checks it has passed and visualizing after pushing into production for any errors, resource utilization and other relevant metrics. Without creating this kind of a dashboard, continuous delivery will remain to be a dream. Remember, your release train needs to have an intelligent real time failure detector!


4 thoughts on “8 Best Practices to keep the Release Train moving

  1. David Mytton December 30, 2012 / 7:59 pm

    Your points essentially boil down to “in-depth, automated testing is a prerequisite to continuous deployment”. The biggest fear with any deploy, automated or not, is that something will break as a result. The only real way to safeguard this is by automating tests as both unit tests of isolated areas of code but also integration tests to ensure that whole features are working. Only then can you be confident in moving forward with continuous deployment, going through all the additional steps you mentioned (rollbacks, monitoring, etc).

  2. beckysoto96 February 5, 2013 / 11:01 pm

    Reblogged this on My Oracle Rants and commented:
    Continuous delivery which is a goal of DEVOPS relies on automated testing. Too often CD is attempted with half thought through processes that rely on tools that don’t perform to the level of functionality required. Manoja Nara outlines the steps required for CD notice the importance of testing and not just any testing automated testing.

  3. beckysoto96 February 5, 2013 / 11:03 pm

    Continuous deployment is successful with automated testing and yet how many companies miss the importance of automated testing along with several of these other steps.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s