The Release Train

Release Engineering develops the capability in an organization to deliver new changes in the product to customer in a confident repeatable manner.

The Release Engineering team develops this process in a manner where subsequent releases can go out out confidently.

What is a Release Train?

A Release Train is a release engineering pattern that helps define how a single release process can be instantiated and followed through.

A Release Train takes the changes in the product through different stages including building, testing, compiling, and finally publishing.

Each team and organization would adapt this concept according to their needs and the nature of the software in question. But here are a few characteristics generally common among Release Teams:

• A Release Train is atomic. Once a train has started, you do not want to bring in any further changes.
• Only one Release Trains runs at a time. More than one concurrent Release Train results in confusion.
• A Release Train doesn't stop for more changes before it departs. Whatever's ready gets on the train. Whatever isn't, gets on the next train.
• If a change already in the Release Train stops it due to a failure or lack of functionality or a discovered bug, it is rejected so the rest of the changes can be delivered to customers.
• A Release Train may contain multiple versions of a single product that need to be patched. This is true for products that need to maintain and support multiple version lines for example. A single Release Train does not necessarily mean changes to only a single version or even a product variant.¹

Stages of the Release Train

Each Release Train typically goes through the following few stages:

1. Pre-work

During this stage, the release team would take inventory of changes submitted by the component teams. A template should typically contain a place to insert relevant release notes and any documentation along with details around whether the change break the customer workflows upon installation. This would also include pull requests that contain code changes and any version bumps for the components and libraries in the product.

Ideally, these pull requests come in to the release train already tested with a recent build of the product, but depending on the maturity of the journey of release engineering, this may not always be possible. We will visit this idea again later in a separate post.

The Release team, being the gatekeepers of the product, should establish guidelines on what kind of changes make into the product. So this is the time to reject any changes that may be breaking the product where drastic changes shouldn't be made.

2. Integration

This is the point where all of the approved requests and pull requests are merged into the mainline branch of the product. In our case, we use Git to maintain production branches in our repository. Once all changes are brought in, an internal build version is assigned to the build so we are ready to build and test.

The build numbers safeguard the teams from confusion if a change needs to be taken out. These build numbers are for internal use and separate from final version increments that customers track. Final product versions are bumped only when the build is completely ready to be shipped to customers.

3. Build, Compile and Test

This is the stage where the core of the machinery triggers to create the next build of the product and compile any necessary components. This build serves as the candidate for the entire release train unless a defect is found. That triggers the removal of the change introducing the defect and a re-run to create the build.

Once a build is ready, the testing begins. In the case of Tanzu Application Service, we test our builds across an array of IAASes and scenarios. This includes all major public cloud services along with fresh installation and upgrade scenarios. We also run tests on a completely internet-less environment for customers who use the product offline.

It's important to note that any defect found while testing should be debugged only enough to pinpoint which change introduced it. The idea is to keep the train moving and debugging to fix while the release train is going means that all other changes have to wait further. We'll go into this further in a later post.

Another note around comprehensiveness of test suites: since Release Engineering is not a QA team, it's important to accommodate tests in the running test suites that aim for 80-90% of the main scenarios. A Release Engineering system cannot provide 100% guaranteed coverage for all use cases in any complex product, and aiming for it is extremely expensive and sets the wrong expectations with the stakeholders.

4. Publish and Distribute

If all systems go green after running these tests, that build is ready to go. Publishing and distributing the build is generally straightforward, but here are a few things that typically happen during this stage:

• The build is uploaded and given access to public or customers in a Marketplace or App Store.
• The version is bumped. It's important to have a clear version policy depending on the needs.
• The documentation for the new version is published, along with release notes.
• Any license files are generated and attached to the product entry or the build. In some cases, it may be necessary to embed this in the build itself.

As you can imagine, this is not a trivial amount of work. And it requires careful deliberation to make sure the product contains the changes and the team has enough confidence to launch it out to the world.

The Release Train pattern provides a simple, repeatable workflow where if followed, removes confusion and makes everyone in the organization follow these stages to track progress. At Tanzu, we've invested in dashboards that allow anyone internally to know when a change will come out. This helps customers be informed as necessary.

1. Though at some point and when the scale demands, you may want to consider dividing up products and teams to handle Release Engineering for different products as necessary. ↩︎