Part 1: Pipelines - Why bother and what are our nightmares and options?


Posted by Willy-Peter Schaub on Sat 19 December 2020

Pipelines enable engineering to continuously deliver value, map and improve their processes and workflows, promoting consistency and reliability across the organisation.

In this series we are going to invite you on our journey of grappling with hundreds of inconsistent and often conflicting continuous delivery pipelines, to evolving to unified pipelines, template-driven pipelines, and eventually self-service automation. We will break down our journey into these multiple parts:

Other pipeline posts:

Pipeline quick reference poster posts:


Why Pipelines?

"Continuous Delivery Pipeline Value Stream Mapping The Continuous Delivery Pipeline (CDP) represents the workflows, activities, and automation needed to shepherd a new piece of functionality from ideation to an on-demand release of value to the end user." - © Scaled Agile, Inc.

As alluded to by the quote from Scaled Agile, we are not talking about pipelines to carry oil, but pipelines that help us automate continuous integration and delivery tasks. Examples include the automation of guardrail automations, such as SonarQube, WhiteSource, and Building Code scans and validations.

CICD Pipeline

Pipelines enable engineering to continuously deliver value, map and improve their processes and workflows, promoting consistency and reliability across the organisation.

Snowflakes

A healthy DevOps mindset promotes the line of autonomy. Above the line the organization defines its vision and governance to ensure alignment with regulatory, legal, and other requirements. Below the line the engineering teams own their process, with full autonomy to plan WHO, WHEN, and HOW they will accomplish their work.

If, however, there is a lack of blueprints, design practices, and governance, each team will design and develop their pipelines slightly differently.

The outcome are unique snowflakes that promote rapid evolution (positive) and a diversity of pipelines that can become hard to maintain, support, and innovate (negative).

CICD Pipeline

With hundreds of continuous delivery pipelines the Sec and Ops in DevSecOps began to buckle detecting and fixing vulnerabilities and other guardrail leaks.

Emergence of Unified Pipelines

In 2018 we decided to grab the pipelines by their valves to tackle the spread of unique pipeline patterns by defining an Unified Azure Pipeline design pattern. The pattern promoted the following principles:

  • Automate everything automatable
  • Build once

We encourage engineering teams to create a release build artifact, with debug symbols published to our symbol server.

  • Continuous integration and delivery
  • Continuous streamlining and improvement
  • Maintain one build definition

Instead of a developer and release pipeline, create one unified pipeline that locks down the higher environments.

  • Maintain one release pipeline definition
  • Scan for vulnerabilities early, often, and fail fast
  • Streamlined approvals

By optimising our approvals, we cut down on the complexity and delay, we inherited from previous years, decades, ...

  • Test early, often, and fail fast
  • Traceability and observability of releases

Nobody wants a "where did this build come from" treasure hunt when joining a 2AM incident call.

What followed was a mind-numbing and expensive era of aligning all snowflakes to the unified design pattern, using the Azure Pipelines GUI editor to manipulate the pipeline json-based configuration. Even though we are using re-usable Task Groups and Variable Groups we had to invest thousands of error-prone clicks - there has to be a better way!?!

We managed to persue our goals of aligning with architecture and security guardrails; consistency through design practices, automation, and collaboration; simplicity to create maintainable pipelines; and enabling and empowering our common engineering system.

Hackathon triggers a course change

A radical hackathon idea in 2019 investigated latest technology trends that promised pipeline-as-code, templates, and other facinating features that promise to enable our ultimate goal of self-service automation. Our hackathon idea was not amongst the winners, but is one of the few ideas that continued to simmer and change the world of our continuous delivery pipelines.

It triggered a pipeline working group, awareness workshops, and even four laptop stickers to highlight unified pipeline, multi-stage, CI+CD, and self-service automation champions.

CICD Pipeline

Welcome YAML based pipelines, which we will introduce in part 2 of this series.


Series Bread Crumbs | Part 1, TOC | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 | Part 7 | Part 8 | Part 9 | Part 10 |