In this series we are going to invite you on our journey of grappling with hundreds of inconsistent and often conflicting continuous delivery pipelines, to evolving to unified pipelines, template-driven pipelines, and eventually self-service automation. We will break down our journey into these multiple parts:
- Part 1: Pipelines - Why bother and what are our nightmares and options? (this)
- Part 2: Pipelines - Introduction, variables and why spaces matter
- Part 3: Pipelines - Basic building blocks as templates and sprinkling on telemetry
- Part 4: Pipelines - Magic of queue time assembly
- Part 5: Pipelines - Blueprints to fuel consistency and enablement
- Part 6: Pipelines - Gotcha! The generic blueprint-based YAML pipeline simplicity
- Part 7: Pipelines - There is more! Simplicity and enablement, courtesy of the app-type blueprint-based YAML pipelines
- Part 8: Pipelines - From CI to CD and beyond in one pipeline
- Part 9: Self-service automation - A dream turns into reality
- Part 10: Pipelines - Meet our second generation app-type blueprints
- Part 11: Our road to OSS Blueprints - Suppress CD when pipeline runs within PR |
Other pipeline posts:
- Azure DevOps Experimentation - YAML Conditionals, Parameters, and Triggers
- How to share variables amongst Azure Pipeline agents
- Pipeline-as-code wrapped with Pull Requests
- Gotchas when share variables with Azure DevOps stages and jobs
- Why do we care about infrastrucure-as-code (IaC)?
Pipeline quick reference poster posts:
- Quick Reference Sheet for Application-type Blueprint-based Pipelines
- Quick Reference Sheet for Pipeline Terminology
- Quick Reference Sheet for YAML and Generic Blueprint-based Pipelines
Why Pipelines?
"Continuous Delivery Pipeline Value Stream Mapping The Continuous Delivery Pipeline (CDP) represents the workflows, activities, and automation needed to shepherd a new piece of functionality from ideation to an on-demand release of value to the end user." - © Scaled Agile, Inc.
As alluded to by the quote from Scaled Agile, we are not talking about pipelines to carry oil, but pipelines that help us automate continuous integration and delivery tasks. Examples include the automation of guardrail automations, such as SonarQube, WhiteSource, and Building Code scans and validations.
Pipelines enable engineering to continuously deliver value, map and improve their processes and workflows, promoting consistency and reliability across the organisation.
Snowflakes
A healthy DevOps mindset promotes the line of autonomy. Above the line the organization defines its vision and governance to ensure alignment with regulatory, legal, and other requirements. Below the line the engineering teams own their process, with full autonomy to plan WHO, WHEN, and HOW they will accomplish their work.
If, however, there is a lack of blueprints, design practices, and governance, each team will design and develop their pipelines slightly differently.
The outcome are unique snowflakes that promote rapid evolution (positive) and a diversity of pipelines that can become hard to maintain, support, and innovate (negative).
With hundreds of continuous delivery pipelines the Sec and Ops in DevSecOps began to buckle detecting and fixing vulnerabilities and other guardrail leaks.
Emergence of Unified Pipelines
In 2018 we decided to grab the pipelines by their valves to tackle the spread of unique pipeline patterns by defining an Unified Azure Pipeline design pattern. The pattern promoted the following principles:
- Automate everything automatable
- Build once
We encourage engineering teams to create a release build artifact, with debug symbols published to our symbol server.
- Continuous integration and delivery
- Continuous streamlining and improvement
- Maintain one build definition
Instead of a developer and release pipeline, create one unified pipeline that locks down the higher environments.
- Maintain one release pipeline definition
- Scan for vulnerabilities early, often, and fail fast
- Streamlined approvals
By optimising our approvals, we cut down on the complexity and delay, we inherited from previous years, decades, ...
- Test early, often, and fail fast
- Traceability and observability of releases
Nobody wants a "where did this build come from" treasure hunt when joining a 2AM incident call.
What followed was a mind-numbing and expensive era of aligning all snowflakes to the unified design pattern, using the Azure Pipelines GUI editor to manipulate the pipeline json-based configuration. Even though we are using re-usable Task Groups and Variable Groups we had to invest thousands of error-prone clicks - there has to be a better way!?!
We managed to persue our goals of aligning with architecture and security guardrails; consistency through design practices, automation, and collaboration; simplicity to create maintainable pipelines; and enabling and empowering our common engineering system.
Hackathon triggers a course change
A radical hackathon idea in 2019 investigated latest technology trends that promised pipeline-as-code, templates, and other facinating features that promise to enable our ultimate goal of self-service automation. Our hackathon idea was not amongst the winners, but is one of the few ideas that continued to simmer and change the world of our continuous delivery pipelines.
It triggered a pipeline working group, awareness workshops, and even four laptop stickers to highlight unified pipeline, multi-stage, CI+CD, and self-service automation champions.
Welcome YAML based pipelines, which we will introduce in part 2 of this series.
Series Bread Crumbs | Part 1, TOC | Part 2 | Part 3 | Part 4 | Part 5 | Part 6 | Part 7 | Part 8 | Part 9 | Part 10 | Part 11 |