I assume that you have read our How to share variables amongst Azure Pipeline agents, which shared a few turbulent moments we experienced while troubleshooting this feature earlier this year. With this post we continue our troubleshooting excursion to highlight a few gotchas that have caused lots of head scratching.
Core Syntax
Here are two reminders to tattoo on your forearm, when referencing variables:
- Within a stage, map variables as:
dependencies.<stage>.<job>.outputs['<step>.<name>'] - Across stages, map variables as:
stageDependencies.<stage>.<job>.outputs['<step>.<name>']
Let us lift the pipeline bonnet and explore.
STEP 1: Define a variable to be shared
We define three variables, named secretValue1, secretValue2, and secretValue3 in our job called StageOneJobOne. Note that we explicitly name two of the steps and leave one as default. Sounds simple, but this will bite is later on.
- job: StageOneJobOne
pool:
vmImage: 'windows-latest'
steps:
- bash: echo "##vso[task.setvariable variable=secretValue1;isOutput=true]BINGO-1!"
name: SetVariable1
- bash: echo "##vso[task.setvariable variable=secretValue2;isOutput=true]BINGO-2!"
- bash: echo "##vso[task.setvariable variable=secretValue3;isOutput=true]BINGO-3!"
name: SetVariable3
- bash: 'env | sort'
The last task, replaces the Display all variables extension, I commonly used to dump all variables. A bonus is that unlike the extension, the bash task runs on any agent.
Extract from task log
...
BASH_SECRETVALUE2: BINGO-2!
...
SETVARIABLE1_SECRETVALUE1: BINGO-1!
SETVARIABLE3_SECRETVALUE3: BINGO-3!
...
If you cannot resolve a variable, add this task to determine if and in which shape it is included. As you may have noted our three variables are mapped slightly differently. The two generated by the explicitly names step have inherited the step names SetVariable1 and SetVariable3, whereas the other assumed the default task name, Bash. Assumptions one of the evil roots of the infamous 2AM-calls!
STEP 2: Reference variable in another job with the same stage
Next we reference the variables in another job and echo their values.
- job: StageOneJobTwo
dependsOn: StageOneJobOne
pool:
vmImage: 'ubuntu-latest'
variables:
var1: $[ dependencies.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
# Gotcha #1
var2: $[ dependencies.StageOneJobOne.outputs['secretValue2'] ]
steps:
- script: echo $(var1)
- script: echo $(var2)
name: GOTCHA_1
Looking at the stage log, we immediately notice that our first var1 variable has been resolved as expected, the second var2 variable is blank?!?
If you refer to your forearm and look at the tattoo for mapping a variable within the stage, you realise we are missing the step name. It is fairly easy to pinpoint this GOTCHA when you use the tools at your disposal, such as the
- bash: 'env | sort'task and the trustworthy log files.
STEP 3: Reference variable in another job in a different stage
Next we reference the variables in another job from another stage and echo the value. The sample shows the use of a stage and a job variable and highlights the importance of using your second tattoo, which uses stageDependencies... instead of dependencies... we used before. In fact, the sample intentionally uses both, to welcome GOTCHA #2.
- stage: StageTwo
dependsOn: StageOne
variables:
varStage: $[ stageDependencies.StageOne.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
jobs:
- job: StageTwoJobOne
pool:
vmImage: 'windows-latest'
variables:
var2: $[ stageDependencies.StageOne.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
var3: $[ dependencies.StageOne.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
steps:
- bash: 'env | sort'
- script: echo $(varStage)
- script: echo $(var2)
- script: echo $(var3)
name: GOTCHA_2
Looking at the job's log file, we immediately notice the GOTCHA.
Troubleshooting Checklist
When things go belly up with your variables, I recommend that you:
- Look at the job logs and check if variables have been prepared correctly.
- Run
- bash: 'env | sort'to display all variables. - Check that your steps creating the variables have a name.
- Check that your steps referencing the variables use the correct mapping as per the mapping tattoo.
- Use the new YAML-pipeline editor and highlight the stage, job, task, and variable names. It highlights reoccurrences very nicely. Saved me a lot of time today when the highlighting, or lack thereof, made me realise that job was named jobe ... easily missed when embedded in mapping hierarchies.
Last, but not least, here is the complete sample code for the pipeline we experimented with. Enjoy!
stages:
# ##################################################################################################
# STAGE ONE
# ##################################################################################################
- stage: StageOne
jobs:
# ------------------------------------------------------------------------------------------------
# STAGE 1 JOB 1
# ------------------------------------------------------------------------------------------------
- job: StageOneJobOne
pool:
vmImage: 'windows-latest'
steps:
- bash: echo "##vso[task.setvariable variable=secretValue1;isOutput=true]BINGO-1!"
name: SetVariable1
- bash: echo "##vso[task.setvariable variable=secretValue2;isOutput=true]BINGO-2!"
- bash: echo "##vso[task.setvariable variable=secretValue3;isOutput=true]BINGO-3!"
name: SetVariable3
- bash: 'env | sort'
# ------------------------------------------------------------------------------------------------
# STAGE 1 JOB 2
# ------------------------------------------------------------------------------------------------
- job: StageOneJobTwo
dependsOn: StageOneJobOne
pool:
vmImage: 'ubuntu-latest'
variables:
var1: $[ dependencies.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
# Gotcha #1
var2: $[ dependencies.StageOneJobOne.outputs['secretValue2'] ]
steps:
- script: echo $(var1)
- script: echo $(var2)
name: GOTCHA_1
# ------------------------------------------------------------------------------------------------
# STAGE 1 JOB 3
# ------------------------------------------------------------------------------------------------
- job: StageOneJobThree
dependsOn:
- StageOneJobOne
- StageOneJobTwo
condition: eq(dependencies.StageOneJobOne.outputs['SetVariable1.secretValue1'], 'BINGO-1!')
pool:
vmImage: 'macOS-latest'
variables:
var1: $[ dependencies.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
steps:
- script: echo $(var1)
# ------------------------------------------------------------------------------------------------
# STAGE 1 JOB 4
# ------------------------------------------------------------------------------------------------
- job: StageOneJobFour
dependsOn:
- StageOneJobOne
- StageOneJobTwo
pool: server
variables:
var1: $[ dependencies.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
# ##################################################################################################
# STAGE TWO
# ##################################################################################################
- stage: StageTwo
dependsOn: StageOne
variables:
varStage: $[ stageDependencies.StageOne.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
jobs:
- job: StageTwoJobOne
pool:
vmImage: 'windows-latest'
variables:
var2: $[ stageDependencies.StageOne.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
var3: $[ dependencies.StageOne.StageOneJobOne.outputs['SetVariable1.secretValue1'] ]
steps:
- bash: 'env | sort'
- script: echo $(varStage)
- script: echo $(var2)
- script: echo $(var3)
name: GOTCHA_2