Testing ARM-templates in DevOps Pipelines

Last updated on July 30, 2021

Recently I have been doing many ARM-deployments with Azure DevOps yml pipelines. One of the challenges I faced was related to the time spent on debugging my ARM templates.

Update 30/7/2021: Remco Vermeer pointed me towards the ARM-template test toolkit, which offers similar functionality however also provides a large number of predefined validations and best-practices right out-of-the-box. I highly recommend to check this out!

I wanted to be able to run simple syntax validation tests on both my local ARM-template files as well as enforce certain outputs that are required as input for dependent pipeline steps. Basic unit and integration tests for my ARM-templates, which is especially useful if you aren’t deploying the files in the same pipeline. For instance when you package several parts of a solution independently and combine those packages later in a deployment pipeline.

After some research I found an interesting approach using the Pester PowerShell-framework. With basic cmdlets I can validate the syntax, the resources being created and the outputs that are generated by the templates.

Since I haven’t found a lot of examples I want to share my solution. First I’ll illustrate the project structure I setup in my repository:

my-solution/
├─ templates/
│  ├─ storageaccount.json
├─ tests/
│  ├─ storageaccount.json.tests.ps1

The storageaccount.json is just a basic storage account ARM-template. It outputs the storage account name and API-version for use in the pipeline:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {},
    "functions": [],
    "variables": {
        "storageAccountName": "storageaccount01",
        "storageAccountApiVersion": "2019-06-01"
    },
    "resources": [
        {
            "name": "[variables('storageAccountName')]",
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "[variables('storageAccountApiVersion')]",
            "location": "[resourceGroup().location]",
            "kind": "StorageV2",
            "sku": {
                "name": "Standard_LRS",
                "tier": "Standard"
            }
        }
    ],
    "outputs": {
        "storageAccountName": {
            "type": "string",
            "value": "[variables('storageAccountName')]"
        },
        "storageAccountApiVersion": {
            "type": "string",
            "value": "[variables('storageAccountApiVersion')]"
        }
    }
}

Next up is the Pester-file, storageaccount.tests.ps1:

BeforeAll {
    $deploymentFile = 'storageaccount.json'
    $TemplatePath = (Get-ChildItem -Filter $deploymentFile -Recurse).FullName
    try {
        $TemplateARM = Get-Content $TemplatePath -Raw -ErrorAction SilentlyContinue
        $Template = ConvertFrom-Json -InputObject $templateARM -ErrorAction SilentlyContinue
    } catch {
        #do nothing, failure will be handled by pester framework
    }
}
Describe 'ARM Template Validation' {
    Context 'File Validation' {
        It 'Template ARM File Exists' {
            Test-Path $templatePath -ErrorAction SilentlyContinue | Should -Be $true
        }

        It 'Is a valid JSON file' {
            $templateARM | ConvertFrom-Json -ErrorAction SilentlyContinue | Should -Not -Be $Null
        }
    }
    Context 'Template Content Validation' { 
        It "Contains all required elements" {
            $Elements = '$schema',
            'contentVersion',
            'functions',
            'outputs',
            'parameters',
            'resources',
            'variables' | Sort-Object                               
            $templateProperties = ($Template | Get-Member -MemberType NoteProperty | ForEach-Object Name).ToLower()
            $templateProperties | Should -Be $Elements
        }
        It "Creates the expected resources" {
            $Elements = 'Microsoft.Storage/storageAccounts'
            $templateResources = $Template.Resources.type
            $templateResources | Should -Be $Elements
        }
        It "Outputs the expected outputs" {
            $Outputs = 'storageAccountName',
            'storageAccountApiVersion' | Sort-Object
            $templateOutputs = $Template.Outputs | Get-Member -MemberType NoteProperty | Foreach-Object Name | Sort-Object
            $templateOutputs | Should -Be $Outputs | Sort-Object 
        }
    }
}

As you can see in the code above, I perform various tests:

• Check if the file exists and if it’s valid JSON
• Check if the file has all the required elements of an ARM-template
• Check if the expected resources are defined
• Check if the expected outputs are defined

With this Pester-file I am able to validate my ARM-templates before committing them to the repository. And I am able to embed the validation in my deployment pipeline:

trigger:
  branches: 
    include:
    - '*'

pool: 
  vmImage: windows-2019

stages:
- stage:
  displayName: Build
  jobs:
  - job:
    displayName: 'Run tests and publish artifacts'
    steps:
      - pwsh: if((Get-Module Pester -ListAvailable).Version -notcontains '5.2.1'){Install-Module Pester -Force}
        displayName: 'Install Pester'
      - pwsh: |
          Import-Module Pester
          Get-ChildItem .\Scripts\ -filter *.psm1 | ForEach-Object {Import-Module $_.FullName}
          $config = [PesterConfiguration]::Default
          $config.TestResult.Enabled = $true
          Invoke-Pester -Configuration $config -ErrorAction Stop
        displayName: 'Run Pester'
        ignoreLASTEXITCODE: true
        failOnStderr: true
        env: 
           SYSTEM_DEFAULTWORKINGDIRECTORY: $(System.DefaultWorkingDirectory)     
      - task: PublishTestResults@2
        displayName: 'Publish Test Results'
        inputs:
          testResultsFormat: NUnit
          testResultsFiles: 'testResults.xml'
          failTaskOnFailedTests: true

Using this approach I can achieve better quality control on ARM-templates, not just for my own work, but also that of my team.