Azure Text Substitution with Special Characters

It’s a common scenario that you need to perform text substitution in a Microsoft Azure pipeline, for example, in order to place a secret from the key-vault into the environment so that a running application can use it to connect to the database. Since it’s a password, it can contain all sorts of characters.

People commonly use the linux “sed” command for this task:

sed "s/find-text/replacement/" filename.yml

Looks simple enough, but suffers from the requirement that you need to perform escaping on the “replacement” text. It’s not obvious to me how to do this escaping in an azure pipeline.

I found this to be an odd and unwanted challenge, so I chose to use Powershell for this task. It has a simple, straightforward syntax, and as far as I can tell, doesn’t try to be “smart” with the text to the extent that you need to perform escaping on the replacement text.

Consider this example using Bash with the sed command:

- task: Bash@3
  displayName: Update output file with secrets variables
  inputs:
    targetType: 'inline'
    workingDirectory: '$(Build.SourcesDirectory)/my-app/manifest/'
    script: |
      sed -i "s/##MY-VARIABLE-NAME##/$(VARIABLE-VALUE)/" output.yml;

sed can run into problems if $(VARIABLE-VALUE) contains, say an “&” character, leading to unexpected substitution results which can “corrupt” the value being configured.

The equivalent powershell task doesn’t need special consideration for the content of the variable as far as I can tell, except of perhaps for quotes. I find this to be an acceptable trade-off.

- task: Powershell@2
  displayName: Updating output file with secrets or variables
  inputs:
    targetType: 'inline'
    workingDirectory: '$(Build.SourcesDirectory)/my-app/manifest/'
      script: |
        $output = Get-Content output.yml -Raw
        $output = $output -replace "##MY-VARIABLE-NAME##", "$(VARIABLE-VALUE)"
        $output | Out-File output.yml

The powershell version accomplishes the same as the sed version and doesn’t suffer care too much about the contents of the replacement string.

This happens over 3 steps:

• Read contents of “output.yml” into $output variable. -Raw flag says don’t process or convert the file in any way
• Replace all occurences of “##MY-VARIABLE-NAME## with the value of $(VARIABLE-VALUE)
• Write the output to the “output.yml” overwriting the previous contents.

If you don’t want to have odd forbidden characters in passwords in your azure pipelines or environment variables, I would avoid using sed and opt for powershell.

Notes

There appears to be a tool called “sd” (search and displace) written in rust that avoids some of the pitfalls of sed, and might be worth using if it’s easily available to you. See: https://github.com/chmln/sd