Secret Management

Background

#

A few years ago I was exposed to the secret managment tool sops, which is a handy utility that can be employed to encrypt sensitive values in your yaml files. Simply this tool allows my secrets to be stored along side my Infrastructure as Code (IaC) or application code that uses secrets for my IT projects. I cannot begin to imagine the amount of hours (or days) this single tool has saved in making credentials available to my projects and preventing me from forgetting credentials for services that I later cannot access because I cannot remember the password. So how does sops work?

SOPS is a tool that allows you to securely store secrets in yaml or json file formats. The key is plaintext and the value is encrypted (think database_password - plaintext: ansdoifn909u03jfjdfopjds - encrypted), so you can see which secrets you have stored in the file without being able to gaze upon the plaintext value of the actual secret. This tool integrates with many key providers to perform encryption of secrets, like AWS KMS or Hashicorp Vault. So if you have a project that requires knowing some secret ahead of time then this tool allows you to easily stash things like API keys, passwords and even certificates in a yaml file with your other files.

To demonstrate how easily SOPS can be used to both store and use secrets in an automation project, I have created a small Ansible project that uses sops to store and retrieve secrets so they can be used at runtime to interact with services that need them. I believe this solution is a nice compromise between using external secret storage like Vault or KMS and saving secrets locally using a GUI-based password manager. Files encrypted by sops can be stored in your git project and version controlled, which is another bonus. Should you ever accidently overwrite a secret that is used in some orphaned system, you can go through your git history and provided you still have access to the key you can then decrypt that missing secret, which is a massive time saver.

Setup

#

To follow the instructions in this guide you will need these tools:

• GPG - for creating the key material that sops will use to encrypt your secrets. (Using this for demonstration purposes and I recommend using another key provider like KMS.)
• SOPS cli tool
• Ansible

A gpg key can be created on your local machine using the gpg --full-generate-key. I use the defaults for the encryption key (ECC/25519) and give the key a name and password.

In order for sops to work it will need a set of creation rules that are stored in a file called .sops.yaml. Essentially this config file specifies which key to use for encryption operations for a specific files, so for example, you can use a separate keys for production and development secrets files. In the script below a .sops.yaml file is created that allows sops to encrypt all files in the group_vars folder. The path_regex can be adjusted as necessary to only encrypt files that a specific extension like .enc for example.

1
2
3

creation_rules:
    -   path_regex: group_vars/*
        pgp: <Your GPG Key ID>

In order to allow Ansible to lookup values using a sops a small python script (plugin) is required. This script tells Ansible to use the python plugin whenever a jinja lookup function is called using ‘sops’ keyword. There may be other ways of achieving this effect but for most of my use cases the script is lightweight and allows me to quickly troubleshoot any issues involving sops within an Ansible playbook. By default Ansible will look for plugins in a lookup_plugins folder that is located with the playbook.

lookup_plugins/sops.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/usr/bin/env python3

import subprocess
import yaml
from ansible.plugins.lookup import LookupBase
from ansible.utils.display import Display


display = Display()

class LookupModule(LookupBase):
    def run(self, terms, variables=None, **kwargs):
        ret = []
        for term in terms:
            try:
                # Use SOPS to decrypt the file
                result = subprocess.run(['sops', '--decrypt', term], 
                                     capture_output=True, 
                                     text=True, 
                                     check=True)
                # Parse the decrypted YAML
                decrypted_data = yaml.safe_load(result.stdout)
                if decrypted_data:
                    ret.append(decrypted_data)
            except subprocess.CalledProcessError as e:
                display.error(f"Failed to decrypt {term}: {e}")
            except yaml.YAMLError as e:
                display.error(f"Failed to parse decrypted YAML from {term}: {e}")
        
        return ret 
EOF

Finally, the remaining files for the Ansible project are displayed in the code blocks below. These files include the main.yaml file under the group_vars folder which contains secrets used by the Ansible Playbook and the Playbook file itself, which contains tasks that display the plaintext of the encrypted secret.

1
2
3
4
5

# Example Secret
some_secret: very nice!
a_list:
- one
- two

group_vars/main.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

---
- name: Test SOPS
  hosts: localhost
  connection: local
  become: false
  tasks:
    - name: Test Secret
      debug:
        msg: "{{ lookup('sops', 'group_vars/main.yml') | json_query('some_secret') }}"
    - name: Test List
      debug: 
        msg: "{{ item }}"
      loop: "{{ lookup('sops', 'group_vars/main.yml') | json_query('a_list')  }}"

site.yml Ansible Playbook

The directory structure should look something like this:

1
2
3
4
5
6
7

.
├── group_vars
│   └── main.yml
├── lookup_plugins
│   └── sops.py
├── site.yml
└── .sops.yaml

Usage

#

At this stage we have setup the project to run a simple playbook that uses sops to decrypt secrets at runtime. However, we have not encrypted the variable file in the group_vars folder using sops, and running the playbook now will result in an error as sops cannot perform a decryption the file that doesn’t have any sops metadata.

1
2
3
4
5
6

$ ansible-playbook site.yml 
...

TASK [Test Secret] ***************************************************************************************
 [ERROR]: Failed to decrypt group_vars/main.yml: Command '['sops', '--decrypt', 'group_vars/main.yml']'
returned non-zero exit status 1.

In order to encrypt the groups_vars/main.yaml we can simply use the sops cli tool to perform the encryption of the file using this command sops -e -i groups_vars/main.yml. The -i encrypts the file in-place rather than outputing the encrypted contents to stdout and the -e flag tells sops to perform an encryption operation. After the file is encrypted, the playbook can be run using the ansible-playbook site.yml command and observe the secrets in the debug output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ ansible-playbook site.yml 
...

TASK [Test Secret] ***************************************************************************************
ok: [localhost] => {
    "msg": "very nice!"
}

TASK [Test List] *****************************************************************************************
ok: [localhost] => (item=one) => {
    "msg": "one"
}
ok: [localhost] => (item=two) => {
    "msg": "two"
}

Summary

#

I think this is a great way to easily manage secrets for small projects. Not only are secrets securely encrypted using your encryption method of choice, but by storing the secrets as yaml where the key is in plaintext you can easily see which secrets you have available, making observing differences in source control easier.

This method isn’t without its flaws however, you must remember to encrypt files before saving them to any remote repository (a pre-commit hook can come in handy to prevent this unwanted situation). And, anyone that has access to the key material can potentially decrypt the file, so if this is a KMS key or Azure Key Vault key that can be potentially accessed by a large number of admins then effort will be required to create policies and auditing to restrict access to only those that need to use the key. It is also worth mentioning that the whole sops file can be decrypted by the key at the time of writing, so it isn’t very granular to restrict access to individual secrets.

words are my own

Appendix

#

Commands to install sops

#

1
2
3
4
5
6
7

#Download the sops binary
curl -LO https://github.com/getsops/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64
# Move the binary in to your PATH
sudo mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops

# Make the binary executable
chmod +x /usr/local/bin/sops