May 16, 2022
SOPS: Secure Secrets as Code
SOPS is a tool that provides the ease of hard-coded plain-text secrets with the security of a cloud-based vault to securely publish and managed secrets from inside your code. SOPS does this by generating encrypted files that can be safely committed to GitHub and bundled into your production deployments. These SOPS files are tracked in your repository to make sharing secrets easy between engineers while doubling as a localized vault running from inside your application at runtime. This makes adding, removing, and rotating secrets easier across both environments and teams by reducing the technical and human costs while avoiding the security implications of hard-coded secrets in open-sourced projects.
Hard-coded plain-text secrets
Committing plain-text secrets to your GitHub repository is the easiest way to manage secrets. If an API key needs to be updated or a new secret is added to your application to support a new feature every engineer can pull the latest working secrets from your main trunk. Very simple. In a similar manner, your production, test, development, staging, preview, or whatever other custom runtime environments you have read straight from your code. There is no concern about how secrets will be injected into the application before runtime or when they will be available. However, committing plain-text secrets only work in high trust environments like a private GitHub repository where engineers are expected to work in good faith. In a low trust environment like an open-sourced project, secrets are easily abused and so must be removed from any public facing code.
Environmental variables
Environmental variables seek to hide plain-text secrets from GitHub. Plain-text secrets are abstracted from the code itself, usually placed in untracked `.env` files, and injected into the application before runtime¹. These `.env` files exist outside of your shared, committed GitHub code and so each engineer and runtime gets its own unique set of untracked secrets. The most obvious human cost of this strategy is felt during onboarding when you must hunt down each of these plain-text secrets or when adding a new required secret to run your application. These examples are typically quick to overcome. But demonstrate the disruptive nature of untracked secrets injected as environmental variables when compared to tracked, hard-coded secrets. To mitigate this decentralized approach to secrets management, untracked secrets become centralized in the form of cloud-based vaults.
Cloud-based vaults
Vaults act as a centralized and secure location to fetch encrypted secrets on demand, which are then decrypted into usable plain-text secrets during runtime². Like hard-coded plain-text secrets inside your code, every engineer can fetch the latest working secrets from your shared vault. Tools like GCP or AWS Secrets Manager makes cloud vaults easier and more secure to setup and maintain long term, but vaults in general require more infrastructure to operate. A bonus of cloud-based vaults, though, is that they are permissions based. Meaning that you can grant engineers or applications access to either all or only some secrets while also easily being able to revoke access when needed. However, to fetch, authenticate, and decrypt secrets stored in a vault you need to pass in valid authentication credentials into your application. Unfortunately, in my experience, these credentials are best injected through untracked environmental variables. But also in my experience this is a worthwhile tradeoff because these credentials are relatively static and fewer in number than the number of secrets to run a feature rich application. This makes cloud-based vaults more appealing than environmental variables. Although, the plain-text secrets which are abstracted form the code are still untracked when compared to hard-coded plain-text secrets. This is the niche that SOPS fills.
SOPS
SOPS takes the best parts of hard-coded secrets, environmental variables, and vaults to securely store your secrets so that they can be published to public GitHub repositories³. SOPS works by using a private key to encrypt your secrets into a `.sops` file. These `.sops` file can then be committed to your public (or private) GitHub repository like hard-coded secrets and are tracked like any other piece of code. This means that when you add, remove, or rotate your secrets, every engineer or build can easily pick up the latest set of working secrets from the main trunk. Using these `.sops` generated files with the handful of untracked environmental variables required to decrypt your secrets lets the `.sops` file act like a vault from inside your applications.
When coupling SOPS with a key-as-a-service provider like AWS Key Management Service (KMS), you can create a more robust cloud-based permissioning system akin to a standard cloud-based vault with little additional overhead. After setting up a private key in the cloud, you can easily configure SOPS to use your new private key to then grant decryption (read) and encryption (write) permissions on an individual and per runtime environment level through the key provider. This let you easily onboard and offboard engineers or only give read permissions to production environments while still managing your secrets as another part of your code³.
Real world example
At ClickFlow, we migrated away from untracked secrets injected as environmental variables to SOPS so that we could confidently test, merge, and deploy our code. With untracked environmental variables, each new microservice we created required every engineer to either request the `.env` file or hunt down the plain-text secrets themselves. We were releasing new microservices and subsequently blocking engineers other daily: so it was obvious we needed to improve our we handled our secrets. For us, we settled on SOPS supported by AWS KMS. SOPS let us codify our development and production secrets for every microservice while allowing AWS KMS to on and offboard collaborators.
This switch from environmental variables to SOPS was night and day for our team because we no longer had to worry about setting up the development environment for microservices that we had little context about. Although, SOPS does have its limitations.
Limitations of SOPS
In the example above, I talk about the human and technical side of SOPS and how those impacted our team. One of the major benefits of SOPS at ClickFlow was the programmatic decryption of `.sops` files to run our microservices. The stated goal of SOPS as a tool is to make managing secrets simply and flexible between people and delivers on this goal exceptionally in my mind. However, these exist a gap between SOPS as a tool and programmatic decryption within many programming languages. Using SOPS with Terraform and Node, I have only come across a handful of community packages to decrypt `.sops` files at runtime:
- https://github.com/carlpett/terraform-provider-sops
- https://www.npmjs.com/package/sops-decoder
- https://www.npmjs.com/package/@1mill/sops (written by me)
Having to write one of those packages myself for Node, a popular language, I can only assume support is also lacking for other languages. This means that if you want to adopt SOPS you likely will have to roll your own decryptor function in your preferred language. If you have control over your runtime, like inside a Docker container, you can likely install and hook into the SOPS tool from your application but I digress. Ultimately this is to say that to programmatically decrypt `.sops` files you will need to explore and likely write a solution that works for your use case.
Conclusion
SOPS makes adding, removing, and rotating secrets easier across both environments and engineers. SOPS (1) keeps your secrets secure even when published to a public GitHub repository³, (2) reduces the number of untracked environmental variables required to run your application by acting like a vault, and (3) lets you grant and revoke access to your secrets when coupled with a provider like AWS KMS³. Even though SOPS does have its limitations around programmatic decryption that you will likely need to solve for your own use case, if you can solve for those limitations, adopting SOPS significantly reduces the technical and human costs of secrets management by making your secrets just another part of your code.
References
[1]: Medlock, Jim. (December 19, 2018). An Introduction to Environment Variables and How to Use Them. https://medium.com/chingu/an-introduction-to-environment-variables-and-how-to-use-them-f602f66d15fa
[2]: HashiCorp Vault. (April 1, 2022). What is a Vault? https://www.vaultproject.io/docs/what-is-vault
[3]: McGrath, Chris. (June 20, 2019). https://oteemo.com/hashicorp-vault-is-overhyped-and-mozilla-sops-with-kms-and-git-is-massively-underrated/
