We live in the age of software as a service, secret keys are everywhere. Yet somehow it is still normal to click-manage secrets. Smart people share them via signal,smarter people, share them via a password manager like Bitwarden. Super-advanced people punch them into GitHub UI and only let github actions have access to secrets. Infra-devops people spin up Vault or AWS secret manager or similar to inject secrets. In all situations secrets have a lifecycle that is completely disconnected from code, causing the two to get out of sync or worse.

Basic problem is that secrets need to be shared between devs and between pieces of infra running on different providers (in my case I have projects split between k8s, smallweb, cloudflare, Deno Deploy + CI on github). It seems completely insane to have copy/pasting secrets between all these providers. Yet the alternative of setting up Hashicorp Vault is even worse.

Last year I was looking for a way to setup a dev/prod split for a team and was breaking my brain on how to maintain separate secrets between dev and prod. Vault seemed like the only cross-infra way to do secrets, but way too operationally complex.

SOPS: for teams

Turns out there is another way: SOPS. I stumbled upon it after spending days searching for alternative secret-management strategies after trying the main-stream options. Eventually, some LLM informed me of existence of SOPS: an ex-Mozilla project to store secrets alongside code.

TLDR:

  1. SOPS is specialized for encrypting key/value formats like JSON, .env, YAML files where keys are stored in plaintext. This enables one to track history of secrets via git log without seeing their values

  2. SOPS-encrypted files are optimized for easiest possible git diffing (which is still hard when one encounters a merge conflict).

  3. Encrypted files have 1 or more “recipients”.

  4. Each SOPS file has an inner key that is encrypted such that every recipient can decrypt it.

  5. SOPS has multiple encryption backends for “recipients”, age is a good one for small-scale/disconnected dev. Age also allows one to treat existing SSH keys like native age keys, thereby leveraging existing [GitHub] SSH key infrastructure. This means that one easily encrypt a secret and share with a whole github team by making a list of ssh-public keys for all teammates. SOPS recently inherited ssh support from age.

  6. One can incrementally graduate to something like AWS KMS to tie in 2-factor, OIDC, etc. SOPS supports using KMS keys as recipients, which means you can leverage IAM roles and policies for more fine-grained access control and auditability.

SOPS seems fairly popular in the Nix community for personal projects. I use it for all my personal and team secret needs.

Hit the SaC

We all know that Infrastructure-as-Code is good. Here is my “recipe” for Secrets-as-Code:

  1. Use my github-to-sops to store secrets in git and share them with rest of your team.

  2. Generate a single age private key for every deployed environment as SOPS_AGE_KEY (eg github actions, cloudflare workers). You can generate a new key pair using age-keygen:

    age-keygen -o key.txt

    This will output the public key to standard output and save the private key to key.txt. Paste the private key once into whatever UI the vendor provides for secret management. Then either keep the public key as the cryptographic identity for that service or have it provide an endpoint that translates the private key into public key.

  3. In code use something like the Go SOPS library or our sops-age npm library to decrypt secrets at runtime. For example:

    import secrets_enc from "./secrets.enc.json" with { type: "json" };
import { decryptSops } from "sops-age";

const secrets = await decryptSops(secrets_enc);

  4. Profit. Now in your code you can choose to load secrets from secrets..enc.json or use whatever strategy that suits you, can even change it easily as your secret management is now always in sync with your code.

  5. Feel free to log the encrypted secrets for diagnostics :)

  6. Note you can still using something like sops exec-env to pass SOPS secrets as traditional env vars.

Store your secrets in the db

Same trick works wonderfully for storing passwords, etc in databases:

  1. Make a REST endpoint that returns age public key

  2. Allow one to write the encrypted key straight into the database (eg via REST, PostgREST or GraphQL). github uses this pattern for their secret management API.

  3. Feel free to log the encrypted value :)

  4. Now everyone can publish secrets using typical REST/database infra but only services with access to private key can decrypt secrets stored in DB

This pattern is sometimes used to implement GDPR compliance where instead of erasing data (which might be hard) one just throws away the keys.

Experience so far

I have been practicing these SOPS workflows for a year now. It was rough-going until SOPS mainlined SSH support a few weeks ago. It’s much easier to learn than Terraform, Ansible and much easier to integrate into IaC frameworks than any other way to do secret injection.

I think it’s amusing that we as an industry internalized that deploying special-snowflake infrastructure by hand is irresponsible, but click-managing secrets is still “THE WAY”.

Preemptive FAQ

Q: Is this secure?

A: Suppose we only rely on SSH keys to gate access to secrets. If one can break/obtain SSH keys to access to your code, then one can exfiltrate all your secrets accessed by code even if they are in another system.

Q: Is this more secure than my $cloud-managed-secret-management?

A: Unlikely. However, it’s likely more secure than an unsupported deployment of a more complex solution(example).

Q: Is SOPS perfect?

A: No. But it’s a time-tested solution with the cleverest/simplest Git integration. I don’t believe that there is a better tool for tracking secrets through code and infra.

Q: Should I trust this Taras guy on security?

A: I’m not an expert. I learned everything I know about security from LLMs and by following FiloSottile and practice.