SOPS and AGE: Practical Secrets Management in GitOps

The Secrets in Git Problem

GitOps means infrastructure as code, versioned in git. But what about secrets—database passwords, API keys, TLS certificates?

The obvious answer: don't commit secrets to git. The practical problem: how do you deploy them? Manual secret management doesn't scale, breaks reproducibility, and creates drift.

SOPS (Secrets OPerationS) solves this. You can commit secrets to git—encrypted—while maintaining the GitOps workflow.

How SOPS Works

SOPS encrypts the values in YAML or JSON files while leaving the structure readable:

# secrets.enc.yaml (encrypted with SOPS)
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: production
stringData:
  username: postgres
  password: ENC[AES256_GCM,data:Qr8yLKJ...,iv:abc123...,tag:xyz789...,type:str]
  database: clouds_of_europe
sops:
  kms: []
  age:
    - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB...
        -----END AGE ENCRYPTED FILE-----

The structure is visible. You can see it's a Secret, see the keys (username, password, database), but the values are encrypted. Only systems with the decryption key can read the actual values.

AGE Keys for Encryption

We use age (Actually Good Encryption) for key management. Age is simpler than GPG—no key servers, no web-of-trust complexity, just a keypair.

Generating keys:

# Generate age keypair
age-keygen -o age.key

# Public key (for encryption): age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
# Private key (for decryption): stored in age.key

Store the private key somewhere secure. For local development, that might be ~/.config/sops/age/keys.txt. For CI/CD, GitHub secrets or equivalent. For Flux in-cluster, a Kubernetes secret.

Configuration with .sops.yaml

At the repository root, .sops.yaml defines encryption rules:

creation_rules:
  # Test environment secrets
  - path_regex: gitops/infrastructure/test/.*.yaml$
    age: age1test...public-key-here

  # Production environment secrets
  - path_regex: gitops/infrastructure/production/.*.yaml$
    age: age1prod...public-key-here

Different environments use different keys. If the test key is compromised, production secrets remain safe. This is per-environment isolation—the same principle as separate kubeconfig files.

Flux Integration

Flux can decrypt SOPS-encrypted files automatically. You configure a decryption secret once:

apiVersion: v1
kind: Secret
metadata:
  name: sops-age
  namespace: flux-system
stringData:
  age.agekey: |
    # created: 2025-01-15T10:30:00Z
    # public key: age1ql3z...
    AGE-SECRET-KEY-1ABCD...

Then reference it in Kustomization resources:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m
  path: ./gitops/infrastructure/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Flux pulls encrypted manifests from git, decrypts them using the age key, and applies them to the cluster. You never handle plaintext secrets in CI pipelines.

Practical Workflow

Encrypting a new secret:

# Create plaintext secret
cat > database-credentials.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
stringData:
  password: "actual-password-here"
EOF

# Encrypt with SOPS
sops -e database-credentials.yaml > database-credentials.enc.yaml

# Commit the encrypted version
git add database-credentials.enc.yaml
git commit -m "Add database credentials for production"
git push

Editing an existing secret:

# SOPS decrypts, opens in editor, re-encrypts on save
sops database-credentials.enc.yaml

# Commit the changes
git add database-credentials.enc.yaml
git commit -m "Rotate database password"
git push

Rule: Always encrypt before commit. We use pre-commit hooks to catch accidentally committed plaintext secrets.

Key Rotation

Rotating keys periodically limits the blast radius if a key is compromised:

# Generate new age key
age-keygen -o age-new.key

# Re-encrypt all secrets with new key
find gitops/ -name "*.enc.yaml" -exec sops rotate -i {} ;

# Update .sops.yaml with new public key
# Update Flux secret with new private key
# Commit and deploy

We rotate keys annually and when team members with key access leave.

What This Enables

Disaster recovery: Entire infrastructure from git, including secrets. Fresh cluster? Clone repo, bootstrap Flux, done.

Audit trail: Every secret change is a git commit. Who rotated the database password? Check git log.

Review process: Secret changes go through pull requests. Someone reviews before production.

No credential sprawl: No secrets in CI configuration, no secrets in chat history, no secrets on developer laptops (except the age key itself).

SOPS with age keys isn't the only secrets solution, but it's the one that makes GitOps actually work for complete infrastructure.

Sources:

#sops #age #secrets #gitops #security