External Secrets Operator

Now let's explore integrating with AWS Secrets Manager using the External Secrets operator. This has already been installed in our EKS cluster:

~$kubectl -n external-secrets get pods

NAME                                                READY   STATUS    RESTARTS   AGE

external-secrets-6d95d66dc8-5trlv                   1/1     Running   0          7m

external-secrets-cert-controller-774dff987b-krnp7   1/1     Running   0          7m

external-secrets-webhook-6565844f8f-jxst8           1/1     Running   0          7m

~$kubectl -n external-secrets get sa

NAME                  SECRETS   AGE

default               0         7m

external-secrets-sa   0         7m

The operator uses a ServiceAccount named external-secrets-sa which is tied to an IAM role via IRSA, providing access to AWS Secrets Manager for retrieving secrets:

~$kubectl -n external-secrets describe sa external-secrets-sa | grep Annotations

Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/eks-workshop-external-secrets-sa-irsa

We need to create a ClusterSecretStore resource - this is a cluster-wide SecretStore that can be referenced by ExternalSecrets from any namespace. Lets inspect the file we will use to create this ClusterSecretStore:

~/environment/eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: "cluster-secret-store"
spec:
  provider:
    aws:
      service: SecretsManager
      region: $AWS_REGION
      auth:
        jwt:
          serviceAccountRef:
            name: "external-secrets-sa"
            namespace: "external-secrets"

Set service: SecretsManager to use AWS Secrets Manager as the secret source

Use the $AWS_REGION environment variable to specify the AWS region where secrets are stored

auth.jwt uses IRSA to authenticate via the external-secrets-sa service account in the external-secrets namespace, which is linked to an IAM role with AWS Secrets Manager permissions

Lets use this file to create the ClusterSecretStore resource.

~$cat ~/environment/eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml \

| envsubst | kubectl apply -f -

Next, we'll create an ExternalSecret that defines what data should be fetched from AWS Secrets Manager and how it should be transformed into a Kubernetes Secret.

Let's inspect the ExternalSecret manifest that will be applied:

~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "catalog-external-secret"
  namespace: "catalog"
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: "cluster-secret-store"
    kind: ClusterSecretStore
  dataFrom:
    - extract:
        key: "$SECRET_NAME"

This tells the External Secrets Operator to pull the secret identified by $SECRET_NAME from the ClusterSecretStore and sync it into the catalog namespace every hour.

We'll also update our catalog Deployment to use these credentials:

Kustomize Patch
Deployment/catalog
Diff

~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../base-application/catalog
  - external-secret.yaml
patches:
  - path: deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/created-by: eks-workshop
    app.kubernetes.io/type: app
  name: catalog
  namespace: catalog
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: service
      app.kubernetes.io/instance: catalog
      app.kubernetes.io/name: catalog
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app.kubernetes.io/component: service
        app.kubernetes.io/created-by: eks-workshop
        app.kubernetes.io/instance: catalog
        app.kubernetes.io/name: catalog
    spec:
      containers:
        - env:
            - name: RETAIL_CATALOG_PERSISTENCE_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: catalog-external-secret
            - name: RETAIL_CATALOG_PERSISTENCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: catalog-external-secret
          envFrom:
            - configMapRef:
                name: catalog
          image: public.ecr.aws/aws-containers/retail-store-sample-catalog:1.2.1
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 3
          name: catalog
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5
            successThreshold: 3
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 512Mi
          securityContext:
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          volumeMounts:
            - mountPath: /tmp
              name: tmp-volume
      securityContext:
        fsGroup: 1000
      serviceAccountName: catalog
      volumes:
        - emptyDir:
            medium: Memory
          name: tmp-volume

         app.kubernetes.io/instance: catalog
         app.kubernetes.io/name: catalog
     spec:
       containers:
-        - envFrom:
+        - env:
+            - name: RETAIL_CATALOG_PERSISTENCE_USER
+              valueFrom:
+                secretKeyRef:
+                  key: username
+                  name: catalog-external-secret
+            - name: RETAIL_CATALOG_PERSISTENCE_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  key: password
+                  name: catalog-external-secret
+          envFrom:
             - configMapRef:
                 name: catalog
-            - secretRef:
-                name: catalog-db
           image: public.ecr.aws/aws-containers/retail-store-sample-catalog:1.2.1
           imagePullPolicy: IfNotPresent
           livenessProbe:
             httpGet:

~$kubectl kustomize ~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/ \

| envsubst | kubectl apply -f-

~$kubectl rollout status -n catalog deployment/catalog --timeout=120s

Let's examine our new ExternalSecret resource:

~$kubectl -n catalog get externalsecrets.external-secrets.io

NAME                      STORE                  REFRESH INTERVAL   STATUS         READY

catalog-external-secret   cluster-secret-store   1h                 SecretSynced   True

The SecretSynced status indicates successful synchronization from AWS Secrets Manager. Let's look at the resource specifications:

~$kubectl -n catalog get externalsecrets.external-secrets.io catalog-external-secret -o yaml | yq '.spec'

dataFrom:

  - extract:

      conversionStrategy: Default

      decodingStrategy: None

      key: eks-workshop-catalog-secret-WDD8yS

refreshInterval: 1h

secretStoreRef:

  kind: ClusterSecretStore

  name: cluster-secret-store

target:

  creationPolicy: Owner

  deletionPolicy: Retain

The configuration references our AWS Secrets Manager secret via the key parameter and the ClusterSecretStore we created earlier. The refreshInterval of 1 hour determines how often the secret values are synchronized.

When we create an ExternalSecret, it automatically creates a corresponding Kubernetes secret:

~$kubectl -n catalog get secrets

NAME                      TYPE     DATA   AGE

catalog-db                Opaque   2      21h

catalog-external-secret   Opaque   2      1m

catalog-secret            Opaque   2      5h40m

This secret is owned by the External Secrets Operator:

~$kubectl -n catalog get secret catalog-external-secret -o yaml | yq '.metadata.ownerReferences'

- apiVersion: external-secrets.io/v1beta1

  blockOwnerDeletion: true

  controller: true

  kind: ExternalSecret

  name: catalog-external-secret

  uid: b8710001-366c-44c2-8e8d-462d85b1b8d7

We can verify our catalog pod is using the new secret values:

~$kubectl -n catalog get pods

NAME                       READY   STATUS    RESTARTS   AGE

catalog-777c4d5dc8-lmf6v   1/1     Running   0          1m

catalog-mysql-0            1/1     Running   0          24h

~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .env'

- name: RETAIL_CATALOG_PERSISTENCE_USER

  valueFrom:

    secretKeyRef:

      key: username

      name: catalog-external-secret

- name: RETAIL_CATALOG_PERSISTENCE_PASSWORD

  valueFrom:

    secretKeyRef:

      key: password

      name: catalog-external-secret

Conclusion

There is no single "best" choice between AWS Secrets and Configuration Provider (ASCP) and External Secrets Operator (ESO) for managing AWS Secrets Manager secrets.

Each tool has distinct advantages:

ASCP can mount secrets directly from AWS Secrets Manager as volumes, avoiding exposure as environment variables, though this requires volume management.
ESO simplifies Kubernetes Secrets lifecycle management and offers cluster-wide SecretStore capability, but doesn't support volume mounting.

Your specific use case should drive the decision, and using both tools can provide maximum flexibility and security in secrets management.

Conclusion​

Conclusion