Avoid Raw Secret Data in Environment Variables

Secrets, as noted in the kubernetes documentation, can be referenced by containers as environment variables or as files mounted on a data volume. Environment variables are very convenient to consume and adhere to the guidance of the 12 Factor App. However, there are accidental disclosure risks for data in environment variables.

Example Setup

To highlight this risk lets set up a small example. The complete code & configuration I reference in this example can be found on github. It contains 2 small golang applications; One of them is the rightful owner of the particular secret and accesses it via an environment variable. The other application reads the data from /var/run/docker.sock, writing all the containers it finds and the configured environment variables to the stdout stream.

Since the owner application requires an environment variable to be present, the kubernetes spec file deployment/owner-pod.yaml for this pod has the following YAML block:

  env:
    - name: KUBE_SECRET
      valueFrom:
          secretKeyRef:
            name: demoserectspy
            key: supersecretvalue

These settings take the value of the supersecretvalue key container in the demosecretspy kubernetes secret and assigns it to the KUBE_SECRET environment variable during pod creation within the cluster. It is a very standard and supported pattern in the kubernetes documentation. However, when we examine what the spy application does, it becomes apparent that this pattern has a flaw.

The spy application does not reference kubernetes secret in its spec file deployment/spy-pod.yaml. It does, however, mount the /var/run/docker.run file from the host, using a volume mount.

  volumes:
    - name: dockersock
      hostPath:
        path: /var/run/docker.sock

Effectively the above volume mount gives the spy pod knowledge about all the containers running on the host. Enumerating the configured environment variables is a single client API call in golang and are returned as a single string array. The spy application outputs all the variables it finds to stdout without any knowledge of their meaning.

time="2019-09-22T18:04:09Z" level=info msg="found container. id: 7429448b3f image: khaines/kubeenvsecrets-secretowner@sha256:d53c8d06f92310514c54fa0bf8a91bf25c89ec39213561c3b3011855352ab14f"
time="2019-09-22T18:04:09Z" level=info msg="found environment variables assigned to container: count=10"
time="2019-09-22T18:04:09Z" level=info msg="KUBE_SECRET=This is a secret value and should not be exposed in plain text or observed by any actor other than those that require the value\n"

This sample output highlights risk from using environment variables for secret values: accidental exposure. In the scope of monitoring & observability, it is a prevalent desire to know what containers are running on what hosts and their details. Software designed to help operations could accidentally view passwords or other secrets, writing them to logs or ship them off-premise to a central service.

Mitigation

A way to reduce this risk is to mount secrets into kubernetes pods using Volume Mounts. Volume mounts are still listed in container details via clients, however not their contents; Therefore, software reporting on container details will not accidentally report a password or other sensitive data.

To help the application stay aligned with 12 Factor App guidance, the environment variable that we started with needs to change. The environment variable can be used to describe the path to the secret, instead of the raw value itself. Using this approach achieves the flexibility of configuration being present via environment variables while adding a layer of indirection to avoid accidental disclosure. The /deployment/safe-owner-pod.yaml file demonstrates this detail:

volumes:
    - name: secretmount
      secret: 
        secretName: demoserectspy
  containers:
  - image: khaines/kubeenvsecrets-safesecretowner
    imagePullPolicy: Always
    name: safesecretowner
    volumeMounts:
      - name: secretmount
        mountPath: "/secrets"
    env:
      - name: KUBE_SECRET
        value: "/secrets/supersecretvalue"

When mounting the secret as a volume within the pod, the key string specified in the secret spec becomes the file name, and the base64 data becomes the content. Therefore for our application to load the correct file for the secret, the ENV value is in an abstract sense: {secret-mount-path}/{secret-key-name}. Because the ENV value is now a path to the data versus the raw value, there is a minor change needed in the application. The change required is to read the file path from the ENV and is shown in the /safesecretowner/main.go file.

Looking once more at the spy pod; the output it creates for the safesecretowner highlights the difference:

time="2019-09-22T18:04:09Z" level=info msg="found container. id: 8e83b7fd5d image: khaines/kubeenvsecrets-safesecretowner@sha256:ac2eb7acac2545c5263ef61ba5651e81dac7eb751f05b9e83c3b9843cc71b24f"
time="2019-09-22T18:04:09Z" level=info msg="found environment variables assigned to container: count=10"
time="2019-09-22T18:04:09Z" level=info msg="KUBE_SECRET=/secrets/supersecretvalue"
In Closing

Sensitive data artifacts are necessary to the function of modern application service environments. Accidental disclosure of this data into logs and other monitoring telemetry is a risk all maintainers of these environments face. It is the hope of this post that the reader takes away some new insight to help improve their current and future deployments.

comments powered by Disqus