Overview

With the increased market use of Kubernetes as a cloud managed service and strict data compliance regulations that include the Payment Card Industry Standard (PCI DSS) and the Health Insurance Portability and Accountability Act (HIPPA), organisations are dependent on encrypted data as a requirement for Kubernetes workloads using persistent storage.

To accommodate these requirements, StorageOS has further improved its core Encryption at Rest functionality in V2.4 with Encryption and Decryption handled by the CPU’s accelerated instruction set instruction set, providing accelerated data at rest for both encryption and decryption for AES-256 with AES-XTS encryption.

StorageOS, a market leader in the field of cloud native persistent storage provides a Kubernetes native CSI interface, facilitating high performance, data availability and advanced storage features that historically were only available in hardware-based enterprise class storage arrays.

The following example demonstrates the use of Encryption at Rest in Kubernetes with StorageOS –

1. Setup and Configuration

For this setup, we have a standard Kubernetes cluster that has StorageOS installed through the one-shot installation process

$ kubectl get nodes
NAME             STATUS   ROLES                      AGE     VERSION
storageos-k8s1   Ready    controlplane,etcd,worker   6d22h   v1.17.17
storageos-k8s2   Ready    controlplane,etcd,worker   6d22h   v1.17.17
storageos-k8s3   Ready    controlplane,etcd,worker   6d22h   v1.17.17

2. Setup of an Encrypted and Unencrypted Volume

We setup both an encrypted and unencrypted volume to show the variance in configuration and how data at rest applies to storage within the Kubernetes infrastructure –

Creating an Encrypted and UnEncrypted PVC/PV - kubectl apply -f- <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: encrypted
  labels:
    storageos.com/encryption: "true"
spec:
  storageClassName: fast
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: unencrypted
spec:
  storageClassName: fast
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF
persistentvolumeclaim/encrypted created
persistentvolumeclaim/unencrypted created

StorageOS simplifies the setup of encrypted volumes with a single change required to the standard workflow.   The use of the encryption label as seen above in the setup of the first Persistent Volume Claim (PVC).

This simple change provides an encrypted volume along with the associated encryption keys required for encryption when using StorageOS.

3. Viewing Encryption Keys

Viewing the volumes with a kubectl describe shows that the encrypted volume has additional annotations relating to the encryption keys used for encryption at rest –

🔍 Showing PVC Annotation - encrypted - kubectl describe pvc/encrypted
Name:          encrypted
Namespace:     default
StorageClass:  fast
Status:        Bound
Volume:        pvc-3fd15067-992e-4d93-b816-f686546025d1
Labels:        storageos.com/encryption=true
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               storageos.com/encryption-secret-name: storageos-volume-key-2af02d26-4913-4a66-9f4a-0aae9589d8d4
               storageos.com/encryption-secret-namespace: default
               storageos.com/storageclass: 40579749-5ae8-44ae-8ae2-55a3858d4e15
               volume.beta.kubernetes.io/storage-provisioner: csi.storageos.com

Correspondingly, the Kubernetes Secret can be viewed to access the encryption key data specific for this volume-
🔍 Showing Encryption Key Secret for PVC encrypted - kubectl get secret storageos-volume-key-2af02d26-4913-4a66-9f4a-
0aae9589d8d4 -o yaml
apiVersion: v1
data:
  hmac: 9jqbnLGA9SfG+AhXzgZ9e49d+VrfJa3ki9C8eZOhubA=
  iv: JS0Cq1+0C/nnkL7plErbsZx8BHo28ies48sURXm4vAo=
  key: 8GkpcbBMQFAGdR4HeiYVsYa86NzFy56xLo/GgObIblt9ryZCGNRMulNv03+ckJP+tEGBEi1Qh1Dd6rokQhoYWw==
  vuk: XYZ2qitwuEFhaJPLTl3u9KSmsljNOod0+VO4FiLlWlzSuoFj1blZJOAZ27lhXz8x9PncX//5dJE3JkiO9VThSloSwh12uYWeKsCBlNbyXZ0=
kind: Secret
metadata:
creationTimestamp: "2021-04-30T10:03:35Z"
labels:
  app.kubernetes.io/component: storageos-api-manager
  app.kubernetes.io/managed-by: storageos-operator
  app.kubernetes.io/name: storageos
  app.kubernetes.io/part-of: storageos
  storageos.com/pvc: encrypted
 name: storageos-volume-key-2af02d26-4913-4a66-9f4a-0aae9589d8d4
 namespace: default
 resourceVersion: "15729433"
 selfLink: /api/v1/namespaces/default/secrets/storageos-volume-key-2af02d26-4913-4a66-9f4a-0aae9589d8d4
 uid: 4ae921d4-1823-4b79-9261-ca1e075fa3b0
type: Opaque

When using encryption with StorageOS, encryption & decryption keys are fully under the control and management of the consumer.  The use of a native Kubernetes construct for key secret management provides the benefit of native integration with Kubernetes KMS providers, therefore providing further compliance and regulation as desired.

Encryption key access, management and availability is a differentiator between StorageOS and other persistent storage services offered by Cloud Providers as each volume has its own individual key whilst being under full control of the end-user.

4. Demonstrating Encryption at Rest with ‘The Great Gatsby’

To fully appreciate Kubernetes Encryption at Rest and the capabilities provided with StorageOS, we make use of ‘F. Scott Fitgerald’s’ famous book, the ‘The Great Gatsby’, a text widely considered as a literary masterpiece that was recreated as a Hollywood film.

The container image spurin/gatby:latest available on DockerHub (source available via GitHub), performs the task of downloading the text of the book to /data/gatsby.txt upon initialisation.

To demonstrate encryption at rest in Kubernetes, we use two StatefulSets, unencrypted and encrypted, each utilising the spurin/gatsby:latest image, with a persistent data volume mounted as /data.  When the container starts, the Great Gatsby text will be saved to the StorageOS persistent volumes.

Creating an UnEncrypted and Encrypted StatefulSet - kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: unencrypted
spec:
  selector:
    matchLabels:
      app: unencrypted
  serviceName: unencrypted
  replicas: 1
  template:
    metadata:
      labels:
        app: unencrypted
    spec:
      volumes:
      - name: unencrypted
        persistentVolumeClaim:
          claimName: unencrypted
      containers:
        - name: unencrypted
          image: spurin/gatsby:latest
          imagePullPolicy: Always
          volumeMounts:
          - mountPath: "/data"
            name: unencrypted
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: encrypted
spec:
  selector:
    matchLabels:
      app: encrypted
  serviceName: encrypted
  replicas: 1
  template:
    metadata:
      labels:
        app: encrypted
    spec:
      volumes:
      - name: encrypted
        persistentVolumeClaim:
          claimName: encrypted
      containers:
        - name: encrypted
          image: spurin/gatsby:latest
          imagePullPolicy: Always
          volumeMounts:
          - mountPath: "/data"
            name: encrypted
---
EOF
statefulset.apps/unencrypted created
statefulset.apps/encrypted created

5. Verifying Data

From the Kubernetes consumption viewpoint, the use of Encryption at Rest is transparent and can be visualised by executing commands on the unencrypted and encrypted pods –

🔍 Checking Gatsby MD5 UnEncrypted - kubectl exec -it unencrypted-0 -- md5sum /data/gatsby.txt
c71f692044dbb4d89df2444ce8cccb35  /data/gatsby.txt

🔍 Checking Gatsby MD5 Encrypted - kubectl exec -it encrypted-0 -- md5sum /data/gatsby.txt
c71f692044dbb4d89df2444ce8cccb35  /data/gatsby.txt

🔍 Checking Gatsby Text UnEncrypted - kubectl exec -it unencrypted-0 -- grep -B1 'put together' /data/gatsby.txt
"They're a rotten crowd," I shouted across the lawn. "You're worth the
whole damn bunch put together."

🔍 Checking Gatsby Text Encrypted - kubectl exec -it encrypted-0 -- grep -B1 'put together' /data/gatsby.txt
"They're a rotten crowd," I shouted across the lawn. "You're worth the
whole damn bunch put together."

In both cases they return the same checksum for the data (c71f692044dbb4d89df2444ce8cccb35), they also both match a famous quote from the book.

To truly see the benefits of Encryption at Rest, we need to look behind the scenes.  For this, we’ll require access to the Kubernetes worker nodes where StorageOS runs.

6.  Data at Rest on the Kubernetes Nodes

In this example our unencrypted data resides on storageos-k8s2 under /var/lib/storageos/data/dev1/vol.201427.*.blob (with these files, relating to the unencrypted /data filesystem).  If we check for the same text from the book using the unix strings command, we can see the following from a data at rest viewpoint –

[email protected]:~# strings /var/lib/storageos/data/dev1/vol.201427.*.blob | grep -B1 -m 1 "put together"
"They're a rotten crowd," I shouted across the lawn. "You're worth the
whole damn bunch put together."

By accessing the Kubernetes worker node, we’re operating at an infrastructure level.  What could be deemed as sensitive information is both stored and easily accessible for those with access to the underlying components.  Should the Kubernetes node be terminated or the infrastructure recycled without due diligence there is a risk of data leakage.

If we were to execute the same command on the volume with encryption at rest (which resides under /var/lib/storageos/data/dev1/vol. 32831.*.blob), the equivalent data is inaccessible, thus safeguarding from data leakage –

[email protected]:~# strings /var/lib/storageos/data/dev1/vol.32831.*.blob | grep -B1 -m 1 "put together"
[email protected]:~#

Running strings against the entire datastore also demonstrates that encryption is in place –

[email protected]:~ # strings /var/lib/storageos/data/dev1/vol.32831.*.blob | head -20
M:d~
p;d|
5SB\9 4_B
T1Ot
#lN]8KX
4Rxc
@'{-
b,'I
p92`
(,^M
ya    Z=
>w` _
^TTA
T^ Y
fdPG
:N]j
Q&Zn
0v/ ]
SnYu
Hg}

Conclusion

For many organisations, data encryption is the barrier that prevents the both the use of Kubernetes and/or Cloud adoption for stateful applications.  StorageOS solves the problem both of encrypted data, and stateful storage.  If you wish to try StorageOS with Encryption, you can test drive it with our StorageOS Forever Free Developer Edition providing 5TB for evaluation purposes.

mm

Author: James Spurin

James Spurin is the Product Evangelist for StorageOS with an extensive career spanning 20 years, covering Kubernetes, automation, containers, software development, storage and unix. During his career James has worked for corporations including Nomura, Goldman Sachs, Dell EMC and Hitachi Data Systems. Outside of his industry experience James is an advocate for open-source software and is known for his popular projects and contributions. He’s also a technical author and his published works have received over 20,000 students across 100+ countries.

  • Customer Case Study: StorageOS provides MSP, Civo with Cloud Native StorageRead Now

  • Blog: On-Premises vs. Cloud Native StorageRead Now

  • Performance Benchmarking Cloud Native Storage Solutions for KubernetesDownload Now

  • On-Demand Webinar: Acceleration Kubernetes Onboarding and Application TransformationWatch Now

  • Cloud Native Live: Kubernetes Clusters need Persistent DataWatch Now