Migrate Zot Registry to Persistent Storage

The Zot Open Container Initiative (OCI) registry uses ephemeral emptyDir storage when persistence is disabled during installation. In this configuration, registry artifacts are lost when the pod restarts or is rescheduled.

This guide walks you through migrating Zot to a PVC so that registry artifacts persist across pod restarts and rescheduling.

warning

Enable persistence during the initial installation for production environments. Migrating from ephemeral to persistent storage requires downtime and a manual data restore. Retrofitting persistence later is more complex than configuring it up front.

Prerequisites

• You have access to the Kubernetes cluster.

• You have the Helm values file for the Zot deployment.

• You have a Container Storage Interface (CSI) and a storage class available on the cluster.

Migrate to Persistent Storage

Check Current Data Size

Before migrating, determine how much data Zot is currently storing so you can appropriately size the PVC.

1. Run the following command to check the current data size.

   kubectl exec --namespace mural-system <zot-pod-name> -- du --summarize --human-readable /var/lib/registry

   Use the output to set the pvc.storage value in the next section, allowing additional capacity for future growth.

Back Up Existing Data

Because kubectl cp can fail with EOF errors when copying large directories, create a tarball inside the pod and then copy it locally.

1. Create the tarball inside the pod.

   kubectl exec --namespace mural-system <zot-pod-name> -- tar --create --gzip --file /tmp/zot-backup.tar.gz --directory /var/lib/registry .

2. Copy the tarball to your local machine.

   kubectl cp mural-system/<zot-pod-name>:/tmp/zot-backup.tar.gz ./zot-backup.tar.gz

   If kubectl cp fails with EOF errors, use the following cat workaround instead.

   kubectl exec --namespace mural-system <zot-pod-name> -- cat /tmp/zot-backup.tar.gz > ./zot-backup.tar.gz

3. Verify the backup is intact.

   tar --list --gzip --file ./zot-backup.tar.gz | head --lines=20

   The output includes meta.db, cache.db, and repository directories with blobs.

Update Helm Values

Make the following changes to your Zot Helm values file.

1. Enable persistence and PVC creation.

   persistence: true
   pvc:
     create: true
     # Name of the PVC to use or create. If not set, the value '$CHART_RELEASE-pvc' is used.
     name: null
     accessModes: ['ReadWriteOnce']
     storage: 20Gi # Adjust based on current data size plus expected growth.
     storageClassName: null # Set to your storage class if not using the default.

2. Set the update strategy to Recreate.

   The Zot chart exposes the strategy field in the Helm values. If your Zot data volume can be mounted by only one pod at a time, such as a ReadWriteOnce PVC or local storage, set strategy.type to Recreate so the current pod is terminated before a replacement pod starts.

   strategy:
     type: Recreate

Redeploy Zot

1. Apply the updated configuration so Zot is redeployed with the new PVC settings.

   For Flux-managed installs, update the values block in the HelmRelease manifest for your mural release and apply the changes to your cluster.

   kubectl apply --filename <path-to-your-mural-helmrelease>.yaml

   If you manage the release manually, use the workflow below.

   warning

   If you do not use Flux, manage the mural-crds chart separately from the mural chart. Apply or upgrade Custom Resource Definitions (CRDs) out of band before you install or upgrade the mural chart. For the manual Helm workflow, refer to Upgrade Manually.

   helm upgrade mural oci://public.ecr.aws/mural/mural --version 1.0.7 \
     --namespace mural-system --values values.yaml --wait

   Zot restarts with an empty PVC mounted at /var/lib/registry.

Restore Data

1. Copy the backup tarball into the new pod.

   kubectl cp ./zot-backup.tar.gz mural-system/<new-zot-pod-name>:/tmp/zot-backup.tar.gz

   If kubectl cp fails, use the following pipe method instead.

   cat ./zot-backup.tar.gz | kubectl exec --stdin --namespace mural-system <new-zot-pod-name> -- tee /tmp/zot-backup.tar.gz > /dev/null

2. Extract the backup into the registry data directory.

   kubectl exec --namespace mural-system <new-zot-pod-name> -- tar --extract --gzip --file /tmp/zot-backup.tar.gz --directory /var/lib/registry

3. Remove the tarball from the pod.

   kubectl exec --namespace mural-system <new-zot-pod-name> -- rm /tmp/zot-backup.tar.gz

Validate

Confirm that the registry is serving the restored artifacts by querying the catalog endpoint.

curl --user '<zot-username>:<zot-password>' https://<your-zot-host>/zot/v2/_catalog # gitleaks:allow

Example Output

{"repositories":["zot/mural-workloads"]}

Next Steps

• Review Helm Chart Configuration for the full set of Zot configuration options, including storage class and access mode settings.

• If you are planning an upgrade, complete the Upgrade PaletteAI guide next.

Summary of Helm Value Changes

Setting	Before	After
persistence	false	true
pvc.create	false	true
pvc.name	null	null (autogenerated as $CHART_RELEASE-pvc)
pvc.storage	8Gi	20Gi
strategy.type	RollingUpdate	Recreate