Migrate Zot Registry to Persistent Storage

The Zot Open Container Initiative (OCI) registry typically uses ephemeral (emptyDir) storage when persistence is disabled during installation. As a result, registry artifacts are lost when the pod restarts or is rescheduled. Starting with PaletteAI v1.1.0, Zot persistence is enabled by default in the bundled Helm values. Use this guide if you installed an earlier PaletteAI version with persistence turned off, or if you disabled the zot.persistence setting and need to migrate to a Persistent Volume Claim (PVC)

This guide walks you through migrating Zot to a PVC so that registry artifacts persist across pod restarts and rescheduling.

warning

Enable persistence on day one for production environments. Migrating from ephemeral to persistent storage requires downtime and a manual data restore. Retrofitting persistence later is significantly more complex than configuring it during initial installation.

Prerequisites

• You have access to the Kubernetes cluster with kubectl.

• You have the Helm values file for the Zot deployment.

• You have a Container Storage Interface (CSI) and a storage class available on the cluster.

Migrate to Persistent Storage

Check Current Data Size

Before migrating, determine how much data Zot is currently storing so you can appropriately size the PVC.

1. Run the following command to check the current data size.

   kubectl exec --namespace mural-system <zot-pod-name> -- du --summarize --human-readable /var/lib/registry

   Use the output to set the pvc.storage value in the next section, allowing additional capacity for future growth.

Back Up Existing Data

Because kubectl cp can fail with EOF errors when copying large directories, create a tarball inside the pod and then copy it locally.

1. Create the tarball inside the pod.

   kubectl exec --namespace mural-system <zot-pod-name> -- tar --create --gzip --file /tmp/zot-backup.tar.gz --directory /var/lib/registry .

2. Copy the tarball to your local machine.

   kubectl cp mural-system/<zot-pod-name>:/tmp/zot-backup.tar.gz ./zot-backup.tar.gz

   If kubectl cp fails with EOF errors, use the following cat workaround instead.

   kubectl exec --namespace mural-system <zot-pod-name> -- cat /tmp/zot-backup.tar.gz > ./zot-backup.tar.gz

3. Verify the backup is intact.

   tar --list --gzip --file ./zot-backup.tar.gz | head --lines=20

   The output includes meta.db, cache.db, and repository directories with blobs.

Update Helm Values

Make the following changes to your Zot Helm values file.

1. Enable persistence and PVC creation.

   persistence: true
   pvc:
     create: true
     # Name of the PVC to use or create. If not set, the value '$CHART_RELEASE-pvc' is used.
     name: null
     accessModes: ['ReadWriteOnce']
     storage: 20Gi # Adjust based on current data size plus expected growth.
     storageClassName: null # Set to your storage class if not using the default.

2. Set the deployment update strategy to Recreate.

The PaletteAI Mural chart deploys Zot as a deployment, and the strategy field in your values controls that workload. Because ReadWriteOnce PVCs can only be mounted by a single pod at a time, using the Recreate strategy ensures the existing pod is terminated before a new one is started. This prevents conflicts during updates. If your installation uses StatefulSet for Zot instead (which is uncommon for this chart), skip this step and follow the update semantics for that controller.

strategy:
  type: Recreate

Redeploy Zot

1. Apply the updated Helm values by running helm upgrade to redeploy Zot.

   The pod restarts with an empty PVC mounted at /var/lib/registry.

Restore Data

1. Copy the backup tarball into the new pod.

   kubectl cp ./zot-backup.tar.gz mural-system/<new-zot-pod-name>:/tmp/zot-backup.tar.gz

   If kubectl cp fails, use the following pipe method instead.

   cat ./zot-backup.tar.gz | kubectl exec --stdin --namespace mural-system <new-zot-pod-name> -- tee /tmp/zot-backup.tar.gz > /dev/null

2. Extract the backup into the registry data directory.

   kubectl exec --namespace mural-system <new-zot-pod-name> -- tar --extract --gzip --file /tmp/zot-backup.tar.gz --directory /var/lib/registry

3. Remove the tarball from the pod.

   kubectl exec --namespace mural-system <new-zot-pod-name> -- rm /tmp/zot-backup.tar.gz

Validate

Confirm the registry is serving the restored artifacts by querying the catalog endpoint.

curl --user '<zot-username>:<zot-password>' https://<your-zot-host>/zot/v2/_catalog # gitleaks:allow

Example Output

{"repositories":["zot/mural-workloads"]}

Next Steps

• Review the Helm Chart Configuration for a full list of Zot configuration options, including storage class and access mode settings.

• If you are planning an upgrade, complete the Upgrade PaletteAI guide next.

Summary of Helm Value Changes

Setting	Before	After
persistence	false	true
pvc.create	false	true
pvc.name	null	null (autogenerated as $CHART_RELEASE-pvc)
pvc.storage	8Gi	20Gi
strategy.type	RollingUpdate	Recreate