Back Up and Restore the Control Plane

The backup scripts capture the PaletteAI control plane from the hub cluster into a Velero archive in Amazon S3. The archive contains Tenants, Projects, Systems, Compute Pools, AI Workloads, Profile Bundles, and VmTemplates. It also contains the Secrets and ConfigMaps these resources depend on.

Workload data is not backed up. Zot registry content, Prometheus metrics, and persistent volume contents stay in the cluster.

info

The scripts install Velero into the hub cluster for the duration of the run and remove it when the run finishes. During a backup they add one label to the objects that must not be restored, and remove that label when the backup completes.

Prerequisites

• Cluster-scoped kubectl access to the hub cluster, including permission to label resources. The scripts label objects for exclusion during a backup and remove the labels afterwards.
• An S3 bucket and credentials with read and write access.
• aws CLI v2, jq, python3, and tar on the host that runs the scripts.
• The velero CLI, or network access to GitHub when you pass --install-velero.

Download the Scripts

curl --location --output paletteai-backup.tar.gz https://docs.palette-ai.com/resources/assets/hosted/scripts/paletteai-backup.tar.gz
tar -xzf paletteai-backup.tar.gz
cd paletteai-backup

The archive contains backup.sh, restore.sh, lib.sh, scrub.sh, scrub.py, restore-modifiers.yaml, and scope.env.example.

Configure

Create scope.env from the example and set the values for this installation.

cp scope.env.example scope.env

scope.env

BUCKET=example-backup-bucket
PREFIX=backups/paletteai
REGION=us-east-1

• BUCKET is required. PREFIX and REGION select the location inside the bucket and default to the values shown. The prefix must be dedicated to these backups, because Velero rejects a location that holds other directories.
• (Optional) EXCLUDE_TENANTS lists Tenant resource names to leave out of the archive entirely. The Tenant, its namespaces, and its Projects are removed.
• (Optional) EXCLUDE_PROJECTS lists project namespaces to leave out.
• (Optional) S3_URL and S3_FORCE_PATH_STYLE point the scripts at an S3-compatible store other than AWS.

Back Up

1. Export AWS credentials for the host. The scripts read them from the environment, from an AWS profile, or from the file named in VELERO_SECRET_FILE. The aws sts get-caller-identity check below is only for AWS. For another S3-compatible backend, supply its credentials through the same environment variables.

   aws sts get-caller-identity

2. Run the backup.

   ./backup.sh --env-file ./scope.env

   The script resolves the namespaces that hold PaletteAI resources, labels the objects that must not be restored, installs Velero, creates the backup, removes the labels, downloads the archive, scrubs it, and uploads the result. Velero is removed at the end of the run.

3. Record the artifact path printed at the end of the run:

   s3://example-backup-bucket/backups/paletteai/backups/paletteai-20261023t183632z/

   Each backup writes a paletteai-<timestamp>z.tar.gz archive and a paletteai-manifest.json file that records the PaletteAI and mural-crds versions, the namespace scope, and the redaction settings.

Archive Contents

The scripts use Velero's native exclusion mechanisms before the archive is written.

• CustomResourceDefinitions are excluded with --exclude-cluster-scoped-resources. The target cluster installs mural-crds first.
• The namespaces of projects listed in EXCLUDE_PROJECTS, and of Tenants listed in EXCLUDE_TENANTS, are left out by Velero's namespace filters.
• Individual objects that must not be restored are labeled velero.io/exclude-from-backup=true for the duration of the backup. Velero skips labeled items, and the labels are removed when the backup finishes:
  • The newest DefinitionRevision per definition. The controller recreates it, and older revisions are the only snapshot so they stay.
  • Generated ConfigMaps (kube-root-ca.crt and mural-project-rbac-*), and controller-owned definition ConfigMaps unless they belong to a kept revision or are cp-snapshot-*.
  • Built-in VmTemplates managed by Hue.
  • Tenants listed in EXCLUDE_TENANTS.
  • ComputePool kubeconfig Secrets and service account token Secrets.

The archive is then rewritten on the host to drop any item that still carries the exclusion label, replace Settings integration credentials with placeholder values, and remove the status embedded in DefinitionRevision specs.

A Velero resource modifier removes the status field, metadata.managedFields, and the last-applied-configuration annotations at restore time, not in the archive.

ConfigMaps named cp-snapshot-* are always kept. Workload Deployments, AvizFabrics, and AvizGPUAllocationRequests are never captured because controllers derive them.

Restore

warning

Restore into a fresh cluster that runs the same mural-crds and mural minor versions as the backup. The script compares the versions in paletteai-manifest.json against the target cluster and stops on a minor mismatch. Pass --force only when you accept the difference.

1. Install mural-crds and mural on the target cluster at the recorded versions.

2. Scale the hue-controllers Deployment to zero replicas so the controllers do not reconcile objects while the restore creates them. Leave hue-webhooks running; the admission webhooks must stay available for the objects the restore creates.

   kubectl scale deployment hue-controllers --namespace mural-system --replicas 0

3. Run the restore. Pass --backup to select a backup, or omit it to use the newest one under the prefix.

   ./restore.sh --env-file ./scope.env --backup paletteai-20261023t183632z

4. Review the restore phase. PartiallyFailed is expected when objects already exist and counts them as warnings; pass --allow-partial to accept it. Partial failures may happen and are generally safe: check that the report lists only skipped or already-existing objects. Failed always stops the run and writes a diagnostics log.

After Restore

1. If the archive was scrubbed, re-enter the integration credentials in System Settings for Palette, Hugging Face, NGC, and Aviz. Redacted values do not authenticate. The step is not needed with --no-redact or when an integration's scrub toggle is 0: the credentials are intact.

2. Verify that the System, Tenants, and Projects report ready.

3. Check that Profile Bundles and Compute Pools reconcile. When the source Palette Cluster Profile is missing, the Profile Bundle controller re-imports it from the cp-snapshot-* ConfigMap, and Compute Pools re-acquire their kubeconfig.

4. Scale hue-controllers back to its previous replica count.

   kubectl scale deployment hue-controllers --namespace mural-system --replicas 1

Troubleshooting

Symptom	Check
cannot access bucket	Host credentials, bucket name, region, and S3_URL
Backup location does not become available	Velero pod logs in the mural-system namespace, and bucket permissions
Backup reports PartiallyFailed	velero backup describe <name> --details; usually a resource type the API rejects
Restore never appears	The prefix is wrong, or Velero cannot list the bucket
Restore reports Failed with an InProgress restore	The Velero pod restarted mid-restore. An OOMKilled pod means the memory limit is too low; raise VELERO_MEMORY_LIMIT and retry
Settings report invalid credentials after restore	Expected: credentials are redacted. Re-enter them
Objects stay labeled after an interrupted backup	Clear the label in the affected namespaces: kubectl label <resource> --namespace <namespace> --selector velero.io/exclude-from-backup=true velero.io/exclude-from-backup-

Pass --keep to backup.sh or restore.sh to leave Velero installed for inspection. Do not run velero uninstall: it deletes the namespace, which is mural-system by default. Re-run restore.sh --cleanup-only to remove a leftover install.

Next Steps

• Configure System Settings to restore integration credentials.
• Troubleshooting the System Resource covers System, Tenant, and Project readiness.