Upgrade PaletteAI

This page explains how to upgrade PaletteAI to a new version. Use the workflow that matches your installation method:

• If you installed PaletteAI with Helm, use the Upgrade with Flux or Upgrade Manually workflow.
• If you installed PaletteAI with the appliance method, skip to Upgrade Appliance. Appliance upgrades use a content bundle downloaded from Artifact Studio.

For Helm-based deployments, PaletteAI consists of two Helm charts that must be upgraded in the following order:

• First, upgrade mural-crds, which contains the Custom Resource Definitions (CRDs).

• Second, upgrade mural, which contains the main application.

info

For production environments, we recommend using Flux or Argo CD to manage the PaletteAI Helm chart lifecycle. Both solutions can be configured to handle CRD upgrades automatically.

Prerequisites

• PaletteAI is installed.

• PaletteAI CLI is installed.

  paletteai --version

• Set the KUBECONFIG environment variable to the path of the PaletteAI hub cluster's kubeconfig file.

  export KUBECONFIG=<kubeconfig-location>

Migrations

The PaletteAI hue controller runs migrations on existing resources at startup to keep them compatible across versions. Each migration records its output to the paletteai-migrations ConfigMap in the mural-system namespace. Migrations run during PaletteAI upgrades to prevent upgrade failures as CRDs evolve between versions.

Version Matrix

The following table lists each migration with the PaletteAI version that introduces it.

Migration Name	PaletteAI Version	Migration Summary
definition-revision-prefix	1.0.3	Changes naming convention for DefinitionRevision CRs.
variable-input-text	1.0.3	Adds support for variable input types text and multiline to VariableSet CRD.
ai-workload-labels	1.1.0	Backfills mandatory classification labels on pre-existing AIWorkload CRs (palette.ai/aiworkload-type, palette.ai/model-deployment-type).
ai-workload-wldcs	1.1.0	Requires new fields profileBundleRef and addonVariantName in AIWorkload.Spec.workloadDeploymentConfigs.
compute-pool-wldcs	1.1.0	Requires new fields profileBundleRef and addonVariantName in ComputePool.Spec.workloadDeploymentConfigs.
tenant-compute-config-ref	1.1.0	Reconciles Tenant.spec.computeConfigRef per RFC-0006. Verifies referenced ComputeConfig exists in the tenant workload namespace, or upserts a default-named one.
compute-pool-deletion-policy	1.3.0	Moves deletionPolicy field in ComputePool CRD.
ai-workload-deletion-policy	1.3.0	Moves deletionPolicy field in AIWorkload CRD.

Validate

After upgrading PaletteAI, verify the migration results recorded by the hue controller. If any migration failed, you must remediate the underlying cause before re-running the migration, otherwise the same failure will resurface.

1. Inspect the paletteai-migrations ConfigMap in the mural-system namespace to review per-migration results. Each migration records its output here, so this ConfigMap is the source of truth for whether the upgrade migrations succeeded.

   kubectl get configmap paletteai-migrations --namespace mural-system --output yaml

2. If a migration reports an error, remediate the underlying cause. Once you have applied a fix, validate it with a dry run. The paletteai migrate command runs in dry-run mode by default and reports what would change without modifying any resources.

   paletteai migrate

3. If the dry run reports no failures, trigger the actual migration to apply the changes. Re-inspect the paletteai-migrations ConfigMap afterward to confirm all migrations succeeded.

   paletteai migrate --dry-run=false

Upgrade with Flux

Use Flux to automatically manage your PaletteAI installation, upgrades, and CRD lifecycle. Flux monitors the OCI registry and upgrades the Helm release when new chart versions appear.

Enablement

1. Create a file named mural-crds-oci-repository.yaml for the mural-crds chart.

   cat << EOF > mural-crds-oci-repository.yaml
   apiVersion: source.toolkit.fluxcd.io/v1
   kind: OCIRepository
   metadata:
     name: mural-crds
     namespace: mural-system
   spec:
     interval: 10m
     ref:
       semver: "0.7.8-hotfix.5"
     url: oci://public.ecr.aws/mural/mural-crds
   EOF

2. Create a file named mural-oci-repository.yaml for the mural chart.

   cat << EOF > mural-oci-repository.yaml
   apiVersion: source.toolkit.fluxcd.io/v1
   kind: OCIRepository
   metadata:
     name: mural
     namespace: mural-system
   spec:
     interval: 10m
     ref:
       semver: "1.1.6"
     url: oci://public.ecr.aws/mural/mural
   EOF

3. If the mural HelmRelease resource already exists on your cluster, suspend it before applying the updated OCIRepository resources. Suspending the HelmRelease prevents the OCIRepository version update from triggering an automatic Helm upgrade before your updated Helm values are in place. Refer to the Flux HelmRelease documentation for more information about suspending reconciliation.

   kubectl patch helmrelease mural --namespace mural-system \
     --type merge --patch '{"spec":{"suspend":true}}'

   warning

   Review the release notes for every version between your current version and the target version before you upgrade. If any of them include breaking changes, such as changes to required Helm values, always suspend the mural HelmRelease first and update your values before resuming it.

4. Apply both OCIRepository resources to your cluster.

   kubectl apply --filename mural-crds-oci-repository.yaml
   kubectl apply --filename mural-oci-repository.yaml

5. Create a file named mural-crds-helm-release.yaml for the mural-crds chart.

   cat << EOF > mural-crds-helm-release.yaml
   apiVersion: helm.toolkit.fluxcd.io/v2
   kind: HelmRelease
   metadata:
     name: mural-crds
     namespace: mural-system
   spec:
     interval: 10m
     chartRef:
       kind: OCIRepository
       name: mural-crds
       namespace: mural-system
     install:
       crds: Create
     upgrade:
       crds: CreateReplace
   EOF

6. Create a file named mural-helm-release.yaml for the mural chart. The dependsOn field ensures that the mural-crds chart is upgraded before the mural chart.

   cat << EOF > mural-helm-release.yaml
   apiVersion: helm.toolkit.fluxcd.io/v2
   kind: HelmRelease
   metadata:
     name: mural
     namespace: mural-system
   spec:
     interval: 10m
     chartRef:
       kind: OCIRepository
       name: mural
       namespace: mural-system
     dependsOn:
       - name: mural-crds
     values:
       # Your custom values here
   EOF

7. Apply both HelmRelease resources to your cluster.

   kubectl apply --filename mural-crds-helm-release.yaml
   kubectl apply --filename mural-helm-release.yaml

8. If you suspended the mural HelmRelease, resume it after the updated values are applied. Flux then reconciles the release and performs the upgrade.

   kubectl patch helmrelease mural --namespace mural-system \
     --type merge --patch '{"spec":{"suspend":false}}'

Validate

1. Confirm that both OCIRepository resources were created and are ready.

   kubectl get ocirepository --namespace mural-system

   Example Output

   NAME          URL                                       READY   STATUS                           AGE
   mural-crds    oci://public.ecr.aws/mural/mural-crds     True    stored artifact for digest ...   10s
   mural         oci://public.ecr.aws/mural/mural          True    stored artifact for digest ...   10s

2. Check that both HelmRelease resources were created.

   kubectl get helmrelease --namespace mural-system

   Example Output

   NAME          READY   STATUS                                                     AGE
   mural-crds    True    Helm upgrade succeeded for release mural-system/mural...   5m
   mural         True    Helm upgrade succeeded for release mural-system/mural...   5m

3. Verify that all PaletteAI pods are in a Running state.

   kubectl get pods --namespace mural-system

Switch Chart Sources to Public ECR

Use this workflow when your hub cluster pulls the mural and mural-crds Helm charts from a private OCI registry (for example, an internal Zot registry on an appliance) and you want to switch to the public AWS ECR registries instead. This applies to Flux-managed hub installations.

The public chart endpoints are:

• oci://public.ecr.aws/mural/mural-crds
• oci://public.ecr.aws/mural/mural

info

Switching chart sources does not change workload OCI registry settings (hue.ociRegistry, fleetConfig.spokes, and related values). Update those separately if you also need to move workload artifacts off a private registry. Refer to Switch from Zot to ECR After Installation for the workload registry switch procedure and Configure ECR Instead of Zot for ECR configuration guidance.

Enablement

1. Confirm that the hub cluster can reach public.ecr.aws over HTTPS (port 443).

2. Update the spec.url field on both hub OCIRepository resources in the mural-system namespace.

   kubectl patch ocirepository mural-crds --namespace mural-system \
     --type merge --patch '{"spec":{"url":"oci://public.ecr.aws/mural/mural-crds"}}'
   
   kubectl patch ocirepository mural --namespace mural-system \
     --type merge --patch '{"spec":{"url":"oci://public.ecr.aws/mural/mural"}}'

   If you manage these resources from manifest files instead of kubectl patch, update the spec.url values in those files and re-apply them.

3. Edit the mural HelmRelease so Brush provisions spoke clusters from the public chart registries as well. Under spec.values, set the following fields:

   brush:
     mural:
       source:
         repository: oci://public.ecr.aws/mural/mural
         basicAuth: {}
         basicAuthSecretName: ''
         certSecretName: ''
     muralCrds:
       source:
         repository: oci://public.ecr.aws/mural/mural-crds
         basicAuth: {}
         basicAuthSecretName: ''
         certSecretName: ''

   Apply the change with your usual workflow, for example:

   kubectl edit helmrelease mural --namespace mural-system

   Clear basicAuthSecretName and certSecretName when switching away from a private registry that required pull credentials or a custom CA. Leave other Brush source settings (such as interval and timeout) unchanged unless your environment requires different values.

   warning

   If your installation uses the PaletteAI appliance installer in airgap mode, the installer manages brush.mural.source and brush.muralCrds.source. Manual edits to those fields will be overwritten on the next installer run unless you also change the deployment mode and registry configuration in Local UI.

4. Wait for Flux to reconcile the updated resources. Flux re-pulls the charts from public ECR and Brush propagates the new chart sources to spoke clusters on its next reconcile.

Validate

1. Confirm that both OCIRepository resources reference the public endpoints and are ready.

   kubectl get ocirepository --all-namespaces

   Example Output

   NAME          URL                                       READY   STATUS                           AGE
   mural-crds    oci://public.ecr.aws/mural/mural-crds     True    stored artifact for digest ...   10m
   mural         oci://public.ecr.aws/mural/mural          True    stored artifact for digest ...   10m

2. Check that both HelmRelease resources reconciled successfully after the registry switch.

   kubectl get helmrelease --all-namespaces

   Both mural-crds and mural should show READY: True.

Switch from Zot to ECR After Installation

If you installed PaletteAI with the default in-cluster Zot registry, you can switch the hub cluster and all spoke clusters to Amazon Elastic Container Registry (ECR) by updating the registry values on your existing installation. This workflow changes the workload artifact registry (hue.ociRegistry and fleetConfig.spokes[i].ociRegistry); to change where the hub pulls the mural and mural-crds Helm charts from, refer to Switch Chart Sources to Public ECR.

PaletteAI reconciles the registry change automatically:

• On the hub cluster, the Hue controllers restart with the new registry endpoint and push the Definition bundles to ECR on the next sync interval, which is five minutes by default.

• The controllers on the hub detect the updated per-spoke registry configuration and update the PaletteAI Helm release on every joined spoke cluster. Each spoke then updates its Flux OCIRepository resources in place and pushes workload artifacts to ECR on its next reconciliation.

Artifacts already stored in Zot are not migrated to ECR. PaletteAI regenerates Definition bundles and workload artifacts from the current cluster state and pushes them to ECR, so no manual artifact migration is required.

warning

Workloads that are already running are not interrupted by the switch. However, deployment updates pause, and Flux may report failed OCIRepository pulls, until the first successful push to ECR completes. Keep Zot enabled until you validate the switch.

Switch Prerequisites

• A PaletteAI installation that uses the default in-cluster Zot registry, with all pods in the release namespace in a Running state.

  kubectl get pods --namespace mural-system

• All items listed in the Configure ECR Instead of Zot prerequisites for your platform. The IAM role with ECR permissions, and, on EKS, the IRSA trust policy for the hue and source-controller service accounts, must exist before you apply the registry change.

• Network connectivity from the hub cluster and every spoke cluster to the ECR endpoint.

• The Helm values file or Flux HelmRelease manifest currently used by your mural release.

Update the Registry Configuration

1. If you set any ociRegistry values under fleetConfig.spokeValuesOverrides.hue, update them to match your ECR configuration. These overrides take precedence over the values in fleetConfig.spokes[i].ociRegistry.

2. Complete all steps in the Configure ECR Instead of Zot enablement section against your existing installation. Make sure to update hue.ociRegistry and every fleetConfig.spokes[i].ociRegistry entry, configure AWS authentication for the hue and source-controller service accounts, remove the basicAuth sections, and apply the updated configuration.

3. Wait for the Hue controllers on the hub cluster to finish rolling out with the new registry configuration.

   kubectl rollout status deployment/hue-controllers --namespace mural-system

Validate the Switch

1. Complete the steps in the Configure ECR Instead of Zot validation section. In the OCIRepository output, confirm that the URL column shows your ECR endpoint.

2. On each spoke cluster, confirm that the OCIRepository resources reference ECR and are ready.

   kubectl get ocirepository --namespace mural-system

3. After you confirm that the hub and all spoke clusters pull from ECR, complete the Configure ECR Instead of Zot next steps to disable the in-cluster Zot registry.

Upgrade Manually

If you prefer to manage upgrades manually, follow these steps to upgrade both charts in the correct order.

danger

The mural-crds chart must be upgraded before the mural chart. Upgrading in the wrong order may cause deployment failures or undefined behavior.

Prerequisites

• PaletteAI is installed.

• helm installed and available in your $PATH.

• kubectl installed and available in your $PATH.

• The KUBECONFIG environment variable is set to the path of the PaletteAI hub cluster's kubeconfig file.

  export KUBECONFIG=<kubeconfig-location>

• Access to the PaletteAI values.yaml file configured for your environment

Enablement

1. Upgrade the mural-crds chart.

   helm template mural-crds oci://public.ecr.aws/mural/mural-crds --version 0.7.8-hotfix.5 --include-crds | kubectl apply --server-side --force-conflicts -f -

   Example Output

   Pulled: public.ecr.aws/mural/mural-crds:0.5.0
   Digest: sha256:406eff86daf9ebdd38a17794390628a353374b6e1a5b856a5f89a702d45dfc79
   customresourcedefinition.apiextensions.k8s.io/clustermanagementaddons.addon.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/managedclusters.cluster.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/managedclustersets.cluster.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/managedclusteraddons.addon.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/managedclustersetbindings.cluster.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/addondeploymentconfigs.addon.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/clusterclaims.cluster.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/placements.cluster.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/addontemplates.addon.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/placementdecisions.cluster.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/addonplacementscores.cluster.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/hubs.fleetconfig.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/spokes.fleetconfig.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/artifactgenerators.source.extensions.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/externalartifacts.source.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/ocirepositories.source.toolkit.fluxcd.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/fluxcdconfigs.fluxcd.open-cluster-management.io serverside-applied
   customresourcedefinition.apiextensions.k8s.io/aiworkloads.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/componentdefinitions.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/computeconfigs.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/computepoolevaluations.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/computepools.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/computes.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/definitionrevisions.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/environments.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/policydefinitions.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/profilebundles.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/projects.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/scalingpolicies.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/settings.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/tenants.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/traitdefinitions.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/variablesets.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/workloaddeployments.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/workloadprofiles.spectrocloud.com serverside-applied
   customresourcedefinition.apiextensions.k8s.io/workloads.spectrocloud.com serverside-applied

2. Upgrade the mural chart. The --wait flag ensures that Helm waits for all resources to be ready before marking the release as successful.

   helm upgrade mural oci://public.ecr.aws/mural/mural --version 1.1.6 \
     --namespace mural-system --values values.yaml --wait

   Example Output

   Pulled: public.ecr.aws/mural/mural:0.7.0
   Digest: sha256:406eff86daf9ebdd38a17794390628a353374b6e1a5b856a5f89a702d45dfc79
   Release "mural" has been upgraded. Happy Helming!
   NAME: mural
   LAST DEPLOYED: Wed Feb 18 21:19:03 2026
   NAMESPACE: mural-system
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None

Validate

1. Check both Helm release statuses.

   helm status mural-crds --namespace mural-system
   helm status mural --namespace mural-system

   Both outputs should show STATUS: deployed and display the latest revision numbers.

2. Verify that all CRDs are present.

   kubectl get crds | grep --extended-regexp "(spectrocloud.com|fluxcd.io|open-cluster-management.io)"

   Example Output

   addondeploymentconfigs.addon.open-cluster-management.io          2026-01-22T19:41:58Z
   addonplacementscores.cluster.open-cluster-management.io          2026-01-22T19:41:58Z
   addontemplates.addon.open-cluster-management.io                  2026-01-22T19:41:58Z
   aiworkloads.spectrocloud.com                                     2026-01-14T18:10:36Z
   appliedmanifestworks.work.open-cluster-management.io             2026-01-14T18:18:10Z
   artifactgenerators.source.extensions.fluxcd.io                   2026-01-20T23:11:13Z
   awscloudconfigs.cluster.spectrocloud.com                         2026-01-14T18:05:25Z
   # ... output omitted for readability

3. Verify that all PaletteAI pods are in a Running state.

   kubectl get pods --namespace mural-system

Upgrade Appliance

Follow these steps to upgrade the PaletteAI Management Appliance using a content bundle downloaded from Artifact Studio. Use this workflow only for appliance-based deployments. If you installed PaletteAI with Helm charts, use Upgrade with Flux or Upgrade Manually.

For additional background, refer to the Palette Management Appliance Upgrade guide.

info

The upgrade process incurs downtime for the PaletteAI management cluster, but workload clusters remain operational.

Prerequisites

• Ensure the PaletteAI management cluster is healthy and you can access the Local UI of the leader node.

• Ensure you can access Artifact Studio to download the installation assets for PaletteAI.

• Download the content bundle for the target PaletteAI version.

Enablement

1. Sign in to Artifact Studio.

   If you do not have access to Artifact Studio, contact your Spectro Cloud representative or open a support ticket.

2. On the Artifact Studio landing page, select the PaletteAI card that matches your deployment.

   • Select Install PaletteAI Enterprise for standard PaletteAI appliance deployments.

   • Select Install PaletteAI VerteX for PaletteAI VerteX deployments.

3. Use the version drop-down to select the target PaletteAI version.

4. In the download modal, download the Content Bundle asset for the target version.

   tip

   For appliance upgrades, download only the Content Bundle asset, not the ISO, Helm chart, or MON Key.

5. Log in to the Local UI of the leader node at https://<node-ip>:5080.

6. From the left main menu, select Content. Select Actions > Upload Content and upload the downloaded content bundle. Wait for the upload and sync to complete.

7. From the left main menu, select Cluster and select the Configuration tab.

8. Select the Update drop-down menu and select Review Changes. Review the proposed changes carefully.

   warning

   Ensure the configured Zot password matches the password used during the initial PaletteAI installation. Some upgrade paths may also require you to re-enter configuration values from the original installation.

9. Select Confirm Changes, then select Update to start the upgrade.

10. After the appliance upgrade completes, update the Flux resources on the management cluster to match the new version.

    Suspend the mural HelmRelease so that the OCIRepository version update does not trigger an automatic Helm upgrade before you update the Helm values. Refer to the Flux HelmRelease documentation for more information about suspending reconciliation.

    kubectl patch helmrelease mural --namespace mural-system \
      --type merge --patch '{"spec":{"suspend":true}}'

    warning

    Review the release notes for every version between your current version and the target version before you upgrade. If any of them include breaking changes, such as changes to required Helm values, always suspend the mural HelmRelease first and update your values before resuming it.

    Update the spec.ref.semver field on the mural-crds and mural OCIRepository resources:

    kubectl patch ocirepository mural-crds --namespace mural-system \
      --type merge --patch '{"spec":{"ref":{"semver":"0.7.8-hotfix.5"}}}'

    kubectl patch ocirepository mural --namespace mural-system \
      --type merge --patch '{"spec":{"ref":{"semver":"1.1.6"}}}'

    Review and update any changed values in the mural HelmRelease:

    kubectl edit helmrelease mural --namespace mural-system

    For Dex local users, post-install credential changes are made under spec.values.dex.config.staticPasswords in this HelmRelease.

    Resume the mural HelmRelease after the updated values are in place. Flux then reconciles the release and performs the upgrade.

    kubectl patch helmrelease mural --namespace mural-system \
      --type merge --patch '{"spec":{"suspend":false}}'

Validate

1. Log in to the Local UI at https://<node-ip>:5080.

2. From the left main menu, select Cluster. Confirm that the palette-ai pack displays the upgraded version and is in a Running status.

3. Check that both HelmRelease resources have reconciled successfully.

   kubectl get helmrelease --namespace mural-system

   Both mural-crds and mural should show READY: True.

Next Steps

• Review the Release Notes for any breaking changes or new configuration options introduced in the upgraded version.

• If you upgraded the appliance, verify that spoke clusters are healthy and reconciling correctly. Refer to the hub and spoke model documentation for spoke topology and platform-specific setup guidance.