Troubleshooting App and Model Deployments

This page provides troubleshooting guidance for common issues when App Deployments and Model Deployments fail to reach the Running status or become unhealthy after deployment.

Both App Deployments and Model Deployments are represented by the AIWorkload custom resource on the hub cluster. The AIWorkload generates one Workload Deployment per configured workload, and each Workload Deployment renders a Workload on the target spoke clusters. Failures can surface at any stage of this chain, so triage proceeds from the AIWorkload down to the individual workloads.

info

Troubleshooting steps on this page use kubectl and jq to inspect Kubernetes resources on the hub cluster. Most checks require read access to the Project namespace where the deployment was created. Flux checks also require read access to the PaletteAI built-in namespace, which defaults to mural-system, and ManagedCluster checks require cluster-scoped read access. If you do not have the required access, contact your platform administrator.

Deployment Status Overview

The AIWorkload reports its overall state in the status.status field, which the UI displays as the App Deployment or Model Deployment status: Provisioning, Running, Unhealthy, Failed, or Deleting.

The status.conditions field explains how the overall state was derived:

• ComputePoolReady - The target Compute Pool is provisioned and ready.

• ModelInPolicy - For Model Deployments from a model catalog, the selected model is allowed by the model access controls configured in the Project or Tenant settings.

• WorkloadDeploymentCreated - The Workload Deployments were generated.

• WorkloadDeploymentReady - The generated Workload Deployments are ready.

Start any investigation by reviewing the deployment in the PaletteAI UI. Select the App Deployment or Model Deployment from the list and review its status details. If the UI does not provide enough detail, inspect the AIWorkload directly.

kubectl get aiworkload <name> --namespace <project-namespace>
kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

The status.workloadDeployments field lists the names of the child Workload Deployments generated by the AIWorkload. Use these names to drill into workload-level failures.

kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.workloadDeployments'

Deployment Stuck in Provisioning

Symptom: The deployment status remains Provisioning and does not transition to Running.

Possible causes:

1. The target Compute Pool is still provisioning or is not ready.

2. The Compute Pool's Environment is not ready, so Workload Deployments cannot be generated.

3. The generated Workload Deployments have not finished rolling out to the spoke clusters.

Resolution:

1. Check the AIWorkload conditions to identify the blocked stage.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

2. If ComputePoolReady is False with a message similar to Waiting for ComputePool to be ready (current: Provisioning), the deployment is waiting on infrastructure. Identify the Compute Pool and check its status.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.computePoolRef'
   kubectl get computepool <computepool-name> --namespace <project-namespace>

   If the Compute Pool is stuck in Provisioning or shows Failed, refer to Troubleshooting Compute Pools to resolve the infrastructure issue first. Deployments that create a new Compute Pool inline can take 15 or more minutes to provision both the infrastructure and the application.

3. If WorkloadDeploymentCreated is False with the message ComputePool Environment not ready, the Compute Pool is up but its Environment has not been published yet. Verify the Environment referenced by the Compute Pool.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status'
   kubectl get environments --namespace <project-namespace>

4. If WorkloadDeploymentReady is False with the message Waiting for WorkloadDeployments to become ready, the workloads are still rolling out. List the child Workload Deployments and check their phase.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.workloadDeployments'
   kubectl get workloaddeployments --namespace <project-namespace>

   A Workload Deployment that stays in rendering or starting, or that reports unhealthy or failed, blocks the deployment from reaching Running. Refer to Workloads Unhealthy or Stuck on the Spoke Cluster to triage the workload rollout.

Deployment Failed

Symptom: The deployment status is Failed.

Cause: One of the AIWorkload conditions is False with a failure reason such as ValidationFailed, ComputePoolFailed, or WorkloadDeploymentFailed. Validation failures are the most common cause and are reported in the condition message.

Resolution:

1. Check the conditions and read the failure message.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

2. Match the failure message against the following common validation errors.

   Message pattern	Meaning and fix
   insufficient GPU <variant>: requested N, available M	The Compute Pool does not have enough free GPUs of the requested variant. Reduce the deployment's hardwareRequests, free capacity by deleting other deployments, or target a Compute Pool with more GPU capacity.
   no GPU capacity available for architecture <arch>	The Compute Pool has no GPU capacity for the requested architecture. Target a Compute Pool that provides GPUs for that architecture.
   GPU variant <variant> not available in ComputePool	The requested GPU variant does not exist in the Compute Pool. Adjust the GPU variant in hardwareRequests or target a different Compute Pool.
   insufficient CPU for architecture <arch>: requested N, available M	The Compute Pool does not have enough free CPU. Reduce the requested CPU or free capacity.
   inference engine mismatch: AIWorkload requires <x> but ComputePool <cp> has <y>	Model Deployments must use an inference engine compatible with the target Compute Pool. Select a Profile Bundle that matches the Compute Pool's inference engine, or target a Compute Pool provisioned with the required engine.

3. Check the Compute Pool allocation to confirm available capacity before redeploying.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status.hardwareCapacity'
   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status.hardwareAllocation'

4. Correct the configuration by adjusting the deployment settings. Refer to Create and Manage App Deployments and Create and Manage Model Deployments.

If the deployment is blocked by a GPU quota rather than physical capacity, the relevant caps are defined on the Project and Tenant. Refer to GPU Quota Exceeded and Insufficient GPU Resources for details on how the caps interact and how to raise them.

Model Deployment Out of Policy

Symptom: A Model Deployment does not start, and the ModelInPolicy condition is False with the reason ModelOutOfPolicy. Updates to an out-of-policy deployment are rejected with an error similar to: AIWorkload is out of policy: the deployed model is no longer authorized under the active model settings; delete and re-deploy to resolve.

Cause: Model Deployments sourced from Hugging Face or NVIDIA NGC are checked against the model access controls configured at the Project or Tenant scope. If the model list is not configured, is disabled, or does not allow the selected model repository or NIM image, PaletteAI does not create any Workload Deployments for the model. Custom model deployments are not subject to registry policy.

Resolution:

1. Read the ModelInPolicy condition message. It identifies the failing registry and scope.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions[] | select(.type == "ModelInPolicy")'

   Example messages:

   • Hugging Face model deployments are disabled at the <scope> scope; enable the Hugging Face model list to deploy Hugging Face models

   • model repository "<repo>" is not in the Hugging Face allow list at the <scope> scope; add it to the model repository allow list (or enable allow-all)

   • NGC image "<repo>/<container>" is not in the NIMs allow list at the <scope> scope; add it to the NIMs allow list (or enable allow-all)

2. Update the model access controls in the Project's Model Management settings, or ask a Tenant admin to update the Tenant-scoped model settings. Refer to Create and Manage Projects for details on Model as a Service Mappings and model access control lists.

3. If the deployment was already created and remains out of policy, delete it and deploy the model again after the allow list is updated.

Workloads Unhealthy or Stuck on the Spoke Cluster

Symptom: The deployment status is Unhealthy, or the WorkloadDeploymentReady condition stays False while the Compute Pool is Running. The child Workload Deployments report a PHASE of rendering, starting, unhealthy, or failed. The full set of phases is rendering, starting, running, unhealthy, failed, and deleting.

Possible causes:

1. A workload component failed to render, validate, or apply.

2. Resources were applied on the spoke cluster, but one or more components are not healthy — for example, pods stuck in ImagePullBackOff or Pending due to GPU scheduling.

3. For Model Deployments, the model download from Hugging Face or the image pull from NVIDIA NGC failed due to missing or invalid integration credentials.

4. The Environment cannot place the workload on any spoke cluster.

Resolution:

1. Check the phase and conditions of each child Workload Deployment.

   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output jsonpath='{.status.phase}'
   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output json | jq '.status.conditions'

   The condition types follow the workload lifecycle: Parsed, Rendered, DryRun, Dispatched, Applied, and Ready. The first condition that is False indicates the failing stage, and its message contains the underlying error.

2. Review the per-cluster workload status aggregated on the Workload Deployment. Each entry reports the phase and component health for one spoke cluster.

   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output json | jq '.status.statuses'

   Unhealthy components include a details message that reports the failing resource — for example, 0 / 1 replicas ready. For workloads with multiple deployment priorities, work backwards to the first priority that is not running and inspect its Ready-<N> condition. Refer to Deployment Priorities for the full triage procedure.

3. For Model Deployments with components stuck pulling images or downloading models, verify the Hugging Face and NVIDIA NGC integration secrets configured in the Settings resource. Refer to Settings Integrations Not Ready for the verification steps.

4. If the Workload Deployment reports that the Environment cannot complete placements, or no spoke clusters were selected, verify the Environment and the Compute Pool's cluster registration.

   kubectl get environments --namespace <project-namespace>
   kubectl describe workloaddeployment <workloaddeployment-name> --namespace <project-namespace>

5. If the failing stage is not clear from conditions and events, review the Hue controller logs on the hub cluster. The controllers run in the Helm release namespace, which defaults to mural-system.

   kubectl get pods --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers
   kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

Deployment Request Rejected

Symptom: Creating or updating a deployment fails immediately with an admission webhook error.

Cause: PaletteAI validates AIWorkload and WorkloadDeployment resources at admission time. Requests that violate the validation rules are rejected before any resources are created.

Resolution:

Match the error message against the following common rejections and correct the manifest or request.

Message pattern	Meaning and fix
computePoolRef and clusterVariant mutual exclusivity error	Set exactly one of spec.computePoolRef or spec.clusterVariant, never both or neither.
label "palette.ai/model-deployment-type" must not be set when "palette.ai/aiworkload-type"="app"	The palette.ai/model-deployment-type label is only valid for Model Deployments. Remove it from App Deployments.
Missing or invalid palette.ai/aiworkload-type label	Set the label to app or model. Model Deployments must also set palette.ai/model-deployment-type to HF, NGC, or custom-model.
ComputePool <namespace>/<name> is not shared with project "<namespace>"	The referenced Compute Pool belongs to another Project and is not shared. Target a Compute Pool in your Project or ask its owner to share it.
variable '<name>' is not defined in any VariableSet	The deployment sets a variable that the workload profile does not define. Remove the variable or add it to the profile.
cannot override locked variable '<name>'	The variable is locked by the workload profile. Remove the override.
user input is required for variable '<name>' but not provided in WorkloadDeployment	A required variable has no value. Provide a value for the variable in the deployment configuration.

If the rejection message references a webhook failure rather than a validation rule, verify that the Hue webhook pods are healthy in the Helm release namespace, which defaults to mural-system.

kubectl get pods --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-webhooks

Cannot Delete Deployment

Symptom: The deployment status stays Deleting and the resource is not removed.

Expected behavior:

By default, PaletteAI deletes deployed resources from spoke clusters before it removes the remaining hub resources. For each child WorkloadDeployment that dispatches manifests through Flux, Hue creates a Flux Kustomization with spec.deletionPolicy: WaitForTermination. During deletion, Flux keeps this Kustomization and its finalizer until the resources it applied finish their configured termination lifecycle.

For a federated deployment without workload orphaning, the foreground deletion chain includes the OCM ManifestWorkReplicaSet, its ManifestWork resources, and the Workload resources on the spoke clusters. An Environment can instead be configured to retain the spoke Workload. Refer to topology-ocm Lifecycle for these settings.

Hue waits until both the Flux Kustomization and OCIRepository are gone, then deletes the deployment's matching artifacts from the OCI registry. Hue removes the WorkloadDeployment finalizer only after the registry cleanup succeeds. Therefore, a Deleting phase can be an expected wait for downstream garbage collection, especially when a spoke cluster is temporarily disconnected.

Possible causes:

1. The workloads on the spoke cluster are still terminating.

2. The deployment created a Compute Pool with deletionPolicy: "delete", and the Compute Pool deletion has not completed.

3. A spoke cluster is disconnected, so OCM cannot confirm that its resources terminated.

4. Flux or Hue encountered an error while deleting a downstream resource.

5. The OCI registry is unavailable, or Hue cannot delete the deployment artifacts.

Resolution:

1. Check whether the child Workload Deployments are still present.

   kubectl get workloaddeployments --namespace <project-namespace>

2. Inspect the phase and Deleting condition of each remaining WorkloadDeployment.

   kubectl get workloaddeployment <workloaddeployment-name> \
     --namespace <project-namespace> \
     --output json | jq '{
       phase: .status.phase,
       deleting: [.status.conditions[] | select(.type == "Deleting")]
     }'

   A Deleting condition with status: "False", reason: "Deleting", and the message Resources deleted, waiting for garbage collection reports cleanup in progress, not a deletion failure. Use the condition reason and message to distinguish this expected wait from a reconciliation error.

3. Inspect the generated Flux Kustomization. Its name is the Workload Deployment name followed by the Project namespace.

   kubectl get kustomization \
     <workloaddeployment-name>-<project-namespace> \
     --namespace <built-in-namespace> \
     --output json | jq '{
       deletionPolicy: .spec.deletionPolicy,
       deletionTimestamp: .metadata.deletionTimestamp,
       finalizers: .metadata.finalizers,
       conditions: .status.conditions
     }'

   The PaletteAI built-in namespace defaults to mural-system. During deletion, the Kustomization can remain with deletionPolicy: "WaitForTermination" and a finalizer while Flux waits for applied resources to terminate. Read its conditions for a concrete Flux error before treating the finalizer as stuck.

4. Inspect the generated Flux OCIRepository.

   kubectl get ocirepository \
     <workloaddeployment-name>-<project-namespace> \
     --namespace <built-in-namespace> \
     --output json | jq '{
       deletionTimestamp: .metadata.deletionTimestamp,
       conditions: .status.conditions
     }'

   A NotFound response is expected after either Flux resource is garbage collected. Hue proceeds only after both resources are absent. If both are absent but the WorkloadDeployment remains, inspect its Deleting condition and the Hue logs for an OCI artifact cleanup error.

5. For deployments on spoke clusters, verify that each target ManagedCluster is available.

   kubectl get workloaddeployment <workloaddeployment-name> \
     --namespace <project-namespace> \
     --output json | jq -r '(.status.statuses // {}) | keys[]'
   kubectl get managedcluster <managedcluster-name>

   Restore connectivity to a temporarily disconnected spoke and allow OCM to finish cleanup. If a manually registered spoke is permanently unreachable, follow Remove an Unreachable Spoke Cluster. Do not manually remove a Compute Pool-managed Spoke; manage it through the Compute Pool lifecycle.

6. After the child Workload Deployments are gone, PaletteAI deletes an owned Compute Pool with deletionPolicy: "delete" and waits for its removal. A shared Compute Pool that another deployment still uses is retained. Check its status and, if the Compute Pool deletion is stuck, refer to Compute Pool Cannot Be Deleted.

   kubectl get computepool --namespace <project-namespace>

7. If deletion does not complete, review the Hue controller logs for errors related to the deployment. The controllers run in the Helm release namespace, which defaults to mural-system.

   kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

   warning

   Finalizers protect resources from premature deletion while cleanup occurs. Do not manually remove finalizers from the WorkloadDeployment or its Flux Kustomization unless controller logs show a known issue and you have consulted with your PaletteAI administrator or support team. Manually removing either finalizer can leave orphaned workloads on spoke clusters.

Resources

• Create and Manage App Deployments

• Create and Manage Model Deployments

• AIWorkloads Concepts

• Deployment Priorities

• Troubleshooting Compute Pools

• Troubleshooting Tenants