Troubleshooting App and Model Deployments

This page provides troubleshooting guidance for common issues when App Deployments and Model Deployments fail to reach the Running status or become unhealthy after deployment.

Both App Deployments and Model Deployments are represented by the AIWorkload custom resource on the hub cluster. The AIWorkload generates one Workload Deployment per configured workload, and each Workload Deployment renders a Workload on the target spoke clusters. Failures can surface at any stage of this chain, so triage proceeds from the AIWorkload down to the individual workloads.

info

Troubleshooting steps on this page use kubectl commands to inspect Kubernetes resources on the hub cluster. You need kubectl access to the hub cluster and the Project namespace where your deployment was created. If you do not have kubectl access, contact your platform administrator.

Deployment Status Overview

The AIWorkload reports its overall state in the status.status field, which the UI displays as the App Deployment or Model Deployment status: Provisioning, Running, Unhealthy, Failed, or Deleting.

The status.conditions field explains how the overall state was derived:

• ComputePoolReady - The target Compute Pool is provisioned and ready.

• ModelInPolicy - For Model Deployments from a model catalog, the selected model is allowed by the model access controls configured in the Project or Tenant settings.

• WorkloadDeploymentCreated - The Workload Deployments were generated.

• WorkloadDeploymentReady - The generated Workload Deployments are ready.

Start any investigation by reviewing the deployment in the PaletteAI UI. Select the App Deployment or Model Deployment from the list and review its status details. If the UI does not provide enough detail, inspect the AIWorkload directly.

kubectl get aiworkload <name> --namespace <project-namespace>
kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

The status.workloadDeployments field lists the names of the child Workload Deployments generated by the AIWorkload. Use these names to drill into workload-level failures.

kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.workloadDeployments'

Deployment Stuck in Provisioning

Symptom: The deployment status remains Provisioning and does not transition to Running.

Possible causes:

1. The target Compute Pool is still provisioning or is not ready.

2. The Compute Pool's Environment is not ready, so Workload Deployments cannot be generated.

3. The generated Workload Deployments have not finished rolling out to the spoke clusters.

Resolution:

1. Check the AIWorkload conditions to identify the blocked stage.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

2. If ComputePoolReady is False with a message similar to Waiting for ComputePool to be ready (current: Provisioning), the deployment is waiting on infrastructure. Identify the Compute Pool and check its status.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.computePoolRef'
   kubectl get computepool <computepool-name> --namespace <project-namespace>

   If the Compute Pool is stuck in Provisioning or shows Failed, refer to Troubleshooting Compute Pools to resolve the infrastructure issue first. Deployments that create a new Compute Pool inline can take 15 or more minutes to provision both the infrastructure and the application.

3. If WorkloadDeploymentCreated is False with the message ComputePool Environment not ready, the Compute Pool is up but its Environment has not been published yet. Verify the Environment referenced by the Compute Pool.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status'
   kubectl get environments --namespace <project-namespace>

4. If WorkloadDeploymentReady is False with the message Waiting for WorkloadDeployments to become ready, the workloads are still rolling out. List the child Workload Deployments and check their phase.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.workloadDeployments'
   kubectl get workloaddeployments --namespace <project-namespace>

   A Workload Deployment that stays in rendering or starting, or that reports unhealthy or failed, blocks the deployment from reaching Running. Refer to Workloads Unhealthy or Stuck on the Spoke Cluster to triage the workload rollout.

Deployment Failed

Symptom: The deployment status is Failed.

Cause: One of the AIWorkload conditions is False with a failure reason such as ValidationFailed, ComputePoolFailed, or WorkloadDeploymentFailed. Validation failures are the most common cause and are reported in the condition message.

Resolution:

1. Check the conditions and read the failure message.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

2. Match the failure message against the following common validation errors.

   Message pattern	Meaning and fix
   insufficient GPU <variant>: requested N, available M	The Compute Pool does not have enough free GPUs of the requested variant. Reduce the deployment's hardwareRequests, free capacity by deleting other deployments, or target a Compute Pool with more GPU capacity.
   no GPU capacity available for architecture <arch>	The Compute Pool has no GPU capacity for the requested architecture. Target a Compute Pool that provides GPUs for that architecture.
   GPU variant <variant> not available in ComputePool	The requested GPU family does not exist in the Compute Pool. Adjust the GPU family in hardwareRequests or target a different Compute Pool.
   insufficient CPU for architecture <arch>: requested N, available M	The Compute Pool does not have enough free CPU. Reduce the requested CPU or free capacity.
   inference engine mismatch: AIWorkload requires <x> but ComputePool <cp> has <y>	Model Deployments must use an inference engine compatible with the target Compute Pool. Select a Profile Bundle that matches the Compute Pool's inference engine, or target a Compute Pool provisioned with the required engine.

3. Check the Compute Pool allocation to confirm available capacity before redeploying.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status.hardwareCapacity'
   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status.hardwareAllocation'

4. Correct the configuration by adjusting the deployment settings. Refer to Create and Manage App Deployments and Create and Manage Model Deployments.

If the deployment is blocked by a GPU quota rather than physical capacity, the relevant caps are defined on the Project and Tenant. Refer to GPU Quota Exceeded and Insufficient GPU Resources for details on how the caps interact and how to raise them.

Model Deployment Out of Policy

Symptom: A Model Deployment does not start, and the ModelInPolicy condition is False with the reason ModelOutOfPolicy. Updates to an out-of-policy deployment are rejected with an error similar to: AIWorkload is out of policy: the deployed model is no longer authorized under the active model settings; delete and re-deploy to resolve.

Cause: Model Deployments sourced from Hugging Face or NVIDIA NGC are checked against the model access controls configured at the Project or Tenant scope. If the model list is not configured, is disabled, or does not allow the selected model repository or NIM image, PaletteAI does not create any Workload Deployments for the model. Custom model deployments are not subject to registry policy.

Resolution:

1. Read the ModelInPolicy condition message. It identifies the failing registry and scope.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions[] | select(.type == "ModelInPolicy")'

   Example messages:

   • Hugging Face model deployments are disabled at the <scope> scope; enable the Hugging Face model list to deploy Hugging Face models

   • model repository "<repo>" is not in the Hugging Face allow list at the <scope> scope; add it to the model repository allow list (or enable allow-all)

   • NGC image "<repo>/<container>" is not in the NIMs allow list at the <scope> scope; add it to the NIMs allow list (or enable allow-all)

2. Update the model access controls in the Project's Model Management settings, or ask a Tenant admin to update the Tenant-scoped model settings. Refer to Create and Manage Projects for details on Model as a Service Mappings and model access control lists.

3. If the deployment was already created and remains out of policy, delete it and deploy the model again after the allow list is updated.

Workloads Unhealthy or Stuck on the Spoke Cluster

Symptom: The deployment status is Unhealthy, or the WorkloadDeploymentReady condition stays False while the Compute Pool is Running. The child Workload Deployments report a PHASE of rendering, unhealthy, or failed.

Possible causes:

1. A workload component failed to render, validate, or apply.

2. Resources were applied on the spoke cluster, but one or more components are not healthy — for example, pods stuck in ImagePullBackOff or Pending due to GPU scheduling.

3. For Model Deployments, the model download from Hugging Face or the image pull from NVIDIA NGC failed due to missing or invalid integration credentials.

4. The Environment cannot place the workload on any spoke cluster.

Resolution:

1. Check the phase and conditions of each child Workload Deployment.

   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output jsonpath='{.status.phase}'
   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output json | jq '.status.conditions'

   The condition types follow the workload lifecycle: Parsed, Rendered, DryRun, Dispatched, Applied, and Ready. The first condition that is False indicates the failing stage, and its message contains the underlying error.

2. Review the per-cluster workload status aggregated on the Workload Deployment. Each entry reports the phase and component health for one spoke cluster.

   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output json | jq '.status.statuses'

   Unhealthy components include a details message that reports the failing resource — for example, 0 / 1 replicas ready. For workloads with multiple deployment priorities, work backwards to the first priority that is not running and inspect its Ready-<N> condition. Refer to Deployment Priorities for the full triage procedure.

3. For Model Deployments with components stuck pulling images or downloading models, verify the Hugging Face and NVIDIA NGC integration secrets configured in the Settings resource. Refer to Settings Integrations Not Ready for the verification steps.

4. If the Workload Deployment reports that the Environment cannot complete placements, or no spoke clusters were selected, verify the Environment and the Compute Pool's cluster registration.

   kubectl get environments --namespace <project-namespace>
   kubectl describe workloaddeployment <workloaddeployment-name> --namespace <project-namespace>

5. If the failing stage is not clear from conditions and events, review the Hue controller logs on the hub cluster. The controllers run in the Helm release namespace, which defaults to mural-system.

   kubectl get pods --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers
   kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

Deployment Request Rejected

Symptom: Creating or updating a deployment fails immediately with an admission webhook error.

Cause: PaletteAI validates AIWorkload and WorkloadDeployment resources at admission time. Requests that violate the validation rules are rejected before any resources are created.

Resolution:

Match the error message against the following common rejections and correct the manifest or request.

Message pattern	Meaning and fix
computePoolRef and clusterVariant mutual exclusivity error	Set exactly one of spec.computePoolRef or spec.clusterVariant, never both or neither.
label "palette.ai/model-deployment-type" must not be set when "palette.ai/aiworkload-type"="app"	The palette.ai/model-deployment-type label is only valid for Model Deployments. Remove it from App Deployments.
Missing or invalid palette.ai/aiworkload-type label	Set the label to app or model. Model Deployments must also set palette.ai/model-deployment-type to HF, NGC, or custom-model.
ComputePool <namespace>/<name> is not shared with project "<namespace>"	The referenced Compute Pool belongs to another Project and is not shared. Target a Compute Pool in your Project or ask its owner to share it.
variable '<name>' is not defined in any VariableSet	The deployment sets a variable that the workload profile does not define. Remove the variable or add it to the profile.
cannot override locked variable '<name>'	The variable is locked by the workload profile. Remove the override.
user input is required for variable '<name>' but not provided in WorkloadDeployment	A required variable has no value. Provide a value for the variable in the deployment configuration.

If the rejection message references a webhook failure rather than a validation rule, verify that the Hue webhook pods are healthy in the Helm release namespace, which defaults to mural-system.

kubectl get pods --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-webhooks

Cannot Delete Deployment

Symptom: The deployment status stays Deleting and the resource is not removed.

Possible causes:

1. The workloads on the spoke cluster are still terminating.

2. The deployment created a Compute Pool with deletionPolicy: "delete", and the Compute Pool deletion has not completed.

3. A finalizer is blocked because a downstream resource cannot be cleaned up.

Resolution:

1. Check whether the child Workload Deployments are still present.

   kubectl get workloaddeployments --namespace <project-namespace>

2. If the deployment created a Compute Pool inline with deletionPolicy: "delete", the Compute Pool must be deleted first. Check its status and, if the Compute Pool deletion is stuck, refer to Compute Pool Cannot Be Deleted.

   kubectl get computepool --namespace <project-namespace>

3. If deletion does not complete, review the Hue controller logs for errors related to the deployment. The controllers run in the Helm release namespace, which defaults to mural-system.

   kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

   warning

   Finalizers protect resources from premature deletion while cleanup occurs. Do not manually remove finalizers unless controller logs show the finalizer is stuck due to a known issue and you have consulted with your PaletteAI administrator or support team. Manually removing finalizers can leave orphaned workloads on spoke clusters.

Resources

• Create and Manage App Deployments

• Create and Manage Model Deployments

• AIWorkloads Concepts

• Deployment Priorities

• Troubleshooting Compute Pools

• Troubleshooting Tenants