Troubleshooting App and Model Deployments

This page provides troubleshooting guidance for common issues when App Deployments and Model Deployments fail to reach the Running status or become unhealthy after deployment.

Both App Deployments and Model Deployments are represented by the AIWorkload custom resource on the hub cluster. The AIWorkload generates one Workload Deployment per configured workload, and each Workload Deployment renders a Workload on the target spoke clusters. Failures can surface at any stage of this chain, so triage proceeds from the AIWorkload down to the individual workloads.

info

Troubleshooting steps on this page use kubectl commands to inspect Kubernetes resources on the hub cluster, and pipe several of them through jq to extract fields from the JSON output. You need kubectl access to the hub cluster and the Project namespace where your deployment was created, plus jq on your local machine. If you do not have kubectl access, contact your platform administrator.

Deployment Status Overview

The AIWorkload reports its overall state in the status.status field, which the UI displays as the App Deployment or Model Deployment status: Provisioning, Running, Unhealthy, Failed, or Deleting.

The status.conditions field explains how the overall state was derived:

• ComputePoolReady - The target Compute Pool is provisioned and ready.

• ModelInPolicy - For Model Deployments from a model catalog, the selected model is allowed by the model access controls configured in the Project or Tenant settings.

• QuotaInPolicy - The effective InferenceQuota resolved for this AIWorkload is in policy and not exhausted. The effective quota is chosen from the deployment, Project, and Tenant scopes; when none is configured, the condition is True.

• WorkloadDeploymentCreated - The Workload Deployments were generated.

• WorkloadDeploymentReady - The generated Workload Deployments are ready.

Start any investigation by reviewing the deployment in the PaletteAI UI. Select the App Deployment or Model Deployment from the list and review its status details. If the UI does not provide enough detail, inspect the AIWorkload directly.

kubectl get aiworkload <name> --namespace <project-namespace>
kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

The status.workloadDeployments field lists the names of the child Workload Deployments generated by the AIWorkload. Use these names to drill into workload-level failures.

kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.workloadDeployments'

Deployment Stuck in Provisioning

Symptom: The deployment status remains Provisioning and does not transition to Running.

Possible causes:

1. The target Compute Pool is still provisioning or is not ready.

2. The Compute Pool's Environment is not ready, so Workload Deployments cannot be generated.

3. The generated Workload Deployments have not finished rolling out to the spoke clusters.

Resolution:

1. Check the AIWorkload conditions to identify the blocked stage.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

2. If ComputePoolReady is False with a message similar to Waiting for ComputePool to be ready (current: Provisioning), the deployment is waiting on infrastructure. Identify the Compute Pool and check its status.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.computePoolRef'
   kubectl get computepool <computepool-name> --namespace <project-namespace>

   If the Compute Pool is stuck in Provisioning or shows Failed, refer to Troubleshooting Compute Pools to resolve the infrastructure issue first. Deployments that create a new Compute Pool inline can take 15 or more minutes to provision both the infrastructure and the application.

3. If WorkloadDeploymentCreated is False with the message ComputePool Environment not ready, the Compute Pool is up but its Environment has not been published yet. Verify the Environment referenced by the Compute Pool.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status'
   kubectl get environments --namespace <project-namespace>

4. If WorkloadDeploymentReady is False with the message Waiting for WorkloadDeployments to become ready, the workloads are still rolling out. List the child Workload Deployments and check their phase.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.workloadDeployments'
   kubectl get workloaddeployments --namespace <project-namespace>

   A Workload Deployment that stays in rendering or starting, or that reports unhealthy or failed, blocks the deployment from reaching Running. Refer to Workloads Unhealthy or Stuck on the Spoke Cluster to triage the workload rollout.

Deployment Failed

Symptom: The deployment status is Failed.

Cause: One of the AIWorkload conditions is False with a failure reason such as ValidationFailed, ComputePoolFailed, or WorkloadDeploymentFailed. Validation failures are the most common cause and are reported in the condition message.

Resolution:

1. Check the conditions and read the failure message.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions'

2. Match the failure message against the following common validation errors.

   Message pattern	Meaning and fix
   insufficient GPU <variant>: requested N, available M	The Compute Pool does not have enough free GPUs of the requested variant. Reduce the deployment's hardwareRequests, free capacity by deleting other deployments, or target a Compute Pool with more GPU capacity.
   no GPU capacity available for architecture <arch>	The Compute Pool has no GPU capacity for the requested architecture. Target a Compute Pool that provides GPUs for that architecture.
   GPU variant <variant> not available in ComputePool	The requested GPU variant does not exist in the Compute Pool. Adjust the GPU variant in hardwareRequests or target a different Compute Pool.
   insufficient CPU for architecture <arch>: requested N, available M	The Compute Pool does not have enough free CPU. Reduce the requested CPU or free capacity.
   inference engine mismatch: AIWorkload requires <x> but ComputePool <cp> has <y>	Model Deployments must use an inference engine compatible with the target Compute Pool. Select a Profile Bundle that matches the Compute Pool's inference engine, or target a Compute Pool provisioned with the required engine.

3. Check the Compute Pool allocation to confirm available capacity before redeploying.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status.hardwareCapacity'
   kubectl get computepool <computepool-name> --namespace <project-namespace> --output json | jq '.status.hardwareAllocation'

4. Correct the configuration by adjusting the deployment settings. Refer to Create and Manage App Deployments and Create and Manage Model Deployments.

If the deployment is blocked by a GPU quota rather than physical capacity, the relevant caps are defined on the Project and Tenant. Refer to GPU Quota Exceeded and Insufficient GPU Resources for details on how the caps interact and how to raise them.

Model Deployment Out of Policy

Symptom: A Model Deployment does not start, and the ModelInPolicy condition is False with the reason ModelOutOfPolicy. Updates to an out-of-policy deployment are rejected with an error similar to: AIWorkload is out of policy: the deployed model is no longer authorized under the active model settings; delete and re-deploy to resolve.

Cause: Model Deployments sourced from Hugging Face or NVIDIA NGC are checked against the model access controls configured at the Project or Tenant scope. The ModelInPolicy condition specifically verifies the model against the access control list. If the model list is not configured, is disabled, or does not allow the selected model repository or NIM image, PaletteAI does not create any Workload Deployments for the model. Custom model deployments are not subject to registry policy.

The access control list denies every resource that is not in the allow list, and the disallow list subtracts from the allow list. The denial reason identifies which list caused the denial: an explicit entry in the disallow list, a deny-all configuration (* in the disallow list), or absence from the allow list.

Resolution:

1. Read the ModelInPolicy condition message. It identifies the failing registry and scope.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions[] | select(.type == "ModelInPolicy")'

   Example messages:

   • Hugging Face model deployments are disabled at the <scope> scope; enable the Hugging Face model list to deploy Hugging Face models

   • model repository "<repo>" is in the Hugging Face disallow list at the <scope> scope; remove it from the disallow list

   • model repository "<repo>" is blocked by deny-all at the <scope> scope; add it to the Hugging Face allow list or turn off deny-all

   • model repository "<repo>" is not in the Hugging Face allow list at the <scope> scope; add it to the allow list or enable allow-all

   • NGC image "<repo>/<container>" is in the NIMs disallow list at the <scope> scope; remove it from the disallow list

   • NGC image "<repo>/<container>" is blocked by deny-all at the <scope> scope; add it to the NIMs allow list or turn off deny-all

   • NGC image "<repo>/<container>" is not in the NIMs allow list at the <scope> scope; add it to the allow list or enable allow-all

   Because the disallow list takes precedence over the allow list, adding the repository or image to the allow list does not override an explicit disallow entry — the resolution depends on the message: for an explicit disallow entry, remove that entry from the disallow list; for deny-all or absence from the allow list, add the repository or image to the allow list of the scope in the message or adjust the * wildcard configuration.

2. Update the model access controls in the Project's Model Management settings, or ask a Tenant admin to update the Tenant-scoped model settings. Refer to Create and Manage Projects for details on Model as a Service Mappings and model access control lists, and to Configure Project Model Settings for allow, disallow, allow-all, and deny-all semantics.

3. If the deployment was already created and remains out of policy, delete it and deploy the model again after the allow list is updated.

Quota Out of Policy

Symptom: The QuotaInPolicy condition on the AIWorkload is False with the reason QuotaOutOfPolicy.

Cause: The effective InferenceQuota configured for the deployment is out of policy or exhausted. InferenceQuotas control request-rate limits and other policy constraints on model deployments. When a quota is unhealthy or its limits are exceeded, the QuotaInPolicy condition reports False, and PaletteAI does not create or update Workload Deployments until the quota recovers.

Resolution:

1. Check the QuotaInPolicy condition message to identify the quota issue.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.status.conditions[] | select(.type == "QuotaInPolicy")'

2. If the deployment references an InferenceQuota, verify the quota resource and its status.

   kubectl get aiworkload <name> --namespace <project-namespace> --output json | jq '.spec.inferenceQuotaRef'
   kubectl get inferencequota <quota-name> --namespace <project-namespace>

3. Wait for the quota to recover, or contact your platform administrator to adjust the InferenceQuota settings. Refer to the InferenceQuota documentation for details on quota configuration and limits.

Workloads Unhealthy or Stuck on the Spoke Cluster

Symptom: The deployment status is Unhealthy, or the WorkloadDeploymentReady condition stays False while the Compute Pool is Running. The child Workload Deployments report a PHASE of rendering, starting, unhealthy, or failed. The full set of phases is rendering, starting, running, unhealthy, failed, and deleting.

Possible causes:

1. A workload component failed to render, validate, or apply.

2. Resources were applied on the spoke cluster, but one or more components are not healthy — for example, pods stuck in ImagePullBackOff or Pending due to GPU scheduling.

3. For Model Deployments, the model download from Hugging Face or the image pull from NVIDIA NGC failed due to missing or invalid integration credentials.

4. The Environment cannot place the workload on any spoke cluster.

Resolution:

1. Check the phase and conditions of each child Workload Deployment.

   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output jsonpath='{.status.phase}'
   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output json | jq '.status.conditions'

   The condition types follow the workload lifecycle: Parsed, Rendered, DryRun, Dispatched, Applied, and Ready. The first condition that is False indicates the failing stage, and its message contains the underlying error.

2. Review the per-cluster workload status aggregated on the Workload Deployment. Each entry reports the phase and component health for one spoke cluster.

   kubectl get workloaddeployment <workloaddeployment-name> --namespace <project-namespace> --output json | jq '.status.statuses'

   Unhealthy components include a details message that reports the failing resource — for example, 0 / 1 replicas ready. For workloads with multiple deployment priorities, work backwards to the first priority that is not running and inspect its Ready-<N> condition. Refer to Deployment Priorities for the full triage procedure.

3. For Model Deployments with components stuck pulling images or downloading models, verify the Hugging Face and NVIDIA NGC integration secrets configured in the Settings resource. Refer to Settings Integrations Not Ready for the verification steps.

4. If the Workload Deployment reports that the Environment cannot complete placements, or no spoke clusters were selected, verify the Environment and the Compute Pool's cluster registration.

   kubectl get environments --namespace <project-namespace>
   kubectl describe workloaddeployment <workloaddeployment-name> --namespace <project-namespace>

5. If the failing stage is not clear from conditions and events, review the Hue controller logs on the hub cluster. The controllers run in the Helm release namespace, which defaults to mural-system.

   kubectl get pods --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers
   kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

Model as a Service Mapping Does Not Resolve

Symptom: The Model Setup step stops with one of the following messages:

• No model as a service mapping was found. Please contact your administrator to configure the mapping in project settings.
• The profile bundle "<name>" mapped to this model exists in more than one namespace (<namespaces>). Ask your administrator to make the profile bundle name unique.

Cause: The first message means no Model as a Service Mapping matched the selected model's source and filters. The second means a mapping matched, but the Profile Bundle name it targets resolves to more than one bundle outside the Model Deployment's namespace. Because a mapping stores only the bundle name, PaletteAI will not choose between same-named bundles in different namespaces.

Resolution:

1. For the first message, ask an administrator to add a Model as a Service Mapping for the source, or to adjust the source filters of the mapping. Refer to Create Model as a Service Mappings.

2. For the second message, rename one of the same-named Profile Bundles so the mapped name resolves to exactly one bundle, or point the mapping at a bundle name that is unique in the scope. Refer to How a Mapping Resolves to a Profile Bundle for the resolution order.

Deployment Request Rejected

Symptom: Creating or updating a deployment fails immediately with an admission webhook error.

Cause: PaletteAI validates AIWorkload and WorkloadDeployment resources at admission time. Requests that violate the validation rules are rejected before any resources are created.

Resolution:

Match the error message against the following common rejections and correct the manifest or request.

Message pattern	Meaning and fix
computePoolRef and clusterVariant mutual exclusivity error	Set exactly one of spec.computePoolRef or spec.clusterVariant, never both or neither.
label "palette.ai/model-deployment-type" must not be set when "palette.ai/aiworkload-type"="app"	The palette.ai/model-deployment-type label is only valid for Model Deployments. Remove it from App Deployments.
Missing or invalid palette.ai/aiworkload-type label	Set the label to app or model. Model Deployments must also set palette.ai/model-deployment-type to HF, NGC, or custom-model.
ComputePool <namespace>/<name> is not shared with project "<namespace>"	The referenced Compute Pool belongs to another Project and is not shared. Target a Compute Pool in your Project or ask its owner to share it.
variable '<name>' is not defined in any VariableSet	The deployment sets a variable that the workload profile does not define. Remove the variable or add it to the profile.
cannot override locked variable '<name>'	The variable is locked by the workload profile. Remove the override.
user input is required for variable '<name>' but not provided in WorkloadDeployment	A required variable has no value. Provide a value for the variable in the deployment configuration.

If the rejection message references a webhook failure rather than a validation rule, verify that the Hue webhook pods are healthy in the Helm release namespace, which defaults to mural-system.

kubectl get pods --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-webhooks

Cannot Delete Deployment

Symptom: The deployment status stays Deleting and the resource is not removed.

Possible causes:

1. The workloads on the spoke cluster are still terminating.

2. The deployment created a Compute Pool with deletionPolicy: "delete", and the Compute Pool deletion has not completed.

3. A finalizer is blocked because a downstream resource cannot be cleaned up.

Resolution:

1. Check whether the child Workload Deployments are still present.

   kubectl get workloaddeployments --namespace <project-namespace>

2. If the deployment created a Compute Pool inline with deletionPolicy: "delete", the Compute Pool must be deleted first. Check its status and, if the Compute Pool deletion is stuck, refer to Compute Pool Cannot Be Deleted.

   kubectl get computepool --namespace <project-namespace>

3. If deletion does not complete, review the Hue controller logs for errors related to the deployment. The controllers run in the Helm release namespace, which defaults to mural-system.

   kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

   warning

   Finalizers protect resources from premature deletion while cleanup occurs. Do not manually remove finalizers unless controller logs show the finalizer is stuck due to a known issue and you have consulted with your PaletteAI administrator or support team. Manually removing finalizers can leave orphaned workloads on spoke clusters.

Resources

• Create and Manage App Deployments

• Create and Manage Model Deployments

• AIWorkloads Concepts

• Deployment Priorities

• Troubleshooting Compute Pools

• Troubleshooting Tenants