Troubleshooting Compute Pools

This page provides troubleshooting guidance for common Compute Pool issues.

info

Troubleshooting steps on this page use kubectl commands to inspect Kubernetes resources on the hub cluster. You need kubectl access to the hub cluster and the Project namespace where your Compute Pool is deployed. If you do not have kubectl access, contact your platform administrator.

Compute Pool Stuck in Provisioning

Symptom: The Compute Pool status remains Provisioning.

Possible causes:

1. No available edge hosts match the resource requirements.

2. Palette cluster deployment fails.

3. Network issues prevent cluster creation.

Resolution:

Start by checking the Compute Pool in the PaletteAI UI. Select the Compute Pool from the list and review the Events tab for error messages and status updates. The UI shows the current status, cluster health, and recent events that can help identify the issue without kubectl.

If the UI does not provide enough detail, use the following kubectl commands.

1. Check Compute Pool conditions.

   kubectl describe computepool <computepool-name> --namespace <project-namespace> | grep -A 10 Conditions

   Expected output for healthy cluster:

   Conditions:
     Type: ClusterDeployed
     Status: True
     Reason: ClusterRunning
     Message: Palette cluster is running
   
     Type: Validated
     Status: True
     Reason: ValidationPassed

   Example failure output:

   Conditions:
     Type: Validated
     Status: False
     Reason: ValidationFailed
     Message: ComputePool validation failed
   
     Type: ClusterDeployed
     Status: False
     Reason: ClusterDeploymentFailed
     Message: Failed to deploy Palette cluster

   Common condition types and failure reasons:

   • Validated / ValidationFailed - ComputePool spec validation failed. Check that referenced resources (ProfileBundle, Settings) exist and are Ready.
   • ClusterDeployed / ClusterDeploymentFailed - Palette cluster deployment failed. Check Settings credentials and available edge hosts.
   • AdminKubeconfigAcquired / KubeconfigFailed - Failed to retrieve the admin kubeconfig from the Palette cluster.
   • SpokeCreated / SpokeCreateFailed - Failed to create the Spoke resource for the cluster.
   • ManagedClusterReady / ManagedClusterNotReady - The OCM ManagedCluster representing the Palette cluster is not ready.
   • EnvironmentCreated / EnvironmentFailed - The Mural Environment failed to be created or become ready.

2. Verify Compute resources have available machines.

   kubectl get compute --namespace <project-namespace> --output yaml | grep -A 20 controlPlaneCompute

   Expected output:

   status:
     controlPlaneCompute:
       - architecture: amd64
         cpuCount: 8
         memory: '32768 MiB'
         instances: 1
         machines:
           abc123: available
         resourceGroups:
           palette.ai: 'true'
     workerCompute:
       - architecture: amd64
         cpuCount: 16
         memory: '65536 MiB'
         instances: 1
         machines:
           def456: available
         family: NVIDIA-A100
         gpuCount: 2
         gpuMemory: '40960 MiB'
         resourceGroups:
           palette.ai: 'true'

   If controlPlaneCompute or workerCompute is empty, no hosts are available with the required resources.

3. Check Palette cluster status.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output jsonpath='{.status.paletteClusterStatuses}' | jq

   Expected output for healthy cluster:

   [
     {
       "name": "ml-cluster",
       "uid": "abc123def456",
       "status": "Running",
       "conditions": [
         {
           "type": "ClusterDeployed",
           "status": "True",
           "reason": "ClusterRunning"
         }
       ]
     }
   ]

   Common status field meanings:

   • status: Cluster lifecycle state (Provisioning, Running, Deleting, Failed, Unhealthy)
   • uid: The Palette cluster UID
   • conditions: Array of condition objects tracking each reconciliation phase
   • kubeconfigSecretName: Name of the Secret containing the cluster kubeconfig

   Example failure output:

   [
     {
       "name": "ml-cluster",
       "uid": "abc123def456",
       "status": "Provisioning",
       "conditions": [
         {
           "type": "ClusterDeployed",
           "status": "False",
           "reason": "ClusterDeploymentFailed",
           "message": "Failed to deploy Palette cluster"
         }
       ]
     }
   ]

4. Verify the ProfileBundle exists.

   kubectl get profilebundle <profilebundle-name> --namespace <project-namespace>

   Expected output:

   NAME                READY   AGE
   edge-profilebundle  True    5d

   If READY is False, describe the ProfileBundle to identify the cause:

   kubectl describe profilebundle <profilebundle-name> --namespace <project-namespace>

VIP Already in Use

Symptom: Compute Pool creation fails due to a VIP conflict.

Cause: The VIP is assigned to another cluster or device.

Resolution:

warning

The VIP address cannot be changed after cluster creation. If you need to use a different VIP, you must delete the Compute Pool and create a new one.

1. For a new Compute Pool (not yet created successfully):

   a. Choose a VIP that is not in use.

   b. Check VIP values in existing Compute Pools to avoid conflicts:

   kubectl get computepools --namespace <project-namespace> --output yaml | grep -A 1 "vip:"

   c. Verify the VIP is not in use on your network:

   # Option 1: Use arping (recommended, more reliable)
   # Requires root/sudo on most systems. No reply expected if VIP is free.
   sudo arping -c 3 <new-vip-address>
   
   # Expected output if VIP is free (no replies):
   # ARPING 10.10.162.130
   # ... (timeout, 0 packets received)
   
   # Option 2: Check if VIP responds to ping (less reliable)
   # Some networks may not respond to ping even if IP is in use
   ping -c 3 <new-vip-address>
   
   # Expected output if VIP is free:
   # ... 100% packet loss
   
   # Option 3: Check ARP table (only shows recently resolved IPs)
   arp -a | grep <new-vip-address>
   
   # Expected output if VIP is free:
   # (no output)

   d. Update your Compute Pool manifest with the new VIP and apply it:

   # Edit the manifest
   vi computepool.yaml
   
   # Update the VIP under spec.clusterVariant.dedicated.paletteClusterDeploymentConfig.edge.vip
   
   # Apply the updated manifest
   kubectl apply --filename computepool.yaml

2. For an existing Compute Pool (already created with wrong VIP):

   The VIP is immutable. You must delete and recreate the Compute Pool:

   a. Back up any workloads or data from the cluster.

   b. Delete the Compute Pool:

   kubectl delete computepool <computepool-name> --namespace <project-namespace>

   c. Verify the Compute Pool is deleted:

   kubectl get computepool <computepool-name> --namespace <project-namespace>

   The cluster will be removed from Palette. Once deletion completes, the command returns a NotFound error.

   d. Update your manifest with the correct VIP.

   e. Create the Compute Pool again:

   kubectl apply --filename computepool.yaml

Insufficient GPU Resources

Symptom: Worker pool creation fails because GPU quota is exceeded.

Cause: A new GPU request is blocked when it would exceed any cap that applies to it. For a Compute Pool in a Project namespace, the relevant caps are:

• Project limits (<project>.spec.gpuResources.limits.<family>) - Caps total GPU usage by all Compute Pools and App Deployments in the Project. Set by the Project owner.
• Tenant projectLimits (<tenant>.spec.gpuResources.projectLimits.<project>.<family>) - Tenant-admin-imposed ceiling on what this specific Project may consume, applied in addition to the Project's own limits. When both this and the Project's own limits are set, the lower of the two is the Project's effective per-Project cap.
• Tenant limits minus tenantReservations - The Tenant's aggregate limits minus any tenantReservations is the combined budget shared across every Project. Enforced at aggregate scope, separate from the per-Project caps above.

Resolution:

1. Check Project GPU limits and current usage.

   kubectl get project <project-name> --namespace <project-namespace> --output jsonpath='{.spec.gpuResources}'
   kubectl get project <project-name> --namespace <project-namespace> --output jsonpath='{.status.gpuUsage}'

2. Check Tenant-side caps that apply to this Project.

   kubectl get tenant <tenant-name> --output jsonpath='{.spec.gpuResources.limits}'
   kubectl get tenant <tenant-name> --output jsonpath="{.spec.gpuResources.projectLimits['<project-name>']}"
   kubectl get tenant <tenant-name> --output jsonpath='{.status.gpuUsage}'

3. Reduce GPU requirements in the Compute Pool, raise the Project's limits (within the Tenant's caps), or request that the Tenant admin raise the Tenant limits or this Project's projectLimits entry.

Compute Pool Cannot Be Deleted

Symptom: Compute Pool deletion fails or does not complete.

Possible causes:

1. AIWorkloads still run on the cluster.

2. A finalizer prevents deletion.

3. Palette cluster deletion fails.

Resolution:

Start by checking the Compute Pool detail page in the PaletteAI UI. The Overview tab shows deployed workloads and cluster status. If workloads are still running, delete them from the Workloads section before retrying the Compute Pool deletion.

If the UI does not provide enough detail, use the following kubectl commands.

1. List AIWorkload references.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output jsonpath='{.status.aiWorkloadRefs}'

2. Delete AIWorkloads.

   kubectl delete aiworkload <aiworkload-name> --namespace <project-namespace>

3. Check finalizers.

   kubectl get computepool <computepool-name> --namespace <project-namespace> --output jsonpath='{.metadata.finalizers}'

   Example output:

   ["spectrocloud.com/computepool-finalizer"]

   warning

   Finalizers protect resources from premature deletion while cleanup occurs. Do not manually remove finalizers unless:

   • You have confirmed all AIWorkloads are deleted.

   • Controller logs show the finalizer is stuck due to a known issue.

   • You have consulted with your PaletteAI administrator or support team.

   Manually removing finalizers can leave orphaned resources in Palette or cause state inconsistencies.

   If you must remove a finalizer (after confirming with support), use:

   kubectl patch computepool <computepool-name> --namespace <project-namespace> \
     --type json -p '[{"op": "remove", "path": "/metadata/finalizers/0"}]'

4. If deletion does not complete, review controller logs.

   First, find the Hue controller pods:

   kubectl get pods --namespace mural-system --selector controller.spectrocloud.com/name=hue-controllers

   Then review the logs:

   kubectl logs --namespace mural-system --selector controller.spectrocloud.com/name=hue-controllers --tail=100 --follow

   If the mural-system namespace or labels are not found, discover the controller deployment:

   kubectl get deployments --all-namespaces | grep -i hue

Node Stuck in Provisioning (Air-Gapped)

Symptom: An edge node remains stuck in a Provisioning state and never transitions to Running in an air-gapped environment.

Possible causes:

1. Pack components are attempting to reach an external endpoint that is unreachable in the air-gapped environment.

2. Registry credentials are not correctly embedded in the node's user-data.

3. Required images are missing from the internal Zot registry (for example, edge-native-byoi, edge-k8s, cni-cilium-oss, or piraeus-operator).

4. Network policy is blocking the node from reaching the internal Zot registry IP or port.

Resolution:

1. Check which images are being pulled on the node.

   crictl images

   Compare the listed images against what is available in your internal Zot registry. Any missing images must be pushed to Zot before cluster deployment.

2. Verify that the containerd registry configuration points to the internal Zot instance.

   cat /etc/containerd/config.toml | grep --after-context=5 registry

   The registry endpoint should resolve to your internal Zot address and port (default 30003), not an external host.

3. Test connectivity from the node to the internal Zot registry.

   curl --insecure https://<vertex-ip>:30003/v2/

   A 200 OK or 401 Unauthorized response confirms the registry endpoint is reachable. A connection refused or timeout indicates a network or firewall issue.

4. Check Palette agent logs for pull failures or registry errors.

   journalctl --unit stylus-agent --follow

   Look for image pull errors, TLS certificate issues, or authentication failures against the registry.

5. Verify that all required pack .zst files were successfully imported into Zot before cluster deployment. Cluster Profiles cannot be imported and packs cannot be resolved until they are present in the registry. Refer to Deploy Profile Bundles in Air-Gapped Environments for the correct import sequence.

GPU Pods Stuck with No NVIDIA Runtime Configured

Symptom: On a GPU edge host, NVIDIA GPU Operator pods remain stuck in Init:CreateContainerError or Init:0/1. The Kubelet reports the following event, repeating roughly every 15 seconds:

Failed: spec.initContainers{driver-validation}:
  Error: failed to get sandbox runtime: no runtime for "nvidia" is configured

Image pulls succeed, so this is not a registry, authentication, or air-gapped problem. It is a Container Runtime Interface (CRI) runtime-registration problem: the nvidia RuntimeClass exists in Kubernetes, but no matching containerd runtime handler is registered behind it.

Possible causes:

1. containerd configuration schema mismatch. The edge host base image ships /etc/containerd/config.toml using schema version = 2 (CRI plugin key io.containerd.grpc.v1.cri). The NVIDIA GPU Operator container-toolkit DaemonSet writes a drop-in at /etc/containerd/conf.d/99-nvidia.toml using schema version = 3 (CRI plugin key io.containerd.cri.v1.runtime). Because the two schema versions differ, containerd loads only the runtimes it finds under the version 2 CRI path—runc—and ignores the nvidia, nvidia-cdi, and nvidia-legacy runtimes from the drop-in. A version = 3 imports tree does not deep-merge the runtimes map; it replaces at the block level.

2. A duplicate containerd service holds the runtime lock. On some hosts, the operating system containerd.service is installed alongside spectro-containerd.service. Both services read the same config.toml and metadata database. When both are active, restarting spectro-containerd lets the operating system service acquire the file lock and lock out the Spectro Cloud service, so runtime changes never take effect.

Resolution:

warning

Editing containerd configuration on an edge host is a low-level operation. Back up the existing configuration before you make changes, and apply the change to one host at a time.

info

These steps use two command-line tools on the edge host: crictl to inspect the container runtime and jq to parse JSON output. If either tool is not available on the host, install it from the linked release page before you continue.

1. Confirm which runtimes containerd has registered.

   sudo crictl --runtime-endpoint=unix:///run/spectro/containerd/containerd.sock info \
     | jq '.config.containerd.runtimes | keys'

   If the output lists only ["runc"] and not nvidia, the NVIDIA runtime handler is not registered. A schema mismatch is the most likely cause, but a duplicate or conflicting containerd service can produce the same result. Complete the service check in the next step before you edit any configuration.

2. Check whether a duplicate containerd service is running.

   systemctl is-active containerd spectro-containerd
   journalctl --unit containerd --since "-5m" | grep "starting containerd"

   If both services are active, or the operating system containerd service restarts on a short loop, a duplicate service is holding the runtime lock. Mask the operating system service as part of the fix below. If only spectro-containerd is active, the cause is the configuration schema mismatch.

3. Confirm the containerd major version before you change the configuration schema.

   containerd --version

   The version = 3 configuration schema is native to containerd 2.x. If the host runs containerd 1.x, keep version = 2 and add the NVIDIA runtimes under the io.containerd.grpc.v1.cri plugin key instead. Applying a version = 3 file to a 1.x host prevents containerd from starting.

4. Back up the current configuration.

   sudo cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak

5. On a containerd 2.x host, edit /etc/containerd/config.toml so that it declares version = 3 and defines every runtime directly rather than relying on the drop-in. Because a version = 3 imports tree does not deep-merge the runtimes map, each runtime must be present in the main file.

   Preserve any settings the host already relies on, such as CNI, registry mirror, snapshotter, and proxy configuration. Add or update only the CRI runtime block shown below and leave unrelated sections in place.

   version = 3
   root = "/var/lib/spectro/containerd"
   state = "/run/spectro/containerd"
   
   [grpc]
     address = "/run/spectro/containerd/containerd.sock"
   
   [plugins]
     [plugins."io.containerd.cri.v1.images".pinned_images]
       sandbox = "registry.k8s.io/pause:3.6"
   
     [plugins."io.containerd.cri.v1.runtime".containerd]
       default_runtime_name = "runc"
   
       [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
         runtime_type = "io.containerd.runc.v2"
         [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
           BinaryName = "/opt/bin/runc"
           SystemdCgroup = true
   
       [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
         runtime_type = "io.containerd.runc.v2"
         [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia.options]
           BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
           SystemdCgroup = true
   
       [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-cdi]
         runtime_type = "io.containerd.runc.v2"
         [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-cdi.options]
           BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi"
           SystemdCgroup = true
   
       [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-legacy]
         runtime_type = "io.containerd.runc.v2"
         [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-legacy.options]
           BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy"
           SystemdCgroup = true

   The GPU Operator drop-in at /etc/containerd/conf.d/99-nvidia.toml can stay on disk. Once the main configuration declares version = 3, containerd reads the runtime definitions from the main file directly.

6. Validate the rendered configuration before you restart the service.

   sudo containerd config dump | grep nvidia

   The command prints the parsed configuration. If it errors or prints nothing, the file is invalid or the NVIDIA runtimes are missing. Correct the file before you continue.

7. If a duplicate operating system containerd service is present, stop and mask it so it cannot reacquire the runtime lock.

   sudo systemctl stop containerd
   sudo systemctl mask containerd

8. Restart the Spectro Cloud containerd service.

   sudo systemctl reset-failed spectro-containerd
   sudo systemctl restart spectro-containerd

   If spectro-containerd fails to start, restore the backup to return the host to its previous state, then recheck the configuration:

   sudo cp -a /etc/containerd/config.toml.bak /etc/containerd/config.toml
   sudo systemctl restart spectro-containerd

9. Verify that all four runtimes are registered.

   sudo crictl --runtime-endpoint=unix:///run/spectro/containerd/containerd.sock info \
     | jq '.config.containerd.runtimes | keys'

   Expected output:

   ["nvidia", "nvidia-cdi", "nvidia-legacy", "runc"]

   The stuck GPU Operator pods transition to Running after the nvidia runtime registers.

info

This mismatch occurs when the edge host base image ships a version = 2 containerd configuration while a newer NVIDIA GPU Operator container-toolkit writes version = 3 drop-ins. The long-term fix is for the edge host base image to ship a version = 3 containerd configuration and to avoid installing a second containerd package. If you manage your own edge base image, align its containerd configuration schema with the version that your GPU Operator toolkit emits.

Resources

• Create and Manage Compute Pools

• Debug Scaling Decisions with ComputePoolEvaluation

• Compute Pool Configuration Reference

• Compute Pool Concepts