Skip to main content
Version: v1.2.x

Troubleshooting Compute Pools

This page provides troubleshooting guidance for common Compute Pool issues.

info

Troubleshooting steps on this page use kubectl commands to inspect Kubernetes resources on the hub cluster. You need kubectl access to the hub cluster and the Project namespace where your Compute Pool is deployed. If you do not have kubectl access, contact your platform administrator.

Compute Pool Stuck in Provisioning​

Symptom: The Compute Pool status remains Provisioning.

Possible causes:

  1. No available edge hosts match the resource requirements.

  2. Palette cluster deployment fails.

  3. Network issues prevent cluster creation.

Resolution:

Start by checking the Compute Pool in the PaletteAI UI. Select the Compute Pool from the list and review the Events tab for error messages and status updates. The UI shows the current status, cluster health, and recent events that can help identify the issue without kubectl.

If the UI does not provide enough detail, use the following kubectl commands.

  1. Check Compute Pool conditions.

    kubectl describe computepool <computepool-name> --namespace <project-namespace> | grep --after-context=10 Conditions

    Expected output for healthy cluster:

    Conditions:
      Type: ClusterDeployed
      Status: True
      Reason: ClusterRunning
      Message: Palette cluster is running

      Type: Validated
      Status: True
      Reason: ValidationPassed

    Example failure output:

    Conditions:
      Type: Validated
      Status: False
      Reason: ValidationFailed
      Message: ComputePool validation failed

      Type: ClusterDeployed
      Status: False
      Reason: ClusterDeploymentFailed
      Message: Failed to deploy Palette cluster

    Common condition types and failure reasons:

    • Validated / ValidationFailed - ComputePool spec validation failed. Check that referenced resources (ProfileBundle, Settings) exist and are Ready.
    • ClusterDeployed / ClusterDeploymentFailed - Palette cluster deployment failed. Check Settings credentials and available edge hosts.
    • AdminKubeconfigAcquired / KubeconfigFailed - Failed to retrieve the admin kubeconfig from the Palette cluster.
    • SpokeCreated / SpokeCreateFailed - Failed to create the Spoke resource for the cluster.
    • ManagedClusterReady / ManagedClusterNotReady - The OCM ManagedCluster representing the Palette cluster is not ready.
    • HelmReleaseCreated, HelmReleaseReady - The Helm release that installs PaletteAI services on the spoke cluster failed to be created or become ready.
    • MachinePoolsReady / PoolProvisioningTimedOut, PoolHostsUnhealthy, PoolProvisioningFailed - One or more machine pool hosts failed to provision or become healthy.
    • EnvironmentCreated / EnvironmentFailed - The Environment failed to be created or become ready.

  2. Verify Compute resources have available machines.

    kubectl get compute --namespace <project-namespace> --output yaml | grep --after-context=20 controlPlaneCompute

    Expected output:

    status:
      controlPlaneCompute:
        - architecture: AMD64
          cpuCount: 8
          memory: '32768 MiB'
          instances: 1
          machines:
            abc123: available
          resourceGroups:
            palette.ai: 'true'
      workerCompute:
        - architecture: AMD64
          cpuCount: 16
          memory: '65536 MiB'
          instances: 1
          machines:
            def456: available
          gpuVariant:
            name: 'NVIDIA A100-SXM4-80GB | 80 GB'
            model: 'NVIDIA A100-SXM4-80GB'
            gpuCount: 2
            perGPUMemory: '80 GB'
            totalGPUMemory: '160 GB'
          resourceGroups:
            palette.ai: 'true'

    If controlPlaneCompute or workerCompute is empty, no hosts are available with the required resources.

  3. Check Palette cluster status.

    kubectl get computepool <computepool-name> --namespace <project-namespace> --output jsonpath='{.status.paletteClusterStatuses}' | jq

    Expected output for healthy cluster:

    [
      {
        "name": "ml-cluster",
        "uid": "abc123def456",
        "status": "Running",
        "conditions": [
          {
            "type": "ClusterDeployed",
            "status": "True",
            "reason": "ClusterRunning"
          }
        ]
      }
    ]

    Common status field meanings:

    • status: Cluster lifecycle state (Provisioning, Running, Deleting, Failed, Unhealthy)
    • uid: The Palette cluster UID
    • conditions: Array of condition objects tracking each reconciliation phase
    • kubeconfigSecretName: Name of the Secret containing the cluster kubeconfig

    Example failure output:

    [
      {
        "name": "ml-cluster",
        "uid": "abc123def456",
        "status": "Provisioning",
        "conditions": [
          {
            "type": "ClusterDeployed",
            "status": "False",
            "reason": "ClusterDeploymentFailed",
            "message": "Failed to deploy Palette cluster"
          }
        ]
      }
    ]

  4. Verify the ProfileBundle exists.

    kubectl get profilebundle <profilebundle-name> --namespace <project-namespace>

    Expected output:

    NAME                READY   AGE
    edge-profilebundle  True    5d

    If READY is False, describe the ProfileBundle to identify the cause:

    kubectl describe profilebundle <profilebundle-name> --namespace <project-namespace>

VIP Already in Use​

Symptom: Compute Pool creation fails due to a VIP conflict.

Cause: The VIP is assigned to another cluster or device.

Resolution:

warning

The VIP address cannot be changed after cluster creation. If you need to use a different VIP, you must delete the Compute Pool and create a new one.

  1. For a new Compute Pool (not yet created successfully):

    a. Choose a VIP that is not in use.

    b. Check VIP values in existing Compute Pools to avoid conflicts:

    kubectl get computepools --namespace <project-namespace> --output yaml | grep --after-context=1 "vip:"

    c. Verify the VIP is not in use on your network:

    # Option 1: Use arping (recommended, more reliable)
    # Requires root/sudo on most systems. No reply expected if VIP is free.
    sudo arping -c 3 <new-vip-address>

    # Expected output if VIP is free (no replies):
    # ARPING 10.10.162.130
    # ... (timeout, 0 packets received)

    # Option 2: Check if VIP responds to ping (less reliable)
    # Some networks may not respond to ping even if IP is in use
    ping -c 3 <new-vip-address>

    # Expected output if VIP is free:
    # ... 100% packet loss

    # Option 3: Check ARP table (only shows recently resolved IPs)
    arp -a | grep <new-vip-address>

    # Expected output if VIP is free:
    # (no output)

    d. Update your Compute Pool manifest with the new VIP and apply it:

    # Edit the manifest
    vi computepool.yaml

    # Update the VIP under spec.clusterVariant.dedicated.paletteClusterDeploymentConfig.edge.vip

    # Apply the updated manifest
    kubectl apply --filename computepool.yaml

  2. For an existing Compute Pool (already created with wrong VIP):

    The VIP is immutable. You must delete and recreate the Compute Pool:

    a. Back up any workloads or data from the cluster.

    b. Delete the Compute Pool:

    kubectl delete computepool <computepool-name> --namespace <project-namespace>

    c. Verify the Compute Pool is deleted:

    kubectl get computepool <computepool-name> --namespace <project-namespace>

    The cluster will be removed from Palette. Once deletion completes, the command returns a NotFound error.

    d. Update your manifest with the correct VIP.

    e. Create the Compute Pool again:

    kubectl apply --filename computepool.yaml

Insufficient GPU Resources​

Symptom: Worker pool creation fails because GPU quota is exceeded.

Cause: A new GPU request is blocked when it would exceed any cap that applies to it. For a Compute Pool in a Project namespace, the relevant caps are:

  • Project limits (<project>.spec.gpuResources.limits.<variant>) - Caps total GPU usage by all Compute Pools and App Deployments in the Project. Set by the Project owner.
  • Tenant projectLimits (<tenant>.spec.gpuResources.projectLimits.<project>.<variant>) - Tenant-admin-imposed ceiling on what this specific Project may consume, applied in addition to the Project's own limits. When both this and the Project's own limits are set, the lower of the two is the Project's effective per-Project cap.
  • Tenant limits minus tenantReservations - The Tenant's aggregate limits minus any tenantReservations is the combined budget shared across every Project. Enforced at aggregate scope, separate from the per-Project caps above.

Resolution:

  1. Check Project GPU limits and current usage.

    kubectl get project <project-name> --namespace <project-namespace> --output jsonpath='{.spec.gpuResources}'
    kubectl get project <project-name> --namespace <project-namespace> --output jsonpath='{.status.gpuUsage}'

  2. Check Tenant-side caps that apply to this Project.

    kubectl get tenant <tenant-name> --output jsonpath='{.spec.gpuResources.limits}'
    kubectl get tenant <tenant-name> --output jsonpath="{.spec.gpuResources.projectLimits['<project-name>']}"
    kubectl get tenant <tenant-name> --output jsonpath='{.status.gpuUsage}'

  3. Reduce GPU requirements in the Compute Pool, raise the Project's limits (within the Tenant's caps), or request that the Tenant admin raise the Tenant limits or this Project's projectLimits entry.

Debug Scaling Decisions with ComputePoolEvaluation​

When a Compute Pool references a Scaling Policy, PaletteAI records its scaling decisions in a ComputePoolEvaluation resource. The resource is a live decision record with two writers: the Scaling Policy controller writes the spec with the observed utilization and target resource counts for each worker pool, and the Compute Pool controller writes status.poolScalingStates as it executes the resulting scaling actions. When autoscaling behaves unexpectedly, inspect this resource first to understand what PaletteAI observed and why it acted, or did not act.

ComputePoolEvaluation Lifecycle​

PaletteAI manages ComputePoolEvaluation resources automatically. You do not create, modify, or delete them.

  • Creation - The Scaling Policy controller creates a ComputePoolEvaluation with the same name and namespace as the Compute Pool once the Compute Pool references the Scaling Policy and utilization metrics for the pool's hosts are available from Prometheus. Until the first metrics arrive, no evaluation exists and the Compute Pool reports a ComputePoolEvaluationAvailable condition with the message Waiting for ComputePoolEvaluation.

  • Updates - The spec is refreshed on every metrics evaluation cycle. It is a snapshot of the latest decision, not a history of past decisions.

  • Removal - When you remove the scalingPolicyRef from a Compute Pool, the Compute Pool controller deletes the ComputePoolEvaluation. Because each evaluation is owned by its Compute Pool, it is also garbage collected automatically when the Compute Pool itself is deleted.

Follow a Scale-Up Decision​

The following example uses a Compute Pool named my-compute-pool that references a combined CPU and GPU Scaling Policy. The worker pool contains two hosts, each with 32 CPUs and four NVIDIA H100 GPUs. GPU utilization has stayed above the 85 percent scale-up threshold for the full scale-up duration, so PaletteAI has decided to add a host.

  1. List the evaluations in the Project namespace.

    kubectl get computepoolevaluations --namespace <my-project-namespace>
    Example Output
    NAME              AGE
    my-compute-pool   3d2h

    Each evaluation shares the name and namespace of its Compute Pool. The resource also has the short name cpe, so kubectl get cpe --namespace my-project-namespace returns the same list.

  2. Retrieve the full resource. You can also use kubectl describe computepoolevaluation <my-compute-pool> --namespace <my-project-namespace>, which prints the same information in a readable tree. This walkthrough uses the YAML output because the field names match the CRD.

    kubectl get computepoolevaluation <my-compute-pool> --namespace <my-project-namespace> --output yaml
    Example ComputePoolEvaluation
    apiVersion: spectrocloud.com/v1alpha1
    kind: ComputePoolEvaluation
    metadata:
      name: my-compute-pool
      namespace: my-project-namespace
      ownerReferences:
        - apiVersion: spectrocloud.com/v1alpha1
          blockOwnerDeletion: true
          controller: true
          kind: ComputePool
          name: my-compute-pool
          uid: 4f8f3f6e-9c1a-4a7e-b1d2-0c9a6a1f2b3c
    spec:
      poolEvaluations:
        ml-cluster/worker-pool-nvidia-amd64-0:
          cpu:
            count: 64
            target: 64
            utilization: 41
          evaluatedAt: '2026-07-03T10:12:04Z'
          gpu:
            'NVIDIA H100 PCIe | 80 GB':
              count: 8
              target: 12
              utilization: 92
          nodeWastes:
            - cpuCount: 32
              gpuCount:
                'NVIDIA H100 PCIe | 80 GB': 4
              host: host-uid-3
              wasteScore: 19100
            - cpuCount: 32
              gpuCount:
                'NVIDIA H100 PCIe | 80 GB': 4
              host: host-uid-4
              wasteScore: 23600
      cpuResourceBounds:
        minCPUCount: 4
        maxCPUCount: 128
      gpuResourceBounds:
        - variant: 'NVIDIA H100 PCIe | 80 GB'
          minGPUCount: 1
          maxGPUCount: 16
      cooldownDuration: 15m
      abortDuration: 30m
    status:
      metricsLastObservedAt:
        ml-cluster/worker-pool-nvidia-amd64-0:
          host-uid-3: '2026-07-03T10:11:58Z'
          host-uid-4: '2026-07-03T10:11:58Z'
      poolScalingStates:
        ml-cluster/worker-pool-nvidia-amd64-0:
          state: Scaling
          details: 'Scaling in progress: 0 completed, 1 in progress, 0 failed. Will abort in 28m30s if not completed'
          actions:
            - type: add
              hostUID: host-uid-5
          scalingStartedAt: '2026-07-03T10:12:05Z'
          stateChangedAt: '2026-07-03T10:12:05Z'
          lastSuccessfulScalingAt: '2026-07-02T16:40:11Z'

  3. Read the decision in spec.poolEvaluations. Entries use the format <cluster-name>/<worker-pool-name>, where the worker pool name matches the allocated machine pool name in the Compute Pool status.

    FieldDescription
    cpuCPU metrics and decision for the pool: utilization is the average CPU utilization percentage across the pool, count is the current total CPU count, and target is the total the controller wants.
    gpuThe same metrics and decision per GPU variant, keyed by variant name, such as NVIDIA H100 PCIe | 80 GB.
    nodeWastesA waste score for every host in the pool, used to pick removal candidates during scale-down. Unused GPUs weigh much more heavily than unused CPUs, and the host with the highest wasteScore is removed first.
    evaluatedAtWhen the metrics evaluation was performed.
    cpuResourceBounds, gpuResourceBounds, cooldownDuration, abortDurationCopied from the Scaling Policy at evaluation time. These are the exact bounds and timing parameters that applied to this decision, which is useful when the referenced policy has changed since.

    Compare target with count for each resource: target greater than count means a scale-up decision, target less than count means a scale-down decision, and equal values mean no action is needed. In the example, GPU utilization of 92 percent exceeded the scale-up threshold for the full scale-up duration, so the controller raised the GPU target from 8 to 12 — one additional host with four GPUs.

  4. Read the execution state in status.poolScalingStates, referencing the same <cluster-name>/<worker-pool-name> as the spec. Each entry is a state machine with the following states.

    StateMeaning
    IdleNo scaling operation is in progress. On each evaluation cycle, the controller compares target with count and starts a scaling operation when they differ.
    ScalingNodes are being added or removed. actions lists each add or remove action with the Edge Host UID, and details reports progress counts and the time remaining before a scale-up is aborted.
    CooldownThe pool waits for cooldownDuration before the next scaling decision. Pools enter cooldown after a scaling operation completes, after an abort, after a manual day-2 change to the pool, and when a pool is first evaluated.
    AbortingA scale-up exceeded abortDuration. Pending hosts that never reached Healthy status are removed, while hosts that provisioned successfully are kept. Scale-down operations are never aborted.
    BlockedAutoscaling is temporarily blocked by another operation in progress on the Compute Pool.

    The remaining status fields add timing and health context:

    • details - A human-readable explanation of the current state, such as adding 2 node(s), Cooldown in progress, 8m30s remaining, or Abort timeout exceeded after 30m2s (1 completed, 1 in progress, 0 failed).
    • stateChangedAt, scalingStartedAt, lastSuccessfulScalingAt - When the pool entered the current state, when the in-progress scaling operation started, and when the last scaling operation completed successfully.
    • metricsLastObservedAt - Per worker pool and per Edge Host UID, the timestamp when utilization metrics for the host were last received from Prometheus. Old timestamps indicate that the host is not reporting metrics.

  5. Confirm that the Compute Pool controller is consuming the evaluation. The Compute Pool reports two related conditions.

    kubectl describe computepool <my-compute-pool> --namespace <my-project-namespace>
    Example Output (Conditions Excerpt)
    Conditions:
      Type:     NodeScalingEnabled
      Status:   True
      Reason:   ScalingPolicyConfigured
      Message:  ScalingPolicy is configured for this ComputePool

      Type:     ComputePoolEvaluationAvailable
      Status:   True
      Reason:   ComputePoolEvaluationFound
      Message:  ComputePoolEvaluation my-project-namespace/my-compute-pool found

Use ComputePoolEvaluation When Debugging​

SymptomWhat to check
No ComputePoolEvaluation existsCheck the ComputePoolEvaluationAvailable condition on the Compute Pool. Reason WaitingForComputePoolEvaluation means no metrics have been evaluated yet, and reason ScalingPolicyNotFound means the scalingPolicyRef points to a policy that does not exist. Also confirm PROMETHEUS_AVAILABLE is True on the Scaling Policy.
The pool does not scale despite high or low loadCompare target with count in spec.poolEvaluations. If they are equal, utilization has not strictly crossed the threshold for the full scale-up or scale-down duration, or the resource bounds cap the target. Also check the pool state: Cooldown pauses decisions, and details shows the remaining cooldown time.
An unexpected host was removed during scale-downReview nodeWastes. The host with the highest wasteScore — the most idle capacity, with unused GPUs weighted most heavily — is removed first.
A scaling operation appears stuckCheck state and details for progress counts. A scale-up is aborted after abortDuration, keeping the hosts that provisioned successfully. A scale-down is never aborted and continues until all removals complete.
Scaling reacts slowly or not at allCheck metricsLastObservedAt. Stale timestamps mean hosts are not reporting metrics. Refer to Configure Prometheus Agent Monitoring to verify the metrics pipeline.
info

ComputePoolEvaluation resources are read-only from a debugging perspective. Manual edits are overwritten on the next evaluation cycle, and a validating webhook rejects evaluations that do not match an existing Compute Pool.

Compute Pool Cannot Be Deleted​

Symptom: Compute Pool deletion fails or does not complete.

Possible causes:

  1. AIWorkloads still run on the cluster.

  2. A finalizer prevents deletion.

  3. Palette cluster deletion fails.

Resolution:

Start by checking the Compute Pool detail page in the PaletteAI UI. The Overview tab shows deployed workloads and cluster status. If workloads are still running, delete them from the Workloads section before retrying the Compute Pool deletion.

If the UI does not provide enough detail, use the following kubectl commands.

  1. List AIWorkload references.

    kubectl get computepool <computepool-name> --namespace <project-namespace> --output jsonpath='{.status.aiWorkloadRefs}'

  2. Delete AIWorkloads.

    kubectl delete aiworkload <aiworkload-name> --namespace <project-namespace>

  3. Check finalizers.

    kubectl get computepool <computepool-name> --namespace <project-namespace> --output jsonpath='{.metadata.finalizers}'

    Example output:

    ["spectrocloud.com/computepool-finalizer"]
    warning

    Finalizers protect resources from premature deletion while cleanup occurs. Do not manually remove finalizers unless:

    • You have confirmed all AIWorkloads are deleted.

    • Controller logs show the finalizer is stuck due to a known issue.

    • You have consulted with your PaletteAI administrator or support team.

    Manually removing finalizers can leave orphaned resources in Palette or cause state inconsistencies.

    If you must remove a finalizer (after confirming with support), use:

    kubectl patch computepool <computepool-name> --namespace <project-namespace> \
      --type json --patch '[{"op": "remove", "path": "/metadata/finalizers/0"}]'

  4. If deletion does not complete, review controller logs.

    First, find the Hue controller pods:

    kubectl get pods --namespace mural-system --selector controller.spectrocloud.com/name=hue-controllers

    Then review the logs:

    kubectl logs --namespace mural-system --selector controller.spectrocloud.com/name=hue-controllers --tail=100 --follow

    If the mural-system namespace or labels are not found, discover the controller deployment:

    kubectl get deployments --all-namespaces | grep --ignore-case hue

Changing Longhorn Replica Count​

The longhornDefaultReplicaCount Cluster Profile variable is immutable. It is set at Compute Pool creation and cannot be edited afterward.

The variable renders into the longhorn Kubernetes StorageClass parameters.numberOfReplicas field, which every CSI-provisioned PersistentVolumeClaim inherits. Kubernetes StorageClass parameters are immutable by design — the API server rejects any change. Allowing edits would also produce a state where new PersistentVolumeClaims use one replica count while existing Longhorn volumes keep another, so the variable is locked to preserve consistency across the cluster.

To use a different replica count, recreate the Compute Pool with the new value. All PersistentVolumeClaims on the cluster are lost, so back up any persistent data first.

info

The replica count must not exceed the number of nodes in the Compute Pool. Use 1 for single-node clusters, 2 for two-node clusters, and 3 for three or more nodes. Setting a higher value leaves PersistentVolumeClaims Pending because Longhorn cannot satisfy replica anti-affinity.

Node Stuck in Provisioning (Air-Gapped)​

Symptom: An edge node remains stuck in a Provisioning state and never transitions to Running in an air-gapped environment.

Possible causes:

  1. Pack components are attempting to reach an external endpoint that is unreachable in the air-gapped environment.

  2. Registry credentials are not correctly embedded in the node's user-data.

  3. Required images are missing from the internal Zot registry (for example, edge-native-byoi, edge-k8s, cni-cilium-oss, or piraeus-operator).

  4. Network policy is blocking the node from reaching the internal Zot registry IP or port.

Resolution:

  1. Check which images are being pulled on the node.

    crictl images

    Compare the listed images against what is available in your internal Zot registry. Any missing images must be pushed to Zot before cluster deployment.

  2. Verify that the containerd registry configuration points to the internal Zot instance.

    cat /etc/containerd/config.toml | grep --after-context=5 registry

    The registry endpoint should resolve to your internal Zot address and port (default 30003), not an external host.

  3. Test connectivity from the node to the internal Zot registry.

    curl --insecure https://<vertex-ip>:30003/v2/

    A 200 OK or 401 Unauthorized response confirms the registry endpoint is reachable. A connection refused or timeout indicates a network or firewall issue.

  4. Check Palette agent logs for pull failures or registry errors.

    journalctl --unit stylus-agent --follow

    Look for image pull errors, TLS certificate issues, or authentication failures against the registry.

  5. Verify that all required pack .zst files were successfully imported into Zot before cluster deployment. Cluster Profiles cannot be imported and packs cannot be resolved until they are present in the registry. Refer to Deploy Profile Bundles in Air-Gapped Environments for the correct import sequence.

GPU Pods Stuck with No NVIDIA Runtime Configured​

Symptom: On a GPU edge host, NVIDIA GPU Operator pods remain stuck in Init:CreateContainerError or Init:0/1. The Kubelet reports the following event, repeating roughly every 15 seconds:

Failed: spec.initContainers{driver-validation}:
  Error: failed to get sandbox runtime: no runtime for "nvidia" is configured

Image pulls succeed, so this is not a registry, authentication, or air-gapped problem. It is a Container Runtime Interface (CRI) runtime-registration problem: the nvidia RuntimeClass exists in Kubernetes, but no matching containerd runtime handler is registered behind it.

Possible causes:

  1. containerd configuration schema mismatch. The edge host base image ships /etc/containerd/config.toml using schema version = 2 (CRI plugin key io.containerd.grpc.v1.cri). The NVIDIA GPU Operator container-toolkit DaemonSet writes a drop-in at /etc/containerd/conf.d/99-nvidia.toml using schema version = 3 (CRI plugin key io.containerd.cri.v1.runtime). Because the two schema versions differ, containerd loads only the runtimes it finds under the version 2 CRI path—runc—and ignores the nvidia, nvidia-cdi, and nvidia-legacy runtimes from the drop-in. A version = 3 imports tree does not deep-merge the runtimes map; it replaces at the block level.

  2. A duplicate containerd service holds the runtime lock. On some hosts, the operating system containerd.service is installed alongside spectro-containerd.service. Both services read the same config.toml and metadata database. When both are active, restarting spectro-containerd lets the operating system service acquire the file lock and lock out the Spectro Cloud service, so runtime changes never take effect.

Resolution:

warning

Editing containerd configuration on an edge host is a low-level operation. Back up the existing configuration before you make changes, and apply the change to one host at a time.

info

These steps use two command-line tools on the edge host: crictl to inspect the container runtime and jq to parse JSON output. If either tool is not available on the host, install it from the linked release page before you continue.

  1. Confirm which runtimes containerd has registered.

    sudo crictl --runtime-endpoint=unix:///run/spectro/containerd/containerd.sock info \
      | jq '.config.containerd.runtimes | keys'

    If the output lists only ["runc"] and not nvidia, the NVIDIA runtime handler is not registered. A schema mismatch is the most likely cause, but a duplicate or conflicting containerd service can produce the same result. Complete the service check in the next step before you edit any configuration.

  2. Check whether a duplicate containerd service is running.

    systemctl is-active containerd spectro-containerd
    journalctl --unit containerd --since "-5m" | grep "starting containerd"

    If both services are active, or the operating system containerd service restarts on a short loop, a duplicate service is holding the runtime lock. Mask the operating system service as part of the fix below. If only spectro-containerd is active, the cause is the configuration schema mismatch.

  3. Confirm the containerd major version before you change the configuration schema.

    containerd --version

    The version = 3 configuration schema is native to containerd 2.x. If the host runs containerd 1.x, keep version = 2 and add the NVIDIA runtimes under the io.containerd.grpc.v1.cri plugin key instead. Applying a version = 3 file to a 1.x host prevents containerd from starting.

  4. Back up the current configuration.

    sudo cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak

  5. On a containerd 2.x host, edit /etc/containerd/config.toml so that it declares version = 3 and defines every runtime directly rather than relying on the drop-in. Because a version = 3 imports tree does not deep-merge the runtimes map, each runtime must be present in the main file.

    Preserve any settings the host already relies on, such as CNI, registry mirror, snapshotter, and proxy configuration. Add or update only the CRI runtime block shown below and leave unrelated sections in place.

    version = 3
    root = "/var/lib/spectro/containerd"
    state = "/run/spectro/containerd"

    [grpc]
      address = "/run/spectro/containerd/containerd.sock"

    [plugins]
      [plugins."io.containerd.cri.v1.images".pinned_images]
        sandbox = "registry.k8s.io/pause:3.6"

      [plugins."io.containerd.cri.v1.runtime".containerd]
        default_runtime_name = "runc"

        [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
            BinaryName = "/opt/bin/runc"
            SystemdCgroup = true

        [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
            SystemdCgroup = true

        [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-cdi]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-cdi.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi"
            SystemdCgroup = true

        [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-legacy]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-legacy.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy"
            SystemdCgroup = true

    The GPU Operator drop-in at /etc/containerd/conf.d/99-nvidia.toml can stay on disk. Once the main configuration declares version = 3, containerd reads the runtime definitions from the main file directly.

  6. Validate the rendered configuration before you restart the service.

    sudo containerd config dump | grep nvidia

    The command prints the parsed configuration. If it errors or prints nothing, the file is invalid or the NVIDIA runtimes are missing. Correct the file before you continue.

  7. If a duplicate operating system containerd service is present, stop and mask it so it cannot reacquire the runtime lock.

    sudo systemctl stop containerd
    sudo systemctl mask containerd

  8. Restart the Spectro Cloud containerd service.

    sudo systemctl reset-failed spectro-containerd
    sudo systemctl restart spectro-containerd

    If spectro-containerd fails to start, restore the backup to return the host to its previous state, then recheck the configuration:

    sudo cp -a /etc/containerd/config.toml.bak /etc/containerd/config.toml
    sudo systemctl restart spectro-containerd

  9. Verify that all four runtimes are registered.

    sudo crictl --runtime-endpoint=unix:///run/spectro/containerd/containerd.sock info \
      | jq '.config.containerd.runtimes | keys'

    Expected output:

    ["nvidia", "nvidia-cdi", "nvidia-legacy", "runc"]

    The stuck GPU Operator pods transition to Running after the nvidia runtime registers.

info

This mismatch occurs when the edge host base image ships a version = 2 containerd configuration while a newer NVIDIA GPU Operator container-toolkit writes version = 3 drop-ins. The long-term fix is for the edge host base image to ship a version = 3 containerd configuration and to avoid installing a second containerd package. If you manage your own edge base image, align its containerd configuration schema with the version that your GPU Operator toolkit emits.

Resources​