Skip to main content
Version: v1.1.x

Troubleshooting the System Resource

This page provides troubleshooting guidance for common issues with the cluster-wide System resource, which defines the top-level PaletteAI configuration that Tenants and Projects inherit, including GPU quotas, model settings, and the system compute.

info

Troubleshooting steps on this page use kubectl commands to inspect Kubernetes resources on the hub cluster. You need cluster-scoped kubectl access to the hub cluster. If you do not have kubectl access, contact your platform administrator.

System Not Ready​

Symptom: The System resource reports READY: false, or Tenants fail checks that depend on system-level configuration.

Cause: One or more System conditions are False. The System becomes ready only after its namespace, Settings, Compute Config, and default compute checks all pass.

Resolution:

  1. Check the System conditions and read the message of each condition that is False. The System resource is a singleton named pai-system.

    kubectl get system pai-system --output json | jq '.status.conditions'

  2. Match the failing condition against the following table.

    ConditionMeaning and fix
    NamespaceEnsuredThe system namespace failed to create. Review the condition message for the underlying error.
    SettingsConfiguredThe referenced Settings resource is missing or not ready. Refer to System Settings Not Ready.
    ComputeConfigValidThe referenced Compute Config does not exist or cannot be resolved. Verify spec.computeConfigRef.
    DefaultComputeReadyThe system default Compute resource is not ready. Refer to System Compute Not Ready.

System Compute Not Ready​

Symptom: The DefaultComputeReady condition is False, and Compute Pools created in the system namespace cannot provision infrastructure.

Possible causes:

  1. The System has no Settings reference, reported as Default Compute not provisioned: System has no SettingsRef.

  2. The referenced Settings has no Palette integration, reported as Default Compute not provisioned: referenced Settings has no Palette integration.

  3. The default Compute resource failed to create or is not ready yet.

Resolution:

  1. Read the DefaultComputeReady condition message. If the default compute was skipped, add a Settings reference with a Palette integration to the System spec.

    kubectl get system pai-system --output json | jq '.status.conditions[] | select(.type == "DefaultComputeReady")'

  2. If the Compute resource exists but is not ready, check its conditions in the system namespace. A SettingsValid condition reporting Palette endpoint is unreachable or Palette authentication failed indicates a Palette integration problem. Refer to Settings Integrations Not Ready.

    kubectl get compute default --namespace <system-namespace> --output json | jq '.status.conditions'

System GPU Oversubscription​

Symptom: The SystemNotOversubscribed condition is False, and GPU usage across Tenants exceeds a system-level cap.

Cause: GPU usage exceeds the System spec.gpuResources configuration for one or more GPU variants. The condition message identifies the exceeded scope:

  • GPU variant '<variant>' usage <n> exceeds system limit <m>. - Total usage across the cluster exceeds the system-wide limit.

  • GPU variant '<variant>' aggregate tenant usage <n> exceeds tenant-pool ceiling <m> - Combined Tenant usage exceeds the budget left after the system reservation.

  • GPU variant '<variant>' system-namespace usage <n> exceeds reservation <m>. - Usage in the system namespace exceeds the systemReservations carve-out.

Unlike Tenants and Projects, oversubscription does not block System readiness. The condition is informational and identifies where capacity must be freed or limits raised.

Resolution:

  1. Check the recorded GPU usage and the oversubscribed variants.

    kubectl get system pai-system --output json | jq '.status.gpuUsage'
    kubectl get system pai-system --output json | jq '.status.oversubscribedVariants'

  2. Reduce usage in the identified scope, raise the System spec.gpuResources.limits, adjust spec.gpuResources.tenantLimits for the over-budget Tenant, or adjust spec.gpuResources.systemReservations. For details on how the caps interact across scopes, refer to GPU Quota Exceeded.

System Settings Not Ready​

Symptom: The SettingsConfigured condition is False, and Tenants or Projects that inherit system-level model settings do not receive them.

Possible causes:

  1. The referenced Settings resource does not exist, reported as Referenced Settings <namespace>/<name> not found.

  2. The referenced Settings resource is not ready, reported as Referenced Settings is not ready.

Resolution:

  1. Verify the Settings reference on the System spec and confirm the resource exists.

    kubectl get system pai-system --output json | jq '.spec.settingsRef'
    kubectl get settings --all-namespaces

  2. If the Settings resource exists but is not ready, triage its integration secrets. Refer to Settings Integrations Not Ready.

Cannot Create a Second System Resource​

Symptom: Creating a System resource is rejected with the error System is a singleton and must be named 'pai-system'.

Cause: The System resource is a cluster-wide singleton. Only one instance named pai-system is allowed.

Resolution:

Edit the existing pai-system resource instead of creating a new one.

kubectl edit system pai-system

Resources​