Skip to main content
Version: v1.1.x

Troubleshooting Projects

This page provides troubleshooting guidance for common issues with Projects, including overall readiness, compute problems, and settings problems.

info

Troubleshooting steps on this page use kubectl commands to inspect Kubernetes resources on the hub cluster. You need kubectl access to the hub cluster. If you do not have kubectl access, contact your platform administrator.

Project Not Ready​

Symptom: The Project reports READY: false, or resources in the Project namespace fail to reconcile.

Cause: One or more Project conditions are False. The Project becomes ready only after its Tenant, Settings, Compute Config, RBAC roles, variable set, default compute, GPU usage, and effective model settings checks all pass.

Resolution:

  1. Check the Project conditions and read the message of each condition that is False.

    kubectl get project <project-name> --namespace <project-namespace> --output json | jq '.status.conditions'

  2. Match the failing condition against the following table.

    ConditionMeaning and fix
    TenantValidThe owning Tenant cannot be resolved or is not ready. A message of Referenced tenant is not ready means you must triage the Tenant first. Refer to Troubleshooting Tenants.
    SettingsValidThe Settings resource cannot be resolved. A message of No SettingsRef configured on Project or Tenant means neither the Project nor its Tenant references a Settings resource. Refer to Project Settings Not Ready.
    ComputeConfigValidThe referenced Compute Config does not exist or cannot be resolved. Verify spec.computeConfigRef points to a Compute Config in the Project namespace or the owning Tenant namespace.
    RolesReadyProject RBAC roles or RoleBindings failed to create, or the RBAC policy ConfigMap is missing or invalid. Review the condition message for the failing resource.
    VariableSetReadyThe Project Variable Set failed to create. Review the condition message.
    DefaultComputeReadyThe Project's default Compute resource is not ready. Refer to Project Compute Not Ready.
    NotOversubscribedGPU usage in the Project exceeds a configured limit. Refer to Project GPU Quota Exceeded.
    EffectiveModelSettingsReadyThe merged Tenant and Project model settings failed to compute. Review the condition message for the failing lookup.

Project Compute Not Ready​

Symptom: The DefaultComputeReady condition is False, and Compute Pools or deployments in the Project cannot provision infrastructure.

Possible causes:

  1. The default Compute resource failed to create, reported as Failed to upsert Compute resource.

  2. The Compute resource exists but is not ready yet, reported as Compute resource is not ready yet.

  3. The Compute resource cannot authenticate with Palette.

Resolution:

  1. Check the default Compute resource in the Project namespace and read its conditions.

    kubectl get compute default --namespace <project-namespace> --output json | jq '.status.conditions'

  2. If the SettingsValid condition on the Compute resource reports Palette endpoint is unreachable or Palette authentication failed, the Palette integration in the resolved Settings resource is misconfigured. Verify the Palette endpoint, API key, and default project ID. Refer to Settings Integrations Not Ready.

  3. If the Compute resource does not exist, verify the Project passed its Settings and Compute Config checks. The default Compute is only provisioned once a Palette integration is available.

Project GPU Quota Exceeded​

Symptom: The NotOversubscribed condition is False with the reason GPUUsageExceedsLimits, or a Compute Pool or deployment request is rejected with a GPU quota error.

Cause: GPU usage in the Project exceeds the Project limits or the Tenant projectLimits cap for one or more GPU variants. The condition message lists each exceeded variant, for example GPU variant '<variant>' usage <n> exceeds Project limit <m> or GPU variant '<variant>' usage <n> exceeds Tenant ProjectLimit <m>.

Resolution:

  1. Check the current GPU usage recorded on the Project.

    kubectl get project <project-name> --namespace <project-namespace> --output json | jq '.status.gpuUsage'

  2. Reduce usage by deleting unused Compute Pools or deployments, raise the Project spec.gpuResources.limits, or ask a Tenant admin to raise the Tenant spec.gpuResources.projectLimits for the Project.

  3. If a Project create or update request is rejected with Project GPU request for variant '<variant>' (<n>) exceeds Tenant limit (<m>), the requested quota is larger than what the Tenant allows. Lower the Project request or ask a Tenant admin to raise the Tenant limit.

For details on how the Project, Tenant, and System caps interact, refer to GPU Quota Exceeded.

Project Settings Not Ready​

Symptom: The SettingsValid condition is False, or model deployments in the Project cannot resolve integrations.

Possible causes:

  1. Neither the Project nor its Tenant references a Settings resource, reported as No SettingsRef configured on Project or Tenant.

  2. The referenced Settings resource exists but is not ready, reported as Project settings are not ready.

Resolution:

  1. Verify the Settings reference on the Project, and fall back to the Tenant reference if the Project does not define one.

    kubectl get project <project-name> --namespace <project-namespace> --output json | jq '.spec.settingsRef'
    kubectl get tenant <tenant-name> --output json | jq '.spec.settingsRef'

  2. If a Settings resource is referenced but not ready, triage its integration secrets. Refer to Settings Integrations Not Ready.

  3. To verify which scope each model integration resolved from, check the effective model settings recorded on the Project. Each entry reports its scope (project or tenant) and whether the Tenant has locked it.

    kubectl get project <project-name> --namespace <project-namespace> --output json | jq '.status.effectiveModelSettings'

Project Creation Rejected​

Symptom: Creating or updating a Project fails immediately with an admission webhook error.

Resolution:

Match the error message against the following common rejections and correct the request.

Message patternMeaning and fix
project name must match its namespace: name=<name>, namespace=<namespace>Projects must be created in a namespace with the same name as the Project. Refer to Create and Manage Projects.
project name '<name>' is already in use by project <namespace>/<name>Project names are unique across the cluster. Choose a different name.
project creation forbidden in the built-in (<ns>) and system (<ns>) namespacesProjects cannot be created in reserved namespaces. Create the Project in its own namespace.
Project GPU request for variant '<variant>' (<n>) exceeds Tenant limit (<m>)The requested GPU quota exceeds the Tenant cap. Refer to Project GPU Quota Exceeded.
cross-project references are not allowedspec.computeConfigRef must reference a Compute Config in the Project's own namespace or the owning Tenant namespace.
persona tier "<tier>" is not defined in the tenant's effective RBAC template catalogThe additionalRoles entry references a persona tier the Tenant does not define. Use a defined tier or update the Tenant RBAC template catalog.

Cannot Delete Project​

Symptom: The Project remains in a deleting state and its namespace is not removed.

Cause: The Project finalizer removes child resources in order before releasing the Project. Deletion waits for all App Deployments, Model Deployments, Compute Pools, and Profile Bundles in the Project namespace to finish their own cleanup.

Resolution:

  1. Check which child resources remain in the Project namespace.

    kubectl get aiworkloads,computepools,profilebundles --namespace <project-namespace>

  2. If a Compute Pool is stuck deleting, refer to Compute Pool Cannot Be Deleted. If a Profile Bundle is stuck deleting, refer to Cannot Delete Profile Bundle.

  3. If deletion still does not complete, review the Hue controller logs for errors related to the Project. The controllers run in the Helm release namespace, which defaults to mural-system.

    kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

Resources​