Troubleshooting Tenants

This page provides troubleshooting guidance for common issues when creating and managing Tenants.

Tenant Not Ready

Symptom: Tenant READY is False.

Possible causes:

1. The referenced Settings resource does not exist or is not ready.

2. PaletteAI cannot create the controller-created Tenant namespace (tenant-<tenant-name>).

3. GPU usage exceeds limits.

Resolution:

1. View Tenant conditions.

   kubectl get tenant <tenant-name> --output json | jq '.status.conditions'

2. If SettingsConfigured is False, verify the Settings resource.

   Check the Settings resource in the auto-generated namespace referenced by .spec.settingsRef.namespace (must be tenant-<tenant-name>).

   # First, get the Settings namespace from the Tenant spec
   kubectl get tenant <tenant-name> --output jsonpath='{.spec.settingsRef.namespace}'
   
   # Then check the Settings resource
   kubectl get settings <settings-name> --namespace <settings-namespace>
   kubectl describe settings <settings-name> --namespace <settings-namespace>

   If the Settings resource shows an IntegrationsConfigured condition with status=False and reason="IntegrationsNotReady", one or more configured integrations have invalid or missing secrets. The condition message lists which integrations are affected. See Settings Integrations Not Ready for resolution steps.

3. If TenantNamespaceCreated is False, verify the controller-created namespace and review events.

   # Check the controller-created namespace (tenant-<name>)
   kubectl get namespace tenant-<tenant-name>
   kubectl describe tenant <tenant-name>

   Debug namespace creation failures:

   First, check recent events to display the actual denial reason:

   # If the namespace exists but Tenant is not Ready
   kubectl get events --namespace tenant-<tenant-name> --sort-by=.lastTimestamp | tail -20
   
   # If namespace does not exist, check controller logs
   kubectl logs --namespace mural-system --selector app=hue-controller --tail=100 | grep --ignore-case "tenant.*<tenant-name>"

   Common causes for namespace creation failure:

   • Namespace already exists with conflicting ownership.

   • Pod Security Admission (PSA) policies blocking namespace creation. Check for pod-security.kubernetes.io labels.

   • Admission webhooks rejecting the namespace. Look for webhook names in events.

   • Finalizers may be blocking deletion. Check for finalizers with the following command:

     kubectl get namespace tenant-<tenant-name> --output yaml | grep finalizers

4. If TenantNotOversubscribed is False, reduce GPU usage.

   kubectl get tenant <tenant-name> --output json | jq '.status.gpuUsage'

5. If ComputeConfigValid is False, verify the ComputeConfig resource.

   The ComputeConfigValid condition indicates that the ComputeConfig referenced by the Tenant's computeConfigRef field exists and is valid in the Tenant workload namespace.

   Common failure reasons:

   • ComputeConfigNotFound: The referenced ComputeConfig does not exist in the Tenant's workload namespace.

   • ValidationFailed: The computeConfigRef.name field is empty or invalid.

   Resolution steps:

   1. Check the Tenant's computeConfigRef.name field to ensure it references a valid ComputeConfig name.

      kubectl get tenant <tenant-name> --output jsonpath='{.spec.computeConfigRef.name}'

   2. Verify that the ComputeConfig resource exists in the Tenant's workload namespace (which follows the pattern tenant-<tenant-name>).

      kubectl get computeconfig <computeconfig-name> --namespace tenant-<tenant-name>

   3. If the ComputeConfig does not exist, create it in the Tenant's workload namespace.

   4. If the reference name is empty, update the Tenant spec with a valid ComputeConfig reference.

6. If DefaultComputeReady is False, verify the Tenant's default Compute resource. Refer to Tenant Compute Not Ready.

Tenant Compute Not Ready

Symptom: The DefaultComputeReady condition is False, and Compute Pools created in the Tenant workload namespace cannot provision infrastructure.

Possible causes:

1. The Tenant has no Settings reference, reported as Default Compute not provisioned: Tenant has no SettingsRef.

2. The referenced Settings has no Palette integration, reported as Default Compute not provisioned: referenced Settings has no Palette integration.

3. The default Compute resource failed to create or is not ready yet.

Resolution:

1. Read the DefaultComputeReady condition message. If the default Compute was skipped, add a Settings reference with a Palette integration to the Tenant spec.

   kubectl get tenant <tenant-name> --output json | jq '.status.conditions[] | select(.type == "DefaultComputeReady")'

2. If the Compute resource exists but is not ready, check its conditions in the Tenant workload namespace. A SettingsValid condition reporting Palette endpoint is unreachable or Palette authentication failed indicates a Palette integration problem. Refer to Settings Integrations Not Ready.

   kubectl get compute default --namespace tenant-<tenant-name> --output json | jq '.status.conditions'

Cannot Delete Tenant

Symptom: The Tenant delete request fails, or the Tenant remains in a deleting state and is not removed.

Possible causes:

1. The admission webhook blocks deletion when child Projects still exist.

2. The Tenant controller cannot finish garbage collection while active Tenant-scoped ComputePools or AIWorkloads remain in the tenant-<tenant-name> namespace.

Resolution:

Child Projects block deletion

If kubectl delete tenant fails with an error indicating child Projects remain, list and delete them:

kubectl get projects --all-namespaces --selector palette.ai/tenant-name=<tenant-name>

info

Projects are automatically labeled with palette.ai/tenant-name during creation. If a Project is missing this label, it may indicate a problem with the Project creation process or a Project created with an older version of PaletteAI.

kubectl delete project <project-name> --namespace <project-namespace>

Verify the child Project count is 0:

kubectl get tenant <tenant-name> --output jsonpath='{.status.childProjectCount}'

Then retry the Tenant deletion.

Active Tenant-scoped workloads block garbage collection

If the Tenant accepts the delete request but remains in a deleting state, check the GarbageCollected condition:

kubectl get tenant <tenant-name> --output json | jq '.status.conditions[] | select(.type == "GarbageCollected")'

When garbage collection is blocked, this condition reports status: "False" and reason: "GarbageCollectionBlocked". The message lists the remaining active ComputePools and AIWorkloads, for example:

cannot garbage-collect Tenant: 1 ComputePool(s) and 2 AIWorkload(s) still exist in namespace "tenant-<tenant-name>"; delete them first

List and delete the remaining active ComputePools and AIWorkloads in the tenant-<tenant-name> namespace:

kubectl get computepools,aiworkloads --namespace tenant-<tenant-name>

kubectl delete computepool <computepool-name> --namespace tenant-<tenant-name>
kubectl delete aiworkload <aiworkload-name> --namespace tenant-<tenant-name>

Only active resources block garbage collection. Resources already terminating (those with a deletionTimestamp set) may still appear in this list but do not require further cleanup. Once only terminating workloads remain, the controller removes the finalizer automatically.

For the full Tenant deletion workflow, refer to Delete Tenants.

GPU Quota Exceeded

Symptom: App Deployments or ComputePools remain pending due to GPU requests.

Cause: A new GPU request is blocked when it would exceed any cap that applies to it. The relevant caps are:

• Project limits (<project>.spec.gpuResources.limits.<family>) - Caps total GPU usage by all Compute Pools and App Deployments in the Project. Set by the Project owner.
• Tenant projectLimits (<tenant>.spec.gpuResources.projectLimits.<project>.<family>) - Tenant-admin-imposed ceiling on what a specific Project may consume, applied in addition to the Project's own limits. When both this and the Project's own limits are set, the lower of the two is the Project's effective per-Project cap.
• Tenant limits (<tenant>.spec.gpuResources.limits.<family>) - Caps combined GPU usage across every Project plus the tenant namespace. Enforced at aggregate scope, separate from the per-Project caps above.
• Tenant tenantReservations (<tenant>.spec.gpuResources.tenantReservations.<family>) - Caps GPU usage by Compute Pools and App Deployments created directly in the tenant namespace (outside of any Project). The remainder (limits - tenantReservations) is the aggregate budget available to all Projects combined.

Resolution:

1. Check current GPU usage and the Tenant's oversubscribed scopes.

   kubectl get tenant <tenant-name> --output json | jq '.status.gpuUsage'
   kubectl get tenant <tenant-name> --output json | jq '.status.tenantGpuUsage'
   kubectl get tenant <tenant-name> --output json | jq '.status.oversubscribedVariants'

   The oversubscribedVariants field identifies which scope is over its cap:

   • Tenant - Combined Project and Tenant-namespace usage exceeds limits for the variant.
   • TenantNamespace - Tenant-namespace usage exceeds tenantReservations for the variant.

2. Check Tenant GPU caps.

   kubectl get tenant <tenant-name> --output json | jq '.spec.gpuResources'

3. Reduce GPU usage or increase the relevant cap.

   Update limits, tenantReservations, or projectLimits as needed and apply the manifest.

   spec:
     gpuResources:
       limits:
         'NVIDIA-A100': 128
       tenantReservations:
         'NVIDIA-A100': 16
       projectLimits:
         project-a:
           'NVIDIA-A100': 64

   kubectl apply --filename tenant.yaml

Invalid Settings Namespace

Symptom: Tenant creation or update is rejected at admission time with an error message similar to: settingsRef.namespace must be 'tenant-<tenant-name>'.

Cause: PaletteAI's webhook validation enforces that settingsRef.namespace must equal the auto-generated Tenant workload namespace format tenant-<tenant-name>. Tenant creation or update requests are rejected at admission time if the namespace does not match. This validation cannot be overridden.

Resolution:

Update the settingsRef.namespace field in the Tenant manifest to match the auto-generated Tenant workload namespace format: tenant-<tenant-name>, where <tenant-name> is the name of your Tenant resource.

Example:

For a Tenant named my-tenant, the correct configuration is:

spec:
  settingsRef:
    name: my-settings
    namespace: tenant-my-tenant

Apply the corrected manifest:

kubectl apply --filename tenant.yaml

Project Name and Namespace Mismatch

Symptom: Project creation or update is rejected at admission time with an error message: project name must match its namespace: name="<project-name>", namespace="<namespace-name>".

Cause: PaletteAI's webhook validation enforces that a Project's metadata.name must match its metadata.namespace. Project creation or update requests are rejected at admission time if the name and namespace do not match. This validation cannot be overridden.

Resolution:

Update the Project manifest so that metadata.name matches metadata.namespace.

Example:

For a Project in namespace my-project, the correct configuration is:

apiVersion: spectrocloud.com/v1alpha1
kind: Project
metadata:
  name: my-project
  namespace: my-project
spec:
  # ... project spec

Apply the corrected manifest:

kubectl apply --filename project.yaml

Settings Integrations Not Ready

Symptom: Settings resource shows IntegrationsConfigured condition with status=False and reason="IntegrationsNotReady". The condition message lists which integrations have invalid or missing secrets.

Cause: One or more configured integrations (Palette, Hugging Face, or NVIDIA) reference secrets that are missing, have incorrect names, or lack required fields.

Resolution:

1. Check the Settings conditions to identify affected integrations.

   kubectl get settings <settings-name> --namespace <settings-namespace> --output json | jq '.status.conditions'

   The condition message identifies which integrations are failing. Example output:

   {
     "type": "IntegrationsConfigured",
     "status": "False",
     "reason": "IntegrationsNotReady",
     "message": "The following integrations have invalid or missing secrets: Hugging Face, NVIDIA"
   }

2. Verify the referenced secrets exist in the correct namespace.

   kubectl get secrets --namespace <settings-namespace>

3. For Palette integrations, verify the secret exists and matches the name in the Settings spec.

   kubectl get secret <palette-secret-name> --namespace <settings-namespace>

4. For Hugging Face integrations, verify the secret exists and contains the required token field.

   kubectl get secret <huggingface-secret-name> --namespace <settings-namespace> --output json | jq '.data'

   The secret must be an Opaque Kubernetes secret with the field specified in the Settings spec.integrations.huggingFace.apiKey.key.

5. For NVIDIA integrations, verify both the NGC API key and image pull secrets exist.

   # Check NGC API key secret
   kubectl get secret <ngc-api-key-secret-name> --namespace <settings-namespace> --output json | jq '.data'
   
   # Check NGC image pull secret
   kubectl get secret <ngc-image-pull-secret-name> --namespace <settings-namespace>

   The API key secret must contain the field specified in spec.integrations.nvidia.ngc.apiKey.key.

6. Ensure secret names match exactly what is specified in the Settings resource.

   kubectl get settings <settings-name> --namespace <settings-namespace> --output yaml | grep -A 5 integrations

7. After correcting the secrets, verify the Settings resource becomes ready.

   kubectl get settings <settings-name> --namespace <settings-namespace>

   The READY column should show True and the IntegrationsConfigured condition should have status=True with reason="IntegrationsValid".

Tenant Admin Groups Do Not Work

Symptom: Users in Tenant admin groups do not have expected permissions in Project namespaces.

Possible causes:

1. OIDC groups do not match your identity provider configuration.

2. PaletteAI did not create RBAC resources.

3. Dex configuration does not match (static users).

Resolution:

1. Verify Dex static user configuration (if applicable).

   kubectl get configmap dex --namespace mural-system --output yaml | grep --after-context=10 staticClients

2. Verify RBAC resources exist in the Project namespace.

   kubectl get roles --namespace <project-namespace> | grep tnt-adm
   kubectl describe role prj-<project-name>-tnt-adm --namespace <project-namespace>

3. Verify the RoleBinding includes the expected groups.

   kubectl get rolebinding --namespace <project-namespace> | grep tnt-adm
   kubectl describe rolebinding <rolebinding-name> --namespace <project-namespace>

4. Verify group membership using impersonation (if OIDC is configured).

   # Test as a user in the tenant admin group
   kubectl auth can-i create projects --as=<username> --as-group=<tenant-admin-group>
   # Expected output: yes

   If the above test succeeds but real user login fails, the issue is likely an identity provider misconfiguration:

   • The user's token does not include the expected group claim. Verify IdP group claim mapping.

   • The Kubernetes API server is not configured to pass through group claims. Verify OIDC configuration in API server flags.

   • Compare --as-group test results with the actual groups returned in a user's token payload.

   To diagnose, decode a user's JWT token and verify the groups claim matches your tenantRoleMapping.groups configuration.

5. If RBAC resources are missing, delete and recreate the Project.

   Warning

   Deleting and recreating a Project will remove all workloads, deployments, and resources in that Project's namespace. Before proceeding:

   • Back up any important data.

   • Document deployed workloads.

   • Ensure you can recreate the Project configuration.

   • Verify no production workloads are running.

Resources

• Create and Manage Tenants

• Tenant Settings

• Tenants and Projects

• Troubleshooting Projects

• Troubleshooting Compute Pools