Create and Manage Tenants

A Tenant represents an organization or major division within your company and serves as the top-level organizational unit in PaletteAI.

A default Tenant is automatically created when you install PaletteAI with global.featureFlags.systemDefaultResources: true in your Helm values. For production workloads with multiple organizations or divisions that require separate GPU quota pools or isolated administrative boundaries, we recommend creating separate Tenants. For guidance on when to create multiple Tenants, refer to When to Create Multiple Tenants.

info

PaletteAI automatically creates a tenant-<name> namespace for internal operations when you create a Tenant. The Settings controller merges Tenant and Project Settings automatically when Tenant.spec.settingsRef (a fully-qualified name-and-namespace reference) is configured. For details on the namespace model, refer to Automatic Resource Creation.

Limitations

• Tenant creation is not available in the PaletteAI user interface at this time. Tenants must be created using Kubernetes manifests and kubectl commands as described in this guide.

• A newly created Tenant does not appear to end users in the PaletteAI UI until at least one Project exists within it and the user's OIDC group is listed in that Project's roleMapping. Tenant visibility is derived from Project access, not from the Tenant itself. Users with no OIDC group membership in any Project under the Tenant cannot access the Tenant in the UI.

Prerequisites

• Install kubectl.

• Set the KUBECONFIG environment variable to the path of the PaletteAI hub cluster kubeconfig file.

  export KUBECONFIG=<kubeconfig-location>

• Ensure you have cluster-admin permissions to manage cluster-scoped Tenant resources.

Create Tenant

Create a Tenant to define organizational boundaries, set GPU quotas, and control access across multiple Projects.

Enablement

1. Create a directory for the Tenant manifests, and then navigate to it.

   mkdir <tenant-name>
   cd <tenant-name>

2. (Optional) Create Tenant-level Settings and Secrets.

   Tenant-level Settings live in the auto-created tenant-<tenant-name> namespace and are referenced by Tenant.spec.settingsRef. The Settings controller automatically merges Tenant and Project Settings. If you skip this step, each Project under the Tenant must define its own settingsRef. Refer to Settings for more information.

   Because the controller creates the tenant-<tenant-name> namespace when it applies the Tenant, create the Tenant first, wait for the namespace to appear, and then apply the Settings and Secret. Alternatively, create the namespace manually before you apply the resources:

   kubectl create namespace tenant-<tenant-name>

   1. Create a Palette Secret in the Tenant namespace. The Secret must be in the same namespace as the Settings resource that references it.

      cat << EOF > palette-secret.yaml
      apiVersion: v1
      kind: Secret
      metadata:
        name: <palette-secret-name>
        namespace: tenant-<tenant-name>
      type: Opaque
      stringData:
        palette: |
          {
            "apiKey": "<your-palette-api-key>",
            "defaultProjectID": "<your-default-project-id>",
            "hostUrl": "<your-palette-host-url>",
            "tenant": "<your-palette-tenant-name>",
            "skipSSLCertificateVerification": false
          }
      EOF

   2. Create a Settings resource in the Tenant namespace that references the Palette Secret.

      cat << EOF > settings.yaml
      apiVersion: spectrocloud.com/v1alpha1
      kind: Settings
      metadata:
        name: <settings-name>
        namespace: tenant-<tenant-name>
      spec:
        integrations:
          palette:
            name: <palette-secret-name>
            namespace: tenant-<tenant-name>
      EOF

   info

   After you create the Tenant, you can manage Tenant-level Settings from the Settings Ref tab on the Tenant settings page. Use the tab to create Tenant-level Settings resources, set default Settings references, and review the Settings used across Projects.

3. Create the Tenant manifest.

   For a complete list of parameters, refer to the Tenant resource spec.

   If you created a Settings resource in step 2, set spec.settingsRef to point to it so the SETTINGS column during validation shows the referenced Settings name. Replace <settings-name> with the value you used in step 2.

   cat << EOF > tenant.yaml
   apiVersion: spectrocloud.com/v1alpha1
   kind: Tenant
   metadata:
     name: <tenant-name>
   spec:
     displayName: '<tenant-display-name>'
     settingsRef:
       namespace: tenant-<tenant-name>
       name: <settings-name>
     tenantRoleMapping:
       groups:
         - '<tenant-admin-group-1>'
         - '<tenant-admin-group-2>'
   EOF

4. Apply the applicable manifests. If you skipped step 2, omit the Settings and Secret commands.

   # Apply Tenant (always required)
   kubectl apply --filename tenant.yaml
   
   # Apply Palette Secret (if created in step 2)
   kubectl apply --filename palette-secret.yaml
   
   # Apply Settings resource (if created in step 2)
   kubectl apply --filename settings.yaml

   Example Output

   tenant.spectrocloud.com/primary-dev created
   secret/dev-palette-secret created
   settings.spectrocloud.com/dev-settings created

Validate

1. Verify the tenant-<name> namespace was created by the controller.

   kubectl get namespace tenant-<tenant-name>

   kubectl get namespace tenant-primary-dev

   Example Output

   NAME                 STATUS   AGE
   tenant-primary-dev   Active   5m

   info

   If you do not create a Tenant-level Settings resource, the Tenant still works, but each Project under the Tenant must specify its own settingsRef.

2. Verify the Tenant exists and is ready. If the Tenant references a Settings resource, the SETTINGS column shows the Settings name.

   kubectl get tenants

   Example Output

   NAME          DISPLAYNAME       READY   SETTINGS       NAMESPACE      PROJECTS   AGE
   default       Default Tenant    true    default        default        1          21d
   primary-dev   Primary Dev       true    dev-settings   primary-dev    0          5m

3. Verify the Tenant configuration.

   kubectl describe tenant <tenant-name>

   The following example shows Display Name, Settings Ref, and Tenant Role Mapping under Spec. It also shows that the Tenant is ready and has 0 child Projects under Status. In Status.Conditions, TenantNamespaceCreated refers to the controller-created tenant-<tenant-name> namespace.

   kubectl describe tenant primary-dev

   Example Output

   Name:         primary-dev
   Namespace:
   Labels:       <none>
   Annotations:  <none>
   API Version:  spectrocloud.com/v1alpha1
   Kind:         Tenant
   Metadata:
     Creation Timestamp:  2026-02-07T15:26:04Z
     Finalizers: spectrocloud.com/tenant-finalizer
     Generation:      1
     Resource Version: 12345678
     UID:              abc-123-def-456
   Spec:
     Display Name: Primary Dev
     Settings Ref:
       Name:       dev-settings
       Namespace:  tenant-primary-dev
     Tenant Role Mapping:
       Groups:
         admin
         sre
   Status:
     Child Project Count: 0
     Conditions:
       Last Transition Time: 2026-02-07T15:26:05Z
       Message:              Settings configured and ready
       Reason:               SettingsConfigured
       Status:               True
       Type:                 SettingsConfigured
       Last Transition Time: 2026-02-07T15:26:05Z
       Message:              Tenant namespace created successfully
       Reason:               TenantNamespaceCreated
       Status:               True
       Type:                 TenantNamespaceCreated
     Ready: true
   Events:  <none>

Modify Tenant

To track changes in version control, update your manifests and apply them with kubectl apply.

Enablement

1. Open the manifest you want to update and make the necessary changes.

   Use the following table to identify what manifest to update for common changes. For the full parameter list, refer to the applicable Resource page.

   Resource	Modifications	Additional Information
   Tenant	Update displayName. Update tenantRoleMapping.groups. Update gpuResources.	Tenants and Projects - GPU Quotas
   Settings	Create a Settings resource in the Tenant namespace (tenant-<name>), and then update the Tenant settingsRef. To remove shared Settings, delete settingsRef from the Tenant spec.	Settings

   The following example adds a Tenant admin group named operations and sets GPU resource limits.

   vi tenant.yaml

   Updated Tenant

   apiVersion: spectrocloud.com/v1alpha1
   kind: Tenant
   metadata:
     name: primary-dev
   spec:
     displayName: 'Primary Dev'
     settingsRef:
       name: dev-settings
       namespace: tenant-primary-dev
     tenantRoleMapping:
       groups:
         - 'admin'
         - 'sre'
         - 'operations'
     gpuResources:
       limits:
         'NVIDIA-A100': 64
         'NVIDIA-H100': 48
         'Default': 24
       requests:
         'NVIDIA-A100': 8
         'NVIDIA-H100': 8
         'Default': 4

2. Save the file and apply the updated manifest.

   kubectl apply --filename <manifest-location>

   kubectl apply --filename tenant.yaml

Validate

Verify that the Tenant reflects your updates.

kubectl describe tenant <tenant-name>

The following example shows the updated Tenant admin group and GPU resource limits.

kubectl describe tenant primary-dev

Example Output

Name: primary-dev
# ... metadata omitted for readability
Spec:
  Display Name: Primary Dev
  Gpu Resources:
    Limits:
      Default: 24
      NVIDIA-A100: 64
      NVIDIA-H100: 48
    Requests:
      Default: 4
      NVIDIA-A100: 8
      NVIDIA-H100: 8
  Settings Ref:
    Name: dev-settings
    Namespace: tenant-primary-dev
  Tenant Role Mapping:
    Groups: admin
      sre
      operations
Status:
  Ready: true
Events: <none>

Delete Tenant

Delete a Tenant when you no longer need it.

warning

PaletteAI prevents you from deleting a Tenant that has child Projects. You must delete all Projects under the Tenant before you can delete the Tenant.

Enablement

1. Verify the Tenant has no child Projects.

   kubectl get tenant <tenant-name> --output jsonpath='{.status.childProjectCount}'

   The output must be 0. If the value is greater than zero, delete the Projects under the Tenant before you continue.

   # List all Projects for this Tenant
   kubectl get projects --all-namespaces --selector palette.ai/tenant-name=<tenant-name>
   
   # Delete each Project
   kubectl delete project <project-name> --namespace <project-namespace>

2. Delete the Tenant.

   kubectl delete tenant <tenant-name>

   kubectl delete tenant primary-dev

3. (Optional) Delete the Tenant namespace.

   PaletteAI does not automatically delete the tenant-<name> namespace when you delete the Tenant. Before deleting it, confirm that no remaining Projects reference Settings resources in that namespace and that you no longer need any Secrets or Settings stored there.

   warning

   Deleting the Tenant namespace permanently deletes all Settings resources and integration Secrets it contains. Settings deletion automatically removes their referenced Secrets. Ensure you no longer need those credentials before proceeding.

   kubectl delete namespace tenant-<tenant-name>

   kubectl delete namespace tenant-primary-dev

Validate

Verify the Tenant no longer exists.

kubectl get tenants

Example Output

NAME      DISPLAYNAME      READY   SETTINGS   NAMESPACE   PROJECTS   AGE
default   Default Tenant   true    default    default     1          21d

Namespace Reference Validation

To enforce Project isolation, PaletteAI admission webhooks validate that resources cannot reference namespaces belonging to other Projects. This prevents scenarios where deleting Project B could break resources in Project A that hold references to namespaces in Project B.

Validation Levels

PaletteAI validates namespace references at four levels, depending on the resource type and field:

1. Same namespace only: Some resource references must be in the same namespace as the owning resource. For example, AIWorkload.spec.computePoolRef.namespace and ComputePool.spec.clusterVariant.imported.environmentRef.namespace must match the owning Project's namespace.

2. Same or system namespace: Some references can be in the resource's namespace or the system namespace (mural-system) for shared resources. For example, AIWorkload.spec.workloadDeploymentConfigs[].workloadProfileRef.namespace and ComputePool.spec.workloadDeploymentConfigs[].workloadProfileRef.namespace can reference either the Project namespace or mural-system.

3. Same, system, or Tenant namespace: Some references can also use the Tenant namespace (tenant-<name>) for Tenant-wide shared resources. For example, AIWorkload.spec.profileBundles[].namespace and ComputePool.spec.profileBundleRef.namespace can reference the Project namespace, mural-system, or the Tenant namespace.

4. Same or Tenant namespace only: Some references can be in the resource's own namespace or the owning Tenant's namespace (tenant-<name>), but not the system namespace. For example, Project.spec.computeConfigRef.namespace must be either the Project namespace or the Tenant namespace.

Affected Resources

The following table shows which resources and fields are subject to namespace reference validation:

Resource	Field	Validation Level
AIWorkload	spec.computePoolRef.namespace	Same namespace only
AIWorkload	spec.profileBundles[].namespace	Same, system, or Tenant namespace
AIWorkload	spec.workloadDeploymentConfigs[].workloadProfileRef.namespace	Same or system namespace
AIWorkload	spec.clusterVariant.*.scalingPolicyRef.namespace	Same or system namespace
ComputePool	spec.clusterVariant.imported.environmentRef.namespace	Same namespace only
ComputePool	spec.profileBundleRef.namespace	Same, system, or Tenant namespace
ComputePool	spec.workloadDeploymentConfigs[].workloadProfileRef.namespace	Same or system namespace
ComputePool	spec.clusterVariant.*.scalingPolicyRef.namespace	Same or system namespace
ProfileBundle	spec.*.workloadProfileRefs[].namespace	Same or system namespace
Project	spec.computeConfigRef.namespace	Same or Tenant namespace only
Settings	spec.integrations.palette.namespace	Same namespace only

Behavior

If you attempt to create or update a resource with an invalid cross-Project namespace reference, the admission webhook rejects the request with a clear error message. For example, if an AIWorkload in Project namespace project-a references a ComputePool in Project namespace project-b, the webhook returns an error similar to the following:

spec.computePoolRef.namespace: namespace "project-b" must match owner namespace "project-a"

To resolve this error, ensure that the namespace reference is valid according to the validation level for that field. Refer to the table above for the permitted namespaces per field.

Next Steps

After you create a Tenant, create at least one Project under it. End users can access a Tenant in the UI only after at least one Project exists within it and their OIDC group is listed in that Project's roleMapping. Projects automatically inherit Tenant Settings when Tenant.spec.settingsRef is configured and the Project does not define its own Settings reference. GPU quotas set at the Tenant level apply to all Projects under that Tenant. PaletteAI grants Tenant admin groups admin permissions in all Project namespaces.

If you encounter issues when creating or managing Tenants, refer to Troubleshooting Tenants.