Skip to main content
Version: v1.2.x

Create and Manage Tenants

A Tenant represents an organization or major division within your company and serves as the top-level organizational unit in PaletteAI.

A default Tenant is automatically created when you install PaletteAI with global.featureFlags.systemDefaultResources: true in your Helm values. For production workloads with multiple organizations or divisions that require separate GPU quota pools or isolated administrative boundaries, we recommend creating separate Tenants. For guidance on when to create multiple Tenants, refer to When to Create Multiple Tenants.

info

PaletteAI automatically creates a tenant-<name> namespace for internal operations when you create a Tenant. The Settings controller merges Tenant and Project Settings automatically when Tenant.spec.settingsRef (a fully-qualified name-and-namespace reference) is configured. For details on the namespace model, refer to Automatic Resource Creation.

You can create Tenants using the PaletteAI UI or using YAML Kubernetes manifests.

Limitations​

  • A newly created Tenant does not appear to end users in the PaletteAI UI until at least one Project exists within it and the user's OIDC group is listed in that Project's roleMapping. Tenant visibility is derived from Project access, not from the Tenant itself. Users with no OIDC group membership in any Project under the Tenant cannot access the Tenant in the UI.

Prerequisites​

  • Access to the system scope of the PaletteAI console with permission to create Tenants. If the Create Tenant button is not displayed, your role does not include the required permission.

  • Palette API credentials for the Tenant's Settings integration.

Create Tenant​

Create a Tenant to define organizational boundaries, set GPU quotas, and control access across multiple Projects.

Enablement​

  1. Log in to the PaletteAI console, and switch to the system scope.

  2. From the left main menu, select Tenants.

  3. Select Create Tenant.

  4. On the Basic information page, enter a display name for the Tenant. (Optional) Add a description, labels, and annotations. Select Next.

  5. On the Tenant Settings page, configure the Tenant's default Settings and integrations. A Palette integration is required; you can also add Hugging Face and NVIDIA NGC integrations to enable model management features. Select Next.

  6. On the Access control page, enter the Tenant Admin Groups. Enter the IdP group name as it appears in your identity provider (for example, Okta Admins), not the canonical pai:tenant:… string. PaletteAI persists the tenant-scoped canonical form on the Tenant resource when you create the Tenant. After upgrade, configure user impersonation groupMap entries so those IdP names map to the canonical strings unless your IdP emits the canonical names directly. Select Next.

  7. (Optional) On the Resource Limits page, set GPU limits and requests per GPU variant to control tenant-wide resource allocation. Select Next.

  8. On the Review page, confirm your configuration, and then select Create Tenant.

Validate​

  1. From the left main menu, select Tenants.

  2. Confirm the new Tenant appears in the list and reports a ready status. The Tenant becomes visible to end users once it contains at least one Project that their OIDC groups can access.

Modify Tenant​

In the PaletteAI console, modify a Tenant from its Tenant Settings page, which covers display values, integrations, user access, resource limits, and more. Refer to Configure Tenant Settings for each tab.

Using the YAML workflow, update your manifests and apply them with kubectl apply to track changes in version control.

Enablement​

  1. Open the manifest you want to update and make the necessary changes.

    Use the following table to identify what manifest to update for common changes. For the full parameter list, refer to the applicable Resource page.

    ResourceModificationsAdditional Information
    TenantUpdate displayName. Update tenantRoleMapping.groups. Update gpuResources.Tenants and Projects - GPU Quotas
    SettingsCreate a Settings resource in the Tenant namespace (tenant-<name>), and then update the Tenant settingsRef. To remove shared Settings, delete settingsRef from the Tenant spec.Settings

    The following example adds a Tenant admin group named operations and sets GPU resource limits.

    vi tenant.yaml
    Updated Tenant
    apiVersion: spectrocloud.com/v1alpha1
    kind: Tenant
    metadata:
      name: primary-dev
    spec:
      displayName: 'Primary Dev'
      computeConfigRef:
        name: default
      settingsRef:
        name: dev-settings
        namespace: tenant-primary-dev
      tenantRoleMapping:
        groups:
          - 'pai:tenant:primary-dev:role:tnt-adm-admin'
          - 'pai:tenant:primary-dev:role:tnt-adm-sre'
          - 'pai:tenant:primary-dev:role:tnt-adm-operations'
      gpuResources:
        limits:
          'NVIDIA A100-SXM4-80GB | 80 GB': 64
          'NVIDIA H100 PCIe | 80 GB': 48
          'Default': 24
        requests:
          'NVIDIA A100-SXM4-80GB | 80 GB': 8
          'NVIDIA H100 PCIe | 80 GB': 8
          'Default': 4
        # Optional. Reserve GPUs exclusively for tenant-namespace resources.
        # Each entry must be <= the matching `limits` entry. Projects share (limits - tenantReservations).
        tenantReservations:
          'NVIDIA A100-SXM4-80GB | 80 GB': 8
        # Optional. Cap individual Projects below the Project's own `limits`.
        # The Project's effective cap is the lower of its own `limits` and the matching `projectLimits` entry.
        projectLimits:
          project-a:
            'NVIDIA A100-SXM4-80GB | 80 GB': 32
          Default:
            'NVIDIA A100-SXM4-80GB | 80 GB': 8

  2. Save the file and apply the updated manifest.

    kubectl apply --filename <manifest-location>
    kubectl apply --filename tenant.yaml

Validate​

Verify that the Tenant reflects your updates.

kubectl describe tenant <tenant-name>

The following example shows the updated Tenant admin group and GPU resource limits.

kubectl describe tenant primary-dev
Example Output
Name: primary-dev
# ... metadata omitted for readability
Spec:
  Display Name: Primary Dev
  Gpu Resources:
    Limits:
      Default: 24
      NVIDIA A100-SXM4-80GB | 80 GB: 64
      NVIDIA H100 PCIe | 80 GB: 48
    Requests:
      Default: 4
      NVIDIA A100-SXM4-80GB | 80 GB: 8
      NVIDIA H100 PCIe | 80 GB: 8
  Settings Ref:
    Name: dev-settings
    Namespace: tenant-primary-dev
  Tenant Role Mapping:
    Groups: pai:tenant:primary-dev:role:tnt-adm-admin
      pai:tenant:primary-dev:role:tnt-adm-sre
      pai:tenant:primary-dev:role:tnt-adm-operations
Status:
  Ready: true
Events: <none>

Namespace Reference Validation​

To enforce Project isolation, PaletteAI admission webhooks validate that resources cannot reference namespaces belonging to other Projects. This prevents scenarios where deleting Project B could break resources in Project A that hold references to namespaces in Project B.

Validation Levels​

PaletteAI validates namespace references at four levels, depending on the resource type and field:

  1. Same namespace only: Some resource references must be in the same namespace as the owning resource. For example, ComputePool.spec.clusterVariant.imported.environmentRef.namespace must match the owning Project's namespace.

  2. Same or System namespace: Some references can be in the resource's namespace or a system namespace (pai-system or mural-system) for shared resources. For example, ProfileBundle.spec.*.workloadProfileRefs[].namespace can reference either the ProfileBundle's namespace or a system namespace.

  3. Same, System, or Tenant namespace: Some references can also use the Tenant namespace (tenant-<name>) for Tenant-wide shared resources. For example, AIWorkload.spec.computePoolRef.namespace can reference the Project namespace, pai-system, or the Tenant namespace when the referenced ComputePool is shared with the Project (via ComputePool.spec.sharedWith). AIWorkload.spec.profileBundles[].namespace, ComputePool.spec.profileBundleRef.namespace, and the workloadDeploymentConfigs[].workloadProfileRef.namespace field on both AIWorkload and ComputePool can reference the Project namespace, pai-system, or the Tenant namespace.

  4. Same or Tenant namespace only: Some references can be in the resource's own namespace or the owning Tenant's namespace (tenant-<name>), but not the system namespace. For example, Project.spec.computeConfigRef.namespace must be either the Project namespace or the Tenant namespace.

Affected Resources​

The following table shows which resources and fields are subject to namespace reference validation:

ResourceFieldValidation Level
AIWorkloadspec.computePoolRef.namespaceSame namespace, System namespace, or Tenant namespace (Tenant when ComputePool is shared with the project)
AIWorkloadspec.profileBundles[].namespaceSame, system, or Tenant namespace
AIWorkloadspec.workloadDeploymentConfigs[].workloadProfileRef.namespaceSame, system, or Tenant namespace
AIWorkloadspec.clusterVariant.*.scalingPolicyRef.namespaceSame, system, Built-in, or Tenant namespace
ComputePoolspec.clusterVariant.imported.environmentRef.namespaceSame namespace only
ComputePoolspec.profileBundleRef.namespaceSame, system, or Tenant namespace
ComputePoolspec.workloadDeploymentConfigs[].workloadProfileRef.namespaceSame, system, or Tenant namespace
ComputePoolspec.clusterVariant.*.scalingPolicyRef.namespaceSame, system, Built-in, or Tenant namespace
ProfileBundlespec.*.workloadProfileRefs[].namespaceSame or system namespace
Projectspec.computeConfigRef.namespaceSame or Tenant namespace only
Settingsspec.integrations.palette.namespaceSame namespace only

Behavior​

If you attempt to create or update a resource with an invalid cross-Project namespace reference, the admission webhook rejects the request with a clear error message. For example, if an AIWorkload in Project namespace project-a references a ComputePool in Project namespace project-b, the webhook returns an error similar to the following:

spec.computePoolRef.namespace: namespace "project-b" must match owner namespace "project-a"

To resolve this error, ensure that the namespace reference is valid according to the validation level for that field. Refer to the table above for the permitted namespaces per field.

Next Steps​

After you create a Tenant, create at least one Project under it. End users can access a Tenant in the UI only after at least one Project exists within it and their OIDC group is listed in that Project's roleMapping. Each Project must define its own Settings resource and Palette integration. Hugging Face and NVIDIA NGC integrations from the Tenant Settings Ref can be shared with Projects when configured. GPU quotas set at the Tenant level apply to all Projects under that Tenant. PaletteAI grants Tenant admin groups admin permissions in all Project namespaces.

If you encounter issues when creating or managing Tenants, refer to Troubleshooting Tenants.