Version: v1.1.x

Tenants and Projects

PaletteAI uses Tenants and Projects to organize teams, control access, and manage GPU resources. This two-level hierarchy lets platform engineering teams set organization-wide policies while giving individual data science teams autonomy over their own workspaces.

For example, artificial intelligence and machine learning (AI/ML) platforms typically serve multiple teams with different needs. Platform teams often need to control who can access what, set resource limits, and manage infrastructure, whereas data science teams need isolated workspaces where they can deploy and manage their own workloads.

Tenants and Projects solve this by separating concerns.

Tenants - Define organizational boundaries and platform-wide policies.
Projects - Provide isolated workspaces for teams to do their work.

Tenants

A Tenant represents an organization or major division within your company. Platform engineering teams typically manage Tenants. Tenants provide the following benefits:

Organizational grouping - Group related Projects under a single administrative unit. For example, a "Research" Tenant might contain Projects for different research teams.
GPU quotas - Set maximum GPU usage across all Projects in the Tenant. This prevents any single team from consuming all available resources.
Platform team access - Grant OpenID Connect (OIDC) groups cluster-wide permissions to manage the Tenant and all its Projects.
Palette integration - Reference a Settings resource that contains Palette API credentials for cluster provisioning.

Automatic Resource Creation

Tenants are cluster-scoped resources; they exist outside of any namespace and can span the entire cluster.

When you create a Tenant, PaletteAI automatically creates a namespace with the name format tenant-<tenant-name> and provisions tenant-scoped RBAC resources (ClusterRoles, ClusterRoleBindings, and shared tenant-namespace Roles) for the OIDC groups specified in spec.tenantRoleMapping. When Projects are created under this Tenant, the Project controller creates per-project RoleBindings to these shared RBAC resources. This ensures platform teams can manage all Projects under their Tenant without manual Role-Based Access Control (RBAC) configuration.

info

When referencing a Settings resource in settingsRef, the namespace must use the auto-generated Tenant namespace format tenant-<tenant-name>. For example, a Tenant named primary-dev must reference Settings in the tenant-primary-dev namespace.

Default admin user

PaletteAI can include a default Dex local admin user when local users are enabled. For the default credentials, customization options, and operational caveats, refer to Local Dex Users.

For production, use OIDC groups instead of static Dex users. Configure Dex in your Helm chart values.

PaletteAI Tenants and Projects hierarchy

Tenant Configuration

Tenants reference a Settings resource and define which OIDC groups have administrative access. You can create a Tenant through the UI or declaratively through a YAML manifest.

The settingsRef field on the Tenant specifies a default Settings resource that Projects can inherit. This provides centralized management of common configurations such as Palette API credentials. Tenant-level Settings can be managed through the Settings Ref tab on the Tenant Settings page, where you can create, view, and manage Settings resources used across Projects.

Tenants must reference a ComputeConfig through the computeConfigRef field. This ComputeConfig resides in the tenant namespace and is used to populate defaults for tenant-scoped Compute Pools and AI Workloads.

Projects can reference the Tenant-level default Settings or define their own Settings resource in their Project namespace. For the details on how Tenant and Project Settings are merged when both are present, refer to Settings and Integrations.

Example Tenant manifest
apiVersion: spectrocloud.com/v1alpha1
kind: Tenant
metadata:
  name: primary-dev
spec:
  displayName: 'Primary Dev'

  # Default compute configuration
  computeConfigRef:
    name: default

  # Palette integration credentials
  settingsRef:
    name: dev-settings
    namespace: tenant-primary-dev

  # OIDC groups with tenant-level access
  tenantRoleMapping:
    groups:
      - admin
      - sre
      - operations

  # GPU quotas for all Projects in this Tenant
  gpuResources:
    limits:
      'NVIDIA-A100': 64
      'NVIDIA-H100': 48
    requests:
      'NVIDIA-A100': 8
      'NVIDIA-H100': 8

When to Create Multiple Tenants

For most deployments, a single Tenant with multiple Projects is sufficient. Projects can reference different Settings resources if they need different Palette credentials.

Consider multiple Tenants if you need any of the following:

Separate GPU quota pools - Each Tenant has independent GPU limits. If different organizations have separate GPU allocations that should not be shared, use separate Tenants.
Isolated administrative boundaries - Tenant admin groups (tenantRoleMapping) receive access to all Projects within that Tenant. Separate Tenants ensure one organization's admins cannot view another organization's Projects.
Independent usage tracking - Tenant status tracks GPU usage per Project. Separate Tenants provide separate usage reports for billing or chargeback.

If your teams can share a common GPU pool and administrative visibility is not a concern, we recommend creating a single Tenant for easier management.

Projects

A Project is a workspace where teams deploy and manage AI/ML applications. Most day-to-day work happens at the Project level. Projects provide the following benefits:

Isolated workspace - Each Project has its own namespace. Resources in one Project cannot reference namespaces belonging to other Projects. Admission webhook validation enforces this at create and update time, preventing scenarios where deleting one Project could silently break resources in another Project. For details on which namespaces each field is permitted to reference, refer to the Namespace Isolation section.
Team access control - Map OIDC groups to viewer, editor, or admin roles within the Project.
Resource scoping - Compute Pools, App Deployments, and configurations are scoped to their Project.
Target namespace control - Define which namespaces on spoke clusters workloads can be deployed to.
GPU budgets - Set GPU limits for the Project (must be within Tenant limits).
Optional Settings override - Use different Palette credentials than the parent Tenant, if needed.

Tenant Visibility

Tenant visibility in the PaletteAI UI is derived from Project access, not from the Tenant itself. A Tenant becomes visible to end users only when the following conditions are met:

At least one Project exists within the Tenant.
The user's OIDC group is listed in that Project's roleMapping (as a viewer, editor, or admin).

Users with no OIDC group membership in any Project under the Tenant cannot access the Tenant in the UI. The UI lists only Tenants the user can reach through Project memberships.

Project Roles

Projects provide three permission levels.

Role	Capabilities
Viewer	Read-only access to all Project resources
Editor	Create and manage App Deployments, Model Deployments, Profile Bundles, and Workload Profiles
Admin	Full access including Project settings and permissions

Refer to Roles and Permissions for an overview of each role, or to the Role Permissions reference for the full Kubernetes permission matrix.

Project Configuration

Projects reference their parent Tenant and define team access. You can create a Project through the UI or declaratively through a Kubernetes manifest. To learn how to create a Project, refer to our Create and Manage Projects guide.

info

Model Management (Model as a Service mappings and model access control lists) is configured after Project creation through the Project Settings page, not during the Project creation wizard.

Example Project manifest
apiVersion: spectrocloud.com/v1alpha1
kind: Project
metadata:
  name: project-a
  namespace: project-a
spec:
  displayName: 'Project A'

  settingsRef:
    name: dev-settings

  # Parent Tenant
  tenantRef:
    name: primary-dev

  # Default compute configuration
  computeConfigRef:
    name: edge-compute-config
    namespace: project-a

  # Team access by OIDC group
  roleMapping:
    viewer:
      - fin-ops
      - auditors
    editor:
      - ml-engineers
      - data-team
    admin:
      - admin
      - sre

  # GPU quotas for this Project
  gpuResources:
    limits:
      'NVIDIA-A100': 64
      'NVIDIA-H100': 48
      'Default': 24
    requests:
      'NVIDIA-A100': 6
      'NVIDIA-H100': 6
      'Default': 4

  # Target namespace restrictions
  admissionConfiguration:
    workloadConfiguration:
      deploymentNamespaces:
        allow:
          - project-a
          - staging
        disallow:
          - cluster-scoped
          - '*'

Target Namespaces

Projects can restrict which namespaces on spoke clusters workloads can be deployed to. This provides security boundaries and prevents workloads from deploying to sensitive system namespaces.

Target namespace restrictions are configured through admissionConfiguration.workloadConfiguration.deploymentNamespaces using allow and deny lists:

Allow list - Namespaces where workloads and their resources can be deployed.
Deny list - Namespaces where workloads and their resources cannot be deployed.

Cluster-Scoped Resources

The special keyword cluster-scoped controls whether workloads can create cluster-scoped resources (resources without a namespace, such as ClusterRoles or CustomResourceDefinitions). Include cluster-scoped in the allow list to permit creation of cluster-scoped resources.

Policy Behavior

Target namespace policies follow these rules:

If the allow list contains *, all namespaces are allowed except those in the deny list.
If the deny list contains *, all namespaces are blocked except those in the allow list.
When both lists are empty or contain only *, PaletteAI denies all namespaces.
The Project's own namespace must always be in the allow list (unless allow-all is enabled).

Example target namespace configuration
admissionConfiguration:
  workloadConfiguration:
    deploymentNamespaces:
      allow:
        - project-a # Project namespace
        - staging # Additional namespace
      disallow:
        - cluster-scoped # Deny cluster-scoped resources
        - '*' # Deny all other namespaces

Namespace Isolation

Admission webhooks enforce that resource references stay within permitted namespace boundaries. Cross-namespace references are rejected at admission time. The following table lists each field and the set of namespaces it may reference.

Resource	Field	Permitted Namespaces
`AIWorkload`	`spec.computePoolRef.namespace`	Project namespace only
`AIWorkload`	`spec.profileBundles[].namespace`	Project namespace, the system namespace, or Tenant namespace
`AIWorkload`	`spec.workloadDeploymentConfigs[].workloadProfileRef.namespace`	Project namespace or the system namespace
`ComputePool`	`spec.clusterVariant.imported.environmentRef.namespace`	Project namespace only
`ComputePool`	`spec.profileBundleRef.namespace`	Project namespace, the system namespace, or Tenant namespace
`ComputePool`	`spec.workloadDeploymentConfigs[].workloadProfileRef.namespace`	Project namespace or the system namespace
`ComputePool`	`spec.clusterVariant.*.scalingPolicyRef.namespace`	Project namespace or the system namespace
`ProfileBundle`	`spec.*.workloadProfileRefs[].namespace`	Project namespace or the system namespace
`Project`	`spec.computeConfigRef.namespace`	Project namespace or Tenant namespace (the system namespace is not permitted)
`Settings`	`spec.integrations.palette.namespace`	Settings namespace only

info

The system namespace, which defaults to mural-system, is the shared PaletteAI namespace. Resources in this namespace are shared across all Projects and Tenants.

GPU Quotas

GPU quotas prevent resource exhaustion and ensure fair allocation across teams. Quotas can be set at the Tenant and Project levels.

Quota Types

Field	Available on	Purpose
`limits`	Tenant, Project	Maximum total GPU usage per GPU family within the scope. Applies to every Compute Pool and App Deployment in scope.
`requests`	Tenant, Project	Maximum GPUs that a single App Deployment or Compute Pool can request per GPU family.
`tenantReservations`	Tenant only	Portion of the Tenant `limits` reserved exclusively for tenant-namespace resources. The remainder (`limits - tenantReservations`) is the budget shared by all Projects.
`projectLimits`	Tenant only	Per-Project ceiling that the Tenant admin can impose, keyed by Project name and then by GPU family. Caps a Project's usage in addition to its own `limits`.

All quota keys are GPU family names (for example, NVIDIA-A100, NVIDIA-H100). Use the special key Default to set a per-family catch-all for any GPU family not listed explicitly.

Example GPU quota
gpuResources:
  limits:
    'NVIDIA-A100': 32 # Up to 32 A100s total within this scope
  requests:
    'NVIDIA-A100': 8 # A single App Deployment or Compute Pool can request up to 8 A100s

How Tenant and Project Quotas Interact

At admission, a Project's limits.<family> and requests.<family> must be less than or equal to the Tenant's limits.<family> and requests.<family> for the same family. If the Tenant only specifies a Default, that catch-all is used.
At runtime, a Project's per-Project cap is the lower of two values: the Project's own gpuResources.limits.<family> and the Tenant's gpuResources.projectLimits[<this Project>].<family>. Whichever is set and lower wins.
The Tenant's gpuResources.limits.<family> is enforced separately at aggregate scope: combined usage across all Projects plus the tenant namespace must not exceed it.
If a Project does not specify limits for a family, no project-level cap is enforced for that Project. The Tenant's aggregate limits (and any projectLimits ceiling) still bound usage.
If a family is constrained by neither the Tenant nor the Project, PaletteAI does not restrict GPU usage for that family.

Tenant Namespace Reservations

Compute Pools and App Deployments can also be created directly in the tenant namespace (tenant-<tenant-name>), outside of any Project. By default, the full Tenant limits budget is available to Projects and tenant-namespace resources have no GPU quota, meaning a Compute Pool or App Deployment that requires GPUs cannot be deployed in the tenant namespace.

To enable tenant-namespace GPU usage, set tenantReservations. Each entry must be less than or equal to the corresponding limits entry for the same family.

Reserving 8 A100s for tenant-namespace usage
gpuResources:
  limits:
    'NVIDIA-A100': 32
  tenantReservations:
    'NVIDIA-A100': 8 # Tenant namespace gets 8; Projects share the remaining 24

The reservation enforces symmetric isolation:

Tenant-namespace resources may not exceed tenantReservations.
Combined GPU usage across all Projects may not exceed (limits - tenantReservations).

Per-Project Ceilings

Use projectLimits to cap how many GPUs a specific Project can consume across all of its Compute Pools and App Deployments, regardless of the Project's own limits.

Tenant capping project-a at 16 A100s and project-b at 8 A100s
gpuResources:
  limits:
    'NVIDIA-A100': 32
  projectLimits:
    project-a:
      'NVIDIA-A100': 16
    project-b:
      'NVIDIA-A100': 8
    Default:
      'NVIDIA-A100': 4 # Any Project not listed gets at most 4 A100s

When both projectLimits[<project>].<family> and the Project's own gpuResources.limits.<family> are set, the Project's effective cap is the lower of the two.

Default Quotas

Use the Default key to set quotas for GPU families not explicitly listed. The Default key acts as a per-family catch-all cap. Each unlisted GPU family individually gets up to the specified limit.

gpuResources:
  limits:
    'NVIDIA-A100': 32
    'Default': 16 # Each unlisted GPU family is limited to 16 total
  requests:
    'NVIDIA-A100': 8
    'Default': 4 # Each unlisted GPU family is limited to 4 per request

GPU Usage and Oversubscription

PaletteAI automatically tracks GPU usage in real time under status.gpuUsage in the respective Tenant and Project Custom Resource Definitions (CRDs). At the Tenant level, status.tenantGpuUsage separately tracks usage by tenant-namespace resources.

GPU tracking at Tenant scope
status:
  gpuUsage:
    project-a:
      'NVIDIA-A100': 12
      'NVIDIA-H100': 8
    project-b:
      'NVIDIA-A100': 4
      'NVIDIA-H100': 4
  tenantGpuUsage:
    'NVIDIA-A100': 4

GPU tracking at Project scope
status:
  gpuUsage:
    'NVIDIA-A100': 12
    'NVIDIA-H100': 8

Use the following commands to check GPU usage at the Tenant and Project levels.

kubectl get tenant <tenant-name> --output jsonpath='{.status.gpuUsage}'
kubectl get tenant <tenant-name> --output jsonpath='{.status.tenantGpuUsage}'
kubectl get project <project-name> --namespace <project-namespace> --output jsonpath='{.status.gpuUsage}'

When concurrent requests cause usage to exceed limits, the following sequence occurs.

The Tenant TenantNotOversubscribed condition or the Project NotOversubscribed condition becomes False.
New GPU requests are blocked until usage drops below limits. App Deployments and Compute Pools requesting GPUs remain in a pending state.
After operators delete existing workloads and usage drops below limits, the condition clears and new requests can proceed.

The Tenant status.oversubscribedVariants field records which GPU variants are oversubscribed and at which scope:

Tenant - Overall tenant usage (Projects + tenant namespace) exceeds limits for the family. Blocks all new GPU resources for that family.
TenantNamespace - Tenant-namespace usage exceeds tenantReservations for the family. Blocks only new tenant-namespace GPU resources for that family; Projects can continue consuming GPUs up to their cap.

Example oversubscribed variants
status:
  oversubscribedVariants:
    'NVIDIA-A100': Tenant
    'NVIDIA-H100': TenantNamespace

Use the following commands to check the oversubscription status at the Tenant and Project levels.

kubectl get tenant <tenant-name> --output jsonpath='{.status.conditions[?(@.type=="TenantNotOversubscribed")]}'
kubectl get tenant <tenant-name> --output jsonpath='{.status.oversubscribedVariants}'
kubectl get project <project-name> --namespace <project-namespace> --output jsonpath='{.status.conditions[?(@.type=="NotOversubscribed")]}'

Default Resources

PaletteAI creates a default Tenant and Project during installation. These provide a starting point for exploration but should be customized for production use.

To disable default resource creation, set global.featureFlags.systemDefaultResources to false in your Helm chart values:

global:
  featureFlags:
    systemDefaultResources: false

Tenants​

Automatic Resource Creation​

Tenant Configuration​

When to Create Multiple Tenants​

Projects​

Tenant Visibility​

Project Roles​

Project Configuration​

Target Namespaces​

Cluster-Scoped Resources​

Policy Behavior​

Namespace Isolation​

GPU Quotas​

Quota Types​

How Tenant and Project Quotas Interact​

Tenant Namespace Reservations​

Per-Project Ceilings​

Default Quotas​

GPU Usage and Oversubscription​

Default Resources​

Tenants

Automatic Resource Creation

Tenant Configuration

When to Create Multiple Tenants

Projects

Tenant Visibility

Project Roles

Project Configuration

Target Namespaces

Cluster-Scoped Resources

Policy Behavior

Namespace Isolation

GPU Quotas

Quota Types

How Tenant and Project Quotas Interact

Tenant Namespace Reservations

Per-Project Ceilings

Default Quotas

GPU Usage and Oversubscription

Default Resources