System Scope

System is the top-level scope in PaletteAI, backed by the pai-system namespace and managed by platform operators. It sits above Tenants and Projects in the scope hierarchy.

You define and deploy the same resource kinds at System scope that you use at Tenant and Project scope — including Compute Pools, AIWorkloads, Profile Bundles, Compute Configs, Scaling Policies, Variable Sets, and Settings. These are real, deployable resources, not templates: a System-scoped Compute Pool provisions infrastructure exactly like a Tenant- or Project-scoped one. System scope is simply the highest, organization-wide tier at which you can create them.

Sharing System Resources

Share System-scoped resources down to Tenants and Projects with the sharedWith field. A consumer that is granted access can view and clone the resource into its own scope. A System-scoped Variable Set applies to every Tenant and Project automatically. For the full sharing model, refer to Sharing Resources.

The System Resource

A single System resource holds organization-wide configuration and defaults. Its metadata.name matches the Helm value global.instanceName (default pai-system). Examples on this page use the default; substitute your global.instanceName if you set one.

The System singleton exposes:

• spec.computeConfigRef — the default Compute Config used to populate defaults for all Tenants, Projects, Compute Pools, and AIWorkloads.
• spec.settingsRef — the default Settings resource for system operations, whose Hugging Face and NVIDIA NGC integrations can be shared down to lower scopes.
• spec.gpuResources — organization-wide GPU quota: system-wide limits and requests that act as ceilings on every Tenant, optional per-Tenant limits, and reservations carved out for system-namespace resources.
• spec.migUISettings and spec.modelSettings — MIG UI gating and system-scope model integrations.

info

Do not confuse the System scope (pai-system), which is user-managed, with the internal platform namespace (mural-system by default). The platform namespace holds installer-managed, built-in resources that cannot be edited in place. Refer to Multi-Tenancy for the distinction.

For how Project users receive read access to shared system resources, refer to Roles and Permissions.

GPU Quotas

At the highest level, the System resource defines system-wide GPU resource management through its spec.gpuResources configuration. This provides the top-level policy that cascades down to all Tenants and Projects.

System-Level Quotas

System defines system-wide GPU resource management with the following fields:

• limits - Total system-wide GPU capacity per variant. This is the absolute ceiling for all GPU usage across the entire PaletteAI instance.

• systemReservations - GPU capacity reserved exclusively for system-namespace workloads per variant. System-namespace resources (Compute Pools and AIWorkloads deployed in the System scope namespace, pai-system) can only consume GPU variants listed in systemReservations, up to the reserved amount.

• tenantLimits - Per-Tenant GPU usage caps enforced at the system level. This is a map keyed by Tenant name (or Default for a catch-all), with each entry specifying per-variant limits. A Tenant's aggregate GPU usage (the sum of all its Projects plus its tenant-namespace resources) must not exceed the limit specified here.

• requests - Per-workload GPU request size caps enforced across all scopes. Each individual Compute Pool or AIWorkload cannot request more GPUs per variant than this ceiling, regardless of whether it lives in the system namespace, a tenant namespace, or a project namespace.

Example System GPU configuration

apiVersion: spectrocloud.com/v1alpha1
kind: System
metadata:
  name: pai-system
spec:
  gpuResources:
    # Total system capacity per GPU variant
    limits:
      'NVIDIA A100-SXM4-80GB | 80 GB': 128
      'NVIDIA H100 PCIe | 80 GB': 96

    # Reserved for system-namespace workloads
    systemReservations:
      'NVIDIA A100-SXM4-80GB | 80 GB': 16
      'NVIDIA H100 PCIe | 80 GB': 8

    # Per-Tenant usage caps
    tenantLimits:
      primary-dev:
        'NVIDIA A100-SXM4-80GB | 80 GB': 64
        'NVIDIA H100 PCIe | 80 GB': 48
      research:
        'NVIDIA A100-SXM4-80GB | 80 GB': 48
        'NVIDIA H100 PCIe | 80 GB': 40
      Default:
        'NVIDIA A100-SXM4-80GB | 80 GB': 32
        'NVIDIA H100 PCIe | 80 GB': 24

    # Per-workload request caps
    requests:
      'NVIDIA A100-SXM4-80GB | 80 GB': 8
      'NVIDIA H100 PCIe | 80 GB': 8

System GPU Utilization Tracking

System tracks system-wide GPU usage in its status fields:

• status.gpuUsage - Maps each Tenant name to its aggregate GPU usage per variant. Each Tenant's aggregate usage is the sum of all its Projects' usage plus its tenant-namespace resources. This mirrors how status.gpuUsage works at the Tenant level (per-Project roll-up), but one scope up.

• status.systemGpuUsage - Tracks GPU usage in the System scope namespace (pai-system) per variant. This reflects the total GPU capacity reserved by all Compute Pools deployed in the system namespace.

Example system GPU usage tracking

status:
  gpuUsage:
    primary-dev:
      'NVIDIA A100-SXM4-80GB | 80 GB': 48
      'NVIDIA H100 PCIe | 80 GB': 32
    research:
      'NVIDIA A100-SXM4-80GB | 80 GB': 36
      'NVIDIA H100 PCIe | 80 GB': 24
  systemGpuUsage:
    'NVIDIA A100-SXM4-80GB | 80 GB': 12
    'NVIDIA H100 PCIe | 80 GB': 4

Use the following commands to check system-wide GPU usage:

kubectl get system pai-system --output jsonpath='{.status.gpuUsage}'
kubectl get system pai-system --output jsonpath='{.status.systemGpuUsage}'

System Oversubscription Detection

The SystemNotOversubscribed condition on System monitors GPU oversubscription at three scopes:

• System scope - When aggregate tenant usage (sum of all Tenants' usage from status.gpuUsage) plus system-namespace usage exceeds limits for a GPU variant, PaletteAI blocks all new GPU requests system-wide for that variant until usage drops below the limit.

• TenantPool scope - When aggregate Tenant usage exceeds the tenant pool (limits less systemReservations) for a GPU variant, encroaching on the system reservation.

• SystemNamespace scope - When system-namespace usage exceeds systemReservations for a GPU variant, PaletteAI blocks only system-namespace GPU requests for that variant. Tenant and Project workloads can continue to consume GPUs up to their respective caps.

The status.oversubscribedVariants field records which GPU variants are oversubscribed and at which scope:

Example system oversubscription

status:
  oversubscribedVariants:
    'NVIDIA A100-SXM4-80GB | 80 GB': System
    'NVIDIA H100 PCIe | 80 GB': SystemNamespace

When a variant is oversubscribed at multiple scopes, the highest-priority value wins: System > TenantPool > SystemNamespace.

Use the following commands to check system oversubscription status:

kubectl get system pai-system --output jsonpath='{.status.conditions[?(@.type=="SystemNotOversubscribed")]}'
kubectl get system pai-system --output jsonpath='{.status.oversubscribedVariants}'

Validation Hierarchy

When you create or update Compute Pools or AIWorkloads, admission webhooks validate GPU requests against multiple levels in the quota hierarchy:

System-level validations:

• Per-workload request caps: Each workload's GPU request per variant must not exceed System.spec.gpuResources.requests.
• Per-Tenant limits: A Tenant's aggregate GPU usage must not exceed System.spec.gpuResources.tenantLimits[<tenant-name>].
• System pool ceiling: Combined GPU usage across all Tenants must not exceed (limits minus systemReservations) per variant.

The same admission pass also enforces the Tenant-level and Project-level limits, requests, and reservations. For those rules, refer to Tenant GPU Quotas and Project GPU Quotas.

At each scope, the effective cap is the most restrictive limit that applies: a Project's GPU usage is bounded by the lower of its own limits, the Tenant's projectLimits for that Project, the Tenant's overall limits, and the system-wide pool ceiling.