Manage Inference Quotas

This guide describes how to create, edit, and delete Inference Quotas. An Inference Quota is a usage budget that the PaletteAI Inference Launchpad gateway enforces on inference requests. Each quota caps the volume a caller can consume over a rolling window, measured in Requests, Tokens, or Cost.

You can manage Inference Quotas at the System, Tenant, or Project scope. The steps below apply at any scope; the navigation path differs by scope, as noted at the start of each procedure.

You can manage Inference Quotas from the PaletteAI console or by applying YAML Kubernetes manifests.

Prerequisites

UI Workflow

• Depending on the scope you plan to manage:

  • System — Platform operator access.
  • Tenant — Tenant admin role. Refer to Tenant Role Permissions.
  • Project — Project admin role. Refer to Project Role Permissions.

• The spectrocloud.com/inferencequotas:create permission for creating per-model quotas, spectrocloud.com/inferencequotas:update for editing limits, and spectrocloud.com/inferencequotas:delete for removing per-model quotas. If a button described in this guide is not displayed, your role does not include the required permission.

YAML Workflow

• Access to the PaletteAI hub cluster with permissions to read, create, update, and delete InferenceQuota resources in the target namespace. The target namespace is:

  • System scope — pai-system
  • Tenant scope — tenant-<tenant-name>
  • Project scope — the Project namespace (matches the Project name)

• At least one Compute Pool at the target scope uses a Launchpad-enabled Profile Bundle. Refer to the PaletteAI Inference Launchpad overview for details on the Launchpad opt-in.

Review the Scope-Total Quota

Each scope has one scope-total quota, created automatically with the scope. Every API key at the scope, and every request from a model at a child scope, draws against this quota.

UI Workflow

1. Log in to the PaletteAI console.

2. Navigate to the settings page for the scope you want to manage:

   • System — From the Projects Menu, select All Projects. From the left main menu, select System Settings.
   • Tenant — From the Projects Menu, select All Projects. From the left main menu, select Tenant Settings.
   • Project — Select the Project. From the left main menu, select Project Settings.

3. Select the Model Management section, and then select the Inference Quotas tab.

4. In the scope-total panel at the top of the page, review the Requests, Tokens, and Cost limits for each window. Empty limits mean the dimension is unlimited.

YAML Workflow

Each scope-total quota is named default regardless of scope — the scope is identified by the namespace the quota lives in.

Example command

kubectl get inferencequota default --namespace <project-name> --output yaml

Edit a Quota's Budget

Every Inference Quota has a budget (spec.budget): the Requests, Tokens, and Cost caps this quota itself may consume. Budget is separate from quota limit ceilings (spec.budgetLimits), which cap what descendant quotas are allowed to set. Configure ceilings in Set Quota Limit Ceilings.

When you lower a budget on a scope-total quota, PaletteAI automatically lowers any descendant scope-total or per-model budget that currently exceeds the new value for that dimension and window. Quotas already at or below the new value, and other dimensions or windows, are left unchanged.

A child budget must still fit the effective parent ceiling (the tighter of the parent's own budget and any budgetLimits that apply). In the UI, values above that ceiling are rejected on save. If you apply YAML that exceeds the ceiling, the API accepts the object and the controller marks it Out of policy (status.outOfPolicy: true) until you correct it.

UI Workflow

1. On the Inference Quotas tab, in the row for the quota you want to edit, open the three-dot menu and select Edit.

2. In the Edit Quota Definition drawer, expand each dimension (Requests, Tokens, Cost) you want to change. The form shows Hour and Day windows. Max allowed under each field is the parent ceiling for that dimension and window. Leave a field empty to leave that window unlimited.

3. Select Confirm. If you lowered a scope-total budget, wait a moment, and verify that over-limit descendant budgets were reduced.

YAML Workflow

Edit the spec.budget block. Each dimension is a list of window entries. window is one of second, minute, hour, or day, and limit is the cap. The console form exposes only Hour and Day; YAML may set any supported window.

Example update to a Project scope-total quota

apiVersion: spectrocloud.com/v1alpha1
kind: InferenceQuota
metadata:
  name: default
  namespace: <project-name>
spec:
  type: project
  budget:
    requests:
      - window: hour
        limit: 2000
      - window: day
        limit: 20000
    tokens:
      - window: day
        limit: 10000000

Apply the manifest:

Example command

kubectl apply --filename inferencequota.yaml

After you lower a parent scope-total spec.budget, wait for reconcile and re-read descendant quotas to confirm over-limit windows were clamped. If a quota remains above an ancestor ceiling, check status.outOfPolicy.

Create a Per-Model Quota

A per-model quota caps usage on a specific set of Model Deployments, in addition to the scope-total quota.

UI Workflow

1. On the Inference Quotas tab, in the per-model quotas panel, select Add Model Quota.

2. In the Quota Definition step, enter an Inference Quota Name. Names must be 63 characters or fewer, contain only lowercase alphanumeric characters or hyphens, and begin and end with an alphanumeric character. The name cannot be changed after creation.

3. Under Quota windows, expand each dimension you want to cap:

   • Requests — Whole-number cap on the number of inference calls.
   • Tokens — Whole-number cap on total input and output tokens.
   • Cost — Fractional dollar cap. Prefixed with $ in the form.

   Enter a limit for the Hour and Day windows you want to enforce. Leave a field empty to mark that window as unlimited.

4. (Optional) In the API Keys step, add one or more API keys to attach to this quota. Refer to Manage API Keys for field-level guidance. You can skip this step and add keys later.

5. In the Review step, confirm the values and select Create.

YAML Workflow

A per-model quota has spec.type: model and lives in the namespace of the scope you want it to apply at — the System namespace (pai-system), a Tenant namespace, or a Project namespace. AIWorkloads that reference the quota via spec.inferenceQuotaRef must live in the same namespace. Use spec.maxModelCount to cap how many AIWorkloads may share the quota.

Example per-model InferenceQuota

apiVersion: spectrocloud.com/v1alpha1
kind: InferenceQuota
metadata:
  name: model-x-budget
  namespace: <project-name>
spec:
  type: model
  maxModelCount: 1
  budget:
    requests:
      - window: hour
        limit: 500
    tokens:
      - window: hour
        limit: 250000
    cost:
      - window: day
        limit: 25.00

Apply the manifest and then attach one or more AIWorkloads to the quota by setting spec.inferenceQuotaRef on each:

Example command

kubectl apply --filename inferencequota.yaml

kubectl patch aiworkload model-x --namespace <project-name> --type merge \
  --patch '{"spec":{"inferenceQuotaRef":{"name":"model-x-budget"}}}'

Set Quota Limit Ceilings

Quota limit ceilings (spec.budgetLimits on a scope-total Inference Quota) cap what other quotas may set in their spec.budget. They do not change this scope's own budget. Use them at System, Tenant, or Project scope:

• Child scope ceilings (budgetLimits.descendantScope) — Cap descendant scope-total budgets. System caps Tenants and Projects; Tenant caps Projects. Not used at Project scope.
• Model ceilings (budgetLimits.model) — Cap per-model budgets at this scope and every scope below it.

Lowering a ceiling does not rewrite descendant budgets. Descendants that already exceed the new ceiling become Out of policy until you edit those budgets down or raise the ceiling again. Deployments stay running; the gateway denies inference against out-of-policy quotas, and editing those deployments is disabled until they are back within policy.

The effective ceiling for a child budget is the tighter of each ancestor's own spec.budget and the applicable budgetLimits entry.

UI Workflow

1. On the same settings page, select the Quota Limits tab (next to Inference Quotas).

2. Edit Model Quota Limits and, at System or Tenant scope, the child-scope section (Tenant Scope Quota Limits at System, Child Scope Quota Limits at Tenant). Leave a field empty for an unlimited ceiling on that window.

3. Select Save changes. If the save would put descendants out of policy, confirm the warning before continuing.

4. On the Inference Quotas tab, bring any Out of policy descendant budgets under the new ceiling, or raise the ceiling again.

YAML Workflow

Patch spec.budgetLimits on the scope-total quota (default in the scope namespace). budgetLimits is not allowed on type: model quotas.

Example System quota with quota limit ceilings

apiVersion: spectrocloud.com/v1alpha1
kind: InferenceQuota
metadata:
  name: default
  namespace: pai-system
spec:
  type: system
  budget:
    requests:
      - window: hour
        limit: 100000
    tokens:
      - window: day
        limit: 500000000
  budgetLimits:
    descendantScope:
      requests:
        - window: hour
          limit: 5000
      tokens:
        - window: day
          limit: 50000000
    model:
      requests:
        - window: hour
          limit: 500
      tokens:
        - window: hour
          limit: 250000

Example command

kubectl apply --filename inferencequota.yaml

After you tighten ceilings, list descendant quotas and check status.outOfPolicy. Admission does not reject an over-ceiling spec.budget; the controller records out-of-policy status on reconcile.

Delete a Per-Model Quota

Deleting a per-model quota removes the additional cap for the models it covered. Any Model Deployment that referenced the deleted quota is automatically rolled over to the scope-total quota at that scope — you do not need to update each deployment. Any API keys attached to the deleted quota are also deleted.

UI Workflow

1. On the Inference Quotas tab, in the row for the quota you want to delete, open the three-dot menu and select Delete.

2. In the confirmation dialog, review the impact summary and select Confirm.

YAML Workflow

Example command

kubectl delete inferencequota <quota-name> --namespace <project-name>

The scope-total quota is created and managed with its scope and cannot be deleted while the scope exists.

Validate

Confirm the quota is in effect:

UI Workflow

1. On the Inference Quotas tab, verify the quota appears in the expected panel and its limits match what you set.

2. Issue an inference call from a client that uses an API key attached to the quota. Refer to the PaletteAI Inference Launchpad documentation for the request format.

3. Return to the Inference Quotas tab and confirm the utilization on the quota row advances after the call.

YAML Workflow

Confirm that the quota is reporting observed usage:

Example command

kubectl get inferencequota <quota-name> --namespace <namespace> --output yaml

Check the status.status, status.used, status.remaining, and status.resetsAt fields. status.status reports the aggregate state as Active, NearLimit, or Exhausted; the usage fields update as the gateway meters inference requests.

Next Steps

• Manage API Keys — Issue credentials that draw against a quota.
• PaletteAI Inference Launchpad — Concept overview, quota types, and scope hierarchy.
• PaletteAI Inference Launchpad documentation — Enforcement at the gateway.