Skip to main content
Version: v1.3.x

PaletteAI Inference Launchpad

PaletteAI Inference Launchpad is Spectro Cloud's inference gateway. It runs on a Kubernetes cluster and serves models to callers over an OpenAI-compatible API, applying authentication and usage metering on every request.

Inference Launchpad is a standalone product. You can run it on its own and define inference quotas and issue API keys directly on the cluster it runs on. When you integrate it with PaletteAI, PaletteAI becomes the central control plane for one or more Launchpad clusters: you define usage budgets and issue credentials once at the PaletteAI hub, and the hub federates them to every Launchpad-enabled Compute Pool. You never manage inference quotas or API keys on the spokes directly. The PaletteAI integration is one way to run Inference Launchpad, not the only way.

This page describes only the PaletteAI-managed workflow. For the gateway itself — how models are exposed, how requests are routed, and how metering telemetry is reported — refer to the PaletteAI Inference Launchpad documentation.

What PaletteAI Manages​

  • Inference Quotas — Usage budgets that cap how much of an AI resource — requests, tokens, or cost — callers can consume over a rolling window. Defined at System, Tenant, or Project scope, either as the scope-total budget that caps all inference at that scope or as a per-model budget that narrows a specific set of Model Deployments.
  • API Keys — Bearer credentials that a client presents to the Inference Launchpad gateway. Each key draws against exactly one Inference Quota. OIDC groups on the key control who may view and manage the credential in PaletteAI; the gateway authenticates callers with the bearer token alone.

Inference Quotas​

An Inference Quota is a usage budget the Inference Launchpad gateway enforces on inference requests. Each quota caps consumption on one or more of three dimensions — Requests, Tokens, and Cost — over one or more rolling windows. If any single dimension hits its limit, the quota is exhausted and further usage is blocked until the window resets.

Types of Inference Quotas​

There are two types of Inference Quotas. Both live at the same scope; the difference is what they cap.

TypeDescriptionUse Case
Scope-totalOne quota per scope that caps the aggregate usage of every model at that scope. Every request from a model at that scope draws against this quota, including requests already counted by any per-model quota.A single, scope-wide budget for all inference the scope is allowed to consume.
Per-modelAn additional quota that caps only the models that opt in through their AIWorkload.spec.inferenceQuotaRef. Requests to those models draw against the per-model quota and every scope-total quota at or above the model's scope.A tighter cap on a specific model — for example, an expensive frontier model — that must not consume the scope-total budget on its own.

Only one scope-total quota exists at each scope, created automatically when the scope is set up. You can add any number of per-model quotas.

Scope Hierarchy​

Inference Quotas follow the PaletteAI scope hierarchy: System is the root, Tenants sit below System, and Projects sit below Tenants. A parent's scope-total quota clamps its descendants:

  • The System scope-total quota caps the total inference across system-scoped Compute Pools and every Tenant.
  • Each Tenant scope-total quota caps the total inference across tenant-scoped Compute Pools and the Projects it contains.
  • Each Project scope-total quota caps the inference issued from that Project.

Per-model quotas at a lower scope cannot exceed the corresponding limit at a parent scope. When you set a limit in the UI, the form displays the maximum value allowed by the parent so you can size the quota within the ceiling.

Quota Dimensions and Windows​

DimensionWhat It Measures
RequestsNumber of inference API calls received by the gateway.
TokensTotal input and output tokens consumed by inference calls.
CostEstimated cost in USD, derived from per-model prices published to the gateway.

Each dimension is limited over one or more windows. Two windows are available in the UI: Hour and Day. A quota can leave a dimension unset — an unset dimension is not enforced.

Budget Reductions, Ceilings, and Out-of-Policy Descendants​

Two fields control hierarchy behavior:

  • spec.budget — What this quota may consume. Lowering a scope-total budget automatically clamps descendant budgets that exceed the new value for that dimension and window.
  • spec.budgetLimits — Ceilings on what descendant scope-total and per-model quotas may set in their budgets. Lowering a ceiling does not rewrite those budgets; over-limit descendants become Out of policy until you edit them down or raise the ceiling.

Out-of-policy deployments stay running, but the gateway denies inference against them, and editing those deployments is disabled until the quota is back within policy. Configure budgets and ceilings in Manage Inference Quotas.

Deleting a per-model quota removes the additional cap for the models it covered; those models fall back to the scope-total quota. Any API keys attached to a deleted per-model quota are also deleted. The scope-total quota itself is created and managed with its scope and cannot be deleted while the scope exists.

API Keys​

An API Key is a bearer credential that a client presents to the Inference Launchpad gateway to authenticate an inference request. Each key draws against exactly one Inference Quota, so every request the key authorizes counts toward that quota's limits.

The API Key Value​

The controller generates the key value on your behalf when the API Key is created. Every value begins with the prefix pai- so it can be quickly identified as being issued by PaletteAI.

The value never appears on the API Key custom resource itself. It is stored in a Kubernetes Secret that the controller creates alongside the API Key, and the UI reads the value out of that Secret each time you press the Copy button. Copy the value from the drawer whenever you need it, hand it to the caller through a secure channel, and treat the API Key like any other bearer token. The value cannot be rotated in place; if the current value is compromised, delete the key and issue a new one.

Inference Quota Binding​

Each API Key is bound to one Inference Quota at the moment you create it, and the binding is immutable. What the key can call depends on which type of quota you bind it to:

  • Attached to a scope-total quota, the key can call any model at that scope.
  • Attached to a per-model quota, the key can call only the models covered by that quota — that is, the AIWorkloads in the quota's own namespace that reference it through spec.inferenceQuotaRef.

To move a key to a different quota, delete it and create a new key against the target quota.

OIDC Groups​

OIDC groups on an API Key are a PaletteAI control-plane concern. They decide which identity-provider groups may view the key and its secret value in the console (and through the Kubernetes API). Every API Key must list at least one OIDC group. The Inference Launchpad gateway does not evaluate OIDC groups; it authenticates inference callers with the bearer token only.

Groups accumulate down the scope hierarchy: every group listed on a higher-scoped key is automatically granted the same control-plane access on every lower-scoped key beneath it, on top of that lower-scoped key's own groups. A member of a System-scope key's group can therefore view any Tenant- or Project-scope key without being named on that key's oidcGroups, and a member of a Tenant-scope key's group can view any Project-scope key in that Tenant. Inheritance supplements the required own groups — it does not replace them.

The Create API Key dialog offers a picker of the OIDC groups PaletteAI already knows about at the current scope. If the group you want does not appear in the list, type its name directly into the field. On child scopes, the dialog also shows a read-only Inherited OIDC Groups section listing the groups already granted access by ancestor keys — because those callers already have control-plane access, you do not need to repeat those groups on the new key.

Expiration and Lifecycle States​

Each API Key has one of the following expiration settings, chosen when the key is created:

  • Never — The key does not expire on its own.
  • 7 / 30 / 60 / 90 days — Predefined windows counted from creation. The dialog shows the exact date each option resolves to.
  • Custom — A calendar date you pick.

Expiration is enforced by the controller. When the current time passes the key's expiresAt, the controller marks the key Expired and the gateway begins rejecting requests it authorizes. You can update the expiration on an existing key without recreating it.

Each key surfaces one of the following states on the Inference Quotas tab.

StateMeaning
ActiveThe key is valid and the gateway accepts requests it authorizes.
ExpiredThe current time is past expiresAt. The gateway rejects requests. The Copy button is disabled for expired keys.
RevokedThe key was administratively revoked. The gateway rejects requests.

Editable Fields​

You can edit an existing API Key's OIDC groups and expiration in place. The key's name and the Inference Quota it draws against are set at creation and cannot be changed. To rebind a key to a different quota or rename it, delete it and create a new key.

Request-Time Authentication​

The client passes the key value as a bearer token in the Authorization header when it calls the Inference Launchpad gateway. The exact request shape — endpoint URL, model routing, and response format — is documented in the PaletteAI Inference Launchpad documentation.

Enable Inference Launchpad on a Cluster​

The gateway itself is deployed by a Launchpad-enabled Profile Bundle. Any Compute Pool whose Profile Bundle carries the palette.ai/launchpad-enabled=true annotation runs the gateway on its spoke cluster and receives the InferenceQuota and InferenceAPIKey resources PaletteAI federates from the hub. Spectro Cloud's AI Launchpad Profile Bundle already carries this annotation. For custom bundles, refer to Profile Bundles.

Where Inference Launchpad Appears in the PaletteAI Console​

Inference Quotas and API Keys are managed from Settings at each scope, under the Model Management section, on the Inference Quotas tab.

ScopeNavigation
SystemSystem Settings → Model Management → Inference Quotas
TenantTenant Settings → Model Management → Inference Quotas
ProjectProject Settings → Model Management → Inference Quotas

At every scope, the tab lists the scope-total quota, the per-model quotas defined at that scope, and — at the System and Tenant scopes — a sub-tab that lists descendants (Tenants under System, Projects under Tenant) so you can review their quotas without leaving the page. The API Keys button on each quota opens a drawer listing the keys attached to that quota.

Permissions​

  • spectrocloud.com/inferencequotas:{create,update,delete} gate the corresponding actions on the Inference Quotas tab.
  • spectrocloud.com/inferenceapikeys:{create,update,delete,get} gate the API key actions on the API Keys drawer. The get permission is also required to copy a key's value.

If an action or button described in the documentation is not displayed, your role does not include the required permission. Contact your administrator to request access.

Next Steps​