4 docs tagged with "inference launchpad"

This guide describes how to configure model pricing at the System scope. Model pricing is the install-wide catalog of per-model rates (USD per 1,000,000 tokens) that the Inference Launchpad gateway uses for cost accounting and that the Cost dimension of Inference Quotas is measured against. PaletteAI federates the catalog to every Launchpad-enabled spoke; each served model that appears in the catalog gets an explicit rate, and every other served model falls back to the Default rate. Without a Default and an explicit price for a model, that model's requests do not contribute to the Cost dimension of any quota.

This guide describes how to deploy an Inference Launchpad cluster from PaletteAI. An Inference Launchpad cluster is a standard Compute Pool whose Profile Bundle opts the spoke in to PaletteAI Inference Launchpad by carrying the palette.ai/launchpad-enabled=true annotation. Once you have such a Compute Pool, deploying a vLLM-backed Model Deployment to it is the same workflow as any other Model Deployment — the Inference Launchpad gateway on the spoke automatically proxies the model.

PaletteAI Inference Launchpad is Spectro Cloud's inference gateway. It runs on a Kubernetes cluster and serves models to callers over an OpenAI-compatible API, applying authentication and usage metering on every request.

This section describes how to manage PaletteAI Inference Launchpad from PaletteAI. Inference Launchpad can also run standalone; the tasks in this section apply only when you use PaletteAI as its central control plane. When you do, PaletteAI is the single place where clusters are provisioned, usage budgets are defined, credentials are issued, and model prices are catalogued, and the hub federates each of those to every Launchpad-enabled Compute Pool.