Skip to main content
Version: v1.2.x

Configure Settings

Settings define integrations and configuration values used by Projects and Compute Pools. For integration types, prerequisites, and examples, refer to Settings and Integrations.

Overview​

You can configure Settings in PaletteAI at three levels:

  • System-level Settings: Settings that you create in the pai-system namespace for system-wide resources. The System resource references a System-level Settings via spec.settingsRef. System-level Settings can share Hugging Face and NVIDIA NGC integrations with selected Tenant and Project combinations.

  • Tenant-level Settings: Settings that you create in the Tenant namespace for Tenant-scoped resources. Tenant-level Settings can share Hugging Face and NVIDIA NGC integrations with selected Projects.

  • Project-level Settings: Settings that you create in a Project namespace. Each Project can have its own Settings that define integrations specific to that Project.

System, Tenant, and Project Settings​

The System, each Tenant, and each Project maintain their own Settings resource, referenced through the settingsRef field on the corresponding System, Tenant, and Project resource. Manage the System Settings Ref from the System Settings page and the Tenant Settings Ref from the Tenant Settings page — each on its Settings Ref tab.

Settings Reference​

The settingsRef field points to the Settings resource for each scope:

  • System — System.spec.settingsRef points to a Settings resource by name in the pai-system namespace. The System-level Settings supplies integrations for system-wide operations and can share integrations with Tenant and Project combinations.

  • Tenant — Tenant.spec.settingsRef points to a Settings resource by name and namespace in the Tenant namespace (tenant-<tenant-name>). It supplies Palette credentials for Tenant-scoped Compute Pools and other Tenant-level operations, and can share Hugging Face and NVIDIA NGC integrations with selected Projects. Projects do not inherit the Tenant Settings Ref — each Project must define its own settingsRef and Palette integration.

  • Project — Project.spec.settingsRef is a name-only reference. Because it has no namespace field, it always resolves to a Settings resource with that name in the Project namespace. It cannot directly target a Settings resource in another namespace, such as the Tenant or System namespace.

Integration Scope​

Hugging Face and NVIDIA NGC integrations cascade System → Tenant → Project: each tier shares its integrations downward through the sharedWith map, a locked integration forces the higher scope's value onto the scopes it is shared with, and otherwise the most-specific scope that defines an integration wins. Palette integrations never cascade — each scope defines its own. The controller writes the resolved configuration to read-only status.effectiveSettings.

For the full cascade, locking, and sharedWith model, refer to Settings and Integrations.

Manage System Settings​

Use the Settings Ref tab on the System Settings page to:

  • Create new System-level Settings and set the System Settings Ref on the System resource.

  • Share Hugging Face and NVIDIA NGC integrations with selected Tenants and Projects.

  • Edit or clone existing Settings.

For the full sharedWith policy language — selector globs, exclusions, and the nil vs {} distinction — refer to Sharing Resources.

Manage Tenant Settings​

Use the Settings Ref tab on the Tenant Settings page to:

  • Create new Tenant-level Settings and set the default Settings reference for the Tenant.

  • Set the Tenant Settings Ref on the Tenant resource.

  • Share Hugging Face and NVIDIA NGC integrations with selected Projects.

  • View and manage Settings at both the Tenant and Project levels.

  • Edit or clone existing Settings.

The Settings Ref tab displays Settings at both the Tenant and Project levels, showing which integrations are in use across your organization.

Integration Secrets​

Each integration in a Settings resource references one or more Kubernetes secrets. PaletteAI enforces strict ownership rules for these secrets to ensure safe lifecycle management.

Secret Namespace​

All integration secrets must reside in the same namespace as their Settings resource. For example, a Settings resource in the project-docs namespace must reference secrets that also exist in project-docs. This applies to all integration types:

  • Palette: spec.integrations.palette.namespace must match the Settings namespace.

  • Hugging Face: The API key secret must be in the Settings namespace.

  • NVIDIA: The NGC API key secret and image pull secret must be in the Settings namespace.

info

When Hugging Face or NVIDIA integrations are shared or merged across scopes, secrets referenced by a System-level Settings resource live in the System namespace (pai-system), secrets referenced by a Tenant-level Settings resource live in the Tenant namespace (tenant-<tenant-name>), and secrets referenced by a Project-level Settings resource live in the Project namespace. Palette integration secrets are always scoped to the System, Tenant, or Project that defines them. Each Settings resource is responsible only for secrets in its own namespace.

Secret Ownership​

Within a namespace, each integration secret is expected to be referenced by only one Settings resource. Duplicate integration secrets (where multiple Settings resources reference the same Palette integration secret or identical contents) are detected during reconciliation and reported through the PaletteIntegrationUnique status condition. PaletteAI resolves the conflict oldest-wins (tenant-scoped Settings are preferred): the winning Settings stays healthy, while each conflicting Settings is marked PaletteIntegrationDuplicated and does not become Ready until the conflict is resolved. Reconciliation may issue nudges to prompt re-evaluation of affected peers.

Automatic Secret Deletion​

When you delete a Settings resource, PaletteAI automatically deletes integration secrets it references, provided no other Settings resource in the same namespace still references them. The controller preserves shared integration secrets while other Settings peers still reference them. Only orphaned secrets (no longer referenced by any Settings) are deleted. If any secret cannot be removed, the Settings resource is retained and the deletion is blocked.

warning

Deleting a Settings resource permanently deletes its integration secrets. Before deleting, ensure you no longer need the credentials stored in those secrets, or copy them elsewhere.

Deletion Protection​

A Settings resource cannot be deleted while a non-deleting Project or Tenant still references it. If you attempt to delete one, PaletteAI returns a validation error that identifies the referencing Project or Tenant. Remove or update the reference first, then delete the Settings resource.

Palette Integration Status Conditions​

PaletteAI reports the PaletteIntegrationUnique condition on every Settings resource with a Palette integration. The condition uses one of these reasons:

PaletteIntegrationDuplicated​

Another Settings is using the same Palette project or tenant. The condition message specifies the conflicting dimension:

  • secret reference — Duplicate secret reference within the namespace.
  • secret contents — Duplicate encoded credentials within the namespace.
  • projectUID — Another Settings within the same PaletteAI tenant uses the same Palette project (hostUrl + defaultProjectID). The message names the conflicting Settings.
  • tenantUID — Another PaletteAI tenant uses the same Palette Tenant. The message names the conflicting Settings.

Example message: Palette integration duplicated by tenant-primary/primary-settings (projectUID)

PaletteTenantMismatch​

The project's resolved Palette tenant UID does not match its parent PaletteAI tenant's resolved Palette tenant UID. This means the Palette credentials belong to a different Palette Tenant than the parent tenant expects.

Example message: Palette tenant uid-abc123 does not match PaletteAI tenant Settings tenant-primary/primary-settings (Palette tenant uid-xyz789)

Resolution: Update the project's Palette secret to use credentials from the same Palette Tenant as the parent tenant.

FailedToVerifyUniqueness​

The Palette integration identity is incomplete — projectUID or tenantUID (or both) could not be resolved. Uniqueness and parity validation cannot proceed until the identity is available.

Example message: Palette integration is invalid or unresolved: missing tenantUID on resolved status

Resolution: Wait for the next reconciliation cycle. If the error persists, verify Palette API connectivity and credential validity.

Check Palette Identity​

To view the resolved Palette identity:

kubectl get settings <name> -n <namespace> -o jsonpath='{.status.paletteStatus}'

When to Use System-Level, Tenant-Level, or Project-Level Settings​

Consider using System-level Settings when:

  • You need centralized Hugging Face or NVIDIA NGC integrations shared across multiple Tenants and Projects.

  • You want to enforce locked integrations that override Tenant and Project settings for compliance or consistency.

  • You need system-wide integration management with optional per-Tenant and per-Project lockdown.

Consider using Tenant-level Settings when:

  • You need Palette credentials for Tenant-scoped Compute Pools or other Tenant-level operations.

  • You want to share Hugging Face or NVIDIA NGC integrations across multiple Projects within a Tenant.

  • You need centralized management of model-registry API keys with optional per-Project lockdown.

Consider using Project-level Settings when:

  • You create a new Project. Every Project requires its own Settings resource and Palette integration.

  • Projects need isolated Palette credentials for security, compliance, or separate Palette tenants.

  • Different teams manage different Projects independently.

Next Steps​

After you configure Settings, refer to the following resources for related Project administration tasks: