Version: v0.2.x

Deploy Profile Bundles in Air-Gapped Environments

To deploy a ProfileBundle in an air-gapped PaletteAI environment, download the bundle archive from Artifact Studio and import it into PaletteAI. Next, mirror the required Helm chart and container images to the internal Zot registry. You must also update the project configuration to use the internal registry. Finally, copy the Zot CA to the workload namespace in the Compute Pool cluster.

Not every Palette pack is published in Artifact Studio. If all packs your profiles depend on are available there, follow the Artifact Studio download and upload steps in this guide. If you require a pack that is not available in Artifact Studio, obtain a Pack tarball from your Spectro Cloud representative and use paletteai mirror export-pack (connected phase) plus paletteai mirror push (air-gap phase) to transfer that pack archive and its chart and image dependencies into your air-gapped OCI registry before you upload and sync pack content in Palette. Command behavior and OCI paths are documented in the PaletteAI CLI reference.

For background information about Profile Bundles and general bundle lifecycle tasks, refer to the Profile Bundles concept page and the Create and Manage Profile Bundles guide.

info

If you deploy through the PaletteAI UI by using the App Deployment or Model Deployment workflows, complete the hub preparation, chart and image mirroring, Project admission update, and spoke-side CA copy steps in this guide before you start the deployment. During the deployment flow, you are prompted to provide values for the variables that the profile bundle requires, and the deployment stores those overrides in spec.workloadDeploymentConfigs[].variables. Patching Patch mural-variables for an Air-Gapped Environment in advance pre-populates the shared internal Zot registry defaults so you do not have to re-enter them for every deployment.

For an end-to-end guide covering appliance setup through deployment, including self-hosted Palette setup, refer to the Self-Hosted Quickstart.

Prerequisites

Access to the hub cluster and spoke cluster kubeconfigs
The paletteai CLI installed on your local machine
Access to Artifact Studio to download packs and bundles
An internal Zot registry deployed and reachable from your environment

Deploy the Profile Bundle

Download the Bundle Archive

Download the ProfileBundle archive from Artifact Studio.
Save the archive locally as <bundle>-<version>.tar.gz.
Extract the archive to a working directory.
```
mkdir --parents ./<bundle>-<version>
tar --extract --gzip --file ./<bundle>-<version>.tar.gz --directory ./<bundle>-<version>
```
The extracted directory contains the manifests and supporting files that you use in the following steps.

Prepare Palette Pack Content

Before you import a bundle that includes cluster profiles, ensure the required Palette packs are available in the appliance's internal pack registry. Because Palette cannot determine the required packs until those packs are already available, identify the required packs from the extracted bundle before you attempt the import.

This pack-upload workflow is separate from the later paletteai mirror step, which mirrors workload Helm charts and container images to the internal Zot registry.

paletteai mirror export-pack and paletteai mirror push are used when you start from a Pack tarball (for example from Spectro Cloud) to mirror that pack and its dependencies into the registry. They are not a substitute for the Artifact Studio pack-bundle download when your packs are all listed there.

Locate the ClusterProfile manifests in the extracted bundle directory.

find ./<bundle>-<version> -type f \( -name '*.yaml' -o -name '*.yml' \) \
  -exec grep --line-number --with-filename "kind: ClusterProfile" {} +

Open each matching ClusterProfile file and identify the required pack names and versions from the packs: array.
```
grep --line-number --context 20 "packs:" <clusterprofile-file>
```
Download the required packs from Artifact Studio.
In Artifact Studio, select Create pack bundle and then Browse Packs.
Use the filters for Product, Version, Cloud type, Layer type, architecture, and FIPS mode to narrow the list to the packs and versions that you identified in the extracted ClusterProfile manifests.
For each required pack, use Select option to choose the correct version. Confirm that each selection appears under Selected items.
Select Download Bundle to download the selected packs, or select Copy all URLs if you want to download them with curl.

Each selected pack is downloaded as <pack-name>.zst with a matching <pack-name>.sig.bin signature file.

The following pack set is required for the ClearML example workflow described in the self-hosted quickstart. Federal Information Processing Standards (FIPS) variants are listed where applicable.
- Edge Native BYOI 2.1.0 (FIPS)
- Edge K8s (PXKe) 1.33.5 (FIPS)
- Cilium 1.17.4 (FIPS)
- Local Path Provisioner 0.0.32 (FIPS)
- Registry Connect 0.1.0
- MetalLB 0.15.2
Log in to Palette Local UI.
In Palette Local UI, go to Content and select Actions > Upload Content.
Upload the downloaded pack archives.
Wait for the uploaded content to sync into the internal Zot registry.
Confirm the uploaded packs appear in Local UI.
Log in to the Palette system console.
Go to Administration > Pack Registries and run a sync for the OCI pack registry.
If using PaletteAI VerteX and your deployment requires a non-FIPS pack such as MetalLB, ensure the tenant setting to allow non-FIPS packs is enabled before continuing.

For more information about downloading packs, refer to the Artifact Studio documentation.

If your imported bundle contains Workload Profiles only, you can skip this section.

PaletteAI UI Workflow

If you are deploying the imported bundle through the PaletteAI UI, the high-level workflow is:

Import the bundle into the hub cluster.
Mirror the required chart and images to the internal Zot registry.
Update project admission to allow the target workload namespace.
Copy the zot-ca Secret into the workload namespace on the spoke cluster.
Start the App Deployment or Model Deployment from the PaletteAI UI.

In this workflow, you are prompted for the variables that the profile bundle requires, and the deployment stores those overrides in spec.workloadDeploymentConfigs[].variables. If you patch the Project VariableSet in advance, the shared internal Zot registry values are already defaulted when you create the deployment.

Import the Profile Bundle

Set KUBECONFIG to the hub cluster kubeconfig.

Import the bundle archive into the target project namespace.

export KUBECONFIG=<hub-kubeconfig>
paletteai studio import --namespace <project-namespace> <bundle>-<version>.tar.gz

If your air-gapped environment requires registry UID remapping during import, use the --registry-map flag described in the PaletteAI CLI reference.

Validate the Import

In PaletteAI, go to Profile Bundles.
Verify that the imported Profile Bundle appears in the list and reaches a ready state.

Review the Palette Cluster Profiles

If the imported bundle includes cluster profiles, verify that Palette imported them into the Palette project specified in your project's Settings resource.

In Palette, open the project associated with the Palette integration that your PaletteAI project uses.
Go to Profiles > Cluster Profiles.
Confirm that the expected Infrastructure, Full, or Add-on cluster profiles from the bundle appear in the list.
Verify that the imported cluster profiles show the expected version and are available for Compute Pool workflows.

If the bundle contains Workload Profiles only, you can skip this section.

Review the `WorkloadProfile`

Before you mirror content or update mural-variables, review the WorkloadProfile in the extracted bundle directory and record the values that you need in the remaining steps.

Locate the WorkloadProfile manifest in the bundle directory.

find ./<bundle>-<version> -type f \( -name '*.yaml' -o -name '*.yml' \) \
  -exec grep --line-number --with-filename "kind: WorkloadProfile" {} +

Open the matching file and identify the chart name, chart version, and the variable names that the profile uses.
```
grep --line-number --extended-regexp 'chart|repository|repo|version|variable' <workloadprofile-file>
```

Record the following values from the WorkloadProfile before you continue:

The chart name.
The chart version.
The variable names that the profile uses for the workload namespace, the internal chart URL, and the internal registry settings.

Air-gapped bundles may already point the chart source at the internal registry, so the extracted WorkloadProfile might not include the original upstream Helm repository URL. If it does not, retrieve it from the bundle source metadata or the original bundle definition used to build the air-gapped version.

Use those values to replace the placeholders and example variable names in the remaining steps.

Mirror the Helm Chart and Images to Zot

Before you deploy workloads from the bundle, mirror the required chart and images to the internal Zot registry. The registry must contain the content referenced by the ClusterProfile and WorkloadProfile resources in the bundle.

Verify that the paletteai CLI is installed and available in your PATH. For installation instructions, refer to the PaletteAI CLI reference.
Choose a local alias for the upstream Helm repository. Use the same alias for both <repo-name> in --chart-url and <repo-url> in the following commands.

Set shell variables for the Zot admin credentials before running the mirroring command.

zot_admin_username=$(kubectl --namespace zot-system get secret zot-secret --output jsonpath='{.data.registryUsername}' | base64 --decode)
zot_admin_password=$(kubectl --namespace zot-system get secret zot-secret --output jsonpath='{.data.registryPassword}' | base64 --decode)

Run paletteai mirror sync to mirror the chart and referenced images to the internal Zot registry. Use the chart name and chart version that you recorded from the WorkloadProfile.
```
paletteai mirror sync \
  --chart-url <repo-name>/<chart-name> \
  --chart-version <chart-version> \
  --release-name <chart-name> \
  --repo-url <repo-url> \
  --dest-registry <VIP>:<zot-nodeport>/spectro-content \
  --dest-username "$zot_admin_username" \
  --dest-password "$zot_admin_password" \
  --dest-insecure
```
If the machine you are using cannot reach both the upstream chart repository and the internal Zot registry at the same time, use the staged paletteai mirror export and paletteai mirror push workflow instead of paletteai mirror sync.

For example, the ClearML chart uses this repository and can be mirrored with the following command:

paletteai mirror sync \
  --chart-url clearml/clearml \
  --chart-version 7.14.7 \
  --release-name clearml \
  --repo-url https://clearml.github.io/clearml-helm-charts \
  --dest-registry <VIP>:<zot-nodeport>/spectro-content \
  --dest-username "$zot_admin_username" \
  --dest-password "$zot_admin_password" \
  --dest-insecure

Validate the Mirrored Content

Verify that Zot lists the mirrored repositories.

curl --silent --insecure --user "$zot_admin_username":"$zot_admin_password" https://<VIP>:<zot-nodeport>/v2/_catalog | jq

Example Output
{
  "repositories": [
    "spectro-content/charts/<chart-name>",
    "spectro-content/images/<image-name>"
  ]
}

Confirm that the expected chart version is present for the mirrored OCI chart.

curl --silent --insecure --user "$zot_admin_username":"$zot_admin_password" \
  https://<VIP>:<zot-nodeport>/v2/spectro-content/charts/<chart-name>/<chart-name>/tags/list | jq

Example Output
{
  "name": "spectro-content/charts/<chart-name>",
  "tags": ["<chart-version>"]
}

Patch Project Admission

By default, project admission allows workload deployments only in approved namespaces. Add the required workload namespace to the project admission allow list.

You can update project admission in the PaletteAI UI. The following CLI example shows the equivalent workflow.

kubectl patch project <project-name> --type=json \
  --patch '[{"op": "add", "path": "/spec/admissionConfiguration/workloadConfiguration/deploymentNamespaces/allow/-", "value": "<workload-namespace>"}]'

For example, if the workload deploys to the clearml namespace and you use the default project, replace the placeholders with default and clearml.

Validate Project Admission

The expected output includes "<workload-namespace>".

kubectl get project <project-name> --output jsonpath='{.spec.admissionConfiguration.workloadConfiguration.deploymentNamespaces.allow}'

Example Output
["<workload-namespace>"]

Patch `mural-variables` for an Air-Gapped Environment

This section is optional. Patch mural-variables if you want shared Zot defaults for repeated deployments, including App Deployment and Model Deployment workflows in the PaletteAI UI.

info

Most profile bundles already define their bundle-specific variables in an inline VariableSet, or you provide them during the deployment flow as deployment-time overrides. Use this section only as a convenience so you do not have to re-enter the common Zot-related values every time you deploy a bundle.

This section applies when you want project-scoped defaults for the internal registry settings, whether you deploy through manual manifests or through the PaletteAI UI by using the App Deployment or Model Deployment workflows. Deployment-specific values in spec.workloadDeploymentConfigs[].variables still take precedence over project-scoped mural-variables, but patching the Project VariableSet in advance lets those shared Zot values appear as defaults across deployments.

When you patch the Project-scoped VariableSet in advance, every deployment that reads from it gets the same shared Zot defaults without requiring you to enter them repeatedly. In the PaletteAI UI, those values are pre-populated at deploy time, while manual workflows can read the same defaults directly from the Project VariableSet. This section covers only the shared Zot variables: zot-cert-secret, zot-auth-secret, and internal-zot-url.

You can update these values in the PaletteAI UI. The following CLI example shows the equivalent workflow.

warning

The following patch replaces the variables array in mural-variables. Review the existing VariableSet before you apply the patch.

Patch only the common Zot-related variables shown below.

kubectl patch variableset mural-variables --namespace <project-namespace> --type='merge' --patch '{
  "spec": {
    "variables": [
      {
        "name": "zot-cert-secret",
        "variableType": "string",
        "value": {"string": "zot-ca"}
      },
      {
        "name": "zot-auth-secret",
        "variableType": "string",
        "value": {"string": "<vip-dashed>-<zot-nodeport>-pull-secret"}
      },
      {
        "name": "internal-zot-url",
        "variableType": "string",
        "value": {"string": "<VIP>:<zot-nodeport>"}
      }
    ]
  }
}'

Replace <VIP> with the internal Zot VIP assigned to your environment. Replace <vip-dashed> with the same VIP, but replace each . with - and append -<zot-nodeport>-pull-secret for the zot-auth-secret value. For example, 10.10.233.0:30003 becomes 10-10-233-0-30003-pull-secret.

If a bundle already defines all of its required variables inline, no additional bundle-specific entries are needed in mural-variables. Add extra entries only when you want reusable Project defaults across repeated UI or manual deployments.

Validate the `VariableSet`

Run the following command to confirm the shared Zot defaults are present.

kubectl get variableset mural-variables --namespace <project-namespace> --output json \
  | jq '.spec.variables[] | select(.name=="zot-cert-secret" or .name=="zot-auth-secret" or .name=="internal-zot-url")'

Example Output
{
  "name": "zot-cert-secret",
  "variableType": "string",
  "value": {
    "string": "zot-ca"
  }
}
{
  "name": "zot-auth-secret",
  "variableType": "string",
  "value": {
    "string": "<vip-dashed>-<zot-nodeport>-pull-secret"
  }
}
{
  "name": "internal-zot-url",
  "variableType": "string",
  "value": {
    "string": "<VIP>:<zot-nodeport>"
  }
}

Copy the Zot CA to the Workload Namespace

The internal Zot CA must exist in the workload namespace on the Compute Pool cluster so workloads can trust the internal registry. You can create the Compute Pool as part of the same App Deployment or Model Deployment flow. However, the source CA Secret is created only after the Compute Pool cluster comes up and a registry CA Secret appears in the mural-system namespace on the spoke cluster. This step is manual because it requires access to the spoke cluster kubeconfig.

Set KUBECONFIG to the spoke cluster kubeconfig.
If this workflow created a new Compute Pool, wait for the registry CA Secret to appear in the mural-system namespace. Depending on how the hub registry was provisioned, the Secret may be named zot-ca or oci-tls-hub.
Create the workload namespace.

Copy the CA certificate from mural-system and recreate it as zot-ca in the workload namespace. The destination Secret must always be named zot-ca, even when the source Secret is oci-tls-hub.

export KUBECONFIG=<spoke-kubeconfig>
until kubectl get secret zot-ca --namespace mural-system >/dev/null 2>&1 \
  || kubectl get secret oci-tls-hub --namespace mural-system >/dev/null 2>&1; do
  sleep 5
done

kubectl create namespace <workload-namespace>

if kubectl get secret zot-ca --namespace mural-system >/dev/null 2>&1; then
  kubectl get secret zot-ca --namespace mural-system --output json \
    | jq '{apiVersion, kind, type, data, metadata: {name: .metadata.name, namespace: "<workload-namespace>"}}' \
    | kubectl apply --filename -
else
  kubectl get secret oci-tls-hub --namespace mural-system --output jsonpath='{.data.ca\.crt}' \
    | base64 --decode > /tmp/ca.crt

  kubectl create secret generic zot-ca --namespace <workload-namespace> \
    --from-file=ca.crt=/tmp/ca.crt \
    --dry-run=client --output yaml \
    | kubectl apply --filename -
fi

Validate the Zot CA Secret

Run the following command to confirm that the zot-ca Secret exists in the workload namespace.

kubectl get secret zot-ca --namespace <workload-namespace>

Example Output
NAME     TYPE     DATA   AGE
zot-ca   Opaque   1      1m

If neither zot-ca nor oci-tls-hub is present in mural-system yet, wait for the Compute Pool to finish provisioning and rerun the copy command. If the workload deployment already started before you copied the Secret, copy the Secret first and then retry the workload deployment.

Next Steps

Deploy or update the workload that uses the imported ProfileBundle.
Review the PaletteAI CLI reference for additional import options, such as --registry-map.
Return to the Create and Manage Profile Bundles guide for general bundle lifecycle tasks.

Prerequisites​

Deploy the Profile Bundle​

Download the Bundle Archive​

Prepare Palette Pack Content​

PaletteAI UI Workflow​

Import the Profile Bundle​

Validate the Import​

Review the Palette Cluster Profiles​

Review the WorkloadProfile​

Mirror the Helm Chart and Images to Zot​

Validate the Mirrored Content​

Patch Project Admission​

Validate Project Admission​

Patch mural-variables for an Air-Gapped Environment​

Validate the VariableSet​

Copy the Zot CA to the Workload Namespace​

Validate the Zot CA Secret​

Next Steps​