Troubleshooting Profile Bundles

This page provides troubleshooting guidance for common issues when Profile Bundles fail to become ready, are rejected at creation, or block deployments.

info

Troubleshooting steps on this page use kubectl commands to inspect Kubernetes resources on the hub cluster. You need kubectl access to the hub cluster and the namespace where the Profile Bundle was created. If you do not have kubectl access, contact your platform administrator.

Profile Bundle Status Overview

The ProfileBundle reports its overall state in the status.status field: Ready or Failed. The status.conditions field explains which processing stage failed:

• DefinitionsReady - Component, trait, and policy definitions were processed.

• RepositoriesReady - Referenced Helm and OCI repositories were registered.

• ClusterProfilesReady - Cluster profiles were validated or imported into Palette.

• WorkloadProfilesReady - Workload profiles were processed.

• RevisionReady - A revision of the bundle content was published.

The status.latestRevision field records the most recently published revision. A Profile Bundle without a latestRevision has not finished publishing and cannot be used by deployments.

kubectl get profilebundle <name> --namespace <namespace>
kubectl get profilebundle <name> --namespace <namespace> --output json | jq '.status.conditions'
kubectl get profilebundle <name> --namespace <namespace> --output json | jq '.status.latestRevision'

Profile Bundle Failed

Symptom: The Profile Bundle status is Failed.

Possible causes:

1. A referenced definition, repository, cluster profile, or workload profile failed to process.

2. The Settings resource for the namespace has no Palette integration, so cluster profiles cannot be validated or imported.

3. A pack in an imported cluster profile is not available in the target Palette registry.

Resolution:

1. Check the conditions and read the message of each condition that is False. The message lists the failing resources and the underlying error.

   kubectl get profilebundle <name> --namespace <namespace> --output json | jq '.status.conditions'

2. Match the failure message against the following common errors.

   Message pattern	Meaning and fix
   no Palette integration configured in Settings	Cluster profile processing requires a Palette integration in the Settings resource for the namespace. Configure one, then re-trigger reconciliation by re-applying the bundle.
   pack '<name>': target registry '<registry>' not found in available registries	A pack in the cluster profile references a Palette registry that is not available. Verify the registry overrides on the bundle and the registries configured in Palette.
   ClusterProfile <name> version <version> already exists with different spec	A cluster profile with the same name and version but different content already exists in Palette. In-place updates are not allowed. Publish the change under a new version.
   failed to find definition revision	A referenced definition has not been published yet. The controller retries automatically. If the condition persists, verify the definition exists and is spelled correctly.
   failed to process <n> repository resources	A Helm or OCI repository failed to register. Verify the repository URL and credentials. Refer to Create Repositories.

3. If the failing stage is not clear from the conditions, review the Hue controller logs in the Helm release namespace, which defaults to mural-system.

   kubectl logs --namespace <release-namespace> --selector controller.spectrocloud.com/name=hue-controllers --tail=100

Profile Bundle Creation or Update Rejected

Symptom: Creating or updating a Profile Bundle fails immediately with an admission webhook error.

Cause: PaletteAI validates Profile Bundle structure, versioning, and content at admission time.

Resolution:

Match the error message against the following common rejections and correct the manifest.

Message pattern	Meaning and fix
ProfileBundle Type is immutable	The bundle type cannot change after creation. Create a new bundle with the desired type.
ProfileBundle revisions are immutable, cannot update spec without also updating the version	Bundles with semantic versioning cannot be edited in place. Bump spec.version along with the content change.
version must not be empty when using semantic versioning	Set spec.version to a semantic version such as 1.0.0.
version must be empty when using basic versioning	Remove spec.version, or switch the bundle to semantic versioning.
version <version> is not a valid semantic version	Use the major.minor.patch form, for example 1.2.0.
cannot mix WorkloadProfile types "<a>" and "<b>" in the same ProfileBundle	Application and model workload profiles cannot share a bundle. Split them into separate bundles.
already exists with different content. Expected hash: <x>, actual hash: <y>	Under basic versioning, a child resource with the same name must keep the same content. Rename the resource or use semantic versioning to publish changed content.
Each InfrastructureVariant must have exactly one ClusterProfile or clusterProfileUID	Provide exactly one of clusterProfile or clusterProfileUID per infrastructure variant.
Palette integration not found or invalid, cannot validate or create ClusterProfiles	The Settings resource for the namespace has no valid Palette integration. Configure one before creating bundles that include cluster profiles.

Deployment Cannot Use Profile Bundle

Symptom: An App Deployment, Model Deployment, or Compute Pool that references the Profile Bundle is rejected or fails validation.

Possible causes:

1. The bundle has not finished publishing, reported as ProfileBundle has no latest revision.

2. A referenced workload profile does not exist in the bundle, reported as WorkloadProfile '<name>' not found in ProfileBundle.

3. The bundle's inference engine does not match the target Compute Pool, reported as inference engine mismatch.

Resolution:

1. Verify the bundle is Ready and has a published revision.

   kubectl get profilebundle <name> --namespace <namespace>
   kubectl get profilebundle <name> --namespace <namespace> --output json | jq '.status.latestRevision'

   If latestRevision is empty, triage the bundle conditions first. Refer to Profile Bundle Failed.

2. If a workload profile is reported as not found, list the workload profiles recorded on the bundle status and correct the reference in the deployment or Compute Pool.

   kubectl get profilebundle <name> --namespace <namespace> --output json | jq '.status.workloadProfileRefs'

3. For inference engine mismatches, refer to Deployment Failed.

Built-in Profile Bundles Missing or Outdated

PaletteAI ships built-in Profile Bundle templates in the Helm release namespace, which defaults to mural-system. During reconciliation of the System, Tenant, and Project resources, the Hue controller clones every template from the release namespace into the corresponding system, tenant, and Project namespaces. The sync is clone-once: a Profile Bundle whose name already exists in the destination namespace is left untouched. This means local edits to a synced copy are never overwritten, and it also means changes to a template do not propagate to namespaces that already hold a copy.

Symptom: A built-in Profile Bundle is missing from a tenant or Project namespace, or an outdated copy remains after the template in the release namespace was updated.

Possible causes:

1. The clone failed during reconciliation. The failure is recorded on the owning resource's BuiltinBundleSyncReady condition with the reason BuiltinBundleSyncFailed and a message similar to failed to sync <n> System ProfileBundle(s). This condition is informational and does not block readiness, so the failure can go unnoticed.

2. The owning System, Tenant, or Project has not reconciled since the template was added to the release namespace.

3. The destination namespace already holds a Profile Bundle with the same name, so the updated template is not re-cloned.

Resolution:

1. Check the BuiltinBundleSyncReady condition on the owning resource and read its message for per-bundle clone errors.

   kubectl get system pai-system --output json | jq '.status.conditions[] | select(.type == "BuiltinBundleSyncReady")'
   kubectl get tenant <tenant-name> --output json | jq '.status.conditions[] | select(.type == "BuiltinBundleSyncReady")'
   kubectl get project <project-name> --namespace <project-namespace> --output json | jq '.status.conditions[] | select(.type == "BuiltinBundleSyncReady")'

2. Compare the templates in the release namespace with the destination namespace to identify missing bundles.

   kubectl get profilebundles --namespace <release-namespace>
   kubectl get profilebundles --namespace <destination-namespace>

3. To pick up an updated template, delete the stale copy in the destination namespace. The next reconciliation of the owning System, Tenant, or Project re-clones it from the release namespace.

   warning

   Deleting the copy discards any local edits made to it, and deletion is blocked while the bundle is referenced by an App Deployment, Model Deployment, or Compute Pool. Refer to Cannot Delete Profile Bundle.

Cannot Delete Profile Bundle

Symptom: Deleting the Profile Bundle is rejected, or the resource remains in a deleting state.

Cause: Deletion is blocked while any App Deployment, Model Deployment, or Compute Pool still references the bundle. The rejection message identifies the blocking resource, for example ProfileBundle used by AIWorkload <namespace>/<name> or ProfileBundle used by ComputePool <namespace>/<name>.

Resolution:

1. Find the resources that still reference the bundle and delete them first.

   kubectl get aiworkloads,computepools --namespace <namespace>

2. Delete the Profile Bundle again once no references remain. Refer to Delete Profile Bundles.

Resources

• Create Profile Bundles

• Import Profile Bundles

• Delete Profile Bundles

• Profile Bundles Concepts

• Troubleshooting App and Model Deployments