Skip to main content
Version: v1.1.x

Spoke Cluster Lifecycle

This page describes the day-2 spoke cluster lifecycle operations that PaletteAI supports after installation, and how to perform them.

Every spoke cluster is represented on the hub cluster by a Spoke resource in the fleetconfig.open-cluster-management.io/v1beta1 API group. The FleetConfig controller on the hub joins each spoke to the fleet using Open Cluster Management (OCM), and the Brush controller then installs the required PaletteAI components on the spoke. Lifecycle operations are driven by creating, updating, and deleting Spoke resources and their associated kubeconfig secrets on the hub cluster.

Supported Operations​

The following table summarizes which spoke lifecycle operations are supported after installation.

OperationSupportDetails
Add a spoke cluster after installationSupportedRegister additional spoke clusters through a Helm upgrade of the hub. Refer to Add a Spoke Cluster After Installation.
Remove or decommission a spoke clusterSupportedDelete the Spoke resource on the hub. Refer to Remove a Spoke Cluster.
Remove an unreachable spoke clusterSupportedForce removal through an annotation. Refer to Remove an Unreachable Spoke Cluster.
Replace or rotate spoke kubeconfig credentialsSupportedUpdate the kubeconfig secret on the hub. Refer to Replace or Rotate Spoke Kubeconfig Credentials.
Recovery of temporarily disconnected spokesAutomaticSpoke agents reconnect on their own. Refer to Disconnected Spoke Clusters.
Rename a spoke or change its registration configuration in placeNot supportedRefer to Unsupported Operations. Remove the spoke and add it again with the new configuration.
Move a spoke to a different hub in placeNot supportedRefer to Unsupported Operations.
info

This page applies to dedicated spoke clusters that you register with the hub yourself through Helm values. Spoke clusters that PaletteAI provisions through Compute Pools are registered and deregistered automatically as part of the Compute Pool lifecycle. Do not manage the Spoke resources of Compute Pools manually; instead, create and delete the Compute Pool itself. Refer to the Deletion Policy section for details on Compute Pool removal.

Add a Spoke Cluster After Installation​

Adding dedicated spoke clusters after installation is supported and does not require reinstalling PaletteAI. The registration mechanism is the same one used during installation: each entry in the fleetConfig.spokes array of the hub's Helm values becomes a Spoke resource on the hub cluster. The Spoke resources are applied by a job that runs after every Helm install and upgrade, so a Helm upgrade with a new spoke entry registers the new cluster.

To add a spoke cluster:

  1. Configure RBAC on the new spoke cluster so the hub's FleetConfig controller can bootstrap the OCM agent. Follow the platform-specific setup guide for your environment:

  2. Create a secret on the hub cluster, in the same namespace as your PaletteAI release, that contains the spoke cluster's kubeconfig. The kubeconfig must use long-lived credentials, not short-lived tokens.

    kubectl create secret generic spoke-1-kubeconfig \
      --from-file=kubeconfig=spoke-1.kubeconfig \
      --namespace mural-system

  3. Add a new entry to the fleetConfig.spokes array in your Helm values. Use the existing entries as a template. Set kubeconfig.inCluster to false, reference the secret you created, and set klusterlet.forceInternalEndpointLookup to false for remote clusters. Every spoke entry requires an ociRegistry section; copy it from your existing spoke entry unless the new spoke uses a different workload registry. Refer to the FleetConfig values guidance in the install guides for the full structure.

    fleetConfig:
      spokes:
        # Keep all existing entries unchanged.
        - name: spoke-1
          kubeconfig:
            context: ''
            inCluster: false
            secretReference:
              name: spoke-1-kubeconfig
              kubeconfigKey: kubeconfig
          klusterlet:
            mode: Default
            forceInternalEndpointLookup: false
          ociRegistry:
            # Copy the ociRegistry configuration from your existing spoke entry.

  4. Apply the updated values with a Helm upgrade of the mural release. For Flux-managed installations, update the values block of your HelmRelease manifest instead and apply it to the hub cluster.

  5. Verify that the spoke cluster joined the fleet. On the hub cluster, check the Spoke resource and its corresponding ManagedCluster.

    kubectl get spokes --namespace mural-system
    kubectl get managedclusters

    The Spoke resource reports its status in the PHASE column, and the new ManagedCluster is listed with JOINED=True and AVAILABLE=True. Once the spoke has joined, the Brush controller installs the PaletteAI spoke components automatically.

info

If the only viable network path is from spoke to hub, you can instead join the spoke cluster out-of-band with clusteradm join, passing --klusterlet-annotation brush-ignore=true so that PaletteAI does not attempt to install its components on the spoke, and then install the required components on the spoke cluster yourself. Refer to Register a Cluster in the OCM documentation for the join procedure.

Remove a Spoke Cluster​

Removing a spoke cluster is supported. Removal is driven by deleting the Spoke resource on the hub cluster, which uninstalls the PaletteAI components from the spoke, unjoins the spoke from the fleet, and cleans up the hub-side resources for that cluster.

To remove a spoke cluster:

  1. Remove or migrate the App Deployments and Model Deployments that target the spoke cluster. Spoke removal does not proceed while workloads that were created outside of a placement are still active on the cluster. Workloads scheduled through placements are descheduled automatically during removal when cleanupConfig.forceClusterDrain is enabled, which is the default.

  2. Delete the Spoke resource on the hub cluster.

    kubectl delete spoke <spoke-name> --namespace mural-system

    Deletion triggers the following sequence:

    • The Brush controller uninstalls the PaletteAI components from the spoke cluster.
    • The OCM agent on the spoke runs the unjoin process and removes the Klusterlet from the spoke cluster.
    • The hub removes the ManagedCluster, its cluster namespace, and related registration resources.

  3. Remove the corresponding entry from the fleetConfig.spokes array in your Helm values, then apply the change with a Helm upgrade. This step is required: the Spoke resources are re-applied from the Helm values after every upgrade, so a spoke that is deleted but still present in the values is registered again on the next upgrade.

  4. Verify that the spoke was removed.

    kubectl get spokes --namespace mural-system
    kubectl get managedclusters

    The deleted Spoke resource and its ManagedCluster are no longer listed.

The cleanupConfig section of each spoke entry controls which resources are garbage collected during removal.

FieldDefaultDescription
purgeKlusterletOperatortrueRemoves the Klusterlet operator and the OCM namespaces from the spoke cluster during unjoin.
purgeKubeconfigSecretfalseDeletes the spoke kubeconfig secret on the hub after the spoke agent has taken over management of the spoke.
purgeAgentNamespacefalseRemoves the agent namespace on the spoke cluster after unjoin.
forceClusterDraintrueDeschedules all placement-managed workloads from the spoke cluster during removal. Manually created workloads must be removed first.

Remove an Unreachable Spoke Cluster​

If a spoke cluster is permanently unreachable, for example because the cluster infrastructure was already destroyed, the normal removal flow cannot complete because the hub waits for the spoke to confirm the unjoin. For this case, PaletteAI supports a force-removal escape hatch through an annotation on the Spoke resource.

kubectl annotate spoke <spoke-name> --namespace mural-system \
  fleetconfig.open-cluster-management.io/force-delete=true
kubectl delete spoke <spoke-name> --namespace mural-system

With the annotation set, the hub cleans up its own resources for the spoke without waiting for the spoke cluster to respond. If the spoke cluster still exists, any PaletteAI and OCM components left on it must be removed manually.

warning

Only use force removal for spoke clusters that cannot be reached anymore. For reachable clusters, use the standard removal procedure so the spoke is cleaned up gracefully.

Replace or Rotate Spoke Kubeconfig Credentials​

Replacing or rotating the stored kubeconfig for a spoke cluster is supported. The hub stores each spoke's kubeconfig in a secret, referenced by spec.kubeconfig.secretReference on the Spoke resource, and reads the secret whenever it needs to act on the spoke cluster directly. Once a spoke has joined the fleet, routine operations run through the agent on the spoke itself; the stored kubeconfig is used during the initial join and as a fallback during removal, so keep the credentials valid for the entire life of the spoke.

To rotate the credentials, replace the contents of the kubeconfig secret in place. The secret lives in the same namespace as the Spoke resource, which is the PaletteAI release namespace by default.

kubectl create secret generic <secret-name> \
  --from-file=kubeconfig=<new-kubeconfig-file> \
  --namespace mural-system \
  --dry-run=client --output yaml | kubectl apply --filename -

The new kubeconfig must reference the same spoke cluster and must use long-lived credentials. You can also point the spoke at a different secret by updating spec.kubeconfig.secretReference on the Spoke resource, or the matching fleetConfig.spokes[].kubeconfig.secretReference Helm value; spec.kubeconfig is one of the few Spoke fields that may be changed after creation.

Disconnected Spoke Clusters​

PaletteAI uses a pull-based model for hub-spoke communication, so a temporary loss of connectivity between a spoke and the hub does not require manual recovery. While a spoke cluster is disconnected:

  • Workloads on the spoke continue to run. Flux operates locally on the spoke and keeps managing the deployed applications.

  • Configuration changes made on the hub are not delivered to the spoke, and status updates stop flowing back. The spoke's ManagedCluster is no longer reported as available on the hub, and hub-side status, such as Compute Pool health, can show as Unknown.

When connectivity is restored, the OCM agents on the spoke reconnect to the hub, pull any pending configuration changes, and resume status reporting. No re-registration is required.

If a spoke cluster is lost permanently, remove it from the fleet using the Remove an Unreachable Spoke Cluster procedure, then add a replacement spoke if needed.

Unsupported Operations​

The following operations are not supported. Where an in-place change is not supported, remove the spoke and add it again with the new configuration.

  • Renaming a spoke or changing its registration configuration in place. After a Spoke resource is created, only a narrow set of fields may be updated: spec.kubeconfig, spec.addOns, spec.cleanupConfig, spec.timeout, spec.logVerbosity, and the annotations, values, and valuesFrom fields under spec.klusterlet. Changes to any other field, including the spoke name, hubRef, the Klusterlet mode, and proxy settings, are rejected by validation.

  • Moving a spoke cluster to a different hub in place. The hubRef of a Spoke resource cannot be changed. Remove the spoke from its current hub, then register it with the new hub.

  • Re-registering a joined spoke in place. There is no supported procedure for resetting or re-joining a spoke that is already registered. If a spoke's registration is broken beyond the credential rotation described on this page, remove the spoke and add it again.