Set Up EKS Environment

To successfully deploy PaletteAI on EKS, specific resources must be created in the AWS accounts where your hub and spoke EKS clusters are located. Additionally, Kubernetes RBAC rules must be configured on your spoke EKS cluster. This guide provides step-by-step instructions for setting up everything required to deploy PaletteAI on an EKS cluster using shell scripts. These scripts enable your spoke clusters to connect to the hub using IAM Roles for Service Accounts (IRSA). The scripts perform the following steps:

• Configure the OIDC provider for both the hub and spoke EKS clusters.
• Create the <HUB_CLUSTER_NAME>_managed-cluster-identity-creator role in the hub AWS account.
• Create the FleetConfigHubIrsa IAM role in the hub AWS account.
• Create the ocm-managed-cluster-<CLUSTER_HASH> IAM role in the spoke AWS account.
• Create the FleetConfigJoin IAM role in the spoke AWS account.
• Set up the mural-system:fleetconfig-join ClusterRole and mural-system:fleetconfig-join-binding ClusterRoleBinding in the spoke EKS cluster.
• Update the aws-auth ConfigMap in the spoke EKS clusters to associate the FleetConfigJoin IAM role with the mural-system:fleetconfig-join user group.

Set Up Hub and Spoke Clusters

Prerequisites

• At least one active EKS cluster. With a single cluster, you can install PaletteAI using the hub-as-spoke pattern; with multiple clusters, you can use the dedicated spoke pattern.

• The following binaries installed on your system:

  • aws CLI - Follow the AWS CLI setup guide to configure the binary for accessing your AWS accounts.

  • eksctl

  • kubectl

  • yq

  • md5sum

  • A text editor of your choice (e.g., vi, nano, etc.)

Enablement

1. Download the scripts required for this setup to an eks-scripts directory by issuing the following commands.

   mkdir -p eks-scripts
   curl --output eks-scripts/env-vars  https://docs.palette-ai.com/resources/assets/hosted/scripts/eks/env-vars
   curl --output eks-scripts/common.sh  https://docs.palette-ai.com/resources/assets/hosted/scripts/eks/common.sh
   curl --output eks-scripts/hub.sh  https://docs.palette-ai.com/resources/assets/hosted/scripts/eks/hub.sh
   curl --output eks-scripts/spoke.sh  https://docs.palette-ai.com/resources/assets/hosted/scripts/eks/spoke.sh

2. Open /eks-scripts/env-vars in a text editor of your choice and configure all environment variables required for the hub.sh script. The following table explains each environment variable and its intended use, as well as which scripts require which variables.

   Variable	Description	Example Value	hub.sh	spoke.sh
   HUB_ACCOUNT_ID	The AWS account ID of the hub EKS cluster.	123456789012	✅	✅
   HUB_CLUSTER_NAME	The name of the hub EKS cluster.	hub	✅	✅
   HUB_REGION	The AWS region that the hub EKS cluster is in.	us-east-1	✅	❌
   SPOKE_ACCOUNT_ID	The AWS account ID of the spoke EKS cluster.	987654321098	❌	✅
   SPOKE_CLUSTER_NAME	The name of the spoke EKS cluster.	spoke	❌	✅
   SPOKE_REGION	The AWS region that the spoke EKS cluster is in.	us-west-2	❌	✅
   SPOKE_KUBECONFIG	The path to the spoke EKS cluster's kubeconfig file.	/path/to/spoke.kubeconfig	❌	✅
   SPOKE_ACCOUNT_ID_LIST	A comma separated list of AWS account IDs for all spoke EKS clusters.	"123456789012","987654321098"	✅	❌
   MURAL_NAMESPACE	The namespace that PaletteAI will be installed in. Default is mural-system.	mural-system	✅	❌

   export HUB_ACCOUNT_ID=<hub-account-id>
   export HUB_CLUSTER_NAME=<hub-cluster-name>
   export HUB_REGION=<hub-region>
   export SPOKE_ACCOUNT_ID=<spoke-account-id>
   export SPOKE_CLUSTER_NAME=<spoke-cluster-name>
   export SPOKE_REGION=<spoke-region>
   export SPOKE_KUBECONFIG=<spoke-kubeconfig>
   export SPOKE_ACCOUNT_ID_LIST=<spoke-account-id-list>
   export MURAL_NAMESPACE=<mural-namespace>

   info

   To configure a single EKS cluster using the hub-as-spoke pattern, use the same EKS cluster details for both the hub and the spoke environment variables.

   Example environment variables for hub-as-spoke pattern

   # Both `spoke.sh` and `hub.sh` require these env vars.
   # Typically they don't ever need to be modified unless you're configuring a new hub cluster.
   export HUB_ACCOUNT_ID=123456789012
   export HUB_CLUSTER_NAME=hub-as-spoke
   
   # Only `spoke.sh` requires these env vars.
   # These env vars should be reset for each spoke you'd like to join the hub.
   export SPOKE_ACCOUNT_ID=123456789012
   export SPOKE_CLUSTER_NAME=hub-as-spoke
   export SPOKE_REGION=us-west-2
   export SPOKE_KUBECONFIG=/path/to/hub-as-spoke.kubeconfig
   
   # Only `hub.sh` requires these env vars.
   # If new spokes are added to new AWS accounts, append the new account IDs to the `SPOKE_ACCOUNT_ID_LIST` env var and re-run `./hub.sh`
   export HUB_REGION=us-west-2
   export SPOKE_ACCOUNT_ID_LIST="123456789012"
   export MURAL_NAMESPACE=mural-system

3. After validating that all environment variables required for hub.sh have been configured, open a shell session with access to the hub AWS account and execute the below command.

   ./eks-scripts/hub.sh

   Example output

   [hub.sh] ✅ FleetConfig Hub EKS setup complete for <HUB_CLUSTER_NAME>.

4. After validating that all environment variables required for spoke.sh have been configured, open a new shell session with access to the spoke AWS account and issue the command below.

   ./eks-scripts/spoke.sh

   Example output

   [spoke.sh] ✅ FleetConfig Spoke EKS setup complete for <SPOKE_CLUSTER_NAME>.

   Repeat this step for each spoke EKS cluster that you would like to join to the hub. You must update the environment variables in /eks-scripts/env-vars and re-run the spoke.sh script for each spoke cluster.

Validate

Files generated by the hub.sh and spoke.sh scripts are stored in the /outputs directory. Those outputs define the IAM roles created in the hub and spoke AWS accounts.

The FleetConfigJoin IAM role should have been created in each spoke AWS account. Ensure that the principal of that IAM role is the ARN of the FleetConfigHubIrsa IAM role which was created in the hub AWS account. The existence of any other value implies that hub.sh and spoke.sh scripts were run out of order. In this case, delete the FleetConfigJoin IAM role from the spoke AWS account, ensure that the FleetConfigHubIrsa IAM role exists on the hub AWS account, and re-run the ./spoke.sh script.

Add New Spoke Clusters

After completing the initial setup, you may want to onboard additional spoke clusters.

Spoke Cluster in New AWS Account

If the new spoke cluster resides in an AWS account that was not included in the original SPOKE_ACCOUNT_ID_LIST environment variable:

1. Update the SPOKE_ACCOUNT_ID_LIST variable to include the AWS account ID of the new spoke cluster.
2. Configure the remaining environment variables required for spoke.sh.
3. Open a shell session with access to the hub AWS account and issue the command ./eks-scripts/hub.sh.
4. Open a shell session with access to the new spoke AWS account and issue the command ./eks-scripts/spoke.sh.

Spoke Cluster in Existing AWS Account

If the new spoke cluster is in an AWS account already listed in SPOKE_ACCOUNT_ID_LIST:

1. Configure the spoke-specific environment variables for the new cluster.
2. Open a shell session with access to the spoke AWS account and issue the command ./eks-scripts/spoke.sh.