Configure ECR Instead of Zot

An Open Container Initiative (OCI) registry is required to store OCI artifacts. Instead of the default in-cluster Zot registry, you can configure the PaletteAI Helm chart to use Amazon Elastic Container Registry (ECR) on self-managed Kubernetes clusters that use AWS EC2 instances. You can configure ECR during installation or afterward. To move an existing installation off Zot, refer to Switch from Zot to ECR After Installation.

Prerequisites

• An existing self-managed Kubernetes cluster that uses AWS EC2 instances.

• Access to the cluster kubeconfig file.

• Downloaded PaletteAI Helm values file.

  curl --output values.yaml --silent https://docs.palette-ai.com/resources/assets/hosted/helm/values.yaml

• Install the following tools on the machine you use to install or upgrade PaletteAI:

  • kubectl version >= 1.31.0

    Set the KUBECONFIG environment variable to the path of the hub cluster kubeconfig file.

    export KUBECONFIG=<kubeconfig-location>

  • Helm version >= 3.17.0 if you plan to use the manual Helm workflow instead of the recommended Flux-managed workflow.

  • AWS CLI

  • A text editor, such as vi

• An AWS IAM role with the following ECR permissions for either a private or public ECR:

  ECR Private

  Minimal inline policy example for private ECR

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "ecr:CompleteLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:InitiateLayerUpload",
          "ecr:BatchCheckLayerAvailability",
          "ecr:PutImage",
          "ecr:BatchGetImage",
          "ecr:GetDownloadUrlForLayer"
        ],
        "Resource": "arn:aws:ecr:<region>:<account-id>:repository/<ecr-namespace>/<ecr-repo-name>"
      },
      {
        "Effect": "Allow",
        "Action": "ecr:GetAuthorizationToken",
        "Resource": "*"
      }
    ]
  }

  ECR Public

  info

  The ECR Public registry's alias is not included in the resource Amazon Resource Name (ARN).

  Minimal inline policy example for ECR Public

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "ecr-public:CompleteLayerUpload",
          "ecr-public:UploadLayerPart",
          "ecr-public:InitiateLayerUpload",
          "ecr-public:BatchCheckLayerAvailability",
          "ecr-public:PutImage"
        ],
        "Resource": "arn:aws:ecr-public::<account-id>:repository/<ecr-repo-name>"
      },
      {
        "Effect": "Allow",
        "Action": [
          "ecr-public:GetAuthorizationToken",
          "sts:GetServiceBearerToken"
        ],
        "Resource": "*"
      }
    ]
  }

• The IAM role is attached to an AWS instance profile, and that instance profile is attached to each EC2 node in the cluster.

Enablement

1. Open the Helm values file in a text editor. This example uses vi.

   vi values.yaml

2. Configure PaletteAI to push OCI artifacts to ECR. Set hue.ociRegistry.provider to aws and hue.ociRegistry.endpoint to your ECR registry URI. The registry URI syntax depends on whether you use a private or public ECR.

   ECR Private

   hue:
     ociRegistry:
       enabled: true
       endpoint: 'oci://<account-id>.dkr.ecr.<region>.amazonaws.com/<ecr-namespace>'
       repository: '<ecr-repo-name>'
       insecure: false
       interval: 5m
       timeout: 60s
       provider: 'aws'

   ECR Public

   hue:
     ociRegistry:
       enabled: true
       endpoint: 'oci://public.ecr.aws/<public-registry-alias>'
       repository: '<ecr-repo-name>'
       insecure: false
       interval: 5m
       timeout: 60s
       provider: 'aws'

3. Update each entry under fleetConfig.spokes[i].ociRegistry so every spoke cluster references the same ECR registry. Set provider to aws.

   ECR Private

   fleetConfig:
     spokes:
       - name: spoke-1
         ociRegistry:
           endpoint: 'oci://<account-id>.dkr.ecr.<region>.amazonaws.com/<ecr-namespace>'
           repository: 'mural-workloads'
           insecure: false
           provider: 'aws'

   ECR Public

   fleetConfig:
     spokes:
       - name: spoke-1
         ociRegistry:
           endpoint: 'oci://public.ecr.aws/<public-registry-alias>'
           repository: 'mural-workloads'
           insecure: false
           provider: 'aws'

   Repeat this step for every spoke cluster you plan to use.

4. On AWS IaaS, Hue and Flux source-controller use the IAM permissions attached to the EC2 node instance profile. No additional service account annotations are required.

5. If you already installed PaletteAI with the default Zot registry, remove the basicAuth section from both the hue and fleetConfig sections in your Helm values. When you use ECR, AWS handles authentication through the node instance profile.

   hue:
     ociRegistry:
       basicAuth:
         username: '<username>'
         password: '<password>'

   fleetConfig:
     spokes:
       - name: spoke-1
         ociRegistry:
           basicAuth:
             username: '<username>'
             password: '<password>'

6. Apply the updated configuration. If you have not installed PaletteAI yet, continue with Install PaletteAI on AWS IaaS and use the updated values.yaml file.

   For Flux-managed installations, update the values block in the HelmRelease manifest for your mural release and apply the changes to the cluster.

   kubectl apply --filename <path-to-your-mural-helmrelease>.yaml

   If you manage the release manually, use the workflow below.

   warning

   If you do not use Flux, manage the mural-crds chart separately from the mural chart. Apply or upgrade Custom Resource Definitions (CRDs) out of band before you install or upgrade the mural chart. For the manual Helm workflow, refer to Upgrade Manually.

   helm upgrade mural oci://public.ecr.aws/mural/mural --version 1.2.2 \
     --namespace mural-system --values values.yaml --wait

Validate

1. If you use Flux, confirm that the OCIRepository resources are ready.

   kubectl get ocirepository --namespace mural-system

2. Check the Hue controller logs for successful OCI pushes.

   kubectl logs --namespace mural-system deployment/hue-controllers | grep --ignore-case "push\|oci"

3. Verify that artifacts exist in ECR.

   ECR Private

   aws ecr describe-images \
     --repository-name <ecr-namespace>/<ecr-repo-name> \
     --region <region>

   ECR Public

   aws ecr-public describe-images \
     --repository-name <ecr-repo-name> \
     --region us-east-1

Next Steps

1. After you confirm that ECR is working, open the PaletteAI values.yaml file and disable Zot.

   vi values.yaml

   zot:
     enabled: false

2. Update your PaletteAI installation to remove Zot.

   For Flux-managed installations, update the values block in the HelmRelease manifest for your mural release and apply the changes to the cluster.

   kubectl apply --filename <path-to-your-mural-helmrelease>.yaml

   If you manage the release manually, use the workflow below.

   warning

   If you do not use Flux, manage the mural-crds chart separately from the mural chart. Apply or upgrade Custom Resource Definitions (CRDs) out of band before you install or upgrade the mural chart. For the manual Helm workflow, refer to Upgrade Manually.

   helm upgrade mural oci://public.ecr.aws/mural/mural --version 1.2.2 \
     --namespace mural-system --values values.yaml --wait

To switch an existing installation from Zot to ECR, refer to Switch from Zot to ECR After Installation.