Skip to main content

Configure ECR Instead of Zot

An Open Container Initiative (OCI) registry is required to store OCI artifacts. Rather than using the default in‑cluster Zot registry, you can configure the PaletteAI Helm chart to use AWS Elastic Container Registry (ECR). ECR works on both AWS EKS and self‑managed Kubernetes on AWS (IaaS) and can be configured during or post-installation.

Prerequisites​

  • An existing self-managed cluster using AWS EC2 instances.

  • Access to your cluster's kubeconfig.

  • PaletteAI's Helm chart downloaded.

    curl --output values.yaml --silent https://docs.palette-ai.com/resources/assets/hosted/helm/values.yaml

  • The following binaries installed on the machine where you are installing or upgrading PaletteAI from:

    • kubectl version >= 1.31.0

      • The KUBECONFIG environment variable set to the path of the PaletteAI hub cluster's kubeconfig file.

        export KUBECONFIG=<kubeconfig-location>

    • helm version >= 3.17

    • AWS CLI

    • A text editor, such as vi, to edit the Helm chart values file.

  • An AWS IAM role with the following ECR permissions for either a private or public ECR.

    Minimal inline policy example for private ECR
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "ecr:CompleteLayerUpload",
            "ecr:UploadLayerPart",
            "ecr:InitiateLayerUpload",
            "ecr:BatchCheckLayerAvailability",
            "ecr:PutImage",
            "ecr:BatchGetImage",
            "ecr:GetDownloadUrlForLayer"
          ],
          "Resource": "arn:aws:ecr:<region>:<account-id>:repository/<ecr-namespace>/<ecr-repo-name>"
        },
        {
          "Effect": "Allow",
          "Action": "ecr:GetAuthorizationToken",
          "Resource": "*"
        }
      ]
    }

  • The above IAM role attached to an AWS instance profile. This instance profile must be attached to each EC2 node that is part of your cluster.

Enablement​

  1. Open the Helm chart values file in a text editor of your choice. This example uses vi.

    vi values.yaml

  2. PaletteAI's Workload controller, Hue, must be able to push OCI artifacts to ECR. To do so, set hue.ociRegistry.provider to aws and specify your ECR's URI for hue.ociRegistry.endpoint. The registry URI syntax varies based on whether you are using a public or private ECR.

    hue:
      ociRegistry:
        enabled: true
        endpoint: 'oci://<account-id>.dkr.ecr.<region>.amazonaws.com/<ecr-namespace>'
        repository: '<ecr-repo-name>'
        insecure: false
        interval: 5m
        timeout: 60s
        provider: 'aws'

  3. Add the following IRSA annotation to Hue's service account.

    hue:
      serviceAccount:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<iam-role-name>

  4. Add the following IRSA annotation to the service account used by Flux's source-controller.

    flux2:
      sourceController:
        serviceAccount:
          create: true
          automount: true
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<iam-role-name>

  5. Each spoke cluster must reference the same OCI registry. Update every entry under fleetConfig.spokes[i].ociRegistry to point to your ECR, and set the provider to aws.

    fleetConfig:
      spokes:
        - name: spoke-1
          ociRegistry:
            endpoint: 'oci://<account-id>.dkr.ecr.<region>.amazonaws.com/<ecr-namespace>'
            repository: 'mural-workloads'
            insecure: false
            provider: 'aws'

    Repeat this step for all spokes you intend to use.

  6. If you have already installed PaletteAI using the default Zot registry, remove the basicAuth section from the hue and fleetConfig sections of your Helm chart. When using ECR with IRSA, authentication is automatically handled by AWS using the service account annotations set in steps 2 - 3.

    hue:
      ociRegistry:
        basicAuth:
          username: 'user'
          password: 'user'
    fleetConfig:
      spokes:
        - name: spoke-1
          ociRegistry:
            basicAuth:
              username: 'user'
              password: 'user'

  7. If you have already installed PaletteAI, update your installation using your modified Helm chart. Otherwise, proceed with the appropriate Install PaletteAI on AWS IaaS or Install PaletteAI on EKS guide.

    helm upgrade mural oci://public.ecr.aws/mural/mural \
      --namespace mural-system \
      --filename values.yaml \
      --wait

Validate​

Once PaletteAI is installed or upgraded, take the following steps to verify that your ECR is correctly configured with PaletteAI.

  1. Check that Flux OCIRepository resources are syncing successfully. Look for READY: True in the output.

    kubectl get ocirepository --namespace mural-system

  2. Check the Hue controller logs for successful OCI pushes.

    kubectl logs --namespace mural-system deployment/hue | grep --ignore-case "push\|oci"

  3. Verify artifacts exist in ECR.

    aws ecr describe-images \
      --repository-name <ecr-namespace>/<ecr-repo-name> \
      --region <region>

  4. After you have validated ECR, open the PaletteAI values.yaml file and disable Zot.

    vi values.yaml
    zot:
      enabled: false

  5. Upgrade your PaletteAI installation to remove Zot.

    helm upgrade mural oci://public.ecr.aws/mural/mural \
      --namespace mural-system \
      --filename values.yaml \
      --wait