Version: v1.0.x

Configure ECR Instead of Zot

An Open Container Initiative (OCI) registry is required to store OCI artifacts. Instead of the default in-cluster Zot registry, you can configure the PaletteAI Helm chart to use Amazon Elastic Container Registry (ECR). Amazon ECR works on both Amazon EKS and self-managed Kubernetes on AWS (IaaS), and you can configure it during installation or afterward.

Prerequisites

AWS IaaS
EKS

An existing self-managed Kubernetes cluster that uses AWS EC2 instances.
Access to the cluster kubeconfig file.

Downloaded PaletteAI Helm values file.

curl --output values.yaml --silent https://docs.palette-ai.com/resources/assets/hosted/helm/values.yaml

Install the following tools on the machine you use to install or upgrade PaletteAI:
- kubectl version >= 1.31.0
  
  Set the KUBECONFIG environment variable to the path of the hub cluster kubeconfig file.
```
export KUBECONFIG=<kubeconfig-location>
```
- Helm version >= 3.17.0 if you plan to use the manual Helm workflow instead of the recommended Flux-managed workflow.
- AWS CLI
- A text editor, such as vi

An AWS IAM role with the following ECR permissions for either a private or public ECR:

ECR Private
ECR Public

Minimal inline policy example for private ECR
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:CompleteLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:InitiateLayerUpload",
        "ecr:BatchCheckLayerAvailability",
        "ecr:PutImage",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ],
      "Resource": "arn:aws:ecr:<region>:<account-id>:repository/<ecr-namespace>/<ecr-repo-name>"
    },
    {
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "*"
    }
  ]
}

info

The ECR Public registry's alias is not included in the resource Amazon Resource Name (ARN).

Minimal inline policy example for ECR Public
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr-public:CompleteLayerUpload",
        "ecr-public:UploadLayerPart",
        "ecr-public:InitiateLayerUpload",
        "ecr-public:BatchCheckLayerAvailability",
        "ecr-public:PutImage"
      ],
      "Resource": "arn:aws:ecr-public::<account-id>:repository/<ecr-repo-name>"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr-public:GetAuthorizationToken",
        "sts:GetServiceBearerToken"
      ],
      "Resource": "*"
    }
  ]
}

The IAM role is attached to an AWS instance profile, and that instance profile is attached to each EC2 node in the cluster.

An existing Amazon EKS cluster.
Access to the cluster kubeconfig file.

Downloaded PaletteAI Helm values file.

curl --output values.yaml --silent https://docs.palette-ai.com/resources/assets/hosted/helm/values.yaml

Install the following tools on the machine you use to install or upgrade PaletteAI:
- kubectl version >= 1.31.0
  
  Set the KUBECONFIG environment variable to the path of the hub cluster kubeconfig file.
```
export KUBECONFIG=<kubeconfig-location>
```
- Helm version >= 3.17.0 if you plan to use the manual Helm workflow instead of the recommended Flux-managed workflow.
- AWS CLI
- A text editor, such as vi
An OpenID Connect identity provider (IdP) in AWS Identity and Access Management (IAM) that uses the OpenID Connect issuer URL for your EKS cluster

An AWS IAM role with the following ECR permissions for either a private or public ECR:

ECR Private
ECR Public

Minimal inline policy example for private ECR
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:CompleteLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:InitiateLayerUpload",
        "ecr:BatchCheckLayerAvailability",
        "ecr:PutImage",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ],
      "Resource": "arn:aws:ecr:<region>:<account-id>:repository/<ecr-namespace>/<ecr-repo-name>"
    },
    {
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "*"
    }
  ]
}

info

The ECR Public registry's alias is not included in the resource Amazon Resource Name (ARN).

Minimal inline policy example for ECR Public
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr-public:CompleteLayerUpload",
        "ecr-public:UploadLayerPart",
        "ecr-public:InitiateLayerUpload",
        "ecr-public:BatchCheckLayerAvailability",
        "ecr-public:PutImage"
      ],
      "Resource": "arn:aws:ecr-public::<account-id>:repository/<ecr-repo-name>"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr-public:GetAuthorizationToken",
        "sts:GetServiceBearerToken"
      ],
      "Resource": "*"
    }
  ]
}

The IAM role includes the following IAM trust policy so the hue and source-controller service accounts can assume the role through IAM Roles for Service Accounts (IRSA)

Example trust policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<provider-id>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<region>.amazonaws.com/id/<provider-id>:sub": [
            "system:serviceaccount:<helm-release-namespace>:hue",
            "system:serviceaccount:<helm-release-namespace>:source-controller"
          ]
        }
      }
    }
  ]
}

Enablement

Open the Helm values file in a text editor. This example uses vi.
```
vi values.yaml
```

Configure Hue to push OCI artifacts to ECR. Set hue.ociRegistry.provider to aws and hue.ociRegistry.endpoint to your ECR registry URI. The registry URI syntax depends on whether you use a private or public ECR.

ECR Private
ECR Public

hue:
  ociRegistry:
    enabled: true
    endpoint: 'oci://<account-id>.dkr.ecr.<region>.amazonaws.com/<ecr-namespace>'
    repository: '<ecr-repo-name>'
    insecure: false
    interval: 5m
    timeout: 60s
    provider: 'aws'

hue:
  ociRegistry:
    enabled: true
    endpoint: 'oci://public.ecr.aws/<public-registry-alias>'
    repository: '<ecr-repo-name>'
    insecure: false
    interval: 5m
    timeout: 60s
    provider: 'aws'

Update each entry under fleetConfig.spokes[i].ociRegistry so every spoke cluster references the same ECR registry. Set provider to aws.

ECR Private
ECR Public

fleetConfig:
  spokes:
    - name: spoke-1
      ociRegistry:
        endpoint: 'oci://<account-id>.dkr.ecr.<region>.amazonaws.com/<ecr-namespace>'
        repository: 'mural-workloads'
        insecure: false
        provider: 'aws'

fleetConfig:
  spokes:
    - name: spoke-1
      ociRegistry:
        endpoint: 'oci://public.ecr.aws/<public-registry-alias>'
        repository: 'mural-workloads'
        insecure: false
        provider: 'aws'

Repeat this step for every spoke cluster you plan to use.

Configure AWS authentication for the hue service account.
- AWS IaaS
- EKS
Skip this step. On AWS IaaS, Hue uses the IAM permissions attached to the EC2 node instance profile.
Add the following IRSA annotation to Hue's service account.
hue: serviceAccount: create: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<iam-role-name>
Configure AWS authentication for the Flux source-controller service account.
- AWS IaaS
- EKS
Skip this step. On AWS IaaS, source-controller uses the IAM permissions attached to the EC2 node instance profile.
Add the following IRSA annotation to the service account used by Flux source-controller.
flux2: sourceController: serviceAccount: create: true automount: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<iam-role-name>

If you already installed PaletteAI with the default Zot registry, remove the basicAuth section from both the hue and fleetConfig sections in your Helm values. When you use ECR, AWS handles authentication through the node instance profile or IRSA.

hue:
  ociRegistry:
    basicAuth:
      username: '<username>'
      password: '<password>'

fleetConfig:
  spokes:
    - name: spoke-1
      ociRegistry:
        basicAuth:
          username: '<username>'
          password: '<password>'

Apply the updated configuration. If you have not installed PaletteAI yet, continue with Install PaletteAI on AWS IaaS or Install PaletteAI on EKS and use the updated values.yaml file.

For Flux-managed installations, update the values block in the HelmRelease manifest for your mural release and apply the changes to the cluster.
```
kubectl apply --filename <path-to-your-mural-helmrelease>.yaml
```
If you manage the release manually, use the workflow below.

warning
If you do not use Flux, manage the mural-crds chart separately from the mural chart. Apply or upgrade Custom Resource Definitions (CRDs) out of band before you install or upgrade the mural chart. For the manual Helm workflow, refer to Upgrade Manually.
```
helm upgrade mural oci://public.ecr.aws/mural/mural --version 1.0.7 \
  --namespace mural-system --values values.yaml --wait
```

Validate

If you use Flux, confirm that the OCIRepository resources are ready.
```
kubectl get ocirepository --namespace mural-system
```

Check the Hue controller logs for successful OCI pushes.

kubectl logs --namespace mural-system deployment/hue | grep --ignore-case "push\|oci"

Verify that artifacts exist in ECR.

ECR Private
ECR Public

aws ecr describe-images \
  --repository-name <ecr-namespace>/<ecr-repo-name> \
  --region <region>

aws ecr-public describe-images \
  --repository-name <ecr-repo-name> \
  --region us-east-1

Next Steps

After you confirm that ECR is working, open the PaletteAI values.yaml file and disable Zot.
```
vi values.yaml
```
```
zot:
  enabled: false
```
Update your PaletteAI installation to remove Zot.

For Flux-managed installations, update the values block in the HelmRelease manifest for your mural release and apply the changes to the cluster.
```
kubectl apply --filename <path-to-your-mural-helmrelease>.yaml
```
If you manage the release manually, use the workflow below.

warning
If you do not use Flux, manage the mural-crds chart separately from the mural chart. Apply or upgrade Custom Resource Definitions (CRDs) out of band before you install or upgrade the mural chart. For the manual Helm workflow, refer to Upgrade Manually.
```
helm upgrade mural oci://public.ecr.aws/mural/mural --version 1.0.7 \
  --namespace mural-system --values values.yaml --wait
```

Prerequisites​

Enablement​

Validate​

Next Steps​

Prerequisites

Enablement

Validate

Next Steps