Version: v1.0.x

Install PaletteAI on GKE

Use this guide to install PaletteAI on Google Kubernetes Engine (GKE). The deployment uses the hub-as-spoke pattern with Zot as the Open Container Initiative (OCI) registry.

Prerequisites

Use a Kubernetes cluster as the PaletteAI hub.
Access to the hub cluster using the built-in Kubernetes cluster-admin ClusterRole.
Minimum Kubernetes versions

Cluster Type Kubernetes Version
Hub >= 1.31.0
Spoke >= 1.31.0
Minimum resource requests

Cluster Type CPU Memory Storage
Hub 3388m 2732 Mi 10Gi
Spoke 1216m 972 Mi 10Gi
Ensure the hub cluster can reach the public AWS Elastic Container Registry (ECR) that hosts the mural and mural-crds charts.
Access to the hub cluster kubeconfig file.
Install Flux controllers on the hub cluster if you plan to use the recommended Flux-managed workflow.
Install the following tools on the machine you use to install or upgrade PaletteAI:
- curl or wget to download the Helm values file.
- A text editor, such as vi, to edit the Helm values file.
- kubectl version >= 1.31.0.
- Helm version >= 3.17.0 if you plan to use the manual Helm workflow instead of the recommended Flux-managed workflow.
Configure the hub cluster Kubernetes API server to trust Dex as an identity provider. PaletteAI deploys Dex as part of the installation. This requirement applies only to the hub cluster, not to spoke clusters. For details, refer to Configure Kubernetes API Server to Trust OpenID Connect (OIDC) Provider.

Cluster Type	Kubernetes Version
Hub	>= 1.31.0
Spoke	>= 1.31.0

Cluster Type	CPU	Memory	Storage
Hub	3388m	2732 Mi	10Gi
Spoke	1216m	972 Mi	10Gi

By default, PaletteAI is configured to terminate HTTPS at the load balancer. To enable this behavior you need:
- A domain name available for PaletteAI.
- The HttpLoadBalancing add-on enabled on the GKE cluster.
- An optional static external IP address for the load balancer.
info
PaletteAI communicates internally over HTTPS. Refer to our Security page for a detailed explanation of the security architecture.
PaletteAI uses Dex as an OIDC provider. Dex provides a number of connectors to allow users to log in to PaletteAI using their existing identity provider. To enable Google Workspace as an OIDC provider for Dex, you must configure a Google Service Account, export a JSON key, and create a Kubernetes Secret containing the JSON key in the same namespace that PaletteAI will be installed. For detailed instructions, refer to the Dex documentation.

Once you download the JSON key, issue the following command to create a Kubernetes Secret containing the JSON key in the same namespace that Mural will be installed.
```
# create the namespace that Mural will be installed in, if it doesn't exist
kubectl create namespace mural-system

# create the secret containing the JSON key
kubectl create secret generic mural-google-service-account --from-file=googleAuth.json=/path/to/googleAuth.json --namespace mural-system
```
To deploy PaletteAI with dedicated GKE spoke clusters, you must configure additional ClusterRoles and ClusterRoleBindings on each spoke. These permissions allow the hub's FleetConfig controller to bootstrap Open Cluster Management (OCM) components on the spoke clusters. Refer to the GKE Spoke Setup guide before proceeding.

Enablement

1. Download the latest Helm chart values file. This example uses curl.
```
curl --output values.yaml --silent https://docs.palette-ai.com/resources/assets/hosted/helm/values.yaml
```
2. Open the Helm chart values file in a text editor of your choice and complete the following sections. This example uses vi.
```
vi values.yaml
```
Global
Use the global section to configure overarching settings for the PaletteAI deployment. Review and modify the following values as necessary.
1. Set global.dns.domain to the primary domain for the deployment. Do not include a protocol. For example, use example.org, not https://example.org.
```
global:
  dns:
    domain: 'example.acme.org'
```
2. In global.auditLogging.basicAuth, change the default username and password for audit logging. These credentials secure the Alertmanager instance that receives audit events. You reuse them when you configure the Base64-encoded Authorization header in the alertmanager section.
```
global:
  auditLogging:
    basicAuth:
      username: '<your-username>'
      password: '<your-password>'
```
  Refer to Audit Logging to learn more about configuring audit logging, querying audit events, and forwarding logs to long-term storage.
3. Configure the metrics collection settings. Provide an existing, external Prometheus server that is reachable from the hub cluster and every spoke cluster. Spoke clusters use Prometheus agents to ship metrics to the server via remote_write.
  
  Set global.metrics.prometheusBaseUrl to the external Prometheus server URL (for example, https://your-external-prometheus:9090). Include only the protocol, host, and port — do not include any API paths.
```
global:
  metrics:
    prometheusBaseUrl: 'https://your-external-prometheus:9090'
    timeout: '5s'
    scrapeInterval: '15s'
    agentType: 'prometheus-agent-minimal'
    username: ''
    password: ''
```
  By default, global.metrics.agentType is set to prometheus-agent-minimal. The minimal agent configuration only collects spoke cluster CPU and GPU utilization metrics. You may change global.metrics.agentType to prometheus-agent to ship all node-exporter and dcgm-exporter metrics from spoke clusters for comprehensive observability.
  
  If your Prometheus server requires basic authentication, configure the username and password fields. Leave these fields blank if authentication is not required.
  
  Refer to Configure Prometheus Agent Monitoring for guidance on agent types, Prometheus and Grafana prerequisites, and GPU metrics.
  
  tip
  If you need to set up a Prometheus server, you may find the Deploy Monitoring Stack guide helpful.
1. Set global.kubernetesProvider to GKE-Ingress.
```
global:
  kubernetesProvider: GKE-Ingress
```
  Complete global configuration section
FleetConfig

To configure a GKE FleetConfig, update the following parameters in the fleetConfig section of your Helm chart.

Parameter	Description
`hub.apiServer`	The hub cluster's API server endpoint. This is found in the hub cluster's kubeconfig file.
`spokes[i].klusterlet.forceInternalEndpointLookup`	Dictates if the internal endpoint is looked up via the cluster-info `ConfigMap` instead of the hub cluster's public API server endpoint.

warning

Do not change the spokes[i].name: hub-as-spoke value if using the hub-as-spoke pattern when installing PaletteAI.

fleetConfig:
  hub:
    apiServer: "https://<public-ip>:<port>"
  spokes:
    - name: hub-as-spoke # do not edit this name if you are using the default hub-as-spoke mode
      klusterlet:
        forceInternalEndpointLookup: false

This is the minimum configuration required to install a hub-as-spoke FleetConfig for GKE. If you are using a dedicated hub with separate spoke clusters, also complete the following steps for each spoke cluster.

Hub with dedicated spoke clusters

Ensure that the spoke cluster has the correct permissions to join the hub cluster. For details, refer to Set Up GKE Spokes.
Create a copy of the spoke kubeconfig.

export KUBECONFIG=spoke-i.kubeconfig
gcloud container clusters get-credentials <spoke-cluster-name> --region <region> --project <project-id>

Upload the copied kubeconfig to a Kubernetes Secret on the hub cluster.

KUBECONFIG=hub.kubeconfig kubectl create secret generic spoke-kubeconfig-i --from-file=kubeconfig=spoke-i.kubeconfig --namespace <spoke-namespace>

Update fleetConfig.spokes[i].kubeconfig to reference the spoke-kubeconfig-i Secret you created in the previous step, and set kubeconfig.inCluster: false.

fleetConfig:
  spokes:
    - kubeconfig:
        # The context to use in the kubeconfig file. Leave empty to use the current context.
        context: ''
        # If set, the kubeconfig will be read from the cluster. Only applicable for same-cluster operations.
        inCluster: false
        # A reference to an existing secret containing a kubeconfig. Must be provided for remote clusters.
        # For same-cluster, must be provided unless InCluster is set to true.
        secretReference:
          # The name of the secret.
          name: 'spoke-kubeconfig-i'
          # The map key to access the kubeconfig.
          kubeconfigKey: 'kubeconfig'

Rename fleetConfig.spokes[i].name from hub-as-spoke to the name you want to use.

Alertmanager

Navigate to the alertmanager section. Update credentials for the alertmanager instance based on the credentials you configured in the global section.

You must provide a Base64-encoded string for the Authorization header. Use the interactive encoder to generate your Base64-encoded string and copy the value to the clipboard.

Username
Password
Base64 Encoded String:

Alternatively, generate the Base64-encoded string using the following command. Replace username and password with the username and password you configured in the global section.
```
echo -n "username:password" | base64
```
The following example shows the livenessProbe and readinessProbe sections with the Base64-encoded string. Replace <your-base64-encoded-string> with the Base64-encoded string you generated.
```
alertmanager:
  livenessProbe:
    httpGet:
      path: /-/healthy
      port: http
      scheme: HTTPS
      httpHeaders:
        - name: Authorization
          value: 'Basic <your-base64-encoded-string>'
  readinessProbe:
    httpGet:
      path: /-/ready
      port: http
      scheme: HTTPS
      httpHeaders:
        - name: Authorization
          value: 'Basic <your-base64-encoded-string>'
```
Complete alertmanager configuration section

For further instructions on accessing audit logs and configuring long-term storage, refer to Audit Logging.

Canvas

To configure the ingress for Canvas, set canvas.ingress.enabled to true. Set canvas.ingress.domain to your domain name. Omit the http:// or https:// prefix. Set canvas.ingress.ingressClassName to gce.

canvas:
  ingress:
    enabled: true
    annotations: {}
    ingressClassName: gce
    domain: replace.with.your.domain # No HTTP/HTTPS prefix.
    matchAllHosts: false
    tls: []

Optionally, add the kubernetes.io/ingress.global-static-ip-name annotation to assign an existing GCP Global Static IP to the ingress.

canvas:
  ingress:
    annotations:
      kubernetes.io/ingress.global-static-ip-name: <static-ip-name>

In canvas.ingress.paths, set the /ai entry pathType to Prefix. Then add a /dex entry with pathType: Prefix, backend.service.name: dex, and backend.service.port.number: 5556. This creates a single Ingress resource for both Canvas and Dex. The GKE Ingress controller uses this resource to create a single load balancer for both services.

canvas:
  ingress:
    paths:
      - path: /ai
        pathType: Prefix
        backend:
          service:
            name: canvas
            port:
              number: 2999
      - path: /dex
        pathType: Prefix
        backend:
          service:
            name: dex
            port:
              number: 5556

Set canvas.enableHTTP to true to let the load balancer terminate Transport Layer Security (TLS). Leave canvas.ingress.tls empty.

canvas:
  enableHTTP: true

The last portion of the Canvas configuration is the OIDC configuration. If you defer configuring OIDC for Dex, you may do the same for Canvas and configure it later.

In the canvas.oidc section, enter a unique string for the sessionSecret. For redirectURL, replace <your-domain> with your domain. Do not remove the /ai/callback path.
```
canvas:
  oidc:
    sessionSecret: '<your-session-secret>'
    sessionDir: '/app/sessions'
    issuerK8sService: 'https://dex.mural-system.svc.cluster.local:5554/dex'
    skipSSLCertificateVerification: true
    redirectURL: 'https://<your-domain>/ai/callback'
```
If you did not configure your Kubernetes cluster to trust Dex as an OIDC provider, then you must configure the canvas.impersonationProxy section to enable user impersonation.

The example below shows how to configure the local Dex user admin@example.com to be mapped to an example Kubernetes group admin. Refer to our Configure User Impersonation guide to learn more about how to configure user impersonation for OIDC groups and other use cases.
Example user impersonation setup
```
canvas:
  impersonationProxy:
    enabled: true
    userMode: 'passthrough'
    groupsMode: 'map'
    userMap: {}
    groupMap: {}
    dexGroupMap:
      'admin@example.com': [ 'admin' ]
```
Complete canvas configuration section

Dex

Dex authenticates users to PaletteAI through SSO. You can configure Dex to connect to an upstream OIDC provider or a local user database. For this installation, you will configure Dex to connect to an OIDC provider. If you want to configure OIDC later, you can do so; however, Dex still requires some basic configuration.

Set dex.config.issuer to your domain. Do not remove the /dex path.
```
dex:
  config:
   issuer: 'https://<your-domain>/dex'
```

This next part may be deferred for later, but we strongly recommend configuring at least one connector. Set the dex.config.connectors to the connectors you want to use. The Dex documentation has examples for each of the connectors. Following is an example configuration for a Google connector.

Example Google Workspace configuration
  dex:
    config:
      connectors:
      - type: google
        id: google
        name: Google Workspace
        config:
          clientID: xxxxxxxxxxxxxxx
          clientSecret: xxxxxxxxxxxxxxx
          redirectURI: https://<your-domain>/dex/callback # Dex callback URL for the authorization code flow; redirects to the application callback URL
          promptType: consent
          hostedDomains:
          - <your-domain> # no HTTP/HTTPS prefix
          groups:
          - admin
          - sre
          serviceAccountFilePath: /etc/google/googleAuth.json # Path to the mounted service account file
          domainToAdminEmail:
            example.com: admin@example.com

Proceed to the dex.config.staticClients section. Replace <your-client-secret> with a unique secret value and <your-domain> with your domain. Do not remove the /ai/callback path for the mural client.

dex:
  config:
    staticClients:
      - id: mural
        redirectURIs:
          - 'https://<your-domain>/ai/callback'
        name: 'mural'
        secret: '<your-client-secret>'
        public: false
        trustedPeers:
          - kubernetes
      - id: kubernetes
        redirectURIs:
          - 'https://<your-domain>'
        name: kubernetes
        secret: '<your-client-secret>'
        public: false
        trustedPeers:
          - mural

Next, configure the dex.config.staticPasswords section. We strongly recommend changing the default user (admin) and password (password) to strong values. The following example is the default user and password in bcrypt format. Remember to use a bcrypt hash generator to generate the password hash. The userID can be any unique string.

warning
If you did not configure any connectors, you must configure at least one static user, which is used to access the PaletteAI UI. Static Dex users automatically inherit admin privileges through the service account. Dex does not support groups for local static users. To use groups for local static users, you must use the User Impersonation feature.
```
dex:
  config:
    staticPasswords:
      - email: 'admin@example.com'
        hash: '$2a$12$Ot2dJ0pmdIC2oXUDW/Ez1OIfhkSzLZIbsumsxkByuU3CUr02DtiC.'
        username: 'admin'
        userID: '08a8684b-db88-4b73-90a9-3cd1661f5466'
```

Add the following volumes and volume mounts to allow the Dex service to access the Google service account file.

warning

Do not remove existing volumes and volume mounts in the values.yaml file.

dex:
  volumes:
    # do not remove existing volumes
    - name: tls-cert-vol
      secret:
        secretName: mural-dex-serving-cert
    - name: google-auth-vol
      secret:
        secretName: mural-google-service-account # The name of the secret that was created earlier

dex:
  volumeMounts:
    # do not remove existing volumeMounts
    - mountPath: /etc/k8s-webhook-certs
      name: tls-cert-vol
      readOnly: true
    - mountPath: /etc/google
      name: google-auth-vol
      readOnly: true

Add the following annotation to the Dex service to allow the Dex service to enable GKE Ingress Load Balancer health checks.
```
dex:
  service:
    annotations:
      cloud.google.com/backend-config: '{"default":"dex-backendconfig"}'
```
Disable Dex's ingress resource. The GKE Ingress controller will use the Ingress resource created for Canvas to create a single load balancer for both Canvas and Dex.
```
dex:
  ingress:
    enabled: false
```
Complete dex configuration section

Flux2

Set flux2.policies.create to false to disable the Flux network policies. These policies, if enabled, prevent ingress traffic from reaching their target services.
```
flux2:
  policies:
    create: false
```
info
This step is not required if the hub and all spoke clusters are configured to use a common, external OCI registry. An external OCI registry is configured in the fleetConfig.spokes[*].ociRegistry and hue.ociRegistry sections of the values.yaml file.

Complete flux2 configuration section

Ingress-Nginx
Disable ingress-nginx. GKE provides a native ingress controller that supports TLS termination at the load balancer.
```
ingress-nginx:
  enabled: false
```
Zot

Set zot.ingress.enabled to true and add the following annotations so the ingress routes traffic to Zot.

zot:
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/use-regex: "true"
      nginx.ingress.kubernetes.io/rewrite-target: /$1

Set zot.ingress.hosts[i].host to the domain you set in the global.dns.domain parameter.

zot:
  ingress:
    hosts:
      - host: my.domain.com
        paths:
          - path: /zot/(.*)

Set zot.httpGet.scheme to HTTP so the HTTP probe succeeds.

zot:
  httpGet:
    scheme: HTTP

Together, these configurations allow you to use a single load balancer instead of a second load balancer dedicated to Zot.

The resulting Ingress resources produce two redirect paths:

Requests for my.domain.com/zot/* are sent to the zot service endpoint with the /zot portion of the path removed.
Requests for my.domain.com/v2 and my.domain.com/v2/* are also sent to the zot service endpoint. Therefore, Zot owns my.domain.com/v2. No other service can expect to serve traffic through that route.

Modify the zot.configFiles.config.json section to remove the tls section. Transport Layer Security (TLS) terminates at the load balancer, so you do not need the tls section.

zot:
  configFiles:
    config.json: |-
      {
        "storage": { "rootDirectory": "/var/lib/registry" },
        "http": { "address": "0.0.0.0", "port": "5000","auth": { "failDelay": 5, "htpasswd": { "path": "/secret/htpasswd" } } },
        "extensions": {"search": {"enable": true}, "ui": {"enable": false}},
        "log": { "level": "debug" }
      }

To add additional users, add them to the htpasswd section. Use the htpasswd utility or a similar tool that can generate a bcrypt hash.

zot:
  secretFiles:
    htpasswd: |-
      admin:$2y$05$vmiurPmJvHylk78HHFWuruFFVePlit9rZWGA/FbZfTEmNRneGJtha
      user:$2y$05$L86zqQDfH5y445dcMlwu6uHv.oXFgT6AiJCwpv3ehr7idc0rI3S2G

If this configuration does not work in your environment, you may set zot.service.type: LoadBalancer and zot.ingress.enabled: false. This requires an additional load balancer and an additional DNS A or CNAME record. You can configure the DNS record after the Helm installation. However, you must pre-configure the following fields to use the correct DNS name:

fleetConfig.spokes[i].ociRegistry.endpoint
fleetConfig.spokeValuesOverrides.hue.ociRegistry.endpoint
hue.ociRegistry.endpoint

tip

If you use a dedicated load balancer for Zot and terminate TLS inside your cluster, the endpoint must include a :5000 suffix when provided to other services that need the registry endpoint (e.g., oci://zot.my.domain.com:5000).

Complete zot configuration section

Install with Flux

Install PaletteAI with Flux to let Flux manage chart ordering and the Custom Resource Definition (CRD) lifecycle for both Helm charts.

Create mural-crds-oci-repository.yaml for the mural-crds chart.

cat << EOF > mural-crds-oci-repository.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: mural-crds
  namespace: mural-system
spec:
  interval: 10m
  ref:
    semver: "0.7.0-hotfix.4"
  url: oci://public.ecr.aws/mural/mural-crds
EOF

Create mural-oci-repository.yaml for the mural chart.

cat << EOF > mural-oci-repository.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: mural
  namespace: mural-system
spec:
  interval: 10m
  ref:
    semver: "1.0.7"
  url: oci://public.ecr.aws/mural/mural
EOF

Apply both OCIRepository resources to your cluster.

kubectl apply --filename mural-crds-oci-repository.yaml
kubectl apply --filename mural-oci-repository.yaml

Create mural-crds-helm-release.yaml for the mural-crds chart.

cat <<'EOF' > mural-crds-helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mural-crds
  namespace: mural-system
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: mural-crds
    namespace: mural-system
  install:
    crds: Create
  upgrade:
    crds: CreateReplace
EOF

Create mural-helm-release.yaml for the mural chart. The dependsOn field ensures that Flux installs mural-crds before mural.

cat <<'EOF' > mural-helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mural
  namespace: mural-system
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: mural
    namespace: mural-system
  dependsOn:
    - name: mural-crds
  values:
    # Paste the contents of your values.yaml file here.
EOF

Open mural-helm-release.yaml and replace the placeholder comment under spec.values with the contents of the values.yaml file for your environment. Keep the inserted YAML indented under spec.values.

Apply both HelmRelease resources to your cluster.

kubectl apply --filename mural-crds-helm-release.yaml
kubectl apply --filename mural-helm-release.yaml

Install with Helm

warning

If you do not use Flux, manage the mural-crds chart separately from the mural chart. Apply or upgrade Custom Resource Definitions (CRDs) out of band before you install or upgrade the mural chart. For the manual Helm workflow, refer to Upgrade Manually.

Install the mural-crds Helm chart first.

helm install mural-crds oci://public.ecr.aws/mural/mural-crds --version 0.7.0-hotfix.4 \
  --namespace mural-system --create-namespace --wait

Example Output
NAME: mural-crds
LAST DEPLOYED: Tue May 27 09:34:33 2025
NAMESPACE: mural-system
STATUS: deployed
REVISION: 1

Install PaletteAI from the mural chart by using your environment's values.yaml file.

helm install mural oci://public.ecr.aws/mural/mural --version 1.0.7 \
  --namespace mural-system --create-namespace --values values.yaml --wait

Example Output
NAME: mural
LAST DEPLOYED: Tue May 27 09:39:48 2025
NAMESPACE: mural-system
STATUS: deployed
REVISION: 1

Configure DNS

Once PaletteAI is deployed, get the IP address of the load balancer the GKE Ingress controller deployed.
```
kubectl get ingress canvas --namespace mural-system
```
Example output
```
NAME     CLASS   HOSTS               ADDRESS   PORTS   AGE
canvas   gce     mural.example.com   YOUR_IP   80      32m
```
Create a DNS record for the ADDRESS load balancer IP. If you are using Cloud DNS, create an A record in your domain's hosted zone and select the load balancer as the target. Refer to Configure Cloud DNS alias record for your target DNS for more information.

info
It may take a few minutes for the DNS changes to take effect.

You have now deployed PaletteAI on GKE. If you are using user impersonation or configured an OIDC provider, you can now log in to PaletteAI. You can also use the default Dex local user.

If you need to update PaletteAI later, review Helm Chart Configuration and then follow Upgrade PaletteAI. For Flux-managed installations, update the mural HelmRelease or the relevant OCIRepository resource and let Flux reconcile the change. For manual Helm installations, follow Upgrade Manually and apply mural-crds out of band before mural.

Validate

Take the following steps to verify that PaletteAI is deployed and configured correctly.

Open a browser and navigate to the domain URL you configured for PaletteAI.
Log in with the default username and password. If you configured Dex with an OIDC connector, log in with your identity provider.

Next Steps

Once you have installed PaletteAI, integrate Palette with PaletteAI by configuring the Settings resource. This resource requires a Palette tenant, project, and API key so PaletteAI can communicate with Palette and deploy AI/ML applications and models to the correct location.

Proceed to the Integrate with Palette guide to learn how to prepare your Palette environment.

Prerequisites​

Enablement​

Global​

FleetConfig​

Alertmanager​

Canvas​

Dex​

Flux2​

Ingress-Nginx​

Zot​

Install with Flux​

Install with Helm​

Configure DNS​

Validate​

Next Steps​

Prerequisites

Enablement

Global

FleetConfig

Alertmanager

Canvas

Dex

Flux2

Ingress-Nginx

Zot

Install with Flux

Install with Helm

Configure DNS

Validate

Next Steps