Version: v1.0.x

Install PaletteAI on Kubernetes

Use this guide to install PaletteAI on a self-managed Kubernetes cluster where you control the API server configuration. The deployment uses the hub-as-spoke pattern with Zot as the Open Container Initiative (OCI) registry. Use this guide if you are installing PaletteAI on:

Self-managed clusters except AWS EC2-based clusters, such as kubeadm, k3s, and RKE
Self-hosted Kubernetes deployments
Edge environments
Any cluster where you can configure the Kubernetes API server to trust Dex as an OIDC provider

warning

If you are installing PaletteAI on AWS or GKE, use a platform-specific guide instead:

AWS IaaS: For self-managed Kubernetes on EC2. Includes AWS-specific ingress and DNS configuration.
AWS EKS: Includes IRSA configuration, AWS load balancer annotations, and impersonation proxy setup.
GKE: Uses native GKE Ingress and includes impersonation proxy setup.

Prerequisites

Use a Kubernetes cluster as the PaletteAI hub.
Access to the hub cluster using the built-in Kubernetes cluster-admin ClusterRole.
Minimum Kubernetes versions

Cluster Type Kubernetes Version
Hub >= 1.31.0
Spoke >= 1.31.0
Minimum resource requests

Cluster Type CPU Memory Storage
Hub 3388m 2732 Mi 10Gi
Spoke 1216m 972 Mi 10Gi
Ensure the hub cluster can reach the public AWS Elastic Container Registry (ECR) that hosts the mural and mural-crds charts.
Access to the hub cluster kubeconfig file.
Install Flux controllers on the hub cluster if you plan to use the recommended Flux-managed workflow.
Install the following tools on the machine you use to install or upgrade PaletteAI:
- curl or wget to download the Helm values file.
- A text editor, such as vi, to edit the Helm values file.
- kubectl version >= 1.31.0.
- Helm version >= 3.17.0 if you plan to use the manual Helm workflow instead of the recommended Flux-managed workflow.
Configure the hub cluster Kubernetes API server to trust Dex as an identity provider. PaletteAI deploys Dex as part of the installation. This requirement applies only to the hub cluster, not to spoke clusters. For details, refer to Configure Kubernetes API Server to Trust OpenID Connect (OIDC) Provider.

Cluster Type	Kubernetes Version
Hub	>= 1.31.0
Spoke	>= 1.31.0

Cluster Type	CPU	Memory	Storage
Hub	3388m	2732 Mi	10Gi
Spoke	1216m	972 Mi	10Gi

Your hub cluster must be able to provision load balancer services. For self-hosted or bare-metal clusters, this requires a load balancer implementation such as MetalLB. For cloud-hosted clusters, ensure the appropriate cloud controller manager is configured.

Enablement

1. Download the latest Helm chart values file. This example uses curl.
```
curl --output values.yaml --silent https://docs.palette-ai.com/resources/assets/hosted/helm/values.yaml
```
2. Open the Helm chart values file in a text editor of your choice and complete the following sections. This example uses vi.
```
vi values.yaml
```
Global
Use the global section to configure overarching settings for the PaletteAI deployment. Review and modify the following values as necessary.
1. Set global.dns.domain to the primary domain for the deployment. Do not include a protocol. For example, use example.org, not https://example.org.
```
global:
  dns:
    domain: 'example.acme.org'
```
2. In global.auditLogging.basicAuth, change the default username and password for audit logging. These credentials secure the Alertmanager instance that receives audit events. You reuse them when you configure the Base64-encoded Authorization header in the alertmanager section.
```
global:
  auditLogging:
    basicAuth:
      username: '<your-username>'
      password: '<your-password>'
```
  Refer to Audit Logging to learn more about configuring audit logging, querying audit events, and forwarding logs to long-term storage.
3. Configure the metrics collection settings. Provide an existing, external Prometheus server that is reachable from the hub cluster and every spoke cluster. Spoke clusters use Prometheus agents to ship metrics to the server via remote_write.
  
  Set global.metrics.prometheusBaseUrl to the external Prometheus server URL (for example, https://your-external-prometheus:9090). Include only the protocol, host, and port — do not include any API paths.
```
global:
  metrics:
    prometheusBaseUrl: 'https://your-external-prometheus:9090'
    timeout: '5s'
    scrapeInterval: '15s'
    agentType: 'prometheus-agent-minimal'
    username: ''
    password: ''
```
  By default, global.metrics.agentType is set to prometheus-agent-minimal. The minimal agent configuration only collects spoke cluster CPU and GPU utilization metrics. You may change global.metrics.agentType to prometheus-agent to ship all node-exporter and dcgm-exporter metrics from spoke clusters for comprehensive observability.
  
  If your Prometheus server requires basic authentication, configure the username and password fields. Leave these fields blank if authentication is not required.
  
  Refer to Configure Prometheus Agent Monitoring for guidance on agent types, Prometheus and Grafana prerequisites, and GPU metrics.
  
  tip
  If you need to set up a Prometheus server, you may find the Deploy Monitoring Stack guide helpful.
Complete global configuration section

Alertmanager
Navigate to the alertmanager section. Update credentials for the alertmanager instance based on the credentials you configured in the global section.

You must provide a Base64-encoded string for the Authorization header. Use the interactive encoder to generate your Base64-encoded string and copy the value to the clipboard.

Username
Password
Base64 Encoded String:

Alternatively, generate the Base64-encoded string using the following command. Replace username and password with the username and password you configured in the global section.
```
echo -n "username:password" | base64
```
The following example shows the livenessProbe and readinessProbe sections with the Base64-encoded string. Replace <your-base64-encoded-string> with the Base64-encoded string you generated.
```
alertmanager:
  livenessProbe:
    httpGet:
      path: /-/healthy
      port: http
      scheme: HTTPS
      httpHeaders:
        - name: Authorization
          value: 'Basic <your-base64-encoded-string>'
  readinessProbe:
    httpGet:
      path: /-/ready
      port: http
      scheme: HTTPS
      httpHeaders:
        - name: Authorization
          value: 'Basic <your-base64-encoded-string>'
```
Complete alertmanager configuration section

For further instructions on accessing audit logs and configuring long-term storage, refer to Audit Logging.

Canvas
Canvas controls the user interface. Review and modify the following values as necessary.
1. To configure the ingress for Canvas, set canvas.ingress.enabled to true. Set canvas.ingress.domain to your domain name. Omit the http:// or https:// prefix.
```
canvas:
  ingress:
    enabled: true
    annotations: {}
    ingressClassName: nginx
    domain: replace.with.your.domain # No HTTP/HTTPS prefix.
    matchAllHosts: false
    tls: []
    paths:
      - path: /ai
        pathType: ImplementationSpecific
        backend:
          service:
            name: canvas
            port:
              number: 2999
```
  Set canvas.enableHTTP to true to let the load balancer terminate Transport Layer Security (TLS). Leave canvas.ingress.tls empty.
```
canvas:
  enableHTTP: true
```
2. The last portion of the Canvas configuration is the OIDC configuration. If you defer configuring OIDC for Dex, you may do the same for Canvas and configure it later.
  
  In the canvas.oidc section, enter a unique string for the sessionSecret. For redirectURL, replace <your-domain> with your domain. Do not remove the /ai/callback path.
```
canvas:
  oidc:
    sessionSecret: '<your-session-secret>'
    sessionDir: '/app/sessions'
    issuerK8sService: 'https://dex.mural-system.svc.cluster.local:5554/dex'
    skipSSLCertificateVerification: true
    redirectURL: 'https://<your-domain>/ai/callback'
```
  If you did not configure your Kubernetes cluster to trust Dex as an OIDC provider, then you must configure the canvas.impersonationProxy section to enable user impersonation.
  
  The example below shows how to configure the local Dex user admin@example.com to be mapped to an example Kubernetes group admin. Refer to our Configure User Impersonation guide to learn more about how to configure user impersonation for OIDC groups and other use cases.
  Example user impersonation setup
```
canvas:
  impersonationProxy:
    enabled: true
    userMode: 'passthrough'
    groupsMode: 'map'
    userMap: {}
    groupMap: {}
    dexGroupMap:
      'admin@example.com': [ 'admin' ]
```
  Complete canvas configuration section
Dex

Dex authenticates users to PaletteAI through SSO. You can configure Dex to connect to an upstream OIDC provider or to a local user database. For this installation, you will configure Dex to connect to an upstream OIDC provider. If you want to configure an OIDC provider later, you can do so; however, Dex still requires some basic configuration.

Set dex.config.issuer to your domain. Do not remove the /dex path.
```
dex:
  config:
   issuer: 'https://<your-domain>/dex'
```

This next part may be deferred for later, but we strongly recommend configuring at least one connector. Set the dex.config.connectors to the connectors you want to use. The Dex documentation has examples for each of the connectors.

Below is an example of an OIDC connector that connects to AWS Cognito. The oidc type can be used for any OIDC provider that does not have a native Dex connector. Different OIDC providers may require different configurations.

Example AWS Cognito configuration
dex:
  config:
    connectors:
      - type: oidc
        id: aws
        name: AWS Cognito
        config:
          issuer: https://cognito-idp.us-east-1.amazonaws.com/us-east-1_xxxxxx
          clientID: xxxxxxxxxxxxxxx
          clientSecret: xxxxxxxxxxxxxxxxx
         redirectURI: https://<your-domain>/dex/callback # Dex callback URL for the authorization code flow; redirects to the application callback URL
          getUserInfo: true
          userNameKey: email
          insecureSkipEmailVerified: true
          insecureEnableGroups: true
          scopes:
            - openid
            - email
            - profile
          promptType: consent
          claimMapping:
            groups: groups

Proceed to the dex.config.staticClients section. Replace <your-client-secret> with a unique secret value and <your-domain> with your domain. Do not remove the /ai/callback path for the mural client.

dex:
  config:
    staticClients:
      - id: mural
        redirectURIs:
          - 'https://<your-domain>/ai/callback'
        name: 'mural'
        secret: '<your-client-secret>'
        public: false
        trustedPeers:
          - kubernetes
      - id: kubernetes
        redirectURIs:
          - 'https://<your-domain>'
        name: kubernetes
        secret: '<your-client-secret>'
        public: false
        trustedPeers:
          - mural

Next, configure the dex.config.staticPasswords section. We strongly recommend changing the default user (admin) and password (password) to strong values. The following example is the default user and password in bcrypt format. Remember to use a bcrypt hash generator to generate the password hash. The userID can be any unique string.

warning
If you did not configure any OIDC connectors, you must configure at least one static user, which is used to access the PaletteAI UI. Static Dex users automatically inherit admin privileges through the service account. Dex does not support groups for local static users. To use groups for local static users, you must use the User Impersonation feature.
```
dex:
  config:
    staticPasswords:
      - email: 'admin@example.com'
        hash: '$2a$12$Ot2dJ0pmdIC2oXUDW/Ez1OIfhkSzLZIbsumsxkByuU3CUr02DtiC.'
        username: 'admin'
        userID: '08a8684b-db88-4b73-90a9-3cd1661f5466'
```

Configure the dex.ingress section to expose Dex. For host, replace <your-domain> with your domain. Do not change the className or the path. Because Transport Layer Security (TLS) is terminated at the load balancer, the tls section is empty.

dex:
  ingress:
    enabled: true
    className: 'nginx'
    annotations: {}
    hosts:
     - host: <your-domain>
        paths:
          - path: /dex
            pathType: ImplementationSpecific
    tls: []

Complete dex configuration section

Flux2

Set flux2.policies.create to false to disable the Flux network policies. These policies, if enabled, prevent ingress traffic from reaching their target services.
```
flux2:
  policies:
    create: false
```
info
This step is not required if the hub and all spoke clusters are configured to use a common, external OCI registry. An external OCI registry is configured in the fleetConfig.spokes[*].ociRegistry and hue.ociRegistry sections of the values.yaml file.

Complete flux2 configuration section

Ingress-Nginx

Configure ingress-nginx.controller.service to use the HTTP listener and terminate TLS at the load balancer. Change the value for targetPorts.https to http.

ingress-nginx:
  controller:
    service:
      targetPorts:
        http: http
        https: http

Install with Flux

Install PaletteAI with Flux to let Flux manage chart ordering and the Custom Resource Definition (CRD) lifecycle for both Helm charts.

Create mural-crds-oci-repository.yaml for the mural-crds chart.

cat << EOF > mural-crds-oci-repository.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: mural-crds
  namespace: mural-system
spec:
  interval: 10m
  ref:
    semver: "0.7.0-hotfix.4"
  url: oci://public.ecr.aws/mural/mural-crds
EOF

Create mural-oci-repository.yaml for the mural chart.

cat << EOF > mural-oci-repository.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: mural
  namespace: mural-system
spec:
  interval: 10m
  ref:
    semver: "1.0.7"
  url: oci://public.ecr.aws/mural/mural
EOF

Apply both OCIRepository resources to your cluster.

kubectl apply --filename mural-crds-oci-repository.yaml
kubectl apply --filename mural-oci-repository.yaml

Create mural-crds-helm-release.yaml for the mural-crds chart.

cat <<'EOF' > mural-crds-helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mural-crds
  namespace: mural-system
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: mural-crds
    namespace: mural-system
  install:
    crds: Create
  upgrade:
    crds: CreateReplace
EOF

Create mural-helm-release.yaml for the mural chart. The dependsOn field ensures that Flux installs mural-crds before mural.

cat <<'EOF' > mural-helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: mural
  namespace: mural-system
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: mural
    namespace: mural-system
  dependsOn:
    - name: mural-crds
  values:
    # Paste the contents of your values.yaml file here.
EOF

Open mural-helm-release.yaml and replace the placeholder comment under spec.values with the contents of the values.yaml file for your environment. Keep the inserted YAML indented under spec.values.

Apply both HelmRelease resources to your cluster.

kubectl apply --filename mural-crds-helm-release.yaml
kubectl apply --filename mural-helm-release.yaml

Install with Helm

warning

If you do not use Flux, manage the mural-crds chart separately from the mural chart. Apply or upgrade Custom Resource Definitions (CRDs) out of band before you install or upgrade the mural chart. For the manual Helm workflow, refer to Upgrade Manually.

Install the mural-crds Helm chart first.

helm install mural-crds oci://public.ecr.aws/mural/mural-crds --version 0.7.0-hotfix.4 \
  --namespace mural-system --create-namespace --wait

Example Output
NAME: mural-crds
LAST DEPLOYED: Tue May 27 09:34:33 2025
NAMESPACE: mural-system
STATUS: deployed
REVISION: 1

Install PaletteAI from the mural chart by using your environment's values.yaml file.

helm install mural oci://public.ecr.aws/mural/mural --version 1.0.7 \
  --namespace mural-system --create-namespace --values values.yaml --wait

Example Output
NAME: mural
LAST DEPLOYED: Tue May 27 09:39:48 2025
NAMESPACE: mural-system
STATUS: deployed
REVISION: 1

Configure DNS

Once PaletteAI is deployed, fetch the EXTERNAL-IP of the load balancer deployed by ingress-nginx-controller.

kubectl get service ingress-nginx-controller --namespace mural-system

Example output
NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.104.129.101   a9d221d65b2fd41b3929574458e8ce05-1177779699.us-east-1.elb.amazonaws.com   80:31952/TCP,443:30926/TCP   41m

Create a DNS record pointing your canvas.ingress.domain configured in values.yaml to the load balancer created by the ingress-nginx controller. Use an A record for IP addresses or a CNAME/alias record for hostnames, depending on your DNS provider's capabilities.

You have now deployed PaletteAI on your Kubernetes cluster. The cluster is configured to trust Dex as an identity provider. If you configured Dex with an OIDC connector, you can now log in to PaletteAI by using your identity provider (IdP). You can also use the default Dex local user.

If you need to update PaletteAI later, review Helm Chart Configuration and then follow Upgrade PaletteAI. For Flux-managed installations, update the mural HelmRelease or the relevant OCIRepository resource and let Flux reconcile the change. For manual Helm installations, follow Upgrade Manually and apply mural-crds out of band before mural.

Validate

Take the following steps to verify that PaletteAI is deployed and configured correctly.

Open a browser and navigate to the domain URL you configured for PaletteAI.
Log in with the default username and password. If you configured Dex with an OIDC connector, log in with your identity provider.

Next Steps

Once you have installed PaletteAI, integrate Palette with PaletteAI by configuring the Settings resource. This resource requires a Palette tenant, project, and API key so PaletteAI can communicate with Palette and deploy AI/ML applications and models to the correct location.

Proceed to the Integrate with Palette guide to learn how to prepare your Palette environment.

Prerequisites​

Enablement​

Global​

Alertmanager​

Canvas​

Dex​

Flux2​

Ingress-Nginx​

Install with Flux​

Install with Helm​

Configure DNS​

Validate​

Next Steps​

Prerequisites

Enablement

Global

Alertmanager

Canvas

Dex