Troubleshoot AI Virtual Machines
When an AI Virtual Machine does not reach Running, the reason is reported in the conditions on the resource. Inspect them first:
kubectl describe aivirtualmachine <vm-name> --namespace <project-namespace>
A failure sets the virtual machine's state to Error and names the cause in a condition message.
Failure Conditions
| Condition | Reason | Meaning |
|---|---|---|
| VMCapableComputePool | NotCapable | The referenced Compute Pool is missing, or its infrastructure kind is not vm. |
| VMCapableComputePool | NotDedicated | The Compute Pool is shared or spans more than one cluster. |
| TemplateValid | TemplateInvalid | The referenced template cannot be used by this virtual machine. |
| Rendered | RenderFailed | The controller could not build the VirtualMachine manifest from the template and your overrides. |
| MachinePoolValid | MachinePoolNotFound | The named machine pool does not exist on the Compute Pool. |
| MachinePoolValid | MachinePoolNotEligible | The machine pool exists, but its nodes cannot run virtual machines. |
| Shipped | ShipFailed | The manifest did not reach the Compute Pool's cluster. |
AI Virtual Machine Reports Unschedulable
Cause: no node in the selected machine pool fits the request right now.
Resolution: this is a wait, not a fault. The virtual machine starts on its own once capacity frees up. To run it on a different machine pool, delete the virtual machine and create a new one, because the machine pool cannot change once a virtual machine is deployed.
Render Fails with a Sizing Error
Cause: a quantity is missing its unit, or uses a unit PaletteAI does not accept.
Resolution: supply a unit, for example 32 GB rather than 32. Supported units are KB/k, KiB/ki, MB/m, MiB/mi, GB/g, GiB/gi, TB/t, and TiB/ti. Refer to Create an AI Virtual Machine.
Template Is Not Usable
Cause: spec.template.name does not name one of the built-in templates, so the controller has nothing to render.
Resolution: use ubuntu-22-04 or fedora-37. These are the templates PaletteAI offers, and they are available on every Compute Pool that runs AI Virtual Machines.
Machine Pool Cannot Run Virtual Machines
Cause: spec.machinePool names a pool the Compute Pool does not have, or a control plane pool whose nodes keep the control plane taint, so the virtual machine never schedules.
Resolution: on a multi-node Compute Pool, select a worker pool. On a single-node Compute Pool, use the control plane machine pool, which is the only one and is always eligible.
Two condition messages map to this section:
machine pool <name> does not exist on this ComputePool
machine pool <name> cannot run VMs
Request Exceeds the Compute Pool Limits
Cause: the request for CPU, memory, or GPUs is above what the Compute Pool allows for a single virtual machine.
Resolution: lower the request to fit the machine pool and apply again. Read the ceiling for each machine pool from the Compute Pool's status.vmCapacityByMachinePool.
SSH Rejects the Password
Cause: the built-in templates expire the pai password, so it stops working after the first sign-in, when a new password was set.
Resolution: use the password you set, or connect over VNC and set a new one. Refer to Change AI Virtual Machine Credentials.
Next Steps
- Manage an AI Virtual Machine: start, stop, clone, or delete it.
- Access an AI Virtual Machine: connect over SSH or VNC.
- AI Virtual Machines: the resource and its states.