Skip to main content

VM health status

When the VM health status shows Unhealthy or Unknown, follow the steps below. For an overview of the statuses themselves, see Check VM status from Home.


Unknown​

The monitoring system can't receive status information from the VM.

Step 1: Check the monitoring agent​

Verify that eci-guest-agent is running inside the VM:

sudo systemctl is-active eci-guest-agent.service
  • Output active: the service is healthy. If the status hasn't changed after 5 minutes, go to step 2
  • Output inactive or failed: restart it
sudo systemctl restart eci-guest-agent.service

After restarting, re-run is-active. Detailed logs are available from sudo systemctl status eci-guest-agent.service; Active: active (running) indicates a healthy service.

Step 2: Check host communication​

If the agent is healthy but the status stays Unknown, test communication with the host directly.

sudo systemctl stop eci-guest-agent.service
socat VSOCK-LISTEN:11190 STDOUT
Restart the service after the test

If you don't stop the agent before testing, you'll see Address already in use. After the test, start the service again to resume monitoring.

sudo systemctl start eci-guest-agent.service
  • Healthy: after the command runs, the cursor waits and after a moment a request like {"command": "cpu-metric", "args": null} arrives.
  • Unhealthy: an error is printed immediately.

Step 3: Contact support​

If the steps above don't resolve it, contact Support with:

  • Service logs: sudo journalctl -eu eci-guest-agent.service --no-pager | tail
  • The error message from the communication test (if applicable)

Unhealthy​

The monitoring system can't properly detect the accelerator (GPU or NPU) allocated to the VM, or the accelerator is not operating correctly. Work through the steps below in order; at each step, check only the item that matches the type of accelerator allocated to your VM.

Step 1: Check that the accelerator hardware is detected​

Use the command for your accelerator type to check whether the device is detected on the PCI bus.

NVIDIA GPU

lspci | grep -i nvidia

Furiosa NPU (Warboy, Renegade)

lspci | grep -i furiosa

Rebellions NPU (ATOM Plus, Max)

lspci | grep -i rebellions

If the device isn't listed, the hardware isn't detected.

Step 2: Check the management tool​

Accelerator metrics are collected with the management tool each vendor provides (the SMI family such as nvidia-smi, rbln-smi, and furiosa-smi, or furiosactl). Check that the tool is installed and on the executable path.

Don't remove the management tool or change its path

The management tool for each accelerator — nvidia-smi, furiosa-smi (or furiosactl), rbln-smi — is also used for metric collection. If you remove it or it can't run because its path changed, metrics for that accelerator can't be collected until it's restored, and the health status stays Unhealthy.

NVIDIA GPU

command -v nvidia-smi

If a path is printed (e.g. /usr/bin/nvidia-smi), the tool is fine. If nothing is printed, it has been removed or its path has changed — if this wasn't intentional, see the GPU driver FAQ to restore it.

Furiosa NPU

command -v furiosa-smi furiosactl

The latest Furiosa SDK (Renegade and others) uses furiosa-smi, and legacy Warboy environments use furiosactl. If nothing is printed, the SDK isn't installed or its path has changed, so reinstall the Furiosa SDK.

Rebellions NPU

command -v rbln-smi

If no path is printed, the RBLN driver and SDK aren't installed or the path has changed, so reinstall the RBLN SDK.

Step 3: Check device status and drivers​

If the management tool is present but the health status is still Unhealthy, check the device status and the driver.

NVIDIA GPU

Check that the GPUs are detected and working, and that they are registered with the NVLink fabric.

nvidia-smi --query-gpu=index,fabric.state,fabric.status --format=csv
  • If the command doesn't run or returns an error, it is an NVIDIA driver / CUDA Toolkit version compatibility problem. For CUDA initialize failed, nvidia-smi failures, and driver / library version mismatches, see the GPU driver FAQ and PyTorch CUDA compatibility issues.
  • On NVLink (NVSwitch) based GPUs (H100, B200, and others), fabric.state must be Completed and fabric.status must be Success. Any other value means fabric initialization hasn't finished, so reboot the VM to reinitialize it. (On GPUs that don't use NVLink, N/A is normal.)

Furiosa NPU

Check that every allocated NPU is listed and in the healthy (alive) state.

furiosa-smi info --format=json # legacy: furiosactl info --format=json

Each device is healthy when its alive value is 1. If alive isn't 1 or a device is missing, recover by reinstalling the Furiosa SDK or rebooting the VM.

Rebellions NPU

Check that every allocated NPU is listed, and check the status and pstate values of each device.

rbln-smi --query -qd npu,device,status,pstate
  • If pstate shows P12, the NPU is in the System Abort (hang) state. Stop the running workloads, then reboot the VM to recover.
  • If a device isn't listed, reinstall the RBLN driver and SDK or reboot the VM.

Step 4: Contact support​

If the steps above don't resolve it, contact Support with:

  • Output of lspci (filtered for the accelerator)
  • The management tool path check result and the tool's output (nvidia-smi / furiosa-smi, furiosactl / rbln-smi)
  • The full error text (for NVIDIA, include the CUDA Toolkit version in use)

Next steps​