Skip to main content

VM diagnostic status

When the VM diagnostic status shows Unhealthy or Unknown, follow the steps below. For an overview of the statuses themselves, see Compute dashboard.

Unknown​

The monitoring system can't receive status information from the VM.

Step 1: Check the monitoring agent​

Verify that eci-guest-agent is running inside the VM:

sudo systemctl is-active eci-guest-agent.service
  • Output active: the service is healthy. If the status hasn't changed after 5 minutes, go to step 2
  • Output inactive or failed: restart it
sudo systemctl restart eci-guest-agent.service

After restarting, re-run is-active. Detailed logs are available from sudo systemctl status eci-guest-agent.service; Active: active (running) indicates a healthy service.

Step 2: Check host communication​

If the agent is healthy but the status stays Unknown, test communication with the host directly.

sudo systemctl stop eci-guest-agent.service
socat VSOCK-LISTEN:11190 STDOUT
Restart the service after the test

If you don't stop the agent before testing, you'll see Address already in use. After the test, start the service again to resume monitoring.

sudo systemctl start eci-guest-agent.service
  • Healthy: after the command runs, the cursor waits and after a moment a request like {"command": "cpu-metric", "args": null} arrives.
  • Unhealthy: an error is printed immediately.

Step 3: Contact support​

If the steps above don't resolve it, contact Support with:

  • Service logs: sudo journalctl -eu eci-guest-agent.service --no-pager | tail
  • The error message from the communication test (if applicable)

Unhealthy​

The monitoring system can't properly detect the GPU allocated to the VM, or the GPU is not operating correctly.

Step 1: Check that the GPU hardware is detected​

lspci | grep -i nvidia

If no NVIDIA device is listed, the hardware isn't detected.

Step 2: Check the nvidia-smi binary​

command -v nvidia-smi
  • A path is printed (e.g. /usr/bin/nvidia-smi): the binary is fine. Skip directly to Step 3.
  • Nothing is printed: nvidia-smi wasn't found. GPU metrics can't be collected in this case, so the diagnostic status stays Unhealthy. The binary may have been removed or its path may have changed — if this wasn't intentional, see the GPU driver FAQ to restore it.
Don't remove nvidia-smi or change its path

nvidia-smi is also used for metric collection. If you remove it or it can't run because its path changed, GPU metrics can't be collected until it's restored.

Step 3: Check driver / CUDA compatibility​

If nvidia-smi exists but fails when you run it (it doesn't work even though you didn't remove it), check NVIDIA driver vs CUDA Toolkit compatibility. For CUDA initialize failed, nvidia-smi failures, and driver / library version mismatches, see the GPU driver FAQ and PyTorch CUDA compatibility issues.

Step 4: Contact support​

If the issue persists, contact Support with:

  • Output of lspci | grep -i nvidia
  • Output of command -v nvidia-smi and nvidia-smi
  • The CUDA Toolkit version in use and the full error text

Next steps​