Operating spot VMs
Overview
Spot VMs let you run on ECI's idle GPU capacity at a discount versus on-demand. The discount varies with supply and demand, and a VM may be reclaimed when capacity tightens, so checkpointing and reclamation detection are mandatory.
In the portal of an organization that is not Elice-integrated, the Spot pricing type and the Spot GPU menu are not shown. For organization types, see Sign up and sign in.
Creating a spot VM
- Under Compute > Virtual Machine, click Create Virtual Machine.
- On the Basics step, set the Pricing Type to Spot.
- In the instance-type list, pick a GPU type whose availability is
{n}currently available. - Fill in the remaining settings and create the VM.
Checking availability
Spot uses idle GPU capacity, so availability fluctuates frequently.
Option 1: From the VM creation screen
Once you select the spot pricing type, availability is shown inline next to each instance type.
| Availability | Description |
|---|---|
{n} currently available | Capacity is secured; the VM can start right away |
| No resources currently available | You can still create the VM, but starting it waits until capacity is secured |
Option 2: The Spot GPU menu
Spot GPU in the left-hand menu shows spot availability and prices for each instance type at a glance.
When capacity is short: waiting to start
If you start a VM while spot capacity is short, the request is not rejected — it waits. The VM detail page shows a Waiting for spot resources notice with the elapsed time, and the VM starts automatically once capacity is secured. Depending on demand, the wait can be long.
If you need to start right away, click Check available resources in the notice, pick another spec that can start now (spot, on-demand, or reserved), then click Change and start. The wait is cancelled, the VM's instance type is replaced with the selected resource, and the VM starts.
Reclamation
When capacity runs short, ECI force-reclaims spot VMs.
Reclamation process
- Once reclamation is decided, the metadata API exposes the scheduled termination time.
- After a 1–2 minute grace period, the VM is force-reclaimed.
Detecting reclamation: the metadata API
From inside the VM, the command below tells you whether reclamation is scheduled.
curl -s --unix-socket /run/eci-guest-agent.sock \
http://localhost/vm/metadata?key=spot_termination_time
- No reclamation scheduled:
404 page not found - Reclamation scheduled: returns the termination time (e.g.
"2026-04-08T05:00:00+00:00")
Reclamation watcher script (run in background)
#!/bin/bash
# spot_watcher.sh
while true; do
HTTP_CODE=$(curl -s -o /tmp/spot_response -w "%{http_code}" \
--unix-socket /run/eci-guest-agent.sock \
http://localhost/vm/metadata?key=spot_termination_time)
if [ "$HTTP_CODE" -eq 200 ]; then
echo "[$(date)] Reclamation scheduled: $(cat /tmp/spot_response)"
/path/to/save_checkpoint.sh # call your checkpoint script
break
fi
sleep 5
done
chmod +x spot_watcher.sh
nohup ./spot_watcher.sh &
Saving checkpoints
Anything in VM memory or local ephemeral storage disappears when reclamation happens. Always save checkpoints to block storage on a regular schedule.
PyTorch checkpoint example
import torch, os
CHECKPOINT_PATH = "/data/checkpoints/checkpoint.pt"
def save_checkpoint(model, optimizer, epoch, step, loss):
os.makedirs(os.path.dirname(CHECKPOINT_PATH), exist_ok=True)
torch.save({
'epoch': epoch,
'step': step,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'loss': loss,
}, CHECKPOINT_PATH)
print(f"[epoch {epoch}] checkpoint saved")
def load_checkpoint(model, optimizer):
if os.path.exists(CHECKPOINT_PATH):
ckpt = torch.load(CHECKPOINT_PATH)
model.load_state_dict(ckpt['model_state_dict'])
optimizer.load_state_dict(ckpt['optimizer_state_dict'])
return ckpt['epoch'], ckpt['step']
return 0, 0 # start from scratch
# Training loop
start_epoch, start_step = load_checkpoint(model, optimizer)
for epoch in range(start_epoch, total_epochs):
for step, batch in enumerate(dataloader, start=start_step):
train_step(model, batch)
if step % 100 == 0:
save_checkpoint(model, optimizer, epoch, step, loss) # save every 100 steps
The grace period (1–2 minutes) may not be enough to finish writing. Combine reclamation detection with regular (every N steps) checkpointing.
Limitations
- Applies only to GPU instance types. CPU-only instances do not have a spot option, and attached block storage, public IPs, and networking are billed at standard rates
- Cannot be joined to a virtual cluster
- The Always On option cannot be used
An existing VM can be switched to spot (or from spot to another type) while it is stopped. A VM that belongs to a virtual cluster cannot be switched to spot. Make the change on the VM detail page under the Settings tab > Pricing · instance type; for the detailed conditions, see Changing pricing type and instance type.
FAQ
My VM shut down out of nowhere.
It was reclaimed because of spot capacity pressure. Set up metadata-API polling and save checkpoints regularly so you can resume after restarting.
I started the VM, but it keeps waiting.
Spot capacity is currently short, so the VM is waiting to start. It starts automatically once capacity is secured. To start right away, switch to another spec from Check available resources in the Waiting for spot resources notice. See When capacity is short: waiting to start for details.
The spot price changed.
The spot price is applied at VM start time, so a restart may pick up a different price.
Next steps
- Terraform spot VM guide: automating spot VMs as IaC
- Pricing model: spot vs on-demand vs reserved