Skip to main content

Operating spot VMs

Overview​

Spot VMs let you run on ECI's idle GPU capacity at a discount versus on-demand. The discount varies with supply and demand, and a VM may be reclaimed when capacity tightens, so checkpointing and reclamation detection are mandatory.

Spot is offered only to Elice-integrated organizations

In the portal of an organization that is not Elice-integrated, the Spot pricing type and the Spot GPU menu are not shown. For organization types, see Sign up and sign in.


Creating a spot VM​

  1. Under Compute > Virtual Machine, click Create Virtual Machine.
  2. On the Basics step, set the Pricing Type to Spot.
  3. In the instance-type list, pick a GPU type whose availability is {n} currently available.
  4. Fill in the remaining settings and create the VM.

Checking availability​

Spot uses idle GPU capacity, so availability fluctuates frequently.

Option 1: From the VM creation screen

Once you select the spot pricing type, availability is shown inline next to each instance type.

AvailabilityDescription
{n} currently availableCapacity is secured; the VM can start right away
No resources currently availableYou can still create the VM, but starting it waits until capacity is secured

Option 2: The Spot GPU menu

Spot GPU in the left-hand menu shows spot availability and prices for each instance type at a glance.


When capacity is short: waiting to start​

If you start a VM while spot capacity is short, the request is not rejected — it waits. The VM detail page shows a Waiting for spot resources notice with the elapsed time, and the VM starts automatically once capacity is secured. Depending on demand, the wait can be long.

If you need to start right away, click Check available resources in the notice, pick another spec that can start now (spot, on-demand, or reserved), then click Change and start. The wait is cancelled, the VM's instance type is replaced with the selected resource, and the VM starts.


Reclamation​

When capacity runs short, ECI force-reclaims spot VMs.

Reclamation process​

  1. Once reclamation is decided, the metadata API exposes the scheduled termination time.
  2. After a 1–2 minute grace period, the VM is force-reclaimed.

Detecting reclamation: the metadata API​

From inside the VM, the command below tells you whether reclamation is scheduled.

curl -s --unix-socket /run/eci-guest-agent.sock \
http://localhost/vm/metadata?key=spot_termination_time
  • No reclamation scheduled: 404 page not found
  • Reclamation scheduled: returns the termination time (e.g. "2026-04-08T05:00:00+00:00")

Reclamation watcher script (run in background)​

#!/bin/bash
# spot_watcher.sh

while true; do
HTTP_CODE=$(curl -s -o /tmp/spot_response -w "%{http_code}" \
--unix-socket /run/eci-guest-agent.sock \
http://localhost/vm/metadata?key=spot_termination_time)

if [ "$HTTP_CODE" -eq 200 ]; then
echo "[$(date)] Reclamation scheduled: $(cat /tmp/spot_response)"
/path/to/save_checkpoint.sh # call your checkpoint script
break
fi
sleep 5
done
chmod +x spot_watcher.sh
nohup ./spot_watcher.sh &

Saving checkpoints​

Data inside the VM is lost on reclamation

Anything in VM memory or local ephemeral storage disappears when reclamation happens. Always save checkpoints to block storage on a regular schedule.

PyTorch checkpoint example​

import torch, os

CHECKPOINT_PATH = "/data/checkpoints/checkpoint.pt"

def save_checkpoint(model, optimizer, epoch, step, loss):
os.makedirs(os.path.dirname(CHECKPOINT_PATH), exist_ok=True)
torch.save({
'epoch': epoch,
'step': step,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'loss': loss,
}, CHECKPOINT_PATH)
print(f"[epoch {epoch}] checkpoint saved")

def load_checkpoint(model, optimizer):
if os.path.exists(CHECKPOINT_PATH):
ckpt = torch.load(CHECKPOINT_PATH)
model.load_state_dict(ckpt['model_state_dict'])
optimizer.load_state_dict(ckpt['optimizer_state_dict'])
return ckpt['epoch'], ckpt['step']
return 0, 0 # start from scratch

# Training loop
start_epoch, start_step = load_checkpoint(model, optimizer)
for epoch in range(start_epoch, total_epochs):
for step, batch in enumerate(dataloader, start=start_step):
train_step(model, batch)
if step % 100 == 0:
save_checkpoint(model, optimizer, epoch, step, loss) # save every 100 steps
Saving only on reclamation detection can be too late

The grace period (1–2 minutes) may not be enough to finish writing. Combine reclamation detection with regular (every N steps) checkpointing.


Limitations​

  • Applies only to GPU instance types. CPU-only instances do not have a spot option, and attached block storage, public IPs, and networking are billed at standard rates
  • Cannot be joined to a virtual cluster
  • The Always On option cannot be used
Transitioning an existing VM to spot

An existing VM can be switched to spot (or from spot to another type) while it is stopped. A VM that belongs to a virtual cluster cannot be switched to spot. Make the change on the VM detail page under the Settings tab > Pricing · instance type; for the detailed conditions, see Changing pricing type and instance type.


FAQ​

My VM shut down out of nowhere.

It was reclaimed because of spot capacity pressure. Set up metadata-API polling and save checkpoints regularly so you can resume after restarting.

I started the VM, but it keeps waiting.

Spot capacity is currently short, so the VM is waiting to start. It starts automatically once capacity is secured. To start right away, switch to another spec from Check available resources in the Waiting for spot resources notice. See When capacity is short: waiting to start for details.

The spot price changed.

The spot price is applied at VM start time, so a restart may pick up a different price.


Next steps​