Skip to main content

Alerts

Overview​

Alerts automatically send a notification when a metric — GPU utilization, memory, disk, etc. — crosses a threshold you've set. They are used to detect training completion, catch problems early, and prevent idle-resource cost leaks.

An alert is made up of three components:

┌─────────────┐ fires ┌────────────────┐ runs ┌─────────┐
│ Alert rule │ ───────▶ │ Alert event │ ──────▶ │ Action │
└─────────────┘ └────────────────┘ └─────────┘
Condition History entry Send email
ComponentRoleWhere to find it
Alert ruleDefines the metric threshold and evaluationMonitoring > Alert > Alert Rules tab
ActionDefines who is notified and how when the rule firesMonitoring > Alert > Actions tab
Alert eventThe history of when the rule firedAlert rule detail > Alert Events tab
Prerequisites
  • Alert.AlertRule.CREATE, Alert.AlertActionTemplate.CREATE permissions. Among the built-in roles, Reader can only view alerts.
  • Create the action first, before creating an alert rule

Evaluation state and rule state​

An alert rule has two kinds of state.

Evaluation stateMeaning
OkThe metric is within the threshold
AlertThreshold exceeded; an alert has fired
No DataNot enough data to evaluate
Rule stateMeaning
ActivatedPeriodically evaluated; fires when conditions are met
PausedEvaluation is suspended; no alerts will fire (settings are kept)

Step 1: Create an action​

First, create the action (currently email) that fires when an alert triggers.

  1. In Monitoring > Alert, open the Actions tab and click Create Action.

  2. Fill in:

    FieldDescription
    Action NameIdentifier (e.g. GPU failure alert)
    Action TypeEmail (webhooks coming soon)
    Recipient EmailsAdd users via search, or enter comma-separated addresses (at least one)
  3. Click Create.

Only email actions are supported today

Integrations with Slack webhooks and other external services are planned.


Step 2: Create an alert rule​

  1. On the Alert Rules tab of Monitoring > Alert, click Create Alert Rule.
  2. Fill in the following:

Basic info​

FieldExample
NameGPU utilization warning
DescriptionAlert when GPU utilization exceeds 90%

Target resource​

FieldDescription
Resource TypeVirtual Machine or Object Storage
ResourceThe VM or bucket to monitor
MetricDepends on the resource type (see below)

VM metrics

CategoryMetric
GPUUtilization (%), memory used/total (MiB), temperature (°C), power (W), clock speed (MHz)
CPUUser · System · Idle · I/O Wait · IRQ · Soft IRQ · Steal · Guest · Nice (%)
MemoryTotal · Available · Active · Inactive · Cached · Buffers · Free (KiB) · Swap
NetworkRX/TX bandwidth (Bps) · bytes · packets · drops · errors
StorageUsage (Bytes) · read/write throughput (Bps) · IOPS · latency (sec)

Object storage metrics

MetricDescription
UsageBucket size in bytes
Object countNumber of objects in the bucket
GET / PUT / COPY / LISTAPI request counts
Multipart uploadsNumber of multipart upload operations

Evaluation conditions​

FieldDescription
Aggregation Methodavg / sum / min / max (AlertAggEnum)
Aggregation Period1m / 5m / 15m / 30m / 1h
Operator> (greater than) / >= (greater than or equal) / < (less than) / <= (less than or equal)
ThresholdThe value that triggers the alert (e.g. 5, 90, 0.85)
Evaluation DatapointsHow many recent datapoints to inspect (e.g. 5): datapoints_to_evaluate
Alert DatapointsHow many of those must breach to fire (e.g. 3): datapoints_to_alert

Example: interval 5m, evaluate 5, alert 3 → measure every 5 minutes over the last 25 minutes; if 3 or more breach the threshold, fire an alert.

Only completed intervals are evaluated

The system aggregates data over each interval into one value before evaluating. An in-progress interval has incomplete data and is excluded from evaluation.

For example, with a 5-minute interval and a 80% GPU utilization threshold, suppose the 10:15–10:20 interval just started and GPU briefly spiked to 95%. The breach is not counted until the 5-minute window closes and the actual average is computed. If that average turns out to be 72%, the interval is classified as Ok.

Preview diagram​

The Evaluation Preview diagram in the form shows how your conditions will be evaluated and updates live as you change them.

  • Green dots: datapoints within the threshold
  • Yellow dots: datapoints that breached the threshold
  • Red dots: points where the alert actually fired (datapoints-to-alert reached)
  • Dashed line: the threshold

Alert rule creation page with the evaluation preview

Pick the action you just created (you can select more than one).

  1. Click Create.

Use-case walkthroughs​

GPU overload alert​

  1. In Monitoring > Alert > Actions, create an email action with recipients.
  2. On the Alert Rules tab, click Create Alert Rule.
  3. Pick the VM to monitor as the target resource.
  4. Set the metric to GPU Utilization, operator >, threshold 90.
  5. Set the evaluation: interval 5 minutes, evaluate 5, alert 3.
  6. Check the Evaluation Preview diagram to confirm the rule behaves as intended.
  7. Select the email action and click Create.

Object storage capacity alert​

  1. If you don't have one already, create an email action first.
  2. In Create Alert Rule, set the resource type to Object Storage.
  3. Pick the bucket to monitor.
  4. Set the metric to Usage, operator >=, threshold to your target capacity (Bytes).
  5. Set evaluation and action, then click Create.

Investigating an alert​

  1. On the Alert Rules tab of Monitoring > Alert, click a rule whose State is Alert.
  2. On the Alert Events tab, check recent state transitions and when they occurred.
  3. On the Metric Chart tab, set the time range to match the time of occurrence to see the metric at that moment.
  4. Once you've identified the cause, take the appropriate action.

Common alert rules​

GoalMetricAgg / intervalOperatorThresholdEvaluate / Alert
Detect training completionGPU utilizationavg / 5m<52 / 2
GPU OOM riskGPU memory usagemax / 1m>903 / 3
Detect VM abnormal shutdownCPU utilizationavg / 5m<11 / 1
Disk-full riskDisk utilizationmax / 5m>851 / 1
Idle GPU (cost leak)GPU utilizationavg / 1h<301 / 1

Checking alert events​

When an alert rule's evaluation state changes, a history entry is recorded automatically. There is no separate page that lists events on their own; check each rule's history on the Alert Events tab of the alert rule detail page.

ColumnDescription
State TransitionOk → Alert / Alert → Ok, etc.
Occurred AtWhen the state changed

Managing alert rules​

The rule list is read-only; its Active column shows whether each rule is activated. Make changes on the detail page that opens when you click a rule.

  • Activate / Pause: use the switch at the top of the detail page (the rule itself is kept). Requires the Alert.AlertRule.UPDATE permission.
  • Edit: change thresholds, conditions, or actions on the Settings tab and Save.
  • Delete: delete the rule at the bottom of the Settings tab. A deleted rule cannot be restored.

A paused rule isn't evaluated and produces no alert events. Pausing rules tied to VMs that are under maintenance is a good way to avoid false alerts.

Rule detail page tabs​

Click a rule in the list to open the detail page, which has five tabs:

TabContents
OverviewCurrent state, last update time, a natural-language summary of the condition, target resource (click to jump to the resource page)
Metric ChartChart of the monitored metric with the threshold. Time range: Last 30 minutes through Last 30 days, or Custom
Alert ActionsAlert actions linked to this rule. Click to jump to the action's detail page
Alert EventsState-transition history (Ok → Alert, Alert → Ok) and when each occurred
SettingsEdit and delete the rule

The action detail page has Overview, Used by rules, and Settings tabs; edit recipient emails and delete the action on the Settings tab.

Settings tab on the alert rule detail page


FAQ​

Evaluation state shows "No Data"

  • Confirm the target VM is in the Running state
  • Confirm the rule is Activated (paused rules are not evaluated)
  • Right after creating a rule, No Data can appear briefly until enough data is collected

I'm not receiving alert emails

  • Confirm the rule has an action attached
  • Confirm the action's recipient email is correct
  • Check your spam / junk folder

What's the difference between "datapoints to evaluate" and "datapoints to alert"?

  • Evaluation Datapoints: how many recent datapoints to look at (the window size)
  • Alert Datapoints: how many of those must breach the threshold to fire (the trigger condition)
  • Setting them equal means every point must breach (strict). Setting "alert" to 1 means a single breach is enough (sensitive)

Can I delete an action?

Deleting an action that other rules are using means those rules no longer send notifications. Check the Used by rules tab on the action detail page first, then delete the action at the bottom of the Settings tab.


Next steps​