Skip to content

M #-: Add notes GPU monitoring #327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Parent: F OpenNebula/engineering#408 AI Factories: New Configuration and Deployment page
wants to merge 2 commits into from
Closed

M #-: Add notes GPU monitoring #327

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1731df0
M #-: Add notes GPU monitoring
josepselga Oct 2, 2025
b3ba738
M #-: Add notes GPU monitoring (MIG vGPU caution note)
josepselga Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions content/product/cloud_system_administration/resource_monitoring/metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,12 @@ The MONITOR information includes the following data:

The metrics above are directly read from and stored in the monitoring database.

{{< alert title="GPU Monitoring Notes" color="success" >}}
- GPU metrics represent combined values across all detected GPUs. This may be a sum or average depending on the metric.
- GPU load can spike very quickly (e.g. during inference). Since probes run at configurable intervals (MONITOR_VM, default 30s in `monitord.conf`), short spikes may not be captured. Lower the interval to capture more details. You can check monitoring configuration [here]({{% relref "../../../product/cloud_system_administration/resource_monitoring/monitoring_system" %}}).
- GPU monitoring metrics may not behave as expected when using MIG or vGPU configurations. In such cases interpretation of the values requires caution.
{{< /alert >}}

Additionally, the following derived metrics are calculated from the stored metrics and used for forecasting. These derived metrics are not stored in the database but are computed on-demand:

| Key | Description |
Expand Down