Iñigo GonzálezIT Operations · Infra

GuíasGuides · Nivel 3Level 3

Grafana: un origen de datos, el primer panel y una alerta que avisa al canal del equipoGrafana: a data source, your first dashboard and an alert that notifies the team channel

Prometheus con node_exporter en los servidores, el panel de disco y CPU que mira todo el equipo, y la regla de alerta que llega a Teams antes de que llame el usuario.Prometheus with node_exporter on the servers, the disk and CPU dashboard the whole team looks at, and the alert rule that reaches Teams before the user calls.

Nivel 3Level 3 · GrafanaGrafana · 4 min4 min read · ActualizadoUpdated 22 oct 202622 Oct 2026

Qué vas a conseguir

Un panel con el estado de los servidores (disco, CPU, memoria, servicios caídos) que se abre en la pantalla de la sala del equipo, y una alerta de "disco por encima del 85 %" que llega al canal de Teams con el nombre del servidor y el valor. Lo que se mide se arregla antes.

Antes de empezar

  • El Grafana de la guía de Docker ya en marcha (o uno nuevo con el mismo Compose).
  • Puerto 9100 abierto en los servidores hacia el servidor de herramientas (node_exporter) y 9090 solo en local.
  • Un webhook entrante en el canal de Teams del equipo (Workflows → "Post to a channel when a webhook request is received"); la URL es un secreto.

1. Prometheus y node_exporter

Grafana pinta; alguien tiene que recoger. Añade Prometheus al Compose y el exportador a los servidores:

docker-compose.yml (fragmento)
  prometheus:    image: prom/prometheus:v2.55.0    volumes:      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro      - prom-data:/prometheus    command: ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.retention.time=90d"]    ports: ["127.0.0.1:9090:9090"]
prometheus.yml
global:  scrape_interval: 30sscrape_configs:  - job_name: linux    static_configs:      - targets:          - srv-web-01.contoso.local:9100          - srv-web-02.contoso.local:9100          - srv-db-01.contoso.local:9100        labels: { site: madrid }
bash · srv-web-01 (o con Ansible en todos)
sudo useradd -rs /bin/false node_exportercurl -sL https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz | sudo tar xz -C /usr/local/bin --strip-components=1 --wildcards '*/node_exporter'sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'[Unit]Description=Prometheus Node ExporterAfter=network.target[Service]User=node_exporterExecStart=/usr/local/bin/node_exporter --collector.systemd[Install]WantedBy=multi-user.targetEOFsudo systemctl enable --now node_exportersudo firewall-cmd --add-port=9100/tcp --permanent && sudo firewall-cmd --reloadcurl -s localhost:9100/metrics | grep -E '^node_filesystem_avail_bytes.*mountpoint="/"'node_filesystem_avail_bytes{device="/dev/mapper/rhel-root",fstype="xfs",mountpoint="/"} 2.0401152e+10

En http://localhost:9090/targets (desde el servidor de herramientas, o por túnel SSH) los tres deben estar en UP. Si uno está en DOWN con connection refused, es el firewall del servidor; con context deadline exceeded, DNS o ruta.

2. El origen de datos y el primer panel

Grafana · Connections
ConnectionsData sourcesAdd data sourcePrometheus
NamePrometheus
Prometheus server URLhttp://prometheus:9090
Save & test
Successfully queried the Prometheus API.
Grafana · New dashboard › Add visualization
DashboardsNewNew dashboardAdd visualizationPrometheus
Query A100 * (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})
Legend{{instance}}
VisualizationBar gauge
UnitPercent (0-100)
Thresholds0 verde · 75 naranja · 85 rojo
TitleDisco raíz usado (%)
Apply
Save dashboard

Tres paneles más y tienes el tablero del equipo: CPU 100 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100, memoria 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) y servicios caídos node_systemd_unit_state{state="failed"} == 1 como tabla. Guarda el dashboard y exporta su JSON al repo de Git: es la copia de seguridad.

Consejo: en lugar de inventar paneles, importa el 1860 (Node Exporter Full) desde Dashboards → New → Import. Es el tablero por servidor que usa medio mundo; el tuyo es la vista de flota.

3. El punto de contacto: Teams

Grafana · Alerting › Contact points
AlertingContact pointsCreate contact point
Nameteams-it-ops
IntegrationMicrosoft Teams
URLhttps://prod-xx.westeurope.logic.azure.com/workflows/… (webhook del canal)
Title{{ .CommonLabels.alertname }} · {{ .CommonLabels.instance }}
Test
Save contact point
AlertingNotification policiesDefault policy
Default contact pointteams-it-ops
Group byalertname, instance

Pulsa Test y mira el canal antes de seguir. Si no llega, el 95 % de las veces es la URL del webhook o que el flujo de Teams está desactivado.

4. La regla de alerta

Grafana · Alerting › Alert rules
AlertingAlert rulesNew alert rule
NameDisco raíz > 85 %
Query A100 * (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})
Expression BReduce · Last · A
Expression CThreshold · B IS ABOVE 85
FolderInfraestructura · Evaluation group: cada 1m · Pending period: 10m
Labelsseverity=warning, team=it-ops
Summary{{ $labels.instance }} tiene el disco raíz al {{ $values.B | printf "%.0f" }} %
Runbook URLhttps://wiki.contoso.local/it/runbooks/disco-lleno
Save rule and exit

El pending period de 10 minutos es lo que evita una alerta por un pico de un tar. La etiqueta severity y la Runbook URL (la página de Confluence del Nivel 2) son lo que convierte una notificación en algo accionable a las tres de la mañana. Comprueba que funciona bajando el umbral a 1 % un minuto y volviéndolo a subir.

5. Lo que revisas cada mes

  • Alerting → Alert rules: reglas en estado Error o NoData (un servidor que ya no existe sigue en el inventario de Prometheus).
  • Alertas que saltaron más de diez veces en el mes y nadie trató: o sobra la alerta o falta el arreglo.
  • Retención de Prometheus frente al disco (du -sh del volumen prom-data).
  • Exportar los dashboards cambiados al repo.

Siguientes pasos

  • Windows: windows_exporter da lo mismo que node_exporter (disco, CPU, servicios); el mismo panel sirve con otro job_name.
  • Añade Elasticsearch (guía de ELK) como origen de datos: la gráfica de errores por hora junto a la de CPU del mismo servidor.
  • Grafana OnCall o la integración con la herramienta de guardia que uses para que las alertas severity=critical llamen por teléfono y las warning se queden en Teams.

¿Falta algo o en tu entorno es distinto? Escríbeme y lo corrijo.

What you'll get

A dashboard with the state of the servers (disk, CPU, memory, failed services) that opens on the team room screen, and a "disk above 85%" alert that lands in the team's Teams channel with the server name and value. What gets measured gets fixed sooner.

Before you start

  • The Grafana from the Docker guide already running (or a new one with the same Compose).
  • Port 9100 open on the servers towards the tools server (node_exporter) and 9090 local only.
  • An incoming webhook in the team's Teams channel (Workflows → "Post to a channel when a webhook request is received"); the URL is a secret.

1. Prometheus and node_exporter

Grafana draws; someone has to collect. Add Prometheus to the Compose and the exporter to the servers:

docker-compose.yml (fragment)
  prometheus:    image: prom/prometheus:v2.55.0    volumes:      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro      - prom-data:/prometheus    command: ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.retention.time=90d"]    ports: ["127.0.0.1:9090:9090"]
prometheus.yml
global:  scrape_interval: 30sscrape_configs:  - job_name: linux    static_configs:      - targets:          - srv-web-01.contoso.local:9100          - srv-web-02.contoso.local:9100          - srv-db-01.contoso.local:9100        labels: { site: madrid }
bash · srv-web-01 (or with Ansible on all of them)
sudo useradd -rs /bin/false node_exportercurl -sL https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz | sudo tar xz -C /usr/local/bin --strip-components=1 --wildcards '*/node_exporter'sudo tee /etc/systemd/system/node_exporter.service >/dev/null <<'EOF'[Unit]Description=Prometheus Node ExporterAfter=network.target[Service]User=node_exporterExecStart=/usr/local/bin/node_exporter --collector.systemd[Install]WantedBy=multi-user.targetEOFsudo systemctl enable --now node_exportersudo firewall-cmd --add-port=9100/tcp --permanent && sudo firewall-cmd --reloadcurl -s localhost:9100/metrics | grep -E '^node_filesystem_avail_bytes.*mountpoint="/"'node_filesystem_avail_bytes{device="/dev/mapper/rhel-root",fstype="xfs",mountpoint="/"} 2.0401152e+10

At http://localhost:9090/targets (from the tools server, or over an SSH tunnel) all three must be UP. If one is DOWN with connection refused, it is the server firewall; with context deadline exceeded, DNS or routing.

2. The data source and the first panel

Grafana · Connections
ConnectionsData sourcesAdd data sourcePrometheus
NamePrometheus
Prometheus server URLhttp://prometheus:9090
Save & test
Successfully queried the Prometheus API.
Grafana · New dashboard › Add visualization
DashboardsNewNew dashboardAdd visualizationPrometheus
Query A100 * (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})
Legend{{instance}}
VisualizationBar gauge
UnitPercent (0-100)
Thresholds0 green · 75 orange · 85 red
TitleRoot disk used (%)
Apply
Save dashboard

Three more panels and you have the team board: CPU 100 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100, memory 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) and failed services node_systemd_unit_state{state="failed"} == 1 as a table. Save the dashboard and export its JSON to the Git repo: that is the backup.

Tip: instead of inventing panels, import 1860 (Node Exporter Full) from Dashboards → New → Import. It is the per-server board half the world uses; yours is the fleet view.

3. The contact point: Teams

Grafana · Alerting › Contact points
AlertingContact pointsCreate contact point
Nameteams-it-ops
IntegrationMicrosoft Teams
URLhttps://prod-xx.westeurope.logic.azure.com/workflows/… (channel webhook)
Title{{ .CommonLabels.alertname }} · {{ .CommonLabels.instance }}
Test
Save contact point
AlertingNotification policiesDefault policy
Default contact pointteams-it-ops
Group byalertname, instance

Press Test and look at the channel before going on. If nothing arrives, 95% of the time it is the webhook URL or the Teams workflow is turned off.

4. The alert rule

Grafana · Alerting › Alert rules
AlertingAlert rulesNew alert rule
NameRoot disk > 85%
Query A100 * (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})
Expression BReduce · Last · A
Expression CThreshold · B IS ABOVE 85
FolderInfrastructure · Evaluation group: every 1m · Pending period: 10m
Labelsseverity=warning, team=it-ops
Summary{{ $labels.instance }} root disk is at {{ $values.B | printf "%.0f" }}%
Runbook URLhttps://wiki.contoso.local/it/runbooks/disk-full
Save rule and exit

The 10-minute pending period is what avoids an alert for a spike from a tar. The severity label and the Runbook URL (the Level 2 Confluence page) are what turn a notification into something actionable at three in the morning. Check it works by dropping the threshold to 1% for a minute and raising it back.

5. What you review each month

  • Alerting → Alert rules: rules in Error or NoData state (a server that no longer exists is still in the Prometheus inventory).
  • Alerts that fired more than ten times in the month and nobody acted on: either the alert is surplus or the fix is missing.
  • Prometheus retention against disk (du -sh of the prom-data volume).
  • Export changed dashboards to the repo.

Next steps

  • Windows: windows_exporter gives the same as node_exporter (disk, CPU, services); the same panel works with another job_name.
  • Add Elasticsearch (ELK guide) as a data source: the errors-per-hour chart next to the CPU chart of the same server.
  • Grafana OnCall or the integration with whatever on-call tool you use, so severity=critical alerts phone someone and warning stays in Teams.

Is something missing, or different in your environment? Get in touch and I'll fix it.

Tech Career Lab · newsletter

¿Te ha servido? Cada dos semanas envío una idea práctica para perfiles IT: LinkedIn, proyectos, networking y procesos de selección.Found this useful? Every two weeks I send one practical idea for IT professionals: LinkedIn, projects, networking and hiring processes.

Apúntate y llévate las 33 ideasSign up and get the 33 ideas →