Prometheus concepts: pull model, exporters, node_exporter and PromQL basics

Prometheus is an open-source monitoring and alerting toolkit. It collects metrics, numbers that change over time such as CPU usage, free disk space or requests per second, and stores them as time series. You query them with PromQL, draw them in Grafana, and turn them into alerts.

The pull model

Most monitoring systems wait for agents to push data. Prometheus does the opposite: it scrapes each target by fetching an HTTP endpoint, usually /metrics, on a fixed interval. The response is plain text, one sample per line:

# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 4.2e+10

Pulling makes it easy to see whether a target is alive: for every scrape Prometheus records a series up with value 1 if the scrape worked and 0 if it failed. It also adds the labels job (the job name from the config) and instance (the host:port scraped). For short-lived batch jobs that end before they can be scraped, there is the Pushgateway.

Series, labels and metric types

A time series is identified by a metric name plus a set of labels (key-value pairs). node_filesystem_avail_bytes{mountpoint="/"} and node_filesystem_avail_bytes{mountpoint="/var"} are two series of the same metric. Labels are what you filter and group by.

• Counter: only goes up (or resets to zero on restart). Requests served, bytes sent, errors. You almost always look at its rate, not its raw value. Names usually end in _total.
• Gauge: goes up and down. Temperature, memory in use, free disk space.
• Histogram: counts observations (for example request durations) in buckets, exposed as _bucket, _sum and _count series; histogram_quantile() computes percentiles from it.
• Summary: similar, but computes quantiles in the client.

Exporters and node_exporter

Applications you write can expose metrics directly with a client library. For everything else there are exporters: small programs that read data from a system and expose it in Prometheus format. The most common is node_exporter, which exposes Linux host metrics (CPU, memory, disks, file systems, network) on port 9100.

# download the current release from the node_exporter releases page, then:
tar xvfz node_exporter-<version>.linux-amd64.tar.gz
cd node_exporter-<version>.linux-amd64
./node_exporter

# from another terminal
curl http://localhost:9100/metrics | grep node_cpu_seconds_total

For production, run it as a systemd service under its own unprivileged user, or install it from your distribution’s packages. Port 9100 should be reachable from the Prometheus server only, not from the internet.

A minimal prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - rules.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets:
          - "192.0.2.10:9100"
          - "192.0.2.11:9100"

Start Prometheus with ./prometheus --config.file=prometheus.yml. Its web UI listens on port 9090; the Status > Targets page shows every target with its health and last error. promtool check config prometheus.yml validates the file before you restart.

PromQL basics

PromQL selects series by name and labels and computes over them. Label matchers are =, !=, =~ (regex) and !~. A selector like up{job="node"} returns the latest value of each matching series (an instant vector); adding [5m] returns the samples of the last five minutes (a range vector), which functions such as rate() need.

up == 0

rate(node_network_receive_bytes_total[5m])

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes

• up == 0: targets whose last scrape failed.
• rate(...[5m]): per-second rate of a counter, averaged over five minutes. Here, received bytes per second per interface.
• The third query is CPU usage in percent per host: the share of time not idle.
• The last query is the fraction of free space per file system, ignoring in-memory file systems.

Alerting rules

Alerting rules are PromQL expressions evaluated on a schedule. When an expression returns results for longer than the for duration, the alert fires and Prometheus sends it to Alertmanager, which groups, silences and routes notifications to email, chat or paging tools.

groups:
  - name: node-alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} of job {{ $labels.job }} is down"
      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.10
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Less than 10% disk space left on {{ $labels.instance }} {{ $labels.mountpoint }}"

Save this as rules.yml next to prometheus.yml (it is referenced in rule_files) and check it with promtool check rules rules.yml. The for clause avoids paging on a single failed scrape.

When Prometheus is not the right tool

Prometheus favors reliability over perfect accuracy: it is ideal for operational monitoring but not for data that must be 100% complete, such as per-request billing. It also stores numbers, not log lines; logs need a separate system.

Official documentation

• Prometheus overview
• Monitoring Linux host metrics with the Node Exporter
• Querying Prometheus