Grafana basics: connect Prometheus, build a dashboard and set up an alert

Grafana is a web application for visualizing data. It does not store metrics itself; it queries data sources such as Prometheus, Loki or SQL databases and shows the results in dashboards. It can also evaluate alert rules and send notifications. This guide assumes Prometheus already scrapes node_exporter on your servers.

Step 1: sign in

Grafana listens on port 3000 by default. Open http://grafana.example.com:3000/, sign in as admin with password admin on a fresh installation, and set a new password when asked. Do this before the server is reachable by anyone else.

Step 2: add Prometheus as a data source

1. In the left menu open Connections and choose Add new connection.
2. Search for Prometheus, select it and click Add new data source.
3. Set Prometheus server URL, for example http://localhost:9090 when both run on the same host. If Grafana runs in a container, localhost means the container itself; use the Prometheus host name or IP instead.
4. Click Save & test. On success Grafana reports that it successfully queried the Prometheus API.

To keep this setting in version control instead of clicking, Grafana can provision data sources from YAML files in the datasources folder of its provisioning directory:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true

Step 3: build a dashboard

1. Go to Dashboards, then New > New dashboard, and add a visualization.
2. Select the Prometheus data source and enter a PromQL query (see below). Choose the Time series visualization and set a title and unit, for example Percent (0-100).
3. Repeat for each panel, then Save the dashboard with a clear name and folder.

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle",instance=~"$instance"}[5m])))

node_memory_MemAvailable_bytes{instance=~"$instance"}

1 - node_filesystem_avail_bytes{instance=~"$instance",fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{instance=~"$instance",fstype!~"tmpfs|overlay"}

The queries above show CPU usage in percent, available memory in bytes and the used fraction of each file system. $instance is a dashboard variable: in the dashboard settings add a variable named instance of type Query with the query label_values(up{job="node"}, instance), and enable multi-value and Include All. A drop-down at the top of the dashboard then switches between hosts.

You do not have to build everything yourself: the Grafana website hosts community dashboards for node_exporter and many other exporters, which you can import from the Dashboards > New > Import page and then adapt.

Step 4: create a contact point

Alerts need somewhere to go. In Alerting (labeled Alerts & IRM in recent versions), open the contact points and create a new one. Pick an integration such as Email, Slack, Microsoft Teams or Webhook, fill in its settings, and use Test to send a test notification.

Step 5: create an alert rule

1. Open Alert rules and choose New alert rule. Give it a name such as Host down.
2. Query: select Prometheus and enter up{job="node"}. Set the condition to fire when the value is below 1. Use Preview to check the result: you get one alert instance per host.
3. Choose a folder and add a label such as severity=critical.
4. Evaluation: pick or create an evaluation group (for example every 1 minute) and set the pending period to 5 minutes, so a single failed scrape does not alert.
5. Notifications: select the contact point from step 4.
6. Add a summary such as {{ $labels.instance }} is not reachable, then click Save rule.

How alert states work

• Normal: the condition is not met.
• Pending: the condition is met, but not yet for the whole pending period.
• Alerting (firing): the condition held for the pending period; a notification is sent.
• Recovering: the condition cleared, but the optional keep firing for window has not passed.
• NoData and Error: the query returned nothing, or failed. Decide in the rule how these should be handled, because a host that disappears entirely may produce no data rather than 0.

Notification policies route alerts to contact points by label, silences mute notifications once (for example during maintenance), and mute timings mute them on a schedule such as weekends. Start with the default policy and one contact point, and add routing only when you need it.

Official documentation

• Configure the Prometheus data source
• Alert rule evaluation
• Configure Grafana-managed alert rules