Skip to main content

Data and monitoring · Advanced · 30 min

How to Send Alerts with Prometheus Alertmanager

By Padding · Updated 16 Nov 2026
Ubuntu 26.04.1 LTS · Prometheus 3.16.0 · Alertmanager 0.35.0

You will write alert rules in Prometheus 3.16, test them with promtool, install Alertmanager 0.35 on Ubuntu 26.04 listening on localhost only, route alerts to a receiver, trigger and resolve a real alert by stopping Node Exporter, and learn silences and an email receiver whose password stays in a file.

On this page

A dashboard only helps when somebody is looking at it. An alert makes the system tell you. Prometheus and Alertmanager split the job in two. Prometheus evaluates your alert rules, which are queries such as "is any target down?". When a rule's query returns something for long enough, Prometheus sends the alert to Alertmanager. Alertmanager decides what to do with it: it groups related alerts into one message, waits briefly so that a burst becomes one notification, repeats reminders at a sensible pace, honours silences you set during maintenance, and sends the result to a receiver such as email or a chat webhook.

In this tutorial you add three rules to the Prometheus from the previous tutorial, test them without a running server, then install Alertmanager and connect the two. A small script on 127.0.0.1 stands in for your chat or email, so you see exactly what Alertmanager sends. You then stop Node Exporter on purpose to make a real alert fire, and start it to see it resolve. Alertmanager is installed from the release file, so it gets no apt updates; the last steps say how to upgrade it.

Step 1 · Write the alert rules

Alert rules live in their own files, which Prometheus loads next to its configuration. Make a folder for them and write three rules:

Bash
sudo install -d -m 0755 /etc/prometheus/rules
sudo tee /etc/prometheus/rules/server.yml > /dev/null <<'YAML'
groups:
  - name: server
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} on {{ $labels.instance }} is down"

      - alert: HighMemoryUse
        expr: 100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Less than 10% memory available on {{ $labels.instance }}"

      - alert: DiskAlmostFull
        expr: |
          node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs|ramfs"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|ramfs"} < 0.10
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Less than 10% disk space free on {{ $labels.mountpoint }} ({{ $labels.instance }})"
YAML
promtool check rules /etc/prometheus/rules/server.yml

promtool prints SUCCESS: 3 rules found. How a rule reads:

  • expr is a PromQL query. The alert is active for every series the query returns. up == 0 returns only the targets whose last scrape failed.
  • for is how long the query must keep returning data before the alert counts as firing. Until then it is pending. It stops a single failed scrape or a short spike from waking anyone. One minute for a down target is short; use longer times for noisy metrics such as memory.
  • labels are added to the alert and are what Alertmanager routes on. severity is a convention you can route on later.
  • annotations hold the human-readable message. {{ $labels.instance }} inserts the label value.

Step 2 · Test the rules without breaking anything

You do not need to crash a server to know a rule works. promtool test rules replays made-up data through your rules and checks which alerts result. This file feeds the InstanceDown rule a target that is up for two minutes and then goes down:

Bash
mkdir -p ~/alert-demo
cat > ~/alert-demo/rules-test.yml <<'YAML'
rule_files:
  - /etc/prometheus/rules/server.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'up{job="node", instance="example:9100"}'
        values: '1 1 0 0 0 0'
    alert_rule_test:
      - eval_time: 2m
        alertname: InstanceDown
        exp_alerts: []
      - eval_time: 4m
        alertname: InstanceDown
        exp_alerts:
          - exp_labels:
              severity: critical
              job: node
              instance: "example:9100"
            exp_annotations:
              summary: "node on example:9100 is down"
YAML
promtool test rules ~/alert-demo/rules-test.yml

The output is SUCCESS. At two minutes the target has only now failed, so the alert is only pending and the list of firing alerts is empty. At four minutes it has been down for more than the one-minute for, so exactly this alert fires. If you change a rule and the test fails, promtool prints the alerts it expected next to the ones it got.

Step 3 · Install Alertmanager

Set the version and architecture again if you opened a new terminal. Then download the release file with its checksum list, check it and install alertmanager and amtool, its command-line tool:

Bash
AM_VERSION=0.35.0
ARCH="$(dpkg --print-architecture)"
mkdir -p ~/monitoring-install
cd ~/monitoring-install
BASE="https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}"
curl -fsSLO "${BASE}/alertmanager-${AM_VERSION}.linux-${ARCH}.tar.gz"
curl -fsSL -o alertmanager-sha256sums.txt "${BASE}/sha256sums.txt"
sha256sum --ignore-missing -c alertmanager-sha256sums.txt
tar -xzf "alertmanager-${AM_VERSION}.linux-${ARCH}.tar.gz"
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/alertmanager" /usr/local/bin/alertmanager
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/amtool" /usr/local/bin/amtool
alertmanager --version | head -n 1

The checksum line ends in OK and the version line starts with alertmanager, version 0.35.0. Create a user for it and its folders. The state folder keeps silences and notification history between restarts:

Bash
sudo useradd --system --no-create-home --shell /usr/sbin/nologin alertmanager
sudo install -d -o root -g alertmanager -m 0750 /etc/alertmanager
sudo install -d -o alertmanager -g alertmanager -m 0750 /var/lib/alertmanager

Step 4 · Make a stand-in receiver

Real receivers are email, Slack, PagerDuty and so on. To see what Alertmanager sends without an account, write a tiny web server that prints each alert it is given. Alertmanager's webhook receiver posts a JSON document, and this script reads the fields you care about:

Bash
cat > ~/alert-demo/webhook-logger.py <<'PY'
import json
from http.server import BaseHTTPRequestHandler, HTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
        payload = json.loads(body)
        for alert in payload["alerts"]:
            print(
                alert["status"],
                alert["labels"].get("alertname"),
                alert["labels"].get("instance"),
                "-",
                alert["annotations"].get("summary"),
                flush=True,
            )
        self.send_response(200)
        self.end_headers()

    def log_message(self, *args):
        pass


HTTPServer(("127.0.0.1", 5001), Handler).serve_forever()
PY
nohup python3 ~/alert-demo/webhook-logger.py > ~/alert-demo/webhook.log 2>&1 &
sleep 1
ss -tln | grep 5001

The listener is 127.0.0.1:5001. The script writes one line per alert to ~/alert-demo/webhook.log.

Step 5 · Configure Alertmanager

The configuration has two parts. A route says how alerts are grouped and timed, and which receiver gets them. A receivers list says how to reach each receiver. This one sends everything to the stand-in script:

Bash
sudo tee /etc/alertmanager/alertmanager.yml > /dev/null <<'YAML'
route:
  receiver: demo-webhook
  group_by: [alertname, instance]
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 4h

receivers:
  - name: demo-webhook
    webhook_configs:
      - url: http://127.0.0.1:5001/
        send_resolved: true
YAML
sudo chgrp alertmanager /etc/alertmanager/alertmanager.yml
sudo chmod 640 /etc/alertmanager/alertmanager.yml
sudo amtool check-config /etc/alertmanager/alertmanager.yml

amtool reports SUCCESS and lists what it found: the global config, the route, and one receiver. What the timings mean:

  • group_by puts alerts with the same alertname and instance into one notification, so ten failing disks on one server are one message.
  • group_wait is how long Alertmanager waits after the first alert of a new group before sending, so alerts that arrive together are sent together. The default is 30 seconds; 10 keeps this demonstration short.
  • group_interval is how long it waits before notifying about changes to a group that it already sent.
  • repeat_interval is how often it reminds you while the alert keeps firing. Four hours is a common choice.
  • send_resolved: true sends a second message when the problem ends, so you know it is over.

Step 6 · Run Alertmanager as a service

Write the unit. --web.listen-address=127.0.0.1:9093 keeps the web interface and API private, since Alertmanager has no login. By default Alertmanager also opens a cluster port, 9094, on every address, for high-availability setups with several Alertmanagers. An empty --cluster.listen-address= turns that off. You do not need it for one instance, and an unneeded open port is a risk:

Bash
sudo tee /etc/systemd/system/alertmanager.service > /dev/null <<'UNIT'
[Unit]
Description=Prometheus Alertmanager
After=network-online.target
Wants=network-online.target

[Service]
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=127.0.0.1:9093 \
  --cluster.listen-address=
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target
UNIT
Bash
sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager
sudo systemctl status alertmanager --no-pager

Check it worked

Bash
curl -s http://127.0.0.1:9093/-/ready
ss -tln | grep -E ':909[34]'

The first prints OK. The listener list shows 127.0.0.1:9093 and, apart from Prometheus's own 127.0.0.1:9090, nothing on 9094. If a line shows 9094, the --cluster.listen-address= option did not take effect.

Step 7 · Connect Prometheus to Alertmanager

Prometheus needs two additions: where Alertmanager is, and which rule files to load. Rewrite its configuration with both. This is the file from the previous tutorial plus the rule_files and alerting sections:

Bash
sudo tee /etc/prometheus/prometheus.yml > /dev/null <<'YAML'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["127.0.0.1:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]

  - job_name: node
    static_configs:
      - targets: ["127.0.0.1:9100"]
YAML
promtool check config /etc/prometheus/prometheus.yml

promtool checks the configuration and the rule files it points to, and prints SUCCESS for both. evaluation_interval is how often rules run; the default is one minute. Reload Prometheus, which re-reads the files without a restart:

Bash
sudo systemctl reload prometheus

Check it worked

Prometheus lists the rules it loaded and the Alertmanagers it found:

Bash
curl -s http://127.0.0.1:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'
curl -s http://127.0.0.1:9090/api/v1/alertmanagers | jq -r '.data.activeAlertmanagers[].url'

You see InstanceDown, HighMemoryUse and DiskAlmostFull, then http://127.0.0.1:9093/api/v2/alerts.

Troubleshooting

  • No Alertmanager in the list: the targets address is wrong, or Prometheus was not reloaded. Check sudo journalctl -u prometheus -n 20 --no-pager for a reload error.
  • The reload says the configuration is invalid: run promtool check config /etc/prometheus/prometheus.yml, which names the line.
  • A rule is missing: the file must end in .yml, and promtool check rules must pass.

Step 8 · Make an alert fire, then resolve

Now cause the problem. Stop Node Exporter. The next scrape of the node target fails, up becomes 0, and the clock for InstanceDown starts:

Bash
sudo systemctl stop node_exporter

Wait until Prometheus shows the alert as firing. This loop asks every five seconds and stops when it fires, which takes between one and two minutes (the for time plus up to a scrape and an evaluation). It gives up after five minutes:

Bash
for i in $(seq 1 60); do
  if curl -s http://127.0.0.1:9090/api/v1/alerts | jq -e '.data.alerts[] | select(.labels.alertname == "InstanceDown" and .state == "firing")' > /dev/null; then
    echo "InstanceDown is firing"
    break
  fi
  sleep 5
done

Prometheus has now sent it to Alertmanager. Ask Alertmanager which alerts it holds, and wait for the notification to reach the stand-in receiver. Alertmanager holds a new group for group_wait (10 seconds) first:

Bash
curl -s http://127.0.0.1:9093/api/v2/alerts | jq -r '.[].labels.alertname'
for i in $(seq 1 30); do
  grep -q '^firing InstanceDown' ~/alert-demo/webhook.log && break
  sleep 2
done
cat ~/alert-demo/webhook.log

The first command prints InstanceDown. The log contains firing InstanceDown 127.0.0.1:9100 - node on 127.0.0.1:9100 is down.

Fix the problem and watch the alert end. Start Node Exporter again:

Bash
sudo systemctl start node_exporter
Bash
for i in $(seq 1 60); do
  grep -q '^resolved InstanceDown' ~/alert-demo/webhook.log && break
  sleep 5
done
cat ~/alert-demo/webhook.log

After the next scrape and the next notification round, about a minute, the log gains resolved InstanceDown 127.0.0.1:9100 - node on 127.0.0.1:9100 is down. That is the whole path working end to end: a failed scrape, a rule, a grouped notification and a resolution message.

Step 9 · Use a real receiver, with the password in a file

To send email, replace the webhook with an email_configs receiver. Alertmanager can read the SMTP password from a file, so the password is not written in the configuration (which you might paste into a ticket or commit to Git). Prompt for the password, then write it to a private file. The file is created with its final permissions before the password goes in:

Bash
read -rsp "SMTP password: " SMTP_PW; echo
Bash
sudo install -m 640 -o root -g alertmanager /dev/null /etc/alertmanager/smtp_password
printf '%s' "$SMTP_PW" | sudo tee /etc/alertmanager/smtp_password > /dev/null

Write the new configuration to a separate file and check it before you use it. Replace the host, addresses and user with your mail provider's values. Port 587 with smtp_require_tls is the usual submission setup:

Bash
cat > ~/alert-demo/alertmanager-email.yml <<'YAML'
global:
  smtp_smarthost: smtp.example.com:587      # replace: your mail provider
  smtp_from: alerts@example.com             # replace
  smtp_auth_username: alerts@example.com    # replace
  smtp_auth_password_file: /etc/alertmanager/smtp_password
  smtp_require_tls: true

route:
  receiver: email
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: email
    email_configs:
      - to: you@example.com                 # replace
        send_resolved: true
YAML
amtool check-config ~/alert-demo/alertmanager-email.yml

amtool reports SUCCESS. To use it, copy the file over /etc/alertmanager/alertmanager.yml, set the same owner and mode as before, and run sudo systemctl reload alertmanager. It can only prove that the file is valid, not that your mail provider accepts it, so send a test alert after switching: stop Node Exporter for two minutes, as above. For chat, Alertmanager has slack_configs with an api_url_file option that reads the webhook address from a file in the same way, and receivers for PagerDuty, Opsgenie, Telegram and others; see https://prometheus.io/docs/alerting/latest/configuration/.

Step 10 · Silence alerts during maintenance

Before planned work, silence the alerts you expect, so nobody is paged for something you caused. A silence matches alerts by label and lasts for a set time. amtool talks to Alertmanager's API:

Bash
ID="$(amtool silence add alertname=InstanceDown --comment="planned maintenance" --duration=1h --author="$(id -un)" --alertmanager.url=http://127.0.0.1:9093)"
amtool silence query --alertmanager.url=http://127.0.0.1:9093
amtool silence expire "$ID" --alertmanager.url=http://127.0.0.1:9093
amtool silence query --alertmanager.url=http://127.0.0.1:9093

silence add prints the new silence's ID. The first query lists it, with your comment and an end time an hour away. After silence expire, the second query lists nothing. Silences are stored in /var/lib/alertmanager, so a restart keeps them. The Alertmanager web page at http://127.0.0.1:9093 shows the same list, and you can open it from your laptop through an SSH tunnel, as in the Grafana tutorial.

Silences are for planned, short things. Alertmanager also has inhibition rules, which stop a minor alert while a major one is already firing: for example, no memory warnings from a server whose InstanceDown alert is active. They are in the same file; see the inhibit_rules section of the configuration reference above.

Step 11 · Choose rules people will act on

Alerts that nobody reads teach people to ignore all of them. A few rules of thumb:

  • Alert on symptoms a person must act on: a service is down, the disk will fill, errors are high. Do not alert on every spike in CPU.
  • Give every alert a for time long enough to ignore blips, and a clear summary that says what is wrong and where.
  • Route by severity: critical pages someone now, warning goes to a channel read in working hours.
  • Review alerts that fired and needed no action, and delete or tune them.
  • Test the delivery path now and then, because a broken receiver fails silently. A rule that always fires, such as vector(1), is the standard heartbeat that an outside service can watch.

Step 12 · Keep Alertmanager updated

Alertmanager came from a release file, so apt upgrade will not update it. Watch https://github.com/prometheus/alertmanager/releases. To upgrade, set AM_VERSION to the new version, repeat the download, checksum and unpack commands, then replace the programs while the service is stopped and start it again:

Bash
sudo systemctl stop alertmanager
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/alertmanager" /usr/local/bin/alertmanager
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/amtool" /usr/local/bin/amtool
sudo systemctl start alertmanager

Run amtool check-config against your configuration with the new amtool first, since options can change between releases.

Step 13 · Remove the demo

Stop the stand-in receiver and remove Alertmanager, the rules and the demo folder. The last commands put Prometheus's configuration back to the one from the previous tutorial. If you want to keep alerting, skip this step. It leaves Prometheus and Node Exporter in place; remove those with the last step of the Prometheus tutorial when the series is finished:

Bash
pkill -f webhook-logger.py
sudo systemctl disable --now alertmanager
sudo rm /etc/systemd/system/alertmanager.service
sudo systemctl daemon-reload
sudo rm /usr/local/bin/alertmanager /usr/local/bin/amtool
sudo rm -r /etc/alertmanager /var/lib/alertmanager /etc/prometheus/rules
sudo userdel alertmanager
rm -rf ~/alert-demo ~/monitoring-install
Bash
sudo tee /etc/prometheus/prometheus.yml > /dev/null <<'YAML'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]

  - job_name: node
    static_configs:
      - targets: ["127.0.0.1:9100"]
YAML
sudo systemctl reload prometheus

Next steps

Need this done for you?

Padding does this work for growing companies, from the first diagnosis to a system that keeps working.

Platform Engineering

Get new tutorials

New tutorials, about twice a month.