Data and monitoring · Advanced · 30 min
How to Send Alerts with Prometheus Alertmanager
By Padding · Updated 16 Nov 2026
Ubuntu 26.04.1 LTS · Prometheus 3.16.0 · Alertmanager 0.35.0
You will write alert rules in Prometheus 3.16, test them with promtool, install Alertmanager 0.35 on Ubuntu 26.04 listening on localhost only, route alerts to a receiver, trigger and resolve a real alert by stopping Node Exporter, and learn silences and an email receiver whose password stays in a file.
On this page
A dashboard only helps when somebody is looking at it. An alert makes the system tell you. Prometheus and Alertmanager split the job in two. Prometheus evaluates your alert rules, which are queries such as "is any target down?". When a rule's query returns something for long enough, Prometheus sends the alert to Alertmanager. Alertmanager decides what to do with it: it groups related alerts into one message, waits briefly so that a burst becomes one notification, repeats reminders at a sensible pace, honours silences you set during maintenance, and sends the result to a receiver such as email or a chat webhook.
In this tutorial you add three rules to the Prometheus from the previous tutorial, test them without a running server, then install Alertmanager and connect the two. A small script on 127.0.0.1 stands in for your chat or email, so you see exactly what Alertmanager sends. You then stop Node Exporter on purpose to make a real alert fire, and start it to see it resolve. Alertmanager is installed from the release file, so it gets no apt updates; the last steps say how to upgrade it.
Step 1 · Write the alert rules
Alert rules live in their own files, which Prometheus loads next to its configuration. Make a folder for them and write three rules:
sudo install -d -m 0755 /etc/prometheus/rules
sudo tee /etc/prometheus/rules/server.yml > /dev/null <<'YAML'
groups:
- name: server
rules:
- alert: InstanceDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} on {{ $labels.instance }} is down"
- alert: HighMemoryUse
expr: 100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 10
for: 5m
labels:
severity: warning
annotations:
summary: "Less than 10% memory available on {{ $labels.instance }}"
- alert: DiskAlmostFull
expr: |
node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs|ramfs"}
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|ramfs"} < 0.10
for: 10m
labels:
severity: warning
annotations:
summary: "Less than 10% disk space free on {{ $labels.mountpoint }} ({{ $labels.instance }})"
YAML
promtool check rules /etc/prometheus/rules/server.ymlpromtool prints SUCCESS: 3 rules found. How a rule reads:
expris a PromQL query. The alert is active for every series the query returns.up == 0returns only the targets whose last scrape failed.foris how long the query must keep returning data before the alert counts as firing. Until then it ispending. It stops a single failed scrape or a short spike from waking anyone. One minute for a down target is short; use longer times for noisy metrics such as memory.labelsare added to the alert and are what Alertmanager routes on.severityis a convention you can route on later.annotationshold the human-readable message.{{ $labels.instance }}inserts the label value.
Step 2 · Test the rules without breaking anything
You do not need to crash a server to know a rule works. promtool test rules replays made-up data through your rules and checks which alerts result. This file feeds the InstanceDown rule a target that is up for two minutes and then goes down:
mkdir -p ~/alert-demo
cat > ~/alert-demo/rules-test.yml <<'YAML'
rule_files:
- /etc/prometheus/rules/server.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'up{job="node", instance="example:9100"}'
values: '1 1 0 0 0 0'
alert_rule_test:
- eval_time: 2m
alertname: InstanceDown
exp_alerts: []
- eval_time: 4m
alertname: InstanceDown
exp_alerts:
- exp_labels:
severity: critical
job: node
instance: "example:9100"
exp_annotations:
summary: "node on example:9100 is down"
YAML
promtool test rules ~/alert-demo/rules-test.ymlThe output is SUCCESS. At two minutes the target has only now failed, so the alert is only pending and the list of firing alerts is empty. At four minutes it has been down for more than the one-minute for, so exactly this alert fires. If you change a rule and the test fails, promtool prints the alerts it expected next to the ones it got.
Step 3 · Install Alertmanager
Set the version and architecture again if you opened a new terminal. Then download the release file with its checksum list, check it and install alertmanager and amtool, its command-line tool:
AM_VERSION=0.35.0
ARCH="$(dpkg --print-architecture)"
mkdir -p ~/monitoring-install
cd ~/monitoring-install
BASE="https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}"
curl -fsSLO "${BASE}/alertmanager-${AM_VERSION}.linux-${ARCH}.tar.gz"
curl -fsSL -o alertmanager-sha256sums.txt "${BASE}/sha256sums.txt"
sha256sum --ignore-missing -c alertmanager-sha256sums.txt
tar -xzf "alertmanager-${AM_VERSION}.linux-${ARCH}.tar.gz"
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/alertmanager" /usr/local/bin/alertmanager
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/amtool" /usr/local/bin/amtool
alertmanager --version | head -n 1The checksum line ends in OK and the version line starts with alertmanager, version 0.35.0. Create a user for it and its folders. The state folder keeps silences and notification history between restarts:
sudo useradd --system --no-create-home --shell /usr/sbin/nologin alertmanager
sudo install -d -o root -g alertmanager -m 0750 /etc/alertmanager
sudo install -d -o alertmanager -g alertmanager -m 0750 /var/lib/alertmanagerStep 4 · Make a stand-in receiver
Real receivers are email, Slack, PagerDuty and so on. To see what Alertmanager sends without an account, write a tiny web server that prints each alert it is given. Alertmanager's webhook receiver posts a JSON document, and this script reads the fields you care about:
cat > ~/alert-demo/webhook-logger.py <<'PY'
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
payload = json.loads(body)
for alert in payload["alerts"]:
print(
alert["status"],
alert["labels"].get("alertname"),
alert["labels"].get("instance"),
"-",
alert["annotations"].get("summary"),
flush=True,
)
self.send_response(200)
self.end_headers()
def log_message(self, *args):
pass
HTTPServer(("127.0.0.1", 5001), Handler).serve_forever()
PY
nohup python3 ~/alert-demo/webhook-logger.py > ~/alert-demo/webhook.log 2>&1 &
sleep 1
ss -tln | grep 5001The listener is 127.0.0.1:5001. The script writes one line per alert to ~/alert-demo/webhook.log.
Step 5 · Configure Alertmanager
The configuration has two parts. A route says how alerts are grouped and timed, and which receiver gets them. A receivers list says how to reach each receiver. This one sends everything to the stand-in script:
sudo tee /etc/alertmanager/alertmanager.yml > /dev/null <<'YAML'
route:
receiver: demo-webhook
group_by: [alertname, instance]
group_wait: 10s
group_interval: 30s
repeat_interval: 4h
receivers:
- name: demo-webhook
webhook_configs:
- url: http://127.0.0.1:5001/
send_resolved: true
YAML
sudo chgrp alertmanager /etc/alertmanager/alertmanager.yml
sudo chmod 640 /etc/alertmanager/alertmanager.yml
sudo amtool check-config /etc/alertmanager/alertmanager.ymlamtool reports SUCCESS and lists what it found: the global config, the route, and one receiver. What the timings mean:
group_byputs alerts with the samealertnameandinstanceinto one notification, so ten failing disks on one server are one message.group_waitis how long Alertmanager waits after the first alert of a new group before sending, so alerts that arrive together are sent together. The default is 30 seconds; 10 keeps this demonstration short.group_intervalis how long it waits before notifying about changes to a group that it already sent.repeat_intervalis how often it reminds you while the alert keeps firing. Four hours is a common choice.send_resolved: truesends a second message when the problem ends, so you know it is over.
Step 6 · Run Alertmanager as a service
Write the unit. --web.listen-address=127.0.0.1:9093 keeps the web interface and API private, since Alertmanager has no login. By default Alertmanager also opens a cluster port, 9094, on every address, for high-availability setups with several Alertmanagers. An empty --cluster.listen-address= turns that off. You do not need it for one instance, and an unneeded open port is a risk:
sudo tee /etc/systemd/system/alertmanager.service > /dev/null <<'UNIT'
[Unit]
Description=Prometheus Alertmanager
After=network-online.target
Wants=network-online.target
[Service]
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager \
--web.listen-address=127.0.0.1:9093 \
--cluster.listen-address=
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
[Install]
WantedBy=multi-user.target
UNITsudo systemctl daemon-reload
sudo systemctl enable --now alertmanager
sudo systemctl status alertmanager --no-pagerCheck it worked
curl -s http://127.0.0.1:9093/-/ready
ss -tln | grep -E ':909[34]'The first prints OK. The listener list shows 127.0.0.1:9093 and, apart from Prometheus's own 127.0.0.1:9090, nothing on 9094. If a line shows 9094, the --cluster.listen-address= option did not take effect.
Step 7 · Connect Prometheus to Alertmanager
Prometheus needs two additions: where Alertmanager is, and which rule files to load. Rewrite its configuration with both. This is the file from the previous tutorial plus the rule_files and alerting sections:
sudo tee /etc/prometheus/prometheus.yml > /dev/null <<'YAML'
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
alertmanagers:
- static_configs:
- targets: ["127.0.0.1:9093"]
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["127.0.0.1:9090"]
- job_name: node
static_configs:
- targets: ["127.0.0.1:9100"]
YAML
promtool check config /etc/prometheus/prometheus.ymlpromtool checks the configuration and the rule files it points to, and prints SUCCESS for both. evaluation_interval is how often rules run; the default is one minute. Reload Prometheus, which re-reads the files without a restart:
sudo systemctl reload prometheusCheck it worked
Prometheus lists the rules it loaded and the Alertmanagers it found:
curl -s http://127.0.0.1:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'
curl -s http://127.0.0.1:9090/api/v1/alertmanagers | jq -r '.data.activeAlertmanagers[].url'You see InstanceDown, HighMemoryUse and DiskAlmostFull, then http://127.0.0.1:9093/api/v2/alerts.
Troubleshooting
- No Alertmanager in the list: the
targetsaddress is wrong, or Prometheus was not reloaded. Checksudo journalctl -u prometheus -n 20 --no-pagerfor a reload error. - The reload says the configuration is invalid: run
promtool check config /etc/prometheus/prometheus.yml, which names the line. - A rule is missing: the file must end in
.yml, andpromtool check rulesmust pass.
Step 8 · Make an alert fire, then resolve
Now cause the problem. Stop Node Exporter. The next scrape of the node target fails, up becomes 0, and the clock for InstanceDown starts:
sudo systemctl stop node_exporterWait until Prometheus shows the alert as firing. This loop asks every five seconds and stops when it fires, which takes between one and two minutes (the for time plus up to a scrape and an evaluation). It gives up after five minutes:
for i in $(seq 1 60); do
if curl -s http://127.0.0.1:9090/api/v1/alerts | jq -e '.data.alerts[] | select(.labels.alertname == "InstanceDown" and .state == "firing")' > /dev/null; then
echo "InstanceDown is firing"
break
fi
sleep 5
donePrometheus has now sent it to Alertmanager. Ask Alertmanager which alerts it holds, and wait for the notification to reach the stand-in receiver. Alertmanager holds a new group for group_wait (10 seconds) first:
curl -s http://127.0.0.1:9093/api/v2/alerts | jq -r '.[].labels.alertname'
for i in $(seq 1 30); do
grep -q '^firing InstanceDown' ~/alert-demo/webhook.log && break
sleep 2
done
cat ~/alert-demo/webhook.logThe first command prints InstanceDown. The log contains firing InstanceDown 127.0.0.1:9100 - node on 127.0.0.1:9100 is down.
Fix the problem and watch the alert end. Start Node Exporter again:
sudo systemctl start node_exporterfor i in $(seq 1 60); do
grep -q '^resolved InstanceDown' ~/alert-demo/webhook.log && break
sleep 5
done
cat ~/alert-demo/webhook.logAfter the next scrape and the next notification round, about a minute, the log gains resolved InstanceDown 127.0.0.1:9100 - node on 127.0.0.1:9100 is down. That is the whole path working end to end: a failed scrape, a rule, a grouped notification and a resolution message.
Step 9 · Use a real receiver, with the password in a file
To send email, replace the webhook with an email_configs receiver. Alertmanager can read the SMTP password from a file, so the password is not written in the configuration (which you might paste into a ticket or commit to Git). Prompt for the password, then write it to a private file. The file is created with its final permissions before the password goes in:
read -rsp "SMTP password: " SMTP_PW; echosudo install -m 640 -o root -g alertmanager /dev/null /etc/alertmanager/smtp_password
printf '%s' "$SMTP_PW" | sudo tee /etc/alertmanager/smtp_password > /dev/nullWrite the new configuration to a separate file and check it before you use it. Replace the host, addresses and user with your mail provider's values. Port 587 with smtp_require_tls is the usual submission setup:
cat > ~/alert-demo/alertmanager-email.yml <<'YAML'
global:
smtp_smarthost: smtp.example.com:587 # replace: your mail provider
smtp_from: alerts@example.com # replace
smtp_auth_username: alerts@example.com # replace
smtp_auth_password_file: /etc/alertmanager/smtp_password
smtp_require_tls: true
route:
receiver: email
group_by: [alertname, instance]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receivers:
- name: email
email_configs:
- to: you@example.com # replace
send_resolved: true
YAML
amtool check-config ~/alert-demo/alertmanager-email.ymlamtool reports SUCCESS. To use it, copy the file over /etc/alertmanager/alertmanager.yml, set the same owner and mode as before, and run sudo systemctl reload alertmanager. It can only prove that the file is valid, not that your mail provider accepts it, so send a test alert after switching: stop Node Exporter for two minutes, as above. For chat, Alertmanager has slack_configs with an api_url_file option that reads the webhook address from a file in the same way, and receivers for PagerDuty, Opsgenie, Telegram and others; see https://prometheus.io/docs/alerting/latest/configuration/.
Step 10 · Silence alerts during maintenance
Before planned work, silence the alerts you expect, so nobody is paged for something you caused. A silence matches alerts by label and lasts for a set time. amtool talks to Alertmanager's API:
ID="$(amtool silence add alertname=InstanceDown --comment="planned maintenance" --duration=1h --author="$(id -un)" --alertmanager.url=http://127.0.0.1:9093)"
amtool silence query --alertmanager.url=http://127.0.0.1:9093
amtool silence expire "$ID" --alertmanager.url=http://127.0.0.1:9093
amtool silence query --alertmanager.url=http://127.0.0.1:9093silence add prints the new silence's ID. The first query lists it, with your comment and an end time an hour away. After silence expire, the second query lists nothing. Silences are stored in /var/lib/alertmanager, so a restart keeps them. The Alertmanager web page at http://127.0.0.1:9093 shows the same list, and you can open it from your laptop through an SSH tunnel, as in the Grafana tutorial.
Silences are for planned, short things. Alertmanager also has inhibition rules, which stop a minor alert while a major one is already firing: for example, no memory warnings from a server whose InstanceDown alert is active. They are in the same file; see the inhibit_rules section of the configuration reference above.
Step 11 · Choose rules people will act on
Alerts that nobody reads teach people to ignore all of them. A few rules of thumb:
- Alert on symptoms a person must act on: a service is down, the disk will fill, errors are high. Do not alert on every spike in CPU.
- Give every alert a
fortime long enough to ignore blips, and a clearsummarythat says what is wrong and where. - Route by
severity:criticalpages someone now,warninggoes to a channel read in working hours. - Review alerts that fired and needed no action, and delete or tune them.
- Test the delivery path now and then, because a broken receiver fails silently. A rule that always fires, such as
vector(1), is the standard heartbeat that an outside service can watch.
Step 12 · Keep Alertmanager updated
Alertmanager came from a release file, so apt upgrade will not update it. Watch https://github.com/prometheus/alertmanager/releases. To upgrade, set AM_VERSION to the new version, repeat the download, checksum and unpack commands, then replace the programs while the service is stopped and start it again:
sudo systemctl stop alertmanager
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/alertmanager" /usr/local/bin/alertmanager
sudo install -o root -g root -m 0755 "alertmanager-${AM_VERSION}.linux-${ARCH}/amtool" /usr/local/bin/amtool
sudo systemctl start alertmanagerRun amtool check-config against your configuration with the new amtool first, since options can change between releases.
Step 13 · Remove the demo
Stop the stand-in receiver and remove Alertmanager, the rules and the demo folder. The last commands put Prometheus's configuration back to the one from the previous tutorial. If you want to keep alerting, skip this step. It leaves Prometheus and Node Exporter in place; remove those with the last step of the Prometheus tutorial when the series is finished:
pkill -f webhook-logger.py
sudo systemctl disable --now alertmanager
sudo rm /etc/systemd/system/alertmanager.service
sudo systemctl daemon-reload
sudo rm /usr/local/bin/alertmanager /usr/local/bin/amtool
sudo rm -r /etc/alertmanager /var/lib/alertmanager /etc/prometheus/rules
sudo userdel alertmanager
rm -rf ~/alert-demo ~/monitoring-installsudo tee /etc/prometheus/prometheus.yml > /dev/null <<'YAML'
global:
scrape_interval: 15s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["127.0.0.1:9090"]
- job_name: node
static_configs:
- targets: ["127.0.0.1:9100"]
YAML
sudo systemctl reload prometheusNext steps
- Look at the same data as graphs: build your first Grafana dashboard.
- Run scheduled checks that can feed metrics: schedule jobs with cron.
- Keep logs from filling the disk you just alerted on: manage log files with logrotate.
- Lock the server's ports down: set up UFW on Ubuntu 26.04.
Related tutorials
- How to Run Your App as a systemd Service on Ubuntu 26.04You will write a systemd unit file that starts your app at boot, restarts it when it crashes, runs it as an unprivileged user, and sends its logs to the journal.
- How to Set Up a UFW Firewall on Ubuntu 26.04You will turn on UFW with a deny-by-default policy, allow SSH before you enable it so you stay connected, open only the web ports your server needs, and learn to list, remove and limit rules.
- How to Schedule Jobs with Cron on LinuxYou will schedule a command to run every minute, then on a real timetable, read the cron syntax, and learn why a job that works in your shell can fail under cron.
Need this done for you?
Padding does this work for growing companies, from the first diagnosis to a system that keeps working.
Get new tutorials