Prometheus is among the most popular tools for server monitoring. It aggregates metrics from potentially countless different sources (provided through so called exporters), stores them in its own time series database and allows them to be queried, e.g. by visualization tools like Grafana. One central aspect of monitoring, besides visualization of data, is alerting. A very common use case for Prometheus is to watch a server’s runtime performance metrics, e.g. its CPU utilization or hard drive usage, and be notified when things go unusual, for instance, when the hard drive is close to running out of free space. Or you might want to use the blackbox_exporter to watch an HTTP or TCP endpoint for being available and get immediate alerts as it goes down. For this, Prometheus integrates with Alertmanager, their own little daemon process whose sole purpose is to send out alert notifications triggered by Prometheus queries.

Alertmanager supports for different receivers, among the most popular of which arguably is email. But there are many more, including the generic webhook one, which is quite powerful.

E-Mail is well and good, but also feels a bit “heavy-weight” and outdated. You might rather want to receive your server monitoring alerts through your favorite messenger, for instance, Telegram. In the following, you’ll learn how to do so with minimal effort by using Telepush. As an example, we are going to set up Prometheus to watch a locally running web service and send a Telegram messages once it’s unavailable.

Prerequisites

I assume you have an up and running setup of Prometheus, Alertmanager and Blackbox Exporter (only required for this example) already.

Obtain a token from Telepush

You will receive notifications through Telepush’s bot. From a technical perspective, an alert takes this way:

1

Prometheus --> Alertmanager --> Telepush API --> Telegram

That is, Alertmanager sends all alerts to Telepush’s web API, which then forwards it to your Telegram chat with the bot. To instruct Telepush where to route which messages, you initially have to get in contact with Telepush’s Telegram bot and obtain a token.

Open a new chat with TelepushBot (Telepush was formerly called MiddlemanBot) and simply send /start. You’ll receive a token. Keep it for the next step.

Configure Alertmanager

Let’s continue with the interesting part and configure Alertmanager to send alerts via Telepush. In my case, the Alertmanager’s config file lives at /opt/alertmanager/alertmanager.yml.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

global:


route:
  group_by: ['alertname']
  group_wait: 10s       
  receiver: 'telepush'  


receivers:
- name: 'telepush'
  webhook_configs:
  - url: 'https://telepush.dev/api/inlets/alertmanager/'    
    http_config:

Then, apply the configuration through a reload:

1

curl -X POST http://localhost:9093/-/reload

🎉 You’re done