|
| 1 | +--- |
| 2 | +title: Mattermost Receiver For Alertmanager in Prometheus-Stack in K8S |
| 3 | +published: true |
| 4 | +description: Add Mattermost Receiver for alertmanager in prometheus-stack in k8s |
| 5 | +tags: 'alertmanager,prometheus,monitoring,kubernetes' |
| 6 | +--- |
| 7 | + |
| 8 | +### Why |
| 9 | + |
| 10 | +Since Prometheus-Stack Alertmanager doesn’t include Mattermost as a built-in receiver, we will integrate it using a webhook via creating a Python relay application that accepts Alertmanager webhook notifications, formats them into a Mattermost-compatible payload, and sends them to a Mattermost incoming webhook URL. |
| 11 | + |
| 12 | +Then we will deploy this relay inside Kubernetes using a Deployment and expose it with a Service so Alertmanager can reliably reach it. |
| 13 | + |
| 14 | +Finally, we will configure Alertmanager using an AlertmanagerConfig resource to define a new receiver and route, so alerts matching our rules (severity/namespace/etc.) get delivered to Mattermost through the relay. |
| 15 | + |
| 16 | +### How |
| 17 | + |
| 18 | +- Prometheus Alert Rules trigger alerts. |
| 19 | +- Alertmanager processes and routes the alerts. |
| 20 | +- Alertmanager sends webhook notifications to the Python Relay Service. |
| 21 | +- The relay formats the alert payload. |
| 22 | +- The relay forwards the formatted message to Mattermost Incoming Webhook. |
| 23 | +- Alerts appear in the designated Mattermost channel. |
| 24 | + |
| 25 | +### Steps |
| 26 | + |
| 27 | +- Create an [Incoming Webhook in Mattermost][mattermost-incoming-webhook]. This webhook URL will be used by the Python relay to deliver alert messages to the desired channel. |
| 28 | + > Ensure you securely store the generated webhook URL, as it will be injected into the relay application via environment variables or Kubernetes secrets. |
| 29 | +
|
| 30 | +- [Python Relay][python-relay-app] Application |
| 31 | + |
| 32 | + > A lightweight webhook server that receives Alertmanager alerts and forwards them to Mattermost. |
| 33 | +
|
| 34 | +- Build the Docker Image [Dockerfile][dockerfile] |
| 35 | + |
| 36 | + Once the relay application is complete, build the image and push it to a container registry accessible by your Kubernetes cluster. |
| 37 | + |
| 38 | +- Deploy in Kubernetes: [K8S Manifests][k8s-manifests] |
| 39 | +- Configure Alertmanager with [AlertmanagerConfig][alertmanagerconfig-manifest] |
| 40 | + > Ensure having a [label][alertmanagerconfig-label] in AlertmanagerConfig to select via prometheus operator |
| 41 | +
|
| 42 | +- Add the Alertmanagerconfig to prometheus `values.yaml` |
| 43 | + |
| 44 | + ```yaml |
| 45 | + ⋮ |
| 46 | + alertmanager: |
| 47 | + alertmanagerSpec: |
| 48 | + alertmanagerConfigSelector: |
| 49 | + matchLabels: |
| 50 | + alertmanagerConfig: alertmanager-mattermost |
| 51 | + ⋮ |
| 52 | + ``` |
| 53 | + |
| 54 | +[mattermost-incoming-webhook]:https://docs.mattermost.com/integrations-guide/incoming-webhooks.html |
| 55 | +[python-relay-app]: https://github.com/cod3mas0n/k8s-alertmanager-mattermost/blob/main/src/app.py |
| 56 | +[dockerfile]: https://github.com/cod3mas0n/k8s-alertmanager-mattermost/blob/main/src/Dockerfile |
| 57 | +[k8s-manifests]: https://github.com/cod3mas0n/k8s-alertmanager-mattermost/tree/main/kubernetes |
| 58 | +[alertmanagerconfig-manifest]: https://github.com/cod3mas0n/k8s-alertmanager-mattermost/blob/main/kubernetes/alertmanagerConfig.yaml |
| 59 | +[alertmanagerconfig-label]: https://github.com/cod3mas0n/k8s-alertmanager-mattermost/blob/main/kubernetes/alertmanagerConfig.yaml#L6-L7 |
0 commit comments