AlertSnitch Python Version

Captures Prometheus AlertManager alerts and writes them in a MySQL or PostgreSQL database for future examination.

Because given a noisy enough alerting environment, offline querying capabilities of triggered alerts is extremely valuable.

How does it work

1. You stand up AlertSnitch using Docker or manually
2. You setup AlertManager to point at it and propagate your alerts in
3. Every alert that gets triggered reaches your database
4. Profit!

graph TD
    A[alertmanager] -->|POST|B(AlertSnitch)
    B --> |Save|C(MySQL/PG Database)
    C -.-|Graph|G[Grafana]
    C -.-|Query|D[MySQL/PG Client]
    style B fill:#f9f,stroke:#333,stroke-width:1px
    style C fill:#00A0A0,stroke:#333,stroke-width:1px
    style D fill:#00C000
    style G fill:#00C000

Features

• Receive webhooks from Prometheus AlertManager with per-project Bearer tokens.
• Store alerts in MySQL or PostgreSQL, with per-project alert tables plus a Grafana summary table.
• Manage projects from the web UI, including project metadata, owners, enabled state, and token rotation.
• Local username/password login with global admin and project admin roles.
• Project-code login for passwordless project self-service: view only that project's dashboard and token, without any mutation permission.
• Admin user management UI for creating, disabling, deleting, and resetting local accounts.
• Project list search and filters by keyword, personnel, environment, and status.
• Provisioned Grafana dashboards with optimized query columns, keyword filtering, personnel columns, and UTC datasource handling.
• Health/readiness probes and Prometheus metrics export.
• Null backend for testing and Docker Compose support for MySQL + AlertSnitch + Grafana.

Quick Start

Using Docker

# Pull and run with MySQL
docker run --rm \
    -p 9567:9567 \
    -e ALERTSNITCH_BACKEND=mysql \
    -e ALERTSNITCH_DSN="mysql+pymysql://user:password@host:3306/alertsnitch" \
    alertsnitch:latest

# Or with PostgreSQL
docker run --rm \
    -p 9567:9567 \
    -e ALERTSNITCH_BACKEND=postgres \
    -e ALERTSNITCH_DSN="postgresql+psycopg2://user:password@host:5432/alertsnitch" \
    alertsnitch:latest

# For testing (null backend - just logs)
docker run --rm \
    -p 9567:9567 \
    -e ALERTSNITCH_BACKEND=null \
    alertsnitch:latest

Kubernetes

Kubernetes manifests are available under deploy/kubernetes/. They mirror the Docker Compose layout by running MySQL, AlertSnitch, and Grafana in one Pod. The default AlertSnitch image is harbor.meos.center/public/alertsnitchpython:v1.8.

cp deploy/kubernetes/secret.example.yaml deploy/kubernetes/secret.yaml
kubectl apply -f deploy/kubernetes/namespace.yaml
sh deploy/kubernetes/create-configmaps.sh
kubectl apply -f deploy/kubernetes/secret.yaml
kubectl apply -k deploy/kubernetes

See deploy/kubernetes/README_CN.md for PVC, Ingress, Alertmanager, Grafana, and operations examples. See docs/GRAFANA_OPERATIONS.md for Grafana MySQL storage and dashboard import operations.

Manual Installation

1. Install dependencies

pip install -r requirements.txt

2. Configure environment variables

cp .env.example .env
# Edit .env with your configuration

3. Initialize the database

python -m alertsnitch.db_init

4. Run AlertSnitch

python -m alertsnitch.app

Or use uvicorn directly:

uvicorn alertsnitch.app:app --host 0.0.0.0 --port 9567

Configuration

Environment Variables

Variable	Description	Default
ALERTSNITCH_BACKEND	Database backend: mysql, postgres, or null	mysql
ALERTSNITCH_DSN	Database connection string	Required
ALERTSNITCH_ADDR	Listen address	0.0.0.0:9567
ALERTSNITCH_DEBUG	Enable debug logging	false
ALERTSNITCH_ADMIN_PASSWORD	Initial/default password for the built-in admin user	Required for first admin bootstrap
ALERTSNITCH_SESSION_HOURS	Web login session lifetime in hours	24
ALERTSNITCH_SESSION_COOKIE_SECURE	Send session cookie only over HTTPS	false

Web UI and Access Model

AlertSnitch supports three access modes:

Access mode	Login method	Permissions
Global admin	Username admin plus password	Manage all projects, rotate tokens, enable/disable/delete projects, and manage local users at /admin/users.
Project admin	Local username/password with role project_admin	Create projects, and manage/delete projects whose projects.personnel contains the username or display name.
Project-code viewer	Project code only, no password	View only /dashboard/{project_code} and /project/token for that project. Can copy the token, but cannot rotate token or modify/delete anything.

Create the default admin and query-helper columns on existing databases:

python scripts/migrate_users.py
python scripts/migrate_alert_query_fields.py

Create or update a project administrator:

python scripts/create_user.py <project_admin_username> '<strong-password>' --role project_admin --display-name '<display-name>'

Project administrator authorization is driven by the projects.personnel column. Use the display name or username there; multiple owners can be separated by commas or semicolons. Projects created by a project administrator automatically include that administrator as an owner.

Global admins can review security and administrative changes at /admin/audit. The audit log records account changes, project create/delete/enable changes, and token rotations. Web sessions are persisted in the database when a SQL backend is enabled, with in-memory storage used only as a fallback.

Database Connection Strings

MySQL:

mysql+pymysql://user:password@localhost:3306/alertsnitch

PostgreSQL:

postgresql+psycopg2://user:password@localhost:5432/alertsnitch

Setting up AlertManager

Configure Prometheus AlertManager to forward alerts to AlertSnitch:

receivers:
- name: alertsnitch
  webhook_configs:
    - url: http://<alertsnitch-host>:9567/webhook
      send_resolved: true
      http_config:
        authorization:
          type: Bearer
          credentials_file: /etc/alertmanager/secrets/alertsnitch-token/token

route:
  routes:
  - receiver: alertsnitch
    continue: true  # Continue to other receivers

API Endpoints

Webhook Receiver

• POST /webhook - Receive alerts from AlertManager

Health Checks

• GET /-/ready - Readiness probe (returns 200 if ready to accept webhooks)
• GET /-/health - Liveness probe (returns 200 if database is reachable)

Metrics

• GET /metrics - Prometheus metrics endpoint

Info

• GET / - Service information

Prometheus Metrics

AlertSnitch exposes the following metrics:

• alertsnitch_webhooks_received_total - Total webhooks received (by status)
• alertsnitch_alerts_processed_total - Total alerts processed (by status and backend)
• alertsnitch_alerts_save_errors_total - Total errors saving alerts
• alertsnitch_webhook_processing_seconds - Webhook processing time histogram
• alertsnitch_database_health - Database health status (gauge)
• alertsnitch_application_ready - Application readiness status (gauge)

Grafana Integration

AlertSnitch ships with Grafana dashboards and provisioning files under grafana/ and deploy/grafana/provisioning/. The MySQL datasource stays on UTC, dashboard timezone is Asia/Shanghai, and current dashboard time filtering is based on updated_at to avoid MySQL TIMESTAMP timezone shifts on historical received_at data.

The dashboards visualize:

• Alert frequency over time
• Most common alerts
• Alert duration
• Alert status distribution
• Project overview, including personnel
• Keyword search across alert name, summary, description, job, instance, namespace, pod, and node

Time Semantics

• starts_at is the Alertmanager alert start time. Use it for durations and MTTR-style analysis.
• updated_at is the AlertSnitch write time. Bundled Grafana dashboards use it for time range filters, trends, recent-window counters, and displayed receive time.
• received_at is retained for compatibility. Historical MySQL deployments may store it as TIMESTAMP, so do not use it for Grafana time filtering.
• AlertSnitch stores timestamps in UTC. MySQL connections are initialized with SET time_zone = '+00:00'; dashboard SQL formats DATE_ADD(updated_at, INTERVAL 8 HOUR) for Beijing-time display.
• Alertmanager may resend long-running alerts with a fresh receive time while keeping the original startsAt; dashboards filtered by starts_at may not show those resends as recently ingested alerts.
• Alerts visible as suppressed or inhibited in Alertmanager may not be delivered to webhook receivers, so they may be absent from AlertSnitch.
• See docs/TIME_HANDLING.md before changing backend timestamp handling or Grafana SQL.

Before importing dashboards into an existing database, run:

python scripts/migrate_alert_query_fields.py
python scripts/refresh_grafana_summary.py

Example query:

SELECT 
  DATE_FORMAT(starts_at, '%Y-%m-%d %H:00:00') as time,
  COUNT(*) as alert_count
FROM alerts
WHERE starts_at >= NOW() - INTERVAL 24 HOUR
GROUP BY time
ORDER BY time;

Testing

Test with curl

# Send a test alert
curl -X POST http://localhost:9567/webhook \
  -H "Authorization: Bearer <project-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4",
    "groupKey": "test-group",
    "status": "firing",
    "receiver": "alertsnitch",
    "groupLabels": {},
    "commonLabels": {},
    "commonAnnotations": {},
    "externalURL": "http://alertmanager:9093",
    "alerts": [
      {
        "status": "firing",
        "labels": {
          "alertname": "TestAlert",
          "severity": "warning"
        },
        "annotations": {
          "summary": "Test alert",
          "description": "This is a test alert"
        },
        "startsAt": "2024-01-01T00:00:00Z",
        "endsAt": null,
        "generatorURL": "http://prometheus:9090/graph",
        "fingerprint": "abc123"
      }
    ]
  }'

Check health

curl http://localhost:9567/-/health
curl http://localhost:9567/-/ready

View metrics

curl http://localhost:9567/metrics

Architecture

The application is built with:

• FastAPI - Modern, fast web framework
• SQLAlchemy - SQL toolkit and ORM
• Pydantic - Data validation
• Prometheus Client - Metrics export
• Uvicorn - ASGI server

Security

⚠️ Warning: AlertSnitch is not meant to be exposed to the internet. It should only run in an internal network reachable by AlertManager.

There is no authentication or encryption provided. Use reverse proxies or network policies for security if needed.

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.