yansonb/improvmx-loki-connector

improvmx-loki-connector

A connector to retrieve email logs from the ImprovMX API and push them to https://grafana.com/oss/loki/.

This service runs continually, fetching logs for all active domains in your ImprovMX account and forwarding them to a Loki instance. It maintains state to ensuring logs are not duplicated and provides a catch-up mechanism for historical data.

Features

• Automated Domain Discovery: Automatically detects and fetches logs for all active domains.
• State Persistence: Tracks the last queried timestamp for each domain to prevent duplicates.
• Smart Catch-Up: Efficiently retrieves historical logs in chunks if the service has been down or for initial backfill.
• Incomplete Log Tracking: Monitors logs that haven't reached terminal states (DELIVERED/HARD-BOUNCE) and re-queries them for new events, ensuring complete event histories.
• Chronological Ordering: Maintains strict chronological order of events in Loki, even across re-queries.
• Configurable: extensive configuration options via environment variables.

Configuration

The application is configured using environment variables:

Kubernetes Probes

The application exposes an HTTP server (default port 8080) with the following endpoints for Kubernetes probes:

• /healthz: Liveness probe. Returns 200 OK.
• /readyz: Readiness probe. Returns 200 OK.

Incomplete Log Tracking

Email logs from ImprovMX often have events that arrive over time. For example, an email may initially have a QUEUED event, followed later by DELIVERED, SOFT-BOUNCE or HARD-BOUNCE events.

This connector automatically:

• Tracks incomplete logs: Logs that haven't reached terminal states (DELIVERED/HARD-BOUNCE) are monitored
• Re-queries for updates: Each cycle checks tracked logs for new events
• Prevents duplicates: Only new events (by index) are ingested to Loki
• Maintains order: All events are sorted chronologically before pushing to Loki
• Auto-cleanup: Logs are removed from tracking after reaching terminal state or 48 hours

The state file includes an incomplete_logs section that persists this tracking across restarts.

Search Window Boundary Enforcement

The connector enforces strict search window boundaries to maintain chronological ordering:

• Events from re-queries that fall outside the current search window are deferred to the next cycle
• This prevents newer events from being ingested before older ones

Development

Local Build

bash
make build

Run

bash
export IMPROVMX_API_TOKEN=your_token
./improvmx-loki-connector

Docker

bash
docker build -t improvmx-loki-connector .
docker run --rm \
  -e IMPROVMX_API_TOKEN=your_token \
  -v $(pwd)/data:/data \
  improvmx-loki-connector

Docker Compose (Loki & Grafana)

To spin up a local observability stack with Loki and Grafana:

bash
docker compose up -d

• Grafana: http://localhost:3000 (Auto-login as Admin enabled)
• Loki: http://localhost:3100

The Loki datasource is automatically provisioned in Grafana.