Alert Doctor

What It Does

Alerts should mean something. Stop getting paged for nothing — Alert Doctor kills noisy alerts, fixes broken thresholds, and closes the coverage gaps that leave real incidents silent.

Alert rules are written once and rot forever: thresholds nobody revisits, duplicates that double-page, latched counters that burn for a day, and silent gaps where real incidents page nobody. Alert Doctor treats your rules like a patient: recurring checkups, evidence-based treatment, and a second opinion on every change.

Every checkup runs the same loop:

1. Proposer reads the live signal and diagnoses and treats what it finds.
2. Apply edits the project's shared alert rules and takes a rollback snapshot.
3. Verify confirms the change is actually live in Prometheus.
4. Judge audits every claim from the cycle.

The Judge's feedback folds into the next checkup, so the loop keeps tightening over time.

Core Capabilities

Evidence-Based Diagnosis

Every checkup reads a full signal window across all of the project's environments — every alert fire, metric series, and incident report — and identifies four patterns:

• Flappers: many short fires that self-resolve (needs a longer for window)
• Constant burners: one fire burning the whole window (a wrong threshold or a latched counter)
• Duplicates: two rules paging for the same failure
• Coverage gaps: incidents where no alert was firing within ±30 minutes

Deliberated Treatment

Before any change ships, the model argues every candidate both ways: the evidence for it, and the strongest honest case against it. A candidate whose counter-argument wins is dropped, and dropped ideas stay visible as an audit trail. Every surviving change cites the exact fires, burn time, or incident that justifies it.

Safe Write-Back

Auto-apply is opt-in. When enabled:

• Changes edit the project's shared alert rules once and release to every selected environment.
• Every write keeps a rollback snapshot.
• A confirmer verifies each rule is actually live in Prometheus afterward. A change that didn't land everywhere is reported honestly as drift, never assumed.

Until you enable Auto-apply, checkups are advice only.

Independent Audit

Checkups alternate between two roles. The Proposer drafts and applies refinements; the Judge, a separate model, audits the previous checkup against the raw signal:

• Flags claims the signal contradicts (grounding errors)
• Rules each change keep, revert, or adjust
• Leaves suggestions the Proposer must answer in the next cycle

The Judge never writes; the Proposer never self-approves.

One-Click Restore

Cycle #0 records your rules exactly as found, before the doctor's first checkup. Restore returns your alerts to any earlier state in one click.

Restore is different from rollback, covered in How It Works: Restore can jump to any earlier cycle, while rollback undoes only the single latest applied change and re-verifies that the undo is live in Prometheus.

Scope Safety

Rules owned by the platform, or defined by multiple alert groups, are read-only. The doctor sees their signal as context but never modifies or removes them. A checkup that produces no usable output fails loudly with the real cause, instead of quietly recording "no changes."

Adaptive Cadence

Leave cadence on Auto: checkups run frequently while rules are actively being tuned, then ease toward daily and weekly as alerting stabilizes (see How It Works for the exact ramp). Hourly, daily, and weekly fixed cadences are also available, plus on-demand runs at any time.

How to Set It Up

1. Open Alert Doctor and pick the project you want treated.

   

2. Choose observation points: the environments to observe. Rules are edited once and released to every environment you select.

   

   If the project is already managed by another case, Alert Doctor rejects the request and names the existing case instead.

3. That's it. The doctor discovers each environment's Prometheus and your incident history on its own, records the Cycle #0 baseline, and is ready for its first checkup.

   

4. Read the first proposal. If you like it, enable Auto-apply to put the doctor on the case: autonomous write-back with rollback snapshots and live verification.

Once a case is running, a few more controls are available from its dashboard:

• Cadence — choose Auto, Hourly, Daily, or Weekly. Auto is the adaptive option described in Core Capabilities.
• Focus — enter a free-text business need in the Focus next field (placeholder: "Target this checkup — e.g. only payments alerts") to scope what the next checkup should prioritize.
• Auto-apply — toggle it on and confirm in the Enable auto-apply dialog. This means every future cycle edits the shared rule base, releases it to every environment in the set, and verifies the change landed. The toggle reads Staying in Praxis when off and Override + release each cycle when on.
• Roll back or re-release — use Roll back to undo the latest applied cycle. You'll be asked why, and that reason is recorded in the audit report. Use Re-release to retry a change that failed to land or drifted.
• Linked incident workspace (optional) — supply a workspace URL and an API key (username optional) so its incident reports feed the doctor's signal alongside the local incident store. Linking without an API key is rejected.
• PDF export — export a report reviewing every change Alert Doctor has made to a project's alert rules.
• Pause, reactivate, or delete — pause or reactivate a case from the dashboard at any time without losing its history or cadence position.

Example Checkup

Cycle #1 · Proposer                    Applied · 3 rules · Verified live

SYMPTOM     PrometheusDiskFull paged 1x and burned 24.1h at value 0.0.
            A day-long page with nothing wrong.

DIAGNOSIS   The expression reads the status metric's polarity backwards;
            two release rules double-page the same failure.

TREATMENT   modify  PrometheusDiskFull: flip == 0 to == 1
                    evidence: 24.1h burn @ 0.0
            remove  MaintenanceReleaseFailure: duplicate of ReleaseFailure
            add     AgentAPIHttp5xxRate: coverage gap from incident 07-08

Cycle #2 · Judge

ASSESSMENT  Sound and well-grounded. The flip is verified by the 24.1h
            burn at value 0.0; all three changes hold up.

SUGGESTIONS advisory · 1: investigate NodeEphemeralStorageWarning flapping
            (13 fires, never near critical).

How It Differs from Incident Responder

Incident Responder is reactive: it investigates what already broke. Alert Doctor is proactive: it continuously tunes the rules so the right things page you in the first place. They compound: Alert Doctor reads Incident Responder's incident history to find coverage gaps.

How It Works

Every checkup follows the same four beats: Diagnose, Deliberate, Treat, Confirm.

A case observes a flat, equal set of environments under one project. There is no privileged "anchor" environment.

Cycles strictly alternate two roles:

1. Proposer (odd cycles) — a faster model drafts a refinement: additions, removals, or modifications to alert rules. If Auto-apply is on, it applies the change. It reads the Judge's feedback from the previous cycle as memory.
2. Judge (even cycles) — a stronger model reviews the Proposer's last cycle against the signal and returns structured feedback. The Judge never applies or writes anything itself.

Each cycle is a single LLM call, with no tool use, fed a pre-fetched signal: the current Prometheus rules, time-series evidence, and recent incident reports. The model is prompted to reason like a doctor: state a symptom and diagnosis, consider differential diagnoses, argue every candidate change both for and against, and only then emit a patch backed by cited evidence.

Every cycle works toward three standing goals:

• Add coverage where a real failure has no alert.
• Remove alerts that are useless, duplicate, or dead.
• Tune alerts that are right in spirit but need threshold, window, scope, or severity adjustment.

The signal lookback window scales with cadence — roughly a day for Auto or Hourly, a week for Daily, two weeks for Weekly — and always covers at least the time since the last run, capped at 14 days. This keeps a paused-and-resumed case from developing a blind spot.

Scheduling runs as a periodic sweep across every organization's cases, skipping organizations over their usage cap. The Auto cadence's ramp is driven by completed-run count, not elapsed time (see Core Capabilities).

When Auto-apply is on, a release runs through these steps:

1. Alert Doctor edits the project's shared alert rule definitions exactly once per cycle, so every environment in the set inherits the change.
2. It snapshots the prior state first, so the change can be rolled back later.
3. It releases each environment in the set one by one.
4. Rule changes are grouped by which underlying resource owns each rule. A rule owned by more than one resource, or a platform-managed rule with no editable owner, is automatically skipped rather than written.
5. A background step re-reads each environment's live Prometheus rules to confirm the write actually landed, rather than trusting the release call alone.
6. Each change is stamped confirmed true, false, or null, and the cycle gets an overall status.

Cycle status after a release:

A release can also fail outright — for example, an unreachable environment — in which case the affected environment is marked separately from a confirmed drift.

Rollback restores the prior snapshot and re-releases it. Only the single latest applied, non-reverted cycle can be rolled back, to avoid clobbering newer changes. Re-release retries pushing an already-applied change without re-editing the base — useful after a partial release or an unreachable environment.

Safety Boundaries

Alert Doctor proposes and, only when explicitly enabled, applies a narrow, reversible class of change. It never touches anything beyond a project's alerting rules.

• Auto-apply is off by default and requires an explicit, separately confirmed opt-in per case. While off, Alert Doctor only proposes changes and never touches a live environment.
• Narrow write scope. Even when enabled, Alert Doctor only ever edits the shared alert-rule resource's definitions. It does not touch other Kubernetes resources, restart workloads, or deploy application code.
• The Judge never writes. Only the Proposer cycle can apply a change; the Judge cycle only reviews and gives feedback.
• Ambiguous rules are skipped, not guessed at. Rules owned by more than one resource, or platform-managed rules with no editable owner, are automatically skipped rather than modified.
• Read-only cluster discovery. The Kubernetes client used for Prometheus discovery is read-only — list and proxy-GET only, with no write path.
• Release-then-confirm. Every write is re-verified by re-reading live Prometheus, rather than trusting the write call alone.
• Every applied cycle is reversible. Rollback is always available, though only the latest applied cycle can be rolled back — an explicit guard against clobbering newer state.
• Deleting a case does not undo its changes. Deleting a case removes only Alert Doctor's own history and configuration. It does NOT revert the environments' already-applied live alert rules.
• Concurrency and usage limits are enforced. Organization-level usage caps gate the unattended scheduler loop, and a single-flight lease prevents two operations — checkup, delete, rollback — from running concurrently on the same case.

Integrations

• Facets Control Plane — the read path for project and environment metadata, and the sole write path for pushing rule changes and releasing environments. Every control-plane operation runs automatically during a checkup when Auto-apply is on.
• Prometheus — read for the current alerting rules and time-series evidence during every checkup, and the target that post-release confirmation re-reads to prove a change landed.
• Praxis incident reports — an optional additional signal source: either the local incident store, when Alert Doctor runs inside the same Praxis deployment as Incident Responder, or a linked remote Praxis workspace's incidents API, connected by supplying a URL and API key, plus an optional username.
• Kubernetes API (via a downloaded Facets kubeconfig) — read-only, used only to auto-discover the in-cluster Prometheus service. It is not a general troubleshooting interface.

Unlike Incident Responder, Alert Doctor has no Slack, Microsoft Teams, Mattermost, email, or other chat and notification integration. Checkup results surface only in the case dashboard and PDF export.

Tip: You can also perform case management, checkups, rollback, and PDF reporting programmatically. See the API Reference for details.

Permissions

All routes require an authenticated Praxis session, validated through a user cookie, and every action is scoped to the caller's organization. Facets API access for reading environments and writing rules is resolved through the organization's Facets integration credential, with an explicit fallback to the user's own personal access token on organizations that only have a personal access token configured. Alert Doctor has no distinct in-app role tiers — such as admin versus viewer — beyond requiring an authenticated organization member.

Troubleshooting

Related

• Incident Responder - Correlates Kubernetes, cloud, and log signals to diagnose infrastructure incidents; Alert Doctor can optionally consume its incident reports as an extra signal.
• K8s Inspector - Natural-language Kubernetes troubleshooting, conceptually adjacent but narrowly focused here on tuning the shared Prometheus alert rule set rather than open-ended cluster Q&A.
• Release Debugger - Diagnoses failed deployments with the same evidence-first, symptom-and-diagnosis framing Alert Doctor's clinical readout uses, but for release failures rather than alerting-rule quality.
• API Reference - Programmatic access to case management, checkups, rollback, and PDF reports.