Deterministic anomaly detection and scoped diagnosis

Production monitoring should decide when to investigate with deterministic, tested rules. An agent may gather logs, recent changes, and hypotheses after a trigger, but it does not define whether an anomaly exists or gain production authority from severity alone. Full marks require versioned policy, replayable evidence, scoped identity, deduplication, and a named human gate.

Scorecard question: How do you monitor production health and detect actionable anomalies?

Maximum-score answer: A version-controlled monitoring policy has tested thresholds, read-only diagnosis, evidence retention, and approval-tier routing.

Version the policy, not a magic filename

Use the repository’s existing monitoring system. A file such as monitoring-policy.yaml is an example, not a standard:

signals:
  checkout_error_rate:
    source: metrics/checkout/errors
    window: 10m
    baseline: same_weekday_28d
    trigger:
      type: relative_and_absolute
      minimum_events: 100
      relative_increase: 2.0
      absolute_rate: 0.02
    action:
      risk_tier: 1
      identity: incident-diagnoser
      permissions: [read_metrics, read_logs, read_deployments]
      output: incident-intake
      dedupe_window: 30m

Choose seasonality, sample size, burn-rate, statistical, or static rules that fit the signal. Do not copy a universal 1σ/2σ/3σ ladder: simple standard-deviation thresholds are not automatically appropriate for skewed, sparse, or seasonal production data.

Required execution path

1. The monitoring platform evaluates the versioned rule.
2. A trigger creates an immutable event with metric window, query, deployment IDs, and timestamps.
3. A dedicated read-only identity gathers bounded telemetry and repository context.
4. The agent drafts hypotheses and an intent.md-compatible intake record with source links.
5. A human or declared low-risk policy accepts, rejects, or escalates the record.
6. Any code change, rollback, merge, or deploy follows its own gate.

The detector and diagnosis should be testable independently. Replaying a historical window must produce the same trigger decision; the model’s narrative may vary and must be treated as analysis, not detection truth.

Prompts to copy

Analyze only the supplied alert evidence, deployment metadata, and read-only logs.
Separate observations from hypotheses. Cite every observation to a query, timestamp,
or commit. Return impact, confidence, missing evidence, and safe next checks.
Do not modify code, configuration, traffic, or production state.

Convert the accepted diagnosis into an incident intake artifact.
Include user impact, evidence links, suspected scope, non-goals,
required human owner, and the next SDLC gate. Do not claim root cause
unless the evidence proves it.

Failure modes

Alert noise overwhelms triage. Add minimum sample sizes, burn-rate or persistence rules, deduplication, and replay tests. An agent summary does not repair a bad detector.

Logs contain prompt injection. Treat telemetry and user content as untrusted data. Restrict tools and require the agent to quote evidence without following embedded instructions.

Severity grants write access. Remove that coupling. Severity may change who is paged; it must not mint credentials or bypass the production gate.

The agent names a root cause too early. Require observations, hypotheses, counter-evidence, and missing data as separate fields.

Verify the control

• Historical normal, drift, spike, and missing-data windows have tests.
• Trigger decisions are reproducible without a model.
• Diagnostic credentials are read-only and separately identifiable.
• Evidence links and tool outputs are retained under the incident policy.
• A dry run cannot merge, deploy, roll back, or mutate production.
• Accepted incidents re-enter the artifact chain through triage.

Feed accepted evidence back

Route accepted intake to Maintain and apply risk tiers from Governance and autonomy.

Q23 · Loop re-entry Turn accepted incident evidence into the next intent.

Metrics Measure useful detection and recovery.