Recurring mechanisms to keep the automation pipeline alive, honest, and improving. Monthly, quarterly, annual cadences with defined owners, inputs, outputs, and recalibration triggers.
Recurring ceremonies with fixed months. Triggered events (post-implementation audits, recalibration) are scheduled ad-hoc but mapped here for planning visibility.
| Ceremony | Cadence | Duration | Attendees | Input | Output |
|---|---|---|---|---|---|
| Monthly Prioritization Review | 1st Thursday of month | 60 min | CoE Lead, Tech Lead, Delivery Lead, 1 rotating process owner | Queue snapshot, delivery status, new submissions, blocker log | Re-ranked queue, blocker resolution plan, intake decisions on new submissions |
| Quarterly Pipeline Review | Last week of quarter | 90 min | CoE Lead, Executive Sponsor, Tech Lead, 2 dept heads | Pipeline health dashboard, completed audit results, score distribution analysis, stale candidate list | Score criteria recalibration decisions, stale candidate pruning, next quarter capacity commitment, strategic alignment refresh |
| Quarterly Discovery Sprint | Week 2 of quarter | 5 business days | CoE Lead + 1 analyst per target department | Department process lists, pain-point interviews, previous audit results | ≥5 new qualified intake submissions per department; updated processes-inventory for untargeted departments |
| Post-Implementation Audit | 90 days after each go-live | 45 min | CoE Lead, Process Owner, Delivery Lead | Live metrics (3 months), projected vs. actual data, user feedback | Completed audit template, lessons-learned entry, recalibration recommendation if gap >30% |
| Annual Criteria Recalibration | January | 3 hours (workshop) | Full CoE team + Sponsor + 1 external facilitator | 12 months of audit data, score distributions, modifier effectiveness analysis, strategic plan for coming year | Updated evaluation anchors, possibly adjusted weights or new modifiers; ratified by Sponsor |
| Mid-Year Strategy Check | June | 90 min | CoE Lead + Sponsor | H1 actuals vs. plan, pipeline depth, team capacity vs. demand | Go/no-go on H2 plan; mid-course correction decisions; resource request if needed |
Completed 90 days after each automation go-live. One audit per process. Results feed the lessons-learned register and trigger recalibration if projected/actual variance exceeds 30% in any dimension.
| Metric | Projected (from intake) | Actual (90-day production) | Variance % | Flag |
|---|---|---|---|---|
| Weekly hours saved | — | — | ||
| FTE equivalent saved | — | — | ||
| Annualized $ savings | — | — | ||
| Error rate (pre-automation) | Baseline | — | — | |
| Error rate (post-automation) | — | — | ||
| Build time (calendar weeks) | — | — | ||
| Transaction volume / week | — | — |
Variance = (Actual − Projected) / Projected × 100. Green flag ≤ ±15%. Yellow flag ±16–30%. Red flag >±30%. Red flags in savings or error rate trigger immediate recalibration review within the next monthly meeting.
How lessons-learned from audits and pipeline reviews systematically improve the evaluation criteria, intake template, AI gate questions, and modifier weights. This is the mechanism that prevents the framework from going stale.
Automatic triggers that force an out-of-cycle recalibration review. If any of these fire, the topic is added to the very next Monthly Prioritization Review agenda — no waiting for the quarterly cycle.
>60% of candidates in the same tier (P1/P2/P3) over any rolling 3-month window. The evaluation criteria aren't discriminating — anchors need tightening or a new dimension is missing.
Any single audit shows >30% variance in savings, hours, or error reduction. Or >2 audits in a rolling 6-month window show >20% variance in the same direction (systematic over-/under-estimation).
AI gate pass rate drops below 40% or rises above 90% over a rolling 6-month window. Too-low means the gate is screening out viable candidates; too-high means the questions aren't discriminating.
Fewer than 5 qualified candidates in the evaluation queue (across all tiers) for >2 consecutive months. The intake funnel is drying up — discovery sprints or outreach need to be intensified.
Any modifier applied to >5 processes shows zero correlation with the outcome it was designed to predict (e.g., Enabler Bonus doesn't correlate with downstream unlocks; Strategic Deadline doesn't correlate with on-time delivery).
A new regulation, internal AI policy update, or material vendor change (e.g., model provider adds data-residency region) affects >2 pipeline candidates. The AI gate questions must be reviewed for continued relevance.
Every recalibration — whether triggered or scheduled — must produce a lightweight decision record. This creates an audit trail of why the framework changed over time and prevents the same debate from recurring.
| Field | Example |
|---|---|
| Date | 2026-09-15 |
| Trigger | Triggered — Score Distribution Collapse. 67% of Q3 candidates scored P2; only 8% scored P1. |
| What changed | Implementation Ease anchors tightened: "Score 5" now requires <2 weeks (was <4 weeks). Effort weight increased from 25% → 30% (drawn from Strategic Alignment, 20% → 15%). |
| Rationale | Audits showed that build-time estimates were systematically 40% too optimistic. Tightening the anchor and increasing the weight penalizes optimistic ease scores and spreads the distribution. |
| Expected effect | P1 tier should contain 15–25% of candidates; P2 40–50%; P3 25–35%. |
| Review date | 2026-12-15 (Q4 Pipeline Review — re-assess distribution) |
| Decision owner | CoE Lead, ratified by Executive Sponsor |
These KPIs are surfaced at every Monthly and Quarterly review. Maintain them in a shared spreadsheet or lightweight BI tool. No one should have to ask "how's the pipeline doing?" — the dashboard answers it.
| Metric | Refresh | Data Source | Alert Threshold |
|---|---|---|---|
| Active Queue Depth | Weekly (automated pull from intake register) | Intake Register | <5 for >2 consecutive months → Trigger Pipeline Depth alert |
| Deliveries (Last 90d) | Monthly | Delivery Log | <1 per month for 3 consecutive months |
| Annualized $ Saved | Monthly (after 90-day audits complete) | Audit Templates | Actual <70% of projected for 2+ consecutive quarters |
| Avg Build Time | Monthly | Delivery Log | Actual >150% of estimated for >3 deliveries in a row |
| AI Gate Pass Rate | Monthly | AI Gate Results Log | <40% or >90% in rolling 6-month → Trigger AI Gate anomaly |
| P1/P2/P3 Split | Monthly | Evaluation Register | >60% in any single tier for 3 months → Trigger Score Distribution collapse |
| Stale Candidate Count | Monthly | Intake Register | >30% of queue is stale → Intensify pruning at next Quarterly Review |
| HITL Review Acceptance Rate (AI only) | Weekly | HITL Dashboard | Drops >10% week-over-week → Possible prompt drift or data shift |
| Feedback Source | Feeds Into | Mechanism | Cadence |
|---|---|---|---|
| Post-Implementation Audit (this step) | Step 2 — Evaluation anchors & weights | Lessons-Learned Register → Monthly Triage → Quarterly Recalibration | Per audit + quarterly |
| Post-Implementation Audit (this step) | Step 1 — Intake Template fields | "What should change in the intake template?" field → Monthly Review adoption | Per audit + monthly |
| AI Gate Conditional-to-Pass conversion rate | Step 3 — AI Gate question scoring | Quarterly Pipeline Review: if CONDITIONAL processes reliably pass with guardrails, question may be too conservative | Quarterly |
| Score distribution analysis | Step 4 — Priority formula & modifier weights | Recalibration triggers (Score Distribution Collapse, Modifier Ineffectiveness) | Monthly monitoring + quarterly action |
| Build-time variance (actual vs. estimated) | Step 2 — Implementation Ease anchors | Quarterly Pipeline Review: systematic over-estimation of ease → tighten anchors | Quarterly |
| Discovery Sprint output | Step 1 — Intake pipeline | New submissions → triage → evaluation queue | Quarterly |
| Annual Strategy Reset | All steps (framework vN+1) | Ratified by Sponsor; new version effective Q2 | Annual (January) |