AI/Tasks/OrchestratorUpgrades/Overview.txt
|
# **Orchestrator Subsystem — Upgrade Phases**
## **Shared Orchestration Contracts** - Phase order: Phase 01 state model and decision pipeline -> Phase 02 cancellation, budgets, and retry semantics -> Phase 03 tool policy and authorization -> Phase 04 validation, repair, and loop progress tracking -> Phase 05 weather and tool-specific routing hardening -> Phase 06 telemetry, memory privacy, and final integration. - Each phase must declare entry criteria, exit criteria, and the artifacts it owns so later phases do not overwrite earlier contracts. - Shared contracts that must remain stable across phases: - run-state semantics - decision schema and validation behavior - tool authorization ordering - cancellation, deadline, and timeout semantics - retry and repair budgets - path and file-operation safety - telemetry redaction and retention rules - compatibility expectations for runtime, platform, and optional capabilities - Verification method for every acceptance criterion must be explicit: unit test, integration test, static check, benchmark, manual review, or runtime diagnostic. - Common output contract for phase prompts: summary, files changed, phase-specific model, tests added/updated, validation results, and step-budget report. - Redaction and resource-limit rules apply to logs, telemetry, replay data, and diagnostics in every phase. - Acceptance tests should use stable naming so a single targeted filter can find the phase-specific coverage. - Mandatory contract baseline for every phase: - Contract version identifiers for run state, decisions, tool results, telemetry events, and memory records. - One authoritative ownership matrix for run state, validation, repair, authorization, retry, execution, finalization, telemetry redaction, and memory precedence. - Explicit precedence rules and conflict outcomes for user intent, memory preference, repair output, policy constraints, and deadline exhaustion. - Numeric resource ceilings for run duration, iterations, tool calls, model calls, output size, repair attempts, loop-history entries, and telemetry retention. - End-to-end acceptance scenarios for blocked mutations, clarifications, cancellations, retry exhaustion, low-confidence weather routing, and prompt-injection resistance. - Rollback and compatibility guidance for major contract changes, especially when entry points or persisted state formats change. - Each phase must define what it owns, what it is forbidden from changing, and which earlier contracts must remain intact. ## **Phase 01 — State Model & Decision Pipeline Architecture** This phase replaces the implicit mutable state machine with explicit transitions and typed orchestration outcomes. ### **Entry Criteria** - The orchestrator boundary and core responsibilities are known. - The phase owns state-model redesign, phase ordering, and decision-flow ownership only. ### **Exit Criteria** - The orchestration state machine is explicit enough to implement without hidden side effects. - Decision phases, policy phases, tool execution phases, and finalization steps are clearly separated. - Tests for invalid state transitions and shared-state mutations are named. ### **Core Objectives** - Replace mutable run-loop state with immutable or clearly reduced state records. - Split `ProcessIterationAsync` into typed phases such as `GenerateDecision`, `ValidateOrRepairDecision`, `ApplyPolicyChecks`, `ResolveToolArguments`, `AuthorizeToolCall`, `ExecuteTool`, `ReduceToolResultIntoState`, and `DecideContinuation`. - Centralize state mutation behind a single reducer instead of scattered `ref` updates. - Define transition results as explicit outcomes: continue, retry, fail, finalize, require clarification, or block. - Make invariant boundaries observable: finalization only succeeds when required file or output conditions are satisfied. ### **Outcome** The Orchestrator becomes a **predictable state machine** rather than a large mutation-heavy loop with hidden assumptions. --- ## **Phase 02 — Cancellation, Budgets & Retry Semantics** This phase makes the orchestrator safe under time pressure, tool hangs, and repeated model failures. ### **Entry Criteria** - Phase 01 state boundaries and phase ordering are stable. - Shared result, retry, and cancellation contracts are available to async consumers. ### **Exit Criteria** - One total run budget, explicit retry reasons, and cancellation/deadline semantics are precise enough for implementation. - Tests for timeout, cancellation, retry exhaustion, and partial side effects are named. ### **Core Objectives** - Add `CancellationToken` support across run lifecycle and tool callbacks. - Introduce a total run budget covering model calls, tool calls, elapsed time, and output tokens. - Replace string-based retry heuristics with structured failure codes and retryability flags. - Add per-tool time limits, run deadlines, and cancellation propagation. - Ensure deterministic failures are not retried unless the repair action changes the underlying cause. ### **Outcome** The Orchestrator becomes **cancellable, budget-aware, and failure-aware** rather than relying on implicit iteration and hope. --- ## **Phase 03 — Tool Policy & Authorization Hardening** This phase turns tool routing into a single registry-based policy layer with explicit authorization gates. ### **Entry Criteria** - The orchestrator decision pipeline and run budget contracts are stable. - Tool inventory and execution policy boundaries must not be reopened here. ### **Exit Criteria** - A centralized `ToolPolicy` registry defines names, schemas, safety, mutation type, execution mode, and output constraints. - Mutation authorization checks are centralized and fail closed. - Negative tests for dangerous paths, unknown tools, and invalid tool arguments are named. ### **Core Objectives** - Create a `ToolPolicy` model with registry metadata for tool category, destructive status, path enforcement, loop behavior, schema validation, and output limits. - Centralize tool validation, authorization, and strict-budget handling behind one policy engine. - Enforce path-root and workspace policy checks before mutation. - Add structured tool execution result models with error codes, retryability, side-effect status, and redacted diagnostics. - Ensure write operations verify actual postconditions instead of only file existence. ### **Outcome** Tool execution becomes a **trusted, policy-enforced boundary** rather than a set of loosely coordinated ad hoc checks. --- ## **Phase 04 — Decision Validation, Repair & Loop Progress Tracking** This phase upgrades validation and guard logic to be deterministic, bounded, and resistant to loops. ### **Entry Criteria** - Shared decision schema and tool policy contracts exist. - Phase 03 authorization rules are stable and must not be weakened. ### **Exit Criteria** - Structured validation errors, revision budgets, and loop-progress budgets are precise enough to implement directly. - Tests covering malformed decisions, repair exhaustion, and repeated CALL loops are named. ### **Core Objectives** - Replace ad hoc parsing and heuristics with a versioned decision schema and structured validation errors. - Add a global decision-repair budget and preserve malformed original decisions for diagnostics. - Add bounded streaming validation for long final answers and tool decisions. - Replace one-call loop guards with bounded recent-history tracking and progress signatures. - Separate same-call, same-target, and same-evidence detection policies. ### **Outcome** The Orchestrator becomes **more resistant to malformed decisions and no-progress loops** without sacrificing controllability. --- ## **Phase 05 — Weather, Search & Tool-Specific Routing Hardening** This phase hardens domain-specific routing logic and provenance checks for NOAA and web discovery flows. ### **Entry Criteria** - Policy, validation, and loop tracking are stable. - Staged tool policy and authorization boundaries are already in place. ### **Exit Criteria** - ZIP and location discovery flows require provenance, confidence checks, and structured candidate scoring. - Tests for low-confidence ZIP extraction and noisy search results are named. ### **Core Objectives** - Strengthen NOAA and ZIP candidate extraction with provenance validation. - Score multiple candidate ZIPs against city/state or requested geography before execution. - Avoid accepting unrelated five-digit values from noisy search content. - Add explicit fallback behavior when confidence is insufficient: ask for the ZIP or force clarification. - Preserve domain-specific routing rules without allowing them to bypass general safety checks. ### **Outcome** Weather-specific tool flows become **more accurate and less mistake-prone** while still honoring general policy boundaries. --- ## **Phase 06 — Telemetry, Memory Privacy & Final Integration** This phase completes the orchestration hardening pass by improving observability, redaction, and system integration. ### **Entry Criteria** - Prior phases have implemented the core policies, validation, retry, and routing boundaries. - Observability may describe behavior but may not redefine safety or authorization semantics. ### **Exit Criteria** - Structured event logging, retention configuration, redaction rules, and regression coverage are concrete enough to test mechanically. - Full phase integration tests cover safety, determinism, and bounded telemetry. ### **Core Objectives** - Add structured event fields: run ID, attempt ID, iteration number, tool name, result size, policy outcome, retry reason, cancellation state, and child diagnostics. - Redact secrets, tokens, authorization headers, and sensitive path data before persistence. - Bound memory retention and add retention/expiration policy by data class. - Ensure memory-derived preferences cannot override current explicit user constraints. - Add regression coverage for prompt injection, memory leakage, loop-control behavior, and concurrent runs on the same orchestrator instance. ### **Outcome** The Orchestrator becomes **observable, privacy-aware, and production-ready** for extended autonomous operation. |