diff --git a/symphony/CLAUDE.md b/symphony/CLAUDE.md index da5dca1..a04d342 100644 --- a/symphony/CLAUDE.md +++ b/symphony/CLAUDE.md @@ -29,9 +29,12 @@ repo to provide the integration. See "Tracker contract" below. ## User flow 1. **First time**: `/symphony-init` in the project root. Interview + write. -2. **Steady state**: `/symphony-tick` is fired by a `/schedule` cron entry (also set up - by init unless the user declines). -3. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates. +2. **Steady state**: `/symphony-tick` is fired by a `/schedule` entry inside the long-lived + orchestrator session (also set up by init unless the user declines). +3. **Other triggers**: ticks can also be fired by remote channels (Discord DM, webhook, + another session). The reconciliation logic handles cross-session liveness via worker + heartbeats. +4. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates. ## Tracker contract @@ -66,31 +69,65 @@ The skill and worker read these instructions when they need to talk to the track the project CLAUDE.md is silent on the tracker, the orchestrator errors with `tracker_integration_missing` and exits the tick. -## State +## Runtime model -Symphony's spec calls for in-memory orchestrator state. Cron-fired ticks are cold starts, -so we persist to `.symphony/state.json` in the consuming repo: +The orchestrator runs inside a **long-lived Claude Code session**. `/schedule` doesn't +cold-start each tick — it sets the prompt for the running session. Tasks spawned in +tick N stay visible to `TaskList` / `TaskGet` in tick N+1 within that session, so the +session itself is the primary store of "what's running right now." -- `running` — issue_id → { worker_task_id, started_at, worktree_path } +Symphony stays open to **other trigger sources** too: a Discord channel could DM the +session and ask for a tick, a webhook hook could trigger one, and a separate user +session could conceivably spawn tasks against the same project. To keep all three +modes honest, two layers carry state: + +1. **`TaskList` / `TaskGet`** — fast path, authoritative for liveness *within this session*. +2. **`.symphony/state.json` + worker heartbeats** — durable record across sessions and + restarts. Authoritative when `TaskList` doesn't know about a task ID. + +### Reconciliation logic + +For each entry in `state.json`'s `running` map: + +- **In our `TaskList`** → it's our task; check status via `TaskGet`; reconcile normally. +- **Not in our `TaskList`** → another session may own it, or the session that spawned it + exited. Read the worker's heartbeat file (`.symphony/heartbeats/.json`). + - **Heartbeat fresh** (within `agent.heartbeat_stale_ms`, default 5min) → leave alone; + another session is owning it. + - **Heartbeat stale** → consider abandoned; mark for retry, drop from `running`. + - **No heartbeat** → ditto. + +This keeps the single-session case simple (TaskList wins) while supporting +multi-session and crash recovery cleanly. + +### State file shape + +- `running` — issue_id → { task_id, session_id, started_at, worktree_path } - `retry_attempts` — issue_id → { attempt, due_at_ms, last_error } - `claimed` — set of issue IDs reserved this tick (cleared on dispatch or release) +- `completed` — set of issue IDs (bookkeeping) -Reconciliation reads the file at the top of every tick, queries `TaskList` to confirm -which workers are actually still running, and rewrites the file before returning. +### Worker heartbeats + +Workers write `.symphony/heartbeats/.json` with `{ timestamp, status, task_id, session_id }` +every minute (configurable via `agent.heartbeat_interval_ms`). On graceful exit they +remove the file. Stale heartbeats — and stale `running` entries with no heartbeat at +all — are how Symphony detects abandonment. ## Mapping to the spec | Symphony component | Claude Code mechanism | |---------------------------|--------------------------------------------------------| -| Polling daemon | `/schedule` cron entry firing `/symphony-tick` | +| Polling daemon | `/schedule` firing `/symphony-tick` in a long-lived session | | Workflow loader | Skill body — reads & parses `WORKFLOW.md` | | Issue tracker client | Project CLAUDE.md instructions + whatever tools fit | -| Orchestrator | Skill, invoked once per tick; state on disk | +| Orchestrator | Skill, runs each tick within the same session | | Workspace manager | `EnterWorktree` per worker | | Agent runner | `Agent` tool with `subagent_type: symphony-worker` | | Hooks | Bash steps inside the worker prompt | -| Concurrency cap | Lockfile + `TaskList` poll inside the dispatcher | +| Concurrency cap | `TaskList` count + `state.json` cross-check | | Retries with backoff | `retry_attempts` entries, due_at compared each tick | +| Liveness across sessions | Worker heartbeats in `.symphony/heartbeats/` | | Logging | `.symphony/logs/-.log` | ## Known divergences from the spec diff --git a/symphony/agents/symphony-worker.md b/symphony/agents/symphony-worker.md index bf4391d..a93f975 100644 --- a/symphony/agents/symphony-worker.md +++ b/symphony/agents/symphony-worker.md @@ -34,16 +34,31 @@ in your summary; the orchestrator will not retry until the project documents it. Bash with `timeout_ms`. Non-zero exit → abort the attempt with a `before_run_failed` error. Do NOT proceed to the agent work. -2. **Do the work.** Follow the rendered prompt. The workflow author is responsible for +2. **Start heartbeat.** Write `/.symphony/heartbeats/.json` with: + ``` + { "task_id": "...", "session_id": "...", "started_at": "...", "status": "running" } + ``` + Update the file's `timestamp` periodically — at least once per minute (the orchestrator's + `agent.heartbeat_interval_ms`, default 60000). The simplest implementation is a + `touch`-like rewrite at natural pause points (after a hook completes, after a test pass, + after a tracker write). The heartbeat is what other Symphony sessions use to know + you're alive when they can't see your task in their own `TaskList`. + +3. **Do the work.** Follow the rendered prompt. The workflow author is responsible for telling you to do things like create a branch, run tests, push, open a PR, comment on the ticket, transition state. Don't infer those steps; if the prompt doesn't say to, don't. -3. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the +4. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the issue payload, treat that as a soft budget — at the limit, stop and hand off with a summary even if work is incomplete. The orchestrator will retry or release. -4. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal. +5. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal. + +6. **Stop heartbeat.** On graceful exit (regardless of status), delete the heartbeat + file. Stale heartbeats are how the orchestrator detects abandonment, so leave a clean + trail behind you. If you crash hard, the file stays and the next reconciliation will + correctly reclaim the issue. ## Output contract diff --git a/symphony/skills/symphony/SKILL.md b/symphony/skills/symphony/SKILL.md index 3f56c4d..91db0fd 100644 --- a/symphony/skills/symphony/SKILL.md +++ b/symphony/skills/symphony/SKILL.md @@ -5,7 +5,15 @@ description: Orchestrator skill for the Symphony plugin. Runs one tick — loads # Symphony orchestrator (one tick) -You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and exits. +You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and returns. +The orchestrator runs in a long-lived session — `/schedule` fires ticks within the +running session rather than cold-starting. Tasks you spawn in tick N stay visible to +`TaskList` in tick N+1 *within the same session*. + +Symphony also stays open to **other trigger sources** (Discord channel, webhook, +another user session). The reconciliation logic uses worker heartbeats to handle +those cases — see step 3. + Spec mapping is in `../../CLAUDE.md`. ## Inputs @@ -41,7 +49,12 @@ their own. ``` .symphony/state.json { - "running": { "": { "task_id": "...", "worktree": "...", "started_at": "..." } }, + "running": { "": { + "task_id": "...", + "session_id": "...", + "worktree": "...", + "started_at": "..." + } }, "retry_attempts": { "": { "attempt": 2, "due_at_ms": 1714000000000, "last_error": "..." } }, "claimed": ["", ...], "completed": ["", ...] @@ -49,16 +62,27 @@ their own. ``` Create empty defaults if missing. Always rewrite atomically (write `state.json.tmp`, rename). +Session ID for entries we own is whatever this session reports; for entries owned by +other sessions, the field stays as the spawner wrote it. ### 3. Reconcile For each entry in `running`: -- Call `TaskList` (or `TaskGet` by id). If the task is no longer running and reported success, - move issue to `completed`, drop from `running`. -- If the task failed, move into `retry_attempts` with exponential backoff (capped by - `agent.max_retry_backoff_ms`). -- Re-fetch the issue's current state from the tracker. If state is in `terminal_states` or - no longer in `active_states`, call `TaskStop` on the worker and drop from `running`. + +**Fast path — task is in our `TaskList`** (we spawned it; same session): +- `TaskGet` for status. If completed successfully → move issue to `completed`, drop from `running`. +- If failed → move to `retry_attempts` with exponential backoff (capped by `agent.max_retry_backoff_ms`). +- Re-fetch issue state from the tracker. If in `terminal_states` or no longer in + `active_states` → `TaskStop` and drop from `running`. + +**Slow path — task is NOT in our `TaskList`** (spawned by another session, or our +session crashed and restarted): +- Read `.symphony/heartbeats/.json`. +- **Heartbeat fresh** (timestamp within `agent.heartbeat_stale_ms`, default 300000 = 5min) + → another session is running it; do NOT dispatch a duplicate; leave alone. +- **Heartbeat stale OR missing** → consider abandoned. Move to `retry_attempts` (backoff + resets to attempt 1 if no prior retries; otherwise increments). Drop from `running`. + Delete the stale heartbeat file. Do NOT call `TaskStop` — we don't own the task. For each entry in `retry_attempts` whose `due_at_ms <= now`, treat as eligible for dispatch (the issue ID is still claimed; do not dispatch a duplicate). @@ -101,7 +125,10 @@ For each candidate up to `slots`: run_in_background: true }) ``` -3. Record `running[issue.id] = { task_id, worktree, started_at: now }`. +3. Record `running[issue.id] = { task_id, session_id, worktree, started_at: now }`. + `session_id` should be a stable identifier for this orchestrator session — used by + other sessions to know who owns this task. If no session ID is available, omit the + field; reconcilers will fall back to heartbeat-only checks. 4. Drop from `retry_attempts` if present. ### 6. Persist & exit diff --git a/symphony/templates/WORKFLOW.md b/symphony/templates/WORKFLOW.md index 93ce491..89cb8c9 100644 --- a/symphony/templates/WORKFLOW.md +++ b/symphony/templates/WORKFLOW.md @@ -37,6 +37,11 @@ agent: max_retry_backoff_ms: 600000 max_concurrent_agents_by_state: "in progress": 2 + # Worker heartbeat cadence — affects how quickly other Symphony sessions + # (Discord triggers, webhooks, separate user sessions) detect abandoned + # workers. Defaults are safe; tighten if you have many concurrent sessions. + heartbeat_interval_ms: 60000 # workers refresh the heartbeat at most this often + heartbeat_stale_ms: 300000 # reconciler treats heartbeats older than this as abandoned --- You are working on issue **{{ issue.identifier }}** — {{ issue.title }}. diff --git a/symphony/templates/state.schema.json b/symphony/templates/state.schema.json index 61c11ae..65f0c31 100644 --- a/symphony/templates/state.schema.json +++ b/symphony/templates/state.schema.json @@ -10,9 +10,10 @@ "type": "object", "required": ["task_id", "worktree", "started_at"], "properties": { - "task_id": { "type": "string" }, - "worktree": { "type": "string" }, - "started_at": { "type": "string", "format": "date-time" }, + "task_id": { "type": "string" }, + "session_id": { "type": ["string", "null"], "description": "Identifier for the orchestrator session that spawned this task. Other sessions reconcile via the heartbeat file when this session's TaskList does not include the task_id." }, + "worktree": { "type": "string" }, + "started_at": { "type": "string", "format": "date-time" }, "issue_identifier": { "type": "string" } } }