fix(symphony): correct runtime model — long-lived session, with multi-session support

Earlier framing assumed each /schedule-fired tick was a cold start
needing on-disk reconciliation. The actual model is a long-lived
orchestrator session where /schedule fires ticks within the running
session — TaskList sees prior-tick tasks just fine. Reframed
accordingly, while leaving the door open to other trigger sources
(Discord channel, webhook, second user session) which DO need
cross-session reconciliation.

Reconciliation logic now has a fast path and a slow path:

- Fast path: task in our TaskList → it's ours → reconcile via TaskGet.
- Slow path: task NOT in our TaskList (another session, or our session
  restarted) → check the worker's heartbeat file. Fresh → another
  session owns it, leave alone. Stale or missing → consider abandoned,
  retry; do not call TaskStop on a task we don't own.

Workers now write .symphony/heartbeats/<issue-id>.json on start,
refresh it at heartbeat_interval_ms cadence (default 60s), and delete
it on graceful exit. Stale heartbeats are how abandonment is detected
across sessions and after crashes.

Also added session_id to state.json running entries so cross-session
reconcilers know who spawned what.

WORKFLOW.md gains heartbeat_interval_ms and heartbeat_stale_ms knobs
under agent: with documented defaults.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
movq
2026-04-29 21:51:23 -05:00
parent 92f005895c
commit ee4e0b236e
5 changed files with 112 additions and 27 deletions

View File

@@ -29,9 +29,12 @@ repo to provide the integration. See "Tracker contract" below.
## User flow ## User flow
1. **First time**: `/symphony-init` in the project root. Interview + write. 1. **First time**: `/symphony-init` in the project root. Interview + write.
2. **Steady state**: `/symphony-tick` is fired by a `/schedule` cron entry (also set up 2. **Steady state**: `/symphony-tick` is fired by a `/schedule` entry inside the long-lived
by init unless the user declines). orchestrator session (also set up by init unless the user declines).
3. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates. 3. **Other triggers**: ticks can also be fired by remote channels (Discord DM, webhook,
another session). The reconciliation logic handles cross-session liveness via worker
heartbeats.
4. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates.
## Tracker contract ## Tracker contract
@@ -66,31 +69,65 @@ The skill and worker read these instructions when they need to talk to the track
the project CLAUDE.md is silent on the tracker, the orchestrator errors with the project CLAUDE.md is silent on the tracker, the orchestrator errors with
`tracker_integration_missing` and exits the tick. `tracker_integration_missing` and exits the tick.
## State ## Runtime model
Symphony's spec calls for in-memory orchestrator state. Cron-fired ticks are cold starts, The orchestrator runs inside a **long-lived Claude Code session**. `/schedule` doesn't
so we persist to `.symphony/state.json` in the consuming repo: cold-start each tick — it sets the prompt for the running session. Tasks spawned in
tick N stay visible to `TaskList` / `TaskGet` in tick N+1 within that session, so the
session itself is the primary store of "what's running right now."
- `running` — issue_id → { worker_task_id, started_at, worktree_path } Symphony stays open to **other trigger sources** too: a Discord channel could DM the
session and ask for a tick, a webhook hook could trigger one, and a separate user
session could conceivably spawn tasks against the same project. To keep all three
modes honest, two layers carry state:
1. **`TaskList` / `TaskGet`** — fast path, authoritative for liveness *within this session*.
2. **`.symphony/state.json` + worker heartbeats** — durable record across sessions and
restarts. Authoritative when `TaskList` doesn't know about a task ID.
### Reconciliation logic
For each entry in `state.json`'s `running` map:
- **In our `TaskList`** → it's our task; check status via `TaskGet`; reconcile normally.
- **Not in our `TaskList`** → another session may own it, or the session that spawned it
exited. Read the worker's heartbeat file (`.symphony/heartbeats/<issue-id>.json`).
- **Heartbeat fresh** (within `agent.heartbeat_stale_ms`, default 5min) → leave alone;
another session is owning it.
- **Heartbeat stale** → consider abandoned; mark for retry, drop from `running`.
- **No heartbeat** → ditto.
This keeps the single-session case simple (TaskList wins) while supporting
multi-session and crash recovery cleanly.
### State file shape
- `running` — issue_id → { task_id, session_id, started_at, worktree_path }
- `retry_attempts` — issue_id → { attempt, due_at_ms, last_error } - `retry_attempts` — issue_id → { attempt, due_at_ms, last_error }
- `claimed` — set of issue IDs reserved this tick (cleared on dispatch or release) - `claimed` — set of issue IDs reserved this tick (cleared on dispatch or release)
- `completed` — set of issue IDs (bookkeeping)
Reconciliation reads the file at the top of every tick, queries `TaskList` to confirm ### Worker heartbeats
which workers are actually still running, and rewrites the file before returning.
Workers write `.symphony/heartbeats/<issue-id>.json` with `{ timestamp, status, task_id, session_id }`
every minute (configurable via `agent.heartbeat_interval_ms`). On graceful exit they
remove the file. Stale heartbeats — and stale `running` entries with no heartbeat at
all — are how Symphony detects abandonment.
## Mapping to the spec ## Mapping to the spec
| Symphony component | Claude Code mechanism | | Symphony component | Claude Code mechanism |
|---------------------------|--------------------------------------------------------| |---------------------------|--------------------------------------------------------|
| Polling daemon | `/schedule` cron entry firing `/symphony-tick` | | Polling daemon | `/schedule` firing `/symphony-tick` in a long-lived session |
| Workflow loader | Skill body — reads & parses `WORKFLOW.md` | | Workflow loader | Skill body — reads & parses `WORKFLOW.md` |
| Issue tracker client | Project CLAUDE.md instructions + whatever tools fit | | Issue tracker client | Project CLAUDE.md instructions + whatever tools fit |
| Orchestrator | Skill, invoked once per tick; state on disk | | Orchestrator | Skill, runs each tick within the same session |
| Workspace manager | `EnterWorktree` per worker | | Workspace manager | `EnterWorktree` per worker |
| Agent runner | `Agent` tool with `subagent_type: symphony-worker` | | Agent runner | `Agent` tool with `subagent_type: symphony-worker` |
| Hooks | Bash steps inside the worker prompt | | Hooks | Bash steps inside the worker prompt |
| Concurrency cap | Lockfile + `TaskList` poll inside the dispatcher | | Concurrency cap | `TaskList` count + `state.json` cross-check |
| Retries with backoff | `retry_attempts` entries, due_at compared each tick | | Retries with backoff | `retry_attempts` entries, due_at compared each tick |
| Liveness across sessions | Worker heartbeats in `.symphony/heartbeats/` |
| Logging | `.symphony/logs/<issue>-<ts>.log` | | Logging | `.symphony/logs/<issue>-<ts>.log` |
## Known divergences from the spec ## Known divergences from the spec

View File

@@ -34,16 +34,31 @@ in your summary; the orchestrator will not retry until the project documents it.
Bash with `timeout_ms`. Non-zero exit → abort the attempt with a `before_run_failed` Bash with `timeout_ms`. Non-zero exit → abort the attempt with a `before_run_failed`
error. Do NOT proceed to the agent work. error. Do NOT proceed to the agent work.
2. **Do the work.** Follow the rendered prompt. The workflow author is responsible for 2. **Start heartbeat.** Write `<repo-root>/.symphony/heartbeats/<issue-id>.json` with:
```
{ "task_id": "...", "session_id": "...", "started_at": "...", "status": "running" }
```
Update the file's `timestamp` periodically — at least once per minute (the orchestrator's
`agent.heartbeat_interval_ms`, default 60000). The simplest implementation is a
`touch`-like rewrite at natural pause points (after a hook completes, after a test pass,
after a tracker write). The heartbeat is what other Symphony sessions use to know
you're alive when they can't see your task in their own `TaskList`.
3. **Do the work.** Follow the rendered prompt. The workflow author is responsible for
telling you to do things like create a branch, run tests, push, open a PR, comment on telling you to do things like create a branch, run tests, push, open a PR, comment on
the ticket, transition state. Don't infer those steps; if the prompt doesn't say to, the ticket, transition state. Don't infer those steps; if the prompt doesn't say to,
don't. don't.
3. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the 4. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the
issue payload, treat that as a soft budget — at the limit, stop and hand off with a issue payload, treat that as a soft budget — at the limit, stop and hand off with a
summary even if work is incomplete. The orchestrator will retry or release. summary even if work is incomplete. The orchestrator will retry or release.
4. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal. 5. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal.
6. **Stop heartbeat.** On graceful exit (regardless of status), delete the heartbeat
file. Stale heartbeats are how the orchestrator detects abandonment, so leave a clean
trail behind you. If you crash hard, the file stays and the next reconciliation will
correctly reclaim the issue.
## Output contract ## Output contract

View File

@@ -5,7 +5,15 @@ description: Orchestrator skill for the Symphony plugin. Runs one tick — loads
# Symphony orchestrator (one tick) # Symphony orchestrator (one tick)
You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and exits. You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and returns.
The orchestrator runs in a long-lived session — `/schedule` fires ticks within the
running session rather than cold-starting. Tasks you spawn in tick N stay visible to
`TaskList` in tick N+1 *within the same session*.
Symphony also stays open to **other trigger sources** (Discord channel, webhook,
another user session). The reconciliation logic uses worker heartbeats to handle
those cases — see step 3.
Spec mapping is in `../../CLAUDE.md`. Spec mapping is in `../../CLAUDE.md`.
## Inputs ## Inputs
@@ -41,7 +49,12 @@ their own.
``` ```
.symphony/state.json .symphony/state.json
{ {
"running": { "<issue_id>": { "task_id": "...", "worktree": "...", "started_at": "..." } }, "running": { "<issue_id>": {
"task_id": "...",
"session_id": "...",
"worktree": "...",
"started_at": "..."
} },
"retry_attempts": { "<issue_id>": { "attempt": 2, "due_at_ms": 1714000000000, "last_error": "..." } }, "retry_attempts": { "<issue_id>": { "attempt": 2, "due_at_ms": 1714000000000, "last_error": "..." } },
"claimed": ["<issue_id>", ...], "claimed": ["<issue_id>", ...],
"completed": ["<issue_id>", ...] "completed": ["<issue_id>", ...]
@@ -49,16 +62,27 @@ their own.
``` ```
Create empty defaults if missing. Always rewrite atomically (write `state.json.tmp`, rename). Create empty defaults if missing. Always rewrite atomically (write `state.json.tmp`, rename).
Session ID for entries we own is whatever this session reports; for entries owned by
other sessions, the field stays as the spawner wrote it.
### 3. Reconcile ### 3. Reconcile
For each entry in `running`: For each entry in `running`:
- Call `TaskList` (or `TaskGet` by id). If the task is no longer running and reported success,
move issue to `completed`, drop from `running`. **Fast path — task is in our `TaskList`** (we spawned it; same session):
- If the task failed, move into `retry_attempts` with exponential backoff (capped by - `TaskGet` for status. If completed successfully → move issue to `completed`, drop from `running`.
`agent.max_retry_backoff_ms`). - If failed → move to `retry_attempts` with exponential backoff (capped by `agent.max_retry_backoff_ms`).
- Re-fetch the issue's current state from the tracker. If state is in `terminal_states` or - Re-fetch issue state from the tracker. If in `terminal_states` or no longer in
no longer in `active_states`, call `TaskStop` on the worker and drop from `running`. `active_states` `TaskStop` and drop from `running`.
**Slow path — task is NOT in our `TaskList`** (spawned by another session, or our
session crashed and restarted):
- Read `.symphony/heartbeats/<issue-id>.json`.
- **Heartbeat fresh** (timestamp within `agent.heartbeat_stale_ms`, default 300000 = 5min)
→ another session is running it; do NOT dispatch a duplicate; leave alone.
- **Heartbeat stale OR missing** → consider abandoned. Move to `retry_attempts` (backoff
resets to attempt 1 if no prior retries; otherwise increments). Drop from `running`.
Delete the stale heartbeat file. Do NOT call `TaskStop` — we don't own the task.
For each entry in `retry_attempts` whose `due_at_ms <= now`, treat as eligible for dispatch For each entry in `retry_attempts` whose `due_at_ms <= now`, treat as eligible for dispatch
(the issue ID is still claimed; do not dispatch a duplicate). (the issue ID is still claimed; do not dispatch a duplicate).
@@ -101,7 +125,10 @@ For each candidate up to `slots`:
run_in_background: true run_in_background: true
}) })
``` ```
3. Record `running[issue.id] = { task_id, worktree, started_at: now }`. 3. Record `running[issue.id] = { task_id, session_id, worktree, started_at: now }`.
`session_id` should be a stable identifier for this orchestrator session — used by
other sessions to know who owns this task. If no session ID is available, omit the
field; reconcilers will fall back to heartbeat-only checks.
4. Drop from `retry_attempts` if present. 4. Drop from `retry_attempts` if present.
### 6. Persist & exit ### 6. Persist & exit

View File

@@ -37,6 +37,11 @@ agent:
max_retry_backoff_ms: 600000 max_retry_backoff_ms: 600000
max_concurrent_agents_by_state: max_concurrent_agents_by_state:
"in progress": 2 "in progress": 2
# Worker heartbeat cadence — affects how quickly other Symphony sessions
# (Discord triggers, webhooks, separate user sessions) detect abandoned
# workers. Defaults are safe; tighten if you have many concurrent sessions.
heartbeat_interval_ms: 60000 # workers refresh the heartbeat at most this often
heartbeat_stale_ms: 300000 # reconciler treats heartbeats older than this as abandoned
--- ---
You are working on issue **{{ issue.identifier }}** — {{ issue.title }}. You are working on issue **{{ issue.identifier }}** — {{ issue.title }}.

View File

@@ -10,9 +10,10 @@
"type": "object", "type": "object",
"required": ["task_id", "worktree", "started_at"], "required": ["task_id", "worktree", "started_at"],
"properties": { "properties": {
"task_id": { "type": "string" }, "task_id": { "type": "string" },
"worktree": { "type": "string" }, "session_id": { "type": ["string", "null"], "description": "Identifier for the orchestrator session that spawned this task. Other sessions reconcile via the heartbeat file when this session's TaskList does not include the task_id." },
"started_at": { "type": "string", "format": "date-time" }, "worktree": { "type": "string" },
"started_at": { "type": "string", "format": "date-time" },
"issue_identifier": { "type": "string" } "issue_identifier": { "type": "string" }
} }
} }