fix(symphony): correct runtime model — long-lived session, with multi-session support
Earlier framing assumed each /schedule-fired tick was a cold start needing on-disk reconciliation. The actual model is a long-lived orchestrator session where /schedule fires ticks within the running session — TaskList sees prior-tick tasks just fine. Reframed accordingly, while leaving the door open to other trigger sources (Discord channel, webhook, second user session) which DO need cross-session reconciliation. Reconciliation logic now has a fast path and a slow path: - Fast path: task in our TaskList → it's ours → reconcile via TaskGet. - Slow path: task NOT in our TaskList (another session, or our session restarted) → check the worker's heartbeat file. Fresh → another session owns it, leave alone. Stale or missing → consider abandoned, retry; do not call TaskStop on a task we don't own. Workers now write .symphony/heartbeats/<issue-id>.json on start, refresh it at heartbeat_interval_ms cadence (default 60s), and delete it on graceful exit. Stale heartbeats are how abandonment is detected across sessions and after crashes. Also added session_id to state.json running entries so cross-session reconcilers know who spawned what. WORKFLOW.md gains heartbeat_interval_ms and heartbeat_stale_ms knobs under agent: with documented defaults. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
movq
@@ -29,9 +29,12 @@ repo to provide the integration. See "Tracker contract" below.
|
||||
## User flow
|
||||
|
||||
1. **First time**: `/symphony-init` in the project root. Interview + write.
|
||||
2. **Steady state**: `/symphony-tick` is fired by a `/schedule` cron entry (also set up
|
||||
by init unless the user declines).
|
||||
3. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates.
|
||||
2. **Steady state**: `/symphony-tick` is fired by a `/schedule` entry inside the long-lived
|
||||
orchestrator session (also set up by init unless the user declines).
|
||||
3. **Other triggers**: ticks can also be fired by remote channels (Discord DM, webhook,
|
||||
another session). The reconciliation logic handles cross-session liveness via worker
|
||||
heartbeats.
|
||||
4. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates.
|
||||
|
||||
## Tracker contract
|
||||
|
||||
@@ -66,31 +69,65 @@ The skill and worker read these instructions when they need to talk to the track
|
||||
the project CLAUDE.md is silent on the tracker, the orchestrator errors with
|
||||
`tracker_integration_missing` and exits the tick.
|
||||
|
||||
## State
|
||||
## Runtime model
|
||||
|
||||
Symphony's spec calls for in-memory orchestrator state. Cron-fired ticks are cold starts,
|
||||
so we persist to `.symphony/state.json` in the consuming repo:
|
||||
The orchestrator runs inside a **long-lived Claude Code session**. `/schedule` doesn't
|
||||
cold-start each tick — it sets the prompt for the running session. Tasks spawned in
|
||||
tick N stay visible to `TaskList` / `TaskGet` in tick N+1 within that session, so the
|
||||
session itself is the primary store of "what's running right now."
|
||||
|
||||
- `running` — issue_id → { worker_task_id, started_at, worktree_path }
|
||||
Symphony stays open to **other trigger sources** too: a Discord channel could DM the
|
||||
session and ask for a tick, a webhook hook could trigger one, and a separate user
|
||||
session could conceivably spawn tasks against the same project. To keep all three
|
||||
modes honest, two layers carry state:
|
||||
|
||||
1. **`TaskList` / `TaskGet`** — fast path, authoritative for liveness *within this session*.
|
||||
2. **`.symphony/state.json` + worker heartbeats** — durable record across sessions and
|
||||
restarts. Authoritative when `TaskList` doesn't know about a task ID.
|
||||
|
||||
### Reconciliation logic
|
||||
|
||||
For each entry in `state.json`'s `running` map:
|
||||
|
||||
- **In our `TaskList`** → it's our task; check status via `TaskGet`; reconcile normally.
|
||||
- **Not in our `TaskList`** → another session may own it, or the session that spawned it
|
||||
exited. Read the worker's heartbeat file (`.symphony/heartbeats/<issue-id>.json`).
|
||||
- **Heartbeat fresh** (within `agent.heartbeat_stale_ms`, default 5min) → leave alone;
|
||||
another session is owning it.
|
||||
- **Heartbeat stale** → consider abandoned; mark for retry, drop from `running`.
|
||||
- **No heartbeat** → ditto.
|
||||
|
||||
This keeps the single-session case simple (TaskList wins) while supporting
|
||||
multi-session and crash recovery cleanly.
|
||||
|
||||
### State file shape
|
||||
|
||||
- `running` — issue_id → { task_id, session_id, started_at, worktree_path }
|
||||
- `retry_attempts` — issue_id → { attempt, due_at_ms, last_error }
|
||||
- `claimed` — set of issue IDs reserved this tick (cleared on dispatch or release)
|
||||
- `completed` — set of issue IDs (bookkeeping)
|
||||
|
||||
Reconciliation reads the file at the top of every tick, queries `TaskList` to confirm
|
||||
which workers are actually still running, and rewrites the file before returning.
|
||||
### Worker heartbeats
|
||||
|
||||
Workers write `.symphony/heartbeats/<issue-id>.json` with `{ timestamp, status, task_id, session_id }`
|
||||
every minute (configurable via `agent.heartbeat_interval_ms`). On graceful exit they
|
||||
remove the file. Stale heartbeats — and stale `running` entries with no heartbeat at
|
||||
all — are how Symphony detects abandonment.
|
||||
|
||||
## Mapping to the spec
|
||||
|
||||
| Symphony component | Claude Code mechanism |
|
||||
|---------------------------|--------------------------------------------------------|
|
||||
| Polling daemon | `/schedule` cron entry firing `/symphony-tick` |
|
||||
| Polling daemon | `/schedule` firing `/symphony-tick` in a long-lived session |
|
||||
| Workflow loader | Skill body — reads & parses `WORKFLOW.md` |
|
||||
| Issue tracker client | Project CLAUDE.md instructions + whatever tools fit |
|
||||
| Orchestrator | Skill, invoked once per tick; state on disk |
|
||||
| Orchestrator | Skill, runs each tick within the same session |
|
||||
| Workspace manager | `EnterWorktree` per worker |
|
||||
| Agent runner | `Agent` tool with `subagent_type: symphony-worker` |
|
||||
| Hooks | Bash steps inside the worker prompt |
|
||||
| Concurrency cap | Lockfile + `TaskList` poll inside the dispatcher |
|
||||
| Concurrency cap | `TaskList` count + `state.json` cross-check |
|
||||
| Retries with backoff | `retry_attempts` entries, due_at compared each tick |
|
||||
| Liveness across sessions | Worker heartbeats in `.symphony/heartbeats/` |
|
||||
| Logging | `.symphony/logs/<issue>-<ts>.log` |
|
||||
|
||||
## Known divergences from the spec
|
||||
|
||||
@@ -34,16 +34,31 @@ in your summary; the orchestrator will not retry until the project documents it.
|
||||
Bash with `timeout_ms`. Non-zero exit → abort the attempt with a `before_run_failed`
|
||||
error. Do NOT proceed to the agent work.
|
||||
|
||||
2. **Do the work.** Follow the rendered prompt. The workflow author is responsible for
|
||||
2. **Start heartbeat.** Write `<repo-root>/.symphony/heartbeats/<issue-id>.json` with:
|
||||
```
|
||||
{ "task_id": "...", "session_id": "...", "started_at": "...", "status": "running" }
|
||||
```
|
||||
Update the file's `timestamp` periodically — at least once per minute (the orchestrator's
|
||||
`agent.heartbeat_interval_ms`, default 60000). The simplest implementation is a
|
||||
`touch`-like rewrite at natural pause points (after a hook completes, after a test pass,
|
||||
after a tracker write). The heartbeat is what other Symphony sessions use to know
|
||||
you're alive when they can't see your task in their own `TaskList`.
|
||||
|
||||
3. **Do the work.** Follow the rendered prompt. The workflow author is responsible for
|
||||
telling you to do things like create a branch, run tests, push, open a PR, comment on
|
||||
the ticket, transition state. Don't infer those steps; if the prompt doesn't say to,
|
||||
don't.
|
||||
|
||||
3. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the
|
||||
4. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the
|
||||
issue payload, treat that as a soft budget — at the limit, stop and hand off with a
|
||||
summary even if work is incomplete. The orchestrator will retry or release.
|
||||
|
||||
4. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal.
|
||||
5. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal.
|
||||
|
||||
6. **Stop heartbeat.** On graceful exit (regardless of status), delete the heartbeat
|
||||
file. Stale heartbeats are how the orchestrator detects abandonment, so leave a clean
|
||||
trail behind you. If you crash hard, the file stays and the next reconciliation will
|
||||
correctly reclaim the issue.
|
||||
|
||||
## Output contract
|
||||
|
||||
|
||||
@@ -5,7 +5,15 @@ description: Orchestrator skill for the Symphony plugin. Runs one tick — loads
|
||||
|
||||
# Symphony orchestrator (one tick)
|
||||
|
||||
You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and exits.
|
||||
You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and returns.
|
||||
The orchestrator runs in a long-lived session — `/schedule` fires ticks within the
|
||||
running session rather than cold-starting. Tasks you spawn in tick N stay visible to
|
||||
`TaskList` in tick N+1 *within the same session*.
|
||||
|
||||
Symphony also stays open to **other trigger sources** (Discord channel, webhook,
|
||||
another user session). The reconciliation logic uses worker heartbeats to handle
|
||||
those cases — see step 3.
|
||||
|
||||
Spec mapping is in `../../CLAUDE.md`.
|
||||
|
||||
## Inputs
|
||||
@@ -41,7 +49,12 @@ their own.
|
||||
```
|
||||
.symphony/state.json
|
||||
{
|
||||
"running": { "<issue_id>": { "task_id": "...", "worktree": "...", "started_at": "..." } },
|
||||
"running": { "<issue_id>": {
|
||||
"task_id": "...",
|
||||
"session_id": "...",
|
||||
"worktree": "...",
|
||||
"started_at": "..."
|
||||
} },
|
||||
"retry_attempts": { "<issue_id>": { "attempt": 2, "due_at_ms": 1714000000000, "last_error": "..." } },
|
||||
"claimed": ["<issue_id>", ...],
|
||||
"completed": ["<issue_id>", ...]
|
||||
@@ -49,16 +62,27 @@ their own.
|
||||
```
|
||||
|
||||
Create empty defaults if missing. Always rewrite atomically (write `state.json.tmp`, rename).
|
||||
Session ID for entries we own is whatever this session reports; for entries owned by
|
||||
other sessions, the field stays as the spawner wrote it.
|
||||
|
||||
### 3. Reconcile
|
||||
|
||||
For each entry in `running`:
|
||||
- Call `TaskList` (or `TaskGet` by id). If the task is no longer running and reported success,
|
||||
move issue to `completed`, drop from `running`.
|
||||
- If the task failed, move into `retry_attempts` with exponential backoff (capped by
|
||||
`agent.max_retry_backoff_ms`).
|
||||
- Re-fetch the issue's current state from the tracker. If state is in `terminal_states` or
|
||||
no longer in `active_states`, call `TaskStop` on the worker and drop from `running`.
|
||||
|
||||
**Fast path — task is in our `TaskList`** (we spawned it; same session):
|
||||
- `TaskGet` for status. If completed successfully → move issue to `completed`, drop from `running`.
|
||||
- If failed → move to `retry_attempts` with exponential backoff (capped by `agent.max_retry_backoff_ms`).
|
||||
- Re-fetch issue state from the tracker. If in `terminal_states` or no longer in
|
||||
`active_states` → `TaskStop` and drop from `running`.
|
||||
|
||||
**Slow path — task is NOT in our `TaskList`** (spawned by another session, or our
|
||||
session crashed and restarted):
|
||||
- Read `.symphony/heartbeats/<issue-id>.json`.
|
||||
- **Heartbeat fresh** (timestamp within `agent.heartbeat_stale_ms`, default 300000 = 5min)
|
||||
→ another session is running it; do NOT dispatch a duplicate; leave alone.
|
||||
- **Heartbeat stale OR missing** → consider abandoned. Move to `retry_attempts` (backoff
|
||||
resets to attempt 1 if no prior retries; otherwise increments). Drop from `running`.
|
||||
Delete the stale heartbeat file. Do NOT call `TaskStop` — we don't own the task.
|
||||
|
||||
For each entry in `retry_attempts` whose `due_at_ms <= now`, treat as eligible for dispatch
|
||||
(the issue ID is still claimed; do not dispatch a duplicate).
|
||||
@@ -101,7 +125,10 @@ For each candidate up to `slots`:
|
||||
run_in_background: true
|
||||
})
|
||||
```
|
||||
3. Record `running[issue.id] = { task_id, worktree, started_at: now }`.
|
||||
3. Record `running[issue.id] = { task_id, session_id, worktree, started_at: now }`.
|
||||
`session_id` should be a stable identifier for this orchestrator session — used by
|
||||
other sessions to know who owns this task. If no session ID is available, omit the
|
||||
field; reconcilers will fall back to heartbeat-only checks.
|
||||
4. Drop from `retry_attempts` if present.
|
||||
|
||||
### 6. Persist & exit
|
||||
|
||||
@@ -37,6 +37,11 @@ agent:
|
||||
max_retry_backoff_ms: 600000
|
||||
max_concurrent_agents_by_state:
|
||||
"in progress": 2
|
||||
# Worker heartbeat cadence — affects how quickly other Symphony sessions
|
||||
# (Discord triggers, webhooks, separate user sessions) detect abandoned
|
||||
# workers. Defaults are safe; tighten if you have many concurrent sessions.
|
||||
heartbeat_interval_ms: 60000 # workers refresh the heartbeat at most this often
|
||||
heartbeat_stale_ms: 300000 # reconciler treats heartbeats older than this as abandoned
|
||||
---
|
||||
|
||||
You are working on issue **{{ issue.identifier }}** — {{ issue.title }}.
|
||||
|
||||
@@ -10,9 +10,10 @@
|
||||
"type": "object",
|
||||
"required": ["task_id", "worktree", "started_at"],
|
||||
"properties": {
|
||||
"task_id": { "type": "string" },
|
||||
"worktree": { "type": "string" },
|
||||
"started_at": { "type": "string", "format": "date-time" },
|
||||
"task_id": { "type": "string" },
|
||||
"session_id": { "type": ["string", "null"], "description": "Identifier for the orchestrator session that spawned this task. Other sessions reconcile via the heartbeat file when this session's TaskList does not include the task_id." },
|
||||
"worktree": { "type": "string" },
|
||||
"started_at": { "type": "string", "format": "date-time" },
|
||||
"issue_identifier": { "type": "string" }
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user