fix(symphony): correct runtime model — long-lived session, with multi-session support
Earlier framing assumed each /schedule-fired tick was a cold start needing on-disk reconciliation. The actual model is a long-lived orchestrator session where /schedule fires ticks within the running session — TaskList sees prior-tick tasks just fine. Reframed accordingly, while leaving the door open to other trigger sources (Discord channel, webhook, second user session) which DO need cross-session reconciliation. Reconciliation logic now has a fast path and a slow path: - Fast path: task in our TaskList → it's ours → reconcile via TaskGet. - Slow path: task NOT in our TaskList (another session, or our session restarted) → check the worker's heartbeat file. Fresh → another session owns it, leave alone. Stale or missing → consider abandoned, retry; do not call TaskStop on a task we don't own. Workers now write .symphony/heartbeats/<issue-id>.json on start, refresh it at heartbeat_interval_ms cadence (default 60s), and delete it on graceful exit. Stale heartbeats are how abandonment is detected across sessions and after crashes. Also added session_id to state.json running entries so cross-session reconcilers know who spawned what. WORKFLOW.md gains heartbeat_interval_ms and heartbeat_stale_ms knobs under agent: with documented defaults. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -29,9 +29,12 @@ repo to provide the integration. See "Tracker contract" below.
|
|||||||
## User flow
|
## User flow
|
||||||
|
|
||||||
1. **First time**: `/symphony-init` in the project root. Interview + write.
|
1. **First time**: `/symphony-init` in the project root. Interview + write.
|
||||||
2. **Steady state**: `/symphony-tick` is fired by a `/schedule` cron entry (also set up
|
2. **Steady state**: `/symphony-tick` is fired by a `/schedule` entry inside the long-lived
|
||||||
by init unless the user declines).
|
orchestrator session (also set up by init unless the user declines).
|
||||||
3. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates.
|
3. **Other triggers**: ticks can also be fired by remote channels (Discord DM, webhook,
|
||||||
|
another session). The reconciliation logic handles cross-session liveness via worker
|
||||||
|
heartbeats.
|
||||||
|
4. **Re-config**: re-run `/symphony-init`; it detects drift and proposes updates.
|
||||||
|
|
||||||
## Tracker contract
|
## Tracker contract
|
||||||
|
|
||||||
@@ -66,31 +69,65 @@ The skill and worker read these instructions when they need to talk to the track
|
|||||||
the project CLAUDE.md is silent on the tracker, the orchestrator errors with
|
the project CLAUDE.md is silent on the tracker, the orchestrator errors with
|
||||||
`tracker_integration_missing` and exits the tick.
|
`tracker_integration_missing` and exits the tick.
|
||||||
|
|
||||||
## State
|
## Runtime model
|
||||||
|
|
||||||
Symphony's spec calls for in-memory orchestrator state. Cron-fired ticks are cold starts,
|
The orchestrator runs inside a **long-lived Claude Code session**. `/schedule` doesn't
|
||||||
so we persist to `.symphony/state.json` in the consuming repo:
|
cold-start each tick — it sets the prompt for the running session. Tasks spawned in
|
||||||
|
tick N stay visible to `TaskList` / `TaskGet` in tick N+1 within that session, so the
|
||||||
|
session itself is the primary store of "what's running right now."
|
||||||
|
|
||||||
- `running` — issue_id → { worker_task_id, started_at, worktree_path }
|
Symphony stays open to **other trigger sources** too: a Discord channel could DM the
|
||||||
|
session and ask for a tick, a webhook hook could trigger one, and a separate user
|
||||||
|
session could conceivably spawn tasks against the same project. To keep all three
|
||||||
|
modes honest, two layers carry state:
|
||||||
|
|
||||||
|
1. **`TaskList` / `TaskGet`** — fast path, authoritative for liveness *within this session*.
|
||||||
|
2. **`.symphony/state.json` + worker heartbeats** — durable record across sessions and
|
||||||
|
restarts. Authoritative when `TaskList` doesn't know about a task ID.
|
||||||
|
|
||||||
|
### Reconciliation logic
|
||||||
|
|
||||||
|
For each entry in `state.json`'s `running` map:
|
||||||
|
|
||||||
|
- **In our `TaskList`** → it's our task; check status via `TaskGet`; reconcile normally.
|
||||||
|
- **Not in our `TaskList`** → another session may own it, or the session that spawned it
|
||||||
|
exited. Read the worker's heartbeat file (`.symphony/heartbeats/<issue-id>.json`).
|
||||||
|
- **Heartbeat fresh** (within `agent.heartbeat_stale_ms`, default 5min) → leave alone;
|
||||||
|
another session is owning it.
|
||||||
|
- **Heartbeat stale** → consider abandoned; mark for retry, drop from `running`.
|
||||||
|
- **No heartbeat** → ditto.
|
||||||
|
|
||||||
|
This keeps the single-session case simple (TaskList wins) while supporting
|
||||||
|
multi-session and crash recovery cleanly.
|
||||||
|
|
||||||
|
### State file shape
|
||||||
|
|
||||||
|
- `running` — issue_id → { task_id, session_id, started_at, worktree_path }
|
||||||
- `retry_attempts` — issue_id → { attempt, due_at_ms, last_error }
|
- `retry_attempts` — issue_id → { attempt, due_at_ms, last_error }
|
||||||
- `claimed` — set of issue IDs reserved this tick (cleared on dispatch or release)
|
- `claimed` — set of issue IDs reserved this tick (cleared on dispatch or release)
|
||||||
|
- `completed` — set of issue IDs (bookkeeping)
|
||||||
|
|
||||||
Reconciliation reads the file at the top of every tick, queries `TaskList` to confirm
|
### Worker heartbeats
|
||||||
which workers are actually still running, and rewrites the file before returning.
|
|
||||||
|
Workers write `.symphony/heartbeats/<issue-id>.json` with `{ timestamp, status, task_id, session_id }`
|
||||||
|
every minute (configurable via `agent.heartbeat_interval_ms`). On graceful exit they
|
||||||
|
remove the file. Stale heartbeats — and stale `running` entries with no heartbeat at
|
||||||
|
all — are how Symphony detects abandonment.
|
||||||
|
|
||||||
## Mapping to the spec
|
## Mapping to the spec
|
||||||
|
|
||||||
| Symphony component | Claude Code mechanism |
|
| Symphony component | Claude Code mechanism |
|
||||||
|---------------------------|--------------------------------------------------------|
|
|---------------------------|--------------------------------------------------------|
|
||||||
| Polling daemon | `/schedule` cron entry firing `/symphony-tick` |
|
| Polling daemon | `/schedule` firing `/symphony-tick` in a long-lived session |
|
||||||
| Workflow loader | Skill body — reads & parses `WORKFLOW.md` |
|
| Workflow loader | Skill body — reads & parses `WORKFLOW.md` |
|
||||||
| Issue tracker client | Project CLAUDE.md instructions + whatever tools fit |
|
| Issue tracker client | Project CLAUDE.md instructions + whatever tools fit |
|
||||||
| Orchestrator | Skill, invoked once per tick; state on disk |
|
| Orchestrator | Skill, runs each tick within the same session |
|
||||||
| Workspace manager | `EnterWorktree` per worker |
|
| Workspace manager | `EnterWorktree` per worker |
|
||||||
| Agent runner | `Agent` tool with `subagent_type: symphony-worker` |
|
| Agent runner | `Agent` tool with `subagent_type: symphony-worker` |
|
||||||
| Hooks | Bash steps inside the worker prompt |
|
| Hooks | Bash steps inside the worker prompt |
|
||||||
| Concurrency cap | Lockfile + `TaskList` poll inside the dispatcher |
|
| Concurrency cap | `TaskList` count + `state.json` cross-check |
|
||||||
| Retries with backoff | `retry_attempts` entries, due_at compared each tick |
|
| Retries with backoff | `retry_attempts` entries, due_at compared each tick |
|
||||||
|
| Liveness across sessions | Worker heartbeats in `.symphony/heartbeats/` |
|
||||||
| Logging | `.symphony/logs/<issue>-<ts>.log` |
|
| Logging | `.symphony/logs/<issue>-<ts>.log` |
|
||||||
|
|
||||||
## Known divergences from the spec
|
## Known divergences from the spec
|
||||||
|
|||||||
@@ -34,16 +34,31 @@ in your summary; the orchestrator will not retry until the project documents it.
|
|||||||
Bash with `timeout_ms`. Non-zero exit → abort the attempt with a `before_run_failed`
|
Bash with `timeout_ms`. Non-zero exit → abort the attempt with a `before_run_failed`
|
||||||
error. Do NOT proceed to the agent work.
|
error. Do NOT proceed to the agent work.
|
||||||
|
|
||||||
2. **Do the work.** Follow the rendered prompt. The workflow author is responsible for
|
2. **Start heartbeat.** Write `<repo-root>/.symphony/heartbeats/<issue-id>.json` with:
|
||||||
|
```
|
||||||
|
{ "task_id": "...", "session_id": "...", "started_at": "...", "status": "running" }
|
||||||
|
```
|
||||||
|
Update the file's `timestamp` periodically — at least once per minute (the orchestrator's
|
||||||
|
`agent.heartbeat_interval_ms`, default 60000). The simplest implementation is a
|
||||||
|
`touch`-like rewrite at natural pause points (after a hook completes, after a test pass,
|
||||||
|
after a tracker write). The heartbeat is what other Symphony sessions use to know
|
||||||
|
you're alive when they can't see your task in their own `TaskList`.
|
||||||
|
|
||||||
|
3. **Do the work.** Follow the rendered prompt. The workflow author is responsible for
|
||||||
telling you to do things like create a branch, run tests, push, open a PR, comment on
|
telling you to do things like create a branch, run tests, push, open a PR, comment on
|
||||||
the ticket, transition state. Don't infer those steps; if the prompt doesn't say to,
|
the ticket, transition state. Don't infer those steps; if the prompt doesn't say to,
|
||||||
don't.
|
don't.
|
||||||
|
|
||||||
3. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the
|
4. **Bounded turns.** Track your own progress. If `agent.max_turns` was provided in the
|
||||||
issue payload, treat that as a soft budget — at the limit, stop and hand off with a
|
issue payload, treat that as a soft budget — at the limit, stop and hand off with a
|
||||||
summary even if work is incomplete. The orchestrator will retry or release.
|
summary even if work is incomplete. The orchestrator will retry or release.
|
||||||
|
|
||||||
4. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal.
|
5. **`after_run` hook.** Run `hooks.after_run` if present. Failure is logged, not fatal.
|
||||||
|
|
||||||
|
6. **Stop heartbeat.** On graceful exit (regardless of status), delete the heartbeat
|
||||||
|
file. Stale heartbeats are how the orchestrator detects abandonment, so leave a clean
|
||||||
|
trail behind you. If you crash hard, the file stays and the next reconciliation will
|
||||||
|
correctly reclaim the issue.
|
||||||
|
|
||||||
## Output contract
|
## Output contract
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,15 @@ description: Orchestrator skill for the Symphony plugin. Runs one tick — loads
|
|||||||
|
|
||||||
# Symphony orchestrator (one tick)
|
# Symphony orchestrator (one tick)
|
||||||
|
|
||||||
You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and exits.
|
You are the dispatcher. Each invocation runs **one** poll-and-dispatch pass and returns.
|
||||||
|
The orchestrator runs in a long-lived session — `/schedule` fires ticks within the
|
||||||
|
running session rather than cold-starting. Tasks you spawn in tick N stay visible to
|
||||||
|
`TaskList` in tick N+1 *within the same session*.
|
||||||
|
|
||||||
|
Symphony also stays open to **other trigger sources** (Discord channel, webhook,
|
||||||
|
another user session). The reconciliation logic uses worker heartbeats to handle
|
||||||
|
those cases — see step 3.
|
||||||
|
|
||||||
Spec mapping is in `../../CLAUDE.md`.
|
Spec mapping is in `../../CLAUDE.md`.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
@@ -41,7 +49,12 @@ their own.
|
|||||||
```
|
```
|
||||||
.symphony/state.json
|
.symphony/state.json
|
||||||
{
|
{
|
||||||
"running": { "<issue_id>": { "task_id": "...", "worktree": "...", "started_at": "..." } },
|
"running": { "<issue_id>": {
|
||||||
|
"task_id": "...",
|
||||||
|
"session_id": "...",
|
||||||
|
"worktree": "...",
|
||||||
|
"started_at": "..."
|
||||||
|
} },
|
||||||
"retry_attempts": { "<issue_id>": { "attempt": 2, "due_at_ms": 1714000000000, "last_error": "..." } },
|
"retry_attempts": { "<issue_id>": { "attempt": 2, "due_at_ms": 1714000000000, "last_error": "..." } },
|
||||||
"claimed": ["<issue_id>", ...],
|
"claimed": ["<issue_id>", ...],
|
||||||
"completed": ["<issue_id>", ...]
|
"completed": ["<issue_id>", ...]
|
||||||
@@ -49,16 +62,27 @@ their own.
|
|||||||
```
|
```
|
||||||
|
|
||||||
Create empty defaults if missing. Always rewrite atomically (write `state.json.tmp`, rename).
|
Create empty defaults if missing. Always rewrite atomically (write `state.json.tmp`, rename).
|
||||||
|
Session ID for entries we own is whatever this session reports; for entries owned by
|
||||||
|
other sessions, the field stays as the spawner wrote it.
|
||||||
|
|
||||||
### 3. Reconcile
|
### 3. Reconcile
|
||||||
|
|
||||||
For each entry in `running`:
|
For each entry in `running`:
|
||||||
- Call `TaskList` (or `TaskGet` by id). If the task is no longer running and reported success,
|
|
||||||
move issue to `completed`, drop from `running`.
|
**Fast path — task is in our `TaskList`** (we spawned it; same session):
|
||||||
- If the task failed, move into `retry_attempts` with exponential backoff (capped by
|
- `TaskGet` for status. If completed successfully → move issue to `completed`, drop from `running`.
|
||||||
`agent.max_retry_backoff_ms`).
|
- If failed → move to `retry_attempts` with exponential backoff (capped by `agent.max_retry_backoff_ms`).
|
||||||
- Re-fetch the issue's current state from the tracker. If state is in `terminal_states` or
|
- Re-fetch issue state from the tracker. If in `terminal_states` or no longer in
|
||||||
no longer in `active_states`, call `TaskStop` on the worker and drop from `running`.
|
`active_states` → `TaskStop` and drop from `running`.
|
||||||
|
|
||||||
|
**Slow path — task is NOT in our `TaskList`** (spawned by another session, or our
|
||||||
|
session crashed and restarted):
|
||||||
|
- Read `.symphony/heartbeats/<issue-id>.json`.
|
||||||
|
- **Heartbeat fresh** (timestamp within `agent.heartbeat_stale_ms`, default 300000 = 5min)
|
||||||
|
→ another session is running it; do NOT dispatch a duplicate; leave alone.
|
||||||
|
- **Heartbeat stale OR missing** → consider abandoned. Move to `retry_attempts` (backoff
|
||||||
|
resets to attempt 1 if no prior retries; otherwise increments). Drop from `running`.
|
||||||
|
Delete the stale heartbeat file. Do NOT call `TaskStop` — we don't own the task.
|
||||||
|
|
||||||
For each entry in `retry_attempts` whose `due_at_ms <= now`, treat as eligible for dispatch
|
For each entry in `retry_attempts` whose `due_at_ms <= now`, treat as eligible for dispatch
|
||||||
(the issue ID is still claimed; do not dispatch a duplicate).
|
(the issue ID is still claimed; do not dispatch a duplicate).
|
||||||
@@ -101,7 +125,10 @@ For each candidate up to `slots`:
|
|||||||
run_in_background: true
|
run_in_background: true
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
3. Record `running[issue.id] = { task_id, worktree, started_at: now }`.
|
3. Record `running[issue.id] = { task_id, session_id, worktree, started_at: now }`.
|
||||||
|
`session_id` should be a stable identifier for this orchestrator session — used by
|
||||||
|
other sessions to know who owns this task. If no session ID is available, omit the
|
||||||
|
field; reconcilers will fall back to heartbeat-only checks.
|
||||||
4. Drop from `retry_attempts` if present.
|
4. Drop from `retry_attempts` if present.
|
||||||
|
|
||||||
### 6. Persist & exit
|
### 6. Persist & exit
|
||||||
|
|||||||
@@ -37,6 +37,11 @@ agent:
|
|||||||
max_retry_backoff_ms: 600000
|
max_retry_backoff_ms: 600000
|
||||||
max_concurrent_agents_by_state:
|
max_concurrent_agents_by_state:
|
||||||
"in progress": 2
|
"in progress": 2
|
||||||
|
# Worker heartbeat cadence — affects how quickly other Symphony sessions
|
||||||
|
# (Discord triggers, webhooks, separate user sessions) detect abandoned
|
||||||
|
# workers. Defaults are safe; tighten if you have many concurrent sessions.
|
||||||
|
heartbeat_interval_ms: 60000 # workers refresh the heartbeat at most this often
|
||||||
|
heartbeat_stale_ms: 300000 # reconciler treats heartbeats older than this as abandoned
|
||||||
---
|
---
|
||||||
|
|
||||||
You are working on issue **{{ issue.identifier }}** — {{ issue.title }}.
|
You are working on issue **{{ issue.identifier }}** — {{ issue.title }}.
|
||||||
|
|||||||
@@ -10,9 +10,10 @@
|
|||||||
"type": "object",
|
"type": "object",
|
||||||
"required": ["task_id", "worktree", "started_at"],
|
"required": ["task_id", "worktree", "started_at"],
|
||||||
"properties": {
|
"properties": {
|
||||||
"task_id": { "type": "string" },
|
"task_id": { "type": "string" },
|
||||||
"worktree": { "type": "string" },
|
"session_id": { "type": ["string", "null"], "description": "Identifier for the orchestrator session that spawned this task. Other sessions reconcile via the heartbeat file when this session's TaskList does not include the task_id." },
|
||||||
"started_at": { "type": "string", "format": "date-time" },
|
"worktree": { "type": "string" },
|
||||||
|
"started_at": { "type": "string", "format": "date-time" },
|
||||||
"issue_identifier": { "type": "string" }
|
"issue_identifier": { "type": "string" }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user