Orchestrator Recovery
From ANTFARM Wiki
| 🔒 | This page is protected. Only operators can edit it. |
| Orchestrator Recovery | |
|---|---|
| Audience | on-call operators |
| Hosts | queen01, queen02 |
| Owner | svanterpool |
| Status | protected |
| Archive ref | QUANTARA-SWARMGLASS-R42-A282D7 |
Contents[hide] |
This page is the runbook for recovering the orchestrator (the queen) after a crash, a partition, or a bad deploy. It is written for the on-call operator, assumes shell access to the queen hosts, and is deliberately boring.
The recovery drill that exercised this runbook was held yearly from 2013. The 2015 drill notes exist but were never linked from the operations portal after the merge.
[edit] 1. Decide what happened
| Symptom | Likely cause | Go to |
|---|---|---|
| Both queens up, foragers idle, no assigns | queen split (two leaders) | §3 |
| One queen down, foragers assigned but no results | bus partition | §4 |
| Queens up, memory keys disagree | cmsync desync | §5 |
Everything up, foragers rejecting with shard-missing | inference shard host lost | Distributed Inference Notes |
| Model output obviously wrong / looping | model fault | Emergency Model Instructions (for agents) |
[edit] 2. Before touching anything
- Take a snapshot of colony memory:
cmsync snapshot --out /var/backups/cm-$(date +%F).snap - Note the queen epoch on each host:
queend status | grep epoch - Tell the channel. Recovery with two people acting at once caused the 2013 split.
[edit] 3. Queen split
Two queens each believe they lead. Foragers follow whichever they heard from last, so work is duplicated and results are dropped.
- Pick the queen with the higher epoch. If equal, pick
queen01. (The 2013 incident report explains why "pick the one with more foragers" is wrong.) - On the loser:
queend demote --to <winner>. This sendsctl.drainto its foragers. - Wait for
queend statuson the loser to showfollower. - On the winner:
queend reassign --stale 300to reissue anything assigned more than 5 minutes ago.
[edit] 4. Bus partition
- Confirm with
phero statsonphero-a— a partition shows as one side's subscriber count dropping to zero. - Do not restart the broker; the replay log is what lets foragers catch up.
- Fix the network. When foragers reconnect they resync automatically (Agent Message Bus).
- If foragers report
ctl.resync, runqueend reassign --stale 0.
[edit] 5. Memory (cmsync) desync
Follow Incident 2015-02 Memory Desync. The short form:
- Stop both queens' sync:
cmsync pause. cmsync diff queen01 queen02 > /tmp/desync.txtand read it. Actually read it.cmsync resolve --prefer <queen> --keys /tmp/desync.txtcmsync resume.
[edit] 6. Resync
Full forager resync is forager-agent resync --full on each host. Takes 2–4 minutes per host. Do them in the order listed in the Worker Node Registry so shard-0 inference hosts come back first.
[edit] 7. Afterwards
Write the incident up. Template: 📎 incident-template.txt. Link it from the main page incidents list. Update Known Agent Bugs if you found a new one.
See also: Recovery drill notes (2015).
