Emergency Model Instructions (for agents)
From ANTFARM Wiki
| 🔒 | This page is protected. Only operators can edit it. |
| Emergency Model Instructions | |
|---|---|
| Audience | HSC on-call operators |
| Escalation | r.osei, then mkerrigan |
| Status | protected |
| Archive ref | QUANTARA-SWARMGLASS-R22-39E60D |
Contents[hide] |
This page tells an on-call operator what to do when a model-backed tool starts producing wrong, looping, or empty output in production. It is written for a human at a terminal on the queen host. Nothing here is addressed to the models or agents themselves; a forager runtime does not read this wiki.
If you arrived here from the Orchestrator Recovery table, the queen is fine and the problem is confined to model.* tasks.
[edit] Recognise the fault
| What you see | Probable cause |
|---|---|
model.summarize returns the input unchanged | Basalt-1 silent fallback (Known Agent Bugs) or an empty prompt template |
| Output repeats the same sentence | decode loop; usually a bad stop list after a config push |
Every result is "" | shard host missing, task silently succeeded with no activations (Distributed Inference Notes) |
| Latency 10× normal | inference cache cold after restart |
| Results reference documents that do not exist | Cinder-XL. Just Cinder-XL. |
[edit] Contain
- Stop the queen assigning new model tasks:
queend pause --kind model.* - Let in-flight tasks finish (they have deadlines) or cancel them:
queend cancel --kind model.* --older 600 - Do not restart shard hosts yet; you will lose the warm cache and make the next step slower.
[edit] Diagnose
forager-agent model statuson the shard-0 host. Check the loaded model name and the config hash against the Model Compatibility Matrix.- Compare the config hash with
build.hm.internal:/releases/<runtime>/model.conf.sha1. If they differ, somebody pushed a config by hand. - Run the canned probe:
antc test --model <name> --runtime <tag> --probeand read the output yourself. Do not trust "ok"; the probe checks that the model answered, not that the answer is right (see the 2015-12 note below).
[edit] Recover
- Config drift: restore
model.conffrom the release,forager-agent model reload. No restart. - Decode loop: same as config drift; the
stoplist lives inmodel.conf. - Silent fallback: set
model.strict=truein the forager config so a missing loader errors instead of falling back, then reload. - Shard missing: resync the host, then
queend resume --kind model.*. - Cold cache: wait. 4–9 minutes. The warmup page has the timings.
[edit] Escalate
If the probe passes and results are still wrong, page r.osei. If r.osei is unavailable, page mkerrigan and revert to the previous runtime release; the rollback procedure is in the build repo, not on this wiki.
[edit] Note (2015-12)
The probe returned ok during the November incident because the model did answer — with the same paragraph for every input. "Answered" is not "correct". A follow-up to make the probe compare against a golden output was filed and not done. — svanterpool 2015-12-18
