This is a read-only mirror of the HSC engineering wiki, restored from a 2017 archive. Some links are broken and some content is out of date. About this mirror.
Incident 2015-02: Memory Desync
From ANTFARM Wiki
| 🗄 | This is an archived page. It is kept for historical reference and is no longer maintained. |
| Incident 2015-02: Memory Desync | |
|---|---|
| Date | 2015-02-19 to 2015-02-22 |
| Duration | 3 days |
| Impact | silently wrong model config, stale index |
| Status | archived |
| Archive ref | QUANTARA-SWARMGLASS-R27-0FD92B |
Contents[hide] |
| 🗄 | This is an archived discussion or record. Do not edit it; start a new thread on the relevant talk page instead. |
[edit] Summary
For three days colony memory keys written on both queens resolved in favour of queen02, whose clock was 41 seconds ahead. Because queen02 was the test colony, its writes included a test model config; model.summarize in production ran with it for the duration. Nobody noticed until inference measurements (Appendix B, runs 020–021 in the raw tables) came out wrong.
[edit] Timeline (UTC)
| Time | Event |
|---|---|
| 02-19 01:10 | ntp on queen02 stops syncing (disk full, log partition). clock drifts. |
| 02-19 22:00 | queen02 runs the nightly test graph, which writes cm://model/conf/active on the test colony. cmsync gossips it to queen01, because the key namespace was shared. |
| 02-19 22:00 | queen01's copy loses the tie: stamps concurrent, wall clock favours queen02 (41s ahead). |
| 02-20 to 02-22 | production foragers read the test config on cache expiry. Summaries get shorter and stranger. |
| 02-22 15:30 | r.osei's measurement run produces numbers 30% off. Investigates. Finds the config hash mismatch. |
| 02-22 16:10 | svanterpool runs cmsync diff, sees 412 keys preferring queen02. |
| 02-22 17:00 | resolved with cmsync resolve --prefer queen01. |
[edit] Root cause
Two rules combined: last-writer-wins by wall clock, and shared key namespace between the production and test colonies. Either alone would have been fine.
[edit] Remediation
- Tie-break changed to prefer the leader queen, then epoch, never wall clock (3.2, AF-67).
- Test colony gets its own key prefix
cm://test/. Enforced by config, not by code. - Clock skew between queens > 5 s now refuses sync (
cmsynclogs and pauses). This produced a new failure mode in April 2015 when both clocks were fine and the log partition filled again. - The desync test harness jbrandt wrote lives on Talk:Memory Synchronization because nobody agreed where else to put it.
[edit] Remediation (runbook form)
This is section 5 of the recovery runbook.
— svanterpool 2015-08-20
