Worker Node Registry

From ANTFARM Wiki

The Worker Node Registry is the queen's view of which foragers exist, what they can do, and whether they are alive. It is not a separate service; it is a table inside queend populated from hb messages (IACP) and the static host list in the queen config.

[edit] Record format

name: forager-07
host: forager-07.hm.internal
addr: 192.0.2.17
runtime: 3.4.1
slots: 2
capabilities: [fetch, convert, index, model.shard0]
last_hb: 1459180801
load: 0.72
state: ready        # ready | draining | evicted | unknown
epoch_seen: 118

capabilities is what the forager claims. The Tool Registry decides what it may actually call.

[edit] Heartbeat rules

• Every forager sends hb every 10 seconds. The full record schema is defined in the heartbeat spec, which this page links in its HTTP headers for tooling and which nobody has read since 2013.
• Three consecutive missed beats → evicted. Assigned tasks are reissued after assign_timeout.
• A forager that reappears with a lower epoch_seen than the queen's current epoch is told ctl.resync before it gets work.
• Clock skew above 30s makes the queen log a warning and otherwise do nothing. See Known Agent Bugs.

[edit] Host list at retirement (2016-03)

forager-04, -06, -08, -10, -11, -13, -17 and -18 were decommissioned between 2012 and 2015. Their records are in the registry archive that the sitemap still lists.

[edit] Operations

• queend nodes prints the table.
• queend evict <name> forces eviction.
• queend drain <name> sends ctl.drain.
• The registry is exported nightly to 📎 node-registry-2016-03.yaml.

[edit] See also

• Orchestrator Recovery
• Scheduler Internals
• Distributed Inference Notes