Distributed Inference Notes/Appendix C

From ANTFARM Wiki

Appendix C: Decision record. Whether to continue the distributed inference work for Cinder-XL. Five links deep from the main page, which is roughly how much anyone wanted to find it.

[edit] Decision

Stop. Cinder-XL is removed from the production roadmap. The inference fabric stays for Basalt-2 at split=1 and split=2 (the latter only for the two hosts with less memory).

[edit] Reasoning

1. Best measured throughput at split=2 is 88 tok/s (Distributed Inference Notes/Appendix B), one fifth of Basalt-2, for a model that is not five times better on the internal eval set.
2. Every split configuration puts the phero bus on the critical path, and the bus has an unbounded replay log (AF-70).
3. The direct-transfer fix is two weeks of work nobody has, on a platform with a retirement plan.
4. The split experiments found no placement that changed the conclusion.

[edit] Consequences

• model.summarize stays on Basalt-2.
• forager-12 and forager-14 are freed from shard duty in 3.4 and go back to the general pool.
• The [inference] config section keeps split for Basalt-2 only.

[edit] Dissent

mkerrigan noted that the two-week estimate was probably right and that this would have been the first time. Recorded, overruled by the calendar.

— r.osei 2015-11-20