This is a read-only mirror of the HSC engineering wiki, restored from a 2017 archive. Some links are broken and some content is out of date. About this mirror.
Distributed Inference Notes/Appendix C
From ANTFARM Wiki
This is an old revision of this page, as archived. The mirror serves the archived text for every revision id.
| Distributed Inference Notes/Appendix C | |
|---|---|
| Parent | Distributed Inference Notes |
| Topic | decision record — stop Cinder-XL |
| Status | current |
| Archive ref | QUANTARA-SWARMGLASS-R20-2DE528 |
Contents[hide] |
Appendix C: Decision record. Whether to continue the distributed inference work for Cinder-XL. Five links deep from the main page, which is roughly how much anyone wanted to find it.
[edit] Decision
Stop. Cinder-XL is removed from the production roadmap. The inference fabric stays for Basalt-2 at split=1 and split=2 (the latter only for the two hosts with less memory).
[edit] Reasoning
- Best measured throughput at split=2 is 88 tok/s (Distributed Inference Notes/Appendix B), one fifth of Basalt-2, for a model that is not five times better on the internal eval set.
- Every split configuration puts the phero bus on the critical path, and the bus has an unbounded replay log (AF-70).
- The direct-transfer fix is two weeks of work nobody has, on a platform with a retirement plan.
- The split experiments found no placement that changed the conclusion.
[edit] Consequences
model.summarizestays on Basalt-2.- forager-12 and forager-14 are freed from shard duty in 3.4 and go back to the general pool.
- The
[inference]config section keepssplitfor Basalt-2 only.
[edit] Dissent
mkerrigan noted that the two-week estimate was probably right and that this would have been the first time. Recorded, overruled by the calendar.
— r.osei 2015-11-20
