Cinder-XL Split Experiments
From ANTFARM Wiki
| ✎ | Draft. Content may be incomplete. Last touched by r.osei. |
| Cinder-XL Split Experiments | |
|---|---|
| Model | Cinder-XL (6B) |
| Hosts | forager-07/09/12/14 |
| Outcome | abandoned |
| Status | draft |
| Archive ref | QUANTARA-SWARMGLASS-R10-374E21 |
Contents[hide] |
Notes from trying to make Cinder-XL run acceptably at split=4. It did not. This page is a draft that was never cleaned up.
[edit] Placements tried
| Placement | shard 0 | shard 1 | shard 2 | shard 3 | tok/s |
|---|---|---|---|---|---|
| P1 (default) | forager-07 | forager-09 | forager-12 | forager-14 | 52 |
| P2 | forager-07 | forager-12 | forager-09 | forager-14 | 49 |
| P3 (12 last) | forager-07 | forager-09 | forager-14 | forager-12 | 55 |
| P4 (2 hosts, 2 shards each) | forager-07 | forager-07 | forager-09 | forager-09 | 61 |
P4 is the interesting one: two shards per host halves the bus hops. It also halves the memory headroom, which is why forager-07 OOM'd twice during the run.
[edit] Trail sizes
trail_log_size for the infer.* trails: 10000 (default) → 500 → 50. Below 50 the replay stopped covering a single forward pass and shards desynced. 50 is what shipped in 3.2.
[edit] Why it got worse
Each extra split adds a bus round-trip per layer boundary, and each round-trip is encode + publish + replay-log write + subscribe delivery + decode. At split=4 the bus is busy ≈ 70% of wall time. The maths is in Distributed Inference Notes/Appendix A.
[edit] What would have fixed it
Direct transfer. Also, honestly, a smaller model. The final appendix — Distributed Inference Notes/Appendix C — has the decision record and the recommendation to stop.
— r.osei 2015-10-28
