AI
SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 01/10/2026, 11:01:48 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 01/10/2026, 08:18:24 UTC.
Briefings
SlideDP is a synchronous data-parallel runtime designed for shared-host multi-GPU systems that enables full-parameter LLM fine-tuning beyond GPU memory limits. It maintains one authoritative host state and decouples communication from state layout to improve throughput. In matched-batch sweeps, SlideDP achieved geometric-mean throughput ratios of 1.46-2.64 times higher than SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s, it approached GPU-resident FSDP2 throughput for Qwen3-14B at a smaller batch size, while with a larger batch it processed over 1M tokens per step and exceeded FSDP2's peak throughput by 11.2%. The system also supports 256K-token sequences for fine-tuning Qwen2.5-72B on four RTX 4090 GPUs.
Key points
- SlideDP maintains one authoritative host state and decouples communication routes from state layout to improve throughput in shared-host multi-GPU systems.
- In matched-batch sweeps, SlideDP achieved geometric-mean throughput ratios of 1.46-2.64 times higher than SlideFormer, MegaTrain, and ZeRO-Offload.
Context and limitations
Evidence covers four H100 systems and four RTX 4090 GPUs; results are from matched-batch sweeps and analytical step-time modeling rather than live production benchmarks.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 01/10/2026, 11:01:48 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeiazb3pu2pdhj5ydvkkjiablp37dx7xqt4k3iagn724w2crklpg4ei