AI
RayOrch: Programming and Executing Lineage-Controlled Multi-Grain Dataflows for Foundation-Model Data Preparation
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 15/09/2026, 20:00:00 UTC
Briefing created: 28/09/2026, 07:25:45 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 28/09/2026, 07:18:24 UTC.
Briefings
RayOrch is a distributed execution engine designed to manage lineage-controlled multi-grain dataflows for foundation model data preparation. It preserves parent-child relationships during execution by validating ordered variable cardinality expansions and matching gathers. The system achieves significant speedups on NVIDIA H20 GPUs, with MinerU scaling from 4 to 64 GPUs yielding 15.14 times speedup and video pipelines from 8 to 64 GPUs yielding 7.82 times speedup. End-to-end time reduction is observed compared to Ray Data and Daft, with 13.1% and 29.0% reductions respectively on MinerU tasks.
Key points
- RayOrch preserves parent-child relationships throughout execution by validating ordered variable cardinality expansions and matching gathers.
- On NVIDIA H20 GPUs, RayOrch achieves 15.14 times speedup when scaling MinerU from 4 to 64 GPUs and 7.82 times speedup when scaling a video pipeline from 8 to 64 GPUs.
Context and limitations
Speedup measurements are specific to NVIDIA H20 GPUs and limited to scaling experiments from 4 to 64 GPUs for MinerU and 8 to 64 GPUs for video pipelines. End-to-end time reductions are reported versus Ray Data and Daft but do not account for all deployment environments or full dataset sizes.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 28/09/2026, 07:25:45 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeidj2esp3zbyjjztqd6xx6njzx3tqducg2bi7sb3akir5hbnvzij4i