AI
PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 30/09/2026, 04:48:51 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/09/2026, 04:18:24 UTC.
Briefings
PlaylistEval is an agentic framework that builds video-language judge benchmarks over 100-hour playlist collections without human annotation. It generates controlled causal degradation questions across seven domains, achieving 93.0% human agreement on a stratified subset of 152 pairs. Evaluations of 17 multimodal models show frontier judges reach 75.4% pairwise accuracy, while open-source models perform significantly lower. The benchmark and evaluation code are available at playlisteval.github.io.
Key points
- PlaylistEval uses an agentic framework to build video-language judge benchmarks over 100-hour playlist collections without human annotation.
- The benchmark contains 630 pairs across seven domains, achieving 93.0% human agreement on a stratified subset of 152 pairs.
Context and limitations
Existing benchmarks are limited to short video segments, often failing to capture full context. Human judgment collection does not scale to ultra-long video collections, and causal degradation questions may not fully represent real-world retrieval challenges.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 04:48:51 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeigpwublpbqyq7gbpxxdivciqlmw2zhpxsoj4xezia4dnx3ohy2uqa