AI

Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 10/11/2026, 08:33:26 UTC.

Briefings

A study evaluated 15 open-source and 10 proprietary LLMs against eight Veteran Subject Matter Experts (SMEs) on annotating 149 veteran-related tweets across five severity categories. Human reviewers showed lower agreement than LLMs, with LLMs achieving 0.8336 ordinal agreement under baseline prompts. GPT-5.6-Terra performed best overall, while Qwen2.5-7B excelled at detecting Acute Crisis with Suicidal Ideation. The study concludes LLMs are promising but require careful evaluation before high-stakes use.

Key points

• Eight Veteran Subject Matter Experts (SMEs) provided annotations for 149 veteran-related tweets.
• GPT-5.6-Terra achieved the best overall performance among all evaluated models.

Context

This research addresses the reliability of Large Language Models (LLMs) in safety-critical mental health tasks by comparing them against human experts. The findings are critical for determining whether LLMs can be safely deployed in high-stakes crisis annotation scenarios.

Limitations

The study relies on a single snapshot from November 2026 and uses preprint data (arXiv:2611.10048v1). Results may vary with different prompt variants, data sources, or model versions not tested in this study.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 10/11/2026, 10:05:07 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiexlp6wf3lndozvomu4grq3thubclu3fyg63fu75b55z4f5yemwem