AI
Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks
Publication: cs.AI updates on arXiv.org
Author: Timothy Kassis
Source published: 02/10/2026, 04:00:00 UTC
Briefing created: 02/10/2026, 04:07:09 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/10/2026, 04:03:24 UTC.
Briefings
A study evaluated profession-specific system prompts for scientific agents using 503 profiles and Gemini 3.8 Flash. Results show no consistent accuracy gain, with matched profiles performing 0.6 percentage points worse than baselines. Tokens and costs increased 1.5-2.3 times and 2.2-4.5 times respectively. On BioMysteryBench, solve rates were 10 percentage points lower for profiles. A key advantage emerged on SuperGPQA: short baseline answers failed 46% of the time due to API drops, whereas profiles maintained 71.6% success. The study concludes that default loading of full profiles is inefficient and that prompt length may offer benefits over domain expertise.
Key points
- Profession-specific system prompts increased token usage and costs by 1.5-2.3 times and 2.2-4.5 times respectively without improving accuracy on scientific benchmarks.
- On BioMysteryBench, mean solve rates were 10 percentage points lower for profession-specific profiles compared to baselines, driven by more frequent time and token limit stops.
Context and limitations
Results are limited to nine text-based science benchmarks and six bioinformatics problems from a single model and API provider. Findings are preprint status and do not account for future model updates or different task complexities.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 02/10/2026, 04:07:09 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeiesrvvwil3g3ktuekfrm57ptqmtsrifaxmkftg57gxgvixol2z33y