PromptBio-Bench

Benchmarking AI Agents on Bioinformatics Tasks

AI agents are increasingly applied to bioinformatics workflows, yet their practical capabilities on real research tasks remain poorly characterized. Existing benchmarks rarely capture the diversity of file formats, multi-step pipelines, and domain-specific evaluation criteria encountered in practice. Here we introduce PromptBio-Bench, a systematic evaluation framework for AI agents on bioinformatics and biostatistics tasks. PromptBio-Bench comprises 244 tasks spanning 27 sub-fields across Bioinformatics and Biostatistics & Data Science, each pairing a natural-language prompt with structured reference outputs scored by a 4-step LLM-assisted pipeline (Match → Detect → Recommend → Compare). We evaluate 3 frontier agents and find that while agents complete the majority of tasks, accuracy varies substantially across fields and file formats, with output omissions and format mismatches as the dominant failure modes.

Two scientific domains, 27 sub-fields

Tasks span the full breadth of computational biology — from raw sequence processing to statistical inference and machine learning.

Bioinformatics

Sequence processing, variant analysis, alignment, epigenomics, transcriptomics, proteomics, single-cell & spatial omics, metagenomics, structural biology, and more.

Genomics Transcriptomics Proteomics Epigenomics Metabolomics Metagenomics Single-Cell & Spatial Omics Structural Bioinformatics Synthetic Biology Immunoinformatics Phylogenetics / Evolution Clinical Bioinformatics Functional & Network Analysis Sequence Data Processing Multi-Omics Integration

Biostatistics & Data Science

Statistical inference, regression, classification, clustering, dimensionality reduction, survival analysis, Bayesian methods, and data visualization for biological research.

Statistical Inference Regression Classification Clustering Dimension Reduction Bayesian Statistics Survival Analysis Longitudinal Analysis Model Evaluation Data Visualization Data Wrangling Descriptive Statistics

Evaluation pipeline

A 4-step LLM-assisted pipeline scores agent outputs against reference answers. Steps 2 and 3 run only once per task — shared across all agents — reducing cost while preserving comparability.

Match

LLM maps agent output files to reference files. Runs per-agent.

→

Detect

Identify the format of each reference file. Shared — runs once.

→

Recommend

LLM selects comparison strategy and parameters per file. Shared — runs once.

→

Compare

Compute similarity (0–1) for each file pair. Runs per-agent.

Final score is the average similarity across all scored reference files.

Supported file formats & comparison strategies

Format	Extensions	Default strategy	Other strategies
FASTA	`.fasta .fa`	exact	approximate, summary
FASTQ	`.fastq .fq`	exact	approximate, summary
BAM / SAM / CRAM	`.bam .sam .cram`	summary	exact, approximate, coverage, variant
BAI / CRAI	`.bai .crai`	functional	summary
TBI / CSI	`.tbi .csi`	functional	summary
FAI	`.fai`	exact	approximate, summary
VCF / BCF	`.vcf .vcf.gz .bcf`	summary	exact, approximate
BED / BedGraph	`.bed .bedgraph .bg .bigbed`	approximate	exact, overlap, correlation, summary
BigWig / Wig	`.bw .bigwig .wig`	summary	exact, approximate, correlation
Table	`.csv .tsv .gct .xlsx`	approximate	exact, summary, semantic
PDB / CIF	`.pdb .cif .mmcif`	exact	approximate, summary
Image	`.png .jpg .pdf .svg`	semantic	—
Text	`.txt .text`	semantic	exact, approximate, numeric, summary

Key findings

Current AI agents are capable but uneven bioinformatics collaborators.

Gap between completion and success. All three agents complete 88–99% of tasks, yet mean accuracy against expert reference files is only 0.72–0.76 — demonstrating that producing an output and producing the correct output are fundamentally different challenges.

Accuracy declines sharply with task difficulty. On high-difficulty tasks, accuracy drops to 0.56–0.69 across all agents, highlighting that current agents struggle with multi-step, domain-specific pipelines requiring chained tool use and correct biological assumptions.

Biomni and ToolsGenie perform comparably; Stella lags in completion. The two top agents show similar overall accuracy, while Stella's 88% completion rate (vs. 98–99% for the others) reveals a distinct failure pattern of not producing output at all rather than producing incorrect output.

Bioinformatics and data science tasks expose different agent weaknesses. Track A (131 bioinformatics tasks: CLI tools, specialist formats such as FASTA/VCF/BAM/PDB) and Track B (113 data science tasks: statistics, ML, visualization) reveal complementary gaps — agents handle structured tabular analysis better than complex genomics pipelines.

LLM-assisted evaluation scales to heterogeneous file outputs. The Match → Detect → Recommend → Compare pipeline scores 13 distinct file formats without hard-coded rules, providing a reusable infrastructure for tracking progress as foundation models and agent frameworks continue to evolve.

Citation

If you find PromptBio-Bench useful, please cite our work.

@article{guo2026promptbio,
  title={PromptBio-Bench: benchmarking LLM-based bioinformatics agents for end-to-end data analysis},
  author={Guo, Wenbin and Zhang, Minzhe and Han, Bowei and Ma, Youjia and Leng, Yang and Hebbar, Shishir and Zhou, Xiaoyuan and Gu, Wenhao and Yang, Xiao and Dhar, Shashi},
  journal={bioRxiv},
  pages={2026--05},
  year={2026},
  publisher={Cold Spring Harbor Laboratory}
}

Tasks & datasets: huggingface.co/datasets/promptbio-ai/promptbio-bench-data

PromptBio-Bench

Two scientific domains, 27 sub-fields

Bioinformatics

Biostatistics & Data Science

Evaluation pipeline

Match

Detect

Recommend

Compare

Supported file formats & comparison strategies

Key findings

Common failure modes

Agents evaluated

Citation

Results explorer

Agents

Completion

Accuracy

Computation cost