FinBen: Financial Benchmarking and Evaluation Suite for Large Language Models and Agents

FinBen brings together the lab’s collaborative research and open-source contributions to financial benchmarking and evaluation for large language models and agents. The project connects benchmark datasets, specialized financial tasks, the Open Financial LLM Leaderboard, and workflow-level evaluation with broader work on AI readiness and governance.

Research Foundations

The original FinBen paper, published in the NeurIPS 2024 Datasets and Benchmarks Track, established a broad evaluation resource spanning 42 datasets and 24 financial tasks. Xiao-Yang Liu served as a corresponding author and contributed to the scientific leadership of this multi-institution collaboration. Its studies included financial text understanding, reasoning, forecasting, decision-making, and agent-based and retrieval-augmented evaluation.

Subsequent collaborative research broadens the evaluation scope. MultiFinBen (ACL 2026) addresses multilingual and multimodal financial tasks. FinAudio evaluates financial speech recognition and audio summarization. FinTagging studies numerical identification and full-scope XBRL concept linking, while FinAuditing evaluates semantic, relational, and mathematical consistency across financial disclosures. Herculean extends evaluation to end-to-end agent workflows for trading, hedging, market insights, and auditing.

Evaluation Infrastructure and Financial Agents

The Open FinLLM Leaderboard translates benchmark research into a shared platform for comparing models and contributing new financial tasks and results. The lab’s Evaluation and Benchmarking Suite for Financial Large Language Models and Agents connects this work to a wider lifecycle of exploration, readiness, and governance. Model comparisons are complemented by evaluation of an agent’s tools, orchestration, execution traces, and task-specific behavior.

Within the FINOS community, the AI Evaluation and Benchmarking Framework organizes financial use cases, risks, and metrics. The FinSight community prototype, developed with Columbia’s SecureFinAI Lab, provides a concrete financial-agent use case for studying multi-agent earnings-call analysis and system-level evaluation.

Open Collaboration

Open collaboration is central to this research direction. Joint work includes the Open Financial LLM Leaderboard with FINOS and Hugging Face, evaluation and governance research with collaborators from the Linux Foundation and PyTorch Foundation, AgentOps work with Red Hat, and documentation with the Rensselaer Center for Open Source. Collaborators include Matt White, Vincent Caldeira, Luca Borella, academic researchers, and student contributors.

The lab’s Open Intelligence work on MOF, MOT, and OpenMDW provides complementary foundations for openness, reproducibility, and model release.

Community Activities

Selected Research

Code, Data, and Community Resources

FINOS Leaderboard Repository · Open Financial LLM Leaderboard · Documentation

Original FinBen Code and Evaluation Resources · FinBen and PIXIU Datasets · MultiFinBen Repository

FINOS AI Evaluation and Benchmarking Framework · FinSight Agent · Leaderboard Launch

Columbia Affiliations
School of Engineering