45–100%
Wide dispersion in decision accuracy
The four tested configurations produced materially different final-decision accuracy on the same controlled task.
2026 Research Programme
Testing LLM reliability in corporate credit decisioning.
Capital Benchmark has built an executable research platform to test whether general-purpose LLMs can reason through structured credit policy, challenging edge cases and final decision aggregation with the reliability required in a bank-controlled process.
Independent research · Executable benchmarks · Capability transfer
Two-minute explainer
How the Capital Benchmark research works.
A short walkthrough of the benchmark methodology, headline results, research-led capability-transfer model and the next phase of bank participation.
Capital Benchmark · LLMs in Credit Decisioning · 2026 Research Programme
What we found
Model choice and implementation architecture matter.
The research separates fluent memo writing from the harder operational question: whether a model applies the supplied policy correctly and reaches the right final decision.
93.7%
Rule accuracy can hide decision failure
Across the 640-memo V2 benchmark, granular policy accuracy materially exceeded aggregate decision accuracy.
100%
Scale result for the selected configuration
GPT-5 retained perfect observed decision accuracy across 1,000 additional public-company obligors.
Business model
Research agenda in, capability transferred out.
Capital Benchmark's operating model is designed to keep research open, independent and actionable. We use a public testing environment to convert a research agenda into a working reference implementation before transferring code, methods and test assets into a bank-controlled environment.
1
Research agendaWe define the questions that matter to credit-risk teams: policy edge cases, model-provider comparisons, architecture choices and operational controls.
2
Public / open testing environmentWe test sample obligor data, policy manuals and LLM configurations in a non-client environment so results can be challenged and discussed without requiring confidential bank data.
3
Capability transfer to banksOnce the reference approach is validated, we transfer code and test data into bank-specific pilots so internal teams can adapt the framework to their own policies, ratings and systems.
Method
Evidence first, implementation second.
We use a structured test harness rather than anecdotal prompts. The benchmark combines a common obligor dataset, generated facility requests, an explicit policy manual and a deterministic ground-truth engine.
The result is an evidence base that can support research, dialogue with credit teams and later institution-specific pilots.