Frontier AI Models Need Frontier Context: Why the Smartest Model Alone Won't Win

Every few weeks a new model tops a public leaderboard, and someone asks why we still invest so heavily in a search stack when "the model can just read everything." At AlphaSense, we run that experiment continuously so our users don’t have to: every major model release, every major feature update, month after month, against real-world tasks. This article summarizes what the current data says about which model wins and why, how we measure it, the critical role of our search and harness, and what we're building next.

The short version: In finance and business research, today’s bottleneck on answer quality is no longer raw model intelligence. It's context. Frontier models are extraordinary reasoners, but they are mediocre searchers, and the gap between those two skills shows up in both quality and cost.

A frontier model doing its own retrieval over our content with raw vector search (a good proxy for your typical market data/content MCP) costs roughly 3x more per question than the same model running in our harness with AlphaSense Search. And the AlphaSense Search-powered answers are preferred more than 2:1 compared to the vector-RAG baseline, all other things being held equal.

Here are a few model-specific takeaways that are discussed in greater detail later in the article:

• GPT-5.6 Sol tops our benchmark, delivering the best overall performance and quality/cost balance when restricting the system to using a single model.
• GPT-5.6 is more token efficient and thus cheaper overall than a leading open-weights model, Kimi K3.
• Opus 5 over-fetches and delivers worse performance that is 5x more expensive than Opus 4.8, showing that upgrading to the latest models is a non-trivial decision and depends on the task at hand. (Incidentally, Opus 5 is great at slide generation.)
• Gemma 4, an open-weights model from Google DeepMind, delivers the same performance at 40x lower cost than Sonnet 5.

The rest of this post, the first of an ongoing series, covers where these gaps come from, how the benchmark works, what we learned about individual models, and the two engineering programs designed to widen the gap: training our own search agent, and pre-computing the answers our users are about to ask for.

Why the Best Models Are Bad Searchers

Hand a frontier model an MCP on financial documents and a hard research question, and you’ll have to design around three failure modes:

Query generation. Real questions arrive with fiscal-period arithmetic, ticker collisions, speaker specificity, and entity type ambiguity (e.g., is JP Morgan a company or a source). Treating everything like a web search doesn’t work in finance.

Prioritization. Embedding distance is indifferent to source authority. An earnings call transcript, a sell-side preview note, a news article, and a 10-Q filing all score similarly on cosine similarity, but their relevance in the eyes of a human analyst is very different depending on the question being asked. You can trust Twitter for your jokes, but not for the numbers going into your model. Today’s LLMs understand what these documents are, but knowing they’re there, finding them efficiently, and prioritizing them effectively requires a specialized harness.

Stopping. Models are surprisingly bad at knowing when to stop searching and when to go deeper. Models often call it quits too soon (retrieval results bring up 2025 Q4 financials on a question of current financial performance) or don’t know when the document they are looking for doesn't exist (like a future quarter’s financial performance). They are trained to compensate for low-precision retrieval with volume: fetch more, skim, re-fetch, and reason over a context window increasingly full of duplicates and noise. Long, diluted contexts degrade answer quality while inflating cost. This is why AlphaSense is 3x cheaper: With a complete search index trained and tuned on tens of millions of searches, freshness, authority, and relevance scores can be returned for query sessions, letting the model know when to stop.

Some of the most capable models we tested were the worst offenders on token discipline (see Opus 5). AlphaSense Search represents more than a decade of data processing, indexing, and ranking engineering that turns out to be exactly what an agent harness needs.

The net effect is that in an optimized setup, the LLM spends its tokens reasoning over the right context instead of hunting for it. Same model, same corpus: about a third of the cost and significantly better answers.

The Advantage This Benchmark Doesn't Measure: Content

Every configuration, including all models across the vector-RAG baseline and AlphaSense Search alike, runs over the same indexed corpus. That isolates the retrieval, harness, and model variables, which is a useful experimental design.

It also means the results measure none of the work upstream of the index: a large proprietary library of expert interviews and channel checks; sell-side research, global filing coverage, event transcripts, and industry trade journals; and the pipelines that normalize, enrich, link, and tag all of it. The gaps reported here are the floor, measuring just the search agent: The real-world delta, content included, is larger than anything in these charts.

Methodology

The benchmark is based on 245 challenging, multi-step finance questions modeled on real analyst workflows: time-anchored, cross-source, and deliberately hard. We create many benchmarks across features and use cases in AlphaSense, and we update them regularly as tasks evolve and complexity increases. You can find sample questions in the Appendix.

For each question, we generate an independent scoring rubric before any model produces an answer. Pre-registering rubrics is useful because it prevents graders from drifting toward whatever the models happen to output, and it lets us encode hard requirements up front.

For example, one question on RevPAR (see Appendix) is scored out of 62 points: 6 mandatory core checks (26 pts), 14 quality signals (36 pts), and 3 penalty checks (−7 pts).

Retrieval quality and answer quality are graded independently by language-model judges. Results are reported as relative scores (vector-RAG baseline = 1.0) and head-to-head preferences.

Results

Holding the baseline model (Opus 4.8) constant and swapping only the retrieval layer moves preference to 1.7:1, the purest measurement of what the AlphaSense Search stack alone contributes. Choosing the best frontier model (GPT-5.6 Sol) on top of AlphaSense retrieval pushes preference to 1.8:1. Leveraging the optimized AlphaSense harness around that model (i.e., query generation, planning, search score pass-through) gets to 2.1:1, and routing each task to the best model for that task type reaches 2.8:1.

The per-model picture, with everything running on AlphaSense Search:

Quality scores are medians from our July 2026 harness run, relative to the vector-RAG baseline of 1.0. Costs in the chart are median per-question totals including retrieval-loop tokens, not list token prices.

A few datapoints deserve commentary:

Orchestration wins over any single model. No model wins more than a plurality of the benchmark questions. Give the smartest model a lot of great tools, and it can forget to use the simple tool for simple tasks. Different models win on different question types: metric extraction, screening, long-form synthesis. That's why per-task routing produces the 2.8x result. Model routing is not only about cost savings; it’s about leveraging the best model for each use-case, and getting an optimal combination of quality, cost, and speed for each task.

Newer is not automatically better. Opus 5 scores below its predecessor Opus 4.8 (1.46 vs. 1.60) at several times the per-question cost. The main issue is that it over-fetches: more retrieval calls, longer contexts, worse synthesis. On the other hand, Opus 5 has proven to be a great model for Slide Generation. This is a strong argument for continuous model evaluation: In-product performance is a property of the model and use case pair, and it drifts with every release. Without a test suite and rigorous testing, regressions can ship to production silently.

Token price does not equal question cost. Kimi K3 is a lot cheaper per token than GPT-5.6 Sol, yet more expensive per question, because it spends many more tokens assembling context before it can answer. When you buy tokens by the million but deliver answers by the task, tokens-to-completion is the metric that matters.

Open weights will play a critical role. Gemma 4-31B matches Sonnet 5 on answer quality at roughly 40x lower cost. Gemma is a great candidate for sub-agent delegation by a frontier lead model. Open weights mean it’s served on Cerebras hardware where it runs 10–20x faster. The combination of good-enough quality for many tasks and very low latency unlocks high-volume and interactive use cases that would be uneconomical on frontier models.

Why This Matters Right Now

With indiscriminate tokenmaxing mostly behind us, and companies looking for cost control and ROI from their AI investments, the answer lies in getting task-by-task model choice and harness optimization right.

This becomes even more critical as we move to longer-running agentic workflows, where all of these quality and cost benefits (and issues) compound: Even if 95% accuracy was acceptable in a single step with human oversight, this compounds to a coin-toss accuracy in a 10-step agentic process.

What's Next

Two programs aim to move the frontier at AlphaSense even further.

Training our next custom search agent. The evaluation machinery above is also, conveniently, a reinforcement learning environment: pre-registered rubrics, independent judges, and millions of search trajectories. We're using it to fine-tune our next proprietary model specialized in context collection: query planning, retrieval-tool use, coverage tracking, and context evaluation. Our objective is model orchestration-level context quality (~2.8x) at up to 40x lower cost.

Frontier models remain critical. Training a custom search agent does not mean moving off frontier models. Planning, deep multi-step reasoning, coding, data analysis, and Excel/PowerPoint generation stay routed to frontier models, exactly as our routing results argue they should. Context collection is one step in the pipeline — currently the most expensive one — and the step where a specialized model, trained on our corpus and our reward signal, beats a generalist.

Pre-computing the questions users are about to ask. The most efficient token is the one never spent. A growing share of our workload leverages recurring structures: supply-chain relationships, channel check signals, and the key debates around companies, industries, and cross-sector themes.

Today, an agent reconstructs these from hundreds of documents, per user, per query, at run-time. We're building offline pipelines that pre-compute them as first-class, queryable datasets — supply-chain graphs assembled across filings, transcripts, and expert calls; channel-check syntheses coordinated by AI interviewers; key-debate ledgers maintained per ticker with the bull and bear evidence attached, updated as new documents land.

Our SuperAnalyst then starts from the pre-computed structure and spends inference only on the delta, each new question, new document, new angle. For these workflows, that's not an incremental saving; it's orders of magnitude, with better consistency, lower latency, and cleaner citations as side effects. Just like standardized fundamental data helped analysts avoid extracting the same data manually from unstructured company disclosures, we are now structuring the next major layer of analysis that everyone has been doing independently. The result: high quality, speed, and efficiency all at the same time.

The Durable Asset

GPT-5.6 Sol holds the quality bar today with good cost efficiency. But as models keep leapfrogging each other, the best choice today might not be the best tomorrow. Your intelligence layer should be able to adapt at a moment's notice, and that's the business we're in. What compounds on our side is everything around the model: the corpus and the pipelines that feed it; a retrieval stack that makes any model 2x better and 3x cheaper; a routing layer that captures the best of each model release; and, underneath all of it, an evaluation harness rigorous enough to catch a flagship regression within days, and to serve as the reward signal for training the next generation of our own agents.

Finance and business demand frontier models with frontier context, and that’s what we’re building at AlphaSense. We will be releasing regular benchmarking of popular closed- and open-source models and tough datasets that represent the work our customers are doing on our platform.

Appendix

Sample Benchmark Questions:

• CardioTTR-Transform did not meet its PEP (announced today). Please can you analyse commentary from BridgeBio, Pfizer, Alnylam, and any other pharmaceutical companies?
• Analyze the key trends emerging this year for firms providing CDL (Commercial Driver's License) training services. Focus on shifts in demand driven by the trucking labor market, impacts of regulatory changes or entry-level driver training (ELDT) requirements, and how training providers are adapting to new technologies such as simulators or online instruction. Discuss trends in tuition pricing, employer-sponsored training programs, and capacity constraints within the industry.
• Give me a detailed rundown of all the major acquisitions completed by Clean Harbors (CLH) - make sure all the data is up to date as of July 2026
• Burberry (BRBY LN) reports its Q1 FY2027 trading update on 17 July 2026 (13 weeks to end-June 2026). Give me the latest sell-side setup: (1) consensus retail comparable store sales growth for Q1 FY27 — total AND by region (Greater China, Asia Pacific/Japan, EMEIA, Americas); (2) each recent broker preview note (June-July 2026) with its Q1 comp estimate, rating and price target; (3) the key investor debates into the print (Burberry Forward brand-reset execution, wholesale H1 growth, Greater China, tourism, gross margin, cost savings). Cite the most recent notes and quote specific numbers.
• Identify the top 7 prominent trends in the global and US chemical industry over the last 8 months as of July 12. Focus specifically on digital innovation (semiconductors, AI), smart operations, shifts in the Middle East landscape, restructuring, and asset or investment readjustments, as well as workforce dynamics. For each trend, explain its strategic and operational implications for the chemical sector.
• Find me commentaries for 2Q26 RevPAR trends and FY26 RevPAR trends (latest) for HLT MAR IHG AC WH CHH and H - one line for 2Q26, one line for FY26, so 14 lines total
• Evaluate the market opportunity and competitive landscape for RFID and eSIM technologies for Seshaasai Technology within the Indian market. Benchmark against global peers to estimate the potential size of the IoT opportunity. Analyze historical pricing and realization trends for these products over the past few years. Identify the key drivers of competitive advantage and switching costs for customers. Address the production and consumption dynamics (global vs. local), the current demand-supply balance, and unit economics including potential for economies or diseconomies of scale. Assess the industry's Porter's Five Forces and utilize economic frameworks such as market structures (oligopoly, monopolistic competition) and game theory to project how competitive dynamics may evolve.

Sample Rubric:

One rubric from our evaluation set (prod_july15th_v1_combined_0001), condensed for readability.

"Find me commentaries for 2Q26 RevPAR trends and FY26 RevPAR trends (latest) for HLT, MAR, IHG, AC, WH, CHH and H - one line for 2Q26, one line for FY26, so 14 lines total."

A hotel-sector research request spanning seven operators — Hilton (HLT), Marriott (MAR), IHG, Accor (AC), Wyndham (WH), Choice Hotels (CHH), and Hyatt (H) — evaluated against evidence available as of July 13, 2026. Fourteen data points are in scope: one 2Q26 line and one FY26 line per company.

The generated rubric contains 23 weighted criteria: 6 mandatory requirements (26 pts) covering completeness, timing, and sourcing; 14 optional quality signals (36 pts) that separate adequate answers from excellent ones; and 3 penalty checks with negative weights for characteristic failure modes. A perfect response scores 62 points.

Each criterion is tagged with the evidence axis it probes:

Mandatory criteria (26 pts)

Optional quality signals (36 pts)

Penalties

What passing vs. failing evidence looks like. Every criterion ships with worked examples. Two representative ones:

Passes: "IHG 2Q26: like-for-like RevPAR +4.1% [source: IHG 2Q26 earnings call]" — a specific figure, on a stated basis, tied to a dated primary source.

Fails: "HLT 2Q26: RevPAR was solid" — no figure, no source; misses c14 and triggers the c22 penalty.