← Back to Evals

Search tools · real agent work

Search API Eval v2

Which web search API helps a PR reviewer (Revera) and a coding agent (OpenCode)?

Run 5 providers + no-search baseline Agent / judge: Claude Opus 5.5

Brave, Exa, Parallel, Firecrawl and Tavily. Model settings: adaptive thinking, medium effort.

GitHub repo Full report

Small samples. Treat comparisons as directional. No blended score: each question is answered separately.

Revera · PR review

Search finds breaking-change bugs.

18/18bug-task runs

Search arms pooled vs. no search 0/6.

Recommended: keep Exa auto. Best rank-1 evidence on review-style lookups (11/12) and no strict false positives. Parallel fast is a close, faster, cheaper backup.

OpenCode · coding

No measurable gain from search on these 4 tasks.

4/4no search

= best search arm 4/4 in the quick pass.

Pick on retrieval and latency: Parallel fast (24/24 evidence@5, ~0.5 s per search).

Retrieval · held-out

Exa and Parallel tie at the top.

  • Exa24/24
  • Parallel24/24
  • Tavily22/24
  • Brave21/24
  • Firecrawl16/24

Only Firecrawl is clearly behind; the rest overlap within noise.

Swipe tables horizontally to compare providers.

Retrieval

Can it find the right evidence?

24 held-out questions, one search + read of the top result each. Evidence means the gold fact’s patterns matched provider-returned text, not merely that a familiar URL appeared.

How often is the evidence there?

Rank 1 vs. anywhere in the top 5. Whiskers: Wilson 95% interval.

Exa 23 / 24of 24
Parallel 20 / 24of 24
Tavily 21 / 22of 24
Brave 14 / 21of 24
Firecrawl 12 / 16of 24
Evidence retrieval counts and evidence@5 confidence intervals
ProviderQuestionsEvidence@1Evidence@5Wilson 95% interval
Exa24232486.2% to 100.0%
Parallel24202486.2% to 100.0%
Tavily24212274.2% to 97.7%
Brave24142169.0% to 95.7%
Firecrawl24121646.7% to 82.0%

Speed without losing evidence

Search only. Dot = p50; line extends to p90. Upper left is better.

50%75%100%0.5 s1.0 s2.0 s4.0 s8.0 sEvidence@5Search latency · log scaleExaParallelTavilyBraveFirecrawl
Search latency and evidence@5
ProviderEvidence@5p50 secondsp90 seconds
Exa24/244.4566.185
Parallel24/240.5310.624
Tavily22/243.3835.476
Brave21/240.5170.559
Firecrawl16/240.9421.763

Provider cost per query

Exa is metered; other prices are estimates. Model spend is separate.

Exa $0.0080 metered
Parallel $0.0020 estimate
Tavily $0.0177 estimate
Brave $0.0050 estimate
Firecrawl $0.0030 estimate

Where the difference comes from

Revera-like lookups separate providers. Traps overlap the two slices.

Evidence@1 / evidence@5 by slice; counts, not percentages
ProviderRevera-liken = 12OpenCode-liken = 12Trapsn = 6
Exa 11 / 1212 / 126 / 6
Parallel 8 / 1212 / 125 / 6
Tavily 9 / 1012 / 126 / 6
Brave 6 / 98 / 123 / 4
Firecrawl 4 / 58 / 112 / 3

Each cell: evidence@1 / evidence@5.

Inspect every question

Hover, focus or select a question to read it in full.

24 questions
1–5 = first evidence rank; stronger tint = earlier— = miss! = top-result read failed

Scroll for more providers; the question stays in view.

First evidence rank for every retrieval question and provider. A dash is a miss. An exclamation mark flags a read error.
QuestionExaParallelTavilyBraveFirecrawl
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:2 first evidence rank
Brave:3 first evidence rank
Firecrawl:—miss

r-pymongo-host-injectionsecurity advisory conditionsmedium

A multi-tenant Python service on pymongo 4.18.1 builds its MongoDB connection string by inserting a tenant-supplied hostname fragment into the host portion of the URI. Is it affected by the 2026 PyMongo connection-string host injection advisory, and which version fixes it?

Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:3 first evidence rank
Tavily:1 first evidence rank
Brave:—miss
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:—miss
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:3 first evidence rank
Tavily:—miss
Brave:1 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:2 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank!
Firecrawl:2 first evidence rank
Exa:4 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:—miss
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:5 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:2 first evidence rank
Tavily:—miss
Brave:1 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank
Firecrawl:4 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:4 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank!
Firecrawl:3 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:2 first evidence rank

A blinded LLM judge, shown the same search results and read page, agreed with the deterministic match (gold fact anywhere in that payload) on all 120 units: 115 supported, 5 insufficient, 0 contradicted.

No search call failed; 2 Brave reads failed in the shared reader. Failures stay in the denominators.

Revera

Does search improve a PR review?

2 real breaking-change bugs, 2 clean PRs. The grid includes confirmation reps; the table keeps the equal-sized quick pass separate.

Task-by-task outcomes

One marker per run, in recorded order: quick pass, then confirmation runs where available.

●Bug found / clean with no finding ○Bug missed ×Strict clean-control finding Small amber dot: an uncertain comment was also filed (not a strict finding)
revera task outcomes, including all confirmation runs
Task Exa Parallel Tavily Brave Firecrawl No search
rv-commander15-negatebugUpdate deployment CLI dependencies and add tags
Exa, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Parallel, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Tavily, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Brave, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Firecrawl, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
No search, quick pass: 0/1 bugs found; 0 strict false positives; 0 uncertain findingsNo search, confirmation 1: 0/1 bugs found; 0 strict false positives; 1 uncertain findingsNo search, confirmation 2: 0/1 bugs found; 0 strict false positives; 0 uncertain findings
rv-got16-http2-agentbugRefresh relay HTTP dependencies and poll timeout
Exa, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Parallel, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Tavily, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Brave, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Firecrawl, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
No search, quick pass: 0/1 bugs found; 0 strict false positives; 1 uncertain findingsNo search, confirmation 1: 0/1 bugs found; 0 strict false positives; 1 uncertain findingsNo search, confirmation 2: 0/1 bugs found; 0 strict false positives; 1 uncertain findings
rv-got16-migratecleanRefresh the snapshot monitor transport
Exa, quick pass: Clean control; 0 strict false positives; 0 uncertain findingsExa, confirmation 1: Clean control; 0 strict false positives; 0 uncertain findingsExa, confirmation 2: Clean control; 0 strict false positives; 1 uncertain findings
Parallel, quick pass: Clean control; 1 strict false positives; 1 uncertain findings
Tavily, quick pass: Clean control; 1 strict false positives; 0 uncertain findings
Brave, quick pass: Clean control; 1 strict false positives; 0 uncertain findings
Firecrawl, quick pass: Clean control; 0 strict false positives; 1 uncertain findingsFirecrawl, confirmation 1: Clean control; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 2: Clean control; 1 strict false positives; 0 uncertain findings
No search, quick pass: Clean control; 0 strict false positives; 0 uncertain findingsNo search, confirmation 1: Clean control; 0 strict false positives; 0 uncertain findingsNo search, confirmation 2: Clean control; 0 strict false positives; 0 uncertain findings
rv-ky2-migratecleanUpdate queue-control HTTP client
Exa, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Parallel, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Tavily, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Brave, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Firecrawl, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
No search, quick pass: Clean control; 0 strict false positives; 0 uncertain findings

Quick pass · equal task counts

First run only. Costs and wall time are means per review; model prices are estimates.

revera quick pass results and mean per-run costs
Provider / profileBugs foundStrict clean-
control findings		UncertainModel $Search $Wall s
Exa auto2/200$0.0726$0.0075metered40.0
Parallel fast2/211$0.1091$0.0018estimate49.1
Tavily advanced2/210$0.1108$0.0220estimate52.4
Brave web2/210$0.0947$0.0063estimate45.7
Firecrawl search2/201$0.1177$0.0040estimate49.4
No search baseline0/201$0.0391$0.0000not used39.6

Every strict false positive was the same comment on the clean rv-got16-migrate PR (that agent: {http2: false} disables session pooling). The premise is correct per the Got 16 docs; adjudicated as a reasonable non-blocking comment, not a hallucination.

No-search reviewers missed both bugs; on Commander 15 the no-search reviewer asserted the old behavior.

An Exa review used about 1.9x the no-search model cost, with similar wall time.

OpenCode

Does search help the repair pass?

4 dependency-upgrade repairs, scored by hidden tests run offline. Confirmation reps focus on oc-ai7.

Task-by-task outcomes

One marker per run, in recorded order: quick pass, then confirmation runs where available.

●All tests pass ○Task failed
opencode task outcomes, including all confirmation runs
Task Exa Parallel Tavily Brave Firecrawl No search
oc-ai7Restore the catalog assistant on the updated AI SDK
Exa, quick pass: All tests pass; 4/4 testsExa, confirmation 1: All tests pass; 4/4 testsExa, confirmation 2: Task failed; 3/4 tests
Parallel, quick pass: Task failed; 3/4 tests
Tavily, quick pass: Task failed; 3/4 tests
Brave, quick pass: Task failed; 3/4 tests
Firecrawl, quick pass: Task failed; 3/4 testsFirecrawl, confirmation 1: All tests pass; 4/4 testsFirecrawl, confirmation 2: Task failed; 3/4 tests
No search, quick pass: All tests pass; 4/4 testsNo search, confirmation 1: Task failed; 3/4 testsNo search, confirmation 2: All tests pass; 4/4 tests
oc-execa10Restore the release command runner after a dependency update
Exa, quick pass: All tests pass; 3/3 tests
Parallel, quick pass: All tests pass; 3/3 tests
Tavily, quick pass: All tests pass; 3/3 tests
Brave, quick pass: All tests pass; 3/3 tests
Firecrawl, quick pass: All tests pass; 3/3 tests
No search, quick pass: All tests pass; 3/3 tests
oc-ky2Repair the inventory API client after its HTTP dependency update
Exa, quick pass: All tests pass; 4/4 tests
Parallel, quick pass: All tests pass; 4/4 tests
Tavily, quick pass: All tests pass; 4/4 tests
Brave, quick pass: All tests pass; 4/4 tests
Firecrawl, quick pass: All tests pass; 4/4 tests
No search, quick pass: All tests pass; 4/4 tests
oc-ts7-configBring the warehouse typecheck configuration forward
Exa, quick pass: All tests pass; 4/4 tests
Parallel, quick pass: All tests pass; 4/4 tests
Tavily, quick pass: All tests pass; 4/4 tests
Brave, quick pass: All tests pass; 4/4 tests
Firecrawl, quick pass: All tests pass; 4/4 tests
No search, quick pass: All tests pass; 4/4 tests

Quick pass · equal task counts

First run only. Costs and wall time are means per task; model prices are estimates.

opencode quick pass results and mean per-run costs
Provider / profileAll passModel $Search $Wall s
Exa auto4/4$0.1049$0.0105metered69.4
Parallel fast3/4$0.1053$0.0013estimate59.8
Tavily advanced3/4$0.1046$0.0160estimate60.7
Brave web3/4$0.0890$0.0063estimate59.0
Firecrawl search3/4$0.1125$0.0027estimate55.6
No search baseline4/4$0.0590$0.0000not used54.5

Every lost run was on oc-ai7, with the same tsc error: the code used a non-generic ToolExecutionOptions, but it is generic in the installed ai@7.0.143 (in the quick pass, the web arms had copied it from docs). In repeat runs oc-ai7 was noisy for every arm (Exa 2/3, Firecrawl 1/3, none 2/3).

Search arms used about 1.8x the model spend of no search, without adding passes. One untested hypothesis: the model already knows these releases.

Recommendations

Keep search useful. Keep it optional.

Revera: keep Exa auto

Exa auto
  • Keep web search opt-in. Keep Exa REST type:auto with highlights + /contents reads.
  • Raise highlight length to 2,000–4,000 chars. Cutting to 1,000 drops Exa from 24/24 to 22/24 evidence@5.
  • Raise the page read cap with query-aware trimming: 6 of 24 Exa reads had the first evidence past 6,000 chars.
  • Parallel fast + /v1/extract as optional failover.

Highlight and read-depth figures: source: INTEGRATION_RECOMMENDATIONS.md

OpenCode: Parallel fast

Parallel fast
  • Parallel fast for interactive search: 24/24 evidence@5 with ~0.5 s search p50.
  • Exa auto as the equally accurate but slower (~4.5 s) alternative.
  • Keep search an optional tool, not run on every task.
  • Update 2026-10-24: Exa is now about as fast; see post-hoc additions.

Not tested

Brave LLM Context (brave:llm-context) was blocked: OPTION_NOT_IN_PLAN on the key's Free AI plan. Listed as blocked, not as a loss. Optional reranker ablation skipped.

Post-hoc additions,

What changed after the frozen evaluation?

The protocol allowed one baseline plus at most one alternative per provider, missing Firecrawl’s Developer Index and Exa’s objective field. These additions are reported separately; the 2026-10-19 results above are unchanged. One rep each, no LLM judge.

Retrieval: same-day controls

Same 24 holdout cases, with all five original profiles rerun as controls. Compare new arms with these controls, not the earlier table. USD/query includes search + read.

Swipe the table horizontally to compare all columns.

ProfileEvidence@1Evidence@5Evidence in readSearch p50 msUSD/query
exa:auto-dynamic new
24/2424/2424/24654$0.0080 metered
exa:auto control
23/2424/2424/24296*$0.0080 metered
exa:lite new
23/2424/2424/24439$0.0050 metered
exa:auto-objective-uncached new
23/2424/2424/24462$0.0080 metered
exa:auto-uncached control
23/2424/2424/24468$0.0080 metered
parallel:fast control
21/2424/2421/24546$0.0020 estimate
brave:web control
14/2421/2420/24494$0.0050 estimate
brave:web-nospell new
14/2421/2420/24466$0.0050 estimate
tavily:advanced control
20/2420/2421/242674$0.0177 estimate
firecrawl:developer new
10/2419/2416/241086$0.0030 estimate
firecrawl:search control
13/2416/2418/24781$0.0030 estimate

* exa:auto is cache-flattered: about half of searches were Exa cache hits; use exa:auto-uncached for latency.

Agents: one run per arm

4 Revera tasks and 4 OpenCode tasks. Model USD means are list-price estimates per task.

Swipe the table horizontally to compare all columns.

ArmRevera bugs found/availableStrict FPsUncertainOpenCode all-pass/tasksRevera model USD meanOpenCode model USD mean
exa:auto-objective2/2003/4$0.0746$0.1053
firecrawl:developer2/2014/4$0.1068$0.1020

Both arms found both bugs. Firecrawl’s uncertain item was the same defensible pooling comment described above. Exa missed oc-ai7, which failed in some runs for every arm on 2026-10-19, including no search; the failing test was not captured, so its cause is unchecked. This track still cannot rank providers.

What the additions show

  • Firecrawl Developer Index improved evidence@5 to 19/24 vs 16/24 for general search. It gained on library and migration questions; both Firecrawl modes found evidence on only 2 of 6 security-advisory cases. Evidence usually came from GitHub issues or PRs, not the canonical page (url@5: 5/24).
  • Exa’s objective field made no measurable difference in the cache-controlled pair: the evidence rank changed in none of the 24 cases.
  • Exa dynamic highlights found evidence at rank 1 on 24/24 with about 9% fewer visible tokens. That is a one-case improvement, within noise.
  • Exa lite matched auto on evidence at about 40% lower cost ($0.005 vs $0.008 per query).
  • Cache-free Exa auto is now 0.47 s search p50, down from 4.5 s on 2026-10-19. That is about as fast as Parallel fast (0.55 s).
  • Brave spellcheck:false returned identical results on all 24 cases.
  • Tavily’s control drifted from 22 to 20/24 evidence@5. Indexes change over time.

Two problems found and fixed

  • Exa’s result cache ignores objective. The first comparison was voided and rerun with cache-busting settings; use the uncached pair above.
  • Fixture dependencies were missing, so the first OpenCode rerun was void. The harness now refuses to run in that state; the tasks were rerun.

Both fixes are recorded in the protocol amendments.

Bottom line

  • Revera: unchanged. Keep Exa auto.
  • OpenCode: Exa and Parallel are now about equal on speed. Exa is slightly better at rank 1 (23 vs 21); Parallel is about 4x cheaper. Exa lite is a middle option. These data do not settle it.
  • Firecrawl users: Developer Index for code questions. Neither Firecrawl mode handled security advisories well. It still trails Exa and Parallel on this set.

Read the full post-hoc analysis

Read the fine print

Method & limitations

Provider fee estimate

$1.72

List-price estimate; Exa portion metered

Agent model estimate

$7.47

List prices, not a proxy bill

Judge model estimate

$8.86

Model total: $16.33

Same tool contract, frozen profiles

One tool contract for all providers: web_search returns at most 5 results; web_read reads only a returned result. Search responses have a 4,000-token cap; reads have a 6,000-token cap.

Deterministic scoring against gold fact patterns. Failures stay in denominators. Profiles were frozen from a pilot, not tuned on the held-out results.

  • exa:auto
  • parallel:fast
  • tavily:advanced
  • brave:web
  • firecrawl:search

Read the protocol

What this evaluation cannot establish

Small samples; only two Revera bugs, both version-upgrade bugs in JS libraries. Do not read the ordering as a universal provider ranking. Provider indexes change over time.

Model costs are list-price estimates, not proxy bills. Anthropic list-price estimate; proxy billing not visible.

Read all limitations

All results come from the frozen evaluation export; the two recommendation ablations are linked to their source above. Inspect the code and artifacts.