Evaluating Jev for Product Search: From Benchmark Claims to Real-World Trade-offs

At Octogen, we offer a product search API for agents and developers. Product search is full of nuances that are easy for a model to blur. A raw hem does not establish that denim is unwashed. Cotton does not establish an Oxford weave. A description of a zipper does not necessarily establish that it is on the side of a boot.

At Octogen, these nuances matter because search needs to retrieve products that satisfy what a shopper actually asked for. Semantically similar descriptions aren’t enough. Our search system works with both source catalog data and generated enrichment that adds structured information about products. That enrichment can make the catalog much more useful for search, but it is still derived evidence: when it conflicts with the source catalog, we treat the source data as authoritative.

We wanted to test one piece of that problem: given an existing set of search candidates, could a model put the best matches near the top?

We learned a few things along the way.

First, benchmark gains are workload-specific. TypeSafe’s published efficiency results were one reason we wanted to evaluate Jev, but the latency and cost differences we observed on product-search reranking were materially smaller. Our task, batching strategy, model settings, and serving harness differed from theirs, so neither benchmark is necessarily wrong. We needed to measure the trade-offs on the workload we actually intended to serve.

Second, the comparison set matters. Our first LLM baseline, GPT-4.1 mini, made Jev look substantially better on relevance. When we added stronger general-purpose models, that quality gap nearly disappeared: GPT-5.6 Terra and Luna with medium reasoning came within one strong-match position of Jev batch10, with Luna matching it in aggregate. Those models were considerably slower and more expensive in our setup, which changed the question from “which model ranks best?” to “which quality/latency/cost trade-off makes sense for production?”

Third, eval design matters as much as the models being compared. We used human relevance judgments as the primary outcome, showed the reviewer the same product evidence available to the models, shuffled the results, and hid model identity, scores, and ranking information. Those choices don’t eliminate subjectivity as we still had one reviewer and a small, repeatedly examined query set, but they reduce several avoidable sources of bias.

Results at a glance

Figures 1–3. Twelve frozen queries and 120 top-ten positions. Latency and provider cost cover fifty-product scoring windows. Historical runs and pricing assumptions describe this evaluation; they are not a controlled reproduction of TypeSafe’s benchmark.

With those principles established, here’s the product-search problem we were trying to evaluate.

The decision we asked the models to make

Retrieval supplied fifty candidates for each of twelve apparel and accessories queries. The set was deliberately small and targeted, with queries spanning different kinds of product constraints. For example, stone-washed raw-hem jeans, brown suede boots with a side zipper, sleeveless poplin midi dresses, a leather crossbody bag with a zip closure and adjustable strap, and block heel sandals with adjustable straps. We froze those lists and their compact product text. The scorer’s job was to assign relevance to the existing candidates and reorder them.

Reranking could promote a good product already present in the window. It could not recover a missing candidate or repair bad catalog evidence upstream.

We used a four-level relevance grading policy:

• 0: Wrong product type or an explicit contradiction of a required feature.

• 1: Related product type, with too little evidence for most defining features.

• 2: Relevant partial match, with an important requested feature still unverified.

• 3: Strong match, with evidence supporting all defining requested features.

Raw catalog facts took precedence over conflicting generated enrichment. Neither the models nor the human reviewer received product images, although the text included fields derived from images and raw catalog data. We evaluated the supplied evidence rather than independently verifying the physical product.

Why we tried Jev

TypeSafe describes Jev as a model for typed, probabilistic decisions. That interface made relevance scoring an interesting application: define a small set of relevance outcomes, score each candidate, and use those probabilities to rank the results. TypeSafe’s launch report describes the model and its intended uses.

We evaluated jev-1.13.0 with one product per request and with ten products per request, which we call batch10. For comparison, we evaluated GPT-4.1 mini, then GPT-5.6 Terra and GPT-5.6 Luna with medium reasoning. Each LLM configuration scored products in batches of ten.

Terra and Luna received the same compact product text and relevance rubric as the earlier configurations. We did not run a prompt search or a tournament across reasoning settings.

The interfaces differed. Jev produced probability-weighted relevance scores; the LLMs returned direct numeric scores under a strict JSON schema. Equal scores retained the original retrieval order. We were comparing complete scoring configurations on our search task, rather than reproducing TypeSafe’s workflow evaluation, which uses probabilistic decisions and model-consensus reference answers.

Our first baseline told a convincing story

We graded the top-ten results by hand. One human reviewer evaluated shuffled query/product pairs with model scores and ranking information hidden. Across the evaluation and follow-ups, 322 unique query/product judgments covered every top-ten result in the six configurations. Shared query/product pairs reused the same judgment; no top-ten labels remain missing.

Each configuration contains 120 result positions: ten results across twelve queries.

The original comparison looked straightforward. Jev batch10 produced 118 strong matches. GPT-4.1 mini produced 104. A fourteen-position difference out of 120 made a compelling case for Jev on this task.

It was also an incomplete story.

Several configurations reached similar relevance

GPT-4.1 mini was a useful control. Treating it as a stand-in for what general-purpose models could do would have been a mistake.

The quality panel in Results at a glance compares all six configurations on the same twelve frozen queries.

With the same frozen candidates and relevance task, Terra produced 117 strong matches and Luna produced 118. Luna closed the aggregate gap with Jev batch10. Jev’s single-product configuration remained one position higher at 119.

Relative to retrieval, Jev batch10 added 25 strong-match positions out of 120. The grade ≥2 column counts partial matches too, so 120 relevant results does not mean every requested feature was supported.

Aggregate equality did not mean identical rankings. Against Jev batch10, Luna won one query on strong-match count, lost one and tied ten. It did better on sandals and worse on side-zipper boots. Terra won one, lost two and tied nine, with losses on raw-hem jeans and poplin dresses.

We also kept the easy cases. The originally reviewed unions for linen trousers and crossbody bags were all strong matches. Removing those two queries as a secondary diagnostic left counts of 73, 99, 98, 84, 97 and 98 out of 100, in the table’s row order. The stronger controls remained close to Jev. All twelve queries stay in the primary result.

Our evidence no longer supported a broad quality advantage over the stronger general-purpose LLM configurations we tested. It supported a much narrower gap between these configurations, which shifted our attention to the engineering trade-offs.

Matching quality wasn’t enough

For Terra and Luna, scoring a fifty-product window required five requests, with at most four batches running concurrently. Windows ran sequentially, and model order alternated across queries.

These are scorer-window measurements, not end-to-end search latency. Neither medium-reasoning configuration met our preliminary one-second target, although that threshold was applied retrospectively; the actual timeout was 120 seconds.

Jev and GPT-4.1 mini in the Results at a glance panel are earlier saved runs, so the cross-provider ratios are historical context rather than a controlled speed benchmark: Luna’s 8.830-second median was about 14.3× the earlier Jev batch10 median of 0.616 seconds, and Terra’s 8.200 seconds was about 13.3×.

The Jev results also show how serving configuration affected the measurement. Under the same four-concurrent-request, ten-starts-per-second limit, the saved single-product run had a median of 5.138 seconds versus 0.616 seconds for batch10: one required fifty requests per window and the other five. Because these measurements include scheduling and pacing, the difference is not a pure inference-speed comparison. The earlier GPT-4.1 mini run measured 1.629 seconds.

A scorer can produce excellent relevance and still be difficult to place directly in a latency-sensitive search path. Twelve observations per configuration are enough to make that concern concrete; they do not establish production tail latency or reliability.

Cost changes the engineering question too

We estimated provider cost from recorded usage. For a fifty-product search, Terra’s estimate was $0.05779, or $0.06943 if every input token incurred the cache-write premium. Luna’s corresponding estimates were $0.00715–$0.00831.

The cost panel in Results at a glance includes every scoring configuration. The saved GPT-4.1 mini estimate was $0.00973 per fifty-product window; Jev batch10 was $0.00177 and Jev single-product $0.00225. These are usage-derived provider estimates at the recorded historical rates, not invoices. Hatched extensions for Terra and Luna represent the all-input cache-write premium assumption, not measured variation. Retrieval cost is outside this comparison.

Using matching accounting assumptions, Terra’s estimated provider cost was about 8.1–8.4× Luna’s. We checked the published rates and cache-write rules on Sep 23, 2026 against the official Terra and Luna documentation.

The full Terra/Luna experiment, including development calls, was estimated at $0.84379–$1.00958. Usage did not separately identify cache writes. Reasoning tokens were already included in completion-token accounting and were not charged twice.

These estimates cover provider usage. They do not establish the cost of running the surrounding search service or the value of improved results. Once multiple models produce similar offline relevance, those operational questions become a larger part of the decision.

Published speedups are workload-specific

TypeSafe’s launch report attributes its headline 193.6× faster and 444.6× cheaper figures to its workflow evaluations. It also cautions that these gains may be near the upper end of real-world improvements. Its LLM wrapper produces structured probabilistic decisions, which TypeSafe notes can be slower and more expensive than returning decisions without probabilities.

Our saved product-search results showed smaller differences. The historical median-latency ratios above were 13.3× for Terra and 14.3× for Luna relative to Jev batch10. Recorded Jev batch10 input usage, priced at the Sep 20, 2026 rate with no output charge, put its estimated provider cost at $0.00177 per fifty-product window. Using unrounded estimates, Terra’s cost was about 32.6–39.2× that historical Jev estimate, and Luna’s about 4.0–4.7×, under the base and all-input cache-write assumptions described above. These are usage-derived cost estimates, not invoices; the runs were on different dates, so neither the timing nor cost ratios constitute a controlled reproduction of TypeSafe’s benchmark.

We were reranking product-search candidates rather than executing TypeSafe’s workflows. Our LLM requests batched ten products and returned direct numeric relevance scores under a strict JSON schema. We used medium reasoning for Terra and Luna, our own concurrency and scheduling limits, and measured complete fifty-product scoring windows. Request shape and serving assumptions change what is being compared.

Published benchmark results can tell us whether a system is worth evaluating. Latency and cost still need to be measured on the task and serving configuration we actually intend to run.

Reranking fewer candidates was faster, but hurt quality

Model choice wasn’t the only way to reduce latency. We also asked whether Jev could rerank fewer products. Instead of scoring all fifty retrieved candidates, we reranked only the first twenty from the same frozen candidate lists.

The smaller window was substantially faster: median scoring time fell from 0.639 seconds to 0.339 seconds, a reduction of about 47%. But relevance also fell. The saved single-product configuration produced 115 strong matches with twenty candidates, compared with 119 with fifty. Three fresh batch10 repetitions showed the same direction.

That failed our predefined no-quality-loss requirement, so we kept the fifty-candidate window for the next engineering phase.

There is an important caveat: this experiment truncated the same frozen fifty-candidate lists rather than running retrieval again with a twenty-product limit. In one query, for example, the smaller window happened to exclude a difficult partial-match sandal, which improved the top-ten score. Removing a difficult candidate is not the same as ranking it correctly.

Batch composition changed which products reached the top ten

Batching made Jev much faster to run, but it also introduced another variable: which products were scored together.

Even when we repeated the same batch10 configuration three times with the same fifty candidates and the same batch layout, the resulting top tens were not identical. On a typical query, nine of the ten products overlapped between runs.

We then deliberately changed the batch composition by reversing product order within batches or shuffling products across batches. The effect became larger: the resulting top tens shared a median of only six of ten products with the saved original run.

Surprisingly, that instability barely changed our aggregate relevance metric. Both altered batching strategies still produced 118 strong matches out of 120 after human review. In other words, different products could reach the top ten while looking almost identical under our headline quality score.

Batching also showed a small quality trade-off relative to scoring products individually. The three fresh batch10 runs produced 117, 117 and 118 strong matches, compared with 119 for the saved single-product configuration.

For us, that means batching is part of the model configuration, not just a serving optimization. If we choose batch10 for its latency advantage, we also need to accept and evaluate the ranking behavior that comes with it rather than selecting whichever batch arrangement happens to score best after the fact.

What we think this experiment shows

On these frozen queries, candidates, product text and human judgments, reranking substantially improved the top ten over retrieval. Jev performed well, but stronger general-purpose LLM configurations narrowed what initially looked like a large quality advantage. At the same time, the medium-reasoning configurations we tested did not fit our preliminary latency target.

The result is not a general winner between model families. It is a more useful engineering picture: several configurations with similar strong-match counts but meaningfully different latency, cost and serving characteristics. That is enough to keep evaluating Jev as a component of the search path.

More broadly, the experiment reinforced why these comparisons need to be run on the workload and configuration you actually intend to serve. The choice of comparison models, batching strategy, candidate window and evaluation setup all affected what we could reasonably conclude.

What it doesn’t show

These twelve targeted queries have been examined repeatedly. They are neither a fresh holdout nor representative production traffic. The batch repetitions are correlated measurements of the same queries, not additional independent queries.

One reviewer supplied the judgments. Related product variants reduce independence, and some labels preserve taxonomy approximations. One case accepted “Fitted” as an approximation for a requested slim fit; we kept the judgment and its caveat rather than silently rewriting the evaluation.

The 322 labels cover every required top-ten result, but not all 600 query/candidate pairs. They cannot support full-pool NDCG, which requires an ideal ranking over that pool. Strong-match counts also do not describe every aspect of ordering within the top ten.

Stable returned Terra/Luna aliases do not establish immutable model snapshots. We tested medium reasoning only; lower-reasoning settings or other prompts would require a separately frozen experiment. Historical timings and usage estimates do not establish live latency, capacity or return on investment.

These search results do not validate category prediction, attribute extraction or section routing. Those decisions need their own evaluations.

What we’re testing next

At our Sep 23, 2026 checkpoint, shadow-integration code was deployed with execution disabled. Controlled replay was still to do, and Jev had not been activated for public ranking.

The next phase asks what happens around the scorer: whether shadow execution changes responses or adds overhead; how realistic concurrency, quotas and cancellations affect cost; whether retained candidate windows preserve pagination without duplicates or omissions; and what users receive when scoring fails or times out. Fresh relevance data and a separate serving decision are also needed before making claims about customer benefit.

Those questions deserve a second post. For this one, the most useful lesson is that benchmark results were only a starting point: the engineering picture became clearer once we measured quality, latency and cost on our own workload against multiple realistic alternatives.