Gorgias ranks itself first in ecommerce AI support, but second in pre-sale, and the company has put both numbers on a public page that anyone can rerun.

The benchmark, released September 25, 2026, covers 8,356 live conversations across 18 named vendors and 212 live storefronts. Results update weekly. The full test harness sits on GitHub. Jason Lemkin, founder of SaaStr, framed the move on X: "Everyone should publish the deepest, most direct competitive evals they can. Will there always be some bias? Yes."

What Gorgias measured

Gorgias earns roughly $100M ARR, with about 80% of that revenue tied to AI support for ecommerce brands. The eval hits every major stage a shopper moves through: a simple first question, cart, shipping, returns, pre-sale discovery.

Each vendor faces the same ten-turn conversation. Gorgias strips vendor and storefront names before its AI judge scores the response. A second adversarial reviewer must agree with the first score at least 90% of the time. The scoring rubric is versioned in the public repo, so no one can quietly change weights to hide a gap.

The setup matters because AI agents drift. A CRM returns the same result on every click. An LLM-backed agent can answer well on Monday and poorly on Thursday, and can degrade after a model update with nobody on the vendor side noticing.

Where Gorgias wins, and where it does not

In support, Gorgias takes the top composite score. In automation rate, the metric support buyers ask about most, Yuma leads on Gorgias's own page. The company published that lead anyway.

In pre-sale, Envive beats Gorgias 72 to 65 on the composite. Gorgias posts the best answer quality in the lane at 76, but averages 18.4 seconds per answer against Envive's 7.9. Twenty-eight percent of Gorgias's shopping answers stretch past 20 seconds, per the report's own latency breakdown.

The weighting choice cost Gorgias first place. Speed carries 25% of the pre-sale score versus 10% in support. Recalculate pre-sale with the support weights and Gorgias finishes first at 74.3. Gorgias kept the tougher weighting because shoppers leave when answers are slow.

How does this differ from a normal vendor benchmark?

Most benchmarks compare what vendors claim their products do. They draw on surveys, demo scripts, and vendor-supplied answers, refreshed once or twice a year. An eval tests what the product actually does.

In Gorgias's case, every one of the 8,356 conversations resolves or does not resolve. Every answer gets graded against a written rubric. The same test runs against every vendor. The cycle repeats weekly.

Lemkin argues this is the new floor for AI vendors. "Buyers can now test AI agents side by side in an afternoon," he wrote. "Agents are starting to build the shortlists."

The frequency is the real differentiator. One snapshot ages in a quarter; a weekly ledger stays current.

Why publishing your own losses earns trust

Buyers already discount vendor charts. Every support leader has seen a dozen decks where the vendor wins, and most ignore all of them. When a vendor prints a category where a competitor leads, buyers give more weight to the categories where it leads.

The team inside Gorgias sees the gap each week. Envive's 7.9 seconds and Yuma's resolution rate now sit in public, alongside Gorgias's own numbers, updated every Monday. The rubric is versioned. The numbers cannot quietly move.

The same dynamic applies to AI-driven shortlisting. Claude, ChatGPT, and procurement agents can read a versioned rubric, cite public weights, and check open code. They cannot do much with a gated PDF. One caveat: the live Gorgias report currently blocks automated readers in its robots.txt, even though the GitHub repo is open. Vendors that fix that step will get cited more.

Public losses make the public wins believable.

What should vendors copy from this template?

Seven steps sit in the Gorgias playbook:

• Test live storefronts, not sandboxes.
• Name every competitor.
• Blind the judge before scoring.
• Open-source the harness so rivals can rerun it.
• Weight for the end customer, even when it costs the top spot.
• Publish the weekly trend, not a single snapshot.
• State the bias up front. Gorgias labels its own project "competitive intelligence" and asks readers to treat cross-vendor numbers as directional.

The competitive eval has crossed from optional to standard. Gorgias just published the first one rigorous enough to copy.