Every few months a model shows up that resets the price floor, and the group chat fills with the same message: we should switch. This quarter it was GLM-5.2. Open weights under an MIT license, 753 billion parameters, and a list price of $1.40 per million input tokens and $4.40 per million output — against Gemini 3.5 Flash at $1.50 / $9.00. On output, where most production spend actually lands, that is roughly half the price. Twenty-odd hosts now serve the same weights, several below Z.ai's own sticker. The token bill says migrate.
We build production systems for a living, so before moving anything we did the boring thing: we built the same three workloads twice — once on each model — and measured what they cost to finish, not what they cost per token. Those are different numbers, and the gap between them is where switching decisions go wrong.
The metric that matters is cost per completed task
Per-token price is an input, not an outcome. What a workload actually costs is a function of how many tokens the model burns to reach an acceptable answer, how long you (or your user, or your engineer) wait for it, and whether it can do the job at all without a second component bolted on. A model that is half the price per token but writes twice as much, or thinks for three times as long, or can't read the file you handed it, is not half the cost. It may not be cheaper at all.
This isn't a novel observation — it's the honest reading of the public benchmark numbers that people like Mohamed Elkholy have been making: cheap tokens, expensive habits. But a benchmark leaderboard reasons about it in the abstract. We wanted the number for the two models a cost-sensitive team is realistically choosing between, on the work they actually run. So we picked three workloads — a chatbot, an agentic coding build, and document extraction — and ran each end to end on both models through the same harness, with the same prompts, scoring each on total token spend, wall-clock to completion, and whether the output met acceptance.
Here is what came back.
Workload 1 — Chatbots: same bill, 3x the wait
We ran the same multi-turn conversation through both models and measured the full response, not just the first token.
The output-price gap is real on paper. It did not reach the invoice. Across the conversation the two models landed on effectively the same total spend — GLM's cheaper per-token rate offset by the tokens it needs to reach the same answer. The reasoning trace is billed at the output rate on both models, and a model that thinks longer produces more billable tokens; independent measurements already flag Gemini 3.5 Flash as verbose (around 75 million output tokens to complete the Artificial Analysis suite), and GLM's own "Max" thinking mode runs near 85k output tokens per task before you dial it back. Verbosity is a cost lever on both sides, and when you let each model answer the same question properly, the bills converge.
What did not converge was latency. GLM-5.2 was 3.01x slower to complete a reply. Time-to-first-token was competitive — the user sees something quickly — but the wait for the finished answer was three times longer. For a customer-facing assistant that is the whole experience. You are paying the same money to make the person on the other end wait three times as long.
For chatbots, speed is the cost, and GLM loses it.
Workload 2 — Agentic coding: the engineer becomes the bill
Chat is a light test. The harder one is agentic engineering work, so we built a real application on each model: a Workforce Operations Service — an HR and time-tracking system — constructed end to end, twice, once on GLM-5.2 and once on Gemini 3.5 Flash. Every ticket in the backlog went to the model as a single task through the agentic harness, and we measured acceptance, total token cost, and the wall-clock time an engineer would actually sit through.
Both models shipped working software of comparable quality. The token bill again favored GLM, modestly. And again the clock told the real story: GLM was 4.4x slower to deliver the same result.
On a chatbot, "slower" costs you a user's patience. On an agentic build, it costs you your most expensive resource. An engineer waiting 4.4x longer for each task to land is not saving money because the API line item dropped a few dollars — the developer-time meter is running the entire time, and it runs at a rate that dwarfs the token savings. This is the exact trap the leaderboards can't show you: the cheaper model produced a more expensive delivery once you count the human standing next to it.
For coding, developer time is the cost, and it erases the saving.
Workload 3 — Documents: a capability gap, not a price gap
The third workload wasn't close, because it wasn't the same job.
GLM-5.2 cannot read a document. It takes text. Hand it a PDF or a scanned form and you need a separate OCR step in front of it to turn the file into text first. Gemini 3.5 Flash reads the file natively — text, image, PDF — as part of the same call.
On clean text that has already been extracted, the two are roughly a wash; GLM is even marginally leaner on the token bill for that narrow case. But "already been extracted" is doing enormous work in that sentence. Real document processing is rarely clean text. The moment your input is an actual file, GLM's price advantage funds a second pipeline you now have to build, run, monitor, and pay for — an OCR component with its own failure modes, its own latency, its own bill, and its own on-call burden. Gemini does it in one hop.
That extra box is not a rounding error. It is a standing piece of infrastructure. For document work, the OCR step is the cost, and it shows up on the org chart before it shows up on the invoice.
Why the cheap model kept losing
None of this is GLM-5.2 being a bad model. It is a genuinely strong open-weight system, and on a batch job with no latency pressure and pre-cleaned inputs, its economics are excellent. The point is narrower and more useful: per-token price is one term in a longer equation, and it's usually not the dominant one.
The dominant terms are the ones the sticker doesn't print:
- Verbosity — you pay per output token, and reasoning traces are output tokens. Half the rate on twice the tokens is the same bill.
- Latency — three to four times slower is either a worse product or an idle engineer. Neither is free, and neither appears on the API invoice.
- Missing modality — a capability the model lacks becomes a component you own, with its own cost and failure surface.
- Retries — in our runs, a cheaper model that needs a second attempt to pass acceptance can cost more than a pricier one that lands first time.
Sum those and you get the headline our own numbers kept producing: GLM won the token bill in all three workloads. The bill was never the cost.
What we actually do about it
The wrong conclusion is "Gemini wins, standardize on Gemini." That's just the same sticker-price mistake pointed the other way. The right conclusion is that the answer is per workload, and it changes as models and prices move — GLM-5.2's cheapest input price already fell ~48% in its first 90 days as hosts competed.
So we don't hard-wire a model into an application. We route. Our delivery stack puts a thin routing and telemetry layer between the agent framework and the model API: it dispatches each workload to the model that's cheapest to complete that workload, measures actual cost-per-completed-task and latency in production rather than trusting a launch-day benchmark, and lets the routing table change without touching application code. GLM-5.2 is a great answer for an overnight batch extraction over clean text. It is the wrong answer for a latency-sensitive assistant. You should be able to use both, and know — from your own telemetry, not a blog post — which is which.
That's the real lesson of the exercise. The question was never "which model is cheaper." It was "cheaper at what, and who's waiting."
Which model for which workload
- Customer-facing chat, agentic coding, anything a human waits on: Gemini 3.5 Flash. The latency gap outweighs the token savings.
- Document and file processing: Gemini 3.5 Flash, unless your inputs are already clean text — native multimodality beats bolting on OCR.
- Latency-tolerant batch work over pre-cleaned text: GLM-5.2, where its per-token economics and open weights genuinely shine.
- Everything, over time: measure it yourself, and route.