LLM Provider Benchmark Methodology

Pricing units

Text input and output prices are shown in US dollars per 1 million tokens. The displayed price is the current price published by OpenRouter; any discount appears separately. Image and video pages keep OpenRouter’s exact billing unit. Other model types are ranked only when their billing units can be compared fairly.

Speed and reliability

Time to first token is the wait before a response starts. Output speed is how many tokens are generated each second. We use OpenRouter’s recent median—the middle measured result—and match it to the exact provider route. Times are displayed in seconds. If OpenRouter has not published enough recent measurements, the table says “Insufficient recent data” and leaves that provider out of speed rankings without hiding it.

Cost and response-time estimates

The comparison charts use 1,000 input and 500 output tokens. Estimated cost equals input tokens × input price + output tokens × output price. Estimated response time equals time to first token + output tokens ÷ output speed. “Cheapest” marks the lowest estimated cost, and “Fastest” marks the shortest estimated response time. “Best cost/speed trade-off” is one provider: the point closest to the ideal lower-left corner after cost and time are each scaled from best to worst. If values tie, the first endpoint in the published list wins the badge.

LLM provider leaderboards

For the overall ranking, each OpenRouter provider can contribute up to five of its popular text models. The weekly view uses OpenRouter’s globally popular model set. Candidate selection determines which models are inspected; it does not let a provider earn a higher rank merely by offering smaller or naturally faster models.

Every complete speed observation is compared only with other providers running the exact same model. For response time and first-token latency, the factor is the same-model median divided by the provider measurement. For throughput, it is the provider measurement divided by the same-model median. A factor above 1.00 means faster than the same-model baseline. The provider rating uses the lower quartile—the slower-performing quarter—of its model factors, and at least three shared measured models are required. Samples below ten shared models are moderated toward 1.00; ten or more receive full weight. This rewards consistency while expressing less confidence in narrow catalogs.

The table also publishes raw catalog medians: estimated time for a 500-token answer, wait until output begins, and generated tokens per second. The Rating basis field says whether a row uses shared-model comparisons or only the provider’s own-model catalog. Raw catalog speed is useful for practical expectations, but it still depends on model mix and is not directly comparable with a model-adjusted rating.

The homepage cards are separate from the model-adjusted leaderboard rating. “Lowest published price” uses the lowest median blended price across each visible provider’s priced candidate models. “Fastest published catalog” uses the lowest raw median 500-token response time. “Best published cost/speed balance” identifies the visible provider closest to the lower-price and lower-response-time ideal using those two raw catalog medians. Providers need at least three relevant priced or speed-measured models to qualify.

Providers such as xAI whose measured proprietary models are unavailable from competing hosts remain visible in the same numbered list using their raw catalog medians. Their Rating basis is “Based on own-model catalog.” They do not receive a model-adjusted rating until enough exact-model comparisons exist.

Multi-model provider comparisons

A model-set comparison asks which provider can serve every selected text model. The cheapest-provider card converts an equal 1,000-input/500-output-token mix across the selected models into a blended price per 1 million tokens. Dedicated cheapest ranking pages use the equivalent total for one request to every model, so their order is identical. Each route breakdown also shows input and output prices per 1 million tokens. Response time, first-token time, and output speed are averaged; reliability uses the lowest published 30-minute uptime because every route must remain available.

For each provider and model, the comparison prefers the provider’s unsuffixed route. If none exists, it uses the lowest-cost route with complete token prices and shows all alternatives in the route breakdown. Only full-coverage providers can win or receive a rank. The recommended cost/speed balance is one provider: the point closest to the normalized lower-cost and faster-response ideal.

Stable routes and variants

Pages use stable OpenRouter model IDs. Temporary ~…latest aliases, OpenRouter’s automatic-routing entries, and every model ID containing :free are excluded from catalogs, search, comparisons, rankings, APIs, and the sitemap.