Developers · Benchmarks
Does OpenLuxe actually save your agent tokens?
We measured it. Real tasks, real models, real API calls — comparing OpenLuxe's shipped MCP tool surface against internal architecture variants and a real free-tier competitor.
CLI v0.8.1 · July 14, 2026 · Grok 4.5 and Gemini 3.5 Flash
These are averages across Grok 4.5 and Gemini 3.5 Flash on the portable task set (the tasks every architecture, including HubSpot, can attempt), 3 trials per cell. Task success is verified against the API's actual resulting state, not the model's self-report.
Tokens per task, by architecture
Average input + output tokens to complete one task. The gap between architectures spans orders of magnitude, so the axis is logarithmic — on a linear scale the efficient bars would be invisible.
- OpenLuxe MCP: The shipped production tool surface — a handful of composable tools (list endpoints, make a request, search, common CRM ops) instead of one tool per endpoint.
- Naive one-tool-per-endpoint: Internal architecture variant: every v1 endpoint exposed as its own tool definition, capped at 200 tools (the model provider’s hard limit) even though the full surface is larger.
- OpenLuxe CLI via shell: Agent shells out to the openluxe CLI binary directly instead of using MCP tool-calling.
- Cold integration: Agent has no OpenLuxe-specific tools at all — only generic web-fetch / HTTP-request tools — and has to discover the docs, read the OpenAPI spec, and wire up auth from scratch.
- HubSpot (free tier): A real external CRM, its real free-tier API, called the same way a well-integrated agent would call it.
Task completion rate
Fewer tokens only matters if the task still gets done. Pass rate is measured by re-reading the record from the API after the run, not by trusting what the model says it did.
Full comparison
Every architecture × model cell for this run. Click a column to sort.
| Architecture | Model | Input tok | Output tok | Total tok ↓ | Turns | Pass rate | vs. OpenLuxe MCP |
|---|---|---|---|---|---|---|---|
| Naive one-tool-per-endpoint | Grok 4.5 | 51,399 | 59 | 51,457 | 2.9 | 87% | 9.4x tokens |
| Naive one-tool-per-endpoint | Gemini 3.5 Flash | 48,390 | 150 | 48,540 | 2.7 | 100% | 9.1x tokens |
| OpenLuxe CLI via shell | Gemini 3.5 Flash | 38,248 | 251 | 38,498 | 8.1 | 80% | 7.3x tokens |
| Cold integration | Gemini 3.5 Flash | 16,234 | 295 | 16,529 | 5.5 | 100% | 3.1x tokens |
| HubSpot (free tier)competitor | Gemini 3.5 Flash | 12,784 | 197 | 12,981 | 4.1 | 80% | 2.4x tokens |
| OpenLuxe CLI via shell | Grok 4.5 | 12,541 | 100 | 12,642 | 5.3 | 93% | 2.3x tokens |
| Cold integration | Grok 4.5 | 7,333 | 135 | 7,468 | 5.0 | 100% | 1.4x tokens |
| OpenLuxe MCPours | Grok 4.5 | 5,432 | 70 | 5,502 | 3.2 | 100% | baseline |
| OpenLuxe MCPours | Gemini 3.5 Flash | 5,129 | 180 | 5,309 | 3.2 | 100% | baseline |
| HubSpot (free tier)competitor | Grok 4.5 | 2,741 | 80 | 2,820 | 2.9 | 60% | 0.5x tokens |
How this was measured
Each architecture is given the same 6 tasks, 3 trials each, run against real model APIs (Grok 4.5 and Gemini 3.5 Flash) with real per-turn token accounting from the provider's own usage response — never estimated.
OpenLuxe MCP is the tool surface that ships in the CLI today. Naive one-tool-per-endpoint, CLI via shell, and cold integration are internal architecture variants run against the same OpenLuxe account and data, isolated so no run's actions can affect another's. HubSpot is a real HubSpot Developer Test Account on the free tier, called the same way a well-integrated agent would call it.
Task success is verified by reading the record back from each system's API after the run and checking it matches what was asked — not by trusting the model's transcript.
Known limitations
6 tasks (5 portable + 1 OpenLuxe-only), 3 trials each, verified against real API state (never LLM self-report). HubSpot pass rate on list_and_count/search_contacts is depressed by a diagnosed tool-description gap (their CONTAINS_TOKEN operator does whole-token not prefix matching), not a capability difference. The naive one-tool-per-endpoint arm is capped at 200 tool definitions by the model providers hard limit.
- The naive one-tool-per-endpoint arm is capped at 200 tool definitions — the model provider's hard limit on a single request — even though OpenLuxe's full public surface is larger. Its real number is a floor, not a ceiling.
- HubSpot's list/search tasks score lower in part because its CONTAINS_TOKEN filter matches whole tokens, not substrings — a real, documented behavior of their API, not a bug we introduced to make the comparison favorable.
- Two models, one account, one time window. This is not a claim about every model or every HubSpot plan.
Try it with your own agent
One credential, one CLI, the same tool surface benchmarked above.
