How We Keep Benchmark Numbers at Zero Fabrication

Strict measured-only enforcement keeps benchmark numbers at zero fabrication throughout the Hax pipeline by admitting only direct machine measurements from bench_harness live probes and auto-collected figures from published tables that explicitly label themselves as measured, resulting in a clean total of 32 records where every single one carries the measured=True flag along with its source and collection date.

In short: The measured-data hub acts as an incorruptible gatekeeper that refuses every estimate or unsourced value before it can reach any published benchmark or analysis.

In AI research and deployment discussions, numbers serve as the primary currency of credibility. A blog that publishes performance claims, hardware specifications, or workflow timings invites scrutiny precisely because those figures influence real choices about which models to run, which GPUs to purchase, and which pipelines to adopt. Hax recognized early that this trust erodes quickly once any number appears fabricated or loosely sourced. The measured-data pipeline therefore functions as the central safeguard, collecting and preserving only what machines actually recorded or what published sources themselves flagged as measured results.

Why obsess over measured numbers?#

The obsession arises because the integrity of the entire content operation rests on those 32 records remaining beyond reproach. Twenty-three of them originate from bench_harness live probes that capture fresh machine data such as the exact 95.6GB VRAM consumption per GPU card and the 120.8ms unified first-response time. Seven more come from systematic sweeps of published tables that carry the word "measured" right in their captions, yielding comparisons such as z-image finishing a generation task in 6s against qwen-image requiring 73s. Two additional records track ComfyUI pool operations, recording a 77.6% generation success rate and the use of 44 samplers under specific rig configurations.

All 32 share the measured=True designation and contain zero estimated or unsourced values. This composition matters because it demonstrates consistent application of the rule rather than selective enforcement. A beginner-friendly way to picture the measured-only store is to imagine a lab notebook where unsourced claims are refused at the door; only entries that arrived with proper documentation and direct observation can ever be cited in later summaries or decision documents. Applying that same refusal policy at scale prevents the slow accumulation of questionable data points that eventually undermines an entire body of work. Readers of the AI blog can therefore treat every cited benchmark as reproducible evidence instead of marketing-adjacent approximation.

How do we block fabrication?#

Blocking happens through deliberate intake filters rather than post-publication corrections. The hub simply will not store a value unless it arrives with either automated bench_harness instrumentation or an auto-extracted entry from a table whose own caption already declares it measured. Any attempt to insert a calculated guess, a vendor brochure claim, or a remembered average gets turned away before it touches the permanent record. This design choice directly protects the current inventory of 23 bench_harness records, 7 published measured records, and 2 ComfyUI records.

Because source and date metadata travel with every entry, the system maintains an auditable chain from raw measurement event to final citation. Transparency about the pipeline further strengthens the approach: by openly describing how many records come from each channel and confirming that all of them satisfy the measured=True requirement, Hax invites external verification instead of asking readers to trust opaque internal processes.

The table above makes the sourcing breakdown visible at a glance and underscores that the 32-record total contains no hidden estimates. Such visibility turns the data hub into both a storage system and a statement of methodological integrity.

Can you build this too?#

Anyone can adopt the same strict standards by following these steps:

1. Force source and date on every figure, so the 23 live-probe records and the 7 published records never lose their context, even months later when someone wants to re-run a comparison.
2. Default to machine measurement and validate human input, prioritizing bench_harness-style automated collection and treating any manually entered number as provisional until an actual run confirms it.
3. Label estimates as "estimate" and keep them out of the main measured set, so softer numbers are never cited alongside hard data.

When these three habits operate together, the resulting collection stays as clean as Hax's current 32 records, all carrying measured=True with full provenance.

Note: measured dates differ per record across the 2026-07-12 and 2026-07-04 collection points, while bench_harness keeps re-measuring hardware parameters continuously so the live-probe portion of the dataset remains up to date.

Reference links

• Schema.org Dataset
• Hugging Face Datasets
• JSON Schema