Benchmarking Quantum ML Models

Fair benchmarking of quantum machine learning demands strong classical baselines, honest resource accounting, and datasets that actually test the claimed strength.

Why benchmarking is hard to do honestly

A quantum model is only interesting if it beats the best classical alternative on a meaningful task, yet many published comparisons use weak baselines, tiny datasets, or metrics that ignore the true cost of running the quantum method. Rigorous benchmarking is the discipline that separates genuine progress from artifacts, and it is where many optimistic claims about quantum advantage fall apart.

Choosing the baseline

The classical baseline must be a well-tuned model, not a strawman. For a quantum kernel classifier, the comparison should include Gaussian and polynomial kernel SVMs with tuned hyperparameters. For a variational classifier, it should include gradient-boosted trees and small neural networks. Beating an untuned or trivial classical model demonstrates nothing.

Accounting for all resources

• Count data-loading cost, which idealized analyses often omit.
• Count the shots needed to reach the reported precision; runtime scales with one over epsilon squared.
• Report qubit count, circuit depth, and whether results assume noiseless simulation or real hardware.
• State whether the setting is fault-tolerant or NISQ, since the comparison changes completely.

Choosing the dataset

A benchmark should test the model where it is claimed to be strong. Quantum kernels may shine on data with hidden quantum-like structure but not on generic tabular data, so reporting only the favorable case is misleading. Best practice includes datasets designed to be classically hard, standard classical benchmarks for fairness, and, where possible, quantum-native data. Cross-validation and multiple random seeds guard against cherry-picking.

# A fair benchmarking harness (schematic)
results = {}
for model in [quantum_model, tuned_svm, gradient_boosting, small_nn]:
    scores = cross_validate(model, X, y, folds=5, seeds=range(10))
    results[model.name] = {
        'accuracy': mean(scores),
        'shots_or_flops': model.resource_count(),  # count the real cost
    }
# Compare accuracy AND resources, not accuracy alone.

The reproducibility standard

Because quantum results depend on hardware calibration, noise, and shot budgets that drift over time, benchmarks should publish full circuit specifications, seeds, shot counts, and mitigation settings so others can reproduce them. A result that cannot be reproduced, or that quietly assumes free data loading and unlimited shots, is not evidence of advantage. Honest benchmarking, more than any single algorithm, is what will tell whether quantum machine learning delivers.