2 days ago
Speed Means Nothing If You're Down: Benchmarking For The Real World
Spencer Kimball is the CEO and co-founder of Cockroach Labs.
For decades, database performance benchmarks like TPC-C (Transaction Processing Performance Council Benchmark C) chased peak throughput under ideal conditions. Systems were scaled vertically and micro-optimized at every layer to game benchmarks and maximize transactions per second (TPS). But times—and technology—have changed.
Today's applications live everywhere: across regions, clouds and availability zones. They're always on, global and expected to perform flawlessly despite disruptions. Users now demand instant banking balances, lag-free global broadcasts, immediate access to patient records and real-time inventory updates across international supply chains. These aren't aspirations; they're baseline expectations, assumed and demanded unequivocally.
Modern workloads are unpredictable, highly distributed and intensely data-driven. Fraud detection, logistics tracking and AI-powered automation now run continuously at a global scale. Autonomous systems assume constant availability and consistency. They don't pause for outages or wait for recovery. They expect the system to be fast, consistent and always available.
Yet legacy benchmarks still focus on raw throughput, rarely accounting for the realities of modern architecture: latency spikes, node crashes or regional outages. The true measure of performance isn't speed under ideal conditions—it's resilience under failure.
If anyone doubts the importance of resilience, recent history is rife with cautionary tales.
• Costco's Black Friday outage in 2019 allegedly cost millions in sales.
• Ticketmaster's 2022 crash during Taylor Swift's ticket sales revealed its infrastructure was unprepared for massive spikes.
• A faulty update in 2024 froze hospital electronic health records, forcing clinicians to revert to paper and risking patient safety.
• CrowdStrike's 2024 outage halted global freight logistics due to insufficient resilience planning.
• Barclays and Capital One outages in 2025 left millions unable to access banking services, demonstrating that even leading institutions aren't immune.
There is a perennial thread running through this representative sampling of recent failures: The rising complexity of modern applications requires a shift in mindset to resilience as a first-order goal.
Modern infrastructure complexity—multiregion, multicloud and governed by stringent regulations like GDPR, PDPA and DORA—demands new benchmarks that prioritize resilience as a fundamental metric, not an afterthought. Benchmarks that ignore this operational and compliance complexity are no longer sufficient.
True resilience testing involves deliberately introducing controlled chaos: killing nodes, dropping disks, simulating outages. The objective is clear: Observe the system's behavior under stress. Does it reroute traffic seamlessly? Are transactions duplicated, delayed or lost? How rapidly does performance return to baseline? These aren't trivial operational details—they're critical indicators of system integrity.
Chaos testing, a methodology where systems are intentionally stressed to uncover hidden vulnerabilities, is gaining traction. However, resilience testing often remains disconnected from standard performance metrics like throughput and latency. Integrating these tests ensures resilience isn't merely theoretical but quantifiable and central to system design.
Just as vehicle performance isn't judged solely by top speed, database benchmarks shouldn't focus exclusively on maximum throughput. Benchmarks must evolve to measure continuity, recovery speed and stability under real-world pressures. Systems engineered with built-in replication, self-healing automation and geographic consensus aren't merely technically superior; they're strategically essential. These aren't just engineering choices; they're risk strategies. And they should be part of how we measure value.
Speed without resilience is meaningless. Challenge your teams and vendors to prove resilience under chaos. If they can't, they're not ready for today's demands.
The ultimate benchmark isn't the fastest lap—it's staying on track when legacy systems are stuck rebuilding in the pit.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
