Benchmarking Learned Systems

Overview

Traditional benchmarks fail to characterize learned systems that overfit to static workloads. We propose new benchmarks that measure adaptability through descriptive statistics and outliers, rather than average metrics, to fairly evaluate the cost and benefits of learned database components.