Confidence curves are used in uncertainty validation to assess how large uncertainties ($u_{ E}$) are associated with large errors ($E$). An oracle curve is commonly used as reference to estimate the quality of the tested datasets. The oracle is a perfect, deterministic, error predictor, such as $|E|=\pm u_{ E}$, which corresponds to a very unlikely error distribution in a probabilistic framework and is unable unable to inform us on the calibration of $u_{ E}$. I propose here to replace the oracle by a probabilistic reference curve, deriving from the more realistic scenario where errors should be random draws from a distribution with standard deviation $u_{ E}$. The probabilistic curve and its confidence interval enable a direct test of the quality of a confidence curve. Paired with the probabilistic reference, a confidence curve can be used to check the calibration and tightness of prediction uncertainties.
Categories: physics.data-an