Dynamic Probability Scoring Rules, Statistical Martingale Testing and Model Selection

We present a novel approach for measuring the quality of a time evolving probability estimates. Examples of such probabilistic estimates are election predictions, weather predictions, or probabilities of some market events... [ view full abstract ]

We present a novel approach for measuring the quality of a time evolving probability estimates. Examples of such probabilistic estimates are election predictions, weather predictions, or probabilities of some market events that appear in hedging of financial products, such as probabilities that a price of an asset will end up over or below a certain level. The basic idea of our approach is that if we have two different probability estimates of the outcome, one can use this discrepancy for setting a trade of these two values against each other. The exact trading price set by this procedure and the corresponding volume is determined by optimization of some utility function that describes the hypothetical behavior of these two bettors. Such optimization procedure finds an equilibrium, where the supply and demand functions of the two agents meet. This creates a sequence of trades that matches every discrepancy that was not reflected in the past trades. We show that the expected profit loss of the true probability series is positive against any other probability sequence regardless of the choice of the utility function. As the true probability evolution is a martingale (conditional expectation of the ultimate outcome), this procedure can be used as a martingale test. In addition,this approach also gives a procedure to select a statistically optimal model, so it can also be used for model selection.