Optimal Experiment Design Demos
These demos use expected information gain to select observations during sequential data collection.
What this is
Optimal experiment design provides a way to fit a statistical model with less data: start with some initial data, then iteratively choose the next measurement to reduce uncertainty about whatever quantity is most relevant for the decision.
Example: in deduplication, instead of labeling random record pairs which are almost all non-matches, we label the pairs that are nearer to the decision boundary and correlate strongly with model parameters.
Typical applications include A/B testing with covariates, record linkage, pricing experiments, adaptive surveys, and ML data labeling.
Demos
Precomputed plots and demos.
The record-linkage demo selects pairs for labeling by expected information gain and fits match probabilities from those labels. It runs against a hosted backend API.
The A/B demo tends to allocate observations to regions of covariate space where the preferred treatment is uncertain and the outcome covaries with the decision of interest. The page contains a 41-step run of decision regions, decision uncertainty, and expected information gain.
Method
A high-level view of sequential design.
Sequential design makes data collection active: choose the next measurement to maximize expected information about a target quantity of interest, accounting for sampling uncertainty and uncertainty in the model.
In practice, this is implemented by an approximation to Bayesian optimal experiment design, evaluating candidate measurements via a data acquisition objective (e.g., expected information gain about a decision boundary or treatment choice), and iterating until the remaining decision uncertainty is below a chosen tolerance.
The acquisition objective can target a treatment rule or another decision-relevant quantity rather than total parameter uncertainty. Uncertainty that cannot change the recommended action is naturally excluded from the objective. The method requires a fitted probabilistic model under which the acquisition objective can be evaluated. In my implementation, one such evaluation has roughly the same order of runtime as one model fit.