Optimal Experiment Design Demos
These demos show how to choose what data to collect next, based on what we expect to learn from it.
What this is
Choosing measurements this way can help us fit a model with less data.
For example, when looking for duplicate records, most randomly chosen pairs won’t match. We can instead label pairs the model expects to learn more from, often ones it isn’t sure about.
Typical applications include A/B testing with covariates, record linkage, pricing experiments, adaptive surveys, and ML data labeling.
Demos
Label pairs of records and see the model learn which ones are likely to match. The demo chooses which pairs to show you next.
The A/B demo tends to allocate observations to regions of covariate space where the preferred treatment is uncertain and the outcome covaries with the decision of interest. The page contains a 41-step run of decision regions, decision uncertainty, and expected information gain.
Method
The model estimates how much each possible measurement would teach us, then chooses the most informative one.
We can focus on a decision, such as which treatment to choose, instead of trying to learn every detail of the model. In my implementation, evaluating one possible measurement takes about as long as fitting the model.