Optimal experiment design

Optimal Experiment Design Demos

These demos show how to choose what data to collect next, based on what we expect to learn from it.

What this is

Choosing measurements this way can help us fit a model with less data.

For example, when looking for duplicate records, most randomly chosen pairs won’t match. We can instead label pairs the model expects to learn more from, often ones it isn’t sure about.

Typical applications include A/B testing with covariates, record linkage, pricing experiments, adaptive surveys, and ML data labeling.

Demos

Method

The model estimates how much each possible measurement would teach us, then chooses the most informative one.

We can focus on a decision, such as which treatment to choose, instead of trying to learn every detail of the model. In my implementation, evaluating one possible measurement takes about as long as fitting the model.