import emat
emat.versions()

emat 0.5.2, plotly 4.14.3

Policy Contrast

The policy contrast viewer, also known as the “AB” viewer, allows you to compare the outcomes of two different sets of policies, to get an understanding of how they differ. The tool runs the model across a distribution of inputs, and displays the resulting distribution of performance measure outputs. By default, uncertainties are modeled using the distributions contained in the model scope, while policy levers are each manipulated to be a specific value. Two sets set of model runs are generated by making a set random draws from all the relevant distributions, and then running the model for every combination of random draws and each set of specific-value inputs. For example, if the number of background random draws is set at 250, then 500 model runs are conducted, 250 each for the two different sets of selected policy levers. The exact same random draws are used for both groups of model runs, so that any variation in the performance measures can be unambiguously linked to the changes in the specific-value inputs, instead of being a result of input stocasticity.

Using this viewer interactively requires a model that is re-runnable in near real time, as every time the “recompute” button is pushed the model will need to be run hundreds of times before the figures can be updated. This speed can be provided either as a core model which runs almost instantaneously, or as a meta-model of a slow-running core model. In this demo, we will use the road test model, which runs fast itself.

import emat.examples
scope, db, model = emat.examples.road_test()

The AB_Viewer object provides the interface for using this tool. To create a viewer, we’ll import AB_Viewer from the emat.analysis package. We the create an instance, providing the runnable model, plus the number of background model runs that will be conducted to simulate the distribution of parameters set to “distribution” mode. Generally a few hundred runs will be sufficient to get a reasonable high-level overview of the distributions; more may be required to support detailed statistical analysis of the results, although such analysis is unnecessary for many exploratory applications.

from emat.analysis import AB_Viewer
ab = AB_Viewer(model, background=250)

The policy contrast viewer has two principal parts, the interface, and figure viewers. The interface has interactive controls for the user to control the tool. Each exogenous uncertainty and policy lever is represented by a row of controls. The left-most toggle, next to the parameter name, controls whether each input is in “distribution” or “specific value” mode. By default, all exogenous uncertainties start in disctribution mode, and all policy levers start in specific value mode, but any input parameter can be set to either mode as desired. This allows, for example, setting one uncertainty to particular high and low values, to visualize threats and opportunities.

ab.interface

When a parameter is set to distribution mode, a readout of the shape and extent of the distribution is provided. There are no other controls for parameters in this mode.

When a parameter is set to specific value mode, an additional set of controls appears in the row. Two sliders are shown along with value read outs. The left and right sliders control the “A” and “B” settings, respectively. The slider handles are also colored green and purple to facilitate coordination with the display figures (shown below). There is a “link” button between the two sliders, which is toggled “on” by default. When activated, the link ensures that both A and B settings for the parameter are the same, so that the difference are relegated only to the unlinked parameters. The user can unlink any number of specific value parameters, and set distinct A and B values.

The outputs from the policy contrast viewer are shown using the get_figures command. This command allow for naming a curated list of specific performance measure outputs, instead of generating figures for all performance measures. The figures take a moment to update each iteration, and displaying too many at once can degrade the performance of the Jupyter notebook.

Each figure contains two plots. The left plot is an asymmetric violin plot, showing the distribution for that performance measure under the A (green, upper half) and B (purple, lower half) policies. The right hand figure shows in red the distribution of the paired differences between the A and B results. These differences are not aggregate differences for A and B, but rather the distribution of the differences for the same random draw in each group.

ab.get_figures('net_benefits', 'time_savings')

Unlike most other TMIP-EMAT visualizations, these figures do not automatically self-update in response to changes in the slider positions in the interface panel. This is because updating these figures requires re-running hundreds of model runs, which can be completed somewhat quickly but not instantly. To prevent lauching those runs every time any slider moves, these figures are intead updated only when you click the “Recompute” button at the bottom of the interface panel.

Alternatively, the sliders for each parameter can be manipulated programatically, using the set_parameter command:

ab.set_parameter('expand_capacity', 40, side='B')