{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cb0e57b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import emat\n",
    "emat.versions()"
   ]
  },
  {
   "cell_type": "raw",
   "id": "1fd2d9f3",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. _methodology-interactive-explorer:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a48f37ee",
   "metadata": {},
   "source": [
    "# Interactive Explorer\n",
    "\n",
    "TMIP-EMAT includes an interactive visualizer, inspired by a \n",
    "[similar tool](https://htmlpreview.github.io/?https://github.com/VisionEval/VisionEval/blob/master/sources/VEScenarioViewer/verpat.html) \n",
    "provided with the [VisionEval](https://visioneval.org) package.\n",
    "To demonstrate the interactive visualizer, we will use the Road Test example model.\n",
    "First, we need to develop and run a design of experiments to have some\n",
    "data to explore.  We'll run 5,000 experiments to get a good size sample of \n",
    "data points to visualize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "174aabd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import emat.examples\n",
    "scope, db, model = emat.examples.road_test()\n",
    "design = model.design_experiments(n_samples=5000)\n",
    "results = model.run_experiments(design)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0958fa2e",
   "metadata": {},
   "source": [
    "One feature of the visualizer is the ability to display not only a number of results,\n",
    "but also to contrast those results against a given \"reference\" model that represents\n",
    "a more traditional single-point forecast of inputs and results.  We'll prepare a\n",
    "reference point here using the `run_reference_experiment` method of the `CoreModel`\n",
    "class, which reads the input parameter defaults (as defined in the scope),\n",
    "and returns both inputs and outputs in a DataFrame (essentially, an experimental\n",
    "design with only a single experiment), suitable for use as the reference point marker in our\n",
    "visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9f180c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "refpoint = model.run_reference_experiment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d100198",
   "metadata": {},
   "source": [
    "The interactive visualizer class can be imported from the `emat.analysis` package.\n",
    "To use it, we create an `Visualizer` instance, giving a scope and a set of \n",
    "experimental results, as well as the reference point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dd8ea40",
   "metadata": {},
   "outputs": [],
   "source": [
    "from emat.analysis import Visualizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fa4b32f",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz = Visualizer(scope=scope, data=results, reference_point=refpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efd1db9a",
   "metadata": {},
   "source": [
    "## Single Dimension Figures\n",
    "\n",
    "To build a complete interactive workspace similar to that provided by VisionEval, we\n",
    "can use the `complete` method of the `Visualizer` instance we created above. This will\n",
    "create a set of histograms illustrating the data in the results computed above. There\n",
    "is one histogram for each policy lever, exogenous uncertainty, and performance measure.\n",
    "\n",
    "A range of data in each histogram can be selected by dragging horizonatally across the \n",
    "figure. For continuous parameters (i.e. float or integer valued parameters) you can \n",
    "select a single contiguous range by dragging across that range, and deselect by double \n",
    "clicking on the figure (or by selecting the entire possible range).  For discrete \n",
    "parameters (i.e. boolean or categorical parameters, identifiable by the larger gaps\n",
    "between the bars in the figure) dragging across the center of any bar toggles whether\n",
    "that bar is selected or not.  This allows non-contiguous selections in categories that\n",
    "have 3 or more possible values.  Like the other figures, any selection can be cleared \n",
    "by double-clicking.\n",
    "\n",
    "Selections can be made simultaneously over any combination of uncertainties, policy levers,\n",
    "and performance measures.  The combination of controls offered can \n",
    "be used interactively to select and highlight only a subset of the experiments in\n",
    "the complete data set.  By manipulating these controls, users can explore the \n",
    "interaction across various inputs and outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79a99865",
   "metadata": {},
   "source": [
    "![Selecting from histograms](interactive-gifs/select-from-histograms-.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1340f939",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a11cc054",
   "metadata": {},
   "source": [
    "It is also possible to display just a small subset of the figures of this interactive viewer.\n",
    "This could be convenient, for example, if there are a very large number of performance measures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09e9c3bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.selectors(['input_flow', 'expand_capacity', 'net_benefits'])"
   ]
  },
  {
   "cell_type": "raw",
   "id": "e99bb238",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "In addition to manipulating the controls interactively, they can also be\n",
    "set programatically from Python code.  To do so, we can define a new :class:`emat.Box`\n",
    "that declares lower and/or upper bounds for any continuous dimensions,\n",
    "as well as the set of allowed (included) value for any discrete dimensions,\n",
    "and then add that new box to this visualizer using the :meth:`Visualizer.add_box` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b7c19f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "box = emat.Box(\"Passable\", scope=scope)\n",
    "box.set_upper_bound('cost_of_capacity_expansion', 400)\n",
    "box.set_lower_bound('time_savings', 5)\n",
    "box.remove_from_allowed_set('debt_type', 'GO Bond')\n",
    "viz.add_box(box)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a56e45",
   "metadata": {},
   "source": [
    "Alternatively, a new box can be created and added to the Visualier\n",
    "with a single :meth:`Visualizer.new_box` command, which\n",
    "passes most keyword arguments through to the :class:`emat.Box` constuctor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2576d74",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.new_box('Profitable', lower_bounds={'net_benefits':0});"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6afe4b8b",
   "metadata": {},
   "source": [
    "Each of these new boxes is added to the `Visualizer` seperately. You can\n",
    "switch between different active boxes using the dropdown selector at the top \n",
    "of the `complete` interface -- this same selector is available within the\n",
    "smaller `status` widget:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0db8a295",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92341c7e",
   "metadata": {},
   "source": [
    "You can also programatically find and change the active box from Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7d0ba61",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.active_selection_name()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59c513b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.set_active_selection_name(\"Passable\")\n",
    "viz.active_selection_name()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d7ca057",
   "metadata": {},
   "source": [
    "When interactively changing bounds by dragging on figures, the currently \n",
    "\"active\" box is modified with the revised bounds.  The entire set of\n",
    "bounds can be cleared at once with the `clear_box` method, which by default \n",
    "clears the settings on the active box selection; give a name to clear the \n",
    "settings from a different box selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c9686a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.clear_box()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288f5493",
   "metadata": {},
   "source": [
    "If instead we want to manipulate an existing box selection, we can access the Box object, \n",
    "manipulate it (e.g. by using `remove_from_allowed_set` or `add_to_allowed_set`), \n",
    "and write it back into the Visualizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52ca59eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "box = viz['Profitable']\n",
    "box.remove_from_allowed_set('debt_type', 'Rev Bond')\n",
    "viz['Profitable'] = box"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3deda633",
   "metadata": {},
   "source": [
    "## Two Dimension Figures"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dafd181",
   "metadata": {},
   "source": [
    "The `Visualizer` object can also create an interactive two-dimensional scatter plot,\n",
    "using the `two_way` method. This method allows the user to specify the variables\n",
    "for both the `x` and `y` axis, and either can be any policy lever, exogenous \n",
    "uncertainty, or performance measure.  These dimensions can be changed interactively\n",
    "later as well.  The resulting scatter plot is linked to the same selection of\n",
    "experiments in the interactive one-dimensional figures shown above, and by default\n",
    "the same experiments are highlighted in the same color scheme in all of these related\n",
    "figures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "550ec202",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.two_way(x='expand_capacity', y='time_savings')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d5041bd",
   "metadata": {},
   "source": [
    "One useful feature of the `two_way` is the ability to manually \"lasso\" a selection of \n",
    "data points. This lasso selection does *not* need to be anything like a rectangular \n",
    "box selection, as we have seen so far.  Once a lasso selection of data points is made \n",
    "in the figure above, you can choose \"Use Manual Selection\" from the `Edit Selection...`\n",
    "menu at right, which will create a new `Visualizer` selection from the selected data.\n",
    "The highlight color changes to signify that this is not an editable rectangular box,\n",
    "and the selected data will be highlighted in *all* figures linked to this `Visualizer`,\n",
    "including the histograms above."
   ]
  },
  {
   "cell_type": "raw",
   "id": "335148cb",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "In addition to the `two_way`, which offers a feature-packed view of two dimensions at a time,\n",
    "there is also a scatter plot matrix :meth:`Visualizer.splom` option, which displays a configurable matrix of similar \n",
    "two dimensional views."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0937c107",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.splom(\n",
    "    rows=('expand_capacity','time_savings','net_benefits'), \n",
    "    cols='L',\n",
    "    reset=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c480d7c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.hmm(\n",
    "    rows=('time_savings','net_benefits'), \n",
    "    cols='L',\n",
    "    show_points=100,\n",
    "    reset=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d8c8782",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.new_selection(\n",
    "    \"time_savings * input_flow > 1000 & cost_of_capacity_expansion < 300\",\n",
    "    name=\"TimeSaved\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa2ecbb8",
   "metadata": {},
   "source": [
    "## Dynamic Feature Scoring\n",
    "\n",
    "EMAT can score the relative importance of inputs for an experiment being within the selection, either for a typical rectangular selection based on thresholds, or for any arbitrary selection. These scores are recomputed and updated in near-real-time as the thresholds are adjusted.\n",
    "\n",
    "When the selection includes rectangular thresholds set on both inputs and outputs, the thresholded inputs are automatically excluded from the scoring algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "595beaa3",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.selection_feature_scores()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5443190",
   "metadata": {},
   "source": [
    "## Using PRIM with the Interactive Explorer\n",
    "\n",
    "The PRIM tools are available directly within the interactive explorer. Simply \n",
    "set a target as shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac76c624",
   "metadata": {},
   "outputs": [],
   "source": [
    "prim = viz.prim(target=\"net_benefits >= 0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd92c516",
   "metadata": {},
   "outputs": [],
   "source": [
    "box1 = prim.find_box()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79741d79",
   "metadata": {},
   "source": [
    "The tradeoff selector is directly integrated into the explorer.  In addition\n",
    "to the information visible by hovering over any point in the tradeoff selector\n",
    "figure, clicking on that point will create a new selection in the explorer, and\n",
    "set all of the interactive constraints \n",
    "to the bounds given by that particular point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c54bf59",
   "metadata": {},
   "outputs": [],
   "source": [
    "box1.tradeoff_selector()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20e17284",
   "metadata": {},
   "outputs": [],
   "source": [
    "box1.select(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b0888aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42bb9b01",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.lever_selectors()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "436b71a9",
   "metadata": {},
   "source": [
    "We can also use PRIM to explore solutions based only on manipulating the\n",
    "policy levers, and not the combination of all inputs (levers & uncertainties)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a22afe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "prim_levers = viz.prim('levers', target=\"Profitable\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2271999a",
   "metadata": {},
   "outputs": [],
   "source": [
    "prim_levers.tradeoff_selector()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1d7ce0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz.parcoords()"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}