{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58ae2878",
   "metadata": {},
   "outputs": [],
   "source": [
    "import emat\n",
    "emat.versions()"
   ]
  },
  {
   "cell_type": "raw",
   "id": "a9c8843e",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. _methodology-splom:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd216667",
   "metadata": {},
   "source": [
    "# Scatter Plot Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2bb089d",
   "metadata": {},
   "source": [
    "Once a series of experiments has been conducted for a core model, it\n",
    "is suggested that an analyst review the results to confirm that the \n",
    "model is behaving as expected.  TMIP-EMAT provides some visualization\n",
    "tools to review results in a graphical manner, which is a generally \n",
    "convenient method to review this experimental data.\n",
    "\n",
    "To demonstrate these tools, we will use the Road Test example model included\n",
    "in TMIP-EMAT.  We can quickly construct and run a design of experiments\n",
    "to exercise this model and populate some results to visualize:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4692e6b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import emat.examples\n",
    "scope, db, model = emat.examples.road_test()\n",
    "design = model.design_experiments()\n",
    "results = model.run_experiments(design)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66f24232",
   "metadata": {},
   "source": [
    "Given this set of experimental results, we can display a scatter plot matrix\n",
    "to see the results.  This is a collection of two-dimensional plots, each\n",
    "showing a contrast between two factors, typically an input parameter (i.e. \n",
    "an uncertainty or a policy lever) and an output performance measure, although\n",
    "it is also possible to plot inputs against inputs or outputs against outputs."
   ]
  },
  {
   "cell_type": "raw",
   "id": "ea806afd",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. _methodology-display-experiments:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6826f25",
   "metadata": {},
   "source": [
    "The `display_experiments` function in the `emat.analysis` sub-package can \n",
    "automatically create a scatter plot matrix that crosses every parameter with\n",
    "every measure, simply by providing the scope and the results.  By default, \n",
    "plots that display levers are shown in blue, plots that show uncertainties\n",
    "are in red. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "571f496b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from emat.analysis import display_experiments\n",
    "display_experiments(scope, results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b39d476",
   "metadata": {},
   "source": [
    "This function also offers the opportunity to identify only a particular \n",
    "subset of parameters or measures to display, using the `rows` and `columns` \n",
    "arguments.  Similar colors are used as the default full display, although\n",
    "if the plot contrasts an uncertainty with a lever the variable on the \n",
    "X axis determines the color; and the plot is green if only measures are shown.\n",
    "Because parameters and measures\n",
    "are all required to have unique names within a scope, it is not necessary\n",
    "to identify which is which, as the `display_experiments` can figure it out\n",
    "automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37e95880",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_experiments(\n",
    "    scope, results, \n",
    "    rows=['input_flow', 'expand_capacity', 'build_travel_time'], \n",
    "    columns=['net_benefits', 'time_savings', 'no_build_travel_time', 'yield_curve', 'expand_capacity'],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "776aa86a",
   "metadata": {},
   "source": [
    "Reviewing these results can be instructive, not only for exploratory analysis\n",
    "but also for validation of the results from the core model.  An analyst can\n",
    "quickly see the direction, magnitude, and shape of various parametric relationships\n",
    "in the model, and easily detect any parameters that are giving unexpected results.\n",
    "For example, in many transportation modeling applications we would expect that\n",
    "most parameters will induce a monotonic response in most performance measures.\n",
    "Observing non-monotonic relationships where we don't expect them is a red flag for\n",
    "the analyst to closely review model outputs, and perhaps the underlying model\n",
    "coding as well, to identify and correct errors. "
   ]
  },
  {
   "cell_type": "raw",
   "id": "2223f710",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. _methodology-contrast-experiments:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "864c912e",
   "metadata": {},
   "source": [
    "## Contrasting Sets of Experiments\n",
    "\n",
    "A similar set of visualizations can be created to contrast two set\n",
    "of experiments derived from the same (or substantially similar) scopes.\n",
    "This is particularly valuable to evaluate the performance of meta-models\n",
    "that are derived from core models, as we can generate scatter plot\n",
    "matrices that show experiments from both the core and meta models.\n",
    "\n",
    "To demonstrate this capability, we'll first create a meta-model from\n",
    "the Road Test core model, then apply that meta-model to a design of\n",
    "5,000 experiments to create a set of meta-model results to visualize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e5c9d94",
   "metadata": {},
   "outputs": [],
   "source": [
    "mm = model.create_metamodel_from_design('lhs', suppress_converge_warnings=True)\n",
    "mm_design = mm.design_experiments(n_samples=5000)\n",
    "mm_results = mm.run_experiments(mm_design)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5d49fca",
   "metadata": {},
   "source": [
    "The `contrast_experiments` function in the `emat.analysis` sub-package can \n",
    "automatically create a scatter plot matrix, using a very similar interface\n",
    "to the `display_experiments` function.  The primary difference between these \n",
    "two functions is that `contrast_experiments` takes two sets of experiments\n",
    "as arguments, instead of one.  The resulting plots are also not colorized\n",
    "based on the roles of each factor in the scope; instead colors are used\n",
    "to differentiate the different datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45e1836d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from emat.analysis import contrast_experiments\n",
    "contrast_experiments(scope, mm_results, results)"
   ]
  },
  {
   "cell_type": "raw",
   "id": "b9e9b604",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. include:: splom-api.irst"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}