{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Writing and Using a Bespoke Model Interface" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import emat\n", "import os\n", "import pandas as pd\n", "import numpy as np\n", "import gzip\n", "import asyncio\n", "from emat.util.show_dir import show_dir, show_file_contents" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook is meant to illustrate the use of TMIP-EMAT's\n", "various modes of operation. It provides an illustration of how to use \n", "TMIP-EMAT and the demo interface to run the command line version\n", "of the [Road Test](https://tmip-emat.github.io/source/emat.examples/RoadTest/road_test_yaml.html) \n", "model. A similar approach can be developed to run\n", "any transportation model that can be run from the command line, including\n", "for proprietary modeling tools that are typically run from a graphical\n", "user interface (GUI) but that provide command line access also." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example notebook, we will activate some logging features. The \n", "same logging utility is written directly into the EMAT and the\n", "[core_files_demo.py](https://github.com/tmip-emat/tmip-emat-bespoke-demo/blob/main/core_files_demo.py) module. This will give us a view of what's happening\n", "inside the code as it runs." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import logging\n", "from emat.util.loggers import log_to_stderr\n", "log = log_to_stderr(logging.INFO)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Connecting to the Model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The interface for this model is located in the `core_files_demo.py`\n", "module, which we will import into this notebook. This file is extensively\n", "documented in comments, and is a great starting point for new users\n", "who want to write an interface for a new bespoke travel demand model." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import core_files_demo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Within this module, you will find a definition for the \n", "`RoadTestFileModel` class.\n", "\n", "We initialize an instance of the model interface object.\n", "If you look at the module code, you'll note the `__init__` function\n", "does a number of things, including creating a temporary directory\n", "to work in, copying the needed files into this temporary directory,\n", "loading the scope, and creating a SQLite database to work within.\n", "For your implementation, you might or might not do any of these steps.\n", "In particular, you'll probably want to use a database that is\n", "not in a temporary location, so that the results will be available\n", "after this notebook is closed." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:03.84] MainProcess/WARNING: changing cwd to /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc\n", "[00:03.88] MainProcess/INFO: running script emat_db_init.sql\n", "[00:03.89] MainProcess/INFO: running script meta_model.sql\n", "[00:03.90] MainProcess/INFO: found no experiments with missing run_id's\n", "[00:03.90] MainProcess/INFO: running script emat_db_init_views.sql\n" ] } ], "source": [ "fx = core_files_demo.RoadTestFileModel()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once we have loaded the `RoadTestFileModel` class, we have\n", "a number of files available in the \"master_directory\" that \n", "was created as that temporary directory:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tmpygcky5rc/\n", "├── road-test-colleague.sqlitedb\n", "├── road-test-demo.db\n", "├── road-test-files/\n", "│ ├── demo-inputs-l.yml\n", "│ └── demo-inputs-x.yml.template\n", "├── road-test-model-config.yml\n", "└── road-test-scope.yml\n" ] } ], "source": [ "show_dir(fx.master_directory.name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Understanding Directories\n", "\n", "The TMIP-EMAT interface design for files-based bespoke models uses\n", "pointers for several directories to control the operation of the \n", "model.\n", "\n", "- **local_directory** \n", " This is the working directory for this instance of TMIP-EMAT,\n", " not that for the core model itself. Typically it can be Python's \n", " usual current working directory, accessible via `os.getcwd()`.\n", " In this directory typically you'll have a TMIP-EMAT model \n", " configuration *yaml* file, a scope definition *yaml* file, and\n", " a sub-directory containing the files needed to run the core model\n", " itself.\n", "\n", "- **model_path** \n", " The relative path from the `local_directory` to the directory where\n", " the core model files are located. When the core model itself is actually\n", " run, this should be to the \"current working directory\" for that run.\n", " The `model_path` must be given in the model config *yaml* file.\n", "\n", "- **rel_output_path** \n", " The relative path from the `model_path` to the directory where\n", " the core model output files are located. The default value of this \n", " path is \"./Outputs\" but this can be overridden by setting \n", " `rel_output_path` in the model config *yaml* file. If the outputs\n", " are comingled with other input files in the core model directory,\n", " this can be set to \".\" (just a dot).\n", "\n", "- **archive_path** \n", " The path where model archive directories can be found. This path\n", " must be given in the model config *yaml* file. It can be given as\n", " an absolute path, or a relative path. If it is a relative path, \n", " it should be relative to the `local_directory`.\n", " \n", "These directories, especially the ones other than the `local_directory`,\n", "are defined in a model configuration *yaml* file. This makes it easy to\n", "change the directory pointers when moving TMIP-EMAT between different\n", "machines that may have different file system structures." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Single Run Operation for Development and Debugging" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before we take on the task of running this model in exploratory mode, we'll\n", "want to make sure that our interface code is working correctly. To check each\n", "of the components of the interface (setup, run, post-process, load-measures,\n", "and archive), we can run each individually in sequence, and inspect the results\n", "to make sure they are correct." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### setup\n", "\n", "This method is the place where the core model *set up* takes place,\n", "including creating or modifying files as necessary to prepare\n", "for a core model run. When running experiments, this method\n", "is called once for each core model experiment, where each experiment\n", "is defined by a set of particular values for both the exogenous\n", "uncertainties and the policy levers. These values are passed to\n", "the experiment only here, and not in the `run` method itself.\n", "This facilitates debugging, as the `setup` method can be used \n", "without the `run` method, as we do here. This allows us to manually\n", "inspect the prepared files and ensure they are correct before\n", "actually running a potentially expensive model.\n", "\n", "Each input exogenous uncertainty or policy lever can potentially\n", "be used to manipulate multiple different aspects of the underlying\n", "core model. For example, a policy lever that includes a number of\n", "discrete future network \"build\" options might trigger the replacement\n", "of multiple related network definition files. Or, a single uncertainty\n", "relating to the cost of fuel might scale both a parameter linked to\n", "the modeled per-mile cost of operating an automobile and the\n", "modeled total cost of fuel used by transit services." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this demo model, running the core model itself in files mode \n", "requires two configuration files to be available, one for levers and\n", "another for uncertainties. These two files are provided in the demo\n", "in two ways: as a runnable base file (for the levers) and as a template\n", "file (for the uncertainties)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The levers file is a *ready-to-use* file (for this demo, in YAML format,\n", "although your model may use a different file format for input files).\n", "It has default values pre-coded into the file, and to modify this \n", "file for use by EMAT the `setup` method needs to parse and edit this\n", "file to swap out the default values for new ones in each experiment.\n", "This can be done using regular expressions (as in this demo), or any other method you\n", "like to edit the file appropriately. The advantage of this approach\n", "is that the base file is ready to use with the core model as-is, facilitating\n", "the use of this file outside the EMAT context." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---\n", "# This file defines lever values for the files-based\n", "# Road Test example. It is intentionally a complex way\n", "# to implement this Python-based model, designed to\n", "# demonstrate how to use a files-based model called\n", "# from the command line.\n", "expand_capacity: 10\n", "amortization_period: 30\n", "interest_rate_lock: False\n", "debt_type: GO Bond\n", "lane_width: 10\n", "mandatory_unused_lever: 42\n", "...\n" ] } ], "source": [ "show_file_contents(fx.master_directory.name, 'road-test-files', 'demo-inputs-l.yml')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By contrast, the uncertainties file is in a *template* format. The\n", "values of the parameters that will be manipulated by EMAT for each \n", "experiment are not given by default values, but instead \n", "each value to be set is indicated in the file by a unique token that is easy to\n", "search and replace, and definitely not something that appear in any script otherwise.\n", "This approach makes the text-substitution code that is used in this module much\n", "simpler and less prone to bugs. But there is a small downside of this approach:\n", "every parameter must definitely be replaced in this process, as the template file\n", "is unusable outside the EMAT context, and also every unique token needs to be replaced. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---\n", "# This file defines uncertainty values for the files-based\n", "# Road Test example. It is intentionally a complex way\n", "# to implement this Python-based model, designed to\n", "# demonstrate how to use a files-based model called\n", "# from the command line.\n", "alpha: __EMAT_PROVIDES_VALUE__ALPHA__\n", "beta: __EMAT_PROVIDES_VALUE__BETA__\n", "input_flow: __EMAT_PROVIDES_VALUE__INPUT_FLOW__\n", "value_of_time: __EMAT_PROVIDES_VALUE__VALUE_OF_TIME__\n", "labor_unit_cost_expansion: __EMAT_PROVIDES_VALUE__LABOR_UNIT_COST_EXPANSION__\n", "materials_unit_cost_expansion: __EMAT_PROVIDES_VALUE__MATERIALS_UNIT_COST_EXPANSION__\n", "interest_rate: __EMAT_PROVIDES_VALUE__INTEREST_RATE__\n", "yield_curve: __EMAT_PROVIDES_VALUE__YIELD_CURVE__\n", "...\n" ] } ], "source": [ "show_file_contents(fx.master_directory.name, 'road-test-files', 'demo-inputs-x.yml.template')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Regardless of which file management system you use, the `setup` method\n", "is the place to make edits to these input files and write them into \n", "your working directory. To do so,\n", "the `setup` method takes one argument: a dictionary containing key-value\n", "pairs that assign a particular value to each input (exogenous uncertainty \n", "or policy lever) that is defined in the model scope. The keys must match \n", "exactly with the names of the parameters given in the scope. \n", "\n", "If you have written your `setup` method to call the super-class `setup`,\n", "you will find that if you give keys as input that are not defined in\n", "the scope, you'll get a KeyError." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:03.94] MainProcess/ERROR: SETUP ERROR: 'name_not_in_scope' not found in scope parameters\n", "[00:03.94] MainProcess/ERROR: KeyError(\"'name_not_in_scope' not found in scope parameters\")\n" ] } ], "source": [ "bad_params = {\n", " 'name_not_in_scope': 'is_a_problem',\n", "}\n", "\n", "try:\n", " fx.setup(bad_params)\n", "except KeyError as error:\n", " log.error(repr(error))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "On the other hand, your custom model may or may not allow you to leave out\n", "some parameters. It is up to you to decide how to handle missing values, \n", "either by setting them at their default values or raising an error. In \n", "normal operation, parameters typically won't be left out from the design\n", "of experiments, so it is not usually important to monitor this carefully.\n", "\n", "In our example module's `setup`, all of the uncertainty values must be given,\n", "because the template file would be unusable otherwise. But the policy levers \n", "can be omitted, and if so they are left at their default values in the \n", "original file. Note that the default values in that file are not strictly\n", "consistent with the default values in the scope file, and TMIP-EMAT does \n", "nothing on its own to address this discrepancy." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:03.97] MainProcess/INFO: RoadTestFileModel SETUP RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n", "[00:03.97] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 1 RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n" ] } ], "source": [ "params = {\n", " 'expand_capacity': 75,\n", " 'amortization_period': 25,\n", " 'debt_type': \"Paygo\",\n", " 'alpha': 0.1234,\n", " 'beta': 4.0,\n", " 'input_flow': 100,\n", " 'value_of_time': 0.075,\n", " 'unit_cost_expansion': 100,\n", " 'interest_rate': 0.035,\n", " 'yield_curve': 0.01,\n", "} # interest_rate_lock is missing, that's ok\n", "\n", "fx.setup(params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After running `setup` successfully, we will have overwritten the \n", "\"demo-inputs-l.yml\" file with new values, and written a new \n", "\"demo-inputs-x.yml\" file into the model working directory with those\n", "values." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tmpygcky5rc/\n", "├── _emat_experiment_id_.yml\n", "├── _emat_parameters_.yml\n", "├── archive/\n", "│ └── scp_EMAT Road Test/\n", "│ └── exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122/\n", "│ └── _emat_start_.log\n", "├── road-test-colleague.sqlitedb\n", "├── road-test-demo.db\n", "├── road-test-files/\n", "│ ├── demo-inputs-l.yml\n", "│ ├── demo-inputs-x.yml\n", "│ └── demo-inputs-x.yml.template\n", "├── road-test-model-config.yml\n", "└── road-test-scope.yml\n" ] } ], "source": [ "show_dir(fx.local_directory)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---\n", "# This file defines lever values for the files-based\n", "# Road Test example. It is intentionally a complex way\n", "# to implement this Python-based model, designed to\n", "# demonstrate how to use a files-based model called\n", "# from the command line.\n", "expand_capacity: 75\n", "amortization_period: 25\n", "interest_rate_lock: False\n", "debt_type: Paygo \n", "lane_width: 10\n", "mandatory_unused_lever: 42\n", "...\n" ] } ], "source": [ "show_file_contents(fx.local_directory, 'road-test-files', 'demo-inputs-l.yml')" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---\n", "# This file defines uncertainty values for the files-based\n", "# Road Test example. It is intentionally a complex way\n", "# to implement this Python-based model, designed to\n", "# demonstrate how to use a files-based model called\n", "# from the command line.\n", "alpha: 0.1234\n", "beta: 4.0\n", "input_flow: 100\n", "value_of_time: 0.075\n", "labor_unit_cost_expansion: 60.0\n", "materials_unit_cost_expansion: 40.0\n", "interest_rate: 0.035\n", "yield_curve: 0.01\n", "...\n" ] } ], "source": [ "show_file_contents(fx.local_directory, 'road-test-files', 'demo-inputs-x.yml')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### run" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `run` method is the place where the core model run takes place.\n", "Note that this method takes no arguments; all the input\n", "exogenous uncertainties and policy levers are delivered to the\n", "core model in the `setup` method, which will be executed prior\n", "to calling this method. This facilitates debugging, as the `setup`\n", "method can be used without the `run` method as we did above, allowing\n", "us to manually inspect the prepared files and ensure they\n", "are correct before actually running a potentially expensive model." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:06.49] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n" ] } ], "source": [ "fx.run()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `RoadTestFileModel` class includes a custom `last_run_logs` method,\n", "which displays both the \"stdout\" and \"stderr\" logs generated by the \n", "model executable during the most recent call to the `run` method.\n", "We can use this method for debugging purposes, to identify why the \n", "core model crashes (if it does crash). In this first test it did not\n", "crash, and the logs look good." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "=== STDOUT ===\n", "[2021-01-31 18:59:50,158] emat.RoadTest.INFO: running emat-road-test-demo\n", "[2021-01-31 18:59:50,164] emat.RoadTest.INFO: emat-road-test-demo completed without errors\n", "\n", "=== END OF LOG ===\n" ] } ], "source": [ "fx.last_run_logs()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### post-process\n", "\n", "There is an (optional) `post_process` step that is separate from the `run` step.\n", "\n", "Post-processing differs from the main model run in two important ways:\n", "\n", "- It can be run to efficiently generate a subset of performance measures.\n", "- It can be run based on archived model main-run core model results.\n", "\n", "Both features are designed to support workflows where new performance \n", "measures are added to the exploratory scope after the main model run(s)\n", "are completed. By allowing the `post_process` method to be run only for a \n", "subset of measures, we can avoid replicating possibly expensive \n", "post-processing steps when we have already completed them, or when they\n", "are not needed for a particular application. \n", "\n", "For example, consider an exploratory modeling activity where the scope \n", "at the time of the initial model run experiments was focused on highway\n", "measures, and transit usage was not explored extensively, and no \n", "network assignment was done for transit trips when the experiments were\n", "initially run. By creating a post-process step to run the transit \n", "network assignment, we can apply that step to existing archived results,\n", "as well as have it run automatically for future model experients\n", "where transit usage is under study, but continue to omit it for future \n", "model experients where we do not need it.\n", "\n", "An optional `measure_names` argument allows the post-processor to\n", "identify which measures need additional computational effort to generate,\n", "and to skip excluded measures that are not currently of interest, or\n", "which have already been computed and do not need to be computed again.\n", "\n", "The post processing is isolated from the main model run to allow it to\n", "be run later using archived model results. When executed directly \n", "after a core model run, it will operate on the results of the model\n", "stored in the local working directory. However, it can also be\n", "used with an optional `output_path` argument, which can be pointed at\n", "a model archive directory instead of the local working directory.\n", "\n", "A consequence of this (and an intentional limitation) is that the \n", "`post_process` method should only use files from the set of files \n", "that are or will be archived from the core model run, and not attempt\n", "to use other non-persistent temporary or intermediate files that \n", "will not be archived." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:06.52] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n" ] } ], "source": [ "fx.post_process()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At this point, the model's output performance measures should be available in one\n", "or more output files that can be read in the next step. For this example, the\n", "results are written to two separate files: 'output_1.csv.gz' and 'output.yaml'." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "build_travel_time: 60.78943107038733\n", "no_build_travel_time: 67.404\n", "time_savings: 6.614568929612666\n", "\n" ] } ], "source": [ "show_file_contents(fx.local_directory, 'road-test-files', \"Outputs\", \"output.yaml\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note in this example, some of the values in the `output_1.csv.gz` file\n", "are intentionally manipulated in a contrived manner, so that there is \n", "some work for the post-processor to do." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ",value_of_time_savings,present_cost_expansion,cost_of_capacity_expansion,net_benefits\n", "exp,1.0508604102769246,,1.3651577800056909,\n", "plain,49.60926697209499,7500.0,311.2700117047018,-261.6607447326066\n", "\n" ] } ], "source": [ "show_file_contents(fx.local_directory, 'road-test-files', \"Outputs\", \"output_1.csv.gz\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### load-measures\n", "\n", "The `load_measures` method is the place to actually reach into\n", "files in the core model's run results and extract performance\n", "measures, returning a dictionary of key-value pairs for the \n", "various performance measures. It takes an optional list giving a \n", "subset of performance measures to load, and like the `post_process` \n", "method also can be pointed at an archive location instead of loading \n", "measures from the local working directory (which is the default).\n", "The `load_measures` method should not do any post-processing\n", "of results (i.e. it should read from but not write to the model\n", "outputs directory)." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'value_of_time_savings': 49.60926697209499,\n", " 'present_cost_expansion': 7500.0,\n", " 'cost_of_capacity_expansion': 311.2700117047018,\n", " 'net_benefits': -261.6607447326066,\n", " 'build_travel_time': 60.78943107038733,\n", " 'no_build_travel_time': 67.404,\n", " 'time_savings': 6.614568929612666}" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fx.load_measures()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You may note that the implementation of `RoadTestFileModel` in the `core_files_demo` module\n", "does not actually include a `load_measures` method itself, but instead inherits this method\n", "from the `FilesCoreModel` superclass. The instructions on how to actually find the relevant\n", "performance measures for this file are instead loaded into table parsers, which are defined\n", "in the `RoadTestFileModel.__init__` constructor. There are [details and illustrations\n", "of how to write and use parsers in the file parsing examples page of the TMIP-EMAT documentation.](https://tmip-emat.github.io/source/emat.models/table_parse_example.html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### archive\n", "\n", "The `archive` method copies the relevant model output files to an archive location for \n", "longer term storage. The particular archive location is based on the experiment id\n", "for a particular experiment, and can be customized if desired by overloading the \n", "`get_experiment_archive_path` method. This customization is not done in this demo,\n", "so the default location is used." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'/var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122'" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fx.get_experiment_archive_path(parameters=params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Actually running the `archive` method should copy any relevant output files\n", "from the `model_path` of the current active model into a subdirectory of `archive_path`." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:06.55] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122\n" ] } ], "source": [ "fx.archive(params)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tmpygcky5rc/\n", "├── _emat_experiment_id_.yml\n", "├── _emat_parameters_.yml\n", "├── archive/\n", "│ └── scp_EMAT Road Test/\n", "│ └── exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122/\n", "│ ├── _emat_start_.log\n", "│ ├── demo-inputs-l.yml\n", "│ ├── demo-inputs-x.yml\n", "│ ├── demo-inputs-x.yml.template\n", "│ ├── emat-road-test.log\n", "│ ├── output.csv.gz\n", "│ ├── output.yaml\n", "│ └── Outputs/\n", "│ ├── output.yaml\n", "│ └── output_1.csv.gz\n", "├── road-test-colleague.sqlitedb\n", "├── road-test-demo.db\n", "├── road-test-files/\n", "│ ├── demo-inputs-l.yml\n", "│ ├── demo-inputs-x.yml\n", "│ ├── demo-inputs-x.yml.template\n", "│ ├── emat-road-test.log\n", "│ ├── output.csv.gz\n", "│ ├── output.yaml\n", "│ └── Outputs/\n", "│ ├── output.yaml\n", "│ └── output_1.csv.gz\n", "├── road-test-model-config.yml\n", "└── road-test-scope.yml\n" ] } ], "source": [ "show_dir(fx.local_directory)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is permissible, but not required, to simply copy the entire contents of the \n", "former to the latter, as is done in this example. However, if the current active model\n", "directory has a lot of boilerplate files that don't change with the inputs, or\n", "if it becomes full of intermediate or temporary files that definitely will never\n", "be used to compute performance measures, it can be advisable to selectively copy\n", "only relevant files. In that case, those files and whatever related sub-directory\n", "tree structure exists in the current active model should be replicated within the\n", "experiments archive directory." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Normal Operation for Running Multiple Experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this demo, we'll create a design of experiments with only 8 experiments.\n", "The `design_experiments` method of the `RoadTestFileModel` object is not defined\n", "in the custom `core_files_demo` written for this model, but rather is a generic\n", "function provide by the TMIP-EMAT main library.\n", "Real applications will typically use a larger number of experiments, but this small number\n", "is sufficient to demonstrate the operation of the tools." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
alphaamortization_periodbetadebt_typeexpand_capacityinput_flowinterest_rateinterest_rate_lockunit_cost_expansionvalue_of_timeyield_curvefree_flow_timeinitial_capacity
experiment
20.134750364.642370Rev Bond24.901018850.039219False143.5197600.0170050.00328060100
30.115907505.242315Rev Bond11.9850221140.025997True107.8367390.1453780.00965960100
40.178456303.510139Paygo72.3995521210.029066True127.8380100.067821-0.00083960100
50.110023444.887030Paygo28.5656371270.034673True125.7998290.0835930.00271460100
60.161977243.865644GO Bond82.8114591350.028634True118.9894470.0549680.00674560100
70.173449214.094118Paygo43.4761721420.033847False133.4807780.1849040.01767460100
80.141973195.331978GO Bond52.445940980.031519False99.3173500.1152280.01475260100
90.193762404.453382Rev Bond92.278968950.037907False103.5434150.0925170.01436060100
\n", "
" ], "text/plain": [ " alpha amortization_period beta debt_type \\\n", "experiment \n", "2 0.134750 36 4.642370 Rev Bond \n", "3 0.115907 50 5.242315 Rev Bond \n", "4 0.178456 30 3.510139 Paygo \n", "5 0.110023 44 4.887030 Paygo \n", "6 0.161977 24 3.865644 GO Bond \n", "7 0.173449 21 4.094118 Paygo \n", "8 0.141973 19 5.331978 GO Bond \n", "9 0.193762 40 4.453382 Rev Bond \n", "\n", " expand_capacity input_flow interest_rate interest_rate_lock \\\n", "experiment \n", "2 24.901018 85 0.039219 False \n", "3 11.985022 114 0.025997 True \n", "4 72.399552 121 0.029066 True \n", "5 28.565637 127 0.034673 True \n", "6 82.811459 135 0.028634 True \n", "7 43.476172 142 0.033847 False \n", "8 52.445940 98 0.031519 False \n", "9 92.278968 95 0.037907 False \n", "\n", " unit_cost_expansion value_of_time yield_curve free_flow_time \\\n", "experiment \n", "2 143.519760 0.017005 0.003280 60 \n", "3 107.836739 0.145378 0.009659 60 \n", "4 127.838010 0.067821 -0.000839 60 \n", "5 125.799829 0.083593 0.002714 60 \n", "6 118.989447 0.054968 0.006745 60 \n", "7 133.480778 0.184904 0.017674 60 \n", "8 99.317350 0.115228 0.014752 60 \n", "9 103.543415 0.092517 0.014360 60 \n", "\n", " initial_capacity \n", "experiment \n", "2 100 \n", "3 100 \n", "4 100 \n", "5 100 \n", "6 100 \n", "7 100 \n", "8 100 \n", "9 100 " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "design1 = fx.design_experiments(design_name='lhs_1', n_samples=8)\n", "design1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `run_experiments` command will automatically run the model once for each experiment in the named design.\n", "The demo command line version of the road test model is (intentionally) a little bit slow, so will take a few\n", "seconds to conduct these eight model experiment runs." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:06.66] MainProcess/INFO: performing 8 scenarios/policies * 1 model(s) = 8 experiments\n", "[00:06.67] MainProcess/INFO: performing experiments sequentially\n", "[00:06.67] MainProcess/INFO: RoadTestFileModel SETUP RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n", "[00:06.68] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 2 RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n", "[00:09.14] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n", "[00:09.15] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n", "[00:09.16] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_002_c9660fe2-6428-11eb-8c2b-acde48001122\n", "[00:09.17] MainProcess/INFO: RAN EXPERIMENT IN 2.50 SECONDS\n", "[00:09.17] MainProcess/INFO: 1 cases completed\n", "[00:09.18] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n", "[00:09.18] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 3 RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n", "[00:11.56] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n", "[00:11.57] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n", "[00:11.58] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_003_cae3c1f2-6428-11eb-8c2b-acde48001122\n", "[00:11.59] MainProcess/INFO: RAN EXPERIMENT IN 2.42 SECONDS\n", "[00:11.59] MainProcess/INFO: 2 cases completed\n", "[00:11.60] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n", "[00:11.60] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 4 RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n", "[00:14.00] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n", "[00:14.01] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n", "[00:14.02] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_004_cc552f1c-6428-11eb-8c2b-acde48001122\n", "[00:14.03] MainProcess/INFO: RAN EXPERIMENT IN 2.44 SECONDS\n", "[00:14.03] MainProcess/INFO: 3 cases completed\n", "[00:14.04] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n", "[00:14.04] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 5 RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n", "[00:16.46] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n", "[00:16.47] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n", "[00:16.48] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_005_cdc944c8-6428-11eb-8c2b-acde48001122\n", "[00:16.48] MainProcess/INFO: RAN EXPERIMENT IN 2.45 SECONDS\n", "[00:16.48] MainProcess/INFO: 4 cases completed\n", "[00:16.49] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n", "[00:16.50] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 6 RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n", "[00:18.99] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n", "[00:19.00] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n", "[00:19.01] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_006_cf3fd362-6428-11eb-8c2b-acde48001122\n", "[00:19.02] MainProcess/INFO: RAN EXPERIMENT IN 2.53 SECONDS\n", "[00:19.02] MainProcess/INFO: 5 cases completed\n", "[00:19.02] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n", "[00:19.03] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 7 RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n", "[00:21.37] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n", "[00:21.39] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n", "[00:21.40] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_007_d0c23324-6428-11eb-8c2b-acde48001122\n", "[00:21.40] MainProcess/INFO: RAN EXPERIMENT IN 2.38 SECONDS\n", "[00:21.40] MainProcess/INFO: 6 cases completed\n", "[00:21.41] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n", "[00:21.42] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 8 RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n", "[00:23.88] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n", "[00:23.89] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n", "[00:23.91] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_008_d22e7272-6428-11eb-8c2b-acde48001122\n", "[00:23.91] MainProcess/INFO: RAN EXPERIMENT IN 2.51 SECONDS\n", "[00:23.91] MainProcess/INFO: 7 cases completed\n", "[00:23.92] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d3ad349e-6428-11eb-8c2b-acde48001122\n", "[00:23.93] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 9 RUNID-d3ad349e-6428-11eb-8c2b-acde48001122\n", "[00:26.40] MainProcess/ERROR: ERROR in run_core_model run 9: Command '['emat-road-test-demo', '--uncs', 'demo-inputs-x.yml', '--levers', 'demo-inputs-l.yml']' returned non-zero exit status 247.\n", "[00:26.40] MainProcess/ERROR: run_core_model ABORT 9\n", "[00:26.40] MainProcess/INFO: RAN EXPERIMENT IN 2.49 SECONDS\n", "[00:26.41] MainProcess/INFO: 8 cases completed\n", "[00:26.41] MainProcess/INFO: experiments finished\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
alphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lockfree_flow_timeinitial_capacityno_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experiment
20.1347504.642370850.017005143.5197600.0392190.00328024.90101836Rev BondFalse6010063.80203161.3543182.4477133.537924-175.134011178.6719353573.788162
30.1159075.2423151140.145378107.8367390.0259970.00965911.98502250Rev BondTrue6010073.82214967.6359616.186188102.52460642.36193160.1626751292.425695
40.1784563.5101391210.067821127.8380100.029066-0.00083972.39955230PaygoTrue6010080.90595963.09026317.815696146.201784-183.229667329.4314519255.414627
50.1100234.8870301270.083593125.7998290.0346730.00271428.56563744PaygoTrue6010081.22893566.21772515.011210159.36338761.56738597.7960013593.552248
60.1619773.8656441350.054968118.9894470.0286340.00674582.81145924GO BondTrue6010091.00472563.01034327.994382207.737074-374.545620582.2826959853.689681
70.1734494.0941181420.184904133.4807780.0338470.01767443.47617221PaygoFalse60100103.73306469.97550733.757557886.351136604.765873281.5852635803.233337
80.1419735.331978980.11522899.3173500.0315190.01475252.44594019GO BondFalse6010067.64845660.8076146.84084277.249238-282.042005359.2912435208.791736
90.1937624.453382950.092517103.5434150.0379070.01436092.27896840Rev BondFalse60100NaNNaNNaNNaNNaNNaNNaN
\n", "
" ], "text/plain": [ " alpha beta input_flow value_of_time \\\n", "experiment \n", "2 0.134750 4.642370 85 0.017005 \n", "3 0.115907 5.242315 114 0.145378 \n", "4 0.178456 3.510139 121 0.067821 \n", "5 0.110023 4.887030 127 0.083593 \n", "6 0.161977 3.865644 135 0.054968 \n", "7 0.173449 4.094118 142 0.184904 \n", "8 0.141973 5.331978 98 0.115228 \n", "9 0.193762 4.453382 95 0.092517 \n", "\n", " unit_cost_expansion interest_rate yield_curve expand_capacity \\\n", "experiment \n", "2 143.519760 0.039219 0.003280 24.901018 \n", "3 107.836739 0.025997 0.009659 11.985022 \n", "4 127.838010 0.029066 -0.000839 72.399552 \n", "5 125.799829 0.034673 0.002714 28.565637 \n", "6 118.989447 0.028634 0.006745 82.811459 \n", "7 133.480778 0.033847 0.017674 43.476172 \n", "8 99.317350 0.031519 0.014752 52.445940 \n", "9 103.543415 0.037907 0.014360 92.278968 \n", "\n", " amortization_period debt_type interest_rate_lock free_flow_time \\\n", "experiment \n", "2 36 Rev Bond False 60 \n", "3 50 Rev Bond True 60 \n", "4 30 Paygo True 60 \n", "5 44 Paygo True 60 \n", "6 24 GO Bond True 60 \n", "7 21 Paygo False 60 \n", "8 19 GO Bond False 60 \n", "9 40 Rev Bond False 60 \n", "\n", " initial_capacity no_build_travel_time build_travel_time \\\n", "experiment \n", "2 100 63.802031 61.354318 \n", "3 100 73.822149 67.635961 \n", "4 100 80.905959 63.090263 \n", "5 100 81.228935 66.217725 \n", "6 100 91.004725 63.010343 \n", "7 100 103.733064 69.975507 \n", "8 100 67.648456 60.807614 \n", "9 100 NaN NaN \n", "\n", " time_savings value_of_time_savings net_benefits \\\n", "experiment \n", "2 2.447713 3.537924 -175.134011 \n", "3 6.186188 102.524606 42.361931 \n", "4 17.815696 146.201784 -183.229667 \n", "5 15.011210 159.363387 61.567385 \n", "6 27.994382 207.737074 -374.545620 \n", "7 33.757557 886.351136 604.765873 \n", "8 6.840842 77.249238 -282.042005 \n", "9 NaN NaN NaN \n", "\n", " cost_of_capacity_expansion present_cost_expansion \n", "experiment \n", "2 178.671935 3573.788162 \n", "3 60.162675 1292.425695 \n", "4 329.431451 9255.414627 \n", "5 97.796001 3593.552248 \n", "6 582.282695 9853.689681 \n", "7 281.585263 5803.233337 \n", "8 359.291243 5208.791736 \n", "9 NaN NaN " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fx.run_experiments(design_name='lhs_1')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Re-running Failed Experiments\n", "\n", "If you pay attention to the logged output, you might notice that one of the \n", "experiments (the last one) failed. We can see `NaN` values in the outputs." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
free_flow_timeinitial_capacityalphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lockno_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experiment
2601000.1347504.642370850.017005143.5197600.0392190.00328024.90101836Rev BondFalse63.80203161.3543182.4477133.537924-175.134011178.6719353573.788162
3601000.1159075.2423151140.145378107.8367390.0259970.00965911.98502250Rev BondTrue73.82214967.6359616.186188102.52460642.36193160.1626751292.425695
4601000.1784563.5101391210.067821127.8380100.029066-0.00083972.39955230PaygoTrue80.90595963.09026317.815696146.201784-183.229667329.4314519255.414627
5601000.1100234.8870301270.083593125.7998290.0346730.00271428.56563744PaygoTrue81.22893566.21772515.011210159.36338761.56738597.7960013593.552248
6601000.1619773.8656441350.054968118.9894470.0286340.00674582.81145924GO BondTrue91.00472563.01034327.994382207.737074-374.545620582.2826959853.689681
7601000.1734494.0941181420.184904133.4807780.0338470.01767443.47617221PaygoFalse103.73306469.97550733.757557886.351136604.765873281.5852635803.233337
8601000.1419735.331978980.11522899.3173500.0315190.01475252.44594019GO BondFalse67.64845660.8076146.84084277.249238-282.042005359.2912435208.791736
9601000.1937624.453382950.092517103.5434150.0379070.01436092.27896840Rev BondFalseNaNNaNNaNNaNNaNNaNNaN
\n", "
" ], "text/plain": [ " free_flow_time initial_capacity alpha beta input_flow \\\n", "experiment \n", "2 60 100 0.134750 4.642370 85 \n", "3 60 100 0.115907 5.242315 114 \n", "4 60 100 0.178456 3.510139 121 \n", "5 60 100 0.110023 4.887030 127 \n", "6 60 100 0.161977 3.865644 135 \n", "7 60 100 0.173449 4.094118 142 \n", "8 60 100 0.141973 5.331978 98 \n", "9 60 100 0.193762 4.453382 95 \n", "\n", " value_of_time unit_cost_expansion interest_rate yield_curve \\\n", "experiment \n", "2 0.017005 143.519760 0.039219 0.003280 \n", "3 0.145378 107.836739 0.025997 0.009659 \n", "4 0.067821 127.838010 0.029066 -0.000839 \n", "5 0.083593 125.799829 0.034673 0.002714 \n", "6 0.054968 118.989447 0.028634 0.006745 \n", "7 0.184904 133.480778 0.033847 0.017674 \n", "8 0.115228 99.317350 0.031519 0.014752 \n", "9 0.092517 103.543415 0.037907 0.014360 \n", "\n", " expand_capacity amortization_period debt_type \\\n", "experiment \n", "2 24.901018 36 Rev Bond \n", "3 11.985022 50 Rev Bond \n", "4 72.399552 30 Paygo \n", "5 28.565637 44 Paygo \n", "6 82.811459 24 GO Bond \n", "7 43.476172 21 Paygo \n", "8 52.445940 19 GO Bond \n", "9 92.278968 40 Rev Bond \n", "\n", " interest_rate_lock no_build_travel_time build_travel_time \\\n", "experiment \n", "2 False 63.802031 61.354318 \n", "3 True 73.822149 67.635961 \n", "4 True 80.905959 63.090263 \n", "5 True 81.228935 66.217725 \n", "6 True 91.004725 63.010343 \n", "7 False 103.733064 69.975507 \n", "8 False 67.648456 60.807614 \n", "9 False NaN NaN \n", "\n", " time_savings value_of_time_savings net_benefits \\\n", "experiment \n", "2 2.447713 3.537924 -175.134011 \n", "3 6.186188 102.524606 42.361931 \n", "4 17.815696 146.201784 -183.229667 \n", "5 15.011210 159.363387 61.567385 \n", "6 27.994382 207.737074 -374.545620 \n", "7 33.757557 886.351136 604.765873 \n", "8 6.840842 77.249238 -282.042005 \n", "9 NaN NaN NaN \n", "\n", " cost_of_capacity_expansion present_cost_expansion \n", "experiment \n", "2 178.671935 3573.788162 \n", "3 60.162675 1292.425695 \n", "4 329.431451 9255.414627 \n", "5 97.796001 3593.552248 \n", "6 582.282695 9853.689681 \n", "7 281.585263 5803.233337 \n", "8 359.291243 5208.791736 \n", "9 NaN NaN " ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = fx.db.read_experiment_all(fx.scope, 'lhs_1')\n", "results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can collect the id's of the failed experiments programmatically. To collect all the experiments that are\n", "missing any performance measure output, we can do this:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Int64Index([9], dtype='int64', name='experiment')" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fails = results.isna().any(axis=1)\n", "failed_experiment_ids = fails.index[fails]\n", "failed_experiment_ids" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When there is an error (thrown as a `subprocess.CalledProcessError`)\n", "during the execution of a `FilesCoreModel`, the output from stdout\n", "and stderr are written to log files in the archive location, instead\n", "of having the legit model outputs written there.\n", "\n", "We can see the log output by reading in the log file, like this:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2021-01-31 19:00:10,065] emat.RoadTest.INFO: running emat-road-test-demo\n", "[2021-01-31 19:00:10,068] emat.RoadTest.ERROR: Random crash, ha ha!\n", "\n" ] } ], "source": [ "error_log = os.path.join(\n", " fx.get_experiment_archive_path(9), \n", " 'error.stdout.log'\n", ")\n", "with open(error_log, 'r') as stdout:\n", " error_log_content = stdout.read()\n", " \n", "print(error_log_content)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we see the log file is explicitly taunting us about \n", "randomly crashing the model run. That's fine -- we wanted to\n", "crash the execution randomly to show what to do in this event, cause it happens \n", "sometimes. Maybe a disk filled up, or there is an intermittent\n", "license problem that causes a failure one in a while. If that's the\n", "case and we can fix it just by re-running, awesome!\n", "\n", "We can load just the failed experiments to try them again." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
free_flow_timeinitial_capacityalphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lock
experiment
9601000.1937624.453382950.092517103.5434150.0379070.0143692.27896840Rev BondFalse
\n", "
" ], "text/plain": [ " free_flow_time initial_capacity alpha beta input_flow \\\n", "experiment \n", "9 60 100 0.193762 4.453382 95 \n", "\n", " value_of_time unit_cost_expansion interest_rate yield_curve \\\n", "experiment \n", "9 0.092517 103.543415 0.037907 0.01436 \n", "\n", " expand_capacity amortization_period debt_type interest_rate_lock \n", "experiment \n", "9 92.278968 40 Rev Bond False " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "failed_experiments = fx.read_experiment_parameters(experiment_ids=failed_experiment_ids)\n", "failed_experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Normally, there is a \"short circuit\" process that will\n", "prevent re-running a core model experiment, instead the performance measure results\n", "will simply be loaded from the database, which is typically much faster than\n", "actually running the core model. But, if the performance measures stored in the\n", "database are junk, we will not want to trigger the short circuit system, and\n", "actually run the full core model again. To do so, we can disable the\n", "short circuit like and re-run the failed experiment. If it failed because of a transient error, \n", "e.g. a disk space problem that's been fixed, then perhaps we can simply re-run the model\n", "and it will work." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:26.53] MainProcess/INFO: performing 1 scenarios/policies * 1 model(s) = 1 experiments\n", "[00:26.54] MainProcess/INFO: performing experiments sequentially\n", "[00:26.54] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d53da118-6428-11eb-8c2b-acde48001122\n", "[00:26.55] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 9 RUNID-d53da118-6428-11eb-8c2b-acde48001122\n", "[00:28.93] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d53da118-6428-11eb-8c2b-acde48001122\n", "[00:28.94] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d53da118-6428-11eb-8c2b-acde48001122\n", "[00:28.95] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_009_d53da118-6428-11eb-8c2b-acde48001122\n", "[00:28.96] MainProcess/INFO: RAN EXPERIMENT IN 2.42 SECONDS\n", "[00:28.96] MainProcess/INFO: 1 cases completed\n", "[00:28.96] MainProcess/INFO: experiments finished\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
alphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lockfree_flow_timeinitial_capacityno_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experiment
90.1937624.453382950.092517103.5434150.0379070.0143692.27896840Rev BondFalse6010069.25157260.5032218.74835176.890662-385.524839462.41559554.87952
\n", "
" ], "text/plain": [ " alpha beta input_flow value_of_time \\\n", "experiment \n", "9 0.193762 4.453382 95 0.092517 \n", "\n", " unit_cost_expansion interest_rate yield_curve expand_capacity \\\n", "experiment \n", "9 103.543415 0.037907 0.01436 92.278968 \n", "\n", " amortization_period debt_type interest_rate_lock free_flow_time \\\n", "experiment \n", "9 40 Rev Bond False 60 \n", "\n", " initial_capacity no_build_travel_time build_travel_time \\\n", "experiment \n", "9 100 69.251572 60.503221 \n", "\n", " time_savings value_of_time_savings net_benefits \\\n", "experiment \n", "9 8.748351 76.890662 -385.524839 \n", "\n", " cost_of_capacity_expansion present_cost_expansion \n", "experiment \n", "9 462.4155 9554.87952 " ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fx.run_experiments(failed_experiments, allow_short_circuit=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Much better! Now we can see we have a more complete set of outputs, without the NaN's. Hooray!" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
free_flow_timeinitial_capacityalphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lockno_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experiment
2601000.1347504.642370850.017005143.5197600.0392190.00328024.90101836Rev BondFalse63.80203161.3543182.4477133.537924-175.134011178.6719353573.788162
3601000.1159075.2423151140.145378107.8367390.0259970.00965911.98502250Rev BondTrue73.82214967.6359616.186188102.52460642.36193160.1626751292.425695
4601000.1784563.5101391210.067821127.8380100.029066-0.00083972.39955230PaygoTrue80.90595963.09026317.815696146.201784-183.229667329.4314519255.414627
5601000.1100234.8870301270.083593125.7998290.0346730.00271428.56563744PaygoTrue81.22893566.21772515.011210159.36338761.56738597.7960013593.552248
6601000.1619773.8656441350.054968118.9894470.0286340.00674582.81145924GO BondTrue91.00472563.01034327.994382207.737074-374.545620582.2826959853.689681
7601000.1734494.0941181420.184904133.4807780.0338470.01767443.47617221PaygoFalse103.73306469.97550733.757557886.351136604.765873281.5852635803.233337
8601000.1419735.331978980.11522899.3173500.0315190.01475252.44594019GO BondFalse67.64845660.8076146.84084277.249238-282.042005359.2912435208.791736
9601000.1937624.453382950.092517103.5434150.0379070.01436092.27896840Rev BondFalse69.25157260.5032218.74835176.890662-385.524839462.4155009554.879520
\n", "
" ], "text/plain": [ " free_flow_time initial_capacity alpha beta input_flow \\\n", "experiment \n", "2 60 100 0.134750 4.642370 85 \n", "3 60 100 0.115907 5.242315 114 \n", "4 60 100 0.178456 3.510139 121 \n", "5 60 100 0.110023 4.887030 127 \n", "6 60 100 0.161977 3.865644 135 \n", "7 60 100 0.173449 4.094118 142 \n", "8 60 100 0.141973 5.331978 98 \n", "9 60 100 0.193762 4.453382 95 \n", "\n", " value_of_time unit_cost_expansion interest_rate yield_curve \\\n", "experiment \n", "2 0.017005 143.519760 0.039219 0.003280 \n", "3 0.145378 107.836739 0.025997 0.009659 \n", "4 0.067821 127.838010 0.029066 -0.000839 \n", "5 0.083593 125.799829 0.034673 0.002714 \n", "6 0.054968 118.989447 0.028634 0.006745 \n", "7 0.184904 133.480778 0.033847 0.017674 \n", "8 0.115228 99.317350 0.031519 0.014752 \n", "9 0.092517 103.543415 0.037907 0.014360 \n", "\n", " expand_capacity amortization_period debt_type \\\n", "experiment \n", "2 24.901018 36 Rev Bond \n", "3 11.985022 50 Rev Bond \n", "4 72.399552 30 Paygo \n", "5 28.565637 44 Paygo \n", "6 82.811459 24 GO Bond \n", "7 43.476172 21 Paygo \n", "8 52.445940 19 GO Bond \n", "9 92.278968 40 Rev Bond \n", "\n", " interest_rate_lock no_build_travel_time build_travel_time \\\n", "experiment \n", "2 False 63.802031 61.354318 \n", "3 True 73.822149 67.635961 \n", "4 True 80.905959 63.090263 \n", "5 True 81.228935 66.217725 \n", "6 True 91.004725 63.010343 \n", "7 False 103.733064 69.975507 \n", "8 False 67.648456 60.807614 \n", "9 False 69.251572 60.503221 \n", "\n", " time_savings value_of_time_savings net_benefits \\\n", "experiment \n", "2 2.447713 3.537924 -175.134011 \n", "3 6.186188 102.524606 42.361931 \n", "4 17.815696 146.201784 -183.229667 \n", "5 15.011210 159.363387 61.567385 \n", "6 27.994382 207.737074 -374.545620 \n", "7 33.757557 886.351136 604.765873 \n", "8 6.840842 77.249238 -282.042005 \n", "9 8.748351 76.890662 -385.524839 \n", "\n", " cost_of_capacity_expansion present_cost_expansion \n", "experiment \n", "2 178.671935 3573.788162 \n", "3 60.162675 1292.425695 \n", "4 329.431451 9255.414627 \n", "5 97.796001 3593.552248 \n", "6 582.282695 9853.689681 \n", "7 281.585263 5803.233337 \n", "8 359.291243 5208.791736 \n", "9 462.415500 9554.879520 " ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = fx.db.read_experiment_all(scope_name=fx.scope.name, design_name='lhs_1')\n", "results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Multiprocessing for Running Multiple Experiments\n", "\n", "The examples above are all single-process demonstrations of using TMIP-EMAT to run core model\n", "experiments. If your core model itself is multi-threaded or otherwise is designed to make \n", "full use of your multi-core CPU, or if a single core model run will otherwise max out some\n", "computational resource (e.g. RAM, disk space) then single process operation should be sufficient.\n", "\n", "If, on the other hand, your core model is such that you can run multiple independent instances of\n", "the model side-by-side on the same machine, then you could benefit from a multiprocessing \n", "approach. This can be accomplished by splitting a design of experiments over several\n", "processes that you start manually, or by using an automatic multiprocessing library such as \n", "`dask.distributed`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running a Subset of Experiments Manually\n", "\n", "Suppose, for example, you wanted to distribute the workload of running experiments over several processes,\n", "or even over several computers. If each process has file system access to the same TMIP-EMAT database of\n", "experiments, we can orchestrate these experiments in parallel by manually splitting up the processes.\n", "\n", "To begin with, we'll have one process create a complete design of experiments, and save it to the \n", "database (which happens automatically here)." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "design2 = fx.design_experiments(design_name='lhs_2', n_samples=8, random_seed=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we can create set up a copy of the same model in a different process, even on a different\n", "machine, as long as we point back to the same original database file. This implies the different\n", "process has access to the file system where the original file is stored. It is valuable to\n", "read and write to the same database file, not just a copy of the file, as this will obviate the need\n", "to sync the experimental data manually afterwards. In this demo, we'll \n", "just create a new directory to work in, but we'll point to the database in the original directory.\n", "Instead of allowing our model to implicitly create a new database file in the new directory, we'll\n", "instantiate a SQLiteDB object pointing to the original database." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:29.10] MainProcess/INFO: running script emat_db_init.sql\n", "[00:29.10] MainProcess/INFO: running script meta_model.sql\n", "[00:29.11] MainProcess/INFO: found no experiments with missing run_id's\n", "[00:29.11] MainProcess/INFO: running script emat_db_init_views.sql\n" ] } ], "source": [ "database_filename = fx.db.database_path\n", "db2 = emat.SQLiteDB(database_filename)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, `db2` is a `emat.SQLiteDB` object, which wraps a *new* connection to the *original* database.\n", "Then, we'll pass that `db2` explicitly to the new `RoadTestFileModel` constructor, which will\n", "create a complete copy of our model (other than the database) in a new directory." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:29.12] MainProcess/WARNING: changing cwd to /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon\n" ] } ], "source": [ "fx2 = core_files_demo.RoadTestFileModel(db=db2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To run a particular slice of a design of experiments, we need to load the experimental design first, \n", "and then pass that slice to the `run_experiments` function, instead of just giving the `design_name`." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "design2 = fx.read_experiment_parameters('lhs_2')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For splitting the work across a number of similarly capable processes or machines,\n", "the double-colon slice is convenient. If, for example, you are splitting the work\n", "over 4 computers, you can run each with slices `0::4`, `1::4`, `2::4`, and `3::4`.\n", "This slices in skip-step manner, so slice below will run every 4th experiment\n", "from the design, starting with experiment index 0 (i.e. the first one). " ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:29.19] MainProcess/INFO: performing 2 scenarios/policies * 1 model(s) = 2 experiments\n", "[00:29.21] MainProcess/INFO: performing experiments sequentially\n", "[00:29.21] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n", "[00:29.22] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 10 RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n", "[00:31.69] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n", "[00:31.70] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n", "[00:31.72] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/archive/scp_EMAT Road Test/exp_010_d6d5244c-6428-11eb-8c2b-acde48001122\n", "[00:31.72] MainProcess/INFO: RAN EXPERIMENT IN 2.51 SECONDS\n", "[00:31.72] MainProcess/INFO: 1 cases completed\n", "[00:31.73] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n", "[00:31.74] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 14 RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n", "[00:34.21] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n", "[00:34.22] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n", "[00:34.23] MainProcess/INFO: RoadTestFileModel ARCHIVE\n", " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/road-test-files\n", " to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/archive/scp_EMAT Road Test/exp_014_d85582c6-6428-11eb-8c2b-acde48001122\n", "[00:34.24] MainProcess/INFO: RAN EXPERIMENT IN 2.51 SECONDS\n", "[00:34.24] MainProcess/INFO: 2 cases completed\n", "[00:34.24] MainProcess/INFO: experiments finished\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
alphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lockfree_flow_timeinitial_capacityno_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experiment
100.1144503.6481041000.034746120.1964320.0283210.0015145.62192734PaygoTrue6010066.86701465.6248451.2421694.316063-17.50278421.818848675.735529
140.1325144.0162631330.107612131.9222900.0359930.01581154.08176022PaygoFalse6010084.99387364.40326920.590604294.700221-37.110353331.8105757134.589600
\n", "
" ], "text/plain": [ " alpha beta input_flow value_of_time \\\n", "experiment \n", "10 0.114450 3.648104 100 0.034746 \n", "14 0.132514 4.016263 133 0.107612 \n", "\n", " unit_cost_expansion interest_rate yield_curve expand_capacity \\\n", "experiment \n", "10 120.196432 0.028321 0.001514 5.621927 \n", "14 131.922290 0.035993 0.015811 54.081760 \n", "\n", " amortization_period debt_type interest_rate_lock free_flow_time \\\n", "experiment \n", "10 34 Paygo True 60 \n", "14 22 Paygo False 60 \n", "\n", " initial_capacity no_build_travel_time build_travel_time \\\n", "experiment \n", "10 100 66.867014 65.624845 \n", "14 100 84.993873 64.403269 \n", "\n", " time_savings value_of_time_savings net_benefits \\\n", "experiment \n", "10 1.242169 4.316063 -17.502784 \n", "14 20.590604 294.700221 -37.110353 \n", "\n", " cost_of_capacity_expansion present_cost_expansion \n", "experiment \n", "10 21.818848 675.735529 \n", "14 331.810575 7134.589600 " ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fx2.run_experiments(design2.iloc[0::4])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Because we have linked the second model instance back to the same database, after\n", "these experiments have finished we can access the results from the original `fx`\n", "instance." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
no_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experimentrun
10d6d5244c-6428-11eb-8c2b-acde4800112266.86701465.6248451.2421694.316063-17.50278421.818848675.735529
14d85582c6-6428-11eb-8c2b-acde4800112284.99387364.40326920.590604294.700221-37.110353331.8105757134.589600
\n", "
" ], "text/plain": [ " no_build_travel_time \\\n", "experiment run \n", "10 d6d5244c-6428-11eb-8c2b-acde48001122 66.867014 \n", "14 d85582c6-6428-11eb-8c2b-acde48001122 84.993873 \n", "\n", " build_travel_time \\\n", "experiment run \n", "10 d6d5244c-6428-11eb-8c2b-acde48001122 65.624845 \n", "14 d85582c6-6428-11eb-8c2b-acde48001122 64.403269 \n", "\n", " time_savings \\\n", "experiment run \n", "10 d6d5244c-6428-11eb-8c2b-acde48001122 1.242169 \n", "14 d85582c6-6428-11eb-8c2b-acde48001122 20.590604 \n", "\n", " value_of_time_savings \\\n", "experiment run \n", "10 d6d5244c-6428-11eb-8c2b-acde48001122 4.316063 \n", "14 d85582c6-6428-11eb-8c2b-acde48001122 294.700221 \n", "\n", " net_benefits \\\n", "experiment run \n", "10 d6d5244c-6428-11eb-8c2b-acde48001122 -17.502784 \n", "14 d85582c6-6428-11eb-8c2b-acde48001122 -37.110353 \n", "\n", " cost_of_capacity_expansion \\\n", "experiment run \n", "10 d6d5244c-6428-11eb-8c2b-acde48001122 21.818848 \n", "14 d85582c6-6428-11eb-8c2b-acde48001122 331.810575 \n", "\n", " present_cost_expansion \n", "experiment run \n", "10 d6d5244c-6428-11eb-8c2b-acde48001122 675.735529 \n", "14 d85582c6-6428-11eb-8c2b-acde48001122 7134.589600 " ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fx.read_experiment_measures('lhs_2')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is important to note that for this manual multiprocessing technique to work, where\n", "different processes run the model simultaneously, each process must be in a seperate \n", "Python instance (e.g. in seperate Jupyter notebooks, not in the same notebook as shown here)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Automatic Multiprocessing for Running Multiple Experiments\n", "\n", "The examples above are all essentially single-process demonstrations of using TMIP-EMAT to run core model\n", "experiments, either by running all in one single process, or by having a user manually instantiate a number \n", "of single processes. If your core model itself is multi-threaded or otherwise is designed to make \n", "full use of your multi-core CPU, or if a single core model run will otherwise max out some\n", "computational resource (e.g. RAM, disk space) then single process operation should be sufficient.\n", "\n", "If, on the other hand, your model is such that you can run multiple independent instances of\n", "the model side-by-side on the same machine, but you don't want to manage the process of manually, \n", "then you could benefit from a multiprocessing approach that uses the `dask.distributed` library. To\n", "demonstrate this, we'll create yet another small design of experiments to run." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
alphaamortization_periodbetadebt_typeexpand_capacityinput_flowinterest_rateinterest_rate_lockunit_cost_expansionvalue_of_timeyield_curvefree_flow_timeinitial_capacity
experiment
180.151341483.676012Rev Bond34.2487921390.037682False128.6276720.0048170.01207960100
190.118509415.309555Paygo76.473048890.035629True125.9943700.0918590.00598760100
200.174355184.727608Rev Bond94.2558931310.030304False139.2570070.0766680.01689660100
210.188622263.703274GO Bond96.9192941250.036478False121.1266720.1101900.01604860100
220.160916334.447292GO Bond64.2478411490.034941False104.4982160.0404060.01036460100
230.148182245.015381Paygo83.2867551280.027313False144.6029770.0613430.01413760100
240.145859433.943408Rev Bond61.4087481170.032835True107.0811500.1500780.00403260100
250.109392325.465265Paygo29.100532980.038449False142.4110650.0577560.00100360100
260.154192374.918319GO Bond55.3839651450.038841True130.1621750.1196890.00157960100
270.106310414.276815GO Bond87.2691341060.036746True113.2064450.0932810.01962360100
280.159061473.630430GO Bond52.4794311020.029875False135.0663240.0752840.01545560100
290.190069455.262368Paygo67.8849391010.026620True118.1478670.1250090.01323460100
300.123809284.504676Rev Bond39.2558731240.037005True121.9248330.0299060.01749560100
310.141084434.570515GO Bond42.644409850.028042False101.8241300.2148480.00530360100
320.116030504.772191Paygo78.7810121200.025253False119.9153950.0246830.01155060100
330.177997224.097314GO Bond92.5354701380.039246True108.7820510.1426030.00461760100
340.121416285.136051Rev Bond48.7935101120.029319False115.5233350.1138980.00031460100
350.168046473.822497Paygo12.1837801350.030787False127.2108230.1314300.00661860100
360.132624303.978002Paygo70.0015961050.025537False98.8712790.0657240.00746160100
370.139045254.317356Rev Bond85.1048771480.035198True134.9326090.045659-0.00233060100
380.196744204.412799Paygo3.928463910.026193False124.2606670.0825190.00348260100
390.164386183.559705Rev Bond2.5637791140.027669True105.5990160.0690890.01823760100
400.171876375.189379Rev Bond43.4167511340.033763True110.2403050.1027070.00966660100
410.102494364.637645GO Bond20.219566900.028766False141.0287350.0364290.01343060100
420.128699394.224275Rev Bond31.087547820.033269True132.5765260.1352450.00920460100
430.194011174.119391Rev Bond25.279767830.031359True137.5022680.0538240.00865460100
440.112179355.082739Paygo57.0019261190.039705True101.5902470.087633-0.00166660100
450.185542214.843869Paygo18.3397441430.034300True96.1906230.1696040.00267560100
460.182904313.879664GO Bond9.336654960.032018False113.6176850.100206-0.00069560100
470.136037155.422761GO Bond13.5678991080.031707True97.7687210.1600560.01862460100
\n", "
" ], "text/plain": [ " alpha amortization_period beta debt_type \\\n", "experiment \n", "18 0.151341 48 3.676012 Rev Bond \n", "19 0.118509 41 5.309555 Paygo \n", "20 0.174355 18 4.727608 Rev Bond \n", "21 0.188622 26 3.703274 GO Bond \n", "22 0.160916 33 4.447292 GO Bond \n", "23 0.148182 24 5.015381 Paygo \n", "24 0.145859 43 3.943408 Rev Bond \n", "25 0.109392 32 5.465265 Paygo \n", "26 0.154192 37 4.918319 GO Bond \n", "27 0.106310 41 4.276815 GO Bond \n", "28 0.159061 47 3.630430 GO Bond \n", "29 0.190069 45 5.262368 Paygo \n", "30 0.123809 28 4.504676 Rev Bond \n", "31 0.141084 43 4.570515 GO Bond \n", "32 0.116030 50 4.772191 Paygo \n", "33 0.177997 22 4.097314 GO Bond \n", "34 0.121416 28 5.136051 Rev Bond \n", "35 0.168046 47 3.822497 Paygo \n", "36 0.132624 30 3.978002 Paygo \n", "37 0.139045 25 4.317356 Rev Bond \n", "38 0.196744 20 4.412799 Paygo \n", "39 0.164386 18 3.559705 Rev Bond \n", "40 0.171876 37 5.189379 Rev Bond \n", "41 0.102494 36 4.637645 GO Bond \n", "42 0.128699 39 4.224275 Rev Bond \n", "43 0.194011 17 4.119391 Rev Bond \n", "44 0.112179 35 5.082739 Paygo \n", "45 0.185542 21 4.843869 Paygo \n", "46 0.182904 31 3.879664 GO Bond \n", "47 0.136037 15 5.422761 GO Bond \n", "\n", " expand_capacity input_flow interest_rate interest_rate_lock \\\n", "experiment \n", "18 34.248792 139 0.037682 False \n", "19 76.473048 89 0.035629 True \n", "20 94.255893 131 0.030304 False \n", "21 96.919294 125 0.036478 False \n", "22 64.247841 149 0.034941 False \n", "23 83.286755 128 0.027313 False \n", "24 61.408748 117 0.032835 True \n", "25 29.100532 98 0.038449 False \n", "26 55.383965 145 0.038841 True \n", "27 87.269134 106 0.036746 True \n", "28 52.479431 102 0.029875 False \n", "29 67.884939 101 0.026620 True \n", "30 39.255873 124 0.037005 True \n", "31 42.644409 85 0.028042 False \n", "32 78.781012 120 0.025253 False \n", "33 92.535470 138 0.039246 True \n", "34 48.793510 112 0.029319 False \n", "35 12.183780 135 0.030787 False \n", "36 70.001596 105 0.025537 False \n", "37 85.104877 148 0.035198 True \n", "38 3.928463 91 0.026193 False \n", "39 2.563779 114 0.027669 True \n", "40 43.416751 134 0.033763 True \n", "41 20.219566 90 0.028766 False \n", "42 31.087547 82 0.033269 True \n", "43 25.279767 83 0.031359 True \n", "44 57.001926 119 0.039705 True \n", "45 18.339744 143 0.034300 True \n", "46 9.336654 96 0.032018 False \n", "47 13.567899 108 0.031707 True \n", "\n", " unit_cost_expansion value_of_time yield_curve free_flow_time \\\n", "experiment \n", "18 128.627672 0.004817 0.012079 60 \n", "19 125.994370 0.091859 0.005987 60 \n", "20 139.257007 0.076668 0.016896 60 \n", "21 121.126672 0.110190 0.016048 60 \n", "22 104.498216 0.040406 0.010364 60 \n", "23 144.602977 0.061343 0.014137 60 \n", "24 107.081150 0.150078 0.004032 60 \n", "25 142.411065 0.057756 0.001003 60 \n", "26 130.162175 0.119689 0.001579 60 \n", "27 113.206445 0.093281 0.019623 60 \n", "28 135.066324 0.075284 0.015455 60 \n", "29 118.147867 0.125009 0.013234 60 \n", "30 121.924833 0.029906 0.017495 60 \n", "31 101.824130 0.214848 0.005303 60 \n", "32 119.915395 0.024683 0.011550 60 \n", "33 108.782051 0.142603 0.004617 60 \n", "34 115.523335 0.113898 0.000314 60 \n", "35 127.210823 0.131430 0.006618 60 \n", "36 98.871279 0.065724 0.007461 60 \n", "37 134.932609 0.045659 -0.002330 60 \n", "38 124.260667 0.082519 0.003482 60 \n", "39 105.599016 0.069089 0.018237 60 \n", "40 110.240305 0.102707 0.009666 60 \n", "41 141.028735 0.036429 0.013430 60 \n", "42 132.576526 0.135245 0.009204 60 \n", "43 137.502268 0.053824 0.008654 60 \n", "44 101.590247 0.087633 -0.001666 60 \n", "45 96.190623 0.169604 0.002675 60 \n", "46 113.617685 0.100206 -0.000695 60 \n", "47 97.768721 0.160056 0.018624 60 \n", "\n", " initial_capacity \n", "experiment \n", "18 100 \n", "19 100 \n", "20 100 \n", "21 100 \n", "22 100 \n", "23 100 \n", "24 100 \n", "25 100 \n", "26 100 \n", "27 100 \n", "28 100 \n", "29 100 \n", "30 100 \n", "31 100 \n", "32 100 \n", "33 100 \n", "34 100 \n", "35 100 \n", "36 100 \n", "37 100 \n", "38 100 \n", "39 100 \n", "40 100 \n", "41 100 \n", "42 100 \n", "43 100 \n", "44 100 \n", "45 100 \n", "46 100 \n", "47 100 " ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "design3 = fx.design_experiments(design_name='lhs_3', n_samples=30, random_seed=3)\n", "design3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The demo module is set up to facilitate distributed multiprocessing. During the `setup`\n", "step, the code detects if it is being run in a distributed \"worker\" environment instead of\n", "in a normal Python environment. If the \"worker\" environment is detected, then a copy\n", "of the entire files-based model is made into the worker's local workspace, and the model\n", "is run there instead of in the master workspace. This allows each worker to edit the files\n", "independently and simultaneously, without disturbing other parallel workers.\n", "\n", "With this small modification, we are ready to run this demo model in parallel subprocesses.\n", "to do, we will use the `async_experiments` method. Leveraging the `asyncio` Python interface\n", "will allow us to run multiple core models in parallel in the background, and monitor the\n", "progress interactively from within a Jupyter notebook." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:34.36] MainProcess/INFO: asynchronous_experiments(max_n_workers=2)\n" ] } ], "source": [ "background = fx.async_experiments(\n", " design=design3,\n", " max_n_workers=2,\n", " stagger_start=5,\n", " batch_size=1,\n", ")" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'30 runs: 30 pending'" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "background.progress() # Initially everything is pending" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:34.37] MainProcess/INFO: AsyncExperimentalDesign.run start\n", "[00:34.37] MainProcess/INFO: initializing default DistributedEvaluator.client\n", "[00:34.37] MainProcess/INFO: max_n_workers=2, actual n_workers=2\n", "[00:34.37] MainProcess/INFO: n_workers=2\n", "[00:35.31] MainProcess/INFO: completed initializing default DistributedEvaluator.client\n", "[00:37.49] MainProcess/INFO: AsyncExperimentalDesign.run dispatching experiments\n", "[00:37.50] MainProcess/INFO: performing 30 scenarios/policies * 1 model(s) = 30 experiments\n", "[00:37.52] MainProcess/INFO: experiments in asynchronous evaluator\n" ] }, { "data": { "text/plain": [ "'30 runs: 2 done, 27 pending, 1 queued'" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "await asyncio.sleep(15)\n", "background.progress()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After 15 seconds, only 3 runs have executed, as the `stagger_start`\n", "allows only one run to start every 5 seconds. Un-started runs remain in the \n", "\"pending\" status. We can see the status of all runs in `status`." ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "experiment\n", "18 done\n", "19 done\n", "20 queued\n", "21 pending\n", "22 pending\n", "23 pending\n", "24 pending\n", "25 pending\n", "26 pending\n", "27 pending\n", "28 pending\n", "29 pending\n", "30 pending\n", "31 pending\n", "32 pending\n", "33 pending\n", "34 pending\n", "35 pending\n", "36 pending\n", "37 pending\n", "38 pending\n", "39 pending\n", "40 pending\n", "41 pending\n", "42 pending\n", "43 pending\n", "44 pending\n", "45 pending\n", "46 pending\n", "47 pending\n", "dtype: object" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "background.status() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For fast running models, may find that `stagger_start` is not needed.\n", "Large and slow models, especially ones that begin with a massive file-copy\n", "operation on a hard disk, may benefit from staggering the task start times,\n", "so the first run has a clear shot at finishing disk actions promptly and\n", "moving to data processing as soon as possible." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we are dissatisfied with this pace for this small and fast model,\n", "the `stagger_start` can be changed dynamically while runs are going.\n", "If we set it to zero, all remaining pending runs will be queued \n", "immediately (or, within a second or so). Queued runs will begin \n", "as soon as a worker process is available to handle them." ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:49.53] MainProcess/INFO: AsyncExperimentalDesign.run dispatching task complete\n", "[00:50.09] MainProcess/INFO: 3 cases completed\n" ] }, { "data": { "text/plain": [ "experiment\n", "18 done\n", "19 done\n", "20 done\n", "21 queued\n", "22 queued\n", "23 queued\n", "24 queued\n", "25 queued\n", "26 queued\n", "27 queued\n", "28 queued\n", "29 queued\n", "30 queued\n", "31 queued\n", "32 queued\n", "33 queued\n", "34 queued\n", "35 queued\n", "36 queued\n", "37 queued\n", "38 queued\n", "39 queued\n", "40 queued\n", "41 queued\n", "42 queued\n", "43 queued\n", "44 queued\n", "45 queued\n", "46 queued\n", "47 queued\n", "dtype: object" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "background.stagger_start = 0\n", "await asyncio.sleep(1)\n", "background.status()" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[00:55.05] MainProcess/INFO: 6 cases completed\n", "[00:58.24] MainProcess/INFO: 9 cases completed\n", "[01:02.97] MainProcess/INFO: 12 cases completed\n" ] }, { "data": { "text/plain": [ "'30 runs: 13 done, 17 queued'" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "await asyncio.sleep(15)\n", "background.progress() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After 15 more seconds, only about 10 more runs are complete, as each run takes around 3 seconds,\n", "and we only have two workers processing the runs.\n", "\n", "The `current_results` method allows us to view the run results that are available currently.\n", "You might notice that runs are not necessarily dispatched in the order they appear\n", "in the table, but we'll get to all of them eventually." ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
free_flow_timeinitial_capacityalphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lockno_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experiment
18601000.1513413.6760121390.004817128.6276720.0376820.01207934.24879248Rev BondFalse90.46719870.31887720.14832113.491094-192.374479205.8655744405.342341
19601000.1185095.309555890.091859125.9943700.0356290.00598776.47304841PaygoTrue63.82989560.1876873.64220829.776845-243.694139273.4709849635.173464
20601000.1743554.7276081310.076668139.2570070.0303040.01689694.25589318Rev BondFalse97.49706461.62434035.872724360.289988-600.998696961.28868413125.793539
21601000.1886223.7032741250.110190121.1266720.0364780.01604896.91929426GO BondFalseNaNNaNNaNNaNNaNNaNNaN
22601000.1609164.4472921490.040406104.4982160.0349410.01036464.24784133GO BondFalseNaNNaNNaNNaNNaNNaNNaN
23601000.1481825.0153811280.061343144.6029770.0273130.01413783.28675524PaygoFalseNaNNaNNaNNaNNaNNaNNaN
24601000.1458593.9434081170.150078107.0811500.0328350.00403261.40874843Rev BondTrueNaNNaNNaNNaNNaNNaNNaN
25601000.1093925.465265980.057756142.4110650.0384490.00100329.10053232PaygoFalseNaNNaNNaNNaNNaNNaNNaN
26601000.1541924.9183191450.119689130.1621750.0388410.00157955.38396537GO BondTrueNaNNaNNaNNaNNaNNaNNaN
27601000.1063104.2768151060.093281113.2064450.0367460.01962387.26913441GO BondTrueNaNNaNNaNNaNNaNNaNNaN
28601000.1590613.6304301020.075284135.0663240.0298750.01545552.47943147GO BondFalseNaNNaNNaNNaNNaNNaNNaN
29601000.1900695.2623681010.125009118.1478670.0266200.01323467.88493945PaygoTrueNaNNaNNaNNaNNaNNaNNaN
30601000.1238094.5046761240.029906121.9248330.0370050.01749539.25587328Rev BondTrueNaNNaNNaNNaNNaNNaNNaN
31601000.1410844.570515850.214848101.8241300.0280420.00530342.64440943GO BondFalseNaNNaNNaNNaNNaNNaNNaN
32601000.1160304.7721911200.024683119.9153950.0252530.01155078.78101250PaygoFalseNaNNaNNaNNaNNaNNaNNaN
\n", "
" ], "text/plain": [ " free_flow_time initial_capacity alpha beta input_flow \\\n", "experiment \n", "18 60 100 0.151341 3.676012 139 \n", "19 60 100 0.118509 5.309555 89 \n", "20 60 100 0.174355 4.727608 131 \n", "21 60 100 0.188622 3.703274 125 \n", "22 60 100 0.160916 4.447292 149 \n", "23 60 100 0.148182 5.015381 128 \n", "24 60 100 0.145859 3.943408 117 \n", "25 60 100 0.109392 5.465265 98 \n", "26 60 100 0.154192 4.918319 145 \n", "27 60 100 0.106310 4.276815 106 \n", "28 60 100 0.159061 3.630430 102 \n", "29 60 100 0.190069 5.262368 101 \n", "30 60 100 0.123809 4.504676 124 \n", "31 60 100 0.141084 4.570515 85 \n", "32 60 100 0.116030 4.772191 120 \n", "\n", " value_of_time unit_cost_expansion interest_rate yield_curve \\\n", "experiment \n", "18 0.004817 128.627672 0.037682 0.012079 \n", "19 0.091859 125.994370 0.035629 0.005987 \n", "20 0.076668 139.257007 0.030304 0.016896 \n", "21 0.110190 121.126672 0.036478 0.016048 \n", "22 0.040406 104.498216 0.034941 0.010364 \n", "23 0.061343 144.602977 0.027313 0.014137 \n", "24 0.150078 107.081150 0.032835 0.004032 \n", "25 0.057756 142.411065 0.038449 0.001003 \n", "26 0.119689 130.162175 0.038841 0.001579 \n", "27 0.093281 113.206445 0.036746 0.019623 \n", "28 0.075284 135.066324 0.029875 0.015455 \n", "29 0.125009 118.147867 0.026620 0.013234 \n", "30 0.029906 121.924833 0.037005 0.017495 \n", "31 0.214848 101.824130 0.028042 0.005303 \n", "32 0.024683 119.915395 0.025253 0.011550 \n", "\n", " expand_capacity amortization_period debt_type \\\n", "experiment \n", "18 34.248792 48 Rev Bond \n", "19 76.473048 41 Paygo \n", "20 94.255893 18 Rev Bond \n", "21 96.919294 26 GO Bond \n", "22 64.247841 33 GO Bond \n", "23 83.286755 24 Paygo \n", "24 61.408748 43 Rev Bond \n", "25 29.100532 32 Paygo \n", "26 55.383965 37 GO Bond \n", "27 87.269134 41 GO Bond \n", "28 52.479431 47 GO Bond \n", "29 67.884939 45 Paygo \n", "30 39.255873 28 Rev Bond \n", "31 42.644409 43 GO Bond \n", "32 78.781012 50 Paygo \n", "\n", " interest_rate_lock no_build_travel_time build_travel_time \\\n", "experiment \n", "18 False 90.467198 70.318877 \n", "19 True 63.829895 60.187687 \n", "20 False 97.497064 61.624340 \n", "21 False NaN NaN \n", "22 False NaN NaN \n", "23 False NaN NaN \n", "24 True NaN NaN \n", "25 False NaN NaN \n", "26 True NaN NaN \n", "27 True NaN NaN \n", "28 False NaN NaN \n", "29 True NaN NaN \n", "30 True NaN NaN \n", "31 False NaN NaN \n", "32 False NaN NaN \n", "\n", " time_savings value_of_time_savings net_benefits \\\n", "experiment \n", "18 20.148321 13.491094 -192.374479 \n", "19 3.642208 29.776845 -243.694139 \n", "20 35.872724 360.289988 -600.998696 \n", "21 NaN NaN NaN \n", "22 NaN NaN NaN \n", "23 NaN NaN NaN \n", "24 NaN NaN NaN \n", "25 NaN NaN NaN \n", "26 NaN NaN NaN \n", "27 NaN NaN NaN \n", "28 NaN NaN NaN \n", "29 NaN NaN NaN \n", "30 NaN NaN NaN \n", "31 NaN NaN NaN \n", "32 NaN NaN NaN \n", "\n", " cost_of_capacity_expansion present_cost_expansion \n", "experiment \n", "18 205.865574 4405.342341 \n", "19 273.470984 9635.173464 \n", "20 961.288684 13125.793539 \n", "21 NaN NaN \n", "22 NaN NaN \n", "23 NaN NaN \n", "24 NaN NaN \n", "25 NaN NaN \n", "26 NaN NaN \n", "27 NaN NaN \n", "28 NaN NaN \n", "29 NaN NaN \n", "30 NaN NaN \n", "31 NaN NaN \n", "32 NaN NaN " ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "background.current_results().head(15)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we want to simply block the Python interpreter until the runs are done,\n", "we can do so by `await`ing the `final_results`." ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "df86a0e2623b41e1a6b2a1d1ca075067", "version_major": 2, "version_minor": 0 }, "text/plain": [ "IntProgress(value=13, max=30)" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[01:06.21] MainProcess/INFO: 15 cases completed\n", "[01:11.06] MainProcess/INFO: 18 cases completed\n", "[01:14.23] MainProcess/INFO: 21 cases completed\n", "[01:19.02] MainProcess/INFO: 24 cases completed\n", "[01:22.12] MainProcess/INFO: 27 cases completed\n", "[01:26.65] MainProcess/INFO: 30 cases completed\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
free_flow_timeinitial_capacityalphabetainput_flowvalue_of_timeunit_cost_expansioninterest_rateyield_curveexpand_capacityamortization_perioddebt_typeinterest_rate_lockno_build_travel_timebuild_travel_timetime_savingsvalue_of_time_savingsnet_benefitscost_of_capacity_expansionpresent_cost_expansion
experiment
18601000.1513413.6760121390.004817128.6276720.0376820.01207934.24879248Rev BondFalse90.46719870.31887720.14832113.491094-192.374479205.8655744405.342341
19601000.1185095.309555890.091859125.9943700.0356290.00598776.47304841PaygoTrue63.82989560.1876873.64220829.776845-243.694139273.4709849635.173464
20601000.1743554.7276081310.076668139.2570070.0303040.01689694.25589318Rev BondFalse97.49706461.62434035.872724360.289988-600.998696961.28868413125.793539
21601000.1886223.7032741250.110190121.1266720.0364780.01604896.91929426GO BondFalse85.85997362.10280023.757173327.226070-334.329873661.55594211739.511583
22601000.1609164.4472921490.040406104.4982160.0349410.01036464.24784133GO BondFalse116.88023666.25996050.620276304.758181-30.577074335.3352546713.784698
23601000.1481825.0153811280.061343144.6029770.0273130.01413783.28675524PaygoFalse90.66518461.46873429.196450229.247199-288.854311518.10151012043.512651
24601000.1458593.9434081170.150078107.0811500.0328350.00403261.40874843Rev BondTrue76.25432162.46052313.793797242.207489-70.587693312.7951836575.719357
25601000.1093925.465265980.057756142.4110650.0384490.00100329.10053232PaygoFalse65.87739161.4552364.42215525.029593-115.116663140.1462564144.237736
26601000.1541924.9183191450.119689130.1621750.0388410.00157955.38396537GO BondTrue117.52706566.58378350.943282884.112291539.879198344.2330947208.897323
27601000.1063104.2768151060.093281113.2064450.0367460.01962387.26913441GO BondTrue68.18379560.5593297.62446675.389314-381.495063456.8843769879.428427
28601000.1590613.6304301020.075284135.0663240.0298750.01545552.47943147GO BondFalse70.25506462.2171898.03787561.722934-256.677328318.4002627088.203858
29601000.1900695.2623681010.125009118.1478670.0266200.01323467.88493945PaygoTrue72.01718060.78651711.230663141.797161-73.766382215.5635438020.460747
30601000.1238094.5046761240.029906121.9248330.0370050.01749539.25587328Rev BondTrue79.57663564.40458315.17205256.263770-210.531699266.7954694786.265822
31601000.1410844.570515850.214848101.8241300.0280420.00530342.64440943GO BondFalse64.02752560.7943553.23317059.044512-139.377225198.4217374342.229844
32601000.1160304.7721911200.024683119.9153950.0252530.01155078.78101250PaygoFalse76.61838161.03863515.57974746.147516-194.872685241.0202019447.056124
33601000.1779974.0973141380.142603108.7820510.0392460.00461792.53547022GO BondTrue99.96608162.72873737.237343732.799774104.331664628.46811010066.198296
34601000.1214165.1360511120.113898115.5233350.0293190.00031448.79351028Rev BondFalse73.03803961.69360511.344433144.715681-169.489549314.2052305636.788951
35601000.1680463.8224971350.131430127.2108230.0307870.00661812.18378047PaygoFalse91.75259680.46075611.291840200.351591159.63538940.7162011549.908642
36601000.1326243.9780021050.06572498.8712790.0255370.00746170.00159630PaygoFalse69.66192861.1703658.49156358.600606-187.746392246.3469986921.147330
37601000.1390454.3173561480.045659134.9326090.035198-0.00233085.10487725Rev BondTrue105.33021863.17577442.154444284.861228-395.777779680.63900711483.423089
38601000.1967444.412799910.082519124.2606670.0261930.0034823.92846320PaygoFalse67.78592266.5684681.2174559.142197-15.63324424.775440488.153405
39601000.1643863.5597051140.069089105.5990160.0276690.0182372.56377918Rev BondTrue75.72462474.3696041.35501910.672342-9.15519219.827534270.732536
40601000.1718765.1893791340.102707110.2403050.0337630.00966643.41675137Rev BondTrue107.09352967.24945039.844079548.362942311.279196237.0837464786.275850
41601000.1024944.637645900.036429141.0287350.0287660.01343020.21956636GO BondFalse63.77261261.6060012.1666117.103389-130.417635137.5210242851.539859
42601000.1286994.224275820.135245132.5765260.0332690.00920431.08754739Rev BondTrue63.33927161.0642412.27503025.230374-175.639567200.8699424121.478933
43601000.1940114.119391830.053824137.5022680.0313590.00865425.27976717Rev BondTrue65.40292662.1351043.26782114.598542-250.630628265.2291693476.025296
44601000.1121795.0827391190.087633101.5902470.039705-0.00166657.00192635PaygoTrue76.29473961.64555914.649179152.766854-30.252812183.0196665790.839697
45601000.1855424.8438691430.16960496.1906230.0343000.00267518.33974421PaygoTrue122.95374387.84751135.106232851.444399765.84594885.5984511764.111444
46601000.1829043.879664960.100206113.6176850.032018-0.0006959.33665431GO BondFalse69.36681166.6251172.74169426.374431-28.13307554.5075061060.809021
47601000.1360375.4227611080.16005697.7687210.0317070.01862413.56789915GO BondTrue72.38964066.2147096.174931106.740422-2.377699109.1181211326.516148
\n", "
" ], "text/plain": [ " free_flow_time initial_capacity alpha beta input_flow \\\n", "experiment \n", "18 60 100 0.151341 3.676012 139 \n", "19 60 100 0.118509 5.309555 89 \n", "20 60 100 0.174355 4.727608 131 \n", "21 60 100 0.188622 3.703274 125 \n", "22 60 100 0.160916 4.447292 149 \n", "23 60 100 0.148182 5.015381 128 \n", "24 60 100 0.145859 3.943408 117 \n", "25 60 100 0.109392 5.465265 98 \n", "26 60 100 0.154192 4.918319 145 \n", "27 60 100 0.106310 4.276815 106 \n", "28 60 100 0.159061 3.630430 102 \n", "29 60 100 0.190069 5.262368 101 \n", "30 60 100 0.123809 4.504676 124 \n", "31 60 100 0.141084 4.570515 85 \n", "32 60 100 0.116030 4.772191 120 \n", "33 60 100 0.177997 4.097314 138 \n", "34 60 100 0.121416 5.136051 112 \n", "35 60 100 0.168046 3.822497 135 \n", "36 60 100 0.132624 3.978002 105 \n", "37 60 100 0.139045 4.317356 148 \n", "38 60 100 0.196744 4.412799 91 \n", "39 60 100 0.164386 3.559705 114 \n", "40 60 100 0.171876 5.189379 134 \n", "41 60 100 0.102494 4.637645 90 \n", "42 60 100 0.128699 4.224275 82 \n", "43 60 100 0.194011 4.119391 83 \n", "44 60 100 0.112179 5.082739 119 \n", "45 60 100 0.185542 4.843869 143 \n", "46 60 100 0.182904 3.879664 96 \n", "47 60 100 0.136037 5.422761 108 \n", "\n", " value_of_time unit_cost_expansion interest_rate yield_curve \\\n", "experiment \n", "18 0.004817 128.627672 0.037682 0.012079 \n", "19 0.091859 125.994370 0.035629 0.005987 \n", "20 0.076668 139.257007 0.030304 0.016896 \n", "21 0.110190 121.126672 0.036478 0.016048 \n", "22 0.040406 104.498216 0.034941 0.010364 \n", "23 0.061343 144.602977 0.027313 0.014137 \n", "24 0.150078 107.081150 0.032835 0.004032 \n", "25 0.057756 142.411065 0.038449 0.001003 \n", "26 0.119689 130.162175 0.038841 0.001579 \n", "27 0.093281 113.206445 0.036746 0.019623 \n", "28 0.075284 135.066324 0.029875 0.015455 \n", "29 0.125009 118.147867 0.026620 0.013234 \n", "30 0.029906 121.924833 0.037005 0.017495 \n", "31 0.214848 101.824130 0.028042 0.005303 \n", "32 0.024683 119.915395 0.025253 0.011550 \n", "33 0.142603 108.782051 0.039246 0.004617 \n", "34 0.113898 115.523335 0.029319 0.000314 \n", "35 0.131430 127.210823 0.030787 0.006618 \n", "36 0.065724 98.871279 0.025537 0.007461 \n", "37 0.045659 134.932609 0.035198 -0.002330 \n", "38 0.082519 124.260667 0.026193 0.003482 \n", "39 0.069089 105.599016 0.027669 0.018237 \n", "40 0.102707 110.240305 0.033763 0.009666 \n", "41 0.036429 141.028735 0.028766 0.013430 \n", "42 0.135245 132.576526 0.033269 0.009204 \n", "43 0.053824 137.502268 0.031359 0.008654 \n", "44 0.087633 101.590247 0.039705 -0.001666 \n", "45 0.169604 96.190623 0.034300 0.002675 \n", "46 0.100206 113.617685 0.032018 -0.000695 \n", "47 0.160056 97.768721 0.031707 0.018624 \n", "\n", " expand_capacity amortization_period debt_type \\\n", "experiment \n", "18 34.248792 48 Rev Bond \n", "19 76.473048 41 Paygo \n", "20 94.255893 18 Rev Bond \n", "21 96.919294 26 GO Bond \n", "22 64.247841 33 GO Bond \n", "23 83.286755 24 Paygo \n", "24 61.408748 43 Rev Bond \n", "25 29.100532 32 Paygo \n", "26 55.383965 37 GO Bond \n", "27 87.269134 41 GO Bond \n", "28 52.479431 47 GO Bond \n", "29 67.884939 45 Paygo \n", "30 39.255873 28 Rev Bond \n", "31 42.644409 43 GO Bond \n", "32 78.781012 50 Paygo \n", "33 92.535470 22 GO Bond \n", "34 48.793510 28 Rev Bond \n", "35 12.183780 47 Paygo \n", "36 70.001596 30 Paygo \n", "37 85.104877 25 Rev Bond \n", "38 3.928463 20 Paygo \n", "39 2.563779 18 Rev Bond \n", "40 43.416751 37 Rev Bond \n", "41 20.219566 36 GO Bond \n", "42 31.087547 39 Rev Bond \n", "43 25.279767 17 Rev Bond \n", "44 57.001926 35 Paygo \n", "45 18.339744 21 Paygo \n", "46 9.336654 31 GO Bond \n", "47 13.567899 15 GO Bond \n", "\n", " interest_rate_lock no_build_travel_time build_travel_time \\\n", "experiment \n", "18 False 90.467198 70.318877 \n", "19 True 63.829895 60.187687 \n", "20 False 97.497064 61.624340 \n", "21 False 85.859973 62.102800 \n", "22 False 116.880236 66.259960 \n", "23 False 90.665184 61.468734 \n", "24 True 76.254321 62.460523 \n", "25 False 65.877391 61.455236 \n", "26 True 117.527065 66.583783 \n", "27 True 68.183795 60.559329 \n", "28 False 70.255064 62.217189 \n", "29 True 72.017180 60.786517 \n", "30 True 79.576635 64.404583 \n", "31 False 64.027525 60.794355 \n", "32 False 76.618381 61.038635 \n", "33 True 99.966081 62.728737 \n", "34 False 73.038039 61.693605 \n", "35 False 91.752596 80.460756 \n", "36 False 69.661928 61.170365 \n", "37 True 105.330218 63.175774 \n", "38 False 67.785922 66.568468 \n", "39 True 75.724624 74.369604 \n", "40 True 107.093529 67.249450 \n", "41 False 63.772612 61.606001 \n", "42 True 63.339271 61.064241 \n", "43 True 65.402926 62.135104 \n", "44 True 76.294739 61.645559 \n", "45 True 122.953743 87.847511 \n", "46 False 69.366811 66.625117 \n", "47 True 72.389640 66.214709 \n", "\n", " time_savings value_of_time_savings net_benefits \\\n", "experiment \n", "18 20.148321 13.491094 -192.374479 \n", "19 3.642208 29.776845 -243.694139 \n", "20 35.872724 360.289988 -600.998696 \n", "21 23.757173 327.226070 -334.329873 \n", "22 50.620276 304.758181 -30.577074 \n", "23 29.196450 229.247199 -288.854311 \n", "24 13.793797 242.207489 -70.587693 \n", "25 4.422155 25.029593 -115.116663 \n", "26 50.943282 884.112291 539.879198 \n", "27 7.624466 75.389314 -381.495063 \n", "28 8.037875 61.722934 -256.677328 \n", "29 11.230663 141.797161 -73.766382 \n", "30 15.172052 56.263770 -210.531699 \n", "31 3.233170 59.044512 -139.377225 \n", "32 15.579747 46.147516 -194.872685 \n", "33 37.237343 732.799774 104.331664 \n", "34 11.344433 144.715681 -169.489549 \n", "35 11.291840 200.351591 159.635389 \n", "36 8.491563 58.600606 -187.746392 \n", "37 42.154444 284.861228 -395.777779 \n", "38 1.217455 9.142197 -15.633244 \n", "39 1.355019 10.672342 -9.155192 \n", "40 39.844079 548.362942 311.279196 \n", "41 2.166611 7.103389 -130.417635 \n", "42 2.275030 25.230374 -175.639567 \n", "43 3.267821 14.598542 -250.630628 \n", "44 14.649179 152.766854 -30.252812 \n", "45 35.106232 851.444399 765.845948 \n", "46 2.741694 26.374431 -28.133075 \n", "47 6.174931 106.740422 -2.377699 \n", "\n", " cost_of_capacity_expansion present_cost_expansion \n", "experiment \n", "18 205.865574 4405.342341 \n", "19 273.470984 9635.173464 \n", "20 961.288684 13125.793539 \n", "21 661.555942 11739.511583 \n", "22 335.335254 6713.784698 \n", "23 518.101510 12043.512651 \n", "24 312.795183 6575.719357 \n", "25 140.146256 4144.237736 \n", "26 344.233094 7208.897323 \n", "27 456.884376 9879.428427 \n", "28 318.400262 7088.203858 \n", "29 215.563543 8020.460747 \n", "30 266.795469 4786.265822 \n", "31 198.421737 4342.229844 \n", "32 241.020201 9447.056124 \n", "33 628.468110 10066.198296 \n", "34 314.205230 5636.788951 \n", "35 40.716201 1549.908642 \n", "36 246.346998 6921.147330 \n", "37 680.639007 11483.423089 \n", "38 24.775440 488.153405 \n", "39 19.827534 270.732536 \n", "40 237.083746 4786.275850 \n", "41 137.521024 2851.539859 \n", "42 200.869942 4121.478933 \n", "43 265.229169 3476.025296 \n", "44 183.019666 5790.839697 \n", "45 85.598451 1764.111444 \n", "46 54.507506 1060.809021 \n", "47 109.118121 1326.516148 " ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "await background.final_results()" ] } ], "metadata": { "jupytext": { "formats": "ipynb,py:percent" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6" }, "pycharm": { "stem_cell": { "cell_type": "raw", "metadata": { "collapsed": false }, "source": [] } }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 4 }