import emat
import yaml
import json

MappingParser Example

In this notebook, we will illustrate the use of a MappingParser with a few simple examples.

from emat.model.core_files.parsers import (
    MappingParser,
    key
)

Parsing a YAML File

First, let’s consider a MappingParser for extracting values from a simple YAML file of traffic counts by time period. We’ll begin by writing such a table as a temporary file to be processed:

sample_file_yaml = """
---
LinkID: 123
LinkName: Yellow Brick Rd.
Toll: 0.30
Count_AM: 3498
Count_MD: 2340
Count_PM: 3821
Count_EV: 1820
...
"""

with open('/tmp/emat_sample_file.yml', 'wt') as f:
    f.write(sample_file_yaml)

If we wanted to read this YAML file one time, we could easily do so using yaml.safe_load:

with open('/tmp/emat_sample_file.yml', 'rt') as fi:
    mapping = yaml.safe_load(fi)

mapping

{'LinkID': 123,
 'LinkName': 'Yellow Brick Rd.',
 'Toll': 0.3,
 'Count_AM': 3498,
 'Count_MD': 2340,
 'Count_PM': 3821,
 'Count_EV': 1820}

It is then simple to manually extract individual values by label, or by position, or we could extract a row total to get a daily total count for a link, or take the mean of a column:

{
    'AM': mapping['Count_AM'],  # one key
    'PM': mapping['Count_PM'],
    'OffPeak': mapping['Count_MD'] + mapping['Count_EV'],  # adding together keys
}

{'AM': 3498, 'PM': 3821, 'OffPeak': 4160}

The MappingParser object makes it easy to combine these instructions to extract the same values from the same file in any model run.

parser = MappingParser(
    'emat_sample_file.yml',
    {
        'AM': key['Count_AM'],  # one key
        'PM': key['Count_PM'],
        'OffPeak': key['Count_MD'] + key['Count_EV'],  # adding together keys
    },
)

We can now execute all these instructions by using the read method of the parser.

parser.read(from_dir='/tmp')

{'AM': 3498.0, 'PM': 3821.0, 'OffPeak': 4160.0}

Using the MappingParser has some advantages over just writing a custom function for each table to be processed. The most important is that we do not need to actually parse anything to access the names of the keys available in the parser’s output.

parser.measure_names

['AM', 'OffPeak', 'PM']

Parsing a JSON File

The default format for a MappingParser input file is YAML, which conveniently can also be used to read performace measures from a JSON file.

with open('/tmp/emat_sample_file.json', 'wt') as f:
    json.dump(mapping, f)

parser = MappingParser(
    'emat_sample_file.json',
    {
        'AM': key['Count_AM'],  # one key
        'PM': key['Count_PM'],
        'OffPeak': key['Count_MD'] + key['Count_EV'],  # adding together keys
    },
)

parser.read(from_dir='/tmp')

{'AM': 3498.0, 'PM': 3821.0, 'OffPeak': 4160.0}

Parsing other File Formats

The MappingParser can also be used for other file types that can be read into a simple Python mapping. For example, consider a mapping encoded as a msgpack.

import msgpack

with open('/tmp/emat_sample_file.msgpk', 'wb') as f:
    msgpack.dump(mapping, f)

To parse this file, we’ll need to write a small reader function that takes a filename and returns the raw mapping.

def msgpack_load(filename):
    with open(filename, 'rb') as fi:
        return msgpack.load(fi)

Then we provide that reader function in the reader_method argument when constucting the MappingParser.

parser = MappingParser(
    'emat_sample_file.msgpk',
    {
        'AM': key['Count_AM'],  # one key
        'PM': key['Count_PM'],
        'OffPeak': key['Count_MD'] + key['Count_EV'],  # adding together keys
    },
    reader_method=msgpack_load
)

parser.read(from_dir='/tmp')

{'AM': 3498.0, 'PM': 3821.0, 'OffPeak': 4160.0}