credit.datasets.gen_1.downscaling_dataset

credit.datasets.gen_1.downscaling_dataset#

Attributes#

Array

Classes#

Sample

Simple class for structuring data for the ML model.

DownscalingDataset

Class that wrangles a collection of DataMaps and their

Module Contents#

credit.datasets.gen_1.downscaling_dataset.Array#
class credit.datasets.gen_1.downscaling_dataset.Sample#

Bases: TypedDict

Simple class for structuring data for the ML model.

x = predictor (input) data y = predictand (target) data

Using typing.TypedDict gives us several advantages:

  1. Single ‘source of truth’ for the type and documentation of each example.

  2. A static type checker can check the types are correct.

Instead of TypedDict, we could use typing.NamedTuple, which would provide runtime checks, but the deal-breaker with Tuples is that they’re immutable so we cannot change the values in the transforms.

input: Array#
target: Array#
class credit.datasets.gen_1.downscaling_dataset.DownscalingDataset#

Bases: torch.utils.data.Dataset

Class that wrangles a collection of DataMaps and their associated DataTransforms for use in ML. Intended to be initialized with **conf[‘data’]; see config/downscaling.yml for an example.

Parameters:

  • rootpath (str) – pathway to the files

  • history_len (int) – number of input timesteps for training

  • forecast_len (int) – number of output timesteps for training

  • valid_history_len (int) – number of input timesteps for validation

  • valid_forecast_len (int) – number of output timesteps for validation all _history_len and _forecast_len arguments default to 1

  • first_date (YYYY-MM-DD string) – starting point of dataset

  • last_date – (YYYY-MM-DD string): ending point of dataset Use first_date and last_date to define a subset of the dataset, e.g., to use 1980-2000 for training and 2001-2010 for testing. Default to first and last timesteps in the files, respectively.

  • datasets (nested dict) – dicts of parameters for initializing DataMaps and their corresponding DataTransforms via **kwargs. These dicts are replaced with actual objects during initialization.

  • image_width (int) – width of dataset in gridcells

  • image_height (int) – height of dataset in gridcells image_width and _height are used to automatically resize different constituent datasets to the same size via expansion & padding. E.g., if declared image size is 120x120, a 55x55 dataset would be expanded by a factor of 2 and padded by 10 to match. Defaults to the width (height) of the widest (tallest) dataset.

  • get_time_from (str) – name of the dataset to pull time coordinates from when creating a sample; used when writing output to netcdf. Defaults to first non-static dataset that has boundary variables.

  • transform (bool) – apply normalizing transforms to samples? Defaults to True

Variables:

  • sample_len – number of timesteps in a sample (sample_len = history_len + forecast_len)

  • len – number of samples in the dataset, which will be less than the number of timesteps in the netcdf files, due to first_date/last_date subsetting and sample_len > 1.

  • mode –

    determines which variables are returned when sampling. Valid values are ‘train’, ‘test’, and ‘predict’. Defaults to ‘train’.

    train: boundary & prognostic variables for input timesteps,

    prognostic and diagnostic for target timesteps

    test: boundary variables for input timesteps,

    prognostic and diagnostic for target timesteps

    predict: boundary variables for input timesteps,

    nothing for target timesteps

  • output – determines the format of samples; see __getitem__ for details. Valid values: ‘by_dset’, ‘by_io’, ‘tensor’. Defaults to ‘tensor’.

  • arrangement – a pandas dataframe that defines how variables are ordered in the tensor. Columns: dataset, dim, var, usage, name

  • tnames – a list of names (structured dataset.var[.z-level]) corresponding to the channels in an output tensor (i.e., prognostic and diagnostic variables only).

rootpath: str#
history_len: int = 1#
forecast_len: int = 1#
valid_history_len: int = 1#
valid_forecast_len: int = 1#
first_date: str = None#
last_date: str = None#
datasets: Dict#
image_width: int = None#
image_height: int = None#
transform: bool = True#
get_time_from: str = None#
__post_init__()#
__len__()#

Number of samples in the dataset. Note that this is smaller than the number of timesteps in the files, both because first_date and last_date may not cover the full range, and because sample length = history_len + forecast_len and ~.__getitem__ only returns complete samples.

__getitem__(index)#

Gets the index’th sample from the dataset. The value of the ‘output’ attribute controls the format of the returned object.

~.output == ‘by_dset’ returns a nested dict structured [dataset][usage][variable] (the format returned by getdata()); the leaf elements are numpy ndarrays covering the full time period (history_len+forecast_len).

~.output == ‘by_io’ splits the ndarrays into history and forecast periods and reorganizes the nested dict to [input/target][dataset.variable] (the format returned by rearrange() and taken as input by to_tensor()). The variables are ordered:

  • first: boundary > prognostic > diagnostic (in-only > in-out > out-only)

  • then: static > 2D > 3D

  • then: order that datasets are defined in config

  • then: alphabetical by variable name

~.output == ‘tensor’ stacks the ndarrays in the z-dimension and converts them to a pair of pyTorch tensors (input and target), returning them as a Sample. It also includes the associated time coordinates in the sample

getdata(dset, index)#

gets data for the index’th sample from dataset dset. Returns a nested dict organized [dataset][usage][variable].

rearrange(items)#

Rearranges a nested dict of data from [dataset][usage][var] to [input/target][dataset.var]. Elements returned depend on ~.mode:

  • train: input contains boundary and prognostic variables, target contains prognostic and diagnostic variables.

  • init: input contains boundary and prognostic variables, target is empty.

  • infer: input contains boundary variables, target is empty.

to_tensor(sample)#

Takes a nested dict organized [input/target][vars], with data arrays (numpy ndarrays) dimensioned [T, Z, Y, X].

Stacks variables along the z-dimension (i.e., so different z-levels of a 3D variable are treated as different variables).

Combines variable stacks into tensors ordered [V, T, Y, X] and returns a Sample where x is historical / input data and y is forecast / target data.

revert(prediction)#

Converts a tensor (ML model output) back into nested dict of numpy arrays (i.e, reverses the getdata -> rearrange -> to_tensor pipeline).

property mode: str#
property output: str#