credit.transforms

credit.transforms#

transforms.py provides transforms.

NormalizeState

Normalize_ERA5_and_Forcing

BridgescalerScaleState

NormalizeState_Quantile

NormalizeTendency

ToTensor

ToTensor_ERA5_and_Forcing

NormalizeState_Quantile_Bridgescalar

ToTensor_BridgeScaler

Attributes#

logger

Classes#

`NormalizeState`	Class to normalize state.
`Normalize_ERA5_and_Forcing`	Class to normalize ERA5 and Forcing Datasets.
`BridgescalerScaleState`	Convert to rescaled tensor using Bridgescaler.
`NormalizeState_Quantile`	Class to use the Quantile scaler functionality.
`NormalizeTendency`	Normalize tendency.
`ToTensor`	Convert variables from xr.Datasets to Pytorch Tensors.
`ToTensor_ERA5_and_Forcing`	Class to convert ERA5 and Forcing Datasets to torch tensor.
`NormalizeState_Quantile_Bridgescalar`	Class to use the bridgescaler Quantile functionality.
`ToTensor_BridgeScaler`	Convert to reshaped tensor.

Functions#

`device_compatible_to`(→ torch.Tensor)	Safely move tensor to device, with float32 casting on MPS (Metal Performance Shaders). Addresses runtime error in OSX about MPS not supporting float64.
`load_transforms`(conf[, scaler_only])	Load transforms.

Module Contents#

credit.transforms.logger#

credit.transforms.device_compatible_to(tensor: torch.Tensor, device: torch.device) → torch.Tensor#

Safely move tensor to device, with float32 casting on MPS (Metal Performance Shaders). Addresses runtime error in OSX about MPS not supporting float64.

Parameters:

tensor (torch.Tensor) – Input tensor to move.
device (torch.device) – Target device.

Returns:

Tensor moved to device (cast to float32 if device is MPS).

Return type:

torch.Tensor

credit.transforms.load_transforms(conf, scaler_only=False)#

Load transforms.

Parameters:

conf (str) – path to config
scaler_only (bool) – True –> retrun scaler; False –> return scaler and ToTensor

Returns:

transform

Return type:

tf.tensor

class credit.transforms.NormalizeState(conf)#

Class to normalize state.

mean_ds#

std_ds#

variables#

surface_variables#

levels#

__call__(sample: credit.data.Sample, inverse: bool = False) → credit.data.Sample#

Normalize via quantile transform.

Normalize via provided scaler file/s.

Parameters:

sample – batch.
inverse – if true, will inverse the transform.

Returns:

transformed type.

Return type:

torch.tensor

transform_dataset(DS: xarray.Dataset) → xarray.Dataset#

transform_array(x: torch.Tensor) → torch.Tensor#

Transform from unscaled to scaled values.

Transform.

Parameters:: x – batch.
Returns:: transformed x.

transform(sample: Dict[str, numpy.ndarray]) → Dict[str, numpy.ndarray]#

Transform from unscaled to scaled values.

Transform.

Parameters:: sample – batch.
Returns:: transformed sample.

inverse_transform(x: torch.Tensor) → torch.Tensor#

Inverse transform between tensor forms.

Inverse transform.

Parameters:: x – batch.
Returns:: inverse transformed x.

class credit.transforms.Normalize_ERA5_and_Forcing(conf)#

Class to normalize ERA5 and Forcing Datasets.

mean_ds#

std_ds#

mean_tensors#

std_tensors#

levels#

varname_upper_air#

num_upper_air#

flag_surface#

flag_dyn_forcing#

flag_diagnostic#

flag_forcing#

flag_static#

__call__(sample: credit.data.Sample, inverse: bool = False) → credit.data.Sample#

Normalize ERA5 and Forcing.

Parameters:

sample – batch.
inverse – whether to transform or inverse transform the sample.

Returns:

transformed and normalized sample.

Return type:

torch.tensor

transform_dataset(DS: xarray.Dataset) → xarray.Dataset#

transform_array(x: torch.Tensor) → torch.Tensor#

Transform of y_pred.

Transform via provided scaler file/s of the prediction variable. Dynamic forcing, forcing, and static vars not transformed.

Parameters:: x – batch.
Returns:: transformed x.

transform(sample: Dict[str, numpy.ndarray]) → Dict[str, numpy.ndarray]#

Transform training batches.

Transform handles forcing & static as follows: - forcing & static don’t need to be transformed; users should transform them and save them to the file - other variables (upper-air, surface, dynamic forcing, diagnostics) need to be transformed

Parameters:: sample – batch.
Returns:: transformed sample.

inverse_transform(x: torch.Tensor) → torch.Tensor#

Inverse transform of y_pred.

Inverse transform of prediction variable. Dynamic forcing, forcing, and static vars not transformed.

Parameters:: x – batch.
Returns:: inverse transformed x.

inverse_transform_input(x: torch.Tensor) → torch.Tensor#

Inverse transform for input x.

Forcing and static variables are not transformed (they were not transformed in the transform function).

Parameters:: x – batch.
Returns:: transformed x.

class credit.transforms.BridgescalerScaleState(conf)#

Bases: object

Convert to rescaled tensor using Bridgescaler.

scaler_file#

variables#

surface_variables#

n_levels#

var_levels = []#

n_surface_variables#

n_3dvar_levels#

scaler_df#

scaler_3d#

scaler_surf#

inverse_transform(x: torch.Tensor) → torch.Tensor#

Inverse transform.

Parameters:: x – batch.
Returns:: inverse transformed batch.

transform_array(x: torch.Tensor) → torch.Tensor#

Transform.

Parameters:: x – batch.
Returns:: transformed batch.

transform(sample: Dict[str, numpy.ndarray]) → Dict[str, numpy.ndarray]#

Transform.

Parameters:: sample – batch.
Returns:: transformed batch.

class credit.transforms.NormalizeState_Quantile(conf)#

Class to use the Quantile scaler functionality.

scaler_file#

variables#

surface_variables#

levels#

scaler_df#

scaler_3ds#

scaler_surfs#

scaler_3d#

scaler_surf#

__call__(sample: credit.data.Sample, inverse: bool = False) → credit.data.Sample#

Normalize via quantile transform.

Normalize via provided scaler file/s.

Parameters:

sample – batch.
inverse – if true, will inverse the transform.

Returns:

transformed type.

Return type:

torch.tensor

inverse_transform(x: torch.Tensor) → torch.Tensor#

Inverse transform.

Parameters:: x – batch.
Returns:: inverse transformed x.

transform(sample: Dict[str, numpy.ndarray]) → Dict[str, numpy.ndarray]#

Transform.

Parameters:: sample – batch.
Returns:: transformed batch.

class credit.transforms.NormalizeTendency(variables, surface_variables, base_path)#

Normalize tendency.

variables#

surface_variables#

base_path#

mean#

std#

transform(tensor, surface_tensor)#

Transform.

Transform input tensor/s.

Parameters:

tensor (torch tensor) – batch.
surface_tensor (torch tensor) – surface batch.

Returns:

transformed torch tensors.

Return type:

torch.Tensor

inverse_transform(tensor, surface_tensor)#

Inverse transform.

Inverse transform input tensor/s.

Parameters:

tensor (torch tensor) – batch.
surface_tensor (torch tensor) – surface batch.

Returns:

inverse transformed torch tensors.

Return type:

torch.Tensor

class credit.transforms.ToTensor(conf)#

Convert variables from xr.Datasets to Pytorch Tensors.

conf#

hist_len#

for_len#

variables#

surface_variables#

allvars#

static_variables#

__call__(sample: credit.data.Sample) → credit.data.Sample#

Convert to reshaped tensor.

Reshape and convert to torch tensor.

Parameters:: sample (interator) – batch.
Returns:: reshaped torch tensor.
Return type:: torch.tensor

class credit.transforms.ToTensor_ERA5_and_Forcing(conf)#

Class to convert ERA5 and Forcing Datasets to torch tensor.

conf#

output_dtype = Ellipsis#

hist_len#

for_len#

flag_surface#

flag_dyn_forcing#

flag_diagnostic#

flag_forcing#

flag_static#

varname_upper_air#

flag_upper_air = True#

num_forcing_static = 0#

__call__(sample: credit.data.Sample) → credit.data.Sample#

Convert variables to input/output torch tensors.

Parameters:: sample (interator) – batch.
Returns:: converted torch tensor.
Return type:: torch.tensor

class credit.transforms.NormalizeState_Quantile_Bridgescalar(conf)#

Class to use the bridgescaler Quantile functionality.

Some hoops have to be jumped thorugh, and the efficiency could be improved if we were to retrain the bridgescaler.

scaler_file#

variables#

surface_variables#

levels#

scaler_df#

scaler_3ds#

scaler_surfs#

scaler_3d#

scaler_surf#

__call__(sample: credit.data.Sample, inverse: bool = False) → credit.data.Sample#

Normalize via quantile transform with bridgescaler.

Normalize via provided scaler file/s.

Parameters:: sample (iterator) – batch.
Returns:: transformed torch tensor.
Return type:: torch.tensor

inverse_transform(x: torch.Tensor) → torch.Tensor#

Inverse transform.

Inverse transform via provided scaler file/s.

Parameters:: x – batch.
Returns:: inverse transformed torch tensor.

transform(sample)#

Transform.

Transform via provided scaler file/s.

Parameters:: sample (iterator) – batch.
Returns:: transformed torch tensor.
Return type:: torch.Tensor

class credit.transforms.ToTensor_BridgeScaler(conf)#

Convert to reshaped tensor.

conf#

hist_len#

for_len#

variables#

surface_variables#

allvars#

static_variables#

latN#

lonN#

levels#

one_shot#

__call__(sample: credit.data.Sample) → credit.data.Sample#

Convert to reshaped tensor.

Reshape and convert to torch tensor.

Parameters:: sample (interator) – batch.
Returns:: reshaped torch tensor.
Return type:: torch.tensor