credit.transforms ================= .. py:module:: credit.transforms Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/credit/transforms/transforms_global/index /autoapi/credit/transforms/transforms_les/index /autoapi/credit/transforms/transforms_quantile/index /autoapi/credit/transforms/transforms_wrf/index Attributes ---------- .. autoapisummary:: credit.transforms.logger Classes ------- .. autoapisummary:: credit.transforms.NormalizeLES credit.transforms.ToTensorLES credit.transforms.NormalizeWRF credit.transforms.ToTensorWRF credit.transforms.Normalize_ERA5_and_Forcing credit.transforms.ToTensor_ERA5_and_Forcing credit.transforms.BridgescalerScaleState credit.transforms.NormalizeState_Quantile_Bridgescalar credit.transforms.ToTensor_BridgeScaler Functions --------- .. autoapisummary:: credit.transforms.load_transforms Package Contents ---------------- .. py:class:: NormalizeLES(conf) .. py:attribute:: mean_ds .. py:attribute:: std_ds .. py:attribute:: mean_tensors .. py:attribute:: std_tensors .. py:attribute:: levels .. py:attribute:: varname_upper_air .. py:attribute:: num_upper_air .. py:attribute:: flag_surface .. py:attribute:: flag_dyn_forcing .. py:attribute:: flag_diagnostic .. py:attribute:: flag_forcing .. py:attribute:: flag_static .. py:method:: __call__(sample, inverse: bool = False) .. py:method:: transform_array(x: torch.Tensor) -> torch.Tensor This function applies to y_pred, so there won't be dynamic forcing, forcing, and static variables. .. py:method:: transform(sample: Dict[str, numpy.ndarray]) -> Dict[str, numpy.ndarray] This function transforms training batches, it handles forcing & static as follows: - forcing & static don't need to be transformed; users should transform them and save them to the file - other variables (upper-air, surface, dynamic forcing, diagnostics) need to be transformed .. py:method:: inverse_transform(x: torch.Tensor) -> torch.Tensor This function applies to y_pred, so there won't be dynamic forcing, forcing, and static vars .. py:method:: inverse_transform_input(x: torch.Tensor) -> torch.Tensor Inverse transform for input x. Forcing and static variables are not transformed (they were not transformed in the transform function). .. py:class:: ToTensorLES(conf) .. py:attribute:: conf .. py:attribute:: output_dtype :value: Ellipsis .. py:attribute:: hist_len .. py:attribute:: for_len .. py:attribute:: flag_surface .. py:attribute:: flag_dyn_forcing .. py:attribute:: flag_diagnostic .. py:attribute:: flag_forcing .. py:attribute:: flag_static .. py:attribute:: varname_upper_air .. py:attribute:: num_forcing_static :value: 0 .. py:method:: __call__(sample) .. py:class:: NormalizeWRF(conf) .. py:attribute:: mean_ds .. py:attribute:: std_ds .. py:attribute:: mean_tensors .. py:attribute:: std_tensors .. py:attribute:: levels .. py:attribute:: varname_upper_air .. py:attribute:: num_upper_air .. py:attribute:: flag_surface .. py:attribute:: flag_dyn_forcing .. py:attribute:: flag_diagnostic .. py:attribute:: flag_forcing .. py:attribute:: flag_static .. py:attribute:: mean_ds_outside .. py:attribute:: std_ds_outside .. py:attribute:: mean_tensors_outside .. py:attribute:: std_tensors_outside .. py:attribute:: levels_outside .. py:attribute:: varname_upper_air_outside .. py:attribute:: num_upper_air_outside .. py:attribute:: flag_surface_outside .. py:method:: __call__(sample, inverse: bool = False) .. py:method:: transform_array(x: torch.Tensor) -> torch.Tensor This function applies to y_pred, so there won't be boundary input, forcing, and static variables. .. py:method:: transform(sample: Dict[str, numpy.ndarray]) -> Dict[str, numpy.ndarray] This function transforms training batches - forcing & static don't need to be transformed; users should transform them and save them to the file - other variables need to be transformed .. py:method:: inverse_transform(x: torch.Tensor) -> torch.Tensor This function applies to y_pred, so there won't be dynamic forcing, forcing, and static vars .. py:class:: ToTensorWRF(conf) .. py:attribute:: conf .. py:attribute:: output_dtype :value: Ellipsis .. py:attribute:: hist_len .. py:attribute:: for_len .. py:attribute:: flag_surface .. py:attribute:: flag_dyn_forcing .. py:attribute:: flag_diagnostic .. py:attribute:: flag_forcing .. py:attribute:: flag_static .. py:attribute:: varname_upper_air .. py:attribute:: num_forcing_static :value: 0 .. py:attribute:: hist_len_outside .. py:attribute:: for_len_outside .. py:attribute:: flag_surface_outside .. py:attribute:: varname_upper_air_outside .. py:method:: __call__(sample) .. py:class:: Normalize_ERA5_and_Forcing(conf) Class to normalize ERA5 and Forcing Datasets. .. py:attribute:: mean_ds .. py:attribute:: std_ds .. py:attribute:: mean_tensors .. py:attribute:: std_tensors .. py:attribute:: levels .. py:attribute:: varname_upper_air .. py:attribute:: num_upper_air .. py:attribute:: flag_surface .. py:attribute:: flag_dyn_forcing .. py:attribute:: flag_diagnostic .. py:attribute:: flag_forcing .. py:attribute:: flag_static .. py:method:: __call__(sample: credit.data.Sample, inverse: bool = False) -> credit.data.Sample Normalize ERA5 and Forcing. :param sample: batch. :param inverse: whether to transform or inverse transform the sample. :returns: transformed and normalized sample. :rtype: torch.tensor .. py:method:: transform_dataset(DS: xarray.Dataset) -> xarray.Dataset .. py:method:: transform_array(x: torch.Tensor) -> torch.Tensor Transform of y_pred. Transform via provided scaler file/s of the prediction variable. Dynamic forcing, forcing, and static vars not transformed. :param x: batch. :returns: transformed x. .. py:method:: transform(sample: Dict[str, numpy.ndarray]) -> Dict[str, numpy.ndarray] Transform training batches. Transform handles forcing & static as follows: - forcing & static don't need to be transformed; users should transform them and save them to the file - other variables (upper-air, surface, dynamic forcing, diagnostics) need to be transformed :param sample: batch. :returns: transformed sample. .. py:method:: inverse_transform(x: torch.Tensor) -> torch.Tensor Inverse transform of y_pred. Inverse transform of prediction variable. Dynamic forcing, forcing, and static vars not transformed. :param x: batch. :returns: inverse transformed x. .. py:method:: inverse_transform_input(x: torch.Tensor) -> torch.Tensor Inverse transform for input x. Forcing and static variables are not transformed (they were not transformed in the transform function). :param x: batch. :returns: transformed x. .. py:method:: _align_coords(DS: xarray.Dataset, ref_ds: xarray.Dataset, rtol: float = 1e-06, atol: float = 1e-08) -> xarray.Dataset Ensure DS has the same lat/lon as ref_ds (to within tolerance), then re-assign DS.coords to exactly match ref_ds.coords. .. py:method:: inverse_transform_dataset(DS: xarray.Dataset, rtol: float = 1e-06, atol: float = 1e-08) -> xarray.Dataset Inverse‐transform DS by (DS * std_ds) + mean_ds, after aligning its coordinates to mean_ds/std_ds. .. py:class:: ToTensor_ERA5_and_Forcing(conf) Class to convert ERA5 and Forcing Datasets to torch tensor. .. py:attribute:: conf .. py:attribute:: output_dtype :value: Ellipsis .. py:attribute:: hist_len .. py:attribute:: for_len .. py:attribute:: flag_surface .. py:attribute:: flag_dyn_forcing .. py:attribute:: flag_diagnostic .. py:attribute:: flag_forcing .. py:attribute:: flag_static .. py:attribute:: varname_upper_air .. py:attribute:: flag_upper_air :value: True .. py:attribute:: num_forcing_static :value: 0 .. py:method:: __call__(sample: credit.data.Sample) -> credit.data.Sample Convert variables to input/output torch tensors. :param sample: batch. :type sample: interator :returns: converted torch tensor. :rtype: torch.tensor .. py:class:: BridgescalerScaleState(conf) Bases: :py:obj:`object` Convert to rescaled tensor using Bridgescaler. .. py:attribute:: scaler_file .. py:attribute:: variables .. py:attribute:: surface_variables .. py:attribute:: n_levels .. py:attribute:: var_levels :value: [] .. py:attribute:: n_surface_variables .. py:attribute:: n_3dvar_levels .. py:attribute:: scaler_df .. py:attribute:: scaler_3d .. py:attribute:: scaler_surf .. py:method:: inverse_transform(x: torch.Tensor) -> torch.Tensor Inverse transform. Inverse transform. :param x: batch. :returns: inverse transformed batch. .. py:method:: transform_array(x: torch.Tensor) -> torch.Tensor Transform. Transform. :param x: batch. :returns: transformed batch. .. py:method:: transform(sample: Dict[str, numpy.ndarray]) -> Dict[str, numpy.ndarray] Transform. Transform. :param sample: batch. :returns: transformed batch. .. py:class:: NormalizeState_Quantile_Bridgescalar(conf) Class to use the bridgescaler Quantile functionality. Some hoops have to be jumped thorugh, and the efficiency could be improved if we were to retrain the bridgescaler. .. py:attribute:: scaler_file .. py:attribute:: variables .. py:attribute:: surface_variables .. py:attribute:: levels .. py:attribute:: scaler_df .. py:attribute:: scaler_3ds .. py:attribute:: scaler_surfs .. py:attribute:: scaler_3d .. py:attribute:: scaler_surf .. py:method:: __call__(sample: credit.data.Sample, inverse: bool = False) -> credit.data.Sample Normalize via quantile transform with bridgescaler. Normalize via provided scaler file/s. :param sample: batch. :type sample: iterator :returns: transformed torch tensor. :rtype: torch.tensor .. py:method:: inverse_transform(x: torch.Tensor) -> torch.Tensor Inverse transform. Inverse transform via provided scaler file/s. :param x: batch. :returns: inverse transformed torch tensor. .. py:method:: transform(sample) Transform. Transform via provided scaler file/s. :param sample: batch. :type sample: iterator :returns: transformed torch tensor. :rtype: torch.Tensor .. py:class:: ToTensor_BridgeScaler(conf) Convert to reshaped tensor. .. py:attribute:: conf .. py:attribute:: hist_len .. py:attribute:: for_len .. py:attribute:: variables .. py:attribute:: surface_variables .. py:attribute:: allvars .. py:attribute:: static_variables .. py:attribute:: latN .. py:attribute:: lonN .. py:attribute:: levels .. py:attribute:: one_shot .. py:method:: __call__(sample: credit.data.Sample) -> credit.data.Sample Convert to reshaped tensor. Reshape and convert to torch tensor. :param sample: batch. :type sample: interator :returns: reshaped torch tensor. :rtype: torch.tensor .. py:data:: logger .. py:function:: load_transforms(conf, scaler_only=False) Load transforms. :param conf: path to config :type conf: str :param scaler_only: True --> retrun scaler; False --> return scaler and ToTensor :type scaler_only: bool :returns: transform :rtype: tf.tensor