credit.datasets
===============

.. py:module:: credit.datasets


Submodules
----------

.. toctree::
   :maxdepth: 1

   /autoapi/credit/datasets/_utils/index
   /autoapi/credit/datasets/base_dataset/index
   /autoapi/credit/datasets/channel_layout/index
   /autoapi/credit/datasets/count_channels/index
   /autoapi/credit/datasets/datamap/index
   /autoapi/credit/datasets/downscaling_dataset/index
   /autoapi/credit/datasets/era5/index
   /autoapi/credit/datasets/era5_multistep/index
   /autoapi/credit/datasets/era5_multistep_batcher/index
   /autoapi/credit/datasets/era5_singlestep/index
   /autoapi/credit/datasets/goes/index
   /autoapi/credit/datasets/hrrr/index
   /autoapi/credit/datasets/hrrr_download/index
   /autoapi/credit/datasets/les_singlestep/index
   /autoapi/credit/datasets/load_dataset_and_dataloader/index
   /autoapi/credit/datasets/local/index
   /autoapi/credit/datasets/mrms/index
   /autoapi/credit/datasets/mrms_download/index
   /autoapi/credit/datasets/multi_source/index
   /autoapi/credit/datasets/om4_multistep_batcher/index
   /autoapi/credit/datasets/realtime_predict/index
   /autoapi/credit/datasets/sequential_multistep/index
   /autoapi/credit/datasets/tisr/index
   /autoapi/credit/datasets/wrf_singlestep/index
   /autoapi/credit/datasets/wrfmultistep/index


Attributes
----------

.. autoapisummary::

   credit.datasets.logger
   credit.datasets._CLASS_SOURCES


Functions
---------

.. autoapisummary::

   credit.datasets.__getattr__
   credit.datasets.set_globals
   credit.datasets.setup_data_loading


Package Contents
----------------

.. py:data:: logger

.. py:data:: _CLASS_SOURCES

.. py:function:: __getattr__(name)

.. py:function:: set_globals(data_config, namespace=None)

   Sets global variables from the provided configuration dictionary in the specified namespace.

   This method updates the global variables in either the given `namespace` or the
   caller's namespace (if `namespace` is not provided). If the `namespace` is not specified,
   it uses the global namespace of the caller (using `sys._getframe(1).f_globals`).

   :param data_config: A dictionary where the keys are the global variable names
                       and the values are the corresponding values to set.
   :type data_config: dict
   :param namespace: The namespace (or dictionary) where the global variables
                     should be set. If not provided, the caller's global namespace is used.
   :type namespace: dict, optional

   The method logs each global variable being created and its name.


.. py:function:: setup_data_loading(conf)

   Sets up the data loading configuration by reading and processing data paths,
    surface, dynamic forcing, and diagnostic files based on the given configuration.

   The function processes the configuration dictionary (`conf`) and performs the following:
   - Globs and filters data files (ERA5, surface, dynamic forcing, diagnostic).
   - Determines the training and validation file sets based on specified years.
   - Sets up variables like historical data length, forecast length, and additional metadata.
   - Returns a dictionary containing all the paths and configuration details for further use.

   :param conf: A dictionary containing configuration details, including data paths,
                variable names, forecast details, and other settings.
   :type conf: dict

   :returns:

             A dictionary containing paths to various datasets and other
                 configuration values used in data loading, such as:
                   * all_ERA_files: All ERA5 dataset files.
                   * train_files: Filtered training dataset files.
                   * valid_files: Filtered validation dataset files.
                   * surface_files: Surface data files, if available.
                   * dyn_forcing_files: Dynamic forcing files, if available.
                   * diagnostic_files: Diagnostic files, if available.
                   * varname_upper_air, varname_surface, varname_dyn_forcing, etc.: Variable names for
                      each data type.
                   * history_len: Length of the history data for training.
                   * forecast_len: Number of steps ahead to forecast.
                   * Other configuration values related to skipping periods, one-shot learning, etc.
   :rtype: data_config (dict)