credit.datasets.gen_2.gfs_download

credit.datasets.gen_2.gfs_download#

Download GFS or GDAS NetCDF files for GFSDataset local mode.

The downloader reads the same single-source data configuration used by GFSDataset, checks which configured timestamps are available in the public Google Cloud bucket, and downloads the atmospheric/surface file pair for each available run. Files are written without renaming, so a local GFSDataset can use the resulting directory directly.

The output layout is:

{base_path}/{system}.YYYYMMDD/HH/atmos/{system}.tHHz.atmanl.nc
{base_path}/{system}.YYYYMMDD/HH/atmos/{system}.tHHz.sfcanl.nc

For forecast output, forecast_hour changes the final names to forms such as atmf003.nc and sfcf003.nc. Downloads use obstore and a thread pool; num_workers controls the number of concurrent file transfers.

Command-line usage:

python -m credit.datasets.gen_2.gfs_download -c config/gfs.yml
python -m credit.datasets.gen_2.gfs_download -c config/gfs.yml --num-workers 8 --overwrite

Attributes#

logger

parser

Functions#

download_gfs(→ None)

Download configured GFS/GDAS files for local GFSDataset use.

Module Contents#

credit.datasets.gen_2.gfs_download.logger#
credit.datasets.gen_2.gfs_download.download_gfs(data_config: dict[str, Any], num_workers: int = 4, overwrite: bool = False) → None#

Download configured GFS/GDAS files for local GFSDataset use.

Parameters:

  • data_config – The top-level data configuration passed to GFSDataset. It must contain exactly one source with dataset_type: "gfs" and a local base_path.

  • num_workers – Number of concurrent file downloads. Defaults to 4.

  • overwrite – If True, download files even when the corresponding local file already exists. Defaults to False.

Raises:

  • KeyError – If the configuration does not contain the required source block.

  • ValueError – If multiple sources, a non-GFS source, or no base_path is supplied.

  • ImportError – If obstore is not installed when the download begins.

Example

>>> from credit.datasets.gen_2.gfs_download import download_gfs
>>> download_gfs(config["data"], num_workers=8)

After downloading, set the source mode to "local" and point base_path at the download directory before constructing GFSDataset.

credit.datasets.gen_2.gfs_download.parser#