credit.datasets.gen_2.hrrr ========================== .. py:module:: credit.datasets.gen_2.hrrr .. autoapi-nested-parse:: hrrr.py ------------------------------------------------------- HRRRDataset: PyTorch Dataset for HRRR GRIB2 data. Supports three HRRR products (``VALID_PRODUCTS``): * ``"wrfprs"`` — pressure-level output (default, ~200 MB/file) * ``"wrfnat"`` — native/hybrid-sigma level output (~200 MB/file, ~65 levels) * ``"wrfsubh"`` — 15-minute sub-hourly surface output (surface vars only) Tensor keys follow the pattern ``{user_provided_name}/{hrrr_product}/{field_type}/{dim}/{varname}`` where *hrrr_product* is product-specific: * ``"wrfprs"`` → ``{user_provided_name}/wrfprs/{field_type}/{dim}/{varname}`` * ``"wrfnat"`` → ``{user_provided_name}/wrfnat/{field_type}/{dim}/{varname}`` * ``"wrfsubh"`` → ``{user_provided_name}/wrfsubh/{field_type}/2d/{varname}`` *dim* is ``"3d"`` for multi-level variables and ``"2d"`` for surface variables. Tensor shapes (before DataLoader batching): 3D variables: ``(n_levels, 1, y, x)`` 2D variables: ``(1, 1, y, x)`` The ``y`` / ``x`` spatial dimensions correspond to HRRR's native Lambert Conformal Conic grid; if ``extent`` is specified they reflect the cropped sub-domain rather than the full CONUS grid (~1059 x 1799). Two S3 path layouts are handled automatically: v1/v2 (before 2018-07-12): s3://noaa-hrrr-bdp-pds/hrrr.{YYYYMMDD}/hrrr.t{HH}z.{product}f{FF:02d}.grib2 v3/v4 (2018-07-12 onward): s3://noaa-hrrr-bdp-pds/hrrr.{YYYYMMDD}/conus/hrrr.t{HH}z.{product}f{FF:02d}.grib2 GRIB2 reading ------------- Both local and remote modes use the same ``.idx`` + byte-range pipeline: *Remote mode*: 1. Fetch the sidecar ``.idx`` inventory (~100 KB) to get exact byte offsets for every GRIB message. 2. Issue one Obstore get_ranges request to pull all the relevant data fields (based on the byte ranges from the idx file). *Local mode*: 1. Reads the ``.idx`` sidecar from disk, 2. Uses ``file.seek()`` + ``file.read()`` — identical byte-range approach, no full-file scan. The ``.idx`` sidecar must be present alongside the grib2; download it with ``hrrr_download.py``. For a typical training sample (5 vars x 6 levels ≈ 30 messages) remote mode transfers ~3 MB instead of ~200 MB (~60-100x reduction). Variable lookup is driven by :data:`VAR_REGISTRY`. Extend it at import time to add variables without subclassing:: from credit.datasets.gen_2.hrrr import VAR_REGISTRY VAR_REGISTRY["MYVAR"] = { "shortName": "myvar", "typeOfLevel": "isobaricInhPa", "idx_name": "MYVAR", "idx_level": None, } Example YAML (wrfprs, local mode):: data: source: Example_HRRR: # User-provided name (arbitrary key) dataset_type: "HRRR" # product: "wrfprs" # Optional for PRS product. Default is "wrfprs". mode: "local" base_path: "/data/hrrr" forecast_hour: 0 levels: [250, 500, 700, 850, 925, 1000] variables: prognostic: vars_3D: [T, U, V, Q, GH] vars_2D: [t2m] extent: [-130, -60, 20, 55] start_datetime: "2021-06-01" end_datetime: "2021-06-05" timestep: "1h" forecast_len: 0 Example YAML (wrfnat, remote mode):: data: source: Example_HRRR_NAT: # User-provided name (arbitrary key) dataset_type: "HRRR" product: "wrfnat" # Options: "wrfprs" (default), "wrfnat", "wrfsubh" mode: "remote" forecast_hour: 0 levels: [10, 20, 30, 40, 50] # hybrid level indices 1-65 variables: prognostic: vars_3D: [T, U, V, Q] start_datetime: "2022-01-01" end_datetime: "2022-01-31" timestep: "1h" forecast_len: 0 Example YAML (wrfsubh, remote mode — 15-min output):: data: source: Example_HRRR_SUBH: # User-provided name (arbitrary key) dataset_type: "HRRR" product: "wrfsubh" # Options: "wrfprs" (default), "wrfnat", "wrfsubh" mode: "remote" variables: prognostic: vars_2D: [t2m, sp, refc] start_datetime: "2022-01-01 00:15" end_datetime: "2022-01-31 00:00" timestep: "15min" forecast_len: 0 Attributes ---------- .. autoapisummary:: credit.datasets.gen_2.hrrr.logger credit.datasets.gen_2.hrrr.VAR_REGISTRY credit.datasets.gen_2.hrrr.VALID_PRODUCTS Classes ------- .. autoapisummary:: credit.datasets.gen_2.hrrr.HRRRDataset Module Contents --------------- .. py:data:: logger .. py:data:: VAR_REGISTRY :type: dict[str, dict[str, str | None]] .. py:data:: VALID_PRODUCTS .. py:class:: HRRRDataset(data_config: dict[str, Any], return_target: bool = False) Bases: :py:obj:`credit.datasets.gen_2.base_dataset.BaseDataset` CREDIT Dataset for HRRR GRIB2 data (wrfprs / wrfnat / wrfsubh). Implements the same field-type semantics as BaseDataset: * ``prognostic`` — input at step 0 and target (autoregressive rollout) * ``dynamic_forcing`` — input at every step; never a target * ``diagnostic`` — target only * ``static`` — input at step 0; never a target, applies to all steps Both modes use ``pygrib`` for GRIB2 decoding. Remote mode fetches the ``.idx`` sidecar and issues parallel HTTP Range requests — no full file download required. See module docstring for full output format, tensor shapes, and YAML configuration examples. :ivar dataset_type: Tensor key - `"HRRR"` :ivar product: Active HRRR product (``"HRRR_PRS" / "wrfprs"``, ``"HRRR_NAT" / "wrfnat"``, or ``"HRRR_SUBH" / "wrfsubh"``) with default value ``"HRRR_PRS"``. :ivar datetimes: DatetimeIndex of valid initialization timestamps. :ivar static_metadata: Dataset-level metadata for MultiSourceDataset. .. py:attribute:: dataset_type .. py:attribute:: product :type: VALID_PRODUCTS .. py:attribute:: mode :type: str .. py:attribute:: base_path :type: str | None .. py:attribute:: forecast_hour :type: int .. py:attribute:: extent :type: list[float] | None .. py:attribute:: global_levels :type: list[int] | None .. py:attribute:: num_decompress_workers :type: int .. py:attribute:: static_metadata :type: dict[str, Any]