{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Preconfigured preprocessors\n", "\n", "For production runs, there are Preprocessor classes that are pre-configured for different input datasets, e.g., CMIP, ERA5 or REMO output (double nesting). These are mostly pre-configured to work with data at DKRZ, e.g., they read input data from the CMIP or ERA5 data pool and mostly process them on the fly without the need to store any duplicated global model data." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The autoreload extension is already loaded. To reload it, use:\n", " %reload_ext autoreload\n" ] } ], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the preprocessor classes, it is usally a good idea to start a dask client to define your preocessing resources. For the preprocessors, it's usually a good idea to avoid multithreading if you want to write a lot of netcdf files. The following should usually work at DKRZ:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2025-03-28 14:20:12,257 - distributed.scheduler - WARNING - Failed to format dashboard link, unknown value: 'JUPYTERHUB_SERVICE_PREFIX'\n" ] } ], "source": [ "from dask.distributed import Client\n", "\n", "client = Client(\n", " dashboard_address=\"localhost:8787\", n_workers=16, threads_per_worker=1\n", ") # mutlithreading does not work well with cdo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, you can create a preprocessor instance, e.g., for ERA5, you can choose the `ERA5Preprocessor`:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from pyremo.preproc import ERA5Preprocessor\n", "\n", "preprocessor = ERA5Preprocessor(\n", " expid=\"000000\",\n", " surflib=\"/work/ch0636/remo/surflibs/cordex/lib_EUR-11_frac.nc\",\n", " domain=\"EUR-11\",\n", " vc=\"vc_49lev\",\n", " scratch=\"/scratch/g/g300046\",\n", " outpath=\"/scratch/g/g300046/000000/xa/{date:%Y}/{date:%m}\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The preprocessor for ERA5 creates intermediate ERA5 NetCDF files (using CDO) in a CF-like format (“gfile”) and stores them in your scratch location. These files are removed automatically after processing and are only used on the fly. Ensure you have enough scratch disk space if preprocessing multiple years." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The easiest way to run the preprocessor is to use the `run` method. If you want to write netcdf files, you should choose `write=True`. The option `compute=True` will immediately start the processing instead of returnd dask delayed objects." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "afiles = preprocessor.run(\n", " \"2000-01-01T00:00:00\", \"2000-02-01T00:00:00\", write=True, compute=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `run` method returns all afiles created by the preprocessor, e.g." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('/scratch/g/g300046/000000/xa/2000/01/a000000a2000010100.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010106.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010112.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010118.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010200.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010206.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010212.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010218.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010300.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010306.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010312.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010318.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010400.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010406.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010412.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010418.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010500.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010506.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010512.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010518.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010600.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010606.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010612.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010618.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010700.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010706.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010712.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010718.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010800.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010806.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010812.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010818.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010900.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010906.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010912.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000010918.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011000.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011006.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011012.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011018.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011100.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011106.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011112.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011118.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011200.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011206.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011212.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011218.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011300.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011306.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011312.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011318.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011400.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011406.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011412.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011418.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011500.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011506.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011512.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011518.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011600.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011606.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011612.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011618.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011700.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011706.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011712.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011718.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011800.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011806.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011812.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011818.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011900.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011906.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011912.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000011918.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012000.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012006.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012012.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012018.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012100.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012106.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012112.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012118.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012200.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012206.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012212.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012218.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012300.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012306.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012312.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012318.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012400.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012406.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012412.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012418.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012500.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012506.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012512.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012518.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012600.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012606.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012612.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012618.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012700.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012706.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012712.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012718.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012800.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012806.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012812.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012818.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012900.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012906.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012912.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000012918.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013000.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013006.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013012.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013018.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013100.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013106.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013112.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000013118.nc',\n", " '/scratch/g/g300046/000000/xa/2000/01/a000000a2000020100.nc')" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "afiles" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 2 }