Pythonic recipe builder example¶

Shows how to author a recipe in Python and serialize it to the same JSON/YAML schema used by stores, catalogs, and the CLI.

In [1]:

import numpy as np

import woodpecker
from woodpecker.recipes import document, fix, recipe
from woodpecker.testing import make_cmip6

import numpy as np import woodpecker from woodpecker.recipes import document, fix, recipe from woodpecker.testing import make_cmip6

Build a recipe with normal Python calls.

In [2]:

cmip6_core = recipe(
    "cmip6.core_units",
    fix("woodpecker.normalize_tas_units_to_kelvin"),
    description="Normalize CMIP6 tas units.",
).match(
    dataset_id_patterns=["CMIP6.CMIP.*.Amon.tas.*"],
    attrs={"project_id": "CMIP6", "activity_id": "CMIP"},
)

cmip6_core.to_payload()

cmip6_core = recipe( "cmip6.core_units", fix("woodpecker.normalize_tas_units_to_kelvin"), description="Normalize CMIP6 tas units.", ).match( dataset_id_patterns=["CMIP6.CMIP.*.Amon.tas.*"], attrs={"project_id": "CMIP6", "activity_id": "CMIP"}, ) cmip6_core.to_payload()

Out[2]:

{'id': 'cmip6.core_units',
 'description': 'Normalize CMIP6 tas units.',
 'match': {'attrs': {'project_id': 'CMIP6', 'activity_id': 'CMIP'},
  'dataset_id_patterns': ['CMIP6.CMIP.*.Amon.tas.*']},
 'steps': [{'id': 'woodpecker.normalize_tas_units_to_kelvin'}]}

Serialize the recipe document. Pass a path to to_yaml(...) or to_json(...) to write a file.

In [3]:

print(cmip6_core.to_yaml())

print(cmip6_core.to_yaml())

recipes:
- id: cmip6.core_units
  description: Normalize CMIP6 tas units.
  match:
    attrs:
      project_id: CMIP6
      activity_id: CMIP
    dataset_id_patterns:
    - CMIP6.CMIP.*.Amon.tas.*
  steps:
  - id: woodpecker.normalize_tas_units_to_kelvin

Use to_model() to run the recipe immediately.

In [4]:

dataset = make_cmip6(overrides={"units": "degC"}, seed=7)
original_values = dataset["tas"].values.copy()

plan_model = cmip6_core.to_model()
findings = woodpecker.recipe.check(dataset, plan_model)

findings.fix_ids

dataset = make_cmip6(overrides={"units": "degC"}, seed=7) original_values = dataset["tas"].values.copy() plan_model = cmip6_core.to_model() findings = woodpecker.recipe.check(dataset, plan_model) findings.fix_ids

Out[4]:

('woodpecker.normalize_tas_units_to_kelvin',)

In [5]:

result = woodpecker.recipe.apply(dataset, plan_model, dry_run=True)

(
    result.stats,
    result.preview,
    dataset["tas"].attrs["units"],
    np.allclose(dataset["tas"].values, original_values),
)

result = woodpecker.recipe.apply(dataset, plan_model, dry_run=True) ( result.stats, result.preview, dataset["tas"].attrs["units"], np.allclose(dataset["tas"].values, original_values), )

Out[5]:

({'attempted': 1,
  'changed': 1,
  'persist_attempted': 0,
  'persisted': 0,
  'persist_failed': 0,
  'preview': [{'path': 'HadGEM3-GC31-LL',
    'fix_id': 'woodpecker.normalize_tas_units_to_kelvin',
    'name': 'Normalize tas-like units to Kelvin',
    'labels': ['risk.value_transformation'],
    'label_titles': ['careful: value transformation'],
    'label_metadata': [{'id': 'risk.value_transformation',
      'title': 'careful: value transformation',
      'description': 'Transforms data or coordinate values.',
      'category': 'risk-medium'}],
    'changed': True}]},
 ({'path': 'HadGEM3-GC31-LL',
   'fix_id': 'woodpecker.normalize_tas_units_to_kelvin',
   'name': 'Normalize tas-like units to Kelvin',
   'labels': ['risk.value_transformation'],
   'label_titles': ['careful: value transformation'],
   'label_metadata': [{'id': 'risk.value_transformation',
     'title': 'careful: value transformation',
     'description': 'Transforms data or coordinate values.',
     'category': 'risk-medium'}],
   'changed': True},),
 'degC',
 True)

In [6]:

write = woodpecker.recipe.apply(dataset, plan_model, dry_run=False)

(
    write.stats,
    dataset["tas"].attrs["units"],
    np.allclose(dataset["tas"].values, original_values + 273.15),
)

write = woodpecker.recipe.apply(dataset, plan_model, dry_run=False) ( write.stats, dataset["tas"].attrs["units"], np.allclose(dataset["tas"].values, original_values + 273.15), )

Out[6]:

({'attempted': 1,
  'changed': 1,
  'persist_attempted': 1,
  'persisted': 1,
  'persist_failed': 0,
  'preview': [{'path': 'HadGEM3-GC31-LL',
    'fix_id': 'woodpecker.normalize_tas_units_to_kelvin',
    'name': 'Normalize tas-like units to Kelvin',
    'labels': ['risk.value_transformation'],
    'label_titles': ['careful: value transformation'],
    'label_metadata': [{'id': 'risk.value_transformation',
      'title': 'careful: value transformation',
      'description': 'Transforms data or coordinate values.',
      'category': 'risk-medium'}],
    'changed': True}]},
 'K',
 True)

A recipe document can contain multiple Python-authored recipes.

In [7]:

plan_document = document(
    cmip6_core,
    recipe("c3s.atlas", fix("atlas.encoding_cleanup")).match(path_patterns=["*atlas*.nc"]),
)

plan_document.to_payload()

plan_document = document( cmip6_core, recipe("c3s.atlas", fix("atlas.encoding_cleanup")).match(path_patterns=["*atlas*.nc"]), ) plan_document.to_payload()

Out[7]:

{'recipes': [{'id': 'cmip6.core_units',
   'description': 'Normalize CMIP6 tas units.',
   'match': {'attrs': {'project_id': 'CMIP6', 'activity_id': 'CMIP'},
    'dataset_id_patterns': ['CMIP6.CMIP.*.Amon.tas.*']},
   'steps': [{'id': 'woodpecker.normalize_tas_units_to_kelvin'}]},
  {'id': 'c3s.atlas',
   'match': {'path_patterns': ['*atlas*.nc']},
   'steps': [{'id': 'atlas.encoding_cleanup'}]}]}