Testing CMORisation on Actual Files

This guide covers comprehensive testing of the CMORisation pipeline using actual model output files.

Overview

The ACCESS-MOPPy test suite includes integration tests that process real data files through the full CMORisation pipeline. These tests validate that variables are correctly transformed to CMIP6/CMIP7 standards.

Key Features:

• Variable-level granularity: Test individual variables, not just entire tables

• Multiple frequencies: Support for monthly, daily, 3-hourly, and 6-hourly data

• CMIP6 & CMIP7: Test both standards with automatic table mapping

• Compliance validation: Verify output against CMOR standards using PrePARE or WCRP compliance-checker

• Flexible filtering: Run tests for specific tables, variables, or frequencies

Setting Up Test Data

External Data Root

Set the ACCESS_MOPPY_DATA_ROOT environment variable to point to your test data directory:

export ACCESS_MOPPY_DATA_ROOT=/path/to/test/data

Expected Directory Structure

The test data should follow this structure:

ACCESS_MOPPY_DATA_ROOT/
├── output1/
│   ├── atmosphere/netCDF/
│   │   ├── *_mon.nc          # Monthly atmospheric data
│   │   ├── *_dai.nc          # Daily atmospheric data
│   │   ├── *_3hr.nc          # 3-hourly atmospheric data
│   │   └── *_6hr.nc          # 6-hourly atmospheric data
│   ├── ocean/
│   │   ├── *-variable-1monthly-mean*.nc  # Monthly ocean (CMIP6 convention)
│   │   └── *-variable-1mon-mean*.nc      # Monthly ocean (alternate convention)
│   └── ice/
│       └── iceh-1monthly-mean_*.nc       # Sea ice monthly data
├── output2/
└── output3/

The test discovery looks for files matching these patterns in any output* subdirectory, so you can have multiple ensemble members or runs.

Running Tests

Before Running Tests

1. Set up the pixi environment:

   cd /path/to/ACCESS-MOPPy
   pixi shell -e dev
   

2. Set the data root (if not already in your shell):

   export ACCESS_MOPPY_DATA_ROOT=/home/romain/PROJECTS/CMIP7_Test_data/esm-historical
   

Basic Test Execution

Run all CMORisation tests:

pytest tests/integration/test_full_cmorisation.py -v

This runs 372 tests: 240 CMIP6 + 132 CMIP7 across all supported tables and variables.

Run with minimal output:

pytest tests/integration/test_full_cmorisation.py -q

Show test collection only (no execution):

pytest tests/integration/test_full_cmorisation.py --collect-only -q

Advanced Filtering

Filter by Variable Name

Test a specific variable across all frequencies:

# Test 'tas' in all tables
pytest tests/integration/test_full_cmorisation.py -k "tas" -v

# Test temperature variables only
pytest tests/integration/test_full_cmorisation.py -k "tas or tasa or tasmin or tasmax" -v

# Test 'tos' (ocean temperature)
pytest tests/integration/test_full_cmorisation.py -k "tos" -v

Filter by Table

Test all variables in a specific table:

# Test all Amon (monthly atmosphere) variables
pytest tests/integration/test_full_cmorisation.py -k "Amon" -v

# Test all ocean variables
pytest tests/integration/test_full_cmorisation.py -k "Omon" -v

# Test all sea ice variables
pytest tests/integration/test_full_cmorisation.py -k "SImon" -v

# Test daily frequency only
pytest tests/integration/test_full_cmorisation.py -k "day" -v

Combined Filtering

Test tas in Amon table only:

pytest tests/integration/test_full_cmorisation.py -k "Amon and tas" -v

Test ocean variables but exclude SImon:

pytest tests/integration/test_full_cmorisation.py -k "Omon or SImon or ocean" -v

Test a specific variable in multiple frequencies:

pytest tests/integration/test_full_cmorisation.py -k "pr" -v
# Tests: Amon-pr, CFmon-pr, day-pr, 6hrPlev-pr, etc.

Filter by CMIP Version

Test CMIP6 only (no CMIP7):

pytest tests/integration/test_full_cmorisation.py -k "not cmip7" -v

Test CMIP7 only:

pytest tests/integration/test_full_cmorisation.py -k "cmip7" -v

# Specific CMIP7 variable
pytest tests/integration/test_full_cmorisation.py -k "cmip7 and tas" -v

Test Granularity Examples

# Single variable test (most specific)
pytest tests/integration/test_full_cmorisation.py::TestFullCMORIntegration::test_cmorisation_variable -k "Amon-ACCESS" -v
# Note: Use the full test ID with the parametrized variable name

# All variables in Amon
pytest tests/integration/test_full_cmorisation.py -k "Amon" -v

# All monthly variables (Amon, Lmon, AERmon, etc.)
pytest tests/integration/test_full_cmorisation.py -k "mon" -v

# All atmospheric variables
pytest tests/integration/test_full_cmorisation.py -k "Amon or day or 3hr or 6hr or Eday or CFday" -v

Understanding Test Output

Successful Test

tests/integration/test_full_cmorisation.py::TestFullCMORIntegration::test_cmorisation_variable[Amon-ACCESS-ESM1-6-CMIP6_Amon.json-CMIP6-tas] PASSED [100%]

What this means: - Variable tas in table Amon was successfully CMORised - Output files were created and validated

Skipped Test

tests/integration/test_full_cmorisation.py::TestFullCMORIntegration::test_cmorisation_variable[Omon-...] SKIPPED
Reason: Ocean data directory not available; set ACCESS_MOPPY_DATA_ROOT

Solutions: 1. Set ACCESS_MOPPY_DATA_ROOT to a directory containing ocean files 2. Ensure ocean files exist at $ACCESS_MOPPY_DATA_ROOT/output*/ocean/

Failed Test

tests/integration/test_full_cmorisation.py::TestFullCMORIntegration::test_cmorisation_variable[Amon-...-tas] FAILED
AssertionError: No output files found for tas in /tmp/cmor_output_Amon_tas

Common causes: - Input data file is missing or malformed - CMORiser encountered a mapping error - Output file wasn’t created for other reasons

Debugging: 1. Check the full error message with -v flag 2. Verify input files exist: ls $ACCESS_MOPPY_DATA_ROOT/output*/atmosphere/netCDF/ 3. Check CMORiser logs if available

Compliance Validation

Default Behavior (PrePARE)

By default, tests validate output using PrePARE:

pytest tests/integration/test_full_cmorisation.py -k "Amon-tas" -v

PrePARE validates: - CMOR table compliance - Dimensions and coordinates - NetCDF structure - Metadata

Using WCRP Compliance Checker

To use the WCRP compliance-checker instead (more strict):

pytest tests/integration/test_full_cmorisation.py -k "Amon-tas" \
    --compliance-tool=wcrp -v

Requirements: - WCRP compliance-checker must be installed - cc-plugin-wcrp must be installed

Verify availability:

compliance-checker --list-tests

Should show wcrp_cmip6:1.0 in the output.

Skipping Compliance Validation

To test CMORisation without compliance validation:

pytest tests/integration/test_full_cmorisation.py -k "Amon" \
    --compliance-tool=none -v

Note: Some tests automatically skip compliance validation: - Omon (ocean) - non-standard grid structures - Ofx (ocean fixed fields) - fixed variables

Practical Testing Workflows

Scenario 1: Test a New Variable

You’ve added a new variable mapping and want to verify it works:

# 1. Run just that variable across all tables
pytest tests/integration/test_full_cmorisation.py -k "mynewvar" -v

# 2. If it passes, run with CMIP7 too
pytest tests/integration/test_full_cmorisation.py -k "mynewvar and cmip7" -v

# 3. Verify compliance output
pytest tests/integration/test_full_cmorisation.py -k "Amon-mynewvar" -v -s

Scenario 2: Validate a Frequency Tier

You want to ensure all daily variables work correctly:

# Test all daily (day) variables
pytest tests/integration/test_full_cmorisation.py -k "day" -v --tb=short

# Count how many pass/fail
pytest tests/integration/test_full_cmorisation.py -k "day" -v --tb=no -q

Scenario 3: Quality Check Before Release

Run comprehensive tests to ensure everything works:

# Test CMIP6 only (stable baseline)
pytest tests/integration/test_full_cmorisation.py -k "not cmip7" \
    --tb=short -v 2>&1 | tee test_results.log

# Check for any failures
grep -E "(FAILED|ERROR)" test_results.log

# If clean, test CMIP7
pytest tests/integration/test_full_cmorisation.py -k "cmip7" \
    --tb=short -v 2>&1 | tee test_results_cmip7.log

Scenario 4: Debug a Single Variable

A test is failing and you need detailed information:

# Run with full traceback and print statements
pytest tests/integration/test_full_cmorisation.py -k "Amon-tas" -vv -s --tb=long

# Show which input files were discovered
pytest tests/integration/test_full_cmorisation.py -k "Amon-tas" -vv -s

Interpreting Variable Names

Test IDs follow the pattern: TableName-VariableName or TableName-VariableName-cmip7

Examples:

Test ID	Meaning	Frequency
Amon-tas	Temperature in Amon (monthly atmosphere)	Monthly
day-pr	Precipitation in daily table	Daily
ocean-tos-cmip7	Ocean temperature (CMIP7 atmos variant)	Monthly
SImon-sithick	Sea ice thickness	Monthly
6hrPlev-ta	Air temperature on pressure levels	6-hourly
Ofx-areacello	Ocean grid cell area (fixed)	Fixed (no time)

CMIP7 tests use these table mappings internally:

• atmos → CMIP6 Amon (monthly atmospheric)

• ocean → CMIP6 Omon (monthly ocean)

• seaIce → CMIP6 SImon (monthly sea ice)

• aerosol → CMIP6 AERmon (monthly aerosol)

When you run a CMIP7 test, it processes using the CMIP6 table but validates against CMIP7 standards.

Troubleshooting

No Tests Collected

Error: .. code-block:: text

collected 0 items

Causes & Solutions:

1. Wrong path:

   # Wrong
   pytest test_full_cmorisation.py
   
   # Correct
   pytest tests/integration/test_full_cmorisation.py
   

2. Pixi environment not activated:

   pixi shell -e dev
   pytest tests/integration/test_full_cmorisation.py
   

3. Invalid -k filter:

   # Wrong
   pytest tests/integration/test_full_cmorisation.py -k "Amon\-tas"
   
   # Correct
   pytest tests/integration/test_full_cmorisation.py -k "Amon and tas"
   

All Tests Skipped

Error: .. code-block:: text

372 skipped, 1 warning in X.XXs Reason: Ocean data directory not available; set ACCESS_MOPPY_DATA_ROOT

Solution:

# Check current setting
echo $ACCESS_MOPPY_DATA_ROOT

# Set it
export ACCESS_MOPPY_DATA_ROOT=/home/romain/PROJECTS/CMIP7_Test_data/esm-historical

# Verify path exists
ls $ACCESS_MOPPY_DATA_ROOT/output*/atmosphere/netCDF/ | head -5

Input Files Not Found

Error: .. code-block:: text

Required input files not available for Omon.tos; set ACCESS_MOPPY_DATA_ROOT

Solutions:

1. Verify data root:

   ls $ACCESS_MOPPY_DATA_ROOT/output*/ocean/
   

2. Check filename conventions:

   Ocean files should match: - *-variable-1monthly-mean*.nc (preferred) - *-variable-1mon-mean*.nc (alternate)

3. List available files:

   find $ACCESS_MOPPY_DATA_ROOT -name "*.nc" | grep ocean | head -10
   

Compliance Validation Failures

Error: .. code-block:: text

WCRP compliance validation failed mandatory checks: - check_dimensions: …

Solutions:

1. Review the specific check:

   Run with verbose output to see full details:

   pytest tests/integration/test_full_cmorisation.py -k "Amon-tas" \
       -vv --tb=long
   

2. Compare against known good output:

   Check if the issue is with your data or the CMORiser:

   # Run with PrePARE instead
   pytest tests/integration/test_full_cmorisation.py -k "Amon-tas" \
       --compliance-tool=prepare -v
   

3. Inspect the output file:

   ncdump -h /tmp/cmor_output_Amon_tas/tas_Amon_*.nc | head -50
   

Running Tests in Batch

Run Multiple Variables Sequentially

pytest tests/integration/test_full_cmorisation.py \
    -k "tas or pr or hus or huss" \
    -v --tb=short

# Output counts
pytest tests/integration/test_full_cmorisation.py \
    -k "tas or pr or hus or huss" \
    -q --tb=no

Test Monthly Data Only

# Run all monthly frequency tables
pytest tests/integration/test_full_cmorisation.py \
    -k "Amon or Lmon or AERmon or Omon or SImon or CFmon or Emon or Ofx" \
    -v --tb=short

# More concise - use 'mon'
pytest tests/integration/test_full_cmorisation.py \
    -k "mon or Emon or Ofx" \
    -v

Subset of Variables by Category

Atmospheric dynamics:

pytest tests/integration/test_full_cmorisation.py \
    -k "Amon and (ua or va or ta or ps or zg)" -v

Radiation:

pytest tests/integration/test_full_cmorisation.py \
    -k "Amon and (rsds or rlds or rsus or rlus)" -v

Water cycle:

pytest tests/integration/test_full_cmorisation.py \
    -k "Amon and (pr or prc or prw or evspsbl)" -v

Performance Considerations

Test Execution Time

• Single variable test: ~1-2 seconds

• All Amon variables: ~30-60 seconds

• All tables (CMIP6): ~3-5 minutes

• All tables (CMIP6 + CMIP7): ~5-8 minutes

Reduce Execution Time

1. Test specific table only:

   pytest tests/integration/test_full_cmorisation.py -k "Amon" -v
   # Instead of running all 372 tests
   

2. Skip compliance validation (faster):

   pytest tests/integration/test_full_cmorisation.py \
       --compliance-tool=none -q
   

3. Use parallel execution (requires pytest-xdist):

   pytest tests/integration/test_full_cmorisation.py -n auto -v
   

Parallel Execution

To install pytest-xdist:

pixi add -e dev pytest-xdist

Run tests in parallel:

pytest tests/integration/test_full_cmorisation.py -n auto -v

Run with 4 workers:

pytest tests/integration/test_full_cmorisation.py -n 4 -v

Tips & Best Practices

1. Always set ACCESS_MOPPY_DATA_ROOT before testing

   Add to your shell profile (.bashrc, .zshrc) to avoid repetition:

   export ACCESS_MOPPY_DATA_ROOT=/home/romain/PROJECTS/CMIP7_Test_data/esm-historical
   

2. Start with a single variable test

   When debugging, narrow the scope:

   # Good for debugging
   pytest tests/integration/test_full_cmorisation.py -k "Amon-tas" -vv -s
   

3. Use –tb=short for cleaner output

   Reduces verbosity while keeping error context:

   pytest tests/integration/test_full_cmorisation.py -k "Amon" --tb=short
   

4. Save test results to file

   For later analysis:

   pytest tests/integration/test_full_cmorisation.py -v \
       2>&1 | tee test_results_$(date +%Y%m%d_%H%M%S).log
   

5. Test CMIP6 before CMIP7

   CMIP6 is more stable; use it as baseline:

   pytest tests/integration/test_full_cmorisation.py \
       -k "not cmip7" -v
   
   # If passing, test CMIP7
   pytest tests/integration/test_full_cmorisation.py \
       -k "cmip7" -v
   

6. Use collection-only to verify filter

   Before running expensive tests:

   pytest tests/integration/test_full_cmorisation.py \
       -k "Amon and (tas or pr)" --collect-only -q
   # Verify you get the tests you expect
   

Additional Resources

• Getting Started - Basic CMORisation usage

• Variable Mapping Reference - Variable mapping details

• CMOR Output Compliance Testing - Compliance validation

• PrePARE Documentation

• CMOR Standards

For help or issues, refer to the project README or open an issue on GitHub.