Quick start#

The shortest path from raw ACCESS output to CMORised NetCDF files on NCI Gadi.

Every ACCESS-MOPPy production run is the same four steps, whatever project you are CMORising for:

1. load the analysis3 environment
2. write a batch config file        (which variables + where the data is)
3. moppy-cmorise my_config.yml      (submits one PBS job per variable)
4. moppy-tui --db .../cmor_tasks.db (watch it run)

This page walks through those four steps with the smallest config that actually works. Once it makes sense, go to the page for your project — each one gives you the metadata block to paste in:


Watch it first (10 minutes)#

CMORising ACCESS output for CMIP7 FastTrack — a quick start with ACCESS-MOPPy walks through this page end to end, from loading the environment to watching the batch run.

Note

The narration voice in the video is synthetic and the script was drafted with AI assistance from this documentation. The workflow it shows is the real, maintained one.


Before you start#

You need:

  • an NCI account, with membership of xp65 (the module) and of the project that owns the model archive you want to read (often p73)

  • a scratch directory you can write to

  • the path to the run you want to CMORise — normally a payu archive root on /g/data


Step 1 — Load the environment#

module use /g/data/xp65/public/modules
module load conda/analysis3

That gives you moppy-cmorise, moppy-tui, and the rest of the command-line tools. Check it worked:

moppy-example-config        # prints a fully commented example config

Note

The first time the default personal publication profile writes output, ACCESS-MOPPy asks for your name, email, organisation, and ORCID, and stores them in ~/.moppy/user.yml. Answer the prompts once on a login node before submitting a personal-profile batch. Official ACCESS Consortium publication uses publication_profile: access-consortium and does not request these details.


Step 2 — Write a batch config#

A batch config is a YAML file that says what to CMORise, where the input is, and what PBS resources each job gets. This is a complete, working minimal example — one variable, ACCESS-ESM1-6, CMIP7:

# my_first_run.yml

# 1. What to CMORise — one PBS job is submitted per entry.
variables:
  - atmos.tas.tavg-h2m-hxy-u.mon.glb

# 2. Who the data belongs to (goes into the file metadata and filenames).
cmip_version: CMIP7
experiment_id: historical
source_id: ACCESS-ESM1-6
variant_label: r1i1p1f1
activity_id: CMIP

# 3. Where the raw model output is, and where output should go.
#    Point input_folder at the archive ROOT — MOPPy finds the files itself.
input_folder: "/g/data/<project>/archive/CMIP7/ACCESS-ESM1-6/historical/<run>"
output_folder: "/scratch/<project>/<user>/moppy_output/first_run"

# 4. PBS resources for each job, and how to set up its environment.
queue: "normalbw"
cpus_per_node: 12
mem: "64GB"
walltime: "04:00:00"
scheduler_options: "#PBS -P <project>"
storage: "gdata/p73+gdata/xp65+scratch/<project>"

worker_init: |
  module use /g/data/xp65/public/modules
  module load conda/analysis3

That is the whole structure. Everything else in the configuration reference is optional tuning on top of these four blocks.

Two things worth knowing straight away:

You do not list input files. Set input_folder to the archive root and MOPPy uses the model’s built-in file-discovery rules to find the right native files for each variable. Only reach for file_patterns if your run has an unusual layout.

Start small. Run one to five variables first and check the output before you submit a list of two hundred. Failures are much easier to read when there are three jobs than when there are three hundred.

To restrict a long run to a few years while you are testing:

start_year: 1850
end_year: 1850

Step 3 — Run it#

From a Gadi login node:

moppy-cmorise my_first_run.yml

The command records every variable in a tracker database, submits a single PBS monitor job, and exits. The monitor job then submits one worker job per variable and retries any that fail.

Terminal showing moppy-cmorise submitting a batch and printing the monitor job ID.

Submitting a batch. The example config behind this screenshot lists three variables, hence three tasks; the paths, user name and PBS job ID are examples too, and yours will differ.#

Because the monitor — not your shell — owns the batch, you can log out straight away. A dropped SSH connection will not stop the run.


Step 4 — Check the status#

The tracker database is written to <output_folder>/cmor_tasks.db. Point the terminal dashboard at it:

moppy-tui --db /scratch/<project>/<user>/moppy_output/first_run/cmor_tasks.db

It refreshes live and shows each variable as pending, running, completed, or failed, with error messages for the failures.

moppy-tui showing one completed variable, one running and one pending.

The same three-variable batch a few minutes in: one variable done, one running, one still queued.#

The batch is finished when the progress bar reaches 100% and nothing is left running or pending:

moppy-tui showing a batch with all three variables completed.

All three variables CMORised. Press q to quit the dashboard.#

Useful variations:

# Only show what went wrong, or what is still going
moppy-tui --status failed,running --db <output_folder>/cmor_tasks.db

# One snapshot and exit — good for a quick check or a cron job
moppy-tui --once --db <output_folder>/cmor_tasks.db

# Full JSON report of the batch, including PBS logs for failed tasks
moppy-batch-report --db <output_folder>/cmor_tasks.db

Step 5 — Look at the output#

CMORised NetCDF files are written under output_folder. Per-variable PBS logs (.out and .err) land in <output_folder>/logs/<variable>/, which is the first place to look when a job fails.

To check a file against the CMIP7 output QC rules:

moppy-qc /path/to/output.nc

Where to go next#

I am working on…

Go to

CMIP7 FastTrack (ACCESS-ESM1-6)

CMIP7 FastTrack quick start

CMIP6Plus

CMIP6Plus quick start

TIPMIP

TIPMIP quick start

And when you outgrow the quick start: