The eval command runs a rolling-origin backtest evaluation on a disease prediction model and exports results in NetCDF format for analysis with scientific tools.
chap eval --model-name <MODEL> --dataset-csv <CSV_FILE> --output-file <OUTPUT.nc> [OPTIONS]
This command evaluates a single model by:
.geojson extension)n-retrain) and generating probabilistic forecasts for each splitThe output NetCDF file can be used with plot-backtest for visualization and export-metrics for metric extraction.
| Parameter | Description |
|---|---|
--model-name |
Model identifier. Can be a local directory path, GitHub URL, or chapkit service URL |
--dataset-csv |
Path to CSV file containing disease data |
--output-file |
Path for output NetCDF file (use .nc extension) |
Control how the evaluation splits and processes data:
| Parameter | Description | Default |
|---|---|---|
--backtest-params.n-periods |
Forecast horizon (number of periods to predict ahead) | 3 |
--backtest-params.n-splits |
Number of test windows to evaluate | 7 |
--backtest-params.stride |
Step size (in periods) between consecutive splits | 1 |
--backtest-params.n-retrain |
Number of times the model is trained, evenly spaced across the splits | 1 |
n-periods=3 means 3-month forecasts.stride=1 means every period gets a forecast; stride=2 skips every other period.Splits are counted back from the end of the dataset, so the last test window ends at the final period. Example: With 36 months of data, n-periods=3, n-splits=7, stride=1:
See How Split Indices Are Calculated for the formula.
Control model execution environment:
| Parameter | Description | Default |
|---|---|---|
--run-config.is-chapkit-model |
Set when using a chapkit REST API model | false |
--run-config.ignore-environment |
Skip automatic environment setup | false |
--run-config.debug |
Enable verbose debug logging | false |
--run-config.log-file |
Path to write log output | None |
--run-config.run-directory-type |
Directory handling: latest, timestamp, or use_existing |
timestamp |
| Parameter | Description | Default |
|---|---|---|
--model-configuration-yaml |
Path to YAML file with model-specific parameters | None |
--historical-context-years |
Years of historical data to include for plotting context | 6 |
--data-source-mapping |
Path to JSON file for column name mapping | None |
--data-configuration |
Path to JSON or YAML file with data settings, such as masked ranges | None |
--dry-run |
Write data files and print commands without executing the model | false |
Use --dry-run to inspect what commands would be executed and verify data files without running the model. This writes all input data (training, historic, and future CSVs) to the run directory and prints the fully-assembled commands for each backtest split.
chap eval \
--model-name ./my_model \
--dataset-csv ./data.csv \
--output-file ./eval.nc \
--dry-run
Example output:
[dry-run] cd runs/my_model/2025-01-15_10-30-00_abc123 && uv run python train.py training_data.csv model
[dry-run] cd runs/my_model/2025-01-15_10-30-00_abc123 && uv run python predict.py model historic_data_2024-01-01.csv future_data_2024-01-01.csv predictions_2024-01-01.csv
[dry-run] cd runs/my_model/2025-01-15_10-30-00_abc123 && uv run python predict.py model historic_data_2024-02-01.csv future_data_2024-02-01.csv predictions_2024-02-01.csv
...
Each predict split writes its own date-suffixed data files, so you can inspect the exact inputs for any split. The printed commands can be copy-pasted into a terminal to run individual steps manually.
When your CSV column names don’t match the names expected by the model, use --data-source-mapping to provide a JSON mapping file.
The JSON file maps model covariate names (keys) to CSV column names (values):
{
"model_covariate_name": "csv_column_name"
}
If your model expects a column named rainfall but your CSV has precipitation_mm:
mapping.json:
{
"rainfall": "precipitation_mm",
"mean_temperature": "temp_avg_celsius"
}
Usage:
chap eval \
--model-name ./my_model \
--dataset-csv ./data.csv \
--output-file ./eval.nc \
--data-source-mapping ./mapping.json
Use --data-configuration to mask disease_cases in periods that should not be used for training, such as COVID years or a reporting outage. On the command line the values are not deleted from the CSV; only the loaded copy is masked. (Creating a dataset with masked ranges through the REST API is different: it deletes the values inside the ranges, see Masked periods.) Inside a masked range the model receives disease_cases as empty (NaN) and an extra column target_masked set to 1. Everywhere else target_masked is 0, so a model can tell a masked period from a genuinely missing value.
The file is JSON or YAML, in the same shape as the maskedRanges field of the REST dataset payload. Both bounds are inclusive and use the dataset’s period type: months as 2021-06 or 202106, weeks as 2021W01 or 2021-W1.
data.json:
{"maskedRanges": [{"start": "2021-06", "end": "2021-08", "name": "reporting outage"}]}
data.yaml:
maskedRanges:
- start: "2021-06"
end: "2021-08"
name: reporting outage
name is optional and only labels the range.
Usage:
chap eval \
--model-name ./my_model \
--dataset-csv ./data.csv \
--output-file ./eval.nc \
--data-configuration ./data.json
The same option is available on chap evaluate-ensemble (with --dataset-csv) and chap validate.
start may not be after end.n_periods + (n_splits - 1) * stride periods. The backtest is rejected before any model runs.target_masked column of its own; that name is reserved.disease_cases. It does have target_masked, always 0, because a forecast period is never masked.On a masked dataset the training, historic and future data all have one extra column, target_masked. It is 1 in masked periods and 0 elsewhere, and always 0 in future data. Because the column is in both history and future, models that stack the two tables or fit on every column keep working. A model that uses every column will pick the indicator up as a covariate. That is harmless, but a model can also use it explicitly, for example to drop masked rows before fitting.
Models that follow the Chap model specification can be loaded from:
https://github.com/dhis2-chap/minimalist_example_r/path/to/model or ./modelchap eval \
--model-name https://github.com/dhis2-chap/minimalist_example_r \
--dataset-csv ./data/vietnam.csv \
--output-file ./eval.nc
Chapkit models are REST API-based prediction services. Use --run-config.is-chapkit-model flag:
From a running service:
chap eval \
--model-name http://localhost:8000 \
--dataset-csv ./data/vietnam.csv \
--output-file ./eval.nc \
--run-config.is-chapkit-model
From a local directory (auto-starts the service):
chap eval \
--model-name /path/to/chapkit-model \
--dataset-csv ./data/vietnam.csv \
--output-file ./eval.nc \
--run-config.is-chapkit-model
When providing a directory path with --run-config.is-chapkit-model, Chap automatically:
uv run fastapi devThe service’s own output is forwarded to Chap’s debug log (--run-config.debug), and the last lines are included in the error message if the service fails to start.
Use --run-config.as-chapkit to serve an MLproject model (a local directory or GitHub URL) with chapkit mlproject run instead of running it with Chap’s runners. The model is then exercised the same way it is in production: through chapkit’s REST API, with its typed user options and adapters.
chap eval \
--model-name https://github.com/dhis2-chap/minimalist_example_uv \
--dataset-csv ./data/vietnam.csv \
--output-file ./eval.nc \
--run-config.as-chapkit
Chap copies the model into the run directory, prepares its environment, starts the service on a free local port, runs the evaluation and stops the service when done. The service never registers with an orchestrator. --dry-run is not supported in this mode, since chapkit runs the model commands itself.
The model’s scripts read config.yml in chap-core’s layout, with the user options under user_option_values. A model whose scripts expect every option at the top level declares config_format: flat under meta_data in its MLproject (chapkit 2.4.0 or newer).
| MLproject environment | What Chap does |
|---|---|
none, or --run-config.ignore-environment |
Runs the commands in the current environment |
uv_env |
Runs uv sync in the run directory and starts chapkit through uv run |
renv_env |
Runs renv::restore() before starting the service |
docker_env, conda_env, python_env |
Not supported |
For a model with a docker_env, start a chapkit image yourself and pass its URL instead:
docker run -p 8000:8000 -v $(pwd):/work ghcr.io/dhis2-chap/chapkit-r-inla-cli
chap eval --model-name http://localhost:8000 --dataset-csv ./data/vietnam.csv --output-file ./eval.nc
The CSV file must contain:
| Column | Description |
|---|---|
time_period |
Time period identifier (ISO format: YYYY-MM for monthly, YYYY-Www for weekly) |
location |
Location identifier matching GeoJSON feature IDs |
disease_cases |
Number of disease cases (target variable) |
| Climate covariates | Model-specific columns (e.g., rainfall, mean_temperature) |
population |
(Optional) Population for the location |
Example CSV:
time_period,location,disease_cases,rainfall,mean_temperature,population
2020-01,region_1,45,120.5,28.3,50000
2020-01,region_2,32,98.2,27.1,35000
2020-02,region_1,52,145.8,29.1,50000
...
Chap automatically looks for a GeoJSON file with the same base name as the CSV:
vietnam_data.csvvietnam_data.geojson (auto-discovered in the same directory)The GeoJSON features must have id properties matching the location values in the CSV.
The output is a NetCDF file containing:
Use plot-backtest to visualize and export-metrics to extract metrics from the output file.
chap eval \
--model-name https://github.com/chap-models/chap_auto_ewars \
--dataset-csv ./example_data/laos_subset.csv \
--output-file ./eval_ewars.nc
chap eval \
--model-name ./my_model \
--dataset-csv ./data.csv \
--output-file ./eval.nc \
--backtest-params.n-periods 6 \
--backtest-params.n-splits 12 \
--backtest-params.stride 2
chap eval \
--model-name https://github.com/dhis2-chap/minimalist_example_r \
--dataset-csv ./data.csv \
--output-file ./eval.nc \
--model-configuration-yaml ./model_config.yaml
chap eval \
--model-name ./my_model \
--dataset-csv ./data.csv \
--output-file ./eval.nc \
--dry-run
chap eval \
--model-name ./my_model \
--dataset-csv ./data.csv \
--output-file ./eval.nc \
--run-config.debug \
--run-config.log-file ./evaluation.log
# Step 1: Evaluate the model
chap eval \
--model-name https://github.com/chap-models/chap_auto_ewars \
--dataset-csv ./data/vietnam.csv \
--output-file ./results/ewars_eval.nc \
--backtest-params.n-splits 10
# Step 2: Generate visualization
chap plot-backtest \
--input-file ./results/ewars_eval.nc \
--output-file ./results/ewars_plot.html
# Step 3: Export metrics
chap export-metrics \
--input-files ./results/ewars_eval.nc \
--output-file ./results/metrics.csv