To use your data with Chap, it must be provided as a CSV file that follows the format described below. A complete working example is available in example_data/laos_subset.csv.
Every dataset must contain these columns:
| Column | Description |
|---|---|
time_period |
Time period identifier (see format below) |
location |
Location identifier (e.g. district name or code) |
disease_cases |
Number of observed disease cases (the target variable) |
Models declare which climate covariates they need (e.g. rainfall, mean_temperature). The CSV must include a column for each covariate required by the model you are running. Covariate columns must not contain missing or NaN values.
Most models also require a population column with the population count for each location and time period. Check the model’s documentation to confirm whether it is needed.
If your CSV uses different column names than those expected by a model, you can provide a JSON mapping file via the --data-source-mapping option instead of renaming columns. See the eval command reference for details.
The table below summarises every column name that Chap recognises:
| Column | Required | Description |
|---|---|---|
time_period |
Yes | Time period in ISO format |
location |
Yes | Location identifier |
disease_cases |
Yes | Observed case count (target) |
population |
Model-dependent | Population for the location |
| covariate columns | Model-dependent | Climate/environmental covariates (e.g. rainfall, mean_temperature) |
target_masked |
Never in input | Added by Chap on datasets with masked periods; reserved, so input data may not contain it. See Masked periods |
The time_period column uses:
YYYY-MM format for monthly data (e.g., 2023-01)YYYY-Wnn format for weekly data (e.g., 2023-W01)All time periods in a dataset must use the same frequency (do not mix monthly and weekly).
Different columns have different rules for missing (NaN) values:
| Column | NaN allowed? | Notes |
|---|---|---|
time_period |
No | Every row must have a valid period |
location |
No | Every row must have a location |
disease_cases |
Yes | Surveillance data may have gaps; Chap tolerates NaN in the target. NaN can also mean the period is masked, see below |
| Covariates | No | Climate/environmental columns must be fully observed |
population |
Yes | Missing values are forward-filled by interpolation when present |
Chap does not perform imputation on covariates. If your data pipeline produces NaN values in covariate columns, those must be resolved before passing the CSV to Chap. Locations with missing covariate values will be rejected during dataset validation.
Every (location, time_period) combination in the dataset must be present as a row. Missing rows are not allowed – if a location covers 12 months, it must have all 12 rows. Chap will raise an error if any location has fewer time periods than others.
A dataset can have masked periods: stretches such as COVID years or a reporting outage that should not be used for training.
!!! warning “Masking deletes data”
When a dataset with masked ranges is created through the API, disease_cases inside the masked periods is stored as empty and the submitted values are deleted. They cannot be recovered or unmasked later. This also applies to every prediction-setup run, which inherits the masked ranges of its backtest’s dataset. To use those periods again, import the data again from its source without the range.
With --data-configuration on the command line the CSV file is left as it is and only the loaded copy is masked. Either way, this is what the model receives:
| Data the model receives | disease_cases |
target_masked |
|---|---|---|
| Training and historic data, masked period | NaN | 1 |
| Training and historic data, other periods | the observed value, or NaN if genuinely missing | 0 |
| Future data | not present | 0 |
The indicator is named target_masked, not after disease_cases, because it marks the period rather than one variable: 1 means the target is masked in that period.
A model that ignores target_masked sees masked periods as missing values, so a model that already handles NaN in the target keeps working. The column lets a model tell the two kinds of NaN apart and handle masked periods on purpose. A model can:
target_masked is 1 before fitting;y ~ .) or that stack historic and future data keep working.Datasets without masked periods have no target_masked column, so models see exactly the same data as before. Masked periods are set when a dataset is created, or with --data-configuration on the command line; see the eval command reference.
Time periods must be consecutive with no gaps. For example, monthly data that jumps from 2023-01 to 2023-03 (skipping February) is invalid.
In addition, every location must have exactly the same set of time periods. If your dataset covers three locations over 12 months, each location must have all 12 monthly rows.
Below is a complete example showing two locations with consecutive monthly periods:
time_period,location,disease_cases,rainfall,mean_temperature,population
2023-01,district_a,150,120.5,28.3,50000
2023-02,district_a,180,95.0,27.8,50000
2023-03,district_a,210,60.2,26.5,50000
2023-01,district_b,90,110.0,29.1,35000
2023-02,district_b,105,88.4,28.6,35000
2023-03,district_b,130,55.7,27.0,35000
time_period,rainfall,mean_temperature,disease_cases,location
2023-01,10,30,200,loc1
2023-02,2,30,100,loc1
time_period,rainfall,mean_temperature,disease_cases,location
2023-W01,12,28,45,loc1
2023-W02,8,29,52,loc1
Use the chap validate command to check that a CSV file meets all the requirements described above before running an evaluation.
The command checks for:
The command exits with code 0 when no errors are found, or code 1 if any errors are detected.
chap validate --dataset-csv example_data/laos_subset.csv
chap validate --dataset-csv example_data/faulty_datasets/missing_covariate_values.csv || true
To also verify that the dataset has the covariates and period type required by a specific model, pass --model-name:
chap validate \
--dataset-csv example_data/laos_subset.csv \
--model-name tests/fixture_models/naive_python_model_uv
If your CSV uses non-standard column names, supply a mapping file with --data-source-mapping:
chap validate \
--dataset-csv example_data/faulty_datasets/custom_column_names.csv \
--data-source-mapping example_data/faulty_datasets/custom_column_names_mapping.json