Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Timing chap-core: chap-devops bench

chap-devops bench puts load on chap-core in two ways:

  • With no --write, it times the requests that the Modeling App sends to chap-core when a person opens a page. Use it to find which page is slow, and to see if it gets slower as the data grows. It changes nothing.
  • With --write, it imports a dataset and runs several backtests on it at the same time, then checks what each one stored. See Backtests under load.

chap-devops bench dataset makes the synthetic datasets for both. See Synthetic datasets.

chap-devops bench                        # every page, 3 times each, 1 at a time
chap-devops bench 12 --repeat 10         # the pages of backtest 12
chap-devops bench --page prediction      # only the prediction page
chap-devops bench --page evaluation,dataset --prediction 7
chap-devops bench --parallel 8           # 8 requests open at the same time

The pages

--page selects the pages, separated by commas. Without it, bench times all of them. The requests follow the Modeling App: each page is a list of steps. The requests of one step go out at the same time, and the next step waits for their answers. Every page starts with /system/info, because the Modeling App waits for it before it shows anything.

PageWhat the Modeling App showsThe requests to chap-core
evaluationOne evaluation, as it opensbacktest info; then configured-models, the plot list, the specification and the version-dependent widgets; then evaluation-entry and actualCases for the whole country (adm0)
compareThe comparison of an evaluationthe backtests list; the specification and compatible-backtests; the info of each baseline that is done; then evaluation-entry and actualCases for the first 10 org units
plotsThe custom plots, after a person picks onethe plot list; facet-coords of each plot; the first subplot of each plot
benchmarkThe benchmark of the backtest specificationthe specification, jobs; then configured-models and the parameters
datasetThe dataset details pagethe dataset of the backtest, with every observation
predictionOne predictionthe prediction and configured-models; then prediction-entry, alerts, alert-policies, and actualCases of the dataset for every org unit of the prediction, with isDatasetId=true

The backtest is the one you give, or the newest in chap-core’s list. The prediction is the one --prediction gives, or the newest. The benchmark and dataset pages use the specification and the dataset of the backtest.

bench first reads the version, the backtest, its specification, the prediction and the plots, and it does not time these reads. With the version, it leaves out the requests that the Modeling App does not send to an older chap-core, for example backtest-parameters before 2.4. If chap-core does not give its version, bench sends all of them.

Some requests are not in bench:

  • the requests to DHIS2 itself, for example the names of the org units;
  • the requests that start only when a person clicks, for example the outbreak scores of an alert policy;
  • the thresholds request of the prediction page, which needs the alert policies of the prediction.

If a page cannot be timed, for example because there is no prediction, bench leaves it out. With -v it says why. If you selected the page with --page, it gives a warning.

The requests go round by round, so with --parallel 1 one request is never sent two times in a row.

The report

The table has one row for each request of each page: PAGE, REQUEST, the count N, FAILED, then MIN, P50, P95 and MAX, then SIZE and GZIP. After the table:

  • timed ...: the count of requests and pages, and the backtest and the prediction.
  • chap-core time per page, from the p50s: for each page, the p50 of the slowest request in each step, added up over the steps. This is the part of opening the page that chap-core takes, with one person on the server. It is not the time a person waits for the page: it leaves out the Modeling App’s own requests to DHIS2 (for example the names of the org units), the time of the DHIS2 route, and the work of the browser. Use it to compare two runs on the same setup.
  • slowest: the request with the highest p50.
  • measured on: what the numbers are from. See What was measured.

The times are the time until the whole answer came and was unpacked. A failed request counts with the time it took, because a person waited that long. Each failed request adds a warning with what chap-core said. --json gives every row with its page and step, min_ms, p50_ms, p95_ms, max_ms, bytes, wire_bytes and the count of each HTTP status. pages gives each page with its steps and its time to open (ms), and skipped gives each page that was left out, with the reason.

gzip

bench asks chap-core for gzip, as the DHIS2 route asks for it. The route unpacks the answer, so the browser gets plain JSON. But chap-core packs every answer that goes to the route, and the times include that work. On a large answer, the time to pack can be more than the time to make the answer.

  • SIZE (bytes in --json) is the unpacked answer, which the browser gets.
  • GZIP (wire_bytes) is the size on the network between chap-core and the DHIS2 route. It is the same as SIZE if chap-core did not pack the answer. chap-core does not pack answers under 1,000 bytes.

All versions of bench ask for gzip. But to compare two chap-core versions, run the same bench version on both.

What was measured

Two runs can only be compared if they ran on the same setup. bench, also with --write, gives a measured on line and server in --json:

  • chap_core_version and revision, from chap-core’s /system/info.
  • api_workers: the gunicorn worker processes of the chap service.
  • worker_processes: the celery worker processes of the worker service. Celery starts one for each CPU it sees (BUGS.md).
  • docker_cpus and docker_memory_bytes: what the docker engine has. On macOS and Windows this is the docker VM.
  • machine and cpus: the machine that chap-devops runs on.
  • gzip: whether chap-core packed an answer with gzip when bench asked for it, as the DHIS2 route asks. bench checks this with one untimed GET /v1/crud/configured-models. It is null when that answer was under 1,000 bytes, which chap-core never packs.
  • transport: the Server and Via headers of that answer. A proxy in front of chap-core, or the DHIS2 route, shows here. Such a proxy can pack or unpack the answers, or hold them back, so its time is in the numbers.

The measured on line says answers packed with gzip, or answers not packed, though gzip was asked for. The second is a sign that something between bench and chap-core unpacks the answers, or that chap-core does not pack them. It names the server when it is not chap-core’s own uvicorn.

bench counts the processes that run, with docker compose top. It does not read the settings, because chap-core can ignore a setting. It counts them only for a chap-core that this deployment runs, so with --url the counts and the docker values are not known.

One at a time and many at a time

A Modeling App page sends several requests at the same time. --parallel 8 does the same. If the server answers one request at a time, the fast requests wait for the slow ones. Then their p50 goes up, but the total time does not go down. For one evaluation (18 org units, 72 months), on chap-core master of 2026-11-05, with the requests of a bench from before the pages:

--parallel 1--parallel 8
total for 125 requests16.1 s14.9 s
backtests p5010 ms737 ms
configured-models p5012 ms748 ms
predicted_vs_actual subplot p50674 ms2.67 s

CPU, memory and the database: –resources

--resources records what the services of this deployment used while bench ran, phase by phase. With --write the phases are import, backtests and prediction (with --predict). Without --write there is one phase, requests.

chap-devops bench --write --model seasonal_median --backtests 1 --predict --resources

For each phase it gives:

  • for chap, worker, postgres and redis: the CPU seconds, the average and the peak cores, and the peak memory, from docker stats;
  • for Postgres: the counters of pg_stat_database that grew in the phase, for example xact_commit, tup_returned, tup_inserted and temp_bytes;
  • for Valkey: total_commands_processed, total_net_input_bytes, total_net_output_bytes, and used_memory_peak at the end.
import (8.58 s): chap 2.2 CPU s (peak 1.0 cores, 729.9 MB), worker 4.9 CPU s (peak 0.8 cores, 1.7 GB), postgres 1.4 CPU s (peak 0.3 cores, 198.3 MB), redis 0.0 CPU s (peak 0.0 cores, 303.6 MB); postgres read 187662 and wrote 187205 rows
backtests (33.74 s): chap 8.2 CPU s (peak 1.0 cores, 784.8 MB), worker 77.3 CPU s (peak 7.8 cores, 3.8 GB), postgres 3.6 CPU s (peak 0.3 cores, 220.0 MB), redis 0.2 CPU s (peak 0.0 cores, 305.6 MB); postgres read 1385708 and wrote 131990 rows

docker stats measures for about a second each time, so that is the interval of the samples. A phase that is shorter than a second has few samples or none. The CPU seconds are the average cores times the length of the phase. The sampling itself uses some CPU, so compare runs that both used --resources, or both did not. --json gives each phase under resources.

--resources needs docker to see chap-core, so it works only for the chap-core of this deployment, not with --url. chap-core’s Postgres does not load pg_stat_statements, so there are no top statements. To get them, add shared_preload_libraries=pg_stat_statements to the postgres service, and run create extension pg_stat_statements in the database.

A second user: –second-user

--second-user shows what the run does to the other people on the server. During --write, a thread asks for GET /v1/crud/configured-models every half second, the list that most pages of the Modeling App ask for first. Each phase of the run gets the p50, p95 and longest time of the requests that started in it:

second user, import: 74 requests of configured-models, p50 13 ms, p95 115 ms, max 4.12 s
second user, backtests: 290 requests of configured-models, p50 18 ms, p95 164 ms, max 647 ms

That run imported 1,500 locations and 156 weeks on chap-core master with one API worker. While chap-core read the import, the second user waited up to 4.12 s for a list that otherwise takes 13 ms. Compare it with more API workers (--workers 1,4). --json gives the phases under second_user.

API workers: –workers

--workers 1,4 runs the same bench once for each count of API worker processes, and puts the numbers side by side. It works with and without --write:

chap-devops bench --page prediction --parallel 8 --workers 1,4
chap-devops bench --write --model seasonal_median --backtests 2 --workers 1,2

For each count, it sets the count as chap-devops restart --api-workers N does: it writes CHAP_API_WORKERS to .env, recreates only the chap service, and waits until chap-core answers. Then it runs the same bench command again with --json. At the end it sets the count back to what it was. It checks every count before the first run: more than one worker needs a chap-core newer than v2.7.0.

This run is from chap-core master (2.8.0.dev0) on an Apple Silicon Mac, with 100 org units and 156 weeks:

MEASURE           1 WORKERS  2 WORKERS
import            2.80 s     2.79 s
import request    768 ms     720 ms
backtest request  982 ms     948 ms
slowest job       6.67 s     6.41 s
whole run         28.79 s    28.06 s
ran the same bench with 2 of 2 API worker counts (1, 2); set back to 1

The rows are the time per page and the p50 of each request, and with --write the import, the requests and the slowest job. --json gives each run whole under runs. --workers changes this deployment’s chap-core, so it cannot be used with --url.

An empty database: –fresh-database

--fresh-database empties chap-core’s database before a --write run, so the run does not find the rows of the runs before it. It stops chap and worker, drops the database and makes it again, and starts them. chap-core makes its tables and its configured models again when it starts.

  • With --write alone, it empties the database once, before the run.
  • With --sizes, it empties the database before each size.
  • With --workers, it empties the database before each count of API workers, so each count starts from the same state.

This deletes all the data in chap-core’s database: use it only on a deployment of its own. It does not work with --url or with --dataset. The closing lines say that the database was emptied, and --json gives fresh_database.

chap-devops bench --write --model seasonal_median --backtests 1 --predict \
  --locations 1500 --periods 365 --fresh-database

Dataset sizes: –sizes

--sizes runs --write once for each dataset size, and puts the numbers side by side. A size is the locations, an x, and the periods with w for weeks or m for months:

chap-devops bench --write --model seasonal_median --backtests 1 \
  --sizes 500x520w,1500x520w,3000x520w,1500x120m --fresh-database --resources

Each size runs as bench --write with its --locations, --periods and --period-type, and with the other options of the command. The other dataset options, such as --covariates and --format, are the same for every size.

--fresh-database empties chap-core’s database before each size, so a run does not find the rows of the run before it. See An empty database.

This run is from chap-core master (2.8.0.dev0) on an Apple Silicon Mac:

MEASURE           50x156w  100x60m
import            4.72 s   4.54 s
import request    562 ms   438 ms
backtest request  409 ms   361 ms
slowest job       4.81 s   3.59 s
whole run         17.84 s  14.87 s
ran 2 of 2 sizes (50x156w, 100x60m), each on an empty database

--json gives each run whole under runs, with its size.

Before and after a change

To see what a change in chap-core does, time the same page before and after it, with the same data and the same bench version:

chap-devops --json bench --page prediction --repeat 10 > before.json
# change chap-core and start it again
chap-devops --json bench --page prediction --repeat 10 > after.json

Compare pages and the p50_ms of each row in the two files.

Another chap-core

--url times a chap-core that this deployment does not run. The token comes from CHAP_API_TOKEN in the shell, as for chap-devops api:

CHAP_API_TOKEN=... chap-devops bench --url https://chap.example.org --repeat 5

Run it while a backtest runs on the worker to see how much a running job slows the pages. Use chap-devops top --record at the same time to see which container is busy.

Backtests under load: –write

--write adds rows to chap-core’s database. Do not use it on a deployment that people use: run it on a deployment of its own.

chap-devops bench --write --model seasonal_median
chap-devops bench --write --model naive_model,seasonal_median --backtests 1
chap-devops bench --write --model seasonal_median --backtests 8 --locations 300 --periods 156
chap-devops bench --write --model 12 --dataset 5   # an existing dataset

The steps follow the Modeling App’s evaluation and benchmark pages:

  1. It makes a synthetic dataset from the dataset options and imports it with make-dataset. With --dataset ID, it uses that dataset instead.
  2. It starts --backtests runs (default 4) at the same time. A run is one create-backtests request with all the models of --model, under one backtest specification, as the benchmark page sends it. It starts all the runs before it polls any job.
  3. Each backtest job queues the baselines that the specification has no run of yet (run_baseline jobs: persistence, seasonal_shift_persistence, global_median, seasonal_median). The benchmark page is complete only when they end, so bench waits for them too.
  4. It polls the jobs until each one has ended, for at most --wait seconds.
  5. For each backtest that ended, the baselines included, it checks what chap-core stored.
  6. It deletes the dataset it made and all the backtests. --keep keeps them. It never deletes a dataset that --dataset named.

--model takes one or more names or ids, separated by commas, as chap-devops api GET /v1/crud/configured-models shows them, for example seasonal_median or chapkit_ewars_model:default. Without it, the error lists the names.

A backtest passes when its job ended with SUCCESS, and chap-core stored one forecast for each org unit, split and forecast period, and stored aggregate metrics. The count of forecasts is the length of evaluation-entry at one quantile. The expected count comes from the backtest row: orgUnits times splitPeriods times nPeriods. Anything else is a failure:

  • the job ended with FAILURE, or it still ran after --wait seconds;
  • chap-core stored fewer or more forecasts than expected, or none;
  • chap-core stored no aggregate metrics.

This run is from chap-core master (2.8.0.dev0) on an Apple Silicon Mac, with 18 org units and 156 weeks:

#  MODEL                                  STATUS   REQUEST  QUEUED  RUN     BACKTEST  FORECASTS  METRICS  RESULT
1  naive_model                            SUCCESS  103 ms   0 ms    2.46 s  1         1512/1512  13       ok
1  seasonal_median                        SUCCESS  103 ms   0 ms    2.98 s  2         1512/1512  13       ok
-  seasonal_median (baseline)             SUCCESS  -        -       2.88 s  4         1512/1512  13       ok
-  global_median (baseline)               SUCCESS  -        -       2.80 s  3         1512/1512  13       ok
-  seasonal_shift_persistence (baseline)  SUCCESS  -        -       3.19 s  5         1512/1512  13       ok
-  persistence (baseline)                 SUCCESS  -        -       3.20 s  6         1512/1512  13       ok
2 of 2 backtests of naive_model, seasonal_median on dataset 2 passed, in 11.11 s
4 of 4 baseline backtests passed

The times of each job:

  • REQUEST: the create-backtests request of its run. chap-core loads the dataset in this request, so it grows with the dataset.
  • QUEUED: from the answer to the request to the start of the job.
  • RUN: from the start to the end of the job, from start_time and end_time in /v1/jobs. This is exact, also for a job that is shorter than a poll.

QUEUED compares the clock of this machine with the clock of chap-core, so it is only exact when they are the same machine. If a backtest fails, the command exits with status 1 and gives a warning with the reason from the job log. --json gives each backtest with its model, baseline, status, request_seconds, queued_seconds, seconds, backtest_id, forecasts, expected_forecasts, metrics and problem. It also gives import_request_seconds, import_seconds and backtest_request_seconds.

A prediction: –predict

--predict also makes one prediction, so the prediction page has something to show. It uses the first backtest that passed, as the Modeling App does:

  1. It makes a prediction setup on the backtest (POST /v1/crud/prediction-setups).
  2. It runs the setup with the synthetic dataset, one object for each value (providedData), with DHIS2 period ids such as 2024W7 (POST /v1/crud/prediction-setups/{id}/run). chap-core stores these values as a new dataset.
  3. It waits for the job, then checks that chap-core stored one forecast for each org unit and forecast period.

--predict cannot be used with --dataset, because the run needs the values of the dataset. With weekly data, give at least 105 periods: chap-core refuses a prediction on 104 weeks (see troubleshooting). Without --keep, bench deletes the prediction, its dataset and the setup before the backtests. With --keep, it gives the command that times the prediction page:

chap-devops bench --write --model seasonal_median --backtests 1 --predict --keep \
  --locations 1500 --periods 365
chap-devops bench --page prediction --prediction 7 --repeat 10

If the prediction fails, the command exits with status 1. --json gives it as prediction, with setup_id, job, status, seconds, prediction_id, dataset_id, entries, expected_entries and problem.

To compare settings, run the same command before and after the change, for example with one worker process and then with four. Use the same seed (the default) to get the same data. If jobs are still running after --wait, chap-devops keeps the rows and says so: chap-devops jobs shows them.

Synthetic datasets: chap-devops bench dataset

chap-devops bench dataset makes a dataset with random values. The same options always give the same dataset.

chap-devops bench dataset                        # 18 locations, 104 weeks, in .
chap-devops bench dataset --out data --locations 2000 --periods 520 --covariates 10
chap-devops bench dataset --import --period-type month --periods 72
OptionDefaultWhat it sets
--locations N18the number of org units, loc01 to loc18
--periods N104the number of periods for each location
--period-typeweekweek (ISO weeks) or month
--cases N100the typical number of cases in one location and period
--covariates N3rainfall, mean_temperature, population, then covariate_4 and more
--seed N1the start of the random values
--formatrowshow --import and --write send the values: rows or columns

The periods always end with the last week or month of 2024. Each location has its own size, season and population. The cases follow the season and the rainfall of the period before, with noise, so a model has something to learn. The values are not realistic.

Without --import, it writes <name>.csv and <name>.geojson to --out (default .). The CSV has the columns that chap eval reads, and chap eval finds the GeoJSON of the same name next to it:

chap-devops bench dataset --out data
chap-devops chap eval --model-name auto_arima_chapkit \
  --dataset-csv data/synthetic-18x104w-c100-k3-s1.csv --output-file eval.nc

The GeoJSON has one square for each location, on a grid in the sea west of Africa, so no one mistakes it for a real place.

With --import, it sends the dataset to chap-core’s make-dataset, waits until chap-core has stored it, and gives the dataset id. --format selects how it sends the values:

  • rows (the default) sends one object for each value (providedData), with DHIS2 period ids such as 2024W7 and 202407. The Modeling App sends a dataset this way, so the import takes the same path in chap-core.
  • columns sends one list for each column (providedDataColumns). The body is about four times smaller. This needs chap-core v2.7.0 or later.

The closing line gives the time of the import, split in two: the request, in which chap-core reads and checks the values, and the job, in which the worker stores them. --json gives request_seconds and import_seconds.

--name sets the name. The default name says what the dataset is made of: synthetic-18x104w-c100-k3-s1 is 18 locations, 104 weeks, 100 cases, 3 covariates and seed 1. bench --write adds -bench-<time> to it.