Timing chap-core: chap-devops bench
chap-devops bench puts load on chap-core in two ways:
- With no
--write, it times the requests that the Modeling App sends to chap-core when a person opens a page. Use it to find which page is slow, and to see if it gets slower as the data grows. It changes nothing. - With
--write, it imports a dataset and runs several backtests on it at the same time, then checks what each one stored. See Backtests under load.
chap-devops bench dataset makes the synthetic datasets for both. See
Synthetic datasets.
chap-devops bench # every page, 3 times each, 1 at a time
chap-devops bench 12 --repeat 10 # the pages of backtest 12
chap-devops bench --page prediction # only the prediction page
chap-devops bench --page evaluation,dataset --prediction 7
chap-devops bench --parallel 8 # 8 requests open at the same time
The pages
--page selects the pages, separated by commas. Without it, bench times
all of them. The requests follow the Modeling App: each page is a list of
steps. The requests of one step go out at the same time, and the next step
waits for their answers. Every page starts with /system/info, because the
Modeling App waits for it before it shows anything.
| Page | What the Modeling App shows | The requests to chap-core |
|---|---|---|
evaluation | One evaluation, as it opens | backtest info; then configured-models, the plot list, the specification and the version-dependent widgets; then evaluation-entry and actualCases for the whole country (adm0) |
compare | The comparison of an evaluation | the backtests list; the specification and compatible-backtests; the info of each baseline that is done; then evaluation-entry and actualCases for the first 10 org units |
plots | The custom plots, after a person picks one | the plot list; facet-coords of each plot; the first subplot of each plot |
benchmark | The benchmark of the backtest specification | the specification, jobs; then configured-models and the parameters |
dataset | The dataset details page | the dataset of the backtest, with every observation |
prediction | One prediction | the prediction and configured-models; then prediction-entry, alerts, alert-policies, and actualCases of the dataset for every org unit of the prediction, with isDatasetId=true |
The backtest is the one you give, or the newest in chap-core’s list. The
prediction is the one --prediction gives, or the newest. The benchmark
and dataset pages use the specification and the dataset of the backtest.
bench first reads the version, the backtest, its specification, the
prediction and the plots, and it does not time these reads. With the
version, it leaves out the requests that the Modeling App does not send to
an older chap-core, for example backtest-parameters before 2.4. If
chap-core does not give its version, bench sends all of them.
Some requests are not in bench:
- the requests to DHIS2 itself, for example the names of the org units;
- the requests that start only when a person clicks, for example the outbreak scores of an alert policy;
- the
thresholdsrequest of the prediction page, which needs the alert policies of the prediction.
If a page cannot be timed, for example because there is no prediction,
bench leaves it out. With -v it says why. If you selected the page with
--page, it gives a warning.
The requests go round by round, so with --parallel 1 one request is never
sent two times in a row.
The report
The table has one row for each request of each page: PAGE, REQUEST, the
count N, FAILED, then MIN, P50, P95 and MAX, then SIZE and
GZIP. After the table:
timed ...: the count of requests and pages, and the backtest and the prediction.chap-core time per page, from the p50s: for each page, the p50 of the slowest request in each step, added up over the steps. This is the part of opening the page that chap-core takes, with one person on the server. It is not the time a person waits for the page: it leaves out the Modeling App’s own requests to DHIS2 (for example the names of the org units), the time of the DHIS2 route, and the work of the browser. Use it to compare two runs on the same setup.slowest: the request with the highest p50.measured on: what the numbers are from. See What was measured.
The times are the time until the whole answer came and was unpacked.
A failed request counts with the time it took, because a person waited that
long. Each failed request adds a warning with what chap-core said. --json
gives every row with its page and step, min_ms, p50_ms, p95_ms,
max_ms, bytes, wire_bytes and the count of each HTTP status. pages
gives each page with its steps and its time to open (ms), and skipped
gives each page that was left out, with the reason.
gzip
bench asks chap-core for gzip, as the DHIS2 route asks for it. The route
unpacks the answer, so the browser gets plain JSON. But chap-core packs every
answer that goes to the route, and the times include that work. On a large
answer, the time to pack can be more than the time to make the answer.
SIZE(bytesin--json) is the unpacked answer, which the browser gets.GZIP(wire_bytes) is the size on the network between chap-core and the DHIS2 route. It is the same asSIZEif chap-core did not pack the answer. chap-core does not pack answers under 1,000 bytes.
All versions of bench ask for gzip. But to compare two chap-core versions,
run the same bench version on both.
What was measured
Two runs can only be compared if they ran on the same setup. bench, also
with --write, gives a measured on line and server in --json:
chap_core_versionandrevision, from chap-core’s/system/info.api_workers: the gunicorn worker processes of thechapservice.worker_processes: the celery worker processes of theworkerservice. Celery starts one for each CPU it sees (BUGS.md).docker_cpusanddocker_memory_bytes: what the docker engine has. On macOS and Windows this is the docker VM.machineandcpus: the machine that chap-devops runs on.gzip: whether chap-core packed an answer with gzip whenbenchasked for it, as the DHIS2 route asks.benchchecks this with one untimedGET /v1/crud/configured-models. It isnullwhen that answer was under 1,000 bytes, which chap-core never packs.transport: theServerandViaheaders of that answer. A proxy in front of chap-core, or the DHIS2 route, shows here. Such a proxy can pack or unpack the answers, or hold them back, so its time is in the numbers.
The measured on line says answers packed with gzip, or answers not packed, though gzip was asked for. The second is a sign that something
between bench and chap-core unpacks the answers, or that chap-core does
not pack them. It names the server when it is not chap-core’s own
uvicorn.
bench counts the processes that run, with docker compose top. It does
not read the settings, because chap-core can ignore a setting. It counts
them only for a chap-core that this deployment runs, so with --url the
counts and the docker values are not known.
One at a time and many at a time
A Modeling App page sends several requests at the same time. --parallel 8
does the same. If the server answers one request at a time, the fast requests
wait for the slow ones. Then their p50 goes up, but the total time does not go
down. For one evaluation (18 org units, 72 months), on chap-core master of
2026-11-05, with the requests of a bench from before the pages:
--parallel 1 | --parallel 8 | |
|---|---|---|
| total for 125 requests | 16.1 s | 14.9 s |
backtests p50 | 10 ms | 737 ms |
configured-models p50 | 12 ms | 748 ms |
predicted_vs_actual subplot p50 | 674 ms | 2.67 s |
CPU, memory and the database: –resources
--resources records what the services of this deployment used while
bench ran, phase by phase. With --write the phases are import,
backtests and prediction (with --predict). Without --write there is
one phase, requests.
chap-devops bench --write --model seasonal_median --backtests 1 --predict --resources
For each phase it gives:
- for
chap,worker,postgresandredis: the CPU seconds, the average and the peak cores, and the peak memory, fromdocker stats; - for Postgres: the counters of
pg_stat_databasethat grew in the phase, for examplexact_commit,tup_returned,tup_insertedandtemp_bytes; - for Valkey:
total_commands_processed,total_net_input_bytes,total_net_output_bytes, andused_memory_peakat the end.
import (8.58 s): chap 2.2 CPU s (peak 1.0 cores, 729.9 MB), worker 4.9 CPU s (peak 0.8 cores, 1.7 GB), postgres 1.4 CPU s (peak 0.3 cores, 198.3 MB), redis 0.0 CPU s (peak 0.0 cores, 303.6 MB); postgres read 187662 and wrote 187205 rows
backtests (33.74 s): chap 8.2 CPU s (peak 1.0 cores, 784.8 MB), worker 77.3 CPU s (peak 7.8 cores, 3.8 GB), postgres 3.6 CPU s (peak 0.3 cores, 220.0 MB), redis 0.2 CPU s (peak 0.0 cores, 305.6 MB); postgres read 1385708 and wrote 131990 rows
docker stats measures for about a second each time, so that is the
interval of the samples. A phase that is shorter than a second has few
samples or none. The CPU seconds are the average cores times the length of
the phase. The sampling itself uses some CPU, so compare runs that both
used --resources, or both did not. --json gives each phase under
resources.
--resources needs docker to see chap-core, so it works only for the
chap-core of this deployment, not with --url. chap-core’s Postgres does
not load pg_stat_statements, so there are no top statements. To get them,
add shared_preload_libraries=pg_stat_statements to the postgres
service, and run create extension pg_stat_statements in the database.
A second user: –second-user
--second-user shows what the run does to the other people on the server.
During --write, a thread asks for GET /v1/crud/configured-models every
half second, the list that most pages of the Modeling App ask for first.
Each phase of the run gets the p50, p95 and longest time of the requests
that started in it:
second user, import: 74 requests of configured-models, p50 13 ms, p95 115 ms, max 4.12 s
second user, backtests: 290 requests of configured-models, p50 18 ms, p95 164 ms, max 647 ms
That run imported 1,500 locations and 156 weeks on chap-core master with
one API worker. While chap-core read the import, the second user waited up
to 4.12 s for a list that otherwise takes 13 ms. Compare it with more API
workers (--workers 1,4). --json gives the phases under second_user.
API workers: –workers
--workers 1,4 runs the same bench once for each count of API worker
processes, and puts the numbers side by side. It works with and without
--write:
chap-devops bench --page prediction --parallel 8 --workers 1,4
chap-devops bench --write --model seasonal_median --backtests 2 --workers 1,2
For each count, it sets the count as chap-devops restart --api-workers N
does: it writes CHAP_API_WORKERS to .env, recreates only the chap
service, and waits until chap-core answers. Then it runs the same bench
command again with --json. At the end it sets the count back to what it
was. It checks every count before the first run: more than one worker needs
a chap-core newer than v2.7.0.
This run is from chap-core master (2.8.0.dev0) on an Apple Silicon Mac,
with 100 org units and 156 weeks:
MEASURE 1 WORKERS 2 WORKERS
import 2.80 s 2.79 s
import request 768 ms 720 ms
backtest request 982 ms 948 ms
slowest job 6.67 s 6.41 s
whole run 28.79 s 28.06 s
ran the same bench with 2 of 2 API worker counts (1, 2); set back to 1
The rows are the time per page and the p50 of each request, and with
--write the import, the requests and the slowest job. --json gives each
run whole under runs. --workers changes this deployment’s chap-core, so
it cannot be used with --url.
An empty database: –fresh-database
--fresh-database empties chap-core’s database before a --write run, so
the run does not find the rows of the runs before it. It stops chap and
worker, drops the database and makes it again, and starts them. chap-core
makes its tables and its configured models again when it starts.
- With
--writealone, it empties the database once, before the run. - With
--sizes, it empties the database before each size. - With
--workers, it empties the database before each count of API workers, so each count starts from the same state.
This deletes all the data in chap-core’s database: use it only on a
deployment of its own. It does not work with --url or with --dataset.
The closing lines say that the database was emptied, and --json gives
fresh_database.
chap-devops bench --write --model seasonal_median --backtests 1 --predict \
--locations 1500 --periods 365 --fresh-database
Dataset sizes: –sizes
--sizes runs --write once for each dataset size, and puts the numbers
side by side. A size is the locations, an x, and the periods with w for
weeks or m for months:
chap-devops bench --write --model seasonal_median --backtests 1 \
--sizes 500x520w,1500x520w,3000x520w,1500x120m --fresh-database --resources
Each size runs as bench --write with its --locations, --periods and
--period-type, and with the other options of the command. The other
dataset options, such as --covariates and --format, are the same for
every size.
--fresh-database empties chap-core’s database before each size, so a run
does not find the rows of the run before it. See
An empty database.
This run is from chap-core master (2.8.0.dev0) on an Apple Silicon Mac:
MEASURE 50x156w 100x60m
import 4.72 s 4.54 s
import request 562 ms 438 ms
backtest request 409 ms 361 ms
slowest job 4.81 s 3.59 s
whole run 17.84 s 14.87 s
ran 2 of 2 sizes (50x156w, 100x60m), each on an empty database
--json gives each run whole under runs, with its size.
Before and after a change
To see what a change in chap-core does, time the same page before and
after it, with the same data and the same bench version:
chap-devops --json bench --page prediction --repeat 10 > before.json
# change chap-core and start it again
chap-devops --json bench --page prediction --repeat 10 > after.json
Compare pages and the p50_ms of each row in the two files.
Another chap-core
--url times a chap-core that this deployment does not run. The token comes
from CHAP_API_TOKEN in the shell, as for chap-devops api:
CHAP_API_TOKEN=... chap-devops bench --url https://chap.example.org --repeat 5
Run it while a backtest runs on the worker to see how much a running job
slows the pages. Use chap-devops top --record
at the same time to see which container is busy.
Backtests under load: –write
--write adds rows to chap-core’s database. Do not use it on a deployment
that people use: run it on a deployment of its own.
chap-devops bench --write --model seasonal_median
chap-devops bench --write --model naive_model,seasonal_median --backtests 1
chap-devops bench --write --model seasonal_median --backtests 8 --locations 300 --periods 156
chap-devops bench --write --model 12 --dataset 5 # an existing dataset
The steps follow the Modeling App’s evaluation and benchmark pages:
- It makes a synthetic dataset from the dataset options
and imports it with
make-dataset. With--dataset ID, it uses that dataset instead. - It starts
--backtestsruns (default 4) at the same time. A run is onecreate-backtestsrequest with all the models of--model, under one backtest specification, as the benchmark page sends it. It starts all the runs before it polls any job. - Each backtest job queues the baselines that the specification has no
run of yet (
run_baselinejobs:persistence,seasonal_shift_persistence,global_median,seasonal_median). The benchmark page is complete only when they end, sobenchwaits for them too. - It polls the jobs until each one has ended, for at most
--waitseconds. - For each backtest that ended, the baselines included, it checks what chap-core stored.
- It deletes the dataset it made and all the backtests.
--keepkeeps them. It never deletes a dataset that--datasetnamed.
--model takes one or more names or ids, separated by commas, as
chap-devops api GET /v1/crud/configured-models shows them, for example
seasonal_median or chapkit_ewars_model:default. Without it, the error
lists the names.
A backtest passes when its job ended with SUCCESS, and chap-core stored one
forecast for each org unit, split and forecast period, and stored aggregate
metrics. The count of forecasts is the length of evaluation-entry at one
quantile. The expected count comes from the backtest row: orgUnits times
splitPeriods times nPeriods. Anything else is a failure:
- the job ended with
FAILURE, or it still ran after--waitseconds; - chap-core stored fewer or more forecasts than expected, or none;
- chap-core stored no aggregate metrics.
This run is from chap-core master (2.8.0.dev0) on an Apple Silicon Mac,
with 18 org units and 156 weeks:
# MODEL STATUS REQUEST QUEUED RUN BACKTEST FORECASTS METRICS RESULT
1 naive_model SUCCESS 103 ms 0 ms 2.46 s 1 1512/1512 13 ok
1 seasonal_median SUCCESS 103 ms 0 ms 2.98 s 2 1512/1512 13 ok
- seasonal_median (baseline) SUCCESS - - 2.88 s 4 1512/1512 13 ok
- global_median (baseline) SUCCESS - - 2.80 s 3 1512/1512 13 ok
- seasonal_shift_persistence (baseline) SUCCESS - - 3.19 s 5 1512/1512 13 ok
- persistence (baseline) SUCCESS - - 3.20 s 6 1512/1512 13 ok
2 of 2 backtests of naive_model, seasonal_median on dataset 2 passed, in 11.11 s
4 of 4 baseline backtests passed
The times of each job:
REQUEST: thecreate-backtestsrequest of its run. chap-core loads the dataset in this request, so it grows with the dataset.QUEUED: from the answer to the request to the start of the job.RUN: from the start to the end of the job, fromstart_timeandend_timein/v1/jobs. This is exact, also for a job that is shorter than a poll.
QUEUED compares the clock of this machine with the clock of chap-core, so
it is only exact when they are the same machine. If a backtest fails, the
command exits with status 1 and gives a warning with the reason from the job
log. --json gives each backtest with its model, baseline, status,
request_seconds, queued_seconds, seconds, backtest_id, forecasts,
expected_forecasts, metrics and problem. It also gives
import_request_seconds, import_seconds and backtest_request_seconds.
A prediction: –predict
--predict also makes one prediction, so the prediction page has
something to show. It uses the first backtest that passed, as the Modeling
App does:
- It makes a prediction setup on the backtest
(
POST /v1/crud/prediction-setups). - It runs the setup with the synthetic dataset, one object for each value
(
providedData), with DHIS2 period ids such as2024W7(POST /v1/crud/prediction-setups/{id}/run). chap-core stores these values as a new dataset. - It waits for the job, then checks that chap-core stored one forecast for each org unit and forecast period.
--predict cannot be used with --dataset, because the run needs the
values of the dataset. With weekly data, give at least 105 periods: chap-core
refuses a prediction on 104 weeks (see
troubleshooting). Without --keep, bench deletes the prediction, its
dataset and the setup before the backtests. With --keep, it gives the
command that times the prediction page:
chap-devops bench --write --model seasonal_median --backtests 1 --predict --keep \
--locations 1500 --periods 365
chap-devops bench --page prediction --prediction 7 --repeat 10
If the prediction fails, the command exits with status 1. --json gives it
as prediction, with setup_id, job, status, seconds,
prediction_id, dataset_id, entries, expected_entries and problem.
To compare settings, run the same command before and after the change, for
example with one worker process and then with four. Use the same seed (the
default) to get the same data. If jobs are still running after --wait,
chap-devops keeps the rows and says so: chap-devops jobs shows them.
Synthetic datasets: chap-devops bench dataset
chap-devops bench dataset makes a dataset with random values. The same
options always give the same dataset.
chap-devops bench dataset # 18 locations, 104 weeks, in .
chap-devops bench dataset --out data --locations 2000 --periods 520 --covariates 10
chap-devops bench dataset --import --period-type month --periods 72
| Option | Default | What it sets |
|---|---|---|
--locations N | 18 | the number of org units, loc01 to loc18 |
--periods N | 104 | the number of periods for each location |
--period-type | week | week (ISO weeks) or month |
--cases N | 100 | the typical number of cases in one location and period |
--covariates N | 3 | rainfall, mean_temperature, population, then covariate_4 and more |
--seed N | 1 | the start of the random values |
--format | rows | how --import and --write send the values: rows or columns |
The periods always end with the last week or month of 2024. Each location has its own size, season and population. The cases follow the season and the rainfall of the period before, with noise, so a model has something to learn. The values are not realistic.
Without --import, it writes <name>.csv and <name>.geojson to --out
(default .). The CSV has the columns that chap eval reads, and chap eval
finds the GeoJSON of the same name next to it:
chap-devops bench dataset --out data
chap-devops chap eval --model-name auto_arima_chapkit \
--dataset-csv data/synthetic-18x104w-c100-k3-s1.csv --output-file eval.nc
The GeoJSON has one square for each location, on a grid in the sea west of Africa, so no one mistakes it for a real place.
With --import, it sends the dataset to chap-core’s make-dataset, waits
until chap-core has stored it, and gives the dataset id. --format selects
how it sends the values:
rows(the default) sends one object for each value (providedData), with DHIS2 period ids such as2024W7and202407. The Modeling App sends a dataset this way, so the import takes the same path in chap-core.columnssends one list for each column (providedDataColumns). The body is about four times smaller. This needs chap-core v2.7.0 or later.
The closing line gives the time of the import, split in two: the request,
in which chap-core reads and checks the values, and the job, in which the
worker stores them. --json gives request_seconds and import_seconds.
--name sets the name. The default name says what the dataset is
made of: synthetic-18x104w-c100-k3-s1 is 18 locations, 104 weeks, 100
cases, 3 covariates and seed 1. bench --write adds -bench-<time> to it.