← Development workflow

7.8 Performance measurement and capacity

KMP has two complementary performance signals: privacy-safe per-tenant hourly aggregates in the platform database and detailed request logs for engineering diagnostics. A local Playwright benchmark supplies repeatable route/concurrency measurements, but its computed “tier” is an experiment result—not a production SKU recommendation, service-level objective, or proof of fleet capacity.

Tenant operational aggregates

After a tenant is resolved, TenantOperationalMetricsService upserts one row per tenant, hour, and bounded route name in tenant_request_metrics_hourly. It records request, error, server-error, slow-request, duration-total, and maximum-duration signals. Route names come from plugin/prefix/controller/action, not raw URLs, query strings, IDs, bodies, or member attributes.

These central aggregates support platform fleet health and tenant-level trend views without querying tenant business databases. Writes are best-effort and do not change an HTTP response when telemetry is unavailable.

Configuration:

Use these aggregates for per-tenant comparisons on a shared deployment. Keep the platform table bounded by retention and indexes; it is operational metadata, not an analytics warehouse.

Detailed request telemetry

When PERF_REQUEST_LOG_ENABLED=true, application middleware emits [request_timing] records for requests above PERF_SLOW_REQUEST_MS (or every request with PERF_LOG_ALL_REQUESTS=true). Fields include:

The detailed log includes raw host/path and can be high-cardinality. Restrict access and retention, and never add query strings, request bodies, identity fields, or secret-bearing headers. PERF_KINGDOM_TAG is a static deployment label retained for compatibility; it is not authoritative per-request tenant attribution on a shared multi-tenant web revision.

Query logging is separately controlled by PERF_DB_QUERY_LOG_ENABLED and the query log channel. Enable it briefly for an investigation, sample appropriately, and review SQL redaction before sending it to a remote sink.

Application Insights transport

ApplicationInsightsLog can send selected log channels through the configured direct or OTLP transport. Representative controls are:

Validate effective configuration without printing the connection string:

bin/cake telemetry_check

Use the smoke-trace option only in an environment where sending telemetry is approved. Dashboard/workbook queries must filter on the emitted channel and telemetry_schema_version; update the deployment workbook when that schema changes.

Local sizing benchmark

The repository wrapper is ./load_test.sh; the implementation is app/scripts/perf/sizing-benchmark.js. It logs in with Playwright, profiles a fixed set of member/grid routes, runs configured virtual-user concurrency levels, samples host/cgroup CPU and memory, and writes a timestamped JSON report under test-results/perf by default.

For the current PostgreSQL multi-tenant local stack:

KMP_BASE_URL=http://kmp.localhost:8080 \
KMP_ENABLE_DB_PROFILE=0 \
KMP_ROUTE_RUNS=5 \
KMP_CONCURRENCY_LEVELS=1,5,10,20 \
./load_test.sh

The benchmark’s optional DB profiler currently uses MySQL’s mysql.slow_log and client. It is not PostgreSQL query profiling; leave it disabled for the supported PostgreSQL stack and use request query counters, PostgreSQL statistics/query plans, or approved managed-database telemetry instead.

The script’s default 127.0.0.1 base URL predates required tenant hosts. Always set KMP_BASE_URL to a registered tenant host. Use only synthetic/local credentials and avoid placing a password in a shared command history.

Interpreting a report

A report contains sequential route timing, concurrency throughput/errors/p95, host/cgroup telemetry, detected risks, and a heuristic app/database split. Its classification is based on the highest tested concurrency with zero errors and p95 below the configured threshold.

Do not extrapolate it linearly to kingdoms or treat the heuristic CPU/memory split as an Azure sizing answer because:

Keep the JSON artifact with the commit/image, environment shape, seed revision, route set, and configuration used. Compare like-for-like reports rather than a single tier label.

Multi-tenant capacity test design

A meaningful pre-production measurement includes:

  1. the exact release image and migration-current platform/tenant schemas;
  2. production-like but synthetic tenant sizes and branch/permission shapes;
  3. concurrent traffic across several tenant hosts, not just many users on one;
  4. cold and warm host-map, ORM metadata, tenant cache, and connection behavior;
  5. active queue/schedule/workflow load plus a backup window;
  6. platform DB, per-tenant DB, web revision, and worker resource/connection metrics; and
  7. an agreed latency/error target and sustained duration with headroom.

Protect isolation while testing: tag synthetic traffic safely, keep output free of member data, confirm tenant A/B results remain separate, and never benchmark a customer production tenant without explicit operational approval.

Capacity signals by component

Signal Likely constraint to investigate
wall time high, CPU low, DB time high query plan/index/connection/database pressure
wall and CPU both high PHP/domain computation, serialization, image/PDF work
query count high but DB time low N+1/ORM round trips and future scale risk
platform DB latency/error across tenants host resolution, metrics/jobs/schedules, central connection pool
only one tenant slow tenant data shape/index drift, tenant DB/storage/mail configuration
worker backlog grows job cost, retry storm, per-cycle bounds, tenant fairness
memory grows across sequential tenants retained table/client/static/context state—a potential isolation defect
errors begin at modest concurrency connection/worker limit, lock contention, downstream throttling

Scale only after identifying the bottleneck. Adding web replicas will not fix a slow query or serialized platform lock; increasing a database tier will not fix an N+1 grid or tenant-context leak.

Investigation workflow

  1. Reproduce one slow route with a correlation ID and correct tenant host.
  2. Compare route aggregate history with detailed duration/query/CPU/memory data.
  3. Inspect the scoped tenant DB plan and indexes; do not query every tenant.
  4. Check worker/platform schedule events for coincident background pressure.
  5. Add a focused regression benchmark/test and make one measurable change.
  6. Repeat on the same environment and then validate in POC with the release image.

Use EXPLAIN (ANALYZE, BUFFERS) only on safe, representative queries in an approved environment. Avoid logging literal SQL parameters containing member information.

Operational claims

Resource settings in templates are starting configuration, not proven capacity. Single-region deployment, unmeasured restore duration, or an untested backup schedule must not be presented as an availability/RTO/RPO guarantee. Establish those claims from sustained telemetry, failover/restore drills, load evidence, and explicit service objectives.

Source map