Blog

Temporal Metrics, the Four-Number Triage, and Capacity Planning on PostgreSQL

When someone says 'Temporal is slow', four numbers asked in a fixed order tell you whether the time is on the queue, in your workers, in your code or inside the server. This post covers that triage, the five server checks behind it, the delta-counter trap that makes rate() lie, and how to size a cluster from writes per workflow, the ladder of ceilings and a step-ramp benchmark.
29 min readPart 7 of 10
On this page
  1. Two families of metrics
  2. The four-number triage for "Temporal is slow"
  3. Why this order
  4. The four numbers as PromQL
  5. When schedule-to-start is high: my workers or the server?
  6. Inside the server: five checks, in order
  7. 1. A rate limiter said no: service_errors_resource_exhausted by cause
  8. 2. Waiting for the shard's IO slot: semaphore_latency
  9. 3. The database: persistence_latency by operation
  10. 4. One hot workflow: lock_latency
  11. 5. GC thrash: cache_usage, memory_heap, CPU
  12. The traps that make metrics lie
  13. Delta vs cumulative
  14. The queue that had "0 pollers"
  15. More server-side traps
  16. What to watch, and the thresholds
  17. Capacity: the smallest ceiling wins
  18. The unit of capacity: writes per workflow
  19. The ladder of ceilings
  20. Never plan for 100 %
  21. Worked example: our real numbers
  22. Prove it with a step-ramp benchmark
  23. Build staging to match production
  24. Choose the workload and ramp it
  25. Record per step
  26. Find the knee
  27. Deriving the numbers from the benchmark
  28. History hosts
  29. shardIOConcurrency: raise until persistence p99 bends
  30. Partitions per hot task queue
  31. Connections: the budget that ties it together
  32. Is it Temporal or PostgreSQL?
  33. The capacity chain, and why more servers can mean less throughput
  34. What I'd tell myself before go-live

TL;DR: Temporal exposes hundreds of metrics, and most of them don't matter at 3 a.m. For "Temporal is slow" I ask four numbers in a fixed order: schedule-to-start (is the task waiting?), worker slots available (are the workers full?), activity execution latency (is it my code?), and server service_latency (is it really Temporal?). Only the last one opens the server, and there I check five things in order: RESOURCE_EXHAUSTED by cause, semaphore_latency, persistence_latency, lock_latency, then heap and GC. On the way, three traps make metrics lie: worker counters exported as delta (so rate() is nonsense; use sum_over_time()), le="1" matching nothing, and long polls showing up as a 60 s "p99". Capacity is the same idea turned around: convert business load into writes per workflow, compute each section's ceiling (pens, History CPU, disk commits, disk space, rate limits, workers), take the smallest, check it with one node dead, and plan peak at 60–70 % of it, because waiting grows as busy ÷ (1 − busy). On our cluster the narrowest section was the disk, at 88 % busy. Then prove it with a step-ramp benchmark and find the knee. And if PostgreSQL is the limit, adding Temporal hosts makes things worse.


The page usually reads the same way: "workflows are slow". You open Grafana. The server exposes hundreds of metrics, each with labels for service, operation, namespace and task queue; the SDK exposes dozens more. Something is always red.

The two easy approaches both waste the first hour. Scrolling every dashboard until something looks wrong finds a CPU spike that wasn't the cause. Looking at CPU and memory tells you a machine is busy, not where a request spent its time.

What I want is a decision tree. A few numbers, asked in a fixed order, where the first ones split the problem into big, cheap halves (is the time in my workers or in the server?) and only then the detailed server metrics. This post is that tree, the traps that make the metrics lie, and the other side of the same coin: working out how much load the cluster can take before it gets slow.

If you're just setting up worker metrics, I wrote separately about getting SDK metrics out of many workers with Prometheus or OTLP. This post assumes the metrics are already flowing.

Two families of metrics

FamilyComes fromExamplesTells you about
SDK metrics (temporal_*)Your workerstemporal_activity_schedule_to_start_latency, temporal_worker_task_slots_available, temporal_activity_execution_latencyThe task queue wait, worker slots, your activity code
Server metricsEach Temporal roleservice_latency, persistence_latency, semaphore_latency, lock_latency, cache_usage, service_errors_resource_exhausted, task_latency_*Shards, the per-shard IO semaphore, caches, rate limiters, the database

Server metrics. Every Temporal process (frontend, history, matching, worker) serves Prometheus metrics on its own port, set by global.metrics.prometheus.listenAddress in the static config, commonly :9090. Every role uses the same names; the service_name label tells them apart. Server counters and histograms are cumulative.

SDK metrics. The TypeScript workers emit metrics from the SDK core, configured once per process with Runtime.install({ telemetryOptions: { metrics: ... } }): either a Prometheus endpoint the SDK serves itself (prometheus: { bindAddress: '0.0.0.0:9464' }) or an OpenTelemetry exporter (otel: { url, ... }). Names start with temporal_. Whether they arrive as delta or cumulative depends on the exporter and its temporality setting, and in our setup they're delta. That one detail breaks half the PromQL people copy from the internet. More on it below.

Metric names and labels move between releases on both the server and the SDK, and SDK histograms may carry a unit suffix (..._seconds_bucket). Before building a dashboard or alert, I grep the real thing on each role and on a worker:

bash
curl -s http://<history-host>:9090/metrics | grep -v '^#' | cut -d'{' -f1 | sort -u | grep -iE 'latency|exhausted|shard'
curl -s http://<worker-host>:9464/metrics | grep -E '^temporal_(activity|worker|workflow)' | cut -d'{' -f1 | sort -u

The four-number triage for "Temporal is slow"

text
1. SCHEDULE-TO-START (SDK)       high? ──▶ the wait is on the task queue
       │                                      │
       │ low                                  ▼
       │                              2. SLOTS AVAILABLE (SDK)
       │                                 0 on all workers → workers full: add slots/workers,
       │                                                       or the activity is slow (go to 3)
       │                                 free slots          → check pollers: 0? wrong queue name?
       │                                                       same delay on EVERY queue? → server dispatch (4)
       ▼
3. ACTIVITY EXECUTION LATENCY (SDK)
       high? ──▶ YOUR code or what it calls (API, DB). Not Temporal.
       │ low
       ▼
4. SERVER LATENCY (service_latency by operation, e.g. StartWorkflowExecution)
       high? ──▶ look inside the server, in this order:
                 • service_errors_resource_exhausted by cause  → a rate limiter
                 • semaphore_latency p95                      → the per-shard IO semaphore
                 • persistence_latency p99 by operation       → the database/disk
                 • lock_latency p99                           → one hot workflow
                 • cache_usage + memory_heap + CPU            → GC thrash

The four-number triage: schedule-to-start, slots available, activity execution latency and server latency, each answer pointing to the next branch, with the five in-server checks at the bottom
The four-number triage: schedule-to-start, slots available, activity execution latency and server latency, each answer pointing to the next branch, with the five in-server checks at the bottom

The sentence I keep in my head: first separate queue time from worker time from server time, then go deeper only where the time actually is.

Why this order

  • Schedule-to-start first, because it's the one number that says whether a task is waiting or being worked on. Your worker measures it, so it covers every hop between "History scheduled the task" and "my worker picked it up".
  • Slots second, because the most common cause of a long queue wait is that every execution slot is taken. It's also the cheapest fix, and it lives in your app, not the server.
  • Activity execution latency third, because if the work itself is slow, the problem is your code or what it calls. Temporal can't make a third-party API faster.
  • Server latency last, because only now do you know the time is really inside Temporal.

The same ordering logic applies to scaling decisions generally: check SDK workers first (most common, cheapest to fix), and check PostgreSQL before scaling any Temporal role, because scaling Temporal in front of a saturated database makes things worse (more on that in the capacity half).

The four numbers as PromQL

Our worker counters and histograms are delta, so these use sum_over_time. If yours are cumulative, swap in rate.

promql
# 1. Schedule-to-start p95, per task queue (activities; the workflow-task version is the same shape)
histogram_quantile(0.95,
  sum by (le, task_queue) (sum_over_time(temporal_activity_schedule_to_start_latency_bucket[5m])))
histogram_quantile(0.95,
  sum by (le, task_queue) (sum_over_time(temporal_workflow_task_schedule_to_start_latency_bucket[5m])))
 
# 2. Free slots per worker type and queue (a gauge: no rate, no sum_over_time). 0 = saturated
min by (task_queue, worker_type) (temporal_worker_task_slots_available)
 
# 3. How long the work itself takes, per activity type
histogram_quantile(0.95,
  sum by (le, activity_type) (sum_over_time(temporal_activity_execution_latency_bucket[5m])))
 
# 4. Server latency per operation (cumulative server histogram: rate is fine). Exclude polls
histogram_quantile(0.99,
  sum by (le, service_name, operation) (rate(service_latency_bucket{operation!~".*Poll.*"}[5m])))

When schedule-to-start is high: my workers or the server?

This is the branch I go down most often, and it splits cleanly:

What you seeMeaningFix
Slots exhausted, worker CPU/memory near the limitWorkers are genuinely fullScale workers
Slots exhausted, worker CPU lowSlots are held by I/O-bound activities waiting on something downstreamRaise slots if the downstream can take it, or fix what the activities wait on
Slots free, pollers starved (few pollers vs partitions, temporal_long_request_failure, Frontend ConcurrentLimit)Workers aren't asking for work fast enoughMore pollers or poller autoscaling; check frontend.namespaceCount and the load balancer idle timeout
Slots free, pollers healthyThe delay is inside the serverHistory transfer-queue lag (task_latency_queue), Matching backlog and sync-match ratio, or PostgreSQL under Matching (persistence_latency for CreateTasks / GetTasks / UpdateTaskQueue)

The trap here is adding Matching hosts because schedule-to-start is high while slots_available is 0. The queue is waiting for your workers; more Matching capacity changes nothing. The pairing worth memorising: high schedule-to-start plus high worker CPU/memory means scale workers; high schedule-to-start plus low worker CPU/memory means more pollers or slots. (How Matching, partitions and pollers actually work is in the Matching and workers post.)

A concrete version of the "slots exhausted, CPU low" row: schedule-to-start is 3 s on an email queue only, every email worker shows 0 free slots, and worker CPU is 4 %. Email activities spend their time waiting on the provider, so every slot stays busy, pollers stop asking for work (a poller only reaches out when a slot is free), and new tasks sit on the queue. temporal_activity_execution_latency for those activities confirms it. If the provider can take more parallel calls, raise maxConcurrentActivityTaskExecutions or add workers; CPU has plenty of room. If the provider itself got slow or is rate-limiting you, more slots just pile more load on it. Fix the HTTP timeout and the provider side instead.

Inside the server: five checks, in order

You only get here when number four says the server's own latency is high. The order goes from "the server is refusing work on purpose" to "the server is struggling".

1. A rate limiter said no: service_errors_resource_exhausted by cause

Frontend and History count every rejection, labelled with the cause: RpsLimit, ConcurrentLimit, PersistenceLimit, BusyWorkflow, and others. The SDK quietly retries these with backoff, so users see slow, not errors. That makes this the single most useful "why is it slow but nothing's failing" metric.

promql
# Label values are CamelCase in current versions; verify with
#   count by (resource_exhausted_cause) (service_errors_resource_exhausted)
sum by (service_name, resource_exhausted_cause) (rate(service_errors_resource_exhausted[5m]))
  • RpsLimit / ConcurrentLimit on Frontend → a namespace or host rate limit. The per-host limit is divided by the number of frontends (the Frontend post covers that division).
  • PersistenceLimit on History → history.persistenceMaxQPS (9,000 per History host by default on current versions) is lower than the database calls your traffic needs. History is throttling its own DB calls.
  • BusyWorkflow → one workflow is getting more updates than it can take (check 4).

Worked example: schedule-to-start is fine, slots are free, but StartWorkflowExecution p99 is 2 s and PersistenceLimit is climbing. Compare history.persistenceMaxQPS × History hosts against sum(rate(persistence_requests{service_name="history"}[5m])). If PostgreSQL has headroom (check persistence_latency and DB CPU/IOPS first), raise it, or use history.persistenceGlobalMaxQPS so the total doesn't change when the host count does. If PostgreSQL is already the wall, raising the limit only moves the queue into the database.

2. Waiting for the shard's IO slot: semaphore_latency

Each History shard has a small number of concurrent persistence slots, history.shardIOConcurrency, default 1. Every write to any workflow on that shard needs one. When writes queue for it, the wait shows up here.

promql
# Waiting for the per-shard IO semaphore; per host, there is NO shard_id label
histogram_quantile(0.95, sum by (le, instance) (rate(semaphore_latency_bucket{operation="ShardInfo"}[5m])))

High here while persistence_latency is fast and stable means the cluster is shard-IO bound: writes are serialising on the semaphore, not waiting on the database. The mechanism and the fix are in the shards and shard IO concurrency post.

3. The database: persistence_latency by operation

promql
# DB round-trip as seen by Temporal (includes connection-pool wait), by operation
histogram_quantile(0.99, sum by (le, operation) (rate(persistence_latency_bucket{service_name="history"}[5m])))

Rising on all operations at once means the database (or its disk, or the network to it) is the wall. Rising on one operation means a table problem, typically bloat on timer_tasks, transfer_tasks or tasks. Note that this number includes waiting for a connection from Temporal's own pool, so a too-small maxConns shows up here too. The tables themselves are covered in the PostgreSQL persistence post.

4. One hot workflow: lock_latency

Only one update to a given workflow runs at a time, behind a per-workflow lock. If thousands of signals hit one workflow ID, they queue for that lock.

promql
# Name varies by version (lock_latency / cache_latency); grep for it
histogram_quantile(0.99, sum by (le) (rate(lock_latency_bucket{service_name="history"}[5m])))
sum(rate(service_errors_resource_exhausted{resource_exhausted_cause="BusyWorkflow"}[5m]))

More hosts can't fix this. One workflow lives on one shard behind one lock. The fix is workflow design: batch signals, or split the entity across several workflow IDs.

5. GC thrash: cache_usage, memory_heap, CPU

History keeps a mutable-state cache in RAM. If it's too big for the memory available, Go's garbage collector runs constantly and eats CPU, and it looks like a capacity problem when it isn't.

promql
max by (instance) (memory_heap{service_name="history"})                              # verify the name on /metrics
histogram_quantile(0.99, sum by (instance, le) (rate(memory_gc_pause_ms_bucket[5m]))) # histogram or summary, by config
sum by (instance) (rate(process_cpu_seconds_total{job="temporal-history"}[5m]))      # the job label is yours

The three memory incidents that look alike (and aren't) are in the persistence and memory post.

The traps that make metrics lie

TrapWhat goes wrongDo this instead
Worker metrics are deltarate() and increase() return made-up numberssum_over_time() on worker counters; rate() is fine on server metrics
le labels are floatsle="1" matches nothing: empty graph, no errorWrite le="1.0"
Task queue labels are sanitised- becomes _ in labels, so describe on the label spelling shows a false "0 pollers"Take queue names from worker config or the CLI, never from labels
AveragesAn average hides the slow tail (the fsync stalls)Always p95/p99 from histograms
Long pollsA poll waits up to ~60 s by design, so "Frontend p99" is just the poll timeoutExclude .*Poll.* from every latency panel and alert

Delta vs cumulative

A cumulative server counter rises steadily so rate() reads its slope correctly, while a delta worker counter goes up and down and every drop looks like a counter reset to rate()
A cumulative server counter rises steadily so rate() reads its slope correctly, while a delta worker counter goes up and down and every drop looks like a counter reset to rate()

rate() and increase() assume a counter only ever grows. On a delta series each sample is just that interval's count, so the line goes up and down. Every drop is read as a counter reset, and the result is nonsense. On delta series, add the samples up instead:

promql
# delta counter: add the samples up
sum(sum_over_time(temporal_activity_execution_failed[5m]))
# delta histogram: percentiles from summed buckets
histogram_quantile(0.95, sum by (le) (sum_over_time(temporal_activity_execution_latency_bucket[5m])))

Or fix it at the source: export cumulative (the SDK's Prometheus endpoint, or OTel with cumulative temporality) and keep using rate() everywhere. Pick one and write it on the dashboard so the next person doesn't have to rediscover it.

The histogram_quantile pattern I reuse for everything has three rules: keep le in the by (...), add any label you want one line per (operation, instance), and put either rate (cumulative) or sum_over_time (delta) inside, never neither.

A panel that's empty with no error is almost always one of three things: le="1" instead of le="1.0" (and you normally shouldn't filter on le inside histogram_quantile at all), rate() on a delta histogram, or a unit suffix you didn't know about (..._seconds_bucket).

The queue that had "0 pollers"

This one nearly cost me a false outage during a deploy. I copied a task queue name from a Grafana label, something like orders_sync, and ran temporal task-queue describe --task-queue orders_sync. Zero pollers. Workflows were progressing fine. The real queue was orders-sync; the metrics pipeline had cleaned - to _, and orders_sync was a different, empty queue that nobody polled.

bash
# Describe it by its REAL name, not the label spelling
temporal task-queue describe --task-queue orders-sync

More server-side traps

  • Shard metrics have no shard_id label. semaphore_latency{operation="ShardInfo"}, numshards_gauge and friends are per host. You find the hot host, not the hot shard.
  • Empty polls are fine when idle. A high poll_timeouts rate means workers are waiting for work.
  • state_transition_count is a histogram. _count counts updates (one per History transaction), while _sum adds up the transitions in each update. Grep what your version records and use one series consistently.
  • Rates need two samples in the window. With a 1 m scrape interval, rate(x[1m]) is often empty. Use at least 4× the scrape interval.

What to watch, and the thresholds

Temporal publishes no alert thresholds. Everything below is either a starting point (a generic operational guess so you have an alert on day one) or benchmark-derived (from your own load test: your normal plus a margin, or a fraction of your measured knee). The plan is to replace the first kind with the second.

MetricRoleStarting thresholdAction
service_latency p99 by operationAllOver 2× benchmark baseline for 10 m (benchmark-derived)Compare roles: the first one to slow down is upstream of the problem
service_errors_resource_exhausted by causeFrontend / HistorySustained over 1/s for 10 mThe cause picks the branch
persistence_latency p99 by operationAllOver 100 ms on write ops for 10 mPG commit latency/CPU/IO; if PG is fine, network and pool waits
persistence_errorsAllAny sustainedPage. PG health, failover state, max_connections
task_latency_queue p95 by task typeHistoryOver 5 s for 15 mTimers and transfers are late: History CPU vs shard IO vs PG
sharditem_created_count, shard_closed_countHistoryAbove 0 outside deploysMembership churn: ports, clock, GC pauses, a flapping host
poll_success_sync ÷ poll_successMatchingUnder 80 % on a normally-synced queueWorkers short, or partition starvation
temporal_*_schedule_to_start_latencySDKp99 over 1 s on latency-sensitive queuesThe triage tree, step 2
temporal_worker_task_slots_availableSDK0 for 10 mRaise slots if CPU allows, else add workers
PG connections, replication lag, dead tuples on queue tablesPostgreSQLConnections over 80 % of max; lag over 10 s; dead tuples growing for hoursAutovacuum tuning on queue tables

The smallest dashboard that's actually useful is two rows: the four triage numbers, then the five server checks. Everything else is drill-down.


The second half of this post is the same machinery pointed the other way. Instead of "where is the time going now?", it's "how much can this cluster take before the time starts going somewhere bad?"

Capacity: the smallest ceiling wins

The question usually arrives as "can the cluster handle Black Friday?" The honest first answer is that it depends on what your workflows do: steps per workflow (each step is about 2 commits), payload size (WAL volume), backlog vs sync match (every backlogged task adds a transaction), timers, signals and visibility usage, the disk's fsync latency, PostgreSQL config, and the Temporal version. Together those swing the cost per workflow by 10× or more. That's also why borrowed figures ("~4,500 state transitions/s on Postgres after tuning") are at best an order-of-magnitude sanity check, never a sizing input.

The principle underneath is simple. A cluster is a pipe made of sections: Frontend, History, Matching, the database disk, your workers. The narrowest section sets the flow. Widening any other section only moves where the water waits. So capacity planning is three steps:

  1. Turn business load into resource load: workflows/s → database writes/s → CPU, disk.
  2. Work out each section's ceiling.
  3. The smallest ceiling is your capacity. Then check it still holds with one node down (N−1).

The unit of capacity: writes per workflow

You can't plan in "requests per second". A start request for a 3-step workflow and one for a 300-step workflow are the same request and a 100× different load. Plan in state transitions (writes) per second at a latency target.

Measure writes per workflow on your own cluster, per workflow type:

promql
# Shard-IO writes per started workflow
  sum(rate(persistence_requests{service_name="history", operation=~"CreateWorkflowExecution|UpdateWorkflowExecution"}[1h]))
/ sum(rate(service_requests{service_name="frontend", operation="StartWorkflowExecution"}[1h]))

One trap here: Prometheus regex matchers are anchored. My own notes first had operation=~"Create|Update.*Execution", and Create alone only matches an operation literally named Create. Spell the operation names out, and add any other write operations your version shows (the task-adding ones, for example) if you want them counted. Also, long-running workflows keep writing long after they start, so measure over a window that covers whole workflows, or count state transitions directly.

From reading the persistence path, one workflow with one activity is about 7 state transitions and about 13 main-DB commits. So 1,000 of those per second is about 13,000 commits/s. That number, not "1,000", is what the disk sees.

The ladder of ceilings

#CeilingHow to compute itNotes
1Shard IO slotsshards × shardIOConcurrency ÷ time each write holds a slotLittle's law
2History CPUops/s per core × cores on the surviving nodesWe measured ~185 ops/s per core
3Disk commitsWhat the disk can fsync per secondFlush latency matters more than headline IOPS
4Disk spacedaily data × retention days, plus room for pg_repack (≈ table size)
5Rate limitsper-host limits × surviving hostsPer-host limits shrink when a host dies
6Workersslots × workers ÷ activity duration

Two things to notice. "Surviving nodes" appears in rows 2 and 5: a ceiling that only holds when everything is up isn't one you can plan on. And row 3 belongs to the database, not History. Losing a History node doesn't change the disk's load at all. Losing the database is a different drill.

Never plan for 100 %

When a resource gets busy, waiting time doesn't grow evenly; it explodes near 100 %. The simplest single-queue model (a rule of thumb, not a law of Temporal):

text
wait ≈ service time × busy ÷ (1 − busy)
 
busy 50 % → wait ≈ 1× the work time
busy 80 % → wait ≈ 4×
busy 90 % → wait ≈ 9×
busy 97 % → wait ≈ 32×

Queueing wait curve: waiting time stays flat until about 70 % busy and then climbs steeply, with our disk at 88 % marked at roughly 7 times the commit time
Queueing wait curve: waiting time stays flat until about 70 % busy and then climbs steeply, with our disk at 88 % marked at roughly 7 times the commit time

So my rule is: plan peak load at 60–70 % of the smallest ceiling, after removing one node. You'll also see "operate at 50–60 % of the knee" later. They don't conflict. The first is measured against a raw ceiling computed on paper (IOPS, cores); the second against a benchmark knee, which is already below the raw ceiling because queueing starts well before 100 %. Both are common operational starting points, not Temporal-published numbers.

The headroom isn't waste. It absorbs N−1 (one of 4 History hosts dies and the other 3 take +33 % each, plus a cache reload storm), shard movement during deploys, retries and timer catch-up after an incident, PostgreSQL vacuum and retention-delete waves, backups, and the fact that your busiest minute isn't your average.

Worked example: our real numbers

Business target: 3,000 workflows per minute, so 50/s. Measured cost: about 12 shard-IO writes per workflow, so 600 writes/s.

CeilingCapacityUsage at 600/sVerdict
Shard IO: 512 shards × 4 slots ÷ 13 ms≈ 157,000 writes/s0.4 %Not a concern
History CPU at ~185 ops/s per core600 ÷ 185 ≈ 3.2 cores needed on survivorsdepends on host sizeFine if N−1 has spare cores
Disk (a P30-class managed disk)≈ 685 commits/s600 ÷ 685 ≈ 88 %Too hot: queueing explodes here
Disk spacedaily GB × 7 daysn/aCheck it
Rate limitsper-host × surviving hostsn/aCheck it
Workersslots × workers ÷ activity durationn/aCheck it

The disk is the narrowest section. At 88 % busy, the curve says waits are about 7× the commit time, and small bursts become big stalls. Checking it the other way: 685 commits/s ÷ 12 writes ≈ 57 workflows/s ≈ 3,400/min at 100 % busy, and ≈ 2,400/min at the 70 % plan line. That matches our real capacity model, which put the disk ceiling around 3,300–4,500 starts/min: the 3,000/min peak fits, but tight. Before taking on more, we'd want a faster disk tier (or WAL on its own disk), or fewer writes per workflow.

Notice what wasn't the problem. On an older cluster with 4 shards and 1 IO slot each at ~10 ms per write, the shard-IO ceiling was ~400 writes/s against ~350/s of demand: 97 % busy, and workflow starts went from 46 ms to 1.4 s. At 512 × 4 the slots moved out of the way and the disk became the narrowest section. The bottleneck moves when you fix one. That's normal, and it's why "add more shards" would not have helped here.

The fixes map one-to-one onto the ladder: shardIOConcurrency (then more shards, which means a new cluster) for the IO slots; a faster disk tier, a separate WAL disk or PostgreSQL tuning for the disk; more or bigger History hosts for CPU; global rate limits instead of per-host ones; more slots or worker processes for workers. And one lever widens every section at once: fewer writes per workflow, via local activities for short steps, fewer and larger activities, batched signals and smaller payloads.

Prove it with a step-ramp benchmark

Numbers on paper are a hypothesis. The benchmark turns them into facts.

Build staging to match production

The benchmark is only as good as its likeness to production. Match the Temporal version and dynamic config; numHistoryShards (immutable, and a 512-shard benchmark says nothing about 4096); shardIOConcurrency, persistence QPS limits and maxConns; the PostgreSQL version, instance class, storage type and IOPS tier and postgresql.conf; whether there's a synchronous standby (a network round trip on every commit); the network path to PostgreSQL (PgBouncer? TLS? same zone?); where visibility lives; and History host size and count.

The disk tier matters as much as the shard count: it sets fsync latency, which sets commit time, which sets how long each shard IO slot is held, which sets the knee. A benchmark on a faster disk shows a knee production can never reach.

The load generator and workers must not be the bottleneck. Give them their own machines and check they're relaxed: worker CPU under ~60 %, free slots, low schedule-to-start at low load.

Choose the workload and ramp it

I use omes, Temporal's load generator. Pick the scenario closest to production (go run ./cmd/omes list-scenarios), or write one with your real payload sizes. Run two profiles: your typical mix and your worst realistic one.

bash
# Workers on dedicated machines (same run-id = same task queue)
go run ./cmd/omes run-worker   --run-id cap-step-200 --language go --server-address temporal-staging:7233 --namespace bench
# Offered load: 200 new workflows/s for 15 minutes
go run ./cmd/omes run-scenario --scenario workflow_with_single_noop_activity --run-id cap-step-200 \
  --server-address temporal-staging:7233 --namespace bench \
  --duration 15m --max-iterations-per-second 200 --max-concurrent 5000

The flags move between commits, so check run-scenario --help on the one you're using.

  • Ramp in doubling steps, then refine: 25 → 50 → 100 → 200 → 400 → 800 workflows/s, then fill in between the last good step and the first bad one.
  • Hold each step 10–15 minutes. That covers 2–3 checkpoints (checkpoint_timeout defaults to 5 min), several autovacuum cycles, and History's shard-info save interval. Annotate step boundaries in Grafana.
  • Soak at the operating point for hours afterwards, ideally across a scaled-down retention boundary. Bloat, vacuum debt and WAL archive backlog only appear over time.

Record per step

From Temporal: achieved state_transition_count rate, service_latency and persistence_latency p99 per operation, RESOURCE_EXHAUSTED by cause, semaphore_latency, SDK schedule-to-start, the sync-match ratio, and CPU/heap/GC per host. From PostgreSQL, a snapshot at each step boundary that I diff:

sql
SELECT now() AS ts,
       (SELECT xact_commit FROM pg_stat_database WHERE datname='temporal')          AS commits,
       (SELECT blks_hit::float/nullif(blks_hit+blks_read,0) FROM pg_stat_database WHERE datname='temporal') AS hit_ratio,
       (SELECT wal_bytes FROM pg_stat_wal)                                           AS wal_bytes,
       (SELECT wal_fpi   FROM pg_stat_wal)                                           AS wal_fpi,
       (SELECT count(*) FROM pg_stat_activity WHERE datname='temporal' AND state='active') AS active,
       (SELECT count(*) FROM pg_stat_activity WHERE datname='temporal' AND wait_event_type IN ('Lock','LWLock','IO')) AS waiting,
       (SELECT count(*) FROM pg_stat_activity WHERE datname IN ('temporal','temporal_visibility')) AS conns;
 

The commit rate should come out at roughly 2 × state transitions/s; it's the unit PostgreSQL actually pays in. For commit and WAL flush latency, use pg_stat_statements on COMMIT (needs track_utility=on) and pg_stat_wal timing (needs track_wal_io_timing).

Find the knee

The knee is the first step where any of these happens:

  • achieved state transitions/s stops following offered load (under ~90 % of expected);
  • p99 service_latency or persistence_latency more than doubles for a less-than-double load increase;
  • service_errors_resource_exhausted appears;
  • schedule-to-start or backlog grows without bound during the hold (still growing, not just higher);
  • PostgreSQL shows saturation: CPU near 100 %, WALSync waits dominating, requested checkpoints, or n_dead_tup climbing step after step.

The resource at its limit at the knee is your weakest link, and it decides what you buy. Then operate at roughly 50–60 % of the knee: tighten toward 70 % only with fast autoscaling and well-understood traffic, loosen toward 40 % if a PostgreSQL failover must be absorbed without a backlog. After that, change one variable per run and re-ramp, keeping the results table in the repo next to the Helm or Terraform that produced it.

Sanity check: if the knee is 5,000 ST/s and production peak is 3,800, you're at 76 % of the knee. Losing a host, a vacuum wave or a retry burst puts you past it.

Deriving the numbers from the benchmark

History hosts

If History CPU or memory saturated first:

text
per_host_capacity = ST/s at the knee ÷ History hosts in the test
hosts_needed      = ceil( target_peak_ST/s ÷ (per_host_capacity × operating_fraction) ) + 1

Illustrative numbers: staging had 3 History hosts and the knee was 6,000 ST/s at 90 % CPU, so 2,000 ST/s per host. Production peak 9,000 ST/s at a 55 % operating fraction: 9,000 ÷ 1,100 = 8.2, so 9, plus one for N+1 = 10 hosts. Then re-check the database before buying anything: 9,000 ST/s is ~18,000 commits/s. If PostgreSQL was at 70 % at 6,000 ST/s, it won't do that.

The N−1 check in plain terms: every node must carry its share plus the dead node's share and still sit at or below ~70 %. Two 4-core History nodes needing 5 cores at peak is fine on a normal day and impossible when one dies (5 ÷ 4 = 125 %). The survivor saturates, its caches grow because it now holds every shard, SDK retries pile on, and latency explodes. Either each node carries all 5 cores alone at 70 % (≈ 8 cores each), or run three 4-core nodes (8 cores survive, 5 ÷ 8 ≈ 62 %).

Memory per host depends on the cache mode, which is version-dependent: from v1.25 one host-level cache (history.hostLevelCacheMaxSize, 128,000 entries) is shared by all shards on a host; up to v1.24 each shard has its own (history.cacheMaxSize, 512 entries), so memory grows with shards per host and losing a host can double a survivor's cache.

The Temporal docs give a starting ratio of 1 History process per 500 shards (4096 shards → about 8 processes). It's a first-deployment starting point, not a per-host limit; the knee decides. And before adding hosts, check shard balance (numshards_gauge), hot shards, cache hit rate and GC pauses, because adding hosts fixes none of those. One History host at 90 % CPU and three at 40 % is imbalance, not capacity. The cluster membership and scaling post goes into that.

shardIOConcurrency: raise until persistence p99 bends

Fix offered load just past the current knee, then sweep 1 → 2 → 4 → 8 (restarting each time, since it's read when a host acquires a shard):

As you raise itMeaningAction
semaphore_latency falls, ST/s rises, persistence_latency p99 flatShards were serialising; the DB had spare capacityKeep it, try the next value
ST/s flat, persistence_latency p99 rises, PG active backends and WALSync waits riseThe queue moved from Temporal's semaphore into PostgreSQLGo back one step. This is the bend
persistence_errors, timeouts or ShardOwnershipLostPool exhaustion or DB overloadBack off; check maxConns

If going from 2 to 4 drops semaphore_latency but ST/s doesn't move and PG active backends double, go back to 2: you added concurrency without throughput. And if semaphore_latency stays high with PostgreSQL and History CPU both unsaturated even after raising it, the cluster needed more shards from the start. Better to learn that in staging, since the shard count can't change later.

Partitions per hot task queue

Benchmark one hot queue at the default 4 partitions and note where schedule-to-start rises or sync-match drops while workers still have free slots. That gives per-partition capacity, which replaces the forum's "~100 tasks/s per partition". Keep numTaskqueueReadPartitions and numTaskqueueWritePartitions equal in steady state (grow read first, shrink write first), and don't raise partitions on cold queues: pollers spread thinner and sync-match drops.

Connections: the budget that ties it together

Each server process opens a pool per datastore:

text
total_conns = Σ over processes ( default.maxConns + visibility.maxConns )
            ≤ max_connections − superuser_reserved − admin/monitoring

If maxConns is unset, Go's database/sql default applies: unlimited. A spike opens connections until PostgreSQL refuses them. Always set maxConns and maxIdleConns on every datastore.

Illustrative: 10 History × (50 + 10) + 3 Matching × (30 + 5) + 3 Frontend × (20 + 20) + 2 Worker × (10 + 10) = 600 + 105 + 120 + 40 = 865 connections. That's far more active backends than a single primary handles well. PostgreSQL throughput peaks at a modest multiple of its core count in active backends, then falls from lock and LWLock contention and context switching. The fix isn't raising max_connections; it's smaller pools, fewer hosts, or a pooler (I wrote up PgBouncer from first principles). Whether Temporal's postgres plugin works behind transaction-mode pooling depends on the server version, so verify yours before relying on it.

Is it Temporal or PostgreSQL?

Three signatures cover most of what I've seen.

History CPU 90 %, PostgreSQL CPU 30 %: History-bound. But confirm first: CPU uneven across hosts is imbalance; high memory_gc_pause_ms with a heap near its limit is GC (more memory or a bigger cache, not more hosts). If it checks out, add History hosts off-peak, one at a time, because ~1/N of shards move and PostgreSQL takes a cold-cache reload burst. Then verify PostgreSQL didn't tip into the next case.

History 30 %, PostgreSQL CPU 95 %, commit latency high: PostgreSQL-bound. Do not add Temporal hosts. In rough order of cost and benefit:

  1. Reduce writes per outcome: payload size, steps per workflow, chatty signals, backlog, visibility usage.
  2. Tune PostgreSQL for the workload: max_wal_size and checkpoint_timeout, wal_compression, per-table autovacuum on queue tables, shared_buffers, commit_delay for group commit, fillfactor on executions.
  3. Move visibility to its own instance.
  4. Scale PostgreSQL vertically, including a faster WAL device if WALSync waits dominate.
  5. Lower Temporal's pressure on purpose (shardIOConcurrency, persistenceMaxQPS, per-namespace rate limits) so overload becomes controlled RESOURCE_EXHAUSTED instead of everything timing out.

Distinguish CPU-bound from flush-bound: 95 % CPU with mostly active backends and no wait event is CPU (queries, index maintenance, vacuum). Low CPU with high commit latency and WALSync / WALWrite waits is the disk. If a ramp shows ST/s going flat between two steps while persistence_latency p99 doubles and PostgreSQL sits at 45 % CPU with mostly WALSync waits, that's the WAL flush path, and more Temporal hosts or more DB CPU do nothing for it.

sql
-- CPU-bound shows mostly no-wait 'active'; flush-bound shows WALSync/WALWrite
SELECT coalesce(wait_event_type,'CPU') AS wtype, coalesce(wait_event,'running') AS wevent, count(*)
FROM pg_stat_activity WHERE datname='temporal' AND state='active' GROUP BY 1,2 ORDER BY 3 DESC;

Matching 90 %, History 30 %, PostgreSQL 40 %: Matching-bound. Check pollers first. Huge worker fleets with many pollers each burn Matching CPU on poll churn; too few pollers push tasks into backlog (low sync-match, extra DB writes). Find whether one queue is hot (loaded_task_queue_partition_count per host, name version-dependent). One hot queue limited by its partitions → raise partitions on that queue. Many queues on too few hosts → add Matching hosts, which only help if there are enough partitions to spread.

The capacity chain, and why more servers can mean less throughput

A request passes Client → Frontend → History (and Matching) → PostgreSQL, and the work is done by your SDK workers polling through Frontend → Matching. The system goes as fast as its slowest link.

The capacity chain from clients through Frontend, History and Matching to PostgreSQL, with load amplifying from 7 calls to about 13 commits per workflow and PostgreSQL as the weakest link in an illustrative example
The capacity chain from clients through Frontend, History and Matching to PostgreSQL, with load amplifying from 7 calls to about 13 commits per workflow and PostgreSQL as the weakest link in an illustrative example

What makes the database the usual weakest link is amplification. For one workflow with one activity (approximate, from reading the source, sync match, cached):

LayerCalls for one workflow
Client → Frontend1 StartWorkflowExecution
Workers → Frontend6 (2 workflow-task polls + 2 completions + 1 activity poll + 1 completion)
Frontend → History / Matching~10
History ↔ Matching3 add-task + 3 record-started
Temporal → main DB≈ 13 commits, ≈ 45–60 statements
Temporal → visibility DB2 upserts, the close touching ~70 indexes
Later, retention≈ 4 commits, ≈ 15 delete statements

The client sends "1 workflow/s"; the database sees ~13 commits/s and ~50 statements/s, plus visibility. 10× the client traffic is 10× the fsyncs, and no Temporal-side scaling changes that ratio. Only changing what the workflow does does.

And this is the counter-intuitive part: when PostgreSQL is the weakest link, adding History (or all-role) hosts can make throughput worse:

  1. More connections on a saturated DB. Each new process brings its own pools. More active backends on a CPU- or lock-bound primary means more context switching and LWLock contention, so less total throughput.
  2. You remove the throttle that was protecting the DB. history.persistenceMaxQPS is per host (9,000 on current versions). Going from 4 to 8 hosts doubles what Temporal will send before throttling itself, so the PersistenceLimit you were hitting disappears and the DB takes the full surge.
  3. Shard movement. Adding a host moves ~1/N of the shards. Each moved shard bumps its range_id, reloads its queue state and starts with a cold cache (around 9 SELECTs per workflow touched, plus history reads), and in-flight writes on the old owner fail with ShardOwnershipLost and retry. All of it lands on the database that was already the bottleneck.
  4. Cache splitting and per-host background load. The working set spreads thinner, hit rates drop, and queue polling, membership heartbeats and namespace refresh all scale with host count.
  5. Matching: hosts beyond the hot queue's partition count do nothing for that queue.

If the database is the limit, more Temporal servers just mean more clients queueing at it together, and it gets slower because more of them crowd it.

What I'd tell myself before go-live

  • For "Temporal is slow": schedule-to-start, slots, activity latency, server latency, in that order. Only the fourth opens the server.
  • Inside the server: RESOURCE_EXHAUSTED by cause, semaphore_latency, persistence_latency, lock_latency, heap/GC.
  • Know whether your worker metrics are delta or cumulative, and write it on the dashboard. Delta means sum_over_time, never rate.
  • Percentiles, not averages. le="1.0", not le="1". Exclude polls from latency panels. Take queue names from config, not labels.
  • Every threshold is a starting point until your own benchmark replaces it. Temporal publishes none.
  • Plan in writes per workflow, not requests. Set maxConns explicitly on every datastore.
  • Plan peak at 60–70 % of the smallest ceiling after N−1, or 50–60 % of the benchmark knee.
  • The paper numbers are a hypothesis; the knee is the fact. Benchmark on the same shard count and disk tier as production.
  • Scale the weakest link only, then re-measure, because it will have moved. If PostgreSQL is full, more Temporal hosts make it worse.

Next in the series: cluster membership, horizontal scaling and which config you can and can't change later.

Keep reading